问:
What is the current Crashlytics version?
This line is automatically generated by Gradle. I want to avoid using +, so I need the most recent stable version of Crashlytics.

```<code>compile "com.crashlytics.android:crashlytics:1.+"
</code>```

答:
The most recent Crashlytics version (for Android) is listed here:
https://fabric.io/kits/android/crashlytics/install

Also, make sure to keep the Gradle plugin up to date by checking this link for the latest version: https://docs.fabric.io/android/changelog.html#fabric-gradle-plugin

Body:
This line is automatically generated by Gradle. I want to avoid using +, so I need the most recent stable version of Crashlytics.

compile "com.crashlytics.android:crashlytics:1.+"


来源: stackexchange

问:
add callback to function(e) Uncaught TypeError: callback is not a function
I have a function which I can not seem to add callback into.

```<code>function ajaxCartFormSubmitted(e) {
  someotherfunctionsinside();
  if (e)...
});

function test () {
    console.log('abraka');

}
</code>```

I try do do

```<code>function ajaxCartFormSubmitted(e, callback) {
  someotherfunctionsinside();
  if (e)...
  if (callback) {
    callback();
  }
});
</code>```

and then calling

```<code>ajaxCartFormSubmitted(test);
</code>```

But I get Uncaught TypeError: callback is not a function. What is the problem? I use jquery 1.3 and cannot upgrade. Thank you.

答:
You have two problems.

1

There are two arguments e and callback.  You have to pass them both if you want the callback since that's your 2nd argument.

Like this: ajaxCartFormSubmitted(event, test)

You can't only pass the 2nd argument and skip the first one.  if you need to do this, write a different function.

2

if it gets called as ajaxCartFormSubmitted(event, 'foo') or ajaxCartFormSubmitted(event, true) then neither 'foo' nor true are functions.  you just need to check that the type of callback is actually a function before you run it

function ajaxCartFormSubmitted(e, callback) {
  someotherfunctionsinside();
  if (e) {
    ...
  }
  if (callback && typeof callback === 'function') {
    callback();
  }
});
```

if that's still not working, add a console.log in there so you can debug it and see what it is.

console.log(typeof callback, callback)

ContentLicense: CC BY-SA 4.0

Body:
I have a function which I can not seem to add callback into.

function ajaxCartFormSubmitted(e) {
  someotherfunctionsinside();
  if (e)...
});

function test () {
    console.log('abraka');

}

I try do do

function ajaxCartFormSubmitted(e, callback) {
  someotherfunctionsinside();
  if (e)...
  if (callback) {
    callback();
  }
});

and then calling

ajaxCartFormSubmitted(test);

But I get Uncaught TypeError: callback is not a function. What is the problem? I use jquery 1.3 and cannot upgrade. Thank you.

Body:
You have two problems.

1

There are two arguments e and callback.  You have to pass them both if you want the callback since that's your 2nd argument.

Like this: ajaxCartFormSubmitted(event, test)

You can't only pass the 2nd argument and skip the first one.  if you need to do this, write a different function.

2

if it gets called as ajaxCartFormSubmitted(event, 'foo') or ajaxCartFormSubmitted(event, true) then neither 'foo' nor true are functions.  you just need to check that the type of callback is actually a function before you run it

function ajaxCartFormSubmitted(e, callback) {
  someotherfunctionsinside();
  if (e) {
    ...
  }
  if (callback && typeof callback === 'function') {
    callback();
  }
});

if that's still not working, add a console.log in there so you can debug it and see what it is.

console.log(typeof callback, callback)


问:
Operator overload for compound dereference and assignment
With C++ operator overloads, it is possible to create a class which emulates a pointer type, so I was experimenting with abstracting FILE read operations with this approach.

```<code>class FilePointer
{
    FILE* file;
    size_t position;
    FilePointer (FILE* file, size_t position)
    {
        this->file = file;
        this->position = position;
    };
    public:
    FilePointer (FILE* file)
    {
        this->file = file;
        this->position = 0;
    };
    FilePointer operator+ (size_t offset)
    {
        return FilePointer(file,position + offset);
    };
    FilePointer operator++ () // Prefix : ++FilePointer
    {
        position++;
        return *this;
    };
    FilePointer operator++ (int) // Postfix : FilePointer++
    {
        FilePointer before = *this;
        position++;
        return before;
    };
    FilePointer operator+= (size_t offset)
    {
        position += offset;
        return *this;
    };
    uint8_t operator* ()
    {
        fseek(file,position,SEEK_SET);
        Uint8 out;
        fread(&out,1,1,file);
        return out;
    };
    uint8_t operator[] (size_t offset)
    {
        return *(*this + offset);
    };

};
</code>```

As can be seen in the above code snippet I was able to find out how to differentiate between the variations of increment operator so that given FilePointer f; f++ and ++f would behave intuitively.

What if I want to use this class for file writes, though? Currently I can grab a byte uint8_t something = f[0]; and it works, but if I want to "set" something, i.e. f[0] = 100;, the overloads as they are will not work for this.

Aside from whether or not this is "good practice" (though feel free to weigh in about that too), is there a way to implement this behavior in operator overloads for a class?

```<code>uint8_t n = f[0];
n++;
f[0] = n;
</code>```

Or, getting even fancier, something like this:

```<code>f[1]++;
</code>```

One way I imagine it could be done is by implementing yet another class which represents a "dereferenced FilePointer", but was curious if it is possible to do with only the overloads in the FilePointer class itself.

答:
One way I imagine it could be done is by implementing yet another class which represents a "dereferenced FilePointer"

That's pretty much the only way to do it. It'd be an object of a "proxy type" that can be converted to uint8_t (possibly implicitly in a case like this, although in general I loath implicit conversions) but can also be assigned to.

It would have to be constructed with a FilePointer*, and be a friend so that it can invoke the appropriate "write" functionality.

It's going to be a bit minging, though, because it could become "dangling" very quickly, writing in the wrong place. You could also store the current position at the time the proxy object is constructed, and seek back to that position iff necessary at the time of writing. But all that seeking is not very performant.

Also reading one byte at a time is sloooooow.

Overall though your attempt is admirable, I would advise against the approach overall. You can at least wrap FILE* in a nice class that does fclose for you if it goes out of scope.

but was curious if it is possible to do with only the overloads in the FilePointer class itself.

I don't think so. Whatever you return from operator[] needs some magic state that can accept a value to "set" and connect to the FILE* machinery. Returning a uint8_t is simply never going to allow you to do that.

Body:
With C++ operator overloads, it is possible to create a class which emulates a pointer type, so I was experimenting with abstracting FILE read operations with this approach.

class FilePointer
{
    FILE* file;
    size_t position;
    FilePointer (FILE* file, size_t position)
    {
        this->file = file;
        this->position = position;
    };
    public:
    FilePointer (FILE* file)
    {
        this->file = file;
        this->position = 0;
    };
    FilePointer operator+ (size_t offset)
    {
        return FilePointer(file,position + offset);
    };
    FilePointer operator++ () // Prefix : ++FilePointer
    {
        position++;
        return *this;
    };
    FilePointer operator++ (int) // Postfix : FilePointer++
    {
        FilePointer before = *this;
        position++;
        return before;
    };
    FilePointer operator+= (size_t offset)
    {
        position += offset;
        return *this;
    };
    uint8_t operator* ()
    {
        fseek(file,position,SEEK_SET);
        Uint8 out;
        fread(&out,1,1,file);
        return out;
    };
    uint8_t operator[] (size_t offset)
    {
        return *(*this + offset);
    };

};

As can be seen in the above code snippet I was able to find out how to differentiate between the variations of increment operator so that given FilePointer f; f++ and ++f would behave intuitively.

What if I want to use this class for file writes, though? Currently I can grab a byte uint8_t something = f[0]; and it works, but if I want to "set" something, i.e. f[0] = 100;, the overloads as they are will not work for this.

Aside from whether or not this is "good practice" (though feel free to weigh in about that too), is there a way to implement this behavior in operator overloads for a class?

uint8_t n = f[0];
n++;
f[0] = n;

Or, getting even fancier, something like this:

f[1]++;

One way I imagine it could be done is by implementing yet another class which represents a "dereferenced FilePointer", but was curious if it is possible to do with only the overloads in the FilePointer class itself.


问:
Error submiting application to orkut sandbox
I am trying to submit an Open Social application to Orkut Sandbox without success. My XML actually looks like this:

```<code><?xml version="1.0" encoding="UTF-8"?>
<Module>
  <ModulePrefs title="App title" author_email="myemal@gmail.com" description="My test application"
        author="My Name" screenshot="http://www.jogodireto.com/280x240/sonic-sky-chase.jpg"
        thumbnail="http://i271.photobucket.com/albums/jj139/Igorstz/120x60-2.png">
        <Require feature="opensocial-data"/>
  </ModulePrefs>

  <Content view="canvas"
    href="http://gadget-doc-examples.googlecode.com/svn/trunk/opensocial-09/mycontent.html">
  </Content>
  <Content view="home">
    <![CDATA[
      Hello, home view!
    ]]>
  </Content>
</Module>
</code>```

And when I try to submit it, I get the following error:

Submit failed. Please ensure that the
  gadget has title, description, author
  email, thumbnail url, and screenshot
  url defined, that you are an owner of
  the gadget, and that the gadget is not
  already submitted to any google
  property.

Is there anything wrong with my xml? It is available in my website if I type in the browser its URL. Looks like I have met these requirements present in the error message.

答: I got it to work, by using this URL to submit the application http://sandbox.orkut.com/Main#Developer and not the one I used before: The "Submit your application" link in http://sandbox.orkut.com/Main#AppDirectory</p>

ContentLicense: CC BY-SA 3.0

Body:
I am trying to submit an Open Social application to Orkut Sandbox without success. My XML actually looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<Module>
  <ModulePrefs title="App title" author_email="myemal@gmail.com" description="My test application"
        author="My Name" screenshot="http://www.jogodireto.com/280x240/sonic-sky-chase.jpg"
        thumbnail="http://i271.photobucket.com/albums/jj139/Igorstz/120x60-2.png">
        <Require feature="opensocial-data"/>
  </ModulePrefs>

  <Content view="canvas"
    href="http://gadget-doc-examples.googlecode.com/svn/trunk/opensocial-09/mycontent.html">
  </Content>
  <Content view="home">
    <![CDATA[
      Hello, home view!
    ]]>
  </Content>
</Module>

And when I try to submit it, I get the following error:

Submit failed. Please ensure that the
  gadget has title, description, author
  email, thumbnail url, and screenshot
  url defined, that you are an owner of
  the gadget, and that the gadget is not
  already submitted to any google
  property.

Is there anything wrong with my xml? It is available in my website if I type in the browser its URL. Looks like I have met these requirements present in the error message.


问:
New error when fitting a ggsn scale-bar to ggmap object
I've recently re-ran my ggmap code and the ggsn scale bar for the ggmap no longer  works. When I remove the scale bar function I can successfully make the ggmap. It was working a few weeks ago, but it now throws the following error:

Error in st_geometry(legend2) <- NULL :    could not find function
  "st_geometry<-"

This is a new error and I'm quite confused by it, since I don't directly call that function.

Below is my ggmap code chunk and dependencies/libraries I'm using

```<code>if(!requireNamespace("devtools"))install.packages("devtools")
devtools::install_github("dkahle/ggmap", ref = "tidyup")
library(ggplot2)
register_google(key = "REDACTED")

devtools::install_github('oswaldosantos/ggsn')

MAD_map <- ggmap(get_map(location=c(-89.410087,43.074243),
                         zoom=14, scale=4, source = "stamen",
                         maptype = "terrain")) +
  ggsn::scalebar(x.min = -89.387949, x.max = -89.387952,
                 y.min =43.090330, y.max = 43.0919, location="topright",
                 dist = .5, transform = TRUE, dist_unit = "km",
                 model = "GRS80", height =.2, st.size = 3, st.dist = .5) +
  geom_point(data = Location_points_DF,
             aes(x=long,y=lat,color= dot.color, size=5),
             shape = c(12, shape.list[c(1,3,5,7,8)]))  +
  scale_shape_manual(values= c(12, shape.list[c(1,3,5,7,8)]))+
  scale_color_manual(values = Location_points_DF$dot.color )  +
  annotate("text", x = Location_points_DF$long, y = Location_points_DF$lat,
           label = c("3","5","6","1","4","2"), size = 5) +
  theme(legend.position="none",
        axis.title.x=element_blank(),
        axis.text.x=element_blank(),
        axis.ticks.x=element_blank(),
        axis.title.y=element_blank(),
        axis.text.y=element_blank(),
        axis.ticks.y=element_blank() )
</code>```

答:
This problem was fixed for me with version 0.5.2: https://github.com/oswaldosantos/ggsn/issues/45

Try out a recent version and see if you are still having an issue: https://github.com/oswaldosantos/ggsn

Ben

Body:
I've recently re-ran my ggmap code and the ggsn scale bar for the ggmap no longer  works. When I remove the scale bar function I can successfully make the ggmap. It was working a few weeks ago, but it now throws the following error:

Error in st_geometry(legend2) <- NULL :    could not find function
  "st_geometry<-"

This is a new error and I'm quite confused by it, since I don't directly call that function.

Below is my ggmap code chunk and dependencies/libraries I'm using

if(!requireNamespace("devtools"))install.packages("devtools")
devtools::install_github("dkahle/ggmap", ref = "tidyup")
library(ggplot2)
register_google(key = "REDACTED")

devtools::install_github('oswaldosantos/ggsn')

MAD_map <- ggmap(get_map(location=c(-89.410087,43.074243),
                         zoom=14, scale=4, source = "stamen",
                         maptype = "terrain")) +
  ggsn::scalebar(x.min = -89.387949, x.max = -89.387952,
                 y.min =43.090330, y.max = 43.0919, location="topright",
                 dist = .5, transform = TRUE, dist_unit = "km",
                 model = "GRS80", height =.2, st.size = 3, st.dist = .5) +
  geom_point(data = Location_points_DF,
             aes(x=long,y=lat,color= dot.color, size=5),
             shape = c(12, shape.list[c(1,3,5,7,8)]))  +
  scale_shape_manual(values= c(12, shape.list[c(1,3,5,7,8)]))+
  scale_color_manual(values = Location_points_DF$dot.color )  +
  annotate("text", x = Location_points_DF$long, y = Location_points_DF$lat,
           label = c("3","5","6","1","4","2"), size = 5) +
  theme(legend.position="none",
        axis.title.x=element_blank(),
        axis.text.x=element_blank(),
        axis.ticks.x=element_blank(),
        axis.title.y=element_blank(),
        axis.text.y=element_blank(),
        axis.ticks.y=element_blank() )


问:
I need get output using excel formulas
I want to get output on below table using Excel function.

I have tried Index match but it only helps to get output for first value, while I have duplicate values.

Database

```<code>  Date      Product Name
01-01-2016  60      A
01-01-2016  54      B
01-01-2016  40      C
01-01-2016  60      D
01-03-2016  47      A
01-03-2016  39      B
01-03-2016  46      C
01-03-2016  42      D
01-02-2016  37      A
01-02-2016  53      B
01-02-2016  25      C
01-02-2016  46      D
01-04-2016  49      A
01-04-2016  47      B
01-04-2016  46      C
01-04-2016  27      D
</code>```

Need a help to fill the below table using Excel formula

Kindly find the below sample output

enter image description here

答:
Using your posted "database" graph (the first one) as a reference and assuming that it starts in cell A1, I put your "required output" data in F1:J5. "A" starts in F2, F1 is blank and the dates are in cells G1:J1.

In G2 put this Array formula and then fill down to the last letter "D": = INDEX($B$1:$B$17,SMALL(IF($C$2:$C$17=$F2,ROW($C$2:$C$17),""),COLUMN()-6)). Then fill across. The "COLUMN()-6" in the Small function assumes that you put your "required out" data in the range F1:J5; otherwise, you will need to change it manually.

Title: I need get output using excel formulas

Body:
I want to get output on below table using Excel function.

I have tried Index match but it only helps to get output for first value, while I have duplicate values.

Database

  Date      Product Name
01-01-2016  60      A
01-01-2016  54      B
01-01-2016  40      C
01-01-2016  60      D
01-03-2016  47      A
01-03-2016  39      B
01-03-2016  46      C
01-03-2016  42      D
01-02-2016  37      A
01-02-2016  53      B
01-02-2016  25      C
01-02-2016  46      D
01-04-2016  49      A
01-04-2016  47      B
01-04-2016  46      C
01-04-2016  27      D

Need a help to fill the below table using Excel formula

Kindly find the below sample output

enter image description here


问:
How do I make an iframe height 100% of it's content not based on aspect ratio using css?
I have an iframe containing a form that is dynamic. Lot of guides show how to make it change height based on a static aspect ratio but the form content changes so that doesn't work.

Is there a way using css to make an iframe or an iframe container 100% the height of it's content?

here's what I currently have but no changes of the height make a difference except static pixel values:

```<code><style>
    .iframe_form_container {
        overflow: hidden;
        position: relative;
        height: 100px;
    }
    .iframe_form {
        position: absolute;
        top: 0;
        left: 0;
        width: 100%;
        height: 100%;
        border: 0px;
    }
</style>
<div class="iframe_form_container">
    <iframe src="*url*" width="100%" id="iframe_form" class="iframe_form" type="text/html" allowTransparency="true"></iframe>
</div>
</code>```

答:
From what I can find the only solution is javascript. It's not a great user experience though.

This guys built an iFrame resizing program that seems to work


问:
Set image inside png transparency
As the title suggests I have a PNG image that has some transparency. I'd like to fill that transparency with a second image (which is currently a JPEG, but it's not a problem to convert it to a PNG).

Every post I have found searching on the Internet was about the "inverse" problem (from an image with a background to an image with transparency), so obviously it did not work out for my situation; for example, I tried

```<code>convert -flatten myimg.png myimg.png
</code>```

(taken from here) and

```<code>convert myimg1.png  -transparent white myimg.png
</code>```

(taken from here).

答:
In ImageMagick 6, if the two images are the same size, then you can just flatten the transparent image over the background image.

Background (lena.jpg):

Transparent (logo_crop_trans.png):

```<code>convert lena.jpg logo_crop_trans.png -flatten lena_logo.jpg
</code>```

If using ImageMagick 7, then change convert to magick.

If you want to anti-alias the transparent image so that it is not so jagged, then use some blur to smooth the outline (Unix syntax):

```<code>convert lena.jpg \( logo_crop_trans.png -channel a -blur 0x1 -level 50x100% +channel \) -compose over -composite lena_logo2.jpg
</code>```

If on Windows remove the \ before the parentheses.

Title: Set image inside png transparency

Body:
As the title suggests I have a PNG image that has some transparency. I'd like to fill that transparency with a second image (which is currently a JPEG, but it's not a problem to convert it to a PNG).

Every post I have found searching on the Internet was about the "inverse" problem (from an image with a background to an image with transparency), so obviously it did not work out for my situation; for example, I tried

convert -flatten myimg.png myimg.png

(taken from here) and

convert myimg1.png  -transparent white myimg.png

(taken from here).

Body:
In ImageMagick 6, if the two images are the same size, then you can just flatten the transparent image over the background image.

Background (lena.jpg):

Transparent (logo_crop_trans.png):

convert lena.jpg logo_crop_trans.png -flatten lena_logo.jpg

If using ImageMagick 7, then change convert to magick.

If you want to anti-alias the transparent image so that it is not so jagged, then use some blur to smooth the outline (Unix syntax):

convert lena.jpg \( logo_crop_trans.png -channel a -blur 0x1 -level 50x100% +channel \) -compose over -composite lena_logo2.jpg

If on Windows remove the \ before the parentheses.


问:
Enclosing a block of text in quotation marks (or any other couple of chars) in a csv
I have a .csv file, and I need to do what follows: for each line, starting from the end of the line, the text that occurs after a certain number of occurrences of a specified character should be enclosed in quotation marks.

I'll try and make myself clearer with an example. Consider the following line of a csv:

```<code>gentlemen, this is a block of text. Thanks!,1,0,0,1
</code>```

I want to enclose in quotation marks the slice of the line which occurs, starting from the end, after the fourth occurrence of a comma. So, it should become:

```<code>"gentlemen, this is a block of text. Thanks!",1,0,0,1
</code>```

I'd like to do it in bash, pure python, or pandas.

答:
With sed:

```<code>$ sed -E 's/(.*)((,.*){4})/"\1"\2/' <<< 'gentlemen, this is a block of text. Thanks!,1,0,0,1'
"gentlemen, this is a block of text. Thanks!",1,0,0,1
</code>```

This uses two capture groups. The important part is (,.*){4}: these are four instances of a comma followed by anything. Because the first group matches greedily, the second group will match the last four commas on the line.

The substitution then puts double quotes around the first group and prints the second group unmodified.

Body:
I have a .csv file, and I need to do what follows: for each line, starting from the end of the line, the text that occurs after a certain number of occurrences of a specified character should be enclosed in quotation marks.

I'll try and make myself clearer with an example. Consider the following line of a csv:

gentlemen, this is a block of text. Thanks!,1,0,0,1

I want to enclose in quotation marks the slice of the line which occurs, starting from the end, after the fourth occurrence of a comma. So, it should become:

"gentlemen, this is a block of text. Thanks!",1,0,0,1

I'd like to do it in bash, pure python, or pandas.

Body:
With sed:

$ sed -E 's/(.*)((,.*){4})/"\1"\2/' <<< 'gentlemen, this is a block of text. Thanks!,1,0,0,1'
"gentlemen, this is a block of text. Thanks!",1,0,0,1

This uses two capture groups. The important part is (,.*){4}: these are four instances of a comma followed by anything. Because the first group matches greedily, the second group will match the last four commas on the line.

The substitution then puts double quotes around the first group and prints the second group unmodified.


问:
Visible redirection nginx
I have a host that looks like this http://loc.www.site.com:42080 I want when the user access to http://loc.www.site.com:42080 it  should  be redirected to http://loc.www.site.com:42080/fr.

This a part of nginx.conf:

```<code>location / {

    location ~ ^/app\.php(/|$) {
        include ez_params.d/ez_fastcgi_params;
        fastcgi_pass engine:9000;
        fastcgi_param SYMFONY_ENV dev;
        fastcgi_param RUNNING_ENV DOCKER;
    }
}
</code>```

答:
This may work</p>

```<code>server {
    listen 42080;
    listen [::]:42080;
    hostname loc.www.site.com;

    location ^/en {
      return 301 http://$http_host$request_uri;
    }
}
</code>```

Title: Visible redirection nginx

Body:
I have a host that looks like this http://loc.www.site.com:42080 I want when the user access to http://loc.www.site.com:42080 it  should  be redirected to http://loc.www.site.com:42080/fr.

This a part of nginx.conf:

location / {

    location ~ ^/app\.php(/|$) {
        include ez_params.d/ez_fastcgi_params;
        fastcgi_pass engine:9000;
        fastcgi_param SYMFONY_ENV dev;
        fastcgi_param RUNNING_ENV DOCKER;
    }
}

Body:
This may work

server {
    listen 42080;
    listen [::]:42080;
    hostname loc.www.site.com;

    location ^/en {
      return 301 http://$http_host$request_uri;
    }
}


问:
Subtract mean of columns grouped by one column
I have a Pandas DataFrame which looks as follows:

```<code>data = {'user_id':[1, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 3],
        'time':[10, 12, 11, 20, 40, 41, 42, 100, 60, 22, 0, 3],
        'height':[1.23, 1.1, 0.5, 10.3, 3.33, 4.55, 2.22, 2.21, 2.25, 7.75, 9.2, 5.5],
        'width':[3, 4, 6, 2, 4, 8, 9, 0, 6, 6, 6, 4]}
df = pd.DataFrame(data)
</code>```

Now, I would like to subtract from every column (except the user_id column) the mean value grouped by user_id. That means for user_id 1 the mean value for time is 13.25 ((10 + 12 + 11 + 20) / 4), so 13.25 is subtracted from 10, 12, 11 and 20 leading to

```<code>'time':[-3.25, -1.25, -2.25, 6.75, 40, 41, 42, 100, 60, 22, 0, 3]
</code>```

This I would like to do for all columns and all users. How can this be done?

答:
You can use groupby.transform to create an array same shape as your dataframe. Then substract that from your time column:

```<code>m = df.groupby('user_id')['time'].transform('mean')

df['time'].add(-m)
</code>```

```<code>0     -3.25
1     -1.25
2     -2.25
3      6.75
4     -1.00
5      0.00
6      1.00
7     63.00
8     23.00
9    -15.00
10   -37.00
11   -34.00
Name: time, dtype: float64
</code>```

Output of m results in:

```<code>print(m)

0     13.25
1     13.25
2     13.25
3     13.25
4     41.00
5     41.00
6     41.00
7     37.00
8     37.00
9     37.00
10    37.00
11    37.00
Name: time, dtype: float64
</code>```

Body:
I have a Pandas DataFrame which looks as follows:

data = {'user_id':[1, 1, 1, 1, 2, 2, 2, 3, 3, 3, 3, 3],
        'time':[10, 12, 11, 20, 40, 41, 42, 100, 60, 22, 0, 3],
        'height':[1.23, 1.1, 0.5, 10.3, 3.33, 4.55, 2.22, 2.21, 2.25, 7.75, 9.2, 5.5],
        'width':[3, 4, 6, 2, 4, 8, 9, 0, 6, 6, 6, 4]}
df = pd.DataFrame(data)

Now, I would like to subtract from every column (except the user_id column) the mean value grouped by user_id. That means for user_id 1 the mean value for time is 13.25 ((10 + 12 + 11 + 20) / 4), so 13.25 is subtracted from 10, 12, 11 and 20 leading to

'time':[-3.25, -1.25, -2.25, 6.75, 40, 41, 42, 100, 60, 22, 0, 3]

This I would like to do for all columns and all users. How can this be done?

Body:
You can use groupby.transform to create an array same shape as your dataframe. Then substract that from your time column:

m = df.groupby('user_id')['time'].transform('mean')

df['time'].add(-m)

0     -3.25
1     -1.25
2     -2.25
3      6.75
4     -1.00
5      0.00
6      1.00
7     63.00
8     23.00
9    -15.00
10   -37.00
11   -34.00
Name: time, dtype: float64

Output of m results in:

print(m)

0     13.25
1     13.25
2     13.25
3     13.25
4     41.00
5     41.00
6     41.00
7     37.00
8     37.00
9     37.00
10    37.00
11    37.00
Name: time, dtype: float64


问:
How to select only few columns from pandas dataframe
I have one json file about ansible inventory where I need to select few columns as dataframe and send email notification.

The following is code I tried:

```<code>import json
import pandas as pd
from pandas.io.json import json_normalize
with open('d:/facts.json') as f:
    d = json.load(f)
mydata = json_normalize(d['ansible_facts'])
mydata.head(1)`
</code>```

Its printing entire records (actually each json will have only one record), but I need to show/select/display only two columns from dataframe. can some one suggest please how to view dataframe with selected columns

Update 1:
I am able to generate required columns now,but only certain column working, but when i mention certain columns, then its saying "not in index"
And also can i have own column custom header lable while printing ?
Working

```<code>import json
import pandas as pd
from pandas.io.json import json_normalize
with open('d:/facts.json') as f:
    d = json.load(f)
mydata = json_normalize(d['ansible_facts'])
mydata.columns = mydata.columns.to_series().apply(lambda x: x.strip())
df1=mydata[['ansible_architecture','ansible_distribution']]
</code>```

But when i mention column as hostname,ansible_distribution, its saying not in index.
Not working

```<code>import json
import pandas as pd
from pandas.io.json import json_normalize
with open('d:/facts.json') as f:
    d = json.load(f)
mydata = json_normalize(d['ansible_facts'])
mydata.columns = mydata.columns.to_series().apply(lambda x: x.strip())
df1=mydata[['hostname','ansible_distribution']]
</code>```

Error:
KeyError: "['hostname'] not in index"

Update2:

Now i am able to fix that issue with below, but I need custom label in output, how to do that

```<code>`import json
import pandas as pd
from pandas.io.json import json_normalize
with open('d:/facts.json') as f:
    d = json.load(f)
mydata = json_normalize(d['ansible_facts'])
mydata.columns = mydata.columns.to_series().apply(lambda x: x.strip())
df1=mydata[['ansible_env.HOSTNAME','ansible_distribution']]`
</code>```

But i need to have custom columname lable in final output like Host,OSversion for above column, how can i do that?

UPDATE 3: now trying to rename columns name before I print it, tried following code but giving error like key error not in index

```<code>import json
import pandas as pd
from tabulate import tabulate
from pandas.io.json import json_normalize
with open('/home/cloud-user/facts.json') as f:
    d = json.load(f)
mydata = json_normalize(d['ansible_facts'])
mydata.columns = mydata.columns.to_series().apply(lambda x: x.strip())

mydata=mydata.rename(columns={"ansible_env.HOSTNAME": "HOSTNAME", "ansible_disrribution": "OSType"})
df1=mydata[['HOSTNAME','OSType']]
print(tabulate(df1, headers='keys', tablefmt='psql'))

Traceback (most recent call last):
  File "ab7.py", line 21, in <module>
    df1=mydata[['HOSTNAME','OSType']]
  File "/usr/lib64/python2.7/site-packages/pandas/core/frame.py", line 2682, in __getitem__
    return self._getitem_array(key)
  File "/usr/lib64/python2.7/site-packages/pandas/core/frame.py", line 2726, in _getitem_array
    indexer = self.loc._convert_to_indexer(key, axis=1)
  File "/usr/lib64/python2.7/site-packages/pandas/core/indexing.py", line 1327, in _convert_to_indexer
    .format(mask=objarr[mask]))
KeyError: "['HOSTNAME' 'OSType'] not in index"
</code>```

But if i dont rename, it working perfectly, But i need most readable column lable. any suggestion please.
without rename stuff code get works and output as below on console

```<code>+----+------------------------+------------------------+
|    | ansible_env.HOSTNAME   | ansible_distribution   |
|----+------------------------+------------------------|
|  0 | ip-xx-xx-xx-xx         | SLES                   |
+----+------------------------+------------------------+
</code>```

Now instead anisble_env.HOSTNAME --> i need lable as HOSTNAME , instead of ansible_distribution --> I need OSType any suggestion please

Update 4:

I fixed issue with below

```<code>df.rename(columns={'ansible_hostname':'HOSTNAME','ansible_distribution':'OS Version','ansible_ip_addresses':'Private IP','ansible_windows_domain':'FQDN'},inplace=True)
</code>```

答:
Select multiple columns as a DataFrame by passing a list to it:

```<code>df[['col_name1', 'col_name2']]
</code>```

For more information try this link:
https://medium.com/dunder-data/selecting-subsets-of-data-in-pandas-6fcd0170be9c

Body:
I have one json file about ansible inventory where I need to select few columns as dataframe and send email notification.

The following is code I tried:

import json
import pandas as pd
from pandas.io.json import json_normalize
with open('d:/facts.json') as f:
    d = json.load(f)
mydata = json_normalize(d['ansible_facts'])
mydata.head(1)`

Its printing entire records (actually each json will have only one record), but I need to show/select/display only two columns from dataframe. can some one suggest please how to view dataframe with selected columns

Update 1:
I am able to generate required columns now,but only certain column working, but when i mention certain columns, then its saying "not in index"
And also can i have own column custom header lable while printing ?
Working

import json
import pandas as pd
from pandas.io.json import json_normalize
with open('d:/facts.json') as f:
    d = json.load(f)
mydata = json_normalize(d['ansible_facts'])
mydata.columns = mydata.columns.to_series().apply(lambda x: x.strip())
df1=mydata[['ansible_architecture','ansible_distribution']]

But when i mention column as hostname,ansible_distribution, its saying not in index.
Not working

import json
import pandas as pd
from pandas.io.json import json_normalize
with open('d:/facts.json') as f:
    d = json.load(f)
mydata = json_normalize(d['ansible_facts'])
mydata.columns = mydata.columns.to_series().apply(lambda x: x.strip())
df1=mydata[['hostname','ansible_distribution']]

Error:
KeyError: "['hostname'] not in index"

Update2:

Now i am able to fix that issue with below, but I need custom label in output, how to do that

`import json
import pandas as pd
from pandas.io.json import json_normalize
with open('d:/facts.json') as f:
    d = json.load(f)
mydata = json_normalize(d['ansible_facts'])
mydata.columns = mydata.columns.to_series().apply(lambda x: x.strip())
df1=mydata[['ansible_env.HOSTNAME','ansible_distribution']]`

But i need to have custom columname lable in final output like Host,OSversion for above column, how can i do that?

UPDATE 3: now trying to rename columns name before I print it, tried following code but giving error like key error not in index

import json
import pandas as pd
from tabulate import tabulate
from pandas.io.json import json_normalize
with open('/home/cloud-user/facts.json') as f:
    d = json.load(f)
mydata = json_normalize(d['ansible_facts'])
mydata.columns = mydata.columns.to_series().apply(lambda x: x.strip())

mydata=mydata.rename(columns={"ansible_env.HOSTNAME": "HOSTNAME", "ansible_disrribution": "OSType"})
df1=mydata[['HOSTNAME','OSType']]
print(tabulate(df1, headers='keys', tablefmt='psql'))

Traceback (most recent call last):
  File "ab7.py", line 21, in <module>
    df1=mydata[['HOSTNAME','OSType']]
  File "/usr/lib64/python2.7/site-packages/pandas/core/frame.py", line 2682, in __getitem__
    return self._getitem_array(key)
  File "/usr/lib64/python2.7/site-packages/pandas/core/frame.py", line 2726, in _getitem_array
    indexer = self.loc._convert_to_indexer(key, axis=1)
  File "/usr/lib64/python2.7/site-packages/pandas/core/indexing.py", line 1327, in _convert_to_indexer
    .format(mask=objarr[mask]))
KeyError: "['HOSTNAME' 'OSType'] not in index"

But if i dont rename, it working perfectly, But i need most readable column lable. any suggestion please.
without rename stuff code get works and output as below on console

+----+------------------------+------------------------+
|    | ansible_env.HOSTNAME   | ansible_distribution   |
|----+------------------------+------------------------|
|  0 | ip-xx-xx-xx-xx         | SLES                   |
+----+------------------------+------------------------+

Now instead anisble_env.HOSTNAME --> i need lable as HOSTNAME , instead of ansible_distribution --> I need OSType any suggestion please

Update 4:

I fixed issue with below

df.rename(columns={'ansible_hostname':'HOSTNAME','ansible_distribution':'OS Version','ansible_ip_addresses':'Private IP','ansible_windows_domain':'FQDN'},inplace=True)

Body:
Select multiple columns as a DataFrame by passing a list to it:

df[['col_name1', 'col_name2']]

For more information try this link:
https://medium.com/dunder-data/selecting-subsets-of-data-in-pandas-6fcd0170be9c


问:
Reactive ("Rx") way to cache and update data
I have a routine which reads data from a db and publishes that data via an IObservable.

After the data has been published, I want to update all those rows which have just been published to stop them being published again.

I am not sure of the "reactive" way of doing this. (Each time I revisit Rx I seem to have this problem!)

I think I need to do 2 things

1) Cache the data as it is published since it'll have the IDs I need to update subsequently - I had wondered whether to use a subject to cache the data being published or whether to have some other routine which wraps what I currently have, subscribing to it, caching it and then republishing

2) Update the data after it has been published. I am really not sure how to build that into the pipeline at all!

I've built this up from various things I've found on line (esp. Lee Campbell for the db polling - thx Lee!) but the other bits I've added on are mine and I may have got it badly wrong. I'm open to suggestions if some parts would be better implemented non-reactively. For example, I've made the db update routine observable, but I don't know it that is really necessary - or whether it is just easier to include it in the pipeline if it is implemented that way..

Here's the relevant bits of code ...

```<code>private IObservable<INotification> Poller() =>
    Observable
        .Timer(_pollingPeriod, _scheduler)
        .SelectMany(_ => NewNotifications(_cx))
        .Timeout(_pollingPeriod + _queryTimeout, Observable.Return(TimeOut.Notification()), _scheduler)
        .Catch<INotification, Exception>(err => Observable.Return(Error.Notification(err)))
        .Repeat();

private IObservable<INotification> NewNotifications(string cx)
{
    try
    {
        return SqlRead<INotification>(cx, NewNotificationsSql(),sdr => EventBuilder(sdr), Empty.Notification());
    }
    catch (Exception ex)
    {
        throw ex;
    }
}

internal static IObservable<T> SqlRead<T>(string cx, string sql, Func<SqlDataReader, T> mapper, T noRows) =>
    Observable.Create<T>(o =>
    {
        using (var conn = new SqlConnection(cx))
        {
            conn.Open();
            using (var cmd = new SqlCommand(sql, conn))
            {
                using (var rdr = cmd.ExecuteReader())
                {
                    if (!rdr.HasRows)
                    {
                        o.OnNext(noRows);
                    }
                    else
                    {
                        while (rdr.Read())
                        {
                            o.OnNext(mapper(rdr));
                        }
                    }
                }
            }
        }
        o.OnCompleted();
        return Disposable.Empty;
    });

internal static IObservable<int> SqlWrite(string cx, string sql) =>
    Observable.Create<int>(o =>
    {
        using (var conn = new SqlConnection(cx))
        {
            conn.Open();
            using (var cmd = new SqlCommand(sql, conn))
            {
                o.OnNext(cmd.ExecuteNonQuery());
            }
        }
        o.OnCompleted();
        return Disposable.Empty;
    });
</code>```

答:
Lets assume you have a UI with list of current notifications

```<code>public class NotificationListViewModel
{
    ObservableCollection<INotification> Items {get;}
}
</code>```

To maintain this collection you need to know how notifications are changing.
Lets have a class to show changes

```<code>enum ChangeType
{
   Add,
   Remove,
   Update
}

class Change<T>
{
    ChangeType Type {get;}
    T Value {get;}
}
</code>```

Now you can expose notifications via changes

```<code>INotificationProvider
{
   public IObservable<Change<INotification>> Notifications {get;}
}
</code>```

Lets add method Update in INotification for convinience

```<code>public class NotificationListViewModel
{
    public NotificationListViewModel(INotificationProvider provider)
    {
        provider.Notifications.Subscribe(change =>
        {
          if(change.Type == ChangeType.Add)
          {
            Items.Add(change.Value);
          }
          if(change.Type == ChangeType.Update)
          {
            Items.First(x => x.Id = change.Value.Id).Update(change.Value);
          }
          if(change.Type == ChangeType.Remove)
          {
            Items.Remove(change.Value);
          }
        });
    }
}
</code>```

And to understand the type of change you receive, you need to maintain the list of existing notifications where you read them from database.

All this code i wrote to show the example of "how can you think about this things".
There is a great library DynamicData where all this ideas are implemented in convinient and optimised way.

Body:
I have a routine which reads data from a db and publishes that data via an IObservable.

After the data has been published, I want to update all those rows which have just been published to stop them being published again.

I am not sure of the "reactive" way of doing this. (Each time I revisit Rx I seem to have this problem!)

I think I need to do 2 things

1) Cache the data as it is published since it'll have the IDs I need to update subsequently - I had wondered whether to use a subject to cache the data being published or whether to have some other routine which wraps what I currently have, subscribing to it, caching it and then republishing

2) Update the data after it has been published. I am really not sure how to build that into the pipeline at all!

I've built this up from various things I've found on line (esp. Lee Campbell for the db polling - thx Lee!) but the other bits I've added on are mine and I may have got it badly wrong. I'm open to suggestions if some parts would be better implemented non-reactively. For example, I've made the db update routine observable, but I don't know it that is really necessary - or whether it is just easier to include it in the pipeline if it is implemented that way..

Here's the relevant bits of code ...

private IObservable<INotification> Poller() =>
    Observable
        .Timer(_pollingPeriod, _scheduler)
        .SelectMany(_ => NewNotifications(_cx))
        .Timeout(_pollingPeriod + _queryTimeout, Observable.Return(TimeOut.Notification()), _scheduler)
        .Catch<INotification, Exception>(err => Observable.Return(Error.Notification(err)))
        .Repeat();

private IObservable<INotification> NewNotifications(string cx)
{
    try
    {
        return SqlRead<INotification>(cx, NewNotificationsSql(),sdr => EventBuilder(sdr), Empty.Notification());
    }
    catch (Exception ex)
    {
        throw ex;
    }
}

internal static IObservable<T> SqlRead<T>(string cx, string sql, Func<SqlDataReader, T> mapper, T noRows) =>
    Observable.Create<T>(o =>
    {
        using (var conn = new SqlConnection(cx))
        {
            conn.Open();
            using (var cmd = new SqlCommand(sql, conn))
            {
                using (var rdr = cmd.ExecuteReader())
                {
                    if (!rdr.HasRows)
                    {
                        o.OnNext(noRows);
                    }
                    else
                    {
                        while (rdr.Read())
                        {
                            o.OnNext(mapper(rdr));
                        }
                    }
                }
            }
        }
        o.OnCompleted();
        return Disposable.Empty;
    });

internal static IObservable<int> SqlWrite(string cx, string sql) =>
    Observable.Create<int>(o =>
    {
        using (var conn = new SqlConnection(cx))
        {
            conn.Open();
            using (var cmd = new SqlCommand(sql, conn))
            {
                o.OnNext(cmd.ExecuteNonQuery());
            }
        }
        o.OnCompleted();
        return Disposable.Empty;
    });

Body:
Lets assume you have a UI with list of current notifications

public class NotificationListViewModel
{
    ObservableCollection<INotification> Items {get;}
}

To maintain this collection you need to know how notifications are changing.
Lets have a class to show changes

enum ChangeType
{
   Add,
   Remove,
   Update
}

class Change<T>
{
    ChangeType Type {get;}
    T Value {get;}
}

Now you can expose notifications via changes

INotificationProvider
{
   public IObservable<Change<INotification>> Notifications {get;}
}

Lets add method Update in INotification for convinience

public class NotificationListViewModel
{
    public NotificationListViewModel(INotificationProvider provider)
    {
        provider.Notifications.Subscribe(change =>
        {
          if(change.Type == ChangeType.Add)
          {
            Items.Add(change.Value);
          }
          if(change.Type == ChangeType.Update)
          {
            Items.First(x => x.Id = change.Value.Id).Update(change.Value);
          }
          if(change.Type == ChangeType.Remove)
          {
            Items.Remove(change.Value);
          }
        });
    }
}

And to understand the type of change you receive, you need to maintain the list of existing notifications where you read them from database.

All this code i wrote to show the example of "how can you think about this things".
There is a great library DynamicData where all this ideas are implemented in convinient and optimised way.


问:
Ordering of query array parameters with Jersey
I am writing REST service, which have array query parameter. RESTful Web Services framework is Jersey 2.23 (but I can upgrade if necessary). URL looks like below:

```<code>http://localhost:8080/restservices/data?sort=id&sort=name&sort=description
</code>```

Here "sort" is array query parameter. The ordering of the parameter values is important because they are used for sorting. This parameter is defined in code like this:

```<code>@QueryParam("sort") final List<String> sort
</code>```

The code works as expected, but I am not sure if the ordering is always preserved.

Can you please help me to understand is it OK to rely on the ordering of the values, or there can be situations when ordering in the sort Java list is different from the ordering in URL?

答:
I believe it is ok. As List is ordered collection.  By the way, you can test this easily as well. Just tried several times.

MKYONG offer a very good example.

Body:
I am writing REST service, which have array query parameter. RESTful Web Services framework is Jersey 2.23 (but I can upgrade if necessary). URL looks like below:

http://localhost:8080/restservices/data?sort=id&sort=name&sort=description

Here "sort" is array query parameter. The ordering of the parameter values is important because they are used for sorting. This parameter is defined in code like this:

@QueryParam("sort") final List<String> sort

The code works as expected, but I am not sure if the ordering is always preserved.

Can you please help me to understand is it OK to rely on the ordering of the values, or there can be situations when ordering in the sort Java list is different from the ordering in URL?


问:
Python regex doesn't match what other regex tools do match
I have following code:

```<code>currencies = ['USD', 'GBP', 'EUR', 'JPY', 'CHF', 'SEK', 'DKK', 'NOK', 'SGD', 'HKD', 'AUD', 'TWD', 'NZD', 'CNY', 'KRW', 'INR', 'CAD', 'VEF', 'EGP', 'THB', 'IDR', 'PKR', 'MYR', 'PHP', 'MXN', 'VND', 'CZK', 'HUF', 'PLN', 'TRY', 'ZAR', 'ILS', 'ARS', 'CLP', 'BRL', 'RUB', 'QAR', 'AED', 'COP', 'PEN', 'CNH', 'KWD', 'SAR']
exclusive_regexp = ".*\/" + ".*|.*\/".join(currencies) + ".*"
searching_regexp = "^(?! (" + exclusive_regexp + ")$)(.*\/.*)$"
searching_regexp = re.compile(searching_regexp)

with open('raw.txt', 'r') as unprocessed_ticks:
    print(re.findall(searching_regexp, unprocessed_ticks.read()))
</code>```

It should find all strings that can be matched with produced regex.

I probed generated regex with regex101.com online tool. it works with python flavor. But in real code it doesn't match anything:
https://regex101.com/r/70uiuE/7
Why it can happen?

答:
A couple of points:

There is no need to escape / in Python (and any other) string patterns (/ should only be escaped when the regex is set via a regex literal with slashes used as regex delimiters)

You are overusing .* pattern, you should minimize the number of alternatives that match at the same location in string

For ^ and $ to match start and end of a line, you must pass the re.M or re.MULTILINE flag to the regex.

I suggest changing the regexps to

```<code>exclusive_regexp = r"/(?:{})".format("|".join(currencies))
searching_regexp = re.compile(r"^(?! .*" + exclusive_regexp + ")[^/\n]*/.*", re.M)
</code>```

and then

```<code>print(searching_regexp.findall(unprocessed_ticks.read()))
</code>```

The exclusive_regexp regex will look tidier now (/(?:USD|GBP|EUR|JPY|CHF|SEK|DKK|NOK|SGD|HKD|AUD|TWD|NZD|CNY|KRW|INR|CAD|VEF|EGP|THB|IDR|PKR|MYR|PHP|MXN|VND|CZK|HUF|PLN|TRY|ZAR|ILS|ARS|CLP|BRL|RUB|QAR|AED|COP|PEN|CNH|KWD|SAR)) and the searching_regexp will match any string that starts with 0+ chars other than /, then has a / and then any amount of any chars but newline ([^/\n]*/.*) that does not start with a space, / and a currency value.

See the Python demo.

Body:
I have following code:

currencies = ['USD', 'GBP', 'EUR', 'JPY', 'CHF', 'SEK', 'DKK', 'NOK', 'SGD', 'HKD', 'AUD', 'TWD', 'NZD', 'CNY', 'KRW', 'INR', 'CAD', 'VEF', 'EGP', 'THB', 'IDR', 'PKR', 'MYR', 'PHP', 'MXN', 'VND', 'CZK', 'HUF', 'PLN', 'TRY', 'ZAR', 'ILS', 'ARS', 'CLP', 'BRL', 'RUB', 'QAR', 'AED', 'COP', 'PEN', 'CNH', 'KWD', 'SAR']
exclusive_regexp = ".*\/" + ".*|.*\/".join(currencies) + ".*"
searching_regexp = "^(?! (" + exclusive_regexp + ")$)(.*\/.*)$"
searching_regexp = re.compile(searching_regexp)

with open('raw.txt', 'r') as unprocessed_ticks:
    print(re.findall(searching_regexp, unprocessed_ticks.read()))

It should find all strings that can be matched with produced regex.

I probed generated regex with regex101.com online tool. it works with python flavor. But in real code it doesn't match anything:
https://regex101.com/r/70uiuE/7
Why it can happen?

Body:
A couple of points:

There is no need to escape / in Python (and any other) string patterns (/ should only be escaped when the regex is set via a regex literal with slashes used as regex delimiters)

You are overusing .* pattern, you should minimize the number of alternatives that match at the same location in string

For ^ and $ to match start and end of a line, you must pass the re.M or re.MULTILINE flag to the regex.

I suggest changing the regexps to

exclusive_regexp = r"/(?:{})".format("|".join(currencies))
searching_regexp = re.compile(r"^(?! .*" + exclusive_regexp + ")[^/\n]*/.*", re.M)

and then

print(searching_regexp.findall(unprocessed_ticks.read()))

The exclusive_regexp regex will look tidier now (/(?:USD|GBP|EUR|JPY|CHF|SEK|DKK|NOK|SGD|HKD|AUD|TWD|NZD|CNY|KRW|INR|CAD|VEF|EGP|THB|IDR|PKR|MYR|PHP|MXN|VND|CZK|HUF|PLN|TRY|ZAR|ILS|ARS|CLP|BRL|RUB|QAR|AED|COP|PEN|CNH|KWD|SAR)) and the searching_regexp will match any string that starts with 0+ chars other than /, then has a / and then any amount of any chars but newline ([^/\n]*/.*) that does not start with a space, / and a currency value.

See the Python demo.


问:
Which Lisp should I learn?

Which Lisp (dialect) should I learn, and why?

The fragmentation between CL and Scheme slows uptake (at least for me!).

So, give me the "true answer", please!

I have tried to read feature comparisons, and they seem to get bogged down in esoterica (that I don't fully understand) like whether the dialect is fully tail-recursive, and the like.  I'm hoping you all (collectively) can make the opaque differences clear.

Things I like

Good library support, good environments, and Unicode support.

Things I dislike

Flamewars. Features that are useful at the beginning, but interfere with long-term learning.

Updates

I've been mostly using MzScheme, which I'm enjoying just fine, once I got readline support going.  I don't run a GUI on Unix, so it seemed to be a fine environment option for me.

I'm also very happy that Clojure has an easy to install .deb package on debian, so it's much easier to play with.  This is a big win.  Despite the hostility of some rather easy-to-anger below, low barrier to entry is a win.  I like being spoonfed.

After reading a lot more of SICP, I do understand the issues around tail recursion much better.

答:
Lisp's biggest problem is that there's no overriding standard everyone sticks to. You don't just have to pick between Common Lisp and Scheme; you have to pick between several incompatible implementations of Common Lisp and several incompatible implementations of Scheme, as well as other little things like Arc and Clojure.

I would recommend starting with MzScheme. It's popular, it's actively developed, it has many libraries offering the features of a modern programming environment, and Scheme is somewhat simpler (not to say better, just simpler) than Common Lisp.

OwnerDisplayName: Gregg Lind

Title: Which Lisp should I learn?

LastEditorDisplayName: Gregg Lind

OwnerDisplayName: Chuck

ContentLicense: CC BY-SA 2.5


问:
Error installing nokogiri, permission denied
When I type "gem install nokogiri" I receive the error:

```<code>ERROR: While executing gem ... (Errno: EACCES)
    Permission denied - C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/
nokogiri-1.6.0-x86-mingw32/lib/nokogiri/1.9/nokogiri.so
</code>```

I have no idea what I need to do to fix this. Any help would be appreciated.

答:
Finally got bundle install to work after installing gems bindex, msgpack, bootsnap, byebug, childprocess, ffi puma, and sqlite3

I came to this conclusion by sudo installing each gem after I attempted to bundle install, once it through an Error it told me what gem I needed

```<code> In Gemfile:

     puma
</code>```

Body:
When I type "gem install nokogiri" I receive the error:

ERROR: While executing gem ... (Errno: EACCES)
    Permission denied - C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/
nokogiri-1.6.0-x86-mingw32/lib/nokogiri/1.9/nokogiri.so

I have no idea what I need to do to fix this. Any help would be appreciated.

Body:
Finally got bundle install to work after installing gems bindex, msgpack, bootsnap, byebug, childprocess, ffi puma, and sqlite3

I came to this conclusion by sudo installing each gem after I attempted to bundle install, once it through an Error it told me what gem I needed

 In Gemfile:

     puma


问:
java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED
I have a service that is expected to execute requests at ~5 or more requests/min. This service depends on Apache AsyncHttpClient. After every few minutes, the clients hits some condition which causes java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED. All requests to the client start failing with same exception message. After service is restarted, this cycle repeats.

It is really hard to debug this problem as the request execution failure surprisingly does not cause a callback to the failed() method of the AsyncResponse.

From what I could gather, there has been a fix HTTPCORE-370 in HttpCore NIO which solved a similar problem in 4.3.2. I am using the following version -

commons-httpclient-3.1.jar

httpasyncclient-4.1.1.jar

httpcore-4.4.4.jar

httpcore-nio-4.4.4.jar

Yet seeing this problem.

答:
If you see OutOfMemoryError before this, try this

-XX:MaxDirectMemorySize=512M

See https://issues.apache.org/jira/browse/HTTPASYNC-104


问:
Selecting from a database from Today onwards
Im trying to select a bunch of stuff from a database. Each have a date and i'd like to only display the ones whose date is equal to today onwards.

$sql="SELECT * FROM bs_events WHERE NOW() ORDER BY eventDate ";

I have tried looking everywhere and cannot find anything.

Thanks for any help guys

答:
try this</p>

```<code>$sql="SELECT * FROM bs_events WHERE NOW() <=  eventDate   ORDER BY eventDate ";
</code>```

Body:
try this

$sql="SELECT * FROM bs_events WHERE NOW() <=  eventDate   ORDER BY eventDate ";


问:
Dask to Flatten Dictionary Column
I am new to Dask and am looking to find a way to flatten a dictionary column in a PANDAS dataframe. Here is a screenshot of the first row of a 16 million-row dataframe:

And here is a sample of the text from three rows:

```<code>{{u'F9_07_PZ_COMP_DIRECT': u'0', u'F9_07_PZ_DIRTRSTKEY_NAME': u'DEBRA MEALY', u'F9_07_PZ_COMP_OTHER': u'0', u'F9_07_PZ_COMP_RELATED': u'0', u'F9_07_PZ_TITLE': u'CHAIR PERSON', u'F9_07_PZ_AVE_HOURS_WEEK': u'1.00', u'F9_07_PC_TRUSTEE_INDIVIDUAL': u'X'}, {u'F9_07_PZ_COMP_DIRECT': u'0', u'F9_07_PZ_DIRTRSTKEY_NAME': u'HELEN GORDON', u'F9_07_PZ_COMP_OTHER': u'0', u'F9_07_PZ_COMP_RELATED': u'0', u'F9_07_PZ_TITLE': u'VICE CHAIR', u'F9_07_PZ_AVE_HOURS_WEEK': u'1.00', u'F9_07_PC_TRUSTEE_INDIVIDUAL': u'X'}, {'F9_07_PC_HIGH_COMP_EMPLOYEE': 'X', 'F9_07_PZ_DIRTRSTKEY_NAME': 'ROB S KHANUJA', 'F9_07_PZ_COMP_OTHER': '14902', 'F9_07_PZ_COMP_RELATED': '0', 'F9_07_PZ_TITLE': 'EXEC. DIR. OPERATIONS', 'F9_07_PZ_AVE_HOURS_WEEK': '40.00', 'F9_07_PZ_COMP_DIRECT': '133173'}}
</code>```

I would normally flatten the Form990PartVIISectionAGrp column with the following code:

    df = pd.concat([df.drop(['Form990PartVIISectionAGrp'], axis=1), df['Form990PartVIISectionAGrp'].swifter.apply(pd.Series)], axis=1)
```

<p>I'm looking to do this in Dask but am getting the following error: "ValueError: The columns in the computed data do not match the columns in the provided metadata."</p>

<p>I am using Python 2.7. I import the relevant packages</p>

<pre class="lang-py prettyprint-override"><code>    from dask import dataframe as dd
    from dask.multiprocessing import get
    from multiprocessing import cpu_count
    nCores = cpu_count()
</code>```

To test the code, I created a random sample of the data:

    dfs = df.sample(1000)
```

<p>And then generate the Dask dataframe:</p>

<pre class="lang-py prettyprint-override"><code>    ddf = dd.from_pandas(dfs, npartitions=nCores)
</code>```

The column is currently in string format so I convert it to a dictionary. Normally, I would just write one line of code:

dfs['Form990PartVIISectionAGrp'] = dfs['Form990PartVIISectionAGrp'].apply(literal_eval)
```

<p>But I've instead tried to do it here in a more 'Dask-like' form so I write the following function and then apply it:</p>

<pre class="lang-py prettyprint-override"><code>    def make_dict(dfs):
        dfs['Form990PartVIISectionAGrp'] = dfs['Form990PartVIISectionAGrp'].apply(literal_eval)
        return dfs
</code>```

    ddf_out = ddf.map_partitions(make_dict, meta=dfs[:0]).compute()
```

<p>This works -- it returns a PANDAS dataframe where the Form990PartVIISectionAGrp column is in dictionary format (it's not any faster than the non-Dask apply, however). </p>

<p><img src="https://i.stack.imgur.com/huxKZ.png" alt="ddf_out"></p>

<p>I then re-create the Dask DF:</p>

<pre class="lang-py prettyprint-override"><code>    ddf = dd.from_pandas(ddf_out, npartitions=nCores)
</code>```

And write a function to flatten the column:

    def flatten(ddf_out):
        ddf_out = pd.concat([ddf_out.drop(['Form990PartVIISectionAGrp'], axis=1), ddf_out['Form990PartVIISectionAGrp'].apply(pd.Series)], axis=1)
        #ddf_out = ddf_out['Form990PartVIISectionAGrp'].apply(pd.Series)
    return ddf_out
```

<p>If I then run this code:</p>

<pre class="lang-py prettyprint-override"><code>    result = ddf.map_partitions(flatten)
</code>```

I get the following output, where the column has not been flattened:

I was also getting errors about missing meta data, and given that the above did not help parse the dictionary column, so I created a list of the columns produced by the plain Python flattening of the columns and used that to create a dictionary of the columns and data types:

metadir = {u'BusinessName': 'O', u'F9_07_PC_FORMER': 'O', u'F9_07_PC_HIGH_COMP_EMPLOYEE': 'O',
       u'F9_07_PC_KEY_EMPLOYEE': 'O', u'F9_07_PC_OFFICER': 'O',
       u'F9_07_PC_TRUSTEE_INDIVIDUAL': 'O', u'F9_07_PC_TRUSTEE_INSTITUTIONAL': 'O',
       u'F9_07_PZ_AVE_HOURS_WEEK': 'O', u'F9_07_PZ_AVE_HOURS_WEEK_RELATED': 'O',
       u'F9_07_PZ_COMP_DIRECT': 'O', u'F9_07_PZ_COMP_OTHER': 'O',
       u'F9_07_PZ_COMP_RELATED': 'O', u'F9_07_PZ_DIRTRSTKEY_NAME': 'O',
       u'F9_07_PZ_TITLE': 'O', u'NameBusiness': 'O', u'URL': 'O'}
```

<p>I then apply the flatten function with this meta data:</p>

<pre class="lang-py prettyprint-override"><code>    result = ddf.map_partitions(flatten, meta=metadir)
</code>```

I get the following output as result:

Running result.columns yields this:

Where this fails is in running compute(), where I get the following error message: "ValueError: The columns in the computed data do not match the columns in the provided metadata." I get the same error whether I write:

result.compute()
```

<p>or </p>

<pre class="lang-py prettyprint-override"><code>result.compute(meta=metadir)
</code>```

I'm not sure what I'm doing wrong here. The columns in result seem to match those in metadir. Any suggestions would be greatly appreciated.

UPDATE:
Here is my stab at updating the flatten function.

    meta = pd.DataFrame(columns=['URL', 'F9_07_PC_TRUSTEE_INDIVIDUAL',
     'F9_07_PZ_DIRTRSTKEY_NAME',
     'F9_07_PZ_COMP_OTHER',
     'F9_07_PZ_COMP_RELATED',
     'F9_07_PZ_TITLE',
     'F9_07_PZ_AVE_HOURS_WEEK',
     'F9_07_PZ_COMP_DIRECT',
     'F9_07_PZ_AVE_HOURS_WEEK_RELATED',
     'F9_07_PC_OFFICER',
     'F9_07_PC_HIGH_COMP_EMPLOYEE',
     'BusinessName',
     'F9_07_PC_KEY_EMPLOYEE',
     'F9_07_PC_TRUSTEE_INSTITUTIONAL',
     'NameBusiness',
     'F9_07_PC_FORMER'], dtype="O")

    def flatten(ddf_out):
        ddf_out = pd.concat([df.drop(['Form990PartVIISectionAGrp'], axis=1), df['Form990PartVIISectionAGrp'].apply(pd.Series)], axis=1)
        for m in meta:
            if m not in ddf_out:
                df[m] = ''
        return ddf_out
```

<p>Then I run:</p>

<pre class="lang-py prettyprint-override"><code>result = ddf.map_partitions(flatten, meta=meta).compute()
</code>```

答:
Given a small- or medium-sized dataset, the plain PANDAS solution would work:

```<code>df = pd.concat([df.drop(['Form990PartVIISectionAGrp'], axis=1), df['Form990PartVIISectionAGrp'].apply(pd.Series)], axis=1)
</code>```

However, with 16 million rows the PANDAS solution won't run on either a Macbook with 16GB of RAM or a Windows machine with 96GB. For that reason I looked to Dask. As seen in the above answer and comments, however, the Dask solution does not work because each observation in my dataset does not necessarily have all of the dictionary keys. Collectively, the 16 million observations of  Form990PartVIISectionAGrp have the 15 keys in the following list:

```<code>  newkeys = ['F9_07_PC_TRUSTEE_INDIVIDUAL',
 'F9_07_PZ_DIRTRSTKEY_NAME',
 'F9_07_PZ_COMP_OTHER',
 'F9_07_PZ_COMP_RELATED',
 'F9_07_PZ_TITLE',
 'F9_07_PZ_AVE_HOURS_WEEK',
 'F9_07_PZ_COMP_DIRECT',
 'F9_07_PZ_AVE_HOURS_WEEK_RELATED',
 'F9_07_PC_OFFICER',
 'F9_07_PC_HIGH_COMP_EMPLOYEE',
 'BusinessName',
 'F9_07_PC_KEY_EMPLOYEE',
 'F9_07_PC_TRUSTEE_INSTITUTIONAL',
 'NameBusiness',
 'F9_07_PC_FORMER']
</code>```

So, my solution involved taking some of the hints provided by @mdurant above and first adding any missing keys to each row:

```<code>for index, row in df[:].iterrows():
    for k in newkeys:
        row['Form990PartVIISectionAGrp'].setdefault(k, np.nan)
</code>```

That took 100 minutes on my Macbook. Based on mdurant's comment, I then saved the dataframe in JSON format:

```<code>df.to_json('df.json', orient='records', lines=True)
</code>```

And read the file into Dask as text:

```<code>import json
import dask.bag as db
b = db.read_text('df.json').map(json.loads)
</code>```

Then create a function to flatten the column:

```<code>def flatten(record):
    return {
    'F9_07_PZ_COMP_OTHER': record['Form990PartVIISectionAGrp']['F9_07_PZ_COMP_OTHER'],
    'F9_07_PZ_COMP_RELATED': record['Form990PartVIISectionAGrp']['F9_07_PZ_COMP_RELATED'],
    'F9_07_PC_TRUSTEE_INDIVIDUAL': record['Form990PartVIISectionAGrp']['F9_07_PC_TRUSTEE_INDIVIDUAL'],
    'F9_07_PZ_DIRTRSTKEY_NAME': record['Form990PartVIISectionAGrp']['F9_07_PZ_DIRTRSTKEY_NAME'],
    'F9_07_PZ_COMP_DIRECT': record['Form990PartVIISectionAGrp']['F9_07_PZ_COMP_DIRECT'],
    'F9_07_PZ_COMP_OTHER': record['Form990PartVIISectionAGrp']['F9_07_PZ_COMP_OTHER'],
    'BusinessName': record['Form990PartVIISectionAGrp']['BusinessName'],
    'F9_07_PC_FORMER': record['Form990PartVIISectionAGrp']['F9_07_PC_FORMER'],
    'F9_07_PC_HIGH_COMP_EMPLOYEE': record['Form990PartVIISectionAGrp']['F9_07_PC_HIGH_COMP_EMPLOYEE'],
    'F9_07_PC_KEY_EMPLOYEE': record['Form990PartVIISectionAGrp']['F9_07_PC_KEY_EMPLOYEE'],
    'F9_07_PC_OFFICER': record['Form990PartVIISectionAGrp']['F9_07_PC_OFFICER'],
    'F9_07_PC_TRUSTEE_INSTITUTIONAL': record['Form990PartVIISectionAGrp']['F9_07_PC_TRUSTEE_INSTITUTIONAL'],
    'F9_07_PZ_AVE_HOURS_WEEK': record['Form990PartVIISectionAGrp']['F9_07_PZ_AVE_HOURS_WEEK'],
    'F9_07_PZ_AVE_HOURS_WEEK_RELATED': record['Form990PartVIISectionAGrp']['F9_07_PZ_AVE_HOURS_WEEK_RELATED'],
    'F9_07_PZ_TITLE': record['Form990PartVIISectionAGrp']['F9_07_PZ_TITLE'],
    'NameBusiness': record['Form990PartVIISectionAGrp']['NameBusiness'],
    'URL': record['URL'],
}
</code>```

I can then apply the function:

```<code>df = b.map(flatten).to_dataframe()
</code>```

And export the data to CSV:

```<code>df.to_csv('compensation*.csv')
</code>```

This works like a charm! In short, based on mdurant's helpful comments above, the keys are 1) to add missing keys to all observations and 2) to not read your data into Dask from PANDAS (use text or CSV instead). Taking care of those two issues led to a good solution to this problem.

Title: Dask to Flatten Dictionary Column

Body:
I am new to Dask and am looking to find a way to flatten a dictionary column in a PANDAS dataframe. Here is a screenshot of the first row of a 16 million-row dataframe:

And here is a sample of the text from three rows:

{{u'F9_07_PZ_COMP_DIRECT': u'0', u'F9_07_PZ_DIRTRSTKEY_NAME': u'DEBRA MEALY', u'F9_07_PZ_COMP_OTHER': u'0', u'F9_07_PZ_COMP_RELATED': u'0', u'F9_07_PZ_TITLE': u'CHAIR PERSON', u'F9_07_PZ_AVE_HOURS_WEEK': u'1.00', u'F9_07_PC_TRUSTEE_INDIVIDUAL': u'X'}, {u'F9_07_PZ_COMP_DIRECT': u'0', u'F9_07_PZ_DIRTRSTKEY_NAME': u'HELEN GORDON', u'F9_07_PZ_COMP_OTHER': u'0', u'F9_07_PZ_COMP_RELATED': u'0', u'F9_07_PZ_TITLE': u'VICE CHAIR', u'F9_07_PZ_AVE_HOURS_WEEK': u'1.00', u'F9_07_PC_TRUSTEE_INDIVIDUAL': u'X'}, {'F9_07_PC_HIGH_COMP_EMPLOYEE': 'X', 'F9_07_PZ_DIRTRSTKEY_NAME': 'ROB S KHANUJA', 'F9_07_PZ_COMP_OTHER': '14902', 'F9_07_PZ_COMP_RELATED': '0', 'F9_07_PZ_TITLE': 'EXEC. DIR. OPERATIONS', 'F9_07_PZ_AVE_HOURS_WEEK': '40.00', 'F9_07_PZ_COMP_DIRECT': '133173'}}

I would normally flatten the Form990PartVIISectionAGrp column with the following code:

    df = pd.concat([df.drop(['Form990PartVIISectionAGrp'], axis=1), df['Form990PartVIISectionAGrp'].swifter.apply(pd.Series)], axis=1)

I'm looking to do this in Dask but am getting the following error: "ValueError: The columns in the computed data do not match the columns in the provided metadata."

I am using Python 2.7. I import the relevant packages

    from dask import dataframe as dd
    from dask.multiprocessing import get
    from multiprocessing import cpu_count
    nCores = cpu_count()

To test the code, I created a random sample of the data:

    dfs = df.sample(1000)

And then generate the Dask dataframe:

    ddf = dd.from_pandas(dfs, npartitions=nCores)

The column is currently in string format so I convert it to a dictionary. Normally, I would just write one line of code:

dfs['Form990PartVIISectionAGrp'] = dfs['Form990PartVIISectionAGrp'].apply(literal_eval)

But I've instead tried to do it here in a more 'Dask-like' form so I write the following function and then apply it:

    def make_dict(dfs):
        dfs['Form990PartVIISectionAGrp'] = dfs['Form990PartVIISectionAGrp'].apply(literal_eval)
        return dfs

    ddf_out = ddf.map_partitions(make_dict, meta=dfs[:0]).compute()

This works -- it returns a PANDAS dataframe where the Form990PartVIISectionAGrp column is in dictionary format (it's not any faster than the non-Dask apply, however).

I then re-create the Dask DF:

    ddf = dd.from_pandas(ddf_out, npartitions=nCores)

And write a function to flatten the column:

    def flatten(ddf_out):
        ddf_out = pd.concat([ddf_out.drop(['Form990PartVIISectionAGrp'], axis=1), ddf_out['Form990PartVIISectionAGrp'].apply(pd.Series)], axis=1)
        #ddf_out = ddf_out['Form990PartVIISectionAGrp'].apply(pd.Series)
    return ddf_out

If I then run this code:

    result = ddf.map_partitions(flatten)

I get the following output, where the column has not been flattened:

I was also getting errors about missing meta data, and given that the above did not help parse the dictionary column, so I created a list of the columns produced by the plain Python flattening of the columns and used that to create a dictionary of the columns and data types:

metadir = {u'BusinessName': 'O', u'F9_07_PC_FORMER': 'O', u'F9_07_PC_HIGH_COMP_EMPLOYEE': 'O',
       u'F9_07_PC_KEY_EMPLOYEE': 'O', u'F9_07_PC_OFFICER': 'O',
       u'F9_07_PC_TRUSTEE_INDIVIDUAL': 'O', u'F9_07_PC_TRUSTEE_INSTITUTIONAL': 'O',
       u'F9_07_PZ_AVE_HOURS_WEEK': 'O', u'F9_07_PZ_AVE_HOURS_WEEK_RELATED': 'O',
       u'F9_07_PZ_COMP_DIRECT': 'O', u'F9_07_PZ_COMP_OTHER': 'O',
       u'F9_07_PZ_COMP_RELATED': 'O', u'F9_07_PZ_DIRTRSTKEY_NAME': 'O',
       u'F9_07_PZ_TITLE': 'O', u'NameBusiness': 'O', u'URL': 'O'}

I then apply the flatten function with this meta data:

    result = ddf.map_partitions(flatten, meta=metadir)

I get the following output as result:

Running result.columns yields this:

Where this fails is in running compute(), where I get the following error message: "ValueError: The columns in the computed data do not match the columns in the provided metadata." I get the same error whether I write:

result.compute()

or

result.compute(meta=metadir)

I'm not sure what I'm doing wrong here. The columns in result seem to match those in metadir. Any suggestions would be greatly appreciated.

UPDATE:
Here is my stab at updating the flatten function.

    meta = pd.DataFrame(columns=['URL', 'F9_07_PC_TRUSTEE_INDIVIDUAL',
     'F9_07_PZ_DIRTRSTKEY_NAME',
     'F9_07_PZ_COMP_OTHER',
     'F9_07_PZ_COMP_RELATED',
     'F9_07_PZ_TITLE',
     'F9_07_PZ_AVE_HOURS_WEEK',
     'F9_07_PZ_COMP_DIRECT',
     'F9_07_PZ_AVE_HOURS_WEEK_RELATED',
     'F9_07_PC_OFFICER',
     'F9_07_PC_HIGH_COMP_EMPLOYEE',
     'BusinessName',
     'F9_07_PC_KEY_EMPLOYEE',
     'F9_07_PC_TRUSTEE_INSTITUTIONAL',
     'NameBusiness',
     'F9_07_PC_FORMER'], dtype="O")

    def flatten(ddf_out):
        ddf_out = pd.concat([df.drop(['Form990PartVIISectionAGrp'], axis=1), df['Form990PartVIISectionAGrp'].apply(pd.Series)], axis=1)
        for m in meta:
            if m not in ddf_out:
                df[m] = ''
        return ddf_out

Then I run:

result = ddf.map_partitions(flatten, meta=meta).compute()

Body:
Given a small- or medium-sized dataset, the plain PANDAS solution would work:

df = pd.concat([df.drop(['Form990PartVIISectionAGrp'], axis=1), df['Form990PartVIISectionAGrp'].apply(pd.Series)], axis=1)

However, with 16 million rows the PANDAS solution won't run on either a Macbook with 16GB of RAM or a Windows machine with 96GB. For that reason I looked to Dask. As seen in the above answer and comments, however, the Dask solution does not work because each observation in my dataset does not necessarily have all of the dictionary keys. Collectively, the 16 million observations of  Form990PartVIISectionAGrp have the 15 keys in the following list:

  newkeys = ['F9_07_PC_TRUSTEE_INDIVIDUAL',
 'F9_07_PZ_DIRTRSTKEY_NAME',
 'F9_07_PZ_COMP_OTHER',
 'F9_07_PZ_COMP_RELATED',
 'F9_07_PZ_TITLE',
 'F9_07_PZ_AVE_HOURS_WEEK',
 'F9_07_PZ_COMP_DIRECT',
 'F9_07_PZ_AVE_HOURS_WEEK_RELATED',
 'F9_07_PC_OFFICER',
 'F9_07_PC_HIGH_COMP_EMPLOYEE',
 'BusinessName',
 'F9_07_PC_KEY_EMPLOYEE',
 'F9_07_PC_TRUSTEE_INSTITUTIONAL',
 'NameBusiness',
 'F9_07_PC_FORMER']

So, my solution involved taking some of the hints provided by @mdurant above and first adding any missing keys to each row:

for index, row in df[:].iterrows():
    for k in newkeys:
        row['Form990PartVIISectionAGrp'].setdefault(k, np.nan)

That took 100 minutes on my Macbook. Based on mdurant's comment, I then saved the dataframe in JSON format:

df.to_json('df.json', orient='records', lines=True)

And read the file into Dask as text:

import json
import dask.bag as db
b = db.read_text('df.json').map(json.loads)

Then create a function to flatten the column:

def flatten(record):
    return {
    'F9_07_PZ_COMP_OTHER': record['Form990PartVIISectionAGrp']['F9_07_PZ_COMP_OTHER'],
    'F9_07_PZ_COMP_RELATED': record['Form990PartVIISectionAGrp']['F9_07_PZ_COMP_RELATED'],
    'F9_07_PC_TRUSTEE_INDIVIDUAL': record['Form990PartVIISectionAGrp']['F9_07_PC_TRUSTEE_INDIVIDUAL'],
    'F9_07_PZ_DIRTRSTKEY_NAME': record['Form990PartVIISectionAGrp']['F9_07_PZ_DIRTRSTKEY_NAME'],
    'F9_07_PZ_COMP_DIRECT': record['Form990PartVIISectionAGrp']['F9_07_PZ_COMP_DIRECT'],
    'F9_07_PZ_COMP_OTHER': record['Form990PartVIISectionAGrp']['F9_07_PZ_COMP_OTHER'],
    'BusinessName': record['Form990PartVIISectionAGrp']['BusinessName'],
    'F9_07_PC_FORMER': record['Form990PartVIISectionAGrp']['F9_07_PC_FORMER'],
    'F9_07_PC_HIGH_COMP_EMPLOYEE': record['Form990PartVIISectionAGrp']['F9_07_PC_HIGH_COMP_EMPLOYEE'],
    'F9_07_PC_KEY_EMPLOYEE': record['Form990PartVIISectionAGrp']['F9_07_PC_KEY_EMPLOYEE'],
    'F9_07_PC_OFFICER': record['Form990PartVIISectionAGrp']['F9_07_PC_OFFICER'],
    'F9_07_PC_TRUSTEE_INSTITUTIONAL': record['Form990PartVIISectionAGrp']['F9_07_PC_TRUSTEE_INSTITUTIONAL'],
    'F9_07_PZ_AVE_HOURS_WEEK': record['Form990PartVIISectionAGrp']['F9_07_PZ_AVE_HOURS_WEEK'],
    'F9_07_PZ_AVE_HOURS_WEEK_RELATED': record['Form990PartVIISectionAGrp']['F9_07_PZ_AVE_HOURS_WEEK_RELATED'],
    'F9_07_PZ_TITLE': record['Form990PartVIISectionAGrp']['F9_07_PZ_TITLE'],
    'NameBusiness': record['Form990PartVIISectionAGrp']['NameBusiness'],
    'URL': record['URL'],
}

I can then apply the function:

df = b.map(flatten).to_dataframe()

And export the data to CSV:

df.to_csv('compensation*.csv')

This works like a charm! In short, based on mdurant's helpful comments above, the keys are 1) to add missing keys to all observations and 2) to not read your data into Dask from PANDAS (use text or CSV instead). Taking care of those two issues led to a good solution to this problem.


问:
Not able to login to Salesforce application through JMeter
Not able to login into Salesforce Application through JMeter.

Please help.

I am providing correct Username and Password but still not able to login successfully.

答: Have you added HTTP Cookie Manager in your jmeter script? If not, can you add it and then give it another shot?</p>


问:
How to convert a Kotlin data class object to map?
Is there any easy way or any standard library method to convert a Kotlin data class object to a map/dictionary of its properties by property names? Can reflection be avoided?</p>

答:
I was using the jackson method, but turns out the performance of this is terrible on Android for first serialization (github issue here). And its dramatically worse for older android versions, (see benchmarks here)

But you can do this much faster with Gson. Conversion in both directions shown here:

```<code>import com.google.gson.Gson
import com.google.gson.reflect.TypeToken

val gson = Gson()

//convert a data class to a map
fun <T> T.serializeToMap(): Map<String, Any> {
    return convert()
}

//convert a map to a data class
inline fun <reified T> Map<String, Any>.toDataClass(): T {
    return convert()
}

//convert an object of type I to type O
inline fun <I, reified O> I.convert(): O {
    val json = gson.toJson(this)
    return gson.fromJson(json, object : TypeToken<O>() {}.type)
}

//example usage
data class Person(val name: String, val age: Int)

fun main() {

    val person = Person("Tom Hanley", 99)

    val map = mapOf(
        "name" to "Tom Hanley",
        "age" to 99
    )

    val personAsMap: Map<String, Any> = person.serializeToMap()

    val mapAsPerson: Person = map.toDataClass()
}
</code>```

Body:
I was using the jackson method, but turns out the performance of this is terrible on Android for first serialization (github issue here). And its dramatically worse for older android versions, (see benchmarks here)

But you can do this much faster with Gson. Conversion in both directions shown here:

import com.google.gson.Gson
import com.google.gson.reflect.TypeToken

val gson = Gson()

//convert a data class to a map
fun <T> T.serializeToMap(): Map<String, Any> {
    return convert()
}

//convert a map to a data class
inline fun <reified T> Map<String, Any>.toDataClass(): T {
    return convert()
}

//convert an object of type I to type O
inline fun <I, reified O> I.convert(): O {
    val json = gson.toJson(this)
    return gson.fromJson(json, object : TypeToken<O>() {}.type)
}

//example usage
data class Person(val name: String, val age: Int)

fun main() {

    val person = Person("Tom Hanley", 99)

    val map = mapOf(
        "name" to "Tom Hanley",
        "age" to 99
    )

    val personAsMap: Map<String, Any> = person.serializeToMap()

    val mapAsPerson: Person = map.toDataClass()
}


问:
How do I change the Hitbox size of a turtle in python turtle graphics?
I dont know how to change the size of a turtle hitbox in python turtle graphics

I haven't tried anything yet because I'm new, and know very little about this. I've tried googling it, though, but nothing popped up.

```<code>from turtle import *
import turtle
from random import randint
import time

screen = turtle.Screen()
screen.setup(1920, 1080)
player = turtle.Turtle()
</code>```

I want to add a button that you have to click to start right here

The game starts right here :

```<code>wn = turtle.Screen()

last_pressed = 'up'

def setup(col, x, y, w, s, shape):

  player.penup()
  player.up()
  player.goto(x,y)
  player.width(w)
  player.turtlesize(s)
  player.color(col)
  player.lt(90)
  player.down()
  wn.onkey(up, "s")
  wn.onkey(left, "d")
  wn.onkey(right, "a")
  wn.onkey(back, "w")
  wn.onkey(quitTurtles, "Escape")
  wn.listen()
  wn.mainloop()
</code>```

答:
This may not be exactly what you are looking for, but this might work in your situation.
Detecting collision in Python turtle game
This is a thread on collision detection between objects and with some tweaking of numbers you could increase the hitbox of the turtle using the abs() function

Body:
I dont know how to change the size of a turtle hitbox in python turtle graphics

I haven't tried anything yet because I'm new, and know very little about this. I've tried googling it, though, but nothing popped up.

from turtle import *
import turtle
from random import randint
import time

screen = turtle.Screen()
screen.setup(1920, 1080)
player = turtle.Turtle()

I want to add a button that you have to click to start right here

The game starts right here :

wn = turtle.Screen()

last_pressed = 'up'

def setup(col, x, y, w, s, shape):

  player.penup()
  player.up()
  player.goto(x,y)
  player.width(w)
  player.turtlesize(s)
  player.color(col)
  player.lt(90)
  player.down()
  wn.onkey(up, "s")
  wn.onkey(left, "d")
  wn.onkey(right, "a")
  wn.onkey(back, "w")
  wn.onkey(quitTurtles, "Escape")
  wn.listen()
  wn.mainloop()


问:
Read specific value of an object
I'm using an api that return this object:

```<code>   {
        "0155894402285712": { "type": "GBUserFieldText", "value": "A0242", "name": "Codice+tessera" },
        "0155894402283800": { "type": "GBUserFieldText", "value": "LZZMRN55L53C003Z", "name": "Codice+Fiscale" }
    }
</code>```

I need to extract the value A0242 and LZZMRN55L53C003Z but the only things that I know are the name "Codice+tessera" and "Codice+fiscale". How can I read these values? Maybe my question is stupid but really I'm losing my brain today...

Thanks

答:
You can use Object.values and then access value key's value

let obj =  {
        "0155894402285712": { "type": "GBUserFieldText", "value": "A0242", "name": "Codice+tessera" },
        "0155894402283800": { "type": "GBUserFieldText", "value": "LZZMRN55L53C003Z", "name": "Codice+Fiscale" }
}

Object.values(obj).forEach(({value})=>{
  console.log(value)
})```

Title: Read specific value of an object

Body:
I'm using an api that return this object:

   {
        "0155894402285712": { "type": "GBUserFieldText", "value": "A0242", "name": "Codice+tessera" },
        "0155894402283800": { "type": "GBUserFieldText", "value": "LZZMRN55L53C003Z", "name": "Codice+Fiscale" }
    }

I need to extract the value A0242 and LZZMRN55L53C003Z but the only things that I know are the name "Codice+tessera" and "Codice+fiscale". How can I read these values? Maybe my question is stupid but really I'm losing my brain today...

Thanks


问:
Using foreign keys when referencing aggregate roots by id
It is a best-practice to reference another aggregate root by ID and not by reference (see "Implementing Domain Driven Design" page 359 and following).

```<code>// direct reference
class AggregateRootA(val b: AggregateRootB)

// reference by ID (preferred)
class AggregateRootA(val b: AggregateRootBId)

class AggregateRootB(val id: AggregateRootBId)
class AggregateRootBId(val id: Long)
</code>```

The central argument for this kind of decoupling is that each aggregate root should be a transaction boundary.

I wonder whether it is a good or bad idea to use a foreign key relationship in the database when using reference by id in the code. This constraint would enforce consistency at the database level, since there couldn't be a database record of AggregateRootA without the referenced record of AggregateRootB. This is essentially what I want, since the object would be invalid otherwise.

Is there any downside of using a foreign key here apart from slightly more cumbersome testing?

答:
This is too long to add as a comment but I'm just adding to what @VoiceOfUnreason has answered so please do accept that answer and not this :)

Just to be pedantic about it: you are referring to DRI (declarative referential integrity) since even a foreign key relationship could be disabled.

There is nothing inherently wrong with DRI since it has a rather valid purpose.  It would, of course, make most sense within the same bounded context as the data would be in the same database.  For different Bounded Contexts you could either forego the enforcement of the relationship altogether and work only on the denormalized data or enforce the relationship by storing the value objects in your Bounded Context's data store.  Aggregates from a "foreign" Bounded Context would be Value Objects in your Bounded Context.  In that case everything that @VoiceOfUnreason mentioned comes into play.  In some instances it may make perfect sense to ensure that certain changes to state occur in a reasonable order.  For instance, you would probably wish to register a customer before activating the phone line.

For cases where you have less strict rules you could go ahead with a weak relationship and perhaps keep track of your process with a status indicating whether the dependent data is waiting for some other bit to complete which is a typical asynchronous/parallel processing paradigm.

Body:
It is a best-practice to reference another aggregate root by ID and not by reference (see "Implementing Domain Driven Design" page 359 and following).

// direct reference
class AggregateRootA(val b: AggregateRootB)

// reference by ID (preferred)
class AggregateRootA(val b: AggregateRootBId)

class AggregateRootB(val id: AggregateRootBId)
class AggregateRootBId(val id: Long)

The central argument for this kind of decoupling is that each aggregate root should be a transaction boundary.

I wonder whether it is a good or bad idea to use a foreign key relationship in the database when using reference by id in the code. This constraint would enforce consistency at the database level, since there couldn't be a database record of AggregateRootA without the referenced record of AggregateRootB. This is essentially what I want, since the object would be invalid otherwise.

Is there any downside of using a foreign key here apart from slightly more cumbersome testing?


问:
Cannot catch orientation changes from "upsidedown" to landscapes on iOS
I need to invalidate my collection view layout when screen orientation changes. Since iPhone X and others have safe area insets; I need my cells' subviews positioned according to screen safe area insets.

If you switch between left-right landscapes from portrait there is no problem. But if you switch from upsidedown I cannot catch the orientation change and automatically cannot

I tried two existing approaches.

First is using NotificationCenter. Second is overriding viewWillTransitionToSize:withTransitionCoordinator: function. It's all the same.

in CollectionViewController:

    NotificationCenter.default.addObserver(forName: UIDevice.orientationDidChangeNotification, object: nil, queue: nil) { (notif) in
      self.collectionViewLayout.invalidateLayout()
    }
```

<p>in Cell:</p>

<pre class="lang-swift prettyprint-override"><code>  override func layoutSubviews() {
    super.layoutSubviews()

    let orientation = UIDevice.current.orientation
    if let insets = UIApplication.shared.keyWindow?.safeAreaInsets {
      if orientation == .landscapeLeft {
        titleLabelLeadingConstraint.constant = insets.left + 20
        titleLabelTrailingConstraint.constant = 20
      } else if orientation == .landscapeRight {
        titleLabelLeadingConstraint.constant = 20
        titleLabelTrailingConstraint.constant = insets.right + 20
      } else {
        titleLabelLeadingConstraint.constant = 20
        titleLabelTrailingConstraint.constant = 20
      }
    }

    print("orin: \(orientation.rawValue) | left: \(titleLabelLeadingConstraint.constant) | right: \(titleLabelTrailingConstraint.constant)")

  }

</code>```

The print() function not being called when you switch from/to upsidedown. But collectionviewlayouts invalidateLayout() function is being called.

答:
OK. At last, I came up with the thing I did not want to do. Observed orientation changes from the cells' themselves.

 override func awakeFromNib() {
    super.awakeFromNib()

    NotificationCenter.default.addObserver(forName: UIDevice.orientationDidChangeNotification, object: nil, queue: nil) { (notif ) in
      // doing my things already stated in the question's itself.
    }
  }
```

Now they can observe all the changes. But I do not know if this much registration is a wise thing; even though collection view reuses cells and -in my case- 7 to 8 cells are on screen at most.

Body:
I need to invalidate my collection view layout when screen orientation changes. Since iPhone X and others have safe area insets; I need my cells' subviews positioned according to screen safe area insets.

If you switch between left-right landscapes from portrait there is no problem. But if you switch from upsidedown I cannot catch the orientation change and automatically cannot

I tried two existing approaches.

First is using NotificationCenter. Second is overriding viewWillTransitionToSize:withTransitionCoordinator: function. It's all the same.

in CollectionViewController:

    NotificationCenter.default.addObserver(forName: UIDevice.orientationDidChangeNotification, object: nil, queue: nil) { (notif) in
      self.collectionViewLayout.invalidateLayout()
    }

in Cell:

  override func layoutSubviews() {
    super.layoutSubviews()

    let orientation = UIDevice.current.orientation
    if let insets = UIApplication.shared.keyWindow?.safeAreaInsets {
      if orientation == .landscapeLeft {
        titleLabelLeadingConstraint.constant = insets.left + 20
        titleLabelTrailingConstraint.constant = 20
      } else if orientation == .landscapeRight {
        titleLabelLeadingConstraint.constant = 20
        titleLabelTrailingConstraint.constant = insets.right + 20
      } else {
        titleLabelLeadingConstraint.constant = 20
        titleLabelTrailingConstraint.constant = 20
      }
    }

    print("orin: \(orientation.rawValue) | left: \(titleLabelLeadingConstraint.constant) | right: \(titleLabelTrailingConstraint.constant)")

  }

The print() function not being called when you switch from/to upsidedown. But collectionviewlayouts invalidateLayout() function is being called.

Body:
OK. At last, I came up with the thing I did not want to do. Observed orientation changes from the cells' themselves.

 override func awakeFromNib() {
    super.awakeFromNib()

    NotificationCenter.default.addObserver(forName: UIDevice.orientationDidChangeNotification, object: nil, queue: nil) { (notif ) in
      // doing my things already stated in the question's itself.
    }
  }

Now they can observe all the changes. But I do not know if this much registration is a wise thing; even though collection view reuses cells and -in my case- 7 to 8 cells are on screen at most.


问:
Why is orientdb oetl import giving me this error
I am trying to import a csv file into orientdb 3.0 I have created and tested the json file and it works with a smaller dataset. But the dataset that I want to import is around a billion rows (six columns)

Following is the user.json file I am using for import with oetl

```<code>{
    "source": { "file": { "path": "d1.csv" } },
    "extractor": { "csv": {} },
    "transformers": [
      { "vertex": { "class": "User" } }
    ],
    "loader": {
      "orientdb": {
         "dbURL": "plocal:/databases/magriwebdoc",
         "dbType": "graph",
         "classes": [
           {"name": "User", "extends": "V"}
         ], "indexes": [
           {"class":"User", "fields":["id:string"], "type":"UNIQUE" }
         ]
      }
    }
  }
</code>```

This is the console output from oetl command:

```<code>2019-05-22 14:31:15:484 INFO  Windows OS is detected, 262144 limit of open files will be set for the disk cache. [ONative]
2019-05-22 14:31:15:647 INFO  8261029888 B/7878 MB/7 GB of physical memory were detected on machine [ONative]
2019-05-22 14:31:15:647 INFO  Detected memory limit for current process is 8261029888 B/7878 MB/7 GB [ONative]
2019-05-22 14:31:15:649 INFO  JVM can use maximum 455MB of heap memory [OMemoryAndLocalPaginatedEnginesInitializer]
2019-05-22 14:31:15:649 INFO  Because OrientDB is running outside a container 12% of memory will be left unallocated according to the setting 'memory.leftToOS' not taking into account heap memory [OMemoryAndLocalPaginatedEnginesInitializer]
2019-05-22 14:31:15:650 INFO  OrientDB auto-config DISKCACHE=6,477MB (heap=455MB os=7,878MB) [orientechnologies]
2019-05-22 14:31:15:652 INFO  System is started under an effective user : `lenovo` [OEngineLocalPaginated]
2019-05-22 14:31:15:670 INFO  WAL maximum segment size is set to 6,144 MB [OrientDBEmbedded]
2019-05-22 14:31:15:701 INFO  BEGIN ETL PROCESSOR [OETLProcessor]
2019-05-22 14:31:15:703 INFO  [file] Reading from file d1.csv with encoding UTF-8 [OETLFileSource]
2019-05-22 14:31:15:703 INFO  Started execution with 1 worker threads [OETLProcessor]
2019-05-22 14:31:16:008 INFO  Page size for WAL located in D:\databases\magriwebdoc is set to 4096 bytes. [OCASDiskWriteAheadLog]
2019-05-22 14:31:16:703 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 1001ms [0 warnings, 0 errors] [OETLProcessor]
2019-05-22 14:31:16:770 INFO  Storage 'plocal:D:\databases/magriwebdoc' is opened under OrientDB distribution : 3.0.18 - Veloce (build 747595e790a081371496f3bb9c57cec395644d82, branch 3.0.x) [OLocalPaginatedStorage]
2019-05-22 14:31:17:703 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 2001ms [0 warnings, 0 errors] [OETLProcessor]
2019-05-22 14:31:17:954 SEVER ETL process has problem:  [OETLProcessor]
2019-05-22 14:31:17:956 INFO  END ETL PROCESSOR [OETLProcessor]
2019-05-22 14:31:17:957 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 2255ms [0 warnings, 0 errors] [OETLProcessor]D:\orientserver\bin>
</code>```

I know the code is right but I am assuming it's more of a memory issue!

Please advise what should I do.

答:
Have you tried improving your memory settings, according to the size of the data that you want to process?

From the documentation, you can custom these properties:

Configuration Environmental Variables (See $ORIENTDB_OPTS_MEMORY parameter)

Performance-Tuning - Memory Settings

Maybe could help you

Body:
I am trying to import a csv file into orientdb 3.0 I have created and tested the json file and it works with a smaller dataset. But the dataset that I want to import is around a billion rows (six columns)

Following is the user.json file I am using for import with oetl

{
    "source": { "file": { "path": "d1.csv" } },
    "extractor": { "csv": {} },
    "transformers": [
      { "vertex": { "class": "User" } }
    ],
    "loader": {
      "orientdb": {
         "dbURL": "plocal:/databases/magriwebdoc",
         "dbType": "graph",
         "classes": [
           {"name": "User", "extends": "V"}
         ], "indexes": [
           {"class":"User", "fields":["id:string"], "type":"UNIQUE" }
         ]
      }
    }
  }

This is the console output from oetl command:

2019-05-22 14:31:15:484 INFO  Windows OS is detected, 262144 limit of open files will be set for the disk cache. [ONative]
2019-05-22 14:31:15:647 INFO  8261029888 B/7878 MB/7 GB of physical memory were detected on machine [ONative]
2019-05-22 14:31:15:647 INFO  Detected memory limit for current process is 8261029888 B/7878 MB/7 GB [ONative]
2019-05-22 14:31:15:649 INFO  JVM can use maximum 455MB of heap memory [OMemoryAndLocalPaginatedEnginesInitializer]
2019-05-22 14:31:15:649 INFO  Because OrientDB is running outside a container 12% of memory will be left unallocated according to the setting 'memory.leftToOS' not taking into account heap memory [OMemoryAndLocalPaginatedEnginesInitializer]
2019-05-22 14:31:15:650 INFO  OrientDB auto-config DISKCACHE=6,477MB (heap=455MB os=7,878MB) [orientechnologies]
2019-05-22 14:31:15:652 INFO  System is started under an effective user : `lenovo` [OEngineLocalPaginated]
2019-05-22 14:31:15:670 INFO  WAL maximum segment size is set to 6,144 MB [OrientDBEmbedded]
2019-05-22 14:31:15:701 INFO  BEGIN ETL PROCESSOR [OETLProcessor]
2019-05-22 14:31:15:703 INFO  [file] Reading from file d1.csv with encoding UTF-8 [OETLFileSource]
2019-05-22 14:31:15:703 INFO  Started execution with 1 worker threads [OETLProcessor]
2019-05-22 14:31:16:008 INFO  Page size for WAL located in D:\databases\magriwebdoc is set to 4096 bytes. [OCASDiskWriteAheadLog]
2019-05-22 14:31:16:703 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 1001ms [0 warnings, 0 errors] [OETLProcessor]
2019-05-22 14:31:16:770 INFO  Storage 'plocal:D:\databases/magriwebdoc' is opened under OrientDB distribution : 3.0.18 - Veloce (build 747595e790a081371496f3bb9c57cec395644d82, branch 3.0.x) [OLocalPaginatedStorage]
2019-05-22 14:31:17:703 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 2001ms [0 warnings, 0 errors] [OETLProcessor]
2019-05-22 14:31:17:954 SEVER ETL process has problem:  [OETLProcessor]
2019-05-22 14:31:17:956 INFO  END ETL PROCESSOR [OETLProcessor]
2019-05-22 14:31:17:957 INFO  + extracted 0 rows (0 rows/sec) - 0 rows -> loaded 0 vertices (0 vertices/sec) Total time: 2255ms [0 warnings, 0 errors] [OETLProcessor]D:\orientserver\bin>

I know the code is right but I am assuming it's more of a memory issue!

Please advise what should I do.