问:
Core dump when global variable used
```<code>#include <iostream>

int sizeWhileBuilding = 0;
int sizeWhileCounting = 0;

struct Node
{
   Node( int theData ) :
      data( theData ), left( NULL ), right( NULL )
   { std::cout << "Node::Node() with data " << theData << std::endl; }

   int data;
   Node* left;
   Node* right;
};

Node* insertNode( Node* root, int data )
{
   if( !root )
   {
      ++sizeWhileBuilding;
      std::cout << "Current size is " << sizeWhileBuilding << std::endl;
      std::cout << "Making new node with " << data << std::endl;
      root = new Node( data );
   }
   else if( root->data < data )
   {
      std::cout << "The node " << root->data << " is lesser than " << data << ". Making right node" << std::endl;
      root->right = insertNode( root->right, data );
   }
   else if( root->data > data )
   {
      std::cout << "The node " << root->data << " is greater than " << data << ". Making left node" << std::endl;
      root->left = insertNode( root->left, data );
   }
   return root;
}

int main()
{
   std::cout << "Making binary tree" << std::endl;

   Node* root;
   if( root )
   {
      std::cout << "Root is NOT null after creation" << std::endl;
   }
   else
   {
      std::cout << "Root is null after creation" << std::endl;
   }
   std::cout << "==============================================================================" << std::endl;
   root = insertNode( root, 10 );

//Problem line below
std::cout << "The size of the tree is " << sizeWhileBuilding << std::endl;
}
</code>```

When I run this code with the "problem line" commented out, it runs and inserts node 10 into the tree. When I run it with the problem line, I see that it coredumps.

When I run without the problem line, I see that the else part at the beginning of main prints "Root is null after creation". With the problementer code here line, "Root is NOT null after creation is printed".

What I don't understand is why the Node* root; initialization depends upon whether the global variable is being used?

答:
```<code>Node* root;
</code>```

root is a local variable and if not initialized its value will be garbage. So sometimes it might be nullptr other times anything else. So if used it will cause  undefined behaviour that may end in coredump. The solution is to assign it:

```<code>Node* root = nullptr;
</code>```

also:

What I don't understand is why the Node* root; initialization depends upon whether the global variable is being used?

if you have undefined behaviour in your code then anything is possible, also such strange behaviour as you describe.

Title: Core dump when global variable used

Body:
#include <iostream>

int sizeWhileBuilding = 0;
int sizeWhileCounting = 0;

struct Node
{
   Node( int theData ) :
      data( theData ), left( NULL ), right( NULL )
   { std::cout << "Node::Node() with data " << theData << std::endl; }

   int data;
   Node* left;
   Node* right;
};

Node* insertNode( Node* root, int data )
{
   if( !root )
   {
      ++sizeWhileBuilding;
      std::cout << "Current size is " << sizeWhileBuilding << std::endl;
      std::cout << "Making new node with " << data << std::endl;
      root = new Node( data );
   }
   else if( root->data < data )
   {
      std::cout << "The node " << root->data << " is lesser than " << data << ". Making right node" << std::endl;
      root->right = insertNode( root->right, data );
   }
   else if( root->data > data )
   {
      std::cout << "The node " << root->data << " is greater than " << data << ". Making left node" << std::endl;
      root->left = insertNode( root->left, data );
   }
   return root;
}

int main()
{
   std::cout << "Making binary tree" << std::endl;

   Node* root;
   if( root )
   {
      std::cout << "Root is NOT null after creation" << std::endl;
   }
   else
   {
      std::cout << "Root is null after creation" << std::endl;
   }
   std::cout << "==============================================================================" << std::endl;
   root = insertNode( root, 10 );

//Problem line below
std::cout << "The size of the tree is " << sizeWhileBuilding << std::endl;
}

When I run this code with the "problem line" commented out, it runs and inserts node 10 into the tree. When I run it with the problem line, I see that it coredumps.

When I run without the problem line, I see that the else part at the beginning of main prints "Root is null after creation". With the problementer code here line, "Root is NOT null after creation is printed".

What I don't understand is why the Node* root; initialization depends upon whether the global variable is being used?

Body:
Node* root;

root is a local variable and if not initialized its value will be garbage. So sometimes it might be nullptr other times anything else. So if used it will cause  undefined behaviour that may end in coredump. The solution is to assign it:

Node* root = nullptr;

also:

What I don't understand is why the Node* root; initialization depends upon whether the global variable is being used?

if you have undefined behaviour in your code then anything is possible, also such strange behaviour as you describe.


来源: stackexchange

问:
How to export as CSV for selected period using AmCharts?
I have an AngularJS app where I have an amchart stock javascript chart. The chart allows the user to select the period of data they want to view, (1-3 months) and allows the user to export their data as CSV or JSON. Whenever I export the data it returns the full 3 months of data. Is there anyway to have the graph export data that correlates to selected period? </p>

答:
You can do this by setting exportSelection to true in your export config:

```<code>var chart = AmCharts.makeChart("chartdiv", {
  // ...
  "export": {
    "enabled": true,
    "exportSelection": true
  }
});
</code>```

Demo

ContentLicense: CC BY-SA 3.0

Body:
You can do this by setting exportSelection to true in your export config:

var chart = AmCharts.makeChart("chartdiv", {
  // ...
  "export": {
    "enabled": true,
    "exportSelection": true
  }
});

Demo


问:
Detecting URLs in Textarea - NOT looking for regex
I am trying to limit the number of URLs someone can add to a textarea. I have a regex that can find all the URLs just fine. Here is the issue I'm running into:

When two links are right next to each other, but separated by a space, the code I've written will not identify the second link. If I separate them by two spaces, then they will be identified. I have no idea what is going on here.

function limitLinks() {
  var counter = 0;
  var url = /\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/?)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\)){0,}(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s\!()\[\]{};:\'\"\.\,<>?«»“”‘’]){0,})/ig;
  var sig = $(".textarea").val();
  var sigsplit = sig.replace(/\n/g, " ").split(" ");
  var links = [];
  for (var i = 0; i < sigsplit.length; i++) {
    if (url.test(sigsplit[i])) {
      counter++;
      links.push(sigsplit[i]);
    }
  }
  console.log("Here's the split text:" + sigsplit);
  console.log("All the links are:" + links);
}

limitLinks();```
<pre class="snippet-code-html lang-html prettyprint-override"><code><script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
<form>
  <textarea class="textarea">foo.com bar.com</textarea>
</form></code>```

With one space separating the URLs, it only identifies the first link. If you add a space to separate the URLs by two spaces, it identifies both. I don't think the regex is the problem here - the regex will find both URLs just fine. It has something to do with the array that is created, I think, when I split the textarea value by spaces.

Any insight would be really appreciated... I've spent too much time trying to figure this out!

答:
With further research, this question is a duplicate of Why RegExp with global flag in Javascript give wrong results?. But I will keep this here as my notes how I got there.

Your regex seems to have some kind of "memory".

Look at this:

```<code>var url = /\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/?)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\)){0,}(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s\!()\[\]{};:\'\"\.\,<>?«»“”‘’]){0,})/ig;

function isLink(str) {
  return url.test(str);
}

console.log('odd.com even.com odd.com even.com odd.com'.split(' ').filter(isLink));
</code>```

It returns ["odd.com", "odd.com", "odd.com"].

Now try this:

```<code>function isLink(str) {
  var url = /\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/?)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\)){0,}(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s\!()\[\]{};:\'\"\.\,<>?«»“”‘’]){0,})/ig;
  return url.test(str);
}

console.log('odd.com even.com odd.com even.com odd.com'.split(' ').filter(isLink));
</code>```

It returns ["odd.com", "even.com", "odd.com", "even.com", "odd.com"].

Fascinating, huh? There's something about your regex that causes it to act differently between executions. By reinstantiating the regex on every function call, it resets that memory. Very strange.

To fix this I removed the g flag at the end.

Here is an even smaller reproducible example:

```<code>var re = /a./g;
console.log('a1 a2 a3 a4 a5 a6'.split(' ').filter(s => re.test(s)));
// returns ["a1", "a3", "a5"]
</code>```

Removing the g flag returns ["a1", "a2", "a3", "a4", "a5", "a6"]

Body:
I am trying to limit the number of URLs someone can add to a textarea. I have a regex that can find all the URLs just fine. Here is the issue I'm running into:

When two links are right next to each other, but separated by a space, the code I've written will not identify the second link. If I separate them by two spaces, then they will be identified. I have no idea what is going on here.

function limitLinks() {
  var counter = 0;
  var url = /\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/?)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\)){0,}(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s\!()\[\]{};:\'\"\.\,<>?«»“”‘’]){0,})/ig;
  var sig = $(".textarea").val();
  var sigsplit = sig.replace(/\n/g, " ").split(" ");
  var links = [];
  for (var i = 0; i < sigsplit.length; i++) {
    if (url.test(sigsplit[i])) {
      counter++;
      links.push(sigsplit[i]);
    }
  }
  console.log("Here's the split text:" + sigsplit);
  console.log("All the links are:" + links);
}

limitLinks();

<form>
  <textarea class="textarea">foo.com bar.com</textarea>
</form>

With one space separating the URLs, it only identifies the first link. If you add a space to separate the URLs by two spaces, it identifies both. I don't think the regex is the problem here - the regex will find both URLs just fine. It has something to do with the array that is created, I think, when I split the textarea value by spaces.

Any insight would be really appreciated... I've spent too much time trying to figure this out!

Body:
With further research, this question is a duplicate of Why RegExp with global flag in Javascript give wrong results?. But I will keep this here as my notes how I got there.

Your regex seems to have some kind of "memory".

Look at this:

var url = /\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/?)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\)){0,}(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s\!()\[\]{};:\'\"\.\,<>?«»“”‘’]){0,})/ig;

function isLink(str) {
  return url.test(str);
}

console.log('odd.com even.com odd.com even.com odd.com'.split(' ').filter(isLink));

It returns ["odd.com", "odd.com", "odd.com"].

Now try this:

function isLink(str) {
  var url = /\b((?:[a-z][\w-]+:(?:\/{1,3}|[a-z0-9%])|www\d{0,3}[.]|[a-z0-9.\-]+[.][a-z]{2,4}\/?)(?:[^\s()<>]+|\(([^\s()<>]+|(\([^\s()<>]+\)))*\)){0,}(?:\(([^\s()<>]+|(\([^\s()<>]+\)))*\)|[^\s\!()\[\]{};:\'\"\.\,<>?«»“”‘’]){0,})/ig;
  return url.test(str);
}

console.log('odd.com even.com odd.com even.com odd.com'.split(' ').filter(isLink));

It returns ["odd.com", "even.com", "odd.com", "even.com", "odd.com"].

Fascinating, huh? There's something about your regex that causes it to act differently between executions. By reinstantiating the regex on every function call, it resets that memory. Very strange.

To fix this I removed the g flag at the end.

Here is an even smaller reproducible example:

var re = /a./g;
console.log('a1 a2 a3 a4 a5 a6'.split(' ').filter(s => re.test(s)));
// returns ["a1", "a3", "a5"]

Removing the g flag returns ["a1", "a2", "a3", "a4", "a5", "a6"]


问:
How to convert NSString to Int so I can add in swift 3.0
I have a value which seems to be an Int but when I do type(of: value) it returns Optional<Any> and after trying many things I figure that data type NSString.

```<code>let sum = data[0]["total1"] + data[0]["total2"]
</code>```

I have two values like that and I want to add them but it does not let me do it with saying

binary operator + cannot be applied to two "Any" operands

or

Value of type "Any" has no member 'intValue'

How do I convert this Any to Int so I can add them?

答:
This is how I do it:</p>

```<code>extension String {
    var asInt: Int? {
        return Int(self)
    }
    var isInt: Bool {
        return Int(self) != nil
    }
}

call it as follows (assuming a property called result):

        let stringOne = "14"
        let stringTwo = "12"
        if  stringOne.isInt && stringTwo.isInt {
            self.result = stringOne.asInt + stringTwo.asInt
        }
</code>```

ContentLicense: CC BY-SA 4.0

Body:
I have a value which seems to be an Int but when I do type(of: value) it returns Optional<Any> and after trying many things I figure that data type NSString.

let sum = data[0]["total1"] + data[0]["total2"]

I have two values like that and I want to add them but it does not let me do it with saying

binary operator + cannot be applied to two "Any" operands

or

Value of type "Any" has no member 'intValue'

How do I convert this Any to Int so I can add them?

Body:
This is how I do it:

extension String {
    var asInt: Int? {
        return Int(self)
    }
    var isInt: Bool {
        return Int(self) != nil
    }
}

call it as follows (assuming a property called result):

        let stringOne = "14"
        let stringTwo = "12"
        if  stringOne.isInt && stringTwo.isInt {
            self.result = stringOne.asInt + stringTwo.asInt
        }


问:
matching the regular expression with the whole string
im kinda strumped in a situation where i need to match a whole string with a regular expression rather than finding if the pattern exists in the string.

suppose if i have a regular expression

```<code>/\\^^\\w+\\$^/
</code>```

what i want is that the code will run through various strings , compare the strings with the regular expression and perform some task if the strings start and end with a ^.

Examples

^hello world^ is a match

my ^hello world^ should not be a match

the php function preg_match matches both of the results

any clues ???

答:
You can use the regex:

```<code>^\^[\w\s]+\^$
</code>```

^ is a regex meta-character which is used as start anchor. To match a literal ^ you need to escape it as \^.

So we have:

^ : Start anchor

\^: A literal ^

[\w\s]+ : space separated words.

\^: A literal ^

$ : End anchor.

Ideone Link

ContentLicense: CC BY-SA 2.5

Body:
im kinda strumped in a situation where i need to match a whole string with a regular expression rather than finding if the pattern exists in the string.

suppose if i have a regular expression

/\\^^\\w+\\$^/

what i want is that the code will run through various strings , compare the strings with the regular expression and perform some task if the strings start and end with a ^.

Examples

^hello world^ is a match

my ^hello world^ should not be a match

the php function preg_match matches both of the results

any clues ???

Body:
You can use the regex:

^\^[\w\s]+\^$

^ is a regex meta-character which is used as start anchor. To match a literal ^ you need to escape it as \^.

So we have:

^ : Start anchor

\^: A literal ^

[\w\s]+ : space separated words.

\^: A literal ^

$ : End anchor.

Ideone Link


问:
How can I add different color overlays to each iteration of the texture in this glsl shader?
So I'm working on this shader right now, and the goal is to have a series of camera based tiles, each overlayed with a different color (think Andy Warhol). I've got the tiles working (side issue - the tiles on the ends are currently being cut off ~50%) but I want to add a color filter to each iteration. I'm looking for the cleanest possible way of doing this. Any ideas?

frag shader:

```<code>#define N 3.0 // number of columns
#define M 3. // number of rows

#import "GPUImageWarholFilter.h"

NSString *const kGPUImageWarholFragmentShaderString = SHADER_STRING
(
 precision highp float;
 varying highp vec2 textureCoordinate;
 uniform sampler2D inputImageTexture;

 void main()
{

    vec4 color = texture2D(inputImageTexture, vec2(fract(textureCoordinate.x * N)/(M/ N), fract(textureCoordinate.y * M) / (M/N)));

    gl_FragColor = color;

}
 );
</code>```

答:
Let's say you're texture is getting applied to a quad and your vertices are:

```<code>float quad[] = {
    0.0, 0.0,
    1.0, 0.0,
    1.0, 0.0, // First triangle
    1.0, 0.0,
    1.0, 0.0,
    1.0, 1.0  // Second triangle
};
</code>```

You can specify texture coordinates which corresponds to each of these vertices and describe the relationship in your vertex array object.

Now, if you quadruple the number of vertices, you'll have access to mid points between the extents of the quad (create four sub-quads) and you can associate (0,0) -> (1,1) texture coordinates with each of these subquads. The effect would be a rendering of the full texture in each subquad.

You could then do math on your vertices in your vertex shader to calculate a color component to assign to each subquad. The color component would be passed to your fragment shader. Use uniforms to specify number of rows and columns then define the color component based on which cell you've calculated the current vertex to be in.

You could also try to compute everything in the fragment shader, but I think it could get expensive pretty fast. Just a hunch though.

You could also try the GL_REPEAT texture parameter, but you'll have less control of the outcome: https://open.gl/textures

Body:
So I'm working on this shader right now, and the goal is to have a series of camera based tiles, each overlayed with a different color (think Andy Warhol). I've got the tiles working (side issue - the tiles on the ends are currently being cut off ~50%) but I want to add a color filter to each iteration. I'm looking for the cleanest possible way of doing this. Any ideas?

frag shader:

#define N 3.0 // number of columns
#define M 3. // number of rows

#import "GPUImageWarholFilter.h"

NSString *const kGPUImageWarholFragmentShaderString = SHADER_STRING
(
 precision highp float;
 varying highp vec2 textureCoordinate;
 uniform sampler2D inputImageTexture;

 void main()
{

    vec4 color = texture2D(inputImageTexture, vec2(fract(textureCoordinate.x * N)/(M/ N), fract(textureCoordinate.y * M) / (M/N)));

    gl_FragColor = color;

}
 );

Body:
Let's say you're texture is getting applied to a quad and your vertices are:

float quad[] = {
    0.0, 0.0,
    1.0, 0.0,
    1.0, 0.0, // First triangle
    1.0, 0.0,
    1.0, 0.0,
    1.0, 1.0  // Second triangle
};

You can specify texture coordinates which corresponds to each of these vertices and describe the relationship in your vertex array object.

Now, if you quadruple the number of vertices, you'll have access to mid points between the extents of the quad (create four sub-quads) and you can associate (0,0) -> (1,1) texture coordinates with each of these subquads. The effect would be a rendering of the full texture in each subquad.

You could then do math on your vertices in your vertex shader to calculate a color component to assign to each subquad. The color component would be passed to your fragment shader. Use uniforms to specify number of rows and columns then define the color component based on which cell you've calculated the current vertex to be in.

You could also try to compute everything in the fragment shader, but I think it could get expensive pretty fast. Just a hunch though.

You could also try the GL_REPEAT texture parameter, but you'll have less control of the outcome: https://open.gl/textures


问:
What is a Pythonic way to remove doubled duplicates in a list but allow triplets/greater?
I have a list in Python

```<code>list1 = [0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1]
</code>```

There are "groupings" of consecutive 1's and 0's. For my purposes, I am only interested in the consecutive 1's. Let's say if there is a solitary 1, e.g.

```<code> ... 0, 0, 1, 0, 0 ...
</code>```

I would like it to be changed into a 0. Similarly, if there are only pairs of 1's, these should become 0s as well. e.g.

```<code>.... 0, 0, 1, 1, 0, 0...
</code>```

should become

```<code>.... 0, 0, 0, 0, 0, 0...
</code>```

But "triplets" of consecutive ones or higher are ok.

I considered keeping track of the counts of 1s via a dictionary, but this feels too complex. The idea would be to iterate through the list, keeping track of the count of consecutive 1s in a list. But, how do you "go back" and switch the 1s to 0s?

```<code>counter_list = []
for i in list1:
    if i == 1:
        counter_list = counter_list + 1
            if len(counter_list)==1 or len(counter_list)==2:
                # now I don't know how to "go back" to change these elements into 0s
</code>```

答:
itertools.groupby can help by grouping the list into runs of 0s or 1s. From there, we can use the length of a group of 1s to decide whether to switch it to 0s, and use itertools.chain.from_iterable to fuse the groups back into one stream:

```<code>import itertools

groups = ((key, list(group)) for key, group in itertools.groupby(list1))

fixed_groups = (group if key==0 or len(group)>2 else [0]*len(group)
                for key, group in groups)

result = list(itertools.chain.from_iterable(fixed_groups))
</code>```

Body:
I have a list in Python

list1 = [0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1]

There are "groupings" of consecutive 1's and 0's. For my purposes, I am only interested in the consecutive 1's. Let's say if there is a solitary 1, e.g.

 ... 0, 0, 1, 0, 0 ...

I would like it to be changed into a 0. Similarly, if there are only pairs of 1's, these should become 0s as well. e.g.

.... 0, 0, 1, 1, 0, 0...

should become

.... 0, 0, 0, 0, 0, 0...

But "triplets" of consecutive ones or higher are ok.

I considered keeping track of the counts of 1s via a dictionary, but this feels too complex. The idea would be to iterate through the list, keeping track of the count of consecutive 1s in a list. But, how do you "go back" and switch the 1s to 0s?

counter_list = []
for i in list1:
    if i == 1:
        counter_list = counter_list + 1
            if len(counter_list)==1 or len(counter_list)==2:
                # now I don't know how to "go back" to change these elements into 0s

Body:
itertools.groupby can help by grouping the list into runs of 0s or 1s. From there, we can use the length of a group of 1s to decide whether to switch it to 0s, and use itertools.chain.from_iterable to fuse the groups back into one stream:

import itertools

groups = ((key, list(group)) for key, group in itertools.groupby(list1))

fixed_groups = (group if key==0 or len(group)>2 else [0]*len(group)
                for key, group in groups)

result = list(itertools.chain.from_iterable(fixed_groups))


问:
Android Studio refactoring issue
I want to refactor the name of Dealerlist to SnapbackDealerlist.

However, Snapbackdealerlist keeps on appearing as a subclass within Dealerlist instead of replacing it.

Does anyone have any ideas on how to fix it? cheers

答: Right click on the file in the Android Studio project pane and click Refactor -> Rename.  This will rename both the class and the file.</p>

Title: Android Studio refactoring issue


问:
Compilation errors in simple Java program with Scanner
```<code>import java.util.*;
public class strings {
    public static void main (String [] args) {
        Scanner keyboard = new Scanner(System.in);
        System.out.print("Type your first integer: ");
        int first = keyboard.next();
        System.out.print("Type your seconds integer : ");
        int second = keyboard.next();
        System.out.print("The sum of your two integers are:");
    }
}
</code>```

<p>I dont know why im getting 2 errors on strings cannot converted to int. </p>

答:
You are using next() method from Scanner class. This method return String and you cannot store string at int type variable. So you can use nextInt() method for int type input or you can just Store the value at String variable and after that you can convert the String variable to int.

Please follow this code:

```<code>import java.util.*;
public class strings {
    public static void main (String [] args) {
        Scanner keyboard = new Scanner(System.in);
        System.out.print("Type your first integer: ");
        int first = keyboard.nextInt();
        System.out.print("Type your seconds integer : ");
        int second = keyboard.nextInt();
        System.out.print("The sum of your two integers are:" +(first+second));
    }
}
</code>```

or Follow this code

```<code>import java.util.*;
public class strings {
    public static void main (String [] args) {
        Scanner keyboard = new Scanner(System.in);
        System.out.print("Type your first integer: ");
        //Get String input and stroe at String variable
        String firstString = keyboard.next();
        //Convert the String variable to int
        int first=Integer.parseInt(firstString);
        System.out.print("Type your seconds integer : ");
        //Get String input and stroe at String variable
        String secondString = keyboard.next();
        //Convert the String variable to int
        int second=Integer.parseInt(secondString);
        System.out.print("The sum of your two integers are:" +(first+second));
    }
}
</code>```

Body:
import java.util.*;
public class strings {
    public static void main (String [] args) {
        Scanner keyboard = new Scanner(System.in);
        System.out.print("Type your first integer: ");
        int first = keyboard.next();
        System.out.print("Type your seconds integer : ");
        int second = keyboard.next();
        System.out.print("The sum of your two integers are:");
    }
}

I dont know why im getting 2 errors on strings cannot converted to int.

Body:
You are using next() method from Scanner class. This method return String and you cannot store string at int type variable. So you can use nextInt() method for int type input or you can just Store the value at String variable and after that you can convert the String variable to int.

Please follow this code:

import java.util.*;
public class strings {
    public static void main (String [] args) {
        Scanner keyboard = new Scanner(System.in);
        System.out.print("Type your first integer: ");
        int first = keyboard.nextInt();
        System.out.print("Type your seconds integer : ");
        int second = keyboard.nextInt();
        System.out.print("The sum of your two integers are:" +(first+second));
    }
}

or Follow this code

import java.util.*;
public class strings {
    public static void main (String [] args) {
        Scanner keyboard = new Scanner(System.in);
        System.out.print("Type your first integer: ");
        //Get String input and stroe at String variable
        String firstString = keyboard.next();
        //Convert the String variable to int
        int first=Integer.parseInt(firstString);
        System.out.print("Type your seconds integer : ");
        //Get String input and stroe at String variable
        String secondString = keyboard.next();
        //Convert the String variable to int
        int second=Integer.parseInt(secondString);
        System.out.print("The sum of your two integers are:" +(first+second));
    }
}


问:
How does reselect affect rendering of components
I don't really understand how does reselect reduces component's rendering. This is what I have without reselect:

```<code>const getListOfSomething = (state) => (
  state.first.list[state.second.activeRecord]
);

const mapStateToProps = (state, ownProps) => {
  console.log(state.first.list, state.second.activeRecord);
  return {
    ...ownProps,
    listOfSomething: getListOfSomething(state)
  }
};
</code>```

It compounds an element from some list based on some value. Render is called each time anything in the state changes, so for example my console.log outputs:

```<code>{}, ""
{}, ""
{}, ""
{}, "1"
{"filled", "1"}
</code>```

because something is going on in the different part of store. Thus the component is rendered 5 times, 2 redundantly.

Using reselect however:

```<code>const getList = state => state.first.list;
const getActiveRecord = state => state.second.activeRecord;
const listOfSomething = (list, activeRecord) => {
  console.log(list, activeRecord);
  return list[activeRecord];
}
const getListOfSomething = createSelector(getList, getActiveRecord, listOfSomething);

const mapStateToProps = (state, ownProps) => {
  console.log(state.first.list, state.second.activeRecord);
  return {
    ...ownProps,
    listOfSomething: getListOfSomething(state)
  }
};
</code>```

Here my first selector console.log outputs:

```<code>{}, ""
{}, "1"
{"filled", "1"}
</code>```

The second:

```<code>{}, ""
{}, ""
{}, ""
{}, "1"
{"filled", "1"}
</code>```

And the component is rendered properly - 3 times !

Why is that so? Why is the component rendered only 3 times? What's exectly going on here?

答:
React-Redux's connect function relies on shallow-equality comparisons.  Each time the store updates and a component's mapState function runs, that connected component checks to see if the contents of the returned object changed.  If mapState returned something different, then the wrapped component must need to re-render.

Reselect uses "memoization", which means it saves a copy of the last inputs and outputs, and if it sees the same inputs twice in a row, it returns the last output rather than recalculating things.  So, a Reselect-based selector function will return the same object references if the inputs didn't change, which means that it's more likely that connect will see that nothing is different and the wrapped component won't re-render.

See the new Redux FAQ section on Immutable Data for more info on how immutability and comparisons work with Redux and React-Redux.

Body:
I don't really understand how does reselect reduces component's rendering. This is what I have without reselect:

const getListOfSomething = (state) => (
  state.first.list[state.second.activeRecord]
);

const mapStateToProps = (state, ownProps) => {
  console.log(state.first.list, state.second.activeRecord);
  return {
    ...ownProps,
    listOfSomething: getListOfSomething(state)
  }
};

It compounds an element from some list based on some value. Render is called each time anything in the state changes, so for example my console.log outputs:

{}, ""
{}, ""
{}, ""
{}, "1"
{"filled", "1"}

because something is going on in the different part of store. Thus the component is rendered 5 times, 2 redundantly.

Using reselect however:

const getList = state => state.first.list;
const getActiveRecord = state => state.second.activeRecord;
const listOfSomething = (list, activeRecord) => {
  console.log(list, activeRecord);
  return list[activeRecord];
}
const getListOfSomething = createSelector(getList, getActiveRecord, listOfSomething);

const mapStateToProps = (state, ownProps) => {
  console.log(state.first.list, state.second.activeRecord);
  return {
    ...ownProps,
    listOfSomething: getListOfSomething(state)
  }
};

Here my first selector console.log outputs:

{}, ""
{}, "1"
{"filled", "1"}

The second:

{}, ""
{}, ""
{}, ""
{}, "1"
{"filled", "1"}

And the component is rendered properly - 3 times !

Why is that so? Why is the component rendered only 3 times? What's exectly going on here?


问:
Common libraries in a large team
Assume you have five products, and all of them use one or more of the company's internal libraries, written by individual developers.

It sounds simple but in practice, I found it to be very difficult to maintain.

How do you deal with the following scenarios:

A developer unintentionally introduces a bug and breaks everything in production.

Every library has to mature, That means the API needs to evolve, so how do you deploy the updated version to production if every developer needs to update/test their code while they are extremely busy on other projects? Is this a resource and time issue?

Version control, deployment,and usage. Would you store this in one global location or force each project to use, say, svn:externals to "tie" a library?

I've found that it is extremely hard to come up with a good strategy. My own pet theory is this:

Each common library has to have a super-thorough set of tests or else it should never be common, even if it means someone else duplicates the effort. Duplicate untested code is better than common untested code (you break only one project).

Each common library has to have a dedicated maintainer (can be offset by a really good test suite in a smaller team).

Each project should check out the version of the library that is known to work with it. This means a developer does not have to get pulled away to update API usage, as the common code gets updated. Which it will be. Every non-trivial piece of code evolves over months and years.

Thank you for your thoughts on this!

答:
You have a competing set of goals here. First, a library of reusable components must be open enough that people from the other projects can easily add to it (or submit components to it). If it's too difficult for them to do that, they'll build their own libraries, and ignore the common one, leading to a lot of duplicate code and wasted effort. On the other hand, you want to control the development of the library enough that you can ensure its quality.

I've been in this position. There's no easy answer. However, there are some heuristics that can help.

Treat the library as an internal project. Release it on regular intervals. Ensure that it has a well-defined release procedure, complete with unit tests and quality assurance. And, most important, release often, so that new submissions to the library show up in the product frequently.

Provide incentives for people to contribute to the library, rather than just making their own internal libraries.

Make it easy for people to contribute to the library, and make the criteria clear-cut and well-defined (e.g., new classes must come with unit tests and documentation).

Put one or two developers in charge of the library, and (IMPORTANT!) allocate time for them to work on it. A library that is treated as an afterthought will quickly become an afterthought.

In short, model the development and maintenance of your internal library after a successful open source library project.

OwnerDisplayName: Andrei

Title: Common libraries in a large team

LastEditorDisplayName: Roger Lipscombe

OwnerDisplayName: Brian Clapper

LastEditorDisplayName: Brian Clapper


问:
How to subset a pandas dataframe on value_counts?
I have the following pandas dataframe

```<code>import pandas as pd
df = pd.read_csv("filename1.csv")

df
    column1  column2   column3
0        10       A          1
1        15       A          1
2        19       B          1
3      5071       B          0
4      5891       B          0
5      3210       B          0
6        12       B          2
7        13       C          2
8        20       C          0
9         5       C          3
10        9       C          3
</code>```

Now, using the function value_counts() will give me the counts of each value in a certain column, e.g.

```<code>df.column3.value_counts()

1   3
2   2
3   2
</code>```

However, I would like to subset a pandas dataframe based on the number of values in a given column. For example, in the above dataframe df, I would like to subset on rows with 3 or more unique values (excluding 0). In this case, the resulting dataframe would be

```<code>df
    column1  column2   column3
0        10       A          1
1        15       A          1
2        19       B          1
</code>```

As the rows for values 2 and 3 only had two rows, i.e. 2, 3 only occurred twice in column3. What is the pandas way to do this?

答:
Alternative solution:</p>

```<code>In [132]: cnt = df.column3.value_counts()

In [133]: cnt
Out[133]:
0    4
1    3
3    2
2    2
Name: column3, dtype: int64

In [134]: v = cnt[(cnt.index != 0) & (cnt >= 3)].index.values

In [135]: v
Out[135]: array([1], dtype=int64)

In [136]: df.query("column3 in @v")
Out[136]:
   column1 column2  column3
0       10       A        1
1       15       A        1
2       19       B        1
</code>```

Body:
I have the following pandas dataframe

import pandas as pd
df = pd.read_csv("filename1.csv")

df
    column1  column2   column3
0        10       A          1
1        15       A          1
2        19       B          1
3      5071       B          0
4      5891       B          0
5      3210       B          0
6        12       B          2
7        13       C          2
8        20       C          0
9         5       C          3
10        9       C          3

Now, using the function value_counts() will give me the counts of each value in a certain column, e.g.

df.column3.value_counts()

1   3
2   2
3   2

However, I would like to subset a pandas dataframe based on the number of values in a given column. For example, in the above dataframe df, I would like to subset on rows with 3 or more unique values (excluding 0). In this case, the resulting dataframe would be

df
    column1  column2   column3
0        10       A          1
1        15       A          1
2        19       B          1

As the rows for values 2 and 3 only had two rows, i.e. 2, 3 only occurred twice in column3. What is the pandas way to do this?

Body:
Alternative solution:

In [132]: cnt = df.column3.value_counts()

In [133]: cnt
Out[133]:
0    4
1    3
3    2
2    2
Name: column3, dtype: int64

In [134]: v = cnt[(cnt.index != 0) & (cnt >= 3)].index.values

In [135]: v
Out[135]: array([1], dtype=int64)

In [136]: df.query("column3 in @v")
Out[136]:
   column1 column2  column3
0       10       A        1
1       15       A        1
2       19       B        1


问:
pyspark.sql.utils.AnalysisException: u'Path does not exist
I am running a spark job with amazon emr using the standard hdfs, not S3 to store my files. I have a hive table in hdfs://user/hive/warehouse/ but it cannot be found when my spark job is ran. I configured the spark property spark.sql.warehouse.dir to reflect that of my hdfs directory and while the yarn logs do say:

```<code>17/03/28 19:54:05 INFO SharedState: Warehouse path is 'hdfs://user/hive/warehouse/'.
</code>```

later on in the logs it says(full log at end of page):

```<code>LogType:stdout
Log Upload Time:Tue Mar 28 19:54:15 +0000 2017
LogLength:854
Log Contents:
Traceback (most recent call last):
  File "test.py", line 25, in <module>
    parquet_example(spark)
  File "test.py", line 9, in parquet_example
    tests = spark.read.parquet("test.parquet")
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/pyspark.zip/pyspark/sql/readwriter.py", line 274, in parquet
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
pyspark.sql.utils.AnalysisException: u'Path does not exist: hdfs://ip-xxx-xx-xx-xxx.ec2.internal:8020/user/hadoop/test.parquet;'
End of LogType:stdout
</code>```

What am i doing wrong for there to be a mismatch in the path?

Here is my hdfs directory for hive/warehouse:

```<code>hdfs dfs -ls

/user/hive/warehouse
Found 1 items
drwxrwxrwt   - hadoop hadoop          0 2017-03-28 18:50 /user/hive/warehouse/test
</code>```

here is what /user/hadoop/ gives me:

```<code>hdfs dfs -ls /user/hadoop/
Found 2 items
drwxr-xr-x   - hadoop hadoop          0 2017-03-28 16:53 /user/hadoop/.hiveJars
drwxr-xr-x   - hadoop hadoop          0 2017-03-28 19:54 /user/hadoop/.sparkStaging
</code>```

And here is my spark job in python:

```<code>from __future__ import print_function
from pyspark.sql import SparkSession
from pyspark.sql import Row

def parquet_example(spark):
    tests = spark.read.parquet("test.parquet")
    tests.createOrReplaceTempView("tests")
    tests_result = spark.sql("SELECT * FROM test")
    tests_result.show()

if __name__ == "__main__":
    warehouseLocation = "hdfs://user/hive/warehouse/"
    spark = SparkSession.builder.appName("example").config("spark.sql.warehouse.dir", warehouseLocation).enableHiveSupport().getOrCreate()

    parquet_example(spark)
    spark.stop()
</code>```

full yarn log:

```<code>Container: container_1490717578939_0012_01_000001 on ip-xxx-xx-xx-xxx.ec2.internal_8041
=========================================================================================
LogType:stderr
Log Upload Time:Tue Mar 28 19:54:15 +0000 2017
LogLength:14054
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/filecache/131/__spark_libs__713193244228500015.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/03/28 19:54:01 INFO SignalUtils: Registered signal handler for TERM
17/03/28 19:54:01 INFO SignalUtils: Registered signal handler for HUP
17/03/28 19:54:01 INFO SignalUtils: Registered signal handler for INT
17/03/28 19:54:02 INFO ApplicationMaster: Preparing Local resources
17/03/28 19:54:03 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1490717578939_0012_000001
17/03/28 19:54:03 INFO SecurityManager: Changing view acls to: yarn,hadoop
17/03/28 19:54:03 INFO SecurityManager: Changing modify acls to: yarn,hadoop
17/03/28 19:54:03 INFO SecurityManager: Changing view acls groups to:
17/03/28 19:54:03 INFO SecurityManager: Changing modify acls groups to:
17/03/28 19:54:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
17/03/28 19:54:03 INFO ApplicationMaster: Starting the user application in a separate Thread
17/03/28 19:54:03 INFO ApplicationMaster: Waiting for spark context initialization...
17/03/28 19:54:03 INFO SparkContext: Running Spark version 2.1.0
17/03/28 19:54:03 INFO SecurityManager: Changing view acls to: yarn,hadoop
17/03/28 19:54:03 INFO SecurityManager: Changing modify acls to: yarn,hadoop
17/03/28 19:54:03 INFO SecurityManager: Changing view acls groups to:
17/03/28 19:54:03 INFO SecurityManager: Changing modify acls groups to:
17/03/28 19:54:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
17/03/28 19:54:03 INFO Utils: Successfully started service 'sparkDriver' on port 33579.
17/03/28 19:54:04 INFO SparkEnv: Registering MapOutputTracker
17/03/28 19:54:04 INFO SparkEnv: Registering BlockManagerMaster
17/03/28 19:54:04 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
17/03/28 19:54:04 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
17/03/28 19:54:04 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/blockmgr-f3713d64-91da-4cb5-9b55-d4a18c607a74
17/03/28 19:54:04 INFO DiskBlockManager: Created local directory at /mnt1/yarn/usercache/hadoop/appcache/application_1490717578939_0012/blockmgr-634c7d4b-026c-4df7-abf4-7846bd7fc958
17/03/28 19:54:04 INFO DiskBlockManager: Created local directory at /mnt2/yarn/usercache/hadoop/appcache/application_1490717578939_0012/blockmgr-19f0a265-755a-42f0-9282-1e3d98a57ab1
17/03/28 19:54:04 INFO MemoryStore: MemoryStore started with capacity 414.4 MB
17/03/28 19:54:04 INFO SparkEnv: Registering OutputCommitCoordinator
17/03/28 19:54:04 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/03/28 19:54:04 INFO Utils: Successfully started service 'SparkUI' on port 37056.
17/03/28 19:54:04 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://xxx.xx.xx.xxx:37056
17/03/28 19:54:04 INFO YarnClusterScheduler: Created YarnClusterScheduler
17/03/28 19:54:04 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1490717578939_0012 and attemptId Some(appattempt_1490717578939_0012_000001)
17/03/28 19:54:04 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
17/03/28 19:54:04 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34414.
17/03/28 19:54:04 INFO NettyBlockTransferService: Server created on xxx.xx.xx.xxx:34414
17/03/28 19:54:04 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
17/03/28 19:54:04 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, xxx.xx.xx.xxx, 34414, None)
17/03/28 19:54:04 INFO BlockManagerMasterEndpoint: Registering block manager xxx.xx.xx.xxx:34414 with 414.4 MB RAM, BlockManagerId(driver, xxx.xx.xx.xxx, 34414, None)
17/03/28 19:54:04 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, xxx.xx.xx.xxx, 34414, None)
17/03/28 19:54:04 INFO BlockManager: external shuffle service port = 7337
17/03/28 19:54:04 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, xxx.xx.xx.xxx, 34414, None)
17/03/28 19:54:05 INFO EventLoggingListener: Logging events to hdfs:///var/log/spark/apps/application_1490717578939_0012_1
17/03/28 19:54:05 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
17/03/28 19:54:05 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
17/03/28 19:54:05 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/03/28 19:54:05 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done
17/03/28 19:54:05 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@xxx.xx.xx.xxx:33579)
17/03/28 19:54:05 INFO ApplicationMaster:
===============================================================================
YARN executor launch context:
  env:
    CLASSPATH -> /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*<CPS>{{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>/usr/lib/hadoop-lzo/lib/*<CPS>/usr/share/aws/emr/emrfs/conf<CPS>/usr/share/aws/emr/emrfs/lib/*<CPS>/usr/share/aws/emr/emrfs/auxlib/*<CPS>/usr/share/aws/emr/lib/*<CPS>/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar<CPS>/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar<CPS>/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar<CPS>/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar<CPS>/usr/lib/spark/yarn/lib/datanucleus-core.jar<CPS>/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar<CPS>/usr/share/aws/emr/cloudwatch-sink/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*<CPS>/usr/lib/hadoop-lzo/lib/*<CPS>/usr/share/aws/emr/emrfs/conf<CPS>/usr/share/aws/emr/emrfs/lib/*<CPS>/usr/share/aws/emr/emrfs/auxlib/*<CPS>/usr/share/aws/emr/lib/*<CPS>/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar<CPS>/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar<CPS>/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar<CPS>/usr/share/aws/emr/cloudwatch-sink/lib/*
    SPARK_YARN_STAGING_DIR -> hdfs://ip-xxx-xx-xx-xxx.ec2.internal:8020/user/hadoop/.sparkStaging/application_1490717578939_0012
    SPARK_USER -> hadoop
    SPARK_YARN_MODE -> true
    PYTHONPATH -> {{PWD}}/pyspark.zip<CPS>{{PWD}}/py4j-0.10.4-src.zip

  command:
    LD_LIBRARY_PATH="/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:$LD_LIBRARY_PATH" \
      {{JAVA_HOME}}/bin/java \
      -server \
      -Xmx5120m \
      '-verbose:gc' \
      '-XX:+PrintGCDetails' \
      '-XX:+PrintGCDateStamps' \
      '-XX:+UseConcMarkSweepGC' \
      '-XX:CMSInitiatingOccupancyFraction=70' \
      '-XX:MaxHeapFreeRatio=70' \
      '-XX:+CMSClassUnloadingEnabled' \
      '-XX:OnOutOfMemoryError=kill -9 %p' \
      -Djava.io.tmpdir={{PWD}}/tmp \
      '-Dspark.history.ui.port=18080' \
      -Dspark.yarn.app.container.log.dir=<LOG_DIR> \
      org.apache.spark.executor.CoarseGrainedExecutorBackend \
      --driver-url \
      spark://CoarseGrainedScheduler@xxx.xx.xx.xxx:33579 \
      --executor-id \
      <executorId> \
      --hostname \
      <hostname> \
      --cores \
      2 \
      --app-id \
      application_1490717578939_0012 \
      --user-class-path \
      file:$PWD/__app__.jar \
      1><LOG_DIR>/stdout \
      2><LOG_DIR>/stderr

  resources:
    py4j-0.10.4-src.zip -> resource { scheme: "hdfs" host: "ip-xxx-xx-xx-xxx.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1490717578939_0012/py4j-0.10.4-src.zip" } size: 74096 timestamp: 1490730839170 type: FILE visibility: PRIVATE
    __spark_conf__ -> resource { scheme: "hdfs" host: "ip-xxx-xx-xx-xxx.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1490717578939_0012/__spark_conf__.zip" } size: 75741 timestamp: 1490730839402 type: ARCHIVE visibility: PRIVATE
    pyspark.zip -> resource { scheme: "hdfs" host: "ip-xxx-xx-xx-xxx.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1490717578939_0012/pyspark.zip" } size: 452353 timestamp: 1490730838849 type: FILE visibility: PRIVATE
    __spark_libs__ -> resource { scheme: "hdfs" host: "ip-xxx-xx-xx-xxx.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1490717578939_0012/__spark_libs__713193244228500015.zip" } size: 196686961 timestamp: 1490730836856 type: ARCHIVE visibility: PRIVATE
    hive-site.xml -> resource { scheme: "hdfs" host: "ip-xxx-xx-xx-xxx.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1490717578939_0012/hive-site.xml" } size: 2375 timestamp: 1490730837023 type: FILE visibility: PRIVATE

===============================================================================
17/03/28 19:54:05 INFO RMProxy: Connecting to ResourceManager at ip-xxx-xx-xx-xxx.ec2.internal/xxx-xx-xx-xxx:8030
17/03/28 19:54:05 INFO YarnRMClient: Registering the ApplicationMaster
17/03/28 19:54:05 INFO SharedState: Warehouse path is 'hdfs://user/hive/warehouse/'.
17/03/28 19:54:05 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
17/03/28 19:54:05 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
17/03/28 19:54:05 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
17/03/28 19:54:06 INFO metastore: Trying to connect to metastore with URI thrift://ip-xxx-xx-xx-xxx.ec2.internal:9083
17/03/28 19:54:06 INFO metastore: Connected to metastore.
17/03/28 19:54:06 INFO SessionState: Created local directory: /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/tmp/yarn
17/03/28 19:54:06 INFO SessionState: Created local directory: /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/tmp/5f653144-e990-45b0-ba73-cdb4d10e9f7a_resources
17/03/28 19:54:06 INFO SessionState: Created HDFS directory: /tmp/hive/hadoop/5f653144-e990-45b0-ba73-cdb4d10e9f7a
17/03/28 19:54:06 INFO SessionState: Created local directory: /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/tmp/yarn/5f653144-e990-45b0-ba73-cdb4d10e9f7a
17/03/28 19:54:06 INFO SessionState: Created HDFS directory: /tmp/hive/hadoop/5f653144-e990-45b0-ba73-cdb4d10e9f7a/_tmp_space.db
17/03/28 19:54:06 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is hdfs://user/hive/warehouse/
17/03/28 19:54:06 ERROR ApplicationMaster: User application exited with status 1
17/03/28 19:54:06 INFO ApplicationMaster: Final app status: FAILED, exitCode: 1, (reason: User application exited with status 1)
17/03/28 19:54:06 INFO SparkContext: Invoking stop() from shutdown hook
17/03/28 19:54:06 INFO SparkUI: Stopped Spark web UI at http://xxx.xx.xx.xxx:37056
17/03/28 19:54:06 INFO YarnClusterSchedulerBackend: Shutting down all executors
17/03/28 19:54:06 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
17/03/28 19:54:06 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
17/03/28 19:54:06 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/03/28 19:54:06 INFO MemoryStore: MemoryStore cleared
17/03/28 19:54:06 INFO BlockManager: BlockManager stopped
17/03/28 19:54:06 INFO BlockManagerMaster: BlockManagerMaster stopped
17/03/28 19:54:06 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/03/28 19:54:06 INFO SparkContext: Successfully stopped SparkContext
17/03/28 19:54:06 INFO ShutdownHookManager: Shutdown hook called
17/03/28 19:54:06 INFO ShutdownHookManager: Deleting directory /mnt1/yarn/usercache/hadoop/appcache/application_1490717578939_0012/spark-3a6db594-2b44-47fe-8e48-4220b93e789a
17/03/28 19:54:06 INFO ShutdownHookManager: Deleting directory /mnt2/yarn/usercache/hadoop/appcache/application_1490717578939_0012/spark-a54516f0-48be-4fdb-899b-bbee998468b1
17/03/28 19:54:06 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/spark-552e3cae-c119-47a5-9c63-34d4df59d072
17/03/28 19:54:06 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/spark-552e3cae-c119-47a5-9c63-34d4df59d072/pyspark-a0240093-16c6-43e4-8f2c-dcef309afe97
End of LogType:stderr

LogType:stdout
Log Upload Time:Tue Mar 28 19:54:15 +0000 2017
LogLength:854
Log Contents:
Traceback (most recent call last):
  File "test.py", line 25, in <module>
    parquet_example(spark)
  File "test.py", line 9, in parquet_example
    tests = spark.read.parquet("test.parquet")
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/pyspark.zip/pyspark/sql/readwriter.py", line 274, in parquet
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
pyspark.sql.utils.AnalysisException: u'Path does not exist: hdfs://ip-xxx-xx-xx-xxx.ec2.internal:8020/user/hadoop/test.parquet;'
End of LogType:stdout
</code>```

答:
The function parquet_example in the question would create a DataFrame from the parquet file test.parquet and query from it by creating a temporary view.

From the comments:

Since the Hive table named test already exists, directly query the table with the created SparkSession

```<code>warehouseLocation = "hdfs://user/hive/warehouse/"
spark = SparkSession \
    .builder \
    .appName("example") \
    .config("spark.sql.warehouse.dir", warehouseLocation) \
    .enableHiveSupport() \
    .getOrCreate()
spark.sql("SELECT * FROM test").show()
</code>```

Body:
I am running a spark job with amazon emr using the standard hdfs, not S3 to store my files. I have a hive table in hdfs://user/hive/warehouse/ but it cannot be found when my spark job is ran. I configured the spark property spark.sql.warehouse.dir to reflect that of my hdfs directory and while the yarn logs do say:

17/03/28 19:54:05 INFO SharedState: Warehouse path is 'hdfs://user/hive/warehouse/'.

later on in the logs it says(full log at end of page):

LogType:stdout
Log Upload Time:Tue Mar 28 19:54:15 +0000 2017
LogLength:854
Log Contents:
Traceback (most recent call last):
  File "test.py", line 25, in <module>
    parquet_example(spark)
  File "test.py", line 9, in parquet_example
    tests = spark.read.parquet("test.parquet")
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/pyspark.zip/pyspark/sql/readwriter.py", line 274, in parquet
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
pyspark.sql.utils.AnalysisException: u'Path does not exist: hdfs://ip-xxx-xx-xx-xxx.ec2.internal:8020/user/hadoop/test.parquet;'
End of LogType:stdout

What am i doing wrong for there to be a mismatch in the path?

Here is my hdfs directory for hive/warehouse:

hdfs dfs -ls

/user/hive/warehouse
Found 1 items
drwxrwxrwt   - hadoop hadoop          0 2017-03-28 18:50 /user/hive/warehouse/test

here is what /user/hadoop/ gives me:

hdfs dfs -ls /user/hadoop/
Found 2 items
drwxr-xr-x   - hadoop hadoop          0 2017-03-28 16:53 /user/hadoop/.hiveJars
drwxr-xr-x   - hadoop hadoop          0 2017-03-28 19:54 /user/hadoop/.sparkStaging

And here is my spark job in python:

from __future__ import print_function
from pyspark.sql import SparkSession
from pyspark.sql import Row

def parquet_example(spark):
    tests = spark.read.parquet("test.parquet")
    tests.createOrReplaceTempView("tests")
    tests_result = spark.sql("SELECT * FROM test")
    tests_result.show()

if __name__ == "__main__":
    warehouseLocation = "hdfs://user/hive/warehouse/"
    spark = SparkSession.builder.appName("example").config("spark.sql.warehouse.dir", warehouseLocation).enableHiveSupport().getOrCreate()

    parquet_example(spark)
    spark.stop()

full yarn log:

Container: container_1490717578939_0012_01_000001 on ip-xxx-xx-xx-xxx.ec2.internal_8041
=========================================================================================
LogType:stderr
Log Upload Time:Tue Mar 28 19:54:15 +0000 2017
LogLength:14054
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/filecache/131/__spark_libs__713193244228500015.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
17/03/28 19:54:01 INFO SignalUtils: Registered signal handler for TERM
17/03/28 19:54:01 INFO SignalUtils: Registered signal handler for HUP
17/03/28 19:54:01 INFO SignalUtils: Registered signal handler for INT
17/03/28 19:54:02 INFO ApplicationMaster: Preparing Local resources
17/03/28 19:54:03 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1490717578939_0012_000001
17/03/28 19:54:03 INFO SecurityManager: Changing view acls to: yarn,hadoop
17/03/28 19:54:03 INFO SecurityManager: Changing modify acls to: yarn,hadoop
17/03/28 19:54:03 INFO SecurityManager: Changing view acls groups to:
17/03/28 19:54:03 INFO SecurityManager: Changing modify acls groups to:
17/03/28 19:54:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
17/03/28 19:54:03 INFO ApplicationMaster: Starting the user application in a separate Thread
17/03/28 19:54:03 INFO ApplicationMaster: Waiting for spark context initialization...
17/03/28 19:54:03 INFO SparkContext: Running Spark version 2.1.0
17/03/28 19:54:03 INFO SecurityManager: Changing view acls to: yarn,hadoop
17/03/28 19:54:03 INFO SecurityManager: Changing modify acls to: yarn,hadoop
17/03/28 19:54:03 INFO SecurityManager: Changing view acls groups to:
17/03/28 19:54:03 INFO SecurityManager: Changing modify acls groups to:
17/03/28 19:54:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
17/03/28 19:54:03 INFO Utils: Successfully started service 'sparkDriver' on port 33579.
17/03/28 19:54:04 INFO SparkEnv: Registering MapOutputTracker
17/03/28 19:54:04 INFO SparkEnv: Registering BlockManagerMaster
17/03/28 19:54:04 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
17/03/28 19:54:04 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
17/03/28 19:54:04 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/blockmgr-f3713d64-91da-4cb5-9b55-d4a18c607a74
17/03/28 19:54:04 INFO DiskBlockManager: Created local directory at /mnt1/yarn/usercache/hadoop/appcache/application_1490717578939_0012/blockmgr-634c7d4b-026c-4df7-abf4-7846bd7fc958
17/03/28 19:54:04 INFO DiskBlockManager: Created local directory at /mnt2/yarn/usercache/hadoop/appcache/application_1490717578939_0012/blockmgr-19f0a265-755a-42f0-9282-1e3d98a57ab1
17/03/28 19:54:04 INFO MemoryStore: MemoryStore started with capacity 414.4 MB
17/03/28 19:54:04 INFO SparkEnv: Registering OutputCommitCoordinator
17/03/28 19:54:04 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
17/03/28 19:54:04 INFO Utils: Successfully started service 'SparkUI' on port 37056.
17/03/28 19:54:04 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://xxx.xx.xx.xxx:37056
17/03/28 19:54:04 INFO YarnClusterScheduler: Created YarnClusterScheduler
17/03/28 19:54:04 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1490717578939_0012 and attemptId Some(appattempt_1490717578939_0012_000001)
17/03/28 19:54:04 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
17/03/28 19:54:04 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34414.
17/03/28 19:54:04 INFO NettyBlockTransferService: Server created on xxx.xx.xx.xxx:34414
17/03/28 19:54:04 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
17/03/28 19:54:04 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, xxx.xx.xx.xxx, 34414, None)
17/03/28 19:54:04 INFO BlockManagerMasterEndpoint: Registering block manager xxx.xx.xx.xxx:34414 with 414.4 MB RAM, BlockManagerId(driver, xxx.xx.xx.xxx, 34414, None)
17/03/28 19:54:04 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, xxx.xx.xx.xxx, 34414, None)
17/03/28 19:54:04 INFO BlockManager: external shuffle service port = 7337
17/03/28 19:54:04 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, xxx.xx.xx.xxx, 34414, None)
17/03/28 19:54:05 INFO EventLoggingListener: Logging events to hdfs:///var/log/spark/apps/application_1490717578939_0012_1
17/03/28 19:54:05 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
17/03/28 19:54:05 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
17/03/28 19:54:05 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
17/03/28 19:54:05 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done
17/03/28 19:54:05 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@xxx.xx.xx.xxx:33579)
17/03/28 19:54:05 INFO ApplicationMaster:
===============================================================================
YARN executor launch context:
  env:
    CLASSPATH -> /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*<CPS>{{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>/usr/lib/hadoop-lzo/lib/*<CPS>/usr/share/aws/emr/emrfs/conf<CPS>/usr/share/aws/emr/emrfs/lib/*<CPS>/usr/share/aws/emr/emrfs/auxlib/*<CPS>/usr/share/aws/emr/lib/*<CPS>/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar<CPS>/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar<CPS>/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar<CPS>/usr/lib/spark/yarn/lib/datanucleus-api-jdo.jar<CPS>/usr/lib/spark/yarn/lib/datanucleus-core.jar<CPS>/usr/lib/spark/yarn/lib/datanucleus-rdbms.jar<CPS>/usr/share/aws/emr/cloudwatch-sink/lib/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*<CPS>/usr/lib/hadoop-lzo/lib/*<CPS>/usr/share/aws/emr/emrfs/conf<CPS>/usr/share/aws/emr/emrfs/lib/*<CPS>/usr/share/aws/emr/emrfs/auxlib/*<CPS>/usr/share/aws/emr/lib/*<CPS>/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar<CPS>/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar<CPS>/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar<CPS>/usr/share/aws/emr/cloudwatch-sink/lib/*
    SPARK_YARN_STAGING_DIR -> hdfs://ip-xxx-xx-xx-xxx.ec2.internal:8020/user/hadoop/.sparkStaging/application_1490717578939_0012
    SPARK_USER -> hadoop
    SPARK_YARN_MODE -> true
    PYTHONPATH -> {{PWD}}/pyspark.zip<CPS>{{PWD}}/py4j-0.10.4-src.zip

  command:
    LD_LIBRARY_PATH="/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:$LD_LIBRARY_PATH" \
      {{JAVA_HOME}}/bin/java \
      -server \
      -Xmx5120m \
      '-verbose:gc' \
      '-XX:+PrintGCDetails' \
      '-XX:+PrintGCDateStamps' \
      '-XX:+UseConcMarkSweepGC' \
      '-XX:CMSInitiatingOccupancyFraction=70' \
      '-XX:MaxHeapFreeRatio=70' \
      '-XX:+CMSClassUnloadingEnabled' \
      '-XX:OnOutOfMemoryError=kill -9 %p' \
      -Djava.io.tmpdir={{PWD}}/tmp \
      '-Dspark.history.ui.port=18080' \
      -Dspark.yarn.app.container.log.dir=<LOG_DIR> \
      org.apache.spark.executor.CoarseGrainedExecutorBackend \
      --driver-url \
      spark://CoarseGrainedScheduler@xxx.xx.xx.xxx:33579 \
      --executor-id \
      <executorId> \
      --hostname \
      <hostname> \
      --cores \
      2 \
      --app-id \
      application_1490717578939_0012 \
      --user-class-path \
      file:$PWD/__app__.jar \
      1><LOG_DIR>/stdout \
      2><LOG_DIR>/stderr

  resources:
    py4j-0.10.4-src.zip -> resource { scheme: "hdfs" host: "ip-xxx-xx-xx-xxx.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1490717578939_0012/py4j-0.10.4-src.zip" } size: 74096 timestamp: 1490730839170 type: FILE visibility: PRIVATE
    __spark_conf__ -> resource { scheme: "hdfs" host: "ip-xxx-xx-xx-xxx.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1490717578939_0012/__spark_conf__.zip" } size: 75741 timestamp: 1490730839402 type: ARCHIVE visibility: PRIVATE
    pyspark.zip -> resource { scheme: "hdfs" host: "ip-xxx-xx-xx-xxx.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1490717578939_0012/pyspark.zip" } size: 452353 timestamp: 1490730838849 type: FILE visibility: PRIVATE
    __spark_libs__ -> resource { scheme: "hdfs" host: "ip-xxx-xx-xx-xxx.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1490717578939_0012/__spark_libs__713193244228500015.zip" } size: 196686961 timestamp: 1490730836856 type: ARCHIVE visibility: PRIVATE
    hive-site.xml -> resource { scheme: "hdfs" host: "ip-xxx-xx-xx-xxx.ec2.internal" port: 8020 file: "/user/hadoop/.sparkStaging/application_1490717578939_0012/hive-site.xml" } size: 2375 timestamp: 1490730837023 type: FILE visibility: PRIVATE

===============================================================================
17/03/28 19:54:05 INFO RMProxy: Connecting to ResourceManager at ip-xxx-xx-xx-xxx.ec2.internal/xxx-xx-xx-xxx:8030
17/03/28 19:54:05 INFO YarnRMClient: Registering the ApplicationMaster
17/03/28 19:54:05 INFO SharedState: Warehouse path is 'hdfs://user/hive/warehouse/'.
17/03/28 19:54:05 INFO Utils: Using initial executors = 0, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
17/03/28 19:54:05 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
17/03/28 19:54:05 INFO HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
17/03/28 19:54:06 INFO metastore: Trying to connect to metastore with URI thrift://ip-xxx-xx-xx-xxx.ec2.internal:9083
17/03/28 19:54:06 INFO metastore: Connected to metastore.
17/03/28 19:54:06 INFO SessionState: Created local directory: /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/tmp/yarn
17/03/28 19:54:06 INFO SessionState: Created local directory: /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/tmp/5f653144-e990-45b0-ba73-cdb4d10e9f7a_resources
17/03/28 19:54:06 INFO SessionState: Created HDFS directory: /tmp/hive/hadoop/5f653144-e990-45b0-ba73-cdb4d10e9f7a
17/03/28 19:54:06 INFO SessionState: Created local directory: /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/tmp/yarn/5f653144-e990-45b0-ba73-cdb4d10e9f7a
17/03/28 19:54:06 INFO SessionState: Created HDFS directory: /tmp/hive/hadoop/5f653144-e990-45b0-ba73-cdb4d10e9f7a/_tmp_space.db
17/03/28 19:54:06 INFO HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is hdfs://user/hive/warehouse/
17/03/28 19:54:06 ERROR ApplicationMaster: User application exited with status 1
17/03/28 19:54:06 INFO ApplicationMaster: Final app status: FAILED, exitCode: 1, (reason: User application exited with status 1)
17/03/28 19:54:06 INFO SparkContext: Invoking stop() from shutdown hook
17/03/28 19:54:06 INFO SparkUI: Stopped Spark web UI at http://xxx.xx.xx.xxx:37056
17/03/28 19:54:06 INFO YarnClusterSchedulerBackend: Shutting down all executors
17/03/28 19:54:06 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
17/03/28 19:54:06 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
 services=List(),
 started=false)
17/03/28 19:54:06 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/03/28 19:54:06 INFO MemoryStore: MemoryStore cleared
17/03/28 19:54:06 INFO BlockManager: BlockManager stopped
17/03/28 19:54:06 INFO BlockManagerMaster: BlockManagerMaster stopped
17/03/28 19:54:06 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/03/28 19:54:06 INFO SparkContext: Successfully stopped SparkContext
17/03/28 19:54:06 INFO ShutdownHookManager: Shutdown hook called
17/03/28 19:54:06 INFO ShutdownHookManager: Deleting directory /mnt1/yarn/usercache/hadoop/appcache/application_1490717578939_0012/spark-3a6db594-2b44-47fe-8e48-4220b93e789a
17/03/28 19:54:06 INFO ShutdownHookManager: Deleting directory /mnt2/yarn/usercache/hadoop/appcache/application_1490717578939_0012/spark-a54516f0-48be-4fdb-899b-bbee998468b1
17/03/28 19:54:06 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/spark-552e3cae-c119-47a5-9c63-34d4df59d072
17/03/28 19:54:06 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/spark-552e3cae-c119-47a5-9c63-34d4df59d072/pyspark-a0240093-16c6-43e4-8f2c-dcef309afe97
End of LogType:stderr

LogType:stdout
Log Upload Time:Tue Mar 28 19:54:15 +0000 2017
LogLength:854
Log Contents:
Traceback (most recent call last):
  File "test.py", line 25, in <module>
    parquet_example(spark)
  File "test.py", line 9, in parquet_example
    tests = spark.read.parquet("test.parquet")
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/pyspark.zip/pyspark/sql/readwriter.py", line 274, in parquet
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
  File "/mnt/yarn/usercache/hadoop/appcache/application_1490717578939_0012/container_1490717578939_0012_01_000001/pyspark.zip/pyspark/sql/utils.py", line 69, in deco
pyspark.sql.utils.AnalysisException: u'Path does not exist: hdfs://ip-xxx-xx-xx-xxx.ec2.internal:8020/user/hadoop/test.parquet;'
End of LogType:stdout

Body:
The function parquet_example in the question would create a DataFrame from the parquet file test.parquet and query from it by creating a temporary view.

From the comments:

Since the Hive table named test already exists, directly query the table with the created SparkSession

warehouseLocation = "hdfs://user/hive/warehouse/"
spark = SparkSession \
    .builder \
    .appName("example") \
    .config("spark.sql.warehouse.dir", warehouseLocation) \
    .enableHiveSupport() \
    .getOrCreate()
spark.sql("SELECT * FROM test").show()


问:
Manually creating interactive PDFs
For reasons tedious to explain, I need to create simple PDFs from a C# application without using any third-party libraries. The PDFs in question need to include text and line drawings, and if possible should include images, radio buttons, and text fields. I've been reading Adobe's documentation, which is for the most part straightforward, but so far have been unable to figure out how to make interactive fields actually appear on the document.

For example, consider the partial test PDF document described below:

```<code>1 0 obj
  << /Type /Catalog
     /Pages 2 0 R
     /AcroForm 6 0 R
  >>
endobj

...

6 0 obj
  << /Fields [ 7 0 R ]
     /DR 5 0 R
  >>
endobj

7 0 obj
  << /FT /Btn
     /Ff 65536
     /Kids [ 8 0 R 9 0 R ]
  >>
endobj

8 0 obj
  << /Parent 7 0 R
     /AS /FieldA
     /AP 10 0 R
  >>
endobj

9 0 obj
  << /Parent 8 0 R
     /AS /FieldB
     /AP 10 0 R
  >>
endobj

10 0 obj
  << /N
    << /FieldA 11 0 R
       /FieldB 12 0 R
       /Off 13 0 R
    >>
  >>
endobj

11 0 obj
  << /Length  59 >>
stream
  BT
    /F1 12 Tf
    0 0 Td
    (Field A) Tj
  ET
endstream
endobj

...
</code>```

Where 5 0 R references a resource dictionary containing the font /F1 and 12 0 R and 13 0 R reference streams similar to 11 0 R with the string replaced by Field B and Off respectively.

What changes do I need to make to the document to cause the fields thus defined to actually appear on a page?

Edit: Since mkl asked, here's the whole pdf (extra line breaks removed to save space)

```<code>%PDF-1.7
%¥±ë

1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj
2 0 obj << /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >> endobj
3 0 obj << /Type /Page /Parent 2 0 R /Resources 5 0 R /Contents 4 0 R >> endobj

4 0 obj << /Length 39 >>
stream
BT /F1 12 Tf 0 0 Td (Hello World) Tj ET
endstream
endobj

5 0 obj << /Font << /F1 << /Type /Font /Subtype /Type1 /BaseFont /Helvetica >> >> >> endobj
6 0 obj << /Fields [ 7 0 R ] /DR 5 0 R >> endobj
7 0 obj << /FT /Btn /Ff 65536 /Kids [ 8 0 R 9 0 R ] >> endobj
8 0 obj << /Parent 7 0 R /AS /FieldA /AP 10 0 R >> endobj
9 0 obj << /Parent 7 0 R /AS /FieldB /AP 10 0 R >> endobj
10 0 obj << /N << /FieldA 11 0 R /FieldB 12 0 R /Off 13 0 R >> >> endobj

11 0 obj << /Length 59 >>
stream
BT /F1 12 Tf 0 0 Td (Field A) Tj ET
endstream
endobj

12 0 obj << /Length 39 >>
stream
BT /F1 12 Tf 0 0 Td (Field B) Tj ET
endstream
endobj

13 0 obj << /Length 39 >>
stream
BT /F1 12 Tf 0 0 Td (Off... ) Tj ET 94
endstream
endobj

xref
0 14
0000000000 65535 f
0000000018 00000 n
0000000068 00000 n
0000000150 00000 n
0000000233 00000 n
0000000327 00000 n
0000000420 00000 n
0000000470 00000 n
0000000533 00000 n
0000000592 00000 n
0000000651 00000 n
0000000727 00000 n
0000000821 00000 n
0000000917 00000 n
trailer << /Root 1 0 R /Size 5 >>
startxref
1013
%%EOF
</code>```

答: Your fields will need to include what are known as "appearances" which are XObjects that contain the drawing instructions used to create the visual appearance of the field. See section 12.5.5 of the PDF Specification. Alternatively, you can set the NeedAppearances property of the AcroForm entry in the document catalog to true. See section 12.7.2 of the PDF specification. Setting this to true will cause a conforming viewer to create the appearances of the fields based on their properties. However, you can't count on most viewers to do this.</p>

Title: Manually creating interactive PDFs

Body:
For reasons tedious to explain, I need to create simple PDFs from a C# application without using any third-party libraries. The PDFs in question need to include text and line drawings, and if possible should include images, radio buttons, and text fields. I've been reading Adobe's documentation, which is for the most part straightforward, but so far have been unable to figure out how to make interactive fields actually appear on the document.

For example, consider the partial test PDF document described below:

1 0 obj
  << /Type /Catalog
     /Pages 2 0 R
     /AcroForm 6 0 R
  >>
endobj

...

6 0 obj
  << /Fields [ 7 0 R ]
     /DR 5 0 R
  >>
endobj

7 0 obj
  << /FT /Btn
     /Ff 65536
     /Kids [ 8 0 R 9 0 R ]
  >>
endobj

8 0 obj
  << /Parent 7 0 R
     /AS /FieldA
     /AP 10 0 R
  >>
endobj

9 0 obj
  << /Parent 8 0 R
     /AS /FieldB
     /AP 10 0 R
  >>
endobj

10 0 obj
  << /N
    << /FieldA 11 0 R
       /FieldB 12 0 R
       /Off 13 0 R
    >>
  >>
endobj

11 0 obj
  << /Length  59 >>
stream
  BT
    /F1 12 Tf
    0 0 Td
    (Field A) Tj
  ET
endstream
endobj

...

Where 5 0 R references a resource dictionary containing the font /F1 and 12 0 R and 13 0 R reference streams similar to 11 0 R with the string replaced by Field B and Off respectively.

What changes do I need to make to the document to cause the fields thus defined to actually appear on a page?

Edit: Since mkl asked, here's the whole pdf (extra line breaks removed to save space)

%PDF-1.7
%¥±ë

1 0 obj << /Type /Catalog /Pages 2 0 R >> endobj
2 0 obj << /Type /Pages /Kids [3 0 R] /Count 1 /MediaBox [0 0 200 200] >> endobj
3 0 obj << /Type /Page /Parent 2 0 R /Resources 5 0 R /Contents 4 0 R >> endobj

4 0 obj << /Length 39 >>
stream
BT /F1 12 Tf 0 0 Td (Hello World) Tj ET
endstream
endobj

5 0 obj << /Font << /F1 << /Type /Font /Subtype /Type1 /BaseFont /Helvetica >> >> >> endobj
6 0 obj << /Fields [ 7 0 R ] /DR 5 0 R >> endobj
7 0 obj << /FT /Btn /Ff 65536 /Kids [ 8 0 R 9 0 R ] >> endobj
8 0 obj << /Parent 7 0 R /AS /FieldA /AP 10 0 R >> endobj
9 0 obj << /Parent 7 0 R /AS /FieldB /AP 10 0 R >> endobj
10 0 obj << /N << /FieldA 11 0 R /FieldB 12 0 R /Off 13 0 R >> >> endobj

11 0 obj << /Length 59 >>
stream
BT /F1 12 Tf 0 0 Td (Field A) Tj ET
endstream
endobj

12 0 obj << /Length 39 >>
stream
BT /F1 12 Tf 0 0 Td (Field B) Tj ET
endstream
endobj

13 0 obj << /Length 39 >>
stream
BT /F1 12 Tf 0 0 Td (Off... ) Tj ET 94
endstream
endobj

xref
0 14
0000000000 65535 f
0000000018 00000 n
0000000068 00000 n
0000000150 00000 n
0000000233 00000 n
0000000327 00000 n
0000000420 00000 n
0000000470 00000 n
0000000533 00000 n
0000000592 00000 n
0000000651 00000 n
0000000727 00000 n
0000000821 00000 n
0000000917 00000 n
trailer << /Root 1 0 R /Size 5 >>
startxref
1013
%%EOF


问:
How to use variables with MongoDB $lookup
Let's say I have 3 collections, cars, bikes, vehicles.

cars collection as:

```<code>{
  {
    "_id": "car1",
    "carBrand": "Audi",
    "color": "blue"
  },
  {
    "_id": "car2",
    "carBrand": "BMW",
    "color": "white"
  }
}
</code>```

bikes collection as:

```<code>{
  {
    "_id": "bike1",
    "bikeBrand": "Audi",
    "color": "red"
  },
  {
    "_id": "bike2",
    "carBrand": "BMW",
    "color": "white"
  }
}
</code>```

and the vehicles collection actually just has references to cars and bikes collections as

```<code>{
  {
    "_id": "vehicle1",
    "vehicleType": "cars",
    "vehicleId": "car1"
  },
  {
    "_id": "vehicle2",
    "vehicleType": "cars",
    "vehicleId": "car2"
  },
  {
    "_id": "vehicle3",
    "vehicleType": "bikes",
    "vehicleId": "bike1"
  },
  {
    "_id": "vehicle4",
    "vehicleType": "bikes",
    "vehicleId": "bike2"
  },

}
</code>```

I'd like to join the vehicles collection with cars and bikes collections. I tried to set "$vehicleType" as a variable to $lookup's from field. However it does not work as I expected. It simply does not join the tables. No errors.

```<code>db.collection.aggregate([{

  $lookup: {
      from: "$vehicleType",
      localField: "vehicleId",
      foreignField: "_id",
      as: "vehicleDetails"
  }

}]);
</code>```

I was expecting to have a result something like this

```<code>{
  {
    "_id": "vehicle1",
    "vehicleType": "cars",
    "vehicleId": "car1",
    "vehicleDetails": {
      "_id": "car1",
      "carBrand": "Audi",
      "color": ""
    }
  },
  {
    "_id": "vehicle2",
    "vehicleType": "cars",
    "vehicleId": "car2",
    "vehicleDetails":    {
      "_id": "car2",
      "carBrand": "BMW",
      "color": "white"

  },
  {
    "_id": "vehicle3",
    "vehicleType": "bikes",
    "vehicleId": "bike1",
    "vehicleDetails":    {
      "_id": "bike1",
      "bikeBrand": "Audi",
      "color": "red"
    }
  },
  {
    "_id": "vehicle4",
    "vehicleType": "bikes",
    "vehicleId": "bike2",
    "vehicleDetails":    {
      "_id": "bike2",
      "carBrand": "BMW",
      "color": "white"
    }
  },

}
</code>```

答:
If cars and bikes don't have common IDs, you can sequentially lookup in separate arrays, then combine them with $setUnion:

```<code>db.vehicles.aggregate([
  {$lookup: {
      from: "cars",
      localField: "vehicleId",
      foreignField: "_id",
      as: "carDetails"
  }},
  {$lookup: {
      from: "bikes",
      localField: "vehicleId",
      foreignField: "_id",
      as: "bikeDetails"
  }},
  {$project: {
     vehicleType: 1,
     vehicleId: 1,
     vehicleDetails:{$setUnion: [ "$carDetails", "$bikeDetails" ]}
  }},
  {$project: {
      carDetails:0,
      bikeDetails:0,
  }}
]);
</code>```

Otherwise you will need to use $facet to filter vehicles by type before lookup:

```<code>db.vehicles.aggregate([
   {
     $facet: {
         "cars": [
            {$match: {"vehicleType": "cars"}},
            {$lookup: {
               from: "cars",
               localField: "vehicleId",
               foreignField: "_id",
               as: "vehicleDetails"
             }},
         ],
         "bikes": [
            {$match: {"vehicleType": "bikes"}},
            {$lookup: {
               from: "bikes",
               localField: "vehicleId",
               foreignField: "_id",
               as: "vehicleDetails"
             }}
         ]
     }
   },
   {$project: {all: {$setUnion: ["$cars", "$bikes"]}}},
   {$unwind: "$all"},
   {$replaceRoot: { newRoot: "$all" }}
])
</code>```

Body:
Let's say I have 3 collections, cars, bikes, vehicles.

cars collection as:

{
  {
    "_id": "car1",
    "carBrand": "Audi",
    "color": "blue"
  },
  {
    "_id": "car2",
    "carBrand": "BMW",
    "color": "white"
  }
}

bikes collection as:

{
  {
    "_id": "bike1",
    "bikeBrand": "Audi",
    "color": "red"
  },
  {
    "_id": "bike2",
    "carBrand": "BMW",
    "color": "white"
  }
}

and the vehicles collection actually just has references to cars and bikes collections as

{
  {
    "_id": "vehicle1",
    "vehicleType": "cars",
    "vehicleId": "car1"
  },
  {
    "_id": "vehicle2",
    "vehicleType": "cars",
    "vehicleId": "car2"
  },
  {
    "_id": "vehicle3",
    "vehicleType": "bikes",
    "vehicleId": "bike1"
  },
  {
    "_id": "vehicle4",
    "vehicleType": "bikes",
    "vehicleId": "bike2"
  },

}

I'd like to join the vehicles collection with cars and bikes collections. I tried to set "$vehicleType" as a variable to $lookup's from field. However it does not work as I expected. It simply does not join the tables. No errors.

db.collection.aggregate([{

  $lookup: {
      from: "$vehicleType",
      localField: "vehicleId",
      foreignField: "_id",
      as: "vehicleDetails"
  }

}]);

I was expecting to have a result something like this

{
  {
    "_id": "vehicle1",
    "vehicleType": "cars",
    "vehicleId": "car1",
    "vehicleDetails": {
      "_id": "car1",
      "carBrand": "Audi",
      "color": ""
    }
  },
  {
    "_id": "vehicle2",
    "vehicleType": "cars",
    "vehicleId": "car2",
    "vehicleDetails":    {
      "_id": "car2",
      "carBrand": "BMW",
      "color": "white"

  },
  {
    "_id": "vehicle3",
    "vehicleType": "bikes",
    "vehicleId": "bike1",
    "vehicleDetails":    {
      "_id": "bike1",
      "bikeBrand": "Audi",
      "color": "red"
    }
  },
  {
    "_id": "vehicle4",
    "vehicleType": "bikes",
    "vehicleId": "bike2",
    "vehicleDetails":    {
      "_id": "bike2",
      "carBrand": "BMW",
      "color": "white"
    }
  },

}

Body:
If cars and bikes don't have common IDs, you can sequentially lookup in separate arrays, then combine them with $setUnion:

db.vehicles.aggregate([
  {$lookup: {
      from: "cars",
      localField: "vehicleId",
      foreignField: "_id",
      as: "carDetails"
  }},
  {$lookup: {
      from: "bikes",
      localField: "vehicleId",
      foreignField: "_id",
      as: "bikeDetails"
  }},
  {$project: {
     vehicleType: 1,
     vehicleId: 1,
     vehicleDetails:{$setUnion: [ "$carDetails", "$bikeDetails" ]}
  }},
  {$project: {
      carDetails:0,
      bikeDetails:0,
  }}
]);

Otherwise you will need to use $facet to filter vehicles by type before lookup:

db.vehicles.aggregate([
   {
     $facet: {
         "cars": [
            {$match: {"vehicleType": "cars"}},
            {$lookup: {
               from: "cars",
               localField: "vehicleId",
               foreignField: "_id",
               as: "vehicleDetails"
             }},
         ],
         "bikes": [
            {$match: {"vehicleType": "bikes"}},
            {$lookup: {
               from: "bikes",
               localField: "vehicleId",
               foreignField: "_id",
               as: "vehicleDetails"
             }}
         ]
     }
   },
   {$project: {all: {$setUnion: ["$cars", "$bikes"]}}},
   {$unwind: "$all"},
   {$replaceRoot: { newRoot: "$all" }}
])


问:
How to iterate map in a template using play framework 1.4.2
I have got a map where the key is a object but the value is a list object. I am using play framework 1.4.2 (using groovy template).

This is the map:

```<code>Map<Object, List<Object>> map = new HashMap<>();
</code>```

How can iterating through the map values in a template?

答:
You can use entrySet() to iterate over the map just like you would do in java (see this SO answer), then create a variable in a groovy script to get the value and key :

```<code>#{list items:map.entrySet(), as:'set' }
  ∗{print the key}*
  ${set.key}
  *{iterate over value list }∗
  #{list items: set.value, as:'itemValue'}
    ${itemValue}
  #{/list}

#{/list}
</code>```

For more information about groovy scripts in play see : Play documentation

Body:
I have got a map where the key is a object but the value is a list object. I am using play framework 1.4.2 (using groovy template).

This is the map:

Map<Object, List<Object>> map = new HashMap<>();

How can iterating through the map values in a template?

Body:
You can use entrySet() to iterate over the map just like you would do in java (see this SO answer), then create a variable in a groovy script to get the value and key :

#{list items:map.entrySet(), as:'set' }
  ∗{print the key}*
  ${set.key}
  *{iterate over value list }∗
  #{list items: set.value, as:'itemValue'}
    ${itemValue}
  #{/list}

#{/list}

For more information about groovy scripts in play see : Play documentation


问:
Setting PYTHONPATH on Windows under Anaconda without elevated privileges
I have a Windows 7 machine with a 32-bit install of anaconda, installed for the local user. The install works as expected: it allows me to run python and pip from the command line, switch environments, etc.

I also have a library of code that I wrote sitting in My Documents. I would like to add an entry like C:\Users\username\Documents\MyLib to my PYTHONPATH. This would normally not be an issue, but I do not have the elevated privileges that this (work-issued) computer requires to modify environment variables through the Windows UI.

I am looking for a way to set PYTHONPATH. I only need it to work within the anaconda environment (i.e., I just want to be able to import my library using that particular interpreter). Solutions using sys.path.append within my scripts are not acceptable. I am, however, OK with doing sys.path.append in some script that will be executed automatically whenever I run python myscript.py.

I suspect that there is a way to get around the privilege restrictions imposed through the Windows UI though, because running conda info -a shows that the Anaconda installer was able to insert items into the PATH, specifically, C:\Users\username\AppData\Local\Continuum\Anaconda3\Library\bin;....

答:
In the Control Panel window for modifying user accounts, there should be an option to change the environment variables just for the current user. It's broken on some versions of Windows, but if it works it's the simplest option.

If that doesn't work, the next simplest option is to use setx.exe in a command prompt. It defaults to the current user. For example:

```<code>setx.exe PYTHONPATH "C:\Users\username\Documents\MyLib"
</code>```

You can also use reg.exe to set the variable manually in the registry. But unlike the above options, this doesn't broadcast a WM_SETTINGCHANGE message to top-level windows. When Explorer sees that message it reloads its environment from the registry. Without it, you'll have to log off and back on again to see the updated environment variable. With that said, here's an example command using reg.exe:

```<code>reg.exe add HKCU\Environment /f /v PYTHONPATH /d "C:\Users\username\Documents\MyLib"
</code>```

I'm not a fan of permanently setting PYTHONPATH because the same variable gets used by every version of Python. I'd rather create a shortcut to a batch script that configures the environment the way I need it for a specific task. For example:

@echo off
set PYTHONPATH=C:\Users\username\Documents\MyLib
C:\Users\username\Documents\MyEnv\Scripts\activate.bat
```

Create a shortcut to this batch script. Then right-click the shortcut and select "Properties". Modify the target to run cmd.exe /k "path\to\the\script.bat".

Body:
In the Control Panel window for modifying user accounts, there should be an option to change the environment variables just for the current user. It's broken on some versions of Windows, but if it works it's the simplest option.

If that doesn't work, the next simplest option is to use setx.exe in a command prompt. It defaults to the current user. For example:

setx.exe PYTHONPATH "C:\Users\username\Documents\MyLib"

You can also use reg.exe to set the variable manually in the registry. But unlike the above options, this doesn't broadcast a WM_SETTINGCHANGE message to top-level windows. When Explorer sees that message it reloads its environment from the registry. Without it, you'll have to log off and back on again to see the updated environment variable. With that said, here's an example command using reg.exe:

reg.exe add HKCU\Environment /f /v PYTHONPATH /d "C:\Users\username\Documents\MyLib"

I'm not a fan of permanently setting PYTHONPATH because the same variable gets used by every version of Python. I'd rather create a shortcut to a batch script that configures the environment the way I need it for a specific task. For example:

@echo off
set PYTHONPATH=C:\Users\username\Documents\MyLib
C:\Users\username\Documents\MyEnv\Scripts\activate.bat

Create a shortcut to this batch script. Then right-click the shortcut and select "Properties". Modify the target to run cmd.exe /k "path\to\the\script.bat".


答:
This is an erosion followed by a dilation, common operations in computer vision:

```<code>>>> from scipy.ndimage.morphology import binary_dilation, binary_erosion
>>> print(list1)
[0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1]
>>> print(binary_dilation(binary_erosion(list1)).astype(int))
[0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
</code>```

The composition of the two operations is called an opening:

```<code>>>> from scipy.ndimage.morphology import binary_opening
>>> print(binary_opening(list1).astype(int))
[0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]
</code>```

Title: What is a Pythonic way to remove doubled duplicates in a list but allow triplets/greater?

Body:
This is an erosion followed by a dilation, common operations in computer vision:

>>> from scipy.ndimage.morphology import binary_dilation, binary_erosion
>>> print(list1)
[0, 0, 0, 1, 1, 1, 0, 0, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 1]
>>> print(binary_dilation(binary_erosion(list1)).astype(int))
[0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]

The composition of the two operations is called an opening:

>>> from scipy.ndimage.morphology import binary_opening
>>> print(binary_opening(list1).astype(int))
[0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1]


问:
Determine Assembler version in CMake?
We need to determine the version of GNU's AS (GAS) before deciding on what to build and how to build it. Some older GAS's, like the ones distributed with CentOS 5 and OpenBSD 6, can't handle SSE4 and above. We need to disable code paths that utilize SSE4, AES-NI, SHA, AVX and BMI.

Our primary build system is GNU Make, and it performs the following:

```<code>HAVE_GAS := $(shell $(CXX) -xc -c /dev/null -Wa,-v -o/dev/null 2>&1 | $(EGREP) -c "GNU assembler")
ifneq ($(HAVE_GAS),0)
  GAS210_OR_LATER := $(shell $(CXX) -xc -c /dev/null -Wa,-v -o/dev/null 2>&1 | $(EGREP) -c "GNU assembler version (2\.[1-9][0-9]|[3-9])")
  GAS217_OR_LATER := $(shell $(CXX) -xc -c /dev/null -Wa,-v -o/dev/null 2>&1 | $(EGREP) -c "GNU assembler version (2\.1[7-9]|2\.[2-9]|[3-9])")
  GAS219_OR_LATER := $(shell $(CXX) -xc -c /dev/null -Wa,-v -o/dev/null 2>&1 | $(EGREP) -c "GNU assembler version (2\.19|2\.[2-9]|[3-9])")
endif
</code>```

According to Cmake's Assembler wiki, CMake has basic support for assembly source files. But it does not discuss how to get the assembler's version.

How do we determine the assembler version when using CMake?

答:
I've run some tests and reviewed CMake's assembler detection code. CMake does

not determine nor set CMAKE_ASM-ATT_COMPILER_VERSION (as it would for e.g. CXX)

only set CMAKE_ASM-ATT_COMPILER_ID

So I've used your example code and just translated it to CMake:

```<code>cmake_minimum_required(VERSION 2.6)
project(AsmTest CXX)

enable_language(ASM-ATT)

if (CMAKE_ASM-ATT_COMPILER_ID STREQUAL "GNU")
    execute_process(
        COMMAND ${CMAKE_ASM-ATT_COMPILER} --version
        OUTPUT_VARIABLE _gas_version
    )
    if (_gas_version MATCHES "GNU.[Aa]ssembler.*(2\.[1-9][0-9]|[3-9])")
        set(GAS210_OR_LATER 1)
    endif()
endif()
message(STATUS "GAS210_OR_LATER = ${GAS210_OR_LATER}")
</code>```

Reference

CMake/Assembler

Title: Determine Assembler version in CMake?

Body:
We need to determine the version of GNU's AS (GAS) before deciding on what to build and how to build it. Some older GAS's, like the ones distributed with CentOS 5 and OpenBSD 6, can't handle SSE4 and above. We need to disable code paths that utilize SSE4, AES-NI, SHA, AVX and BMI.

Our primary build system is GNU Make, and it performs the following:

HAVE_GAS := $(shell $(CXX) -xc -c /dev/null -Wa,-v -o/dev/null 2>&1 | $(EGREP) -c "GNU assembler")
ifneq ($(HAVE_GAS),0)
  GAS210_OR_LATER := $(shell $(CXX) -xc -c /dev/null -Wa,-v -o/dev/null 2>&1 | $(EGREP) -c "GNU assembler version (2\.[1-9][0-9]|[3-9])")
  GAS217_OR_LATER := $(shell $(CXX) -xc -c /dev/null -Wa,-v -o/dev/null 2>&1 | $(EGREP) -c "GNU assembler version (2\.1[7-9]|2\.[2-9]|[3-9])")
  GAS219_OR_LATER := $(shell $(CXX) -xc -c /dev/null -Wa,-v -o/dev/null 2>&1 | $(EGREP) -c "GNU assembler version (2\.19|2\.[2-9]|[3-9])")
endif

According to Cmake's Assembler wiki, CMake has basic support for assembly source files. But it does not discuss how to get the assembler's version.

How do we determine the assembler version when using CMake?

Body:
I've run some tests and reviewed CMake's assembler detection code. CMake does

not determine nor set CMAKE_ASM-ATT_COMPILER_VERSION (as it would for e.g. CXX)

only set CMAKE_ASM-ATT_COMPILER_ID

So I've used your example code and just translated it to CMake:

cmake_minimum_required(VERSION 2.6)
project(AsmTest CXX)

enable_language(ASM-ATT)

if (CMAKE_ASM-ATT_COMPILER_ID STREQUAL "GNU")
    execute_process(
        COMMAND ${CMAKE_ASM-ATT_COMPILER} --version
        OUTPUT_VARIABLE _gas_version
    )
    if (_gas_version MATCHES "GNU.[Aa]ssembler.*(2\.[1-9][0-9]|[3-9])")
        set(GAS210_OR_LATER 1)
    endif()
endif()
message(STATUS "GAS210_OR_LATER = ${GAS210_OR_LATER}")

Reference

CMake/Assembler