问:
How to save arbitrarily-structured documents with Java DynamoDBMapper class
I'm using Amazon's DynamoDBMapper Java class to save data to a DynamoDB table. This code needs to work for data structured in multiple different ways, so I would like to stay away from writing particularly structure-specific code. For this reason, I store the code as JSON objects in Java -- which are basically glorified HashMaps.

I would like to store these JSON objects into DynamoDB as Dynamo's relatively new JSON Document type.

The way the DynamoDBMapper API works is essentially that you write a Java class (typically a POJO), then add some annotations, then pass your objects of that class into DynamoDBMapper so that it can then put items into the database with the structure of the Java class. This works well for many aspects of what I'm doing, but not with the fact that I want these classes to contain arbitrarily-structured JSON documents. This is the way you're meant to store JSON documents using DynamoDBMapper, and as you can see, it doesn't allow for the structure of the documents to be arbitrary.

I realize I could use Dynamo's putItem() to pass the jsons as Strings into Item objects -- I just wanted to see if what I want to do is possible with DynamoDBMapper before I shift my approach.

答:
Here's how I came up with my answer of how to store arbitrary Map objects in DynamoDB. This is extremely useful for archiving REST API responses that have been unmarshaled to foreign objects. I'm personally using this to archive REST responses from the PayPal Payment API. I don't care what variables they use in their REST API or the structure of their POJO / beans. I just want to make sure I save everything.

```<code>@DynamoDBTable(tableName = "PaymentResponse")
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.PROPERTY)
@JsonSubTypes({
    @JsonSubTypes.Type(value = PayPalPaymentResponse.class, name = "PayPalPaymentResponse"),
    @JsonSubTypes.Type(value = BatchPayPalPaymentResponse.class, name = "BatchPayPalPaymentResponse")}
    )
public abstract class PaymentResponse {

    // store any arbitrary REST resrponse data in map form so we don't have to worry about the
    // structure or the actual response itself
    protected Map<String, String> paymentResponseData = Maps.newHashMap();

    public PaymentResponse(PaymentResponseType paymentResponseType) {
        this.paymentResponseType = paymentResponseType;
    }

    public Map<String, String> getPaymentResponseData() { return paymentResponseData; }

    public void setPaymentResponseData(Map<String, String> paymentResponseData) { this.paymentResponseData = paymentResponseData; }

    @Override
    public String toString() {
        return Arrays.toString(paymentResponseData.entrySet().toArray());
    }
}

public class ConverterUtils {

      public static BatchPayPalPaymentResponse getBatchPayPalPaymentResponse(PayoutBatch payoutBatch) throws IOException {
        //read in the PayoutBatch response data and convert it first to a JSON string and then convert the
        //JSON string into a Map<String, String>
        Map<String, String> responseData = objectMapper.readValue(objectMapper.writeValueAsString(payoutBatch), new TypeReference<Map<String, String>>() {});
        BatchPayPalPaymentResponse batchPayPalPaymentResponse = new BatchPayPalPaymentResponse(responseData);
        return batchPayPalPaymentResponse;
    }

    public static PayPalPaymentResponse getSinglePayPalPaymentResponse(PayoutItemDetails payoutItemDetails) throws IOException {
        //read in the paypal PayoutItemDetails response data and convert it first to a JSON string and then convert the
        //JSON string into a Map<String, String>
        Map<String, String> responseData = objectMapper.readValue(objectMapper.writeValueAsString(payoutItemDetails), new TypeReference<Map<String, String>>() {});
        PayPalPaymentResponse payPalPaymentResponse = new PayPalPaymentResponse(responseData);
        return payPalPaymentResponse;

    }
}

public class BatchPayPalPaymentResponse extends PaymentResponse {
    public BatchPayPalPaymentResponse(Map<String, String> responseData) {
        super(responseData);
    }
    ....
    ....
    ....
}

public class PayPalPaymentResponse extends PaymentResponse {
    public PayPalPaymentResponse(Map<String, String> responseData) {
        super(responseData);
    }
    ....
    ....
    ....
}
</code>```

Now you can just call mapper.save(instanceOfPaymentResponse). Note that my code also includes how to use a Jackson parser to pick and choose which sub-class of PaymentResponse to unmarshal too. That's because I use a DynamoDBTypeConverter to marshal my class to a string before putting it into the database.

Finally, I'll throw in my converter for completeness so it all hopefully makes sense.

```<code>public class PaymentResponseConverter implements DynamoDBTypeConverter<String, PaymentResponse> {

    private static final ObjectMapper objectMapper = new ObjectMapper();
    static {
        objectMapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);
    }

    @Override
    public String convert(PaymentResponse object) {
        try {
            return objectMapper.writeValueAsString(object);
        } catch (JsonProcessingException e) {
            throw new IllegalArgumentException(String.format("Received invalid instance of PaymentResponse and cannot marshal it to a string (%s)", e.getMessage()));
        }
    }

    @Override
    public PaymentResponse unconvert(String object) {
        try {
            return objectMapper.readValue(object, PaymentResponse.class);
        } catch (IOException e) {
            throw new IllegalArgumentException(String.format("Unable to convert JSON to instance of PaymentResponse. This is a fatal error. (%s)", e.getMessage()));
        }
    }
}
</code>```

Body:
Here's how I came up with my answer of how to store arbitrary Map objects in DynamoDB. This is extremely useful for archiving REST API responses that have been unmarshaled to foreign objects. I'm personally using this to archive REST responses from the PayPal Payment API. I don't care what variables they use in their REST API or the structure of their POJO / beans. I just want to make sure I save everything.

@DynamoDBTable(tableName = "PaymentResponse")
@JsonIgnoreProperties(ignoreUnknown = true)
@JsonTypeInfo(use = JsonTypeInfo.Id.NAME, include = JsonTypeInfo.As.PROPERTY)
@JsonSubTypes({
    @JsonSubTypes.Type(value = PayPalPaymentResponse.class, name = "PayPalPaymentResponse"),
    @JsonSubTypes.Type(value = BatchPayPalPaymentResponse.class, name = "BatchPayPalPaymentResponse")}
    )
public abstract class PaymentResponse {

    // store any arbitrary REST resrponse data in map form so we don't have to worry about the
    // structure or the actual response itself
    protected Map<String, String> paymentResponseData = Maps.newHashMap();

    public PaymentResponse(PaymentResponseType paymentResponseType) {
        this.paymentResponseType = paymentResponseType;
    }

    public Map<String, String> getPaymentResponseData() { return paymentResponseData; }

    public void setPaymentResponseData(Map<String, String> paymentResponseData) { this.paymentResponseData = paymentResponseData; }

    @Override
    public String toString() {
        return Arrays.toString(paymentResponseData.entrySet().toArray());
    }
}

public class ConverterUtils {

      public static BatchPayPalPaymentResponse getBatchPayPalPaymentResponse(PayoutBatch payoutBatch) throws IOException {
        //read in the PayoutBatch response data and convert it first to a JSON string and then convert the
        //JSON string into a Map<String, String>
        Map<String, String> responseData = objectMapper.readValue(objectMapper.writeValueAsString(payoutBatch), new TypeReference<Map<String, String>>() {});
        BatchPayPalPaymentResponse batchPayPalPaymentResponse = new BatchPayPalPaymentResponse(responseData);
        return batchPayPalPaymentResponse;
    }

    public static PayPalPaymentResponse getSinglePayPalPaymentResponse(PayoutItemDetails payoutItemDetails) throws IOException {
        //read in the paypal PayoutItemDetails response data and convert it first to a JSON string and then convert the
        //JSON string into a Map<String, String>
        Map<String, String> responseData = objectMapper.readValue(objectMapper.writeValueAsString(payoutItemDetails), new TypeReference<Map<String, String>>() {});
        PayPalPaymentResponse payPalPaymentResponse = new PayPalPaymentResponse(responseData);
        return payPalPaymentResponse;

    }
}

public class BatchPayPalPaymentResponse extends PaymentResponse {
    public BatchPayPalPaymentResponse(Map<String, String> responseData) {
        super(responseData);
    }
    ....
    ....
    ....
}

public class PayPalPaymentResponse extends PaymentResponse {
    public PayPalPaymentResponse(Map<String, String> responseData) {
        super(responseData);
    }
    ....
    ....
    ....
}

Now you can just call mapper.save(instanceOfPaymentResponse). Note that my code also includes how to use a Jackson parser to pick and choose which sub-class of PaymentResponse to unmarshal too. That's because I use a DynamoDBTypeConverter to marshal my class to a string before putting it into the database.

Finally, I'll throw in my converter for completeness so it all hopefully makes sense.

public class PaymentResponseConverter implements DynamoDBTypeConverter<String, PaymentResponse> {

    private static final ObjectMapper objectMapper = new ObjectMapper();
    static {
        objectMapper.setSerializationInclusion(JsonInclude.Include.NON_NULL);
    }

    @Override
    public String convert(PaymentResponse object) {
        try {
            return objectMapper.writeValueAsString(object);
        } catch (JsonProcessingException e) {
            throw new IllegalArgumentException(String.format("Received invalid instance of PaymentResponse and cannot marshal it to a string (%s)", e.getMessage()));
        }
    }

    @Override
    public PaymentResponse unconvert(String object) {
        try {
            return objectMapper.readValue(object, PaymentResponse.class);
        } catch (IOException e) {
            throw new IllegalArgumentException(String.format("Unable to convert JSON to instance of PaymentResponse. This is a fatal error. (%s)", e.getMessage()));
        }
    }
}


来源: stackexchange

问:
Kube-proxy fails to retrieve node info - invalid nodeIP
I've been trying to setup a Kubernetes cluster for a few months now, but I have no luck so far.

I'm trying to set it up on 4 bare metal PCs running coreOS. I've just clean installed everything again, but I get to the same problem as before. I'm following this tutorial. I think I've configured everything correctly, but am not 100% sure. When I reboot any of the machines, kubelet and flanneld services are running, but I see the following errors for them when checking service status with systemctl status:

kubelet error: Process: 1246 ExecStartPre=/usr/bin/rkt rm --uuid-file=/var/run/kubelet-pod.uuid (code=exited, status=254)

flanneld error: Process: 1057 ExecStartPre=/usr/bin/rkt rm --uuid-file=/var/lib/coreos/flannel-wrapper.uuid (code=exited, status=254)

If I restart both services, they work, or at least look like they work - I get no errors.

Everything else seems to work fine, so the only problem (I think) left are the kube-proxy service on all nodes.

If I run kubectl get pods I see all pods running:

```<code>$ kubectl get pods
NAME                                   READY     STATUS    RESTARTS   AGE
kube-apiserver-kubernetes-4            1/1       Running   4          6m
kube-controller-manager-kubernetes-4   1/1       Running   6          6m
kube-proxy-kubernetes-1                1/1       Running   4          18h
kube-proxy-kubernetes-2                1/1       Running   5          26m
kube-proxy-kubernetes-3                1/1       Running   4          19m
kube-proxy-kubernetes-4                1/1       Running   4          18h
kube-scheduler-kubernetes-4            1/1       Running   6          18h
</code>```

The answer to this question suggest to check if kubectl get node returns same names that are registered on kubelet. As far as I checked the logs, nodes are registered correctly, and this is the output of kubectl get node

```<code>$ kubectl get node
NAME           STATUS                        AGE       VERSION
kubernetes-1   Ready                         18h       v1.6.1+coreos.0
kubernetes-2   Ready                         36m       v1.6.1+coreos.0
kubernetes-3   Ready                         29m       v1.6.1+coreos.0
kubernetes-4   Ready,SchedulingDisabled      18h       v1.6.1+coreos.0
</code>```

The tutorial I've used (linked above) suggest I use --hostname-override but I couldn't get node info on master node (kubernetes-4) If i tried to curl it locally. So I removed it and I can get node info normally now.

Someone suggested it might be a flannel problem and that I should check the flannel ports. Using netstat -lntu I get the following output:

```<code>Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN
tcp        0      0 MASTER_IP:2379          0.0.0.0:*               LISTEN
tcp        0      0 MASTER_IP:2380          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN
tcp6       0      0 :::4194                 :::*                    LISTEN
tcp6       0      0 :::10250                :::*                    LISTEN
tcp6       0      0 :::10251                :::*                    LISTEN
tcp6       0      0 :::10252                :::*                    LISTEN
tcp6       0      0 :::10255                :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 :::443                  :::*                    LISTEN
udp        0      0 0.0.0.0:8472            0.0.0.0:*
</code>```

So I assume the ports are fine?

Also etcd2 works, etcdctl cluster-health shows that all nodes are healthy

This is the part of cloud-config that starts the etcd2 on reboot, besides that I only store ssh keys and node username/password/groups in it:

```<code>#cloud-config

coreos:
  etcd2:
    name: "kubernetes-4"
    initial-advertise-peer-urls: "http://NODE_IP:2380"
    listen-peer-urls: "http://NODE_IP:2380"
    listen-client-urls: "http://NODE_IP,http://127.0.0.1:2379"
    advertise-client-urls: "http://NODE_IP:2379"
    initial-cluster-token: "etcd-cluster-1"
    initial-cluster: "kubernetes-4=http://MASTER_IP:2380,kubernetes-1=http://WORKER_1_IP:2380,kubernetes-2=http://WORKER_2_IP:2380,kubernetes-3=http://WORKER_3_IP:2380"
    initial-cluster-state: "new"
  units:
    - name: etcd2.service
      command: start
</code>```

This is the content of /etc/flannel/options.env file:

```<code>FLANNELD_IFACE=NODE_IP
FLANNELD_ETCD_ENDPOINTS=http://MASTER_IP:2379,http://WORKER_1_IP:2379,http://WORKER_2_IP:2379,http://WORKER_3_IP:2379
</code>```

The same endpoints are under --etcd-servers in kube-apiserver.yaml file

Any ideas/suggestion what could be the problem? Also if there are some details missing let me know, I'll add them to the post.

Edit: I forgot to include kube-proxy logs.

Master node kube-proxy log:

```<code>$ kubectl logs kube-proxy-kubernetes-4
I0615 07:47:45.250631       1 server.go:225] Using iptables Proxier.
W0615 07:47:45.286923       1 server.go:469] Failed to retrieve node info: Get http://127.0.0.1:8080/api/v1/nodes/kubernetes-4: dial tcp 127.0.0.1:8080: getsockopt: connection refused
W0615 07:47:45.303576       1 proxier.go:304] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
W0615 07:47:45.303593       1 proxier.go:309] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0615 07:47:45.303646       1 server.go:249] Tearing down userspace rules.
E0615 07:47:45.357276       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: Get http://127.0.0.1:8080/api/v1/endpoints?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E0615 07:47:45.357278       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
</code>```

Worker nodes kube-proxy log:

```<code>$ kubectl logs kube-proxy-kubernetes-1
I0615 07:47:33.667025       1 server.go:225] Using iptables Proxier.
W0615 07:47:33.697387       1 server.go:469] Failed to retrieve node info: Get https://MASTER_IP/api/v1/nodes/kubernetes-1: dial tcp MASTER_IP:443: getsockopt: connection refused
W0615 07:47:33.712718       1 proxier.go:304] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
W0615 07:47:33.712734       1 proxier.go:309] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0615 07:47:33.712773       1 server.go:249] Tearing down userspace rules.
E0615 07:47:33.787122       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: Get https://MASTER_IP/api/v1/endpoints?resourceVersion=0: dial tcp MASTER_IP:443: getsockopt: connection refused
E0615 07:47:33.787144       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: Get https://MASTER_IP/api/v1/services?resourceVersion=0: dial tcp MASTER_IP:443: getsockopt: connection refused
</code>```

答:
Did you try the scripts here? These are the condensed versions of the tutorial you used, for various platforms. The scripts worked perfectly for me on bare metal for k8s v1.6.4. I have a tweaked script with better encryption.

kube-apiserver isn't running which explains the error dial tcp 127.0.0.1:8080: getsockopt: connection refused. When I was debugging kube-apiserver, this was what I would do in the node:

Remove /etc/kubernetes/manifests/kube-apiserver.yaml.

Manually run a hyperkube container. Depending on your config, you will have to mount additional volumes (ie. -v) to expose files to the container. Update the image version to the one you use.

docker run --net=host -it -v /etc/kubernetes/ssl:/etc/kubernetes/ssl quay.io/coreos/hyperkube:v1.6.2_coreos.0

The above command will launch a shell in the hyperkube container. Now, launch kube-apiserver with the flags in your kube-apiserver.yaml manifest. It should look similar to this example:

/hyperkube apiserver \
--bind-address=0.0.0.0 \
--etcd-cafile=/etc/kubernetes/ssl/apiserver/ca.pem \
--etcd-certfile=/etc/kubernetes/ssl/apiserver/client.pem \
--etcd-keyfile=/etc/kubernetes/ssl/apiserver/client-key.pem \
--etcd-servers=https://10.246.40.20:2379,https://10.246.40.21:2379,https://10.246.40.22:2379 \
...

In any case, I suggest that you tear down the cluster and try the scripts first. It might just work ootb.

ContentLicense: CC BY-SA 3.0

Body:
I've been trying to setup a Kubernetes cluster for a few months now, but I have no luck so far.

I'm trying to set it up on 4 bare metal PCs running coreOS. I've just clean installed everything again, but I get to the same problem as before. I'm following this tutorial. I think I've configured everything correctly, but am not 100% sure. When I reboot any of the machines, kubelet and flanneld services are running, but I see the following errors for them when checking service status with systemctl status:

kubelet error: Process: 1246 ExecStartPre=/usr/bin/rkt rm --uuid-file=/var/run/kubelet-pod.uuid (code=exited, status=254)

flanneld error: Process: 1057 ExecStartPre=/usr/bin/rkt rm --uuid-file=/var/lib/coreos/flannel-wrapper.uuid (code=exited, status=254)

If I restart both services, they work, or at least look like they work - I get no errors.

Everything else seems to work fine, so the only problem (I think) left are the kube-proxy service on all nodes.

If I run kubectl get pods I see all pods running:

$ kubectl get pods
NAME                                   READY     STATUS    RESTARTS   AGE
kube-apiserver-kubernetes-4            1/1       Running   4          6m
kube-controller-manager-kubernetes-4   1/1       Running   6          6m
kube-proxy-kubernetes-1                1/1       Running   4          18h
kube-proxy-kubernetes-2                1/1       Running   5          26m
kube-proxy-kubernetes-3                1/1       Running   4          19m
kube-proxy-kubernetes-4                1/1       Running   4          18h
kube-scheduler-kubernetes-4            1/1       Running   6          18h

The answer to this question suggest to check if kubectl get node returns same names that are registered on kubelet. As far as I checked the logs, nodes are registered correctly, and this is the output of kubectl get node

$ kubectl get node
NAME           STATUS                        AGE       VERSION
kubernetes-1   Ready                         18h       v1.6.1+coreos.0
kubernetes-2   Ready                         36m       v1.6.1+coreos.0
kubernetes-3   Ready                         29m       v1.6.1+coreos.0
kubernetes-4   Ready,SchedulingDisabled      18h       v1.6.1+coreos.0

The tutorial I've used (linked above) suggest I use --hostname-override but I couldn't get node info on master node (kubernetes-4) If i tried to curl it locally. So I removed it and I can get node info normally now.

Someone suggested it might be a flannel problem and that I should check the flannel ports. Using netstat -lntu I get the following output:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN
tcp        0      0 MASTER_IP:2379          0.0.0.0:*               LISTEN
tcp        0      0 MASTER_IP:2380          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN
tcp6       0      0 :::4194                 :::*                    LISTEN
tcp6       0      0 :::10250                :::*                    LISTEN
tcp6       0      0 :::10251                :::*                    LISTEN
tcp6       0      0 :::10252                :::*                    LISTEN
tcp6       0      0 :::10255                :::*                    LISTEN
tcp6       0      0 :::22                   :::*                    LISTEN
tcp6       0      0 :::443                  :::*                    LISTEN
udp        0      0 0.0.0.0:8472            0.0.0.0:*

So I assume the ports are fine?

Also etcd2 works, etcdctl cluster-health shows that all nodes are healthy

This is the part of cloud-config that starts the etcd2 on reboot, besides that I only store ssh keys and node username/password/groups in it:

#cloud-config

coreos:
  etcd2:
    name: "kubernetes-4"
    initial-advertise-peer-urls: "http://NODE_IP:2380"
    listen-peer-urls: "http://NODE_IP:2380"
    listen-client-urls: "http://NODE_IP,http://127.0.0.1:2379"
    advertise-client-urls: "http://NODE_IP:2379"
    initial-cluster-token: "etcd-cluster-1"
    initial-cluster: "kubernetes-4=http://MASTER_IP:2380,kubernetes-1=http://WORKER_1_IP:2380,kubernetes-2=http://WORKER_2_IP:2380,kubernetes-3=http://WORKER_3_IP:2380"
    initial-cluster-state: "new"
  units:
    - name: etcd2.service
      command: start

This is the content of /etc/flannel/options.env file:

FLANNELD_IFACE=NODE_IP
FLANNELD_ETCD_ENDPOINTS=http://MASTER_IP:2379,http://WORKER_1_IP:2379,http://WORKER_2_IP:2379,http://WORKER_3_IP:2379

The same endpoints are under --etcd-servers in kube-apiserver.yaml file

Any ideas/suggestion what could be the problem? Also if there are some details missing let me know, I'll add them to the post.

Edit: I forgot to include kube-proxy logs.

Master node kube-proxy log:

$ kubectl logs kube-proxy-kubernetes-4
I0615 07:47:45.250631       1 server.go:225] Using iptables Proxier.
W0615 07:47:45.286923       1 server.go:469] Failed to retrieve node info: Get http://127.0.0.1:8080/api/v1/nodes/kubernetes-4: dial tcp 127.0.0.1:8080: getsockopt: connection refused
W0615 07:47:45.303576       1 proxier.go:304] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
W0615 07:47:45.303593       1 proxier.go:309] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0615 07:47:45.303646       1 server.go:249] Tearing down userspace rules.
E0615 07:47:45.357276       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: Get http://127.0.0.1:8080/api/v1/endpoints?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused
E0615 07:47:45.357278       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: Get http://127.0.0.1:8080/api/v1/services?resourceVersion=0: dial tcp 127.0.0.1:8080: getsockopt: connection refused

Worker nodes kube-proxy log:

$ kubectl logs kube-proxy-kubernetes-1
I0615 07:47:33.667025       1 server.go:225] Using iptables Proxier.
W0615 07:47:33.697387       1 server.go:469] Failed to retrieve node info: Get https://MASTER_IP/api/v1/nodes/kubernetes-1: dial tcp MASTER_IP:443: getsockopt: connection refused
W0615 07:47:33.712718       1 proxier.go:304] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
W0615 07:47:33.712734       1 proxier.go:309] clusterCIDR not specified, unable to distinguish between internal and external traffic
I0615 07:47:33.712773       1 server.go:249] Tearing down userspace rules.
E0615 07:47:33.787122       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:49: Failed to list *api.Endpoints: Get https://MASTER_IP/api/v1/endpoints?resourceVersion=0: dial tcp MASTER_IP:443: getsockopt: connection refused
E0615 07:47:33.787144       1 reflector.go:201] k8s.io/kubernetes/pkg/proxy/config/api.go:46: Failed to list *api.Service: Get https://MASTER_IP/api/v1/services?resourceVersion=0: dial tcp MASTER_IP:443: getsockopt: connection refused


问:
Faster/Piped version of Stream.CopyTo
I was shocked to see that the default implementation of Stream.CopyTo is not piped. No piping means that no reading on the source is done whilst writing to the destination.

This means that in most cases performance is half of what it could be. (Copy Filestream from C: to another HDD)

How do I implement a fast piped CopyTo?

答:
This is the most robust implementation I could think of:</p>

```<code>public void CopyTo_Fast(Stream Destination, int Buffersize)
{
    byte[] BufferA = new byte[Buffersize];
    byte[] BufferB = new byte[Buffersize];
    int Abytes = Read(BufferA, 0, BufferA.Length);//read first buffer
    if (Abytes < Buffersize) //trivial case stream too small for two buffers
    {
        Destination.Write(BufferA, 0, Abytes);
        return;
    }
    int Bbytes = 0;
    Task Writer;
    Task Reader;

    while(true) //Read to end
    {
        Writer = Task.Run(() => Destination.Write(BufferA, 0, Abytes));
        Reader = Task.Run(() => Bbytes = Read(BufferB, 0, Buffersize));
        Task.WaitAll(Writer, Reader);
        if (Bbytes == 0) return;

        Writer = Task.Run(() => Destination.Write(BufferB, 0, Bbytes));
        Reader = Task.Run(() => Abytes = Read(BufferA, 0, Buffersize));
        Task.WaitAll(Writer, Reader);
        if (Abytes == 0) return;
    }
}
</code>```

Title: Faster/Piped version of Stream.CopyTo

Body:
This is the most robust implementation I could think of:

public void CopyTo_Fast(Stream Destination, int Buffersize)
{
    byte[] BufferA = new byte[Buffersize];
    byte[] BufferB = new byte[Buffersize];
    int Abytes = Read(BufferA, 0, BufferA.Length);//read first buffer
    if (Abytes < Buffersize) //trivial case stream too small for two buffers
    {
        Destination.Write(BufferA, 0, Abytes);
        return;
    }
    int Bbytes = 0;
    Task Writer;
    Task Reader;

    while(true) //Read to end
    {
        Writer = Task.Run(() => Destination.Write(BufferA, 0, Abytes));
        Reader = Task.Run(() => Bbytes = Read(BufferB, 0, Buffersize));
        Task.WaitAll(Writer, Reader);
        if (Bbytes == 0) return;

        Writer = Task.Run(() => Destination.Write(BufferB, 0, Bbytes));
        Reader = Task.Run(() => Abytes = Read(BufferA, 0, Buffersize));
        Task.WaitAll(Writer, Reader);
        if (Abytes == 0) return;
    }
}


问:
javascript merge array of objects, resulting object values in array
Would like to merge an array of objects resulting in an object of unique keys and array of values (duplication of values is ok). Solutions in vanilla JS or lodash preferred.

eg - from this:

```<code>[{
  a: 1,
  b: 2
}, {
  a: 1,
  c: 3
}]
</code>```

to this:

```<code>{
  a: [1, 1],
  b: [2],
  c: [3]
}
</code>```

答:
You can use lodash#mergeWith wrapped in a  lodash#spread to make lodash#mergeWith treat an array as a list of arguments. We use lodash#concat as a supporting function to concatenate an empty object (to avoid mutating the objects in the collection), the collection, and the customizer function that merges the entire collection. The customizer is composed using lodash#flow, wherein its first argument is lodash#concat that only accepts an arity of 2 using lodash#ary and the second argument uses lodash#compact -- It removes all undefined values in an array.

```<code>var result = _.spread(_.mergeWith)(
  _.concat({}, data, _.flow(_.ary(_.concat, 2), _.compact))
);
</code>```

var data = [{
  "a": 1,
  "b": 2
}, {
  "a": 1,
  "c": 3
}];

var result = _.spread(_.mergeWith)(
  _.concat({}, data, _.flow(_.ary(_.concat, 2), _.compact))
);

console.log(result);```
<pre class="snippet-code-html lang-html prettyprint-override"><code><script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.4/lodash.min.js"></script></code>```

Body:
Would like to merge an array of objects resulting in an object of unique keys and array of values (duplication of values is ok). Solutions in vanilla JS or lodash preferred.

eg - from this:

[{
  a: 1,
  b: 2
}, {
  a: 1,
  c: 3
}]

to this:

{
  a: [1, 1],
  b: [2],
  c: [3]
}

Body:
You can use lodash#mergeWith wrapped in a  lodash#spread to make lodash#mergeWith treat an array as a list of arguments. We use lodash#concat as a supporting function to concatenate an empty object (to avoid mutating the objects in the collection), the collection, and the customizer function that merges the entire collection. The customizer is composed using lodash#flow, wherein its first argument is lodash#concat that only accepts an arity of 2 using lodash#ary and the second argument uses lodash#compact -- It removes all undefined values in an array.

var result = _.spread(_.mergeWith)(
  _.concat({}, data, _.flow(_.ary(_.concat, 2), _.compact))
);

var data = [{
  "a": 1,
  "b": 2
}, {
  "a": 1,
  "c": 3
}];

var result = _.spread(_.mergeWith)(
  _.concat({}, data, _.flow(_.ary(_.concat, 2), _.compact))
);

console.log(result);

<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.4/lodash.min.js"></script>


问:
Setting S3 to post notifications to SQS when new item is created
I am trying to do (what should be) a simple task.

Create an S3 Bucket

Create an SQS Queue

Have the S3 bucket put a message in the SQS Queue whenever a new item is added.

I am getting stuck on attaching the two.

I've created the queue, and it works well.  I've created the bucket, and it works well.

Now i'm trying to add a policy to the queue that will let the bucket send notifications.

Following this guide:

http://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html

I am trying to call this method:

```<code>  public SetQueueAttributesResponse SetQueueAttributes(Dictionary<string, string> attributes)
    {
        var setQueueAttributesRequest = new SetQueueAttributesRequest()
        {
            QueueUrl = QueueUri.ToString(),
            Attributes = attributes
        };

        return SqsClient.SetQueueAttributes(setQueueAttributesRequest);
    }
</code>```

And the attributes property has a single key/value pair:

```<code>Policy

{
"Version" : "7/5/2017",
"Id"      : "S3-Post",
"Statement" : [
    {
        "Sid" : "S3-Post",
        "Effect" : "Allow",
        "Principal" : {
            "AWS" : "*"
        },
        "Action"    : [
            "sqs:SendMessage",
            "sqs:GetQueueUrl"
        ],
        "Resource"  : "arn:aws:sqs:us-west-2:[REDACTED]:testqueue2",
        "Condition" : {
            "ArnLike" : {
                "aws:SourceArn" : "arn:aws:s3:*:*:*"
            }
        }
    }
]
</code>```

}

(Yes, I widened the filter condition slightly, but got this same failure with a bucket name established.)

I get this exception:

Amazon.SQS.AmazonSQSException was unhandled

ErrorCode=InvalidAttributeValue

HResult=-2146233088

Message=Invalid value for the parameter Policy.

RequestId=0ac6201a-17f5-5b63-a5ac-71cd42b10481

Source=AWSSDK.Core

Can anyone see my error, or can anyone point me to a more functional example?

答:
Version refers to the version of IAM policy language in use, not a revision of your policy.

There's only one value you want to use: "2012-10-17".

http://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html#Version

Body:
I am trying to do (what should be) a simple task.

Create an S3 Bucket

Create an SQS Queue

Have the S3 bucket put a message in the SQS Queue whenever a new item is added.

I am getting stuck on attaching the two.

I've created the queue, and it works well.  I've created the bucket, and it works well.

Now i'm trying to add a policy to the queue that will let the bucket send notifications.

Following this guide:

http://docs.aws.amazon.com/AmazonS3/latest/dev/ways-to-add-notification-config-to-bucket.html

I am trying to call this method:

  public SetQueueAttributesResponse SetQueueAttributes(Dictionary<string, string> attributes)
    {
        var setQueueAttributesRequest = new SetQueueAttributesRequest()
        {
            QueueUrl = QueueUri.ToString(),
            Attributes = attributes
        };

        return SqsClient.SetQueueAttributes(setQueueAttributesRequest);
    }

And the attributes property has a single key/value pair:

Policy

{
"Version" : "7/5/2017",
"Id"      : "S3-Post",
"Statement" : [
    {
        "Sid" : "S3-Post",
        "Effect" : "Allow",
        "Principal" : {
            "AWS" : "*"
        },
        "Action"    : [
            "sqs:SendMessage",
            "sqs:GetQueueUrl"
        ],
        "Resource"  : "arn:aws:sqs:us-west-2:[REDACTED]:testqueue2",
        "Condition" : {
            "ArnLike" : {
                "aws:SourceArn" : "arn:aws:s3:*:*:*"
            }
        }
    }
]

}

(Yes, I widened the filter condition slightly, but got this same failure with a bucket name established.)

I get this exception:

Amazon.SQS.AmazonSQSException was unhandled

ErrorCode=InvalidAttributeValue

HResult=-2146233088

Message=Invalid value for the parameter Policy.

RequestId=0ac6201a-17f5-5b63-a5ac-71cd42b10481

Source=AWSSDK.Core

Can anyone see my error, or can anyone point me to a more functional example?


问:
OrderBy in SQL Server with custom "max" integer
I have table in SQL Server which contains a sort column of integers. All values in the column are positive. I want to sort the table such that I chose a "max" value (like 3) and all values are returned decreasing relative to 3. The "descending" order doesn't actually matter much as long as each item in the database is listed.

Database example:

```<code>Value SortColumn
A     1
B     5
C     4
D     3
E     0
F     2
</code>```

Desired output (with 3 as "max"):

```<code>Value SortColumn
D        3
C        4
B        5
E        0
A        1
F        2
</code>```

It seems that this similar post might be in the right direction, but it's not quite what I want....

答:
I think you want:</p>

```<code>order by abs(sortcolumn - 3) asc
</code>```

Body:
I have table in SQL Server which contains a sort column of integers. All values in the column are positive. I want to sort the table such that I chose a "max" value (like 3) and all values are returned decreasing relative to 3. The "descending" order doesn't actually matter much as long as each item in the database is listed.

Database example:

Value SortColumn
A     1
B     5
C     4
D     3
E     0
F     2

Desired output (with 3 as "max"):

Value SortColumn
D        3
C        4
B        5
E        0
A        1
F        2

It seems that this similar post might be in the right direction, but it's not quite what I want....

Body:
I think you want:

order by abs(sortcolumn - 3) asc


问:
Second MySQL database PHP from WordPress page
I am building a new WP site and want to create a couple of pages that access data from a separate mysql database. I want to insert the PHP into the pages using a plugin like Insert PHP.

I've installed the plugin but, I cannot seem to get the database to connect from within the WP page.

First off, I am not even sure where to put the second mysql file on the same server. It has a different login and PW than the WP site.

Should I put it in the \var\www\html folder? I don't want to replace the DB I am using, just send some queries out to another one on the same server to get some data.

I've added PHP with the insert PHP plugin and I get an access denied message:

Connection failed: Access denied for user 'root'@'localhost'

Here is the code I use:

```<code>$servername = "localhost";
$username = "xxxx";
$password = "xxxx";
$dbname = "xxxx";

// Create connection
$conn = new mysqli($servername, $username, $password);
// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}
</code>```

Please help and TY.

答:
maybe your root password is incorrect. Try this one
https://dev.mysql.com/doc/refman/5.7/en/resetting-permissions.html</p>

Body:
I am building a new WP site and want to create a couple of pages that access data from a separate mysql database. I want to insert the PHP into the pages using a plugin like Insert PHP.

I've installed the plugin but, I cannot seem to get the database to connect from within the WP page.

First off, I am not even sure where to put the second mysql file on the same server. It has a different login and PW than the WP site.

Should I put it in the \var\www\html folder? I don't want to replace the DB I am using, just send some queries out to another one on the same server to get some data.

I've added PHP with the insert PHP plugin and I get an access denied message:

Connection failed: Access denied for user 'root'@'localhost'

Here is the code I use:

$servername = "localhost";
$username = "xxxx";
$password = "xxxx";
$dbname = "xxxx";

// Create connection
$conn = new mysqli($servername, $username, $password);
// Check connection
if ($conn->connect_error) {
    die("Connection failed: " . $conn->connect_error);
}

Please help and TY.


问:
urlconnection javax.net.ssl.SSLException: java.lang.IllegalStateException
im trying to use an UrlConnection to send audio (represented as a base64 byte array - as requested by the Google Cloud Speech API). I am doing this on a Raspberry Pi with ubuntu. The connection does work if I use it on my laptop (also with ubuntu). I am using java with an URLConnection. The crash happens on OutputStream out = urlConnection.getOutputStream();

Java Code:

```<code>// URL to google + parameter with the API key.
    URL googleUrl = new URL(GOOGLE_RECOGNIZER_URL + "?key=" + APIKEY);

    // Open New URL connection channel.
    URLConnection urlConnection;

    urlConnection = googleUrl.openConnection();

    // we want to do output. (input is automaticly set on true).
    urlConnection.setDoOutput(true);

    // No caching.
    urlConnection.setUseCaches(false);

    // Add headers.
    urlConnection.setRequestProperty("Content-Type", "application/json");

// For sending the recorded data to google it needs to be first
    // converted to a byte array.
    byte[] array1 = AudioToByteArray.readAudioFileData(file);

    // And next convert the byte array to a base64 string.
    String audioString = Base64.getEncoder().encodeToString(array1);

    // Make (JSON) Strings for configuration and location of audio file.
    String testConfig = "\"config\": { \"encoding\":\"LINEAR16\", \"sampleRate\": 16000, \"languageCode\": \"nl-NL\"}";
    String testAudioLoc = "\"audio\": { \"content\": \"" + audioString + "\" }";
    String together = "{ " + testConfig + "," + testAudioLoc + " }";

    // Send the string to google.
    OutputStream out = urlConnection.getOutputStream();

    OutputStreamWriter wr = new OutputStreamWriter(out);
    wr.write(together);
    wr.flush();
    wr.close();
    out.close();
</code>```

Error:

```<code>javax.net.ssl.SSLException: java.lang.IllegalStateException
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1906)
at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1889)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1410)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
at main.ContactGoogleCloudSpeechAPI.writeOut(ContactGoogleCloudSpeechAPI.java:98)
at main.ContactGoogleCloudSpeechAPI.contactGoogleUrlCon(ContactGoogleCloudSpeechAPI.java:41)
at main.SpeechRecognition.contactAPI(SpeechRecognition.java:184)
at main.SpeechRecognition.mainLoop(SpeechRecognition.java:104)
at main.SpeechRecognition.run(SpeechRecognition.java:81)
at main.SpeechRecognition.main(SpeechRecognition.java:62)
</code>```

答:
I was getting the same exception, and realized that I was using openJDK

So the solution is to uninstall openJDK and install oracle JDK

You can check which JDK version are you using the following command:

```<code>java -version
</code>```

Body:
im trying to use an UrlConnection to send audio (represented as a base64 byte array - as requested by the Google Cloud Speech API). I am doing this on a Raspberry Pi with ubuntu. The connection does work if I use it on my laptop (also with ubuntu). I am using java with an URLConnection. The crash happens on OutputStream out = urlConnection.getOutputStream();

Java Code:

// URL to google + parameter with the API key.
    URL googleUrl = new URL(GOOGLE_RECOGNIZER_URL + "?key=" + APIKEY);

    // Open New URL connection channel.
    URLConnection urlConnection;

    urlConnection = googleUrl.openConnection();

    // we want to do output. (input is automaticly set on true).
    urlConnection.setDoOutput(true);

    // No caching.
    urlConnection.setUseCaches(false);

    // Add headers.
    urlConnection.setRequestProperty("Content-Type", "application/json");

// For sending the recorded data to google it needs to be first
    // converted to a byte array.
    byte[] array1 = AudioToByteArray.readAudioFileData(file);

    // And next convert the byte array to a base64 string.
    String audioString = Base64.getEncoder().encodeToString(array1);

    // Make (JSON) Strings for configuration and location of audio file.
    String testConfig = "\"config\": { \"encoding\":\"LINEAR16\", \"sampleRate\": 16000, \"languageCode\": \"nl-NL\"}";
    String testAudioLoc = "\"audio\": { \"content\": \"" + audioString + "\" }";
    String together = "{ " + testConfig + "," + testAudioLoc + " }";

    // Send the string to google.
    OutputStream out = urlConnection.getOutputStream();

    OutputStreamWriter wr = new OutputStreamWriter(out);
    wr.write(together);
    wr.flush();
    wr.close();
    out.close();

Error:

javax.net.ssl.SSLException: java.lang.IllegalStateException
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1906)
at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1889)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1410)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
at main.ContactGoogleCloudSpeechAPI.writeOut(ContactGoogleCloudSpeechAPI.java:98)
at main.ContactGoogleCloudSpeechAPI.contactGoogleUrlCon(ContactGoogleCloudSpeechAPI.java:41)
at main.SpeechRecognition.contactAPI(SpeechRecognition.java:184)
at main.SpeechRecognition.mainLoop(SpeechRecognition.java:104)
at main.SpeechRecognition.run(SpeechRecognition.java:81)
at main.SpeechRecognition.main(SpeechRecognition.java:62)

Body:
I was getting the same exception, and realized that I was using openJDK

So the solution is to uninstall openJDK and install oracle JDK

You can check which JDK version are you using the following command:

java -version


答:
You could use a CASE expression:

```<code>SELECT
    Value, SortColumn
FROM yourTable
ORDER BY
    CASE WHEN SortColumn = 3 THEN 0 ELSE 1 END,
    SortColumn DESC
</code>```

Title: OrderBy in SQL Server with custom "max" integer

Body:
You could use a CASE expression:

SELECT
    Value, SortColumn
FROM yourTable
ORDER BY
    CASE WHEN SortColumn = 3 THEN 0 ELSE 1 END,
    SortColumn DESC


问:
Architect a Cloudera CDH cluster on AWS: instances and storage
I have some doubts about a deployment of CDH on AWS. I read the reference architecture doc and other material I found on Cloudera Engineering Blog but I need some more suggestions about it.

1) Is the CDH deployment available only for some kind of instances or I can deploy it on all the AWS instance types?

2) Assuming I want to create a cluster that will be active 24x7. For a long-running cluster I understood it's better to have a cluster based on local-storage instances. If we consider a cluster of 2PBs I think that d2.8xlarge should be the best choice for the datanodes. About the Master Nodes: - if I want to deploy only 3 Master Nodes, is it better to have them as local-storage instances too or as EBS attached instances to be able to react quickly to a possible Master Node failure? - are there some best practice about the master node instance type (EBS or local-storage)? About the Data Nodes: - if a data node fails, Has the CDH some automated mechanism to automatically spin-up a new instance and connect it to the cluster in order to restore the cluster without down-times? Have we to build a script from scratch to do this thing? About the Edge Nodes: - are there some best practice about the instance type (EBS or local-storage)?

3) If I want to do a backup of the cluster on S3: - when I do a distcp from the CDH to S3, can I move the data directly on Glacier instead of the normal S3? If I have some compression applied on the data (e.g. snappy, gzip, etc.) and I do a distcp to S3: - Is the space occupied on S3 the same or the distcp command decompress the data for the copy?

If I have a cluster based on EBS attached instances: - is it possible to snapshot the disks and re-attach a datanode with the EBS disks rebuilt from the snapshot?

4) If I have the Data Nodes deployed as r4.8xlarge and I need more horsepower, is it possible to scale-up the cluster from r4.8xlarge to a r4.16xlarge on-the-fly? Attaching and detaching the disks in few mins?

Thanks a lot for the clarifications, I hope my doubts will help also other users.

答:
1) There's no explicit restriction on instance types where CDH components will work, but you'd need to pick types with a minimum of horsepower. For example, I don't expect that a micro size instance would work for much of anything. A type that is too small will generally cause daemons to run out of memory. The reference architecture has suggested instance types for certain situations.

2) You should stick with EBS for the root volume of instance types. There are a few reasons, including that newer instance types don't even support local instance storage for the root disk.

CDH doesn't have a mechanism for replacing data nodes when they fail. You could roll something yourself, possibly with help from Cloudera Director.

3) You can set up lifecycle rules for data in S3 to migrate it from the standard storage class into Glacier over time, or you can just write directly to Glacier; it doesn't look like direct Glacier access can be done through the s3a connector. I'm pretty sure distcp and S3 won't fiddle with compression; what you copy is opaque to S3 for sure. You can snapshot EBS volumes (root or additionally attached), then detach them and re-attach them to a different instance; this isn't necessarily a great way to back up datanodes vs. the distcp route, because each datanode is unique and has changing data as the cluster runs.

4) You can resize EBS-backed EC2 instances without detaching and re-attaching disks. You do have to stop an instance to resize it.


问:
Search for file in directory with multiple directories
Here's my goal. I want to be able to pass a parent directory and a filename to a method that searches for that specific file in the directory and any sub-directories. Below is the code I have been working with but can not get it to do exactly what I want. It will find the file I specify but will not return anything.

```<code>private static File findFile(File dir, String name) {
    String file     = "";
    File[] dirlist  = dir.listFiles();

    search:
        for(int i = 0; i < dirlist.length; i++) {
            if(dirlist[i].isDirectory()) {
                findFile(dirlist[i], name);
            } else if(dirlist[i].getName().matches(name)) {
                file = dirlist[i].toString();
                break search;
            }
        }

    return new File(file);
}
</code>```

I know that when the method finds a directory and calls itself it resets the file variable which is where I am storing the found file. So that is why I am getting a blank return. I am not sure how to accomplish this goal or if it's even possible.

答:
In fact there are many solutions to do the job.
I assume that you want to find a unique file (or the first one) found in a directory tree that matches with the fileName.
It is a problem of optimization because there are multiple ways to explore solutions, and we want to find an acceptable solution.

1- Solution using FileUtils.listFiles

```<code>public static File searchFileWithFileUtils(final File file, final String fileName) {
    File target = null;
    if(file.isDirectory()) {
        Collection<File> files = FileUtils.listFiles(file, null, true);
        for (File currFile : files) {
            if (currFile.isFile() && currFile.getName().equals(fileName)) {
                target = currFile;
                break;
            }
        }
    }
    return target;
}
</code>```

The solution using the library FileUtils is not a suitable solution because the method FileUtils#listFiles() loads all the directory/folder tree (the cost is expensive !).
We don't need to know all the tree, we can choose a better algorithm which stops when the file is found.

2- Recursive Solution

```<code>public static File searchFileRecursive(final File file, final String search) {
    if (file.isDirectory()) {
        File[] files = file.listFiles();
        for (File f : files) {
            File target = searchFileRecursive(f, search);
            if(target != null) {
                return target;
            }
        }
    } else {
        if (search.equals(file.getName())) {
            return file;
        }
    }
    return null;
}
</code>```

The algorithm tests if the file exists inside any folder. If not, it tries subfolder of the current folder... recursively. If the file is not found in the current branch it tries another subfolder.

The exploration is deep, and for any file in a deepness of 1 the algorithm will explore the entirety of previous subfolders (previous branches are completely explored !).
This algorithm has the best performances for files in a deep location inside the first branch.

In the majority of cases, the file location is not deep, so let explore another algorithm that works in most of cases.

3- Fastest Solution : exploration by deepness

```<code>public static File searchFileByDeepness(final String directoryName, final String fileName) {
    File target = null;
    if(directoryName != null && fileName != null) {
        File directory = new File(directoryName);
        if(directory.isDirectory()) {
            File file = new File(directoryName, fileName);
            if(file.isFile()) {
                target = file;
            }
            else {
                List<File> subDirectories = getSubDirectories(directory);
                do {
                    List<File> subSubDirectories = new ArrayList<File>();
                    for(File subDirectory : subDirectories) {
                        File fileInSubDirectory = new File(subDirectory, fileName);
                        if(fileInSubDirectory.isFile()) {
                            return fileInSubDirectory;
                        }
                        subSubDirectories.addAll(getSubDirectories(subDirectory));
                    }
                    subDirectories = subSubDirectories;
                } while(subDirectories != null && ! subDirectories.isEmpty());
            }
        }
    }
    return target;
}

private static List<File> getSubDirectories(final File directory) {
    File[] subDirectories = directory.listFiles(new FilenameFilter() {
        @Override
        public boolean accept(final File current, final String name) {
            return new File(current, name).isDirectory();
        }
    });
    return Arrays.asList(subDirectories);
}
</code>```

For each deepness, the algorithm searches the file inside all folders of the same level. If the file is not found, it tries the next level (deepness++).
Due to the parallel exploration (symmetry), this solution is suitable in most of cases.

Comparison:

```<code>public class FileLocationFinder {

    public static void main(final String[] args) {
        String rootFolder = args[0];
        String fileName = args[1];

        long start = System.currentTimeMillis();
        File target = searchFileWithFileUtils(new File(rootFolder), fileName);
        System.out.println(target.getAbsolutePath());
        System.out.println("Duration: " + (System.currentTimeMillis() - start) + "ms");

        start = System.currentTimeMillis();
        target = searchFileRecursive(new File(rootFolder), fileName);
        System.out.println(target.getAbsolutePath());
        System.out.println("Duration: " + (System.currentTimeMillis() - start) + "ms");

        start = System.currentTimeMillis();
        target = searchFileByDeepness(rootFolder, fileName);
        System.out.println(target.getAbsolutePath());
        System.out.println("Duration: " + (System.currentTimeMillis() - start) + "ms");
    }

    // Solution with FileUtils#listFiles
    //--------------------------------------------

    public static File searchFileWithFileUtils(final File file, final String fileName) {
        File target = null;
        if(file.isDirectory()) {
            Collection<File> files = FileUtils.listFiles(file, null, true);
            for (File currFile : files) {
                if (currFile.isFile() && currFile.getName().equals(fileName)) {
                    target = currFile;
                    break;
                }
            }
        }
        return target;
    }

    // Recursive solution
    //--------------------------------------------

    public static File searchFileRecursive(final File file, final String search) {
        if (file.isDirectory()) {
            File[] files = file.listFiles();
            for (File f : files) {
                File target = searchFileRecursive(f, search);
                if(target != null) {
                    return target;
                }
            }
        } else {
            if (search.equals(file.getName())) {
                return file;
            }
        }
        return null;
    }

    // Fastest solution
    //--------------------------------------------

    public static File searchFileByDeepness(final String directoryName, final String fileName) {
        File target = null;
        if(directoryName != null && fileName != null) {
            File directory = new File(directoryName);
            if(directory.isDirectory()) {
                File file = new File(directoryName, fileName);
                if(file.isFile()) {
                    target = file;
                }
                else {
                    List<File> subDirectories = getSubDirectories(directory);
                    do {
                        List<File> subSubDirectories = new ArrayList<File>();
                        for(File subDirectory : subDirectories) {
                            File fileInSubDirectory = new File(subDirectory, fileName);
                            if(fileInSubDirectory.isFile()) {
                                return fileInSubDirectory;
                            }
                            subSubDirectories.addAll(getSubDirectories(subDirectory));
                        }
                        subDirectories = subSubDirectories;
                    } while(subDirectories != null && ! subDirectories.isEmpty());
                }
            }
        }
        return target;
    }

    private static List<File> getSubDirectories(final File directory) {
        File[] subDirectories = directory.listFiles(new FilenameFilter() {
            @Override
            public boolean accept(final File current, final String name) {
                return new File(current, name).isDirectory();
            }
        });
        return Arrays.asList(subDirectories);
    }
}
</code>```

Result:

searchFileWithFileUtils: 20186ms | searchFileRecursive: 1134ms | searchFileByDeepness: 16ms

[EDIT]
You can also use Java 8 Files API to do this job :

```<code>public static File searchFileJava8(final String rootFolder, final String fileName) {
    File target = null;
    Path root = Paths.get(rootFolder);
    try (Stream<Path> stream = Files.find(root, Integer.MAX_VALUE, (path, attr) ->
            path.getFileName().toString().equals(fileName))) {
        Optional<Path> path = stream.findFirst();
        if(path.isPresent()) {
            target = path.get().toFile();
        }
    }
    catch (IOException e) {
    }
    return target;
}
</code>```

But the execution time is not better (994ms).

ContentLicense: CC BY-SA 2.5

Body:
Here's my goal. I want to be able to pass a parent directory and a filename to a method that searches for that specific file in the directory and any sub-directories. Below is the code I have been working with but can not get it to do exactly what I want. It will find the file I specify but will not return anything.

private static File findFile(File dir, String name) {
    String file     = "";
    File[] dirlist  = dir.listFiles();

    search:
        for(int i = 0; i < dirlist.length; i++) {
            if(dirlist[i].isDirectory()) {
                findFile(dirlist[i], name);
            } else if(dirlist[i].getName().matches(name)) {
                file = dirlist[i].toString();
                break search;
            }
        }

    return new File(file);
}

I know that when the method finds a directory and calls itself it resets the file variable which is where I am storing the found file. So that is why I am getting a blank return. I am not sure how to accomplish this goal or if it's even possible.

Body:
In fact there are many solutions to do the job.
I assume that you want to find a unique file (or the first one) found in a directory tree that matches with the fileName.
It is a problem of optimization because there are multiple ways to explore solutions, and we want to find an acceptable solution.

1- Solution using FileUtils.listFiles

public static File searchFileWithFileUtils(final File file, final String fileName) {
    File target = null;
    if(file.isDirectory()) {
        Collection<File> files = FileUtils.listFiles(file, null, true);
        for (File currFile : files) {
            if (currFile.isFile() && currFile.getName().equals(fileName)) {
                target = currFile;
                break;
            }
        }
    }
    return target;
}

The solution using the library FileUtils is not a suitable solution because the method FileUtils#listFiles() loads all the directory/folder tree (the cost is expensive !).
We don't need to know all the tree, we can choose a better algorithm which stops when the file is found.

2- Recursive Solution

public static File searchFileRecursive(final File file, final String search) {
    if (file.isDirectory()) {
        File[] files = file.listFiles();
        for (File f : files) {
            File target = searchFileRecursive(f, search);
            if(target != null) {
                return target;
            }
        }
    } else {
        if (search.equals(file.getName())) {
            return file;
        }
    }
    return null;
}

The algorithm tests if the file exists inside any folder. If not, it tries subfolder of the current folder... recursively. If the file is not found in the current branch it tries another subfolder.

The exploration is deep, and for any file in a deepness of 1 the algorithm will explore the entirety of previous subfolders (previous branches are completely explored !).
This algorithm has the best performances for files in a deep location inside the first branch.

In the majority of cases, the file location is not deep, so let explore another algorithm that works in most of cases.

3- Fastest Solution : exploration by deepness

public static File searchFileByDeepness(final String directoryName, final String fileName) {
    File target = null;
    if(directoryName != null && fileName != null) {
        File directory = new File(directoryName);
        if(directory.isDirectory()) {
            File file = new File(directoryName, fileName);
            if(file.isFile()) {
                target = file;
            }
            else {
                List<File> subDirectories = getSubDirectories(directory);
                do {
                    List<File> subSubDirectories = new ArrayList<File>();
                    for(File subDirectory : subDirectories) {
                        File fileInSubDirectory = new File(subDirectory, fileName);
                        if(fileInSubDirectory.isFile()) {
                            return fileInSubDirectory;
                        }
                        subSubDirectories.addAll(getSubDirectories(subDirectory));
                    }
                    subDirectories = subSubDirectories;
                } while(subDirectories != null && ! subDirectories.isEmpty());
            }
        }
    }
    return target;
}

private static List<File> getSubDirectories(final File directory) {
    File[] subDirectories = directory.listFiles(new FilenameFilter() {
        @Override
        public boolean accept(final File current, final String name) {
            return new File(current, name).isDirectory();
        }
    });
    return Arrays.asList(subDirectories);
}

For each deepness, the algorithm searches the file inside all folders of the same level. If the file is not found, it tries the next level (deepness++).
Due to the parallel exploration (symmetry), this solution is suitable in most of cases.

Comparison:

public class FileLocationFinder {

    public static void main(final String[] args) {
        String rootFolder = args[0];
        String fileName = args[1];

        long start = System.currentTimeMillis();
        File target = searchFileWithFileUtils(new File(rootFolder), fileName);
        System.out.println(target.getAbsolutePath());
        System.out.println("Duration: " + (System.currentTimeMillis() - start) + "ms");

        start = System.currentTimeMillis();
        target = searchFileRecursive(new File(rootFolder), fileName);
        System.out.println(target.getAbsolutePath());
        System.out.println("Duration: " + (System.currentTimeMillis() - start) + "ms");

        start = System.currentTimeMillis();
        target = searchFileByDeepness(rootFolder, fileName);
        System.out.println(target.getAbsolutePath());
        System.out.println("Duration: " + (System.currentTimeMillis() - start) + "ms");
    }

    // Solution with FileUtils#listFiles
    //--------------------------------------------

    public static File searchFileWithFileUtils(final File file, final String fileName) {
        File target = null;
        if(file.isDirectory()) {
            Collection<File> files = FileUtils.listFiles(file, null, true);
            for (File currFile : files) {
                if (currFile.isFile() && currFile.getName().equals(fileName)) {
                    target = currFile;
                    break;
                }
            }
        }
        return target;
    }

    // Recursive solution
    //--------------------------------------------

    public static File searchFileRecursive(final File file, final String search) {
        if (file.isDirectory()) {
            File[] files = file.listFiles();
            for (File f : files) {
                File target = searchFileRecursive(f, search);
                if(target != null) {
                    return target;
                }
            }
        } else {
            if (search.equals(file.getName())) {
                return file;
            }
        }
        return null;
    }

    // Fastest solution
    //--------------------------------------------

    public static File searchFileByDeepness(final String directoryName, final String fileName) {
        File target = null;
        if(directoryName != null && fileName != null) {
            File directory = new File(directoryName);
            if(directory.isDirectory()) {
                File file = new File(directoryName, fileName);
                if(file.isFile()) {
                    target = file;
                }
                else {
                    List<File> subDirectories = getSubDirectories(directory);
                    do {
                        List<File> subSubDirectories = new ArrayList<File>();
                        for(File subDirectory : subDirectories) {
                            File fileInSubDirectory = new File(subDirectory, fileName);
                            if(fileInSubDirectory.isFile()) {
                                return fileInSubDirectory;
                            }
                            subSubDirectories.addAll(getSubDirectories(subDirectory));
                        }
                        subDirectories = subSubDirectories;
                    } while(subDirectories != null && ! subDirectories.isEmpty());
                }
            }
        }
        return target;
    }

    private static List<File> getSubDirectories(final File directory) {
        File[] subDirectories = directory.listFiles(new FilenameFilter() {
            @Override
            public boolean accept(final File current, final String name) {
                return new File(current, name).isDirectory();
            }
        });
        return Arrays.asList(subDirectories);
    }
}

Result:

searchFileWithFileUtils: 20186ms | searchFileRecursive: 1134ms | searchFileByDeepness: 16ms

[EDIT]
You can also use Java 8 Files API to do this job :

public static File searchFileJava8(final String rootFolder, final String fileName) {
    File target = null;
    Path root = Paths.get(rootFolder);
    try (Stream<Path> stream = Files.find(root, Integer.MAX_VALUE, (path, attr) ->
            path.getFileName().toString().equals(fileName))) {
        Optional<Path> path = stream.findFirst();
        if(path.isPresent()) {
            target = path.get().toFile();
        }
    }
    catch (IOException e) {
    }
    return target;
}

But the execution time is not better (994ms).


问:
OkHttp - detect image download timeout
In my android app I using glide for image loading and OkHttp for network operations.

I would like to show error image using glide if image download takes longer than timeout (eg. 10s). I used GlideModule to connect OkHttpClient with glide but it seems that readTimeout method from OkHttpClient is not working as I thought - it's not waiting for image download completion but it's only waiting for bytes to read.

So my question is: How I can abort okhttp request if image download operation takes more than set timeout? I tried to do this in Interceptor but without success for that moment.

答:
This isn’t implemented in OkHttp. We’re tracking whole operation timeouts here.

If you’re impatient you can use a ScheduledExecutorService or similar to cancel the call after a specified duration.

ContentLicense: CC BY-SA 4.0

Title: OkHttp - detect image download timeout


问:
unable to install package with bower
I am trying to use a build script to build a project. The build runs successfully if ran on developers machine, on my machine or I run it manually on the teamcity agent but fails if the the build is ran through teamcity on the same agent.
This is the error that I am getting.

"git ls-remote --tags --heads https://github.com/jquery/jquery-dist.git";, exit code of #128

I saw a solution to run the command
"git config --global url."https://".insteadOf git://"

this also did not resolve the problem.

the error comes when the command "bower install" is executed. the content of bower.json are

{
  "name": "asp.net",
  "private": true,
  "dependencies": {
    "bootstrap": "3.3.6",
    "jquery": "2.2.0",
    "jquery-validation": "1.16.0",
    "jquery-validation-unobtrusive": "3.2.6",
    "jquery-ui": "1.11.4",
    "Font-Awesome": "4.6.3"
  }
}

The Agent is a windows server 2016 with Visual studio 2017 enterprise installed. I installed nodes v6.11.0LTS manually and bower 1.8.0 using npm installed.

BTW the error keeps coming on different packages, Its not always on the same package name.
I also tried cache clean but It did not work as well.

答:
Visual studio/msbuild was using different version of node and eventually npm, bower and git although the latest version were already installed on the Agent but were not being used by msbuild.
had to go in to the webexternal folder of visual studio and update node and then bower.
search for git in visual studio folder mine was at "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\cmd" had to update git from visual studio.

So although both git and node latest version were installed and were in the path, ms build used the version which were available in visual studio. updated git from visual studio and updated node.js by going into the web external folder of visual studio openign and cmd and ran command to update node. once node was updated I installed bower. Then msbuild started using the latest versions and I did not face the problem again.
BTW I restarted the Agent after the installation as well, dont know if that helped as well.

Title: unable to install package with bower


问:
Finding regions of higher numbers in a matrix
I am working on a project to detect certain objects in an aerial image, and as part of this I am trying to utilize elevation data for the image. I am working with Digital Elevation Models (DEMs), basically a matrix of elevation values. When I am trying to detect trees, for example, I want to search for tree-shaped regions that are higher than their surrounding terrain. Here is an example of a tree in a DEM heatmap:

https://i.stack.imgur.com/pIvlv.png

I want to be able to find small regions like that that are higher than their surroundings.

I am using OpenCV and GDAL for my actual image processing. Do either of those already contain techniques for what I'm trying to accomplish? If not, can you point me in the right direction? Some ideas I've had are going through each pixel and calculating the rate of change in relation to it's surrounding pixels, which would hopefully mean that pixels with high rates change/steep slopes would signify an edge of a raised area.

Note that the elevations will change from image to image, and this needs to work with any elevation. So the ground might be around 10 meters in one image but 20 meters in another.

答:
Supposing you can put the DEM information into a 2D Mat where each "pixel" has the elevation value, you can find local maximums by applying dilate and then substract the result from the original image.

There's a related post with code examples in: http://answers.opencv.org/question/28035/find-local-maximum-in-1d-2d-mat/


问:
What does 'Template error at line 0' mean in django?
I have got an error on a django site that i am working on which is a zero division error. I couldn't find the source of the error which I am thinking that is because of likely mathfilter but still not sure.

What I am asking here is that django gives that error with saying it occurs at line 0 which I don't know what that exactly mean.

This is the beginning of the template file:

```<code>1   {% extends "main.html" %}
2   {% include "ru.html" %}
3   {% load staticfiles %}
4   {% load mathfilters %}
5   {% load humanize %}
</code>```

Only thing that occurs to me when I see this (error at line 0) is that to look at the main.html file. But I couldn't find anything that might cause this zero division error.

How to debug?

What might cause this error?

What does error at line 0 means and which file should i debug then?

Which filters might cause this?

Any possible version problem?

答: Exception ZeroDivisionError is raised when the second argument of a division or modulo operation is zero.There must be such expression in your "main.html" file or other files that link it.Post your main.html code so that I could figure out.</p>

Body:
I have got an error on a django site that i am working on which is a zero division error. I couldn't find the source of the error which I am thinking that is because of likely mathfilter but still not sure.

What I am asking here is that django gives that error with saying it occurs at line 0 which I don't know what that exactly mean.

This is the beginning of the template file:

1   {% extends "main.html" %}
2   {% include "ru.html" %}
3   {% load staticfiles %}
4   {% load mathfilters %}
5   {% load humanize %}

Only thing that occurs to me when I see this (error at line 0) is that to look at the main.html file. But I couldn't find anything that might cause this zero division error.

How to debug?

What might cause this error?

What does error at line 0 means and which file should i debug then?

Which filters might cause this?

Any possible version problem?


问:
find a pattern in files and rename them
I use this command to find files with a given pattern and then rename them to something else

```<code>find . -name '*-GHBAG-*' -exec bash -c 'echo mv $0 ${0/GHBAG/stream-agg}' {} \;
</code>```

As I run this command, I see some outputs like this

```<code>mv ./report-GHBAG-1B ./report-stream-agg-1B
mv ./reoprt-GHBAG-0.5B ./report-stream-agg-0.5B
</code>```

However at the end, when I run ls, I see the old file names.

答:
This works for my needs, replacing all matching files or file types. Be warned, this is a very greedy search

```<code># bashrc
function file_replace() {
  for file in $(find . -type f -name "$1*"); do
    mv $file $(echo "$file" | sed "s/$1/$2/");
  done
}
</code>```

I will usually run with find . -type f -name "MYSTRING*" in advance to check the matches out before replacing.

For example:

```<code>file_replace "Slider.js" "RangeSlider.ts"

renamed:    packages/react-ui-core/src/Form/Slider.js -> packages/react-ui-core/src/Form/RangeSlider.ts
renamed:    stories/examples/Slider.js -> stories/examples/RangeSlider.ts
</code>```

or ditch the filetype to make it even greedier

```<code>file_replace Slider RangeSlider

renamed:    packages/react-ui-core/src/Form/Slider.js -> packages/react-ui-core/src/Form/RangeSlider.js
renamed:    stories/examples/Slider.js -> stories/examples/RangeSlider.js
renamed:    stories/theme/Slider.css -> stories/theme/RangeSlider.css
</code>```

Title: find a pattern in files and rename them

Body:
I use this command to find files with a given pattern and then rename them to something else

find . -name '*-GHBAG-*' -exec bash -c 'echo mv $0 ${0/GHBAG/stream-agg}' {} \;

As I run this command, I see some outputs like this

mv ./report-GHBAG-1B ./report-stream-agg-1B
mv ./reoprt-GHBAG-0.5B ./report-stream-agg-0.5B

However at the end, when I run ls, I see the old file names.

Body:
This works for my needs, replacing all matching files or file types. Be warned, this is a very greedy search

# bashrc
function file_replace() {
  for file in $(find . -type f -name "$1*"); do
    mv $file $(echo "$file" | sed "s/$1/$2/");
  done
}

I will usually run with find . -type f -name "MYSTRING*" in advance to check the matches out before replacing.

For example:

file_replace "Slider.js" "RangeSlider.ts"

renamed:    packages/react-ui-core/src/Form/Slider.js -> packages/react-ui-core/src/Form/RangeSlider.ts
renamed:    stories/examples/Slider.js -> stories/examples/RangeSlider.ts

or ditch the filetype to make it even greedier

file_replace Slider RangeSlider

renamed:    packages/react-ui-core/src/Form/Slider.js -> packages/react-ui-core/src/Form/RangeSlider.js
renamed:    stories/examples/Slider.js -> stories/examples/RangeSlider.js
renamed:    stories/theme/Slider.css -> stories/theme/RangeSlider.css


问:
Unity3D update local assetBundle
In my project, I have a bunch of items stored as assetBundles. In my server, I stored a list of item including their id and version. So ideally, using UnityWebRequest.GetAssetBundle(uri, versionId)I just get the item lists every time I open the app, if first opened, it will download every item; if secondly open, it will only download the one I have updated the version number on the server. Everything is very easy.

But now, I want to store those assetBundles at local first so that people don't need to download with cellular data.Is there an easy way to manage those assetBundles?

答:
You 'dont' keep a local text file. you have the version number inside the local bundle. You have a text file on the server. then get the text file, compare that with the version in the bundle and update if necessary.

In the server, you can have an assetbundle with only a textasset to secure that too.

you can build the asset bundles and have them in the streamingAssets folder or anywhere and use AssetBundle.LoadFromFile()

Title: Unity3D update local assetBundle


问:
How to export all AD user accounts in a specified OU
I'm attempting to export all AD users within an OU to a csv file. However, I do not know how to fill in the first two given variables in a sample script I had pulled from a website.

Given that my LDAP connection string is LDAP://OU=Admin Accounts, OU=Test Accounts,dc=sample,dc=domain,dc=com, how would I fill out the variables for $path, and $pathexist?

Below is the sample script:

```<code>PROCESS #This is where the script executes
{
$path = Split-Path -parent "$CSVReportPath\*.*"
$pathexist = Test-Path -Path $path
If ($pathexist -eq $false)
{New-Item -type directory -Path $path}

$reportdate = Get-Date -Format ssddmmyyyy

$csvreportfile = $path + "\ALLADUsers_$reportdate.csv"

#import the ActiveDirectory Module
Import-Module ActiveDirectory

#Perform AD search. The quotes "" used in $SearchLoc is essential
#Without it, Export-ADUsers returuned error
              Get-ADUser -server $ADServer -searchbase "$SearchLoc" -Properties * -Filter * |
              Select-Object @{Label = "First Name";Expression = {$_.GivenName}},
              @{Label = "Last Name";Expression = {$_.Surname}},
              @{Label = "Display Name";Expression = {$_.DisplayName}},
              @{Label = "Logon Name";Expression = {$_.sAMAccountName}},
              @{Label = "Full address";Expression = {$_.StreetAddress}},
              @{Label = "City";Expression = {$_.City}},
              @{Label = "State";Expression = {$_.st}},
              @{Label = "Post Code";Expression = {$_.PostalCode}},
              @{Label = "Country/Region";Expression = {if (($_.Country -eq 'GB')  ) {'United Kingdom'} Else {''}}},
              @{Label = "Job Title";Expression = {$_.Title}},
              @{Label = "Company";Expression = {$_.Company}},
              @{Label = "Description";Expression = {$_.Description}},
              @{Label = "Department";Expression = {$_.Department}},
              @{Label = "Office";Expression = {$_.OfficeName}},
              @{Label = "Phone";Expression = {$_.telephoneNumber}},
              @{Label = "Email";Expression = {$_.Mail}},
              @{Label = "Manager";Expression = {%{(Get-AdUser $_.Manager -server $ADServer -Properties DisplayName).DisplayName}}},
              @{Label = "Account Status";Expression = {if (($_.Enabled -eq 'TRUE')  ) {'Enabled'} Else {'Disabled'}}}, # the 'if statement# replaces $_.Enabled
              @{Label = "Last LogOn Date";Expression = {$_.lastlogondate}} |

              #Export CSV report
              Export-Csv -Path $csvreportfile -NoTypeInformation
}
</code>```

答:
That sample seems a little extreme.  This should work:

```<code>$csvreportfile = "C:\Temp\Output.csv"
$ADServer = "DC1.domain.com"
Get-ADUser -server $ADServer -searchbase "OU=Admin Accounts, OU=Test Accounts,dc=sample,dc=domain,dc=com" -Properties * -Filter * |
Select-Object @{Label = "First Name";Expression = {$_.GivenName}},
@{Label = "Last Name";Expression = {$_.Surname}},
@{Label = "Display Name";Expression = {$_.DisplayName}},
@{Label = "Logon Name";Expression = {$_.sAMAccountName}},
@{Label = "Full address";Expression = {$_.StreetAddress}},
@{Label = "City";Expression = {$_.City}},
@{Label = "State";Expression = {$_.st}},
@{Label = "Post Code";Expression = {$_.PostalCode}},
@{Label = "Country/Region";Expression = {if (($_.Country -eq 'GB')  ) {'United Kingdom'} Else {''}}},
@{Label = "Job Title";Expression = {$_.Title}},
@{Label = "Company";Expression = {$_.Company}},
@{Label = "Description";Expression = {$_.Description}},
@{Label = "Department";Expression = {$_.Department}},
@{Label = "Office";Expression = {$_.OfficeName}},
@{Label = "Phone";Expression = {$_.telephoneNumber}},
@{Label = "Email";Expression = {$_.Mail}},
@{Label = "Manager";Expression = {%{(Get-AdUser $_.Manager -server $ADServer -Properties DisplayName).DisplayName}}},
@{Label = "Account Status";Expression = {if (($_.Enabled -eq 'TRUE')  ) {'Enabled'} Else {'Disabled'}}}, # the 'if statement# replaces $_.Enabled
@{Label = "Last LogOn Date";Expression = {$_.lastlogondate}} |

#Export CSV report
Export-Csv -Path $csvreportfile -NoTypeInformation
</code>```

To explain the $Path and $Pathexist variables, it is just checking to see if the path for the CSV export exists.  If it doesn't exist, it creates it.  You would need to declare $CSVReportPath as a variable as in:

```<code>$CSVReportPath = "C:\Temp"
</code>```

Body:
I'm attempting to export all AD users within an OU to a csv file. However, I do not know how to fill in the first two given variables in a sample script I had pulled from a website.

Given that my LDAP connection string is LDAP://OU=Admin Accounts, OU=Test Accounts,dc=sample,dc=domain,dc=com, how would I fill out the variables for $path, and $pathexist?

Below is the sample script:

PROCESS #This is where the script executes
{
$path = Split-Path -parent "$CSVReportPath\*.*"
$pathexist = Test-Path -Path $path
If ($pathexist -eq $false)
{New-Item -type directory -Path $path}

$reportdate = Get-Date -Format ssddmmyyyy

$csvreportfile = $path + "\ALLADUsers_$reportdate.csv"

#import the ActiveDirectory Module
Import-Module ActiveDirectory

#Perform AD search. The quotes "" used in $SearchLoc is essential
#Without it, Export-ADUsers returuned error
              Get-ADUser -server $ADServer -searchbase "$SearchLoc" -Properties * -Filter * |
              Select-Object @{Label = "First Name";Expression = {$_.GivenName}},
              @{Label = "Last Name";Expression = {$_.Surname}},
              @{Label = "Display Name";Expression = {$_.DisplayName}},
              @{Label = "Logon Name";Expression = {$_.sAMAccountName}},
              @{Label = "Full address";Expression = {$_.StreetAddress}},
              @{Label = "City";Expression = {$_.City}},
              @{Label = "State";Expression = {$_.st}},
              @{Label = "Post Code";Expression = {$_.PostalCode}},
              @{Label = "Country/Region";Expression = {if (($_.Country -eq 'GB')  ) {'United Kingdom'} Else {''}}},
              @{Label = "Job Title";Expression = {$_.Title}},
              @{Label = "Company";Expression = {$_.Company}},
              @{Label = "Description";Expression = {$_.Description}},
              @{Label = "Department";Expression = {$_.Department}},
              @{Label = "Office";Expression = {$_.OfficeName}},
              @{Label = "Phone";Expression = {$_.telephoneNumber}},
              @{Label = "Email";Expression = {$_.Mail}},
              @{Label = "Manager";Expression = {%{(Get-AdUser $_.Manager -server $ADServer -Properties DisplayName).DisplayName}}},
              @{Label = "Account Status";Expression = {if (($_.Enabled -eq 'TRUE')  ) {'Enabled'} Else {'Disabled'}}}, # the 'if statement# replaces $_.Enabled
              @{Label = "Last LogOn Date";Expression = {$_.lastlogondate}} |

              #Export CSV report
              Export-Csv -Path $csvreportfile -NoTypeInformation
}

Body:
That sample seems a little extreme.  This should work:

$csvreportfile = "C:\Temp\Output.csv"
$ADServer = "DC1.domain.com"
Get-ADUser -server $ADServer -searchbase "OU=Admin Accounts, OU=Test Accounts,dc=sample,dc=domain,dc=com" -Properties * -Filter * |
Select-Object @{Label = "First Name";Expression = {$_.GivenName}},
@{Label = "Last Name";Expression = {$_.Surname}},
@{Label = "Display Name";Expression = {$_.DisplayName}},
@{Label = "Logon Name";Expression = {$_.sAMAccountName}},
@{Label = "Full address";Expression = {$_.StreetAddress}},
@{Label = "City";Expression = {$_.City}},
@{Label = "State";Expression = {$_.st}},
@{Label = "Post Code";Expression = {$_.PostalCode}},
@{Label = "Country/Region";Expression = {if (($_.Country -eq 'GB')  ) {'United Kingdom'} Else {''}}},
@{Label = "Job Title";Expression = {$_.Title}},
@{Label = "Company";Expression = {$_.Company}},
@{Label = "Description";Expression = {$_.Description}},
@{Label = "Department";Expression = {$_.Department}},
@{Label = "Office";Expression = {$_.OfficeName}},
@{Label = "Phone";Expression = {$_.telephoneNumber}},
@{Label = "Email";Expression = {$_.Mail}},
@{Label = "Manager";Expression = {%{(Get-AdUser $_.Manager -server $ADServer -Properties DisplayName).DisplayName}}},
@{Label = "Account Status";Expression = {if (($_.Enabled -eq 'TRUE')  ) {'Enabled'} Else {'Disabled'}}}, # the 'if statement# replaces $_.Enabled
@{Label = "Last LogOn Date";Expression = {$_.lastlogondate}} |

#Export CSV report
Export-Csv -Path $csvreportfile -NoTypeInformation

To explain the $Path and $Pathexist variables, it is just checking to see if the path for the CSV export exists.  If it doesn't exist, it creates it.  You would need to declare $CSVReportPath as a variable as in:

$CSVReportPath = "C:\Temp"


问:
Does throwing an Exception have to cause the program to terminate
Does throwing an Exception have to cause the program to terminate?

I think its no, I just want to make sure

答:
In Java and .NET, if you don't handle an exception, it will most like cause your program to terminate.

Simply throwing an exception will not terminate the program, as such. It is what happens after it was thrown that determines what will occur.