问:
How to make https request with ssl certificate in Retrofit
I have a .p12 certificate file, and I use the SSL Converter to convert it to a .pem certificate file. Then I use that pem certificate file in my android code like this:

```<code>OkHttpClient okHttpClient = new OkHttpClient();
        try {
            CertificateFactory cf = CertificateFactory.getInstance("X.509");
            InputStream instream = context.getResources().openRawResource(R.raw.pem_certificate);
            Certificate ca;
            ca = cf.generateCertificate(instream);
            KeyStore kStore = KeyStore.getInstance(KeyStore.getDefaultType());
            kStore.load(null, null);
            kStore.setCertificateEntry("ca", ca);
            TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
            tmf.init(kStore);
            SSLContext sslContext = SSLContext.getInstance("TLS");
            sslContext.init(null, tmf.getTrustManagers(), null);
            okHttpClient.setSslSocketFactory(sslContext.getSocketFactory());
        } catch (CertificateException
                | KeyStoreException
                | NoSuchAlgorithmException
                | IOException
                | KeyManagementException e) {
            e.printStackTrace();
        }

        baseURL = endpoint;
        RestAdapter restAdapter = new RestAdapter.Builder()
                .setEndpoint(baseURL)
                .setClient(new OkClient(okHttpClient))
                .build();

        service = restAdapter.create(DishService.class);
</code>```

But this code doesn't work. It was failed at the line "ca = cf.generateCertificate(instream);" with CertificateException message.

答:
Step1: Put your security certificate inside raw directory like src/main/res/raw/client_certificate.cer

Step2: Create a SSLContext object with this certificate like

```<code>SSLContext sslContext = createCertificate(getApplicationContext().getResources().openRawResource(R.raw.client_certificate));
</code>```

Step3: Add the created SSLContext object to your OkHttpClient builder.

```<code>addClientCertificate(okHttpClientBuilder);
</code>```

Step4: Set the OkHttpClient builder to retrofit client like.

```<code>retrofitBuilder.client(okHttpClientBuilder.build());
</code>```

Here is complete implementation for all above mentioned steps in RetrofitManager.java file.

RetrofitManager.java

```<code>import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

import java.io.IOException;
import java.io.InputStream;
import java.security.GeneralSecurityException;
import java.security.KeyManagementException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.util.concurrent.TimeUnit;

import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.TrustManagerFactory;
import javax.net.ssl.X509TrustManager;
import java.security.cert.Certificate;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.logging.HttpLoggingInterceptor;
import retrofit2.Retrofit;
import retrofit2.adapter.rxjava.RxJavaCallAdapterFactory;
import retrofit2.converter.gson.GsonConverterFactory;

/**
 * Created by Rajoo Kannaujiya on 02/16/2020.
 */

public class RetrofitManager {
    private static final String URL = "https://stackoverflow.com"; // Fetch url from config file.
    private static final String TLS = "TLS";
    private static final String SSL = "SSL";
    private static final String CA = "ca";
    private static final String X_509 = "X.509";
    private static final String PROD = "prod";
    private static X509TrustManager trustManager = null;
    private static final String CONTENT_TYPE = "Content-Type";
    private static final String CONTENT_TYPE_VALUE = "application/json";
    public static final String BUILD_TYPE = "prod"; // Fetch build type from config file (Whether it is a dev, qa, stage or prod build).

    public static Retrofit getRetrofitForAPIUrl() {
        return getRetrofit(URL);
    }

    private static Retrofit getRetrofit(String url) {
        Gson gson = new GsonBuilder().setLenient().create();

        Retrofit.Builder retrofitBuilder = new Retrofit.Builder()
                .addCallAdapterFactory(RxJavaCallAdapterFactory.create())
                .addConverterFactory(GsonConverterFactory.create(gson))
                .baseUrl(url);

        HttpLoggingInterceptor logging = new HttpLoggingInterceptor();
        logging.level(HttpLoggingInterceptor.Level.BODY);

        OkHttpClient.Builder okHttpClientBuilder = new OkHttpClient.Builder();
        okHttpClientBuilder.addInterceptor(chain -> {
            Request origReq = chain.request();
            Request.Builder requestBuilder = origReq.newBuilder()
                    .addHeader(CONTENT_TYPE, CONTENT_TYPE_VALUE);

            Request request = requestBuilder.build();
            return chain.proceed(request);
        });

        addClientCertificate(okHttpClientBuilder);

        OkHttpClient.Builder okHttpBuilder = okHttpClientBuilder
                .connectTimeout(10, TimeUnit.SECONDS)
                .writeTimeout(20, TimeUnit.SECONDS)
                .readTimeout(30, TimeUnit.SECONDS)
                .addInterceptor(logging);
        retrofitBuilder.client(okHttpBuilder.build());

        return retrofitBuilder.build();
    }

    /**
     * Method to create SSLContext object for the client certificate and return it.
     *
     * @param clientCertificate
     * @return SSLContext Object
     */
    private static SSLContext getSSLContextForClientCertificate(InputStream clientCertificate) {
        SSLContext sslContext = null;
        // If given client certificate is for production environment only, Then check put check for build type.
        if (PROD.contentEquals(BUILD_TYPE)) {
            // Creating X.509 certificate factory instance
            CertificateFactory cf = null;
            try {
                cf = CertificateFactory.getInstance(X_509);
            } catch (CertificateException e) {
                e.printStackTrace();
            }
            // Generating client certificate
            Certificate ca = null;
            try {
                try {
                    if (cf != null) {
                        ca = cf.generateCertificate(clientCertificate);
                    }
                } catch (CertificateException e) {
                    e.printStackTrace();
                }
            } finally {
                try {
                    clientCertificate.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
            // Creating a KeyStore containing our trusted CAs
            String keyStoreType = KeyStore.getDefaultType();
            KeyStore keyStore = null;
            try {
                keyStore = KeyStore.getInstance(keyStoreType);
            } catch (KeyStoreException e) {
                e.printStackTrace();
            }
            if (keyStore != null) {
                try {
                    keyStore.load(null, null);
                } catch (CertificateException | IOException | NoSuchAlgorithmException e) {
                    e.printStackTrace();
                }
                try {
                    keyStore.setCertificateEntry(CA, ca);
                } catch (KeyStoreException e) {
                    e.printStackTrace();
                }
            }
            // Creating a TrustManager that trusts the CAs in our KeyStore
            String tmfAlgorithm = TrustManagerFactory.getDefaultAlgorithm();
            TrustManagerFactory tmf = null;
            try {
                tmf = TrustManagerFactory.getInstance(tmfAlgorithm);
            } catch (NoSuchAlgorithmException e) {
                e.printStackTrace();
            }
            try {
                if (tmf != null) {
                    tmf.init(keyStore);
                }
            } catch (KeyStoreException e) {
                e.printStackTrace();
            }
            // Creating an SSLContext instance that uses our TrustManager
            try {
                sslContext = SSLContext.getInstance(TLS);
            } catch (NoSuchAlgorithmException e) {
                e.printStackTrace();
            }
            try {
                if (sslContext != null && tmf != null) {
                    sslContext.init(null, tmf.getTrustManagers(), null);
                }
            } catch (KeyManagementException e) {
                e.printStackTrace();
            }
        } else { // For non prod environment build. This section is useful only if you don't have client certificate for lower environment.
            try {
                sslContext = SSLContext.getInstance(SSL);
                sslContext.init(null, new TrustManager[]{getTrustManager()}, new java.security.SecureRandom());
            } catch (NoSuchAlgorithmException | KeyManagementException e) {
                e.printStackTrace();
            }
        }
        trustManager = getTrustManager();
        return sslContext;
    }

    /**
     * Method to add client certificate to OkHttpClient.Builder object.
     *
     * @param okHttpClientBuilder
     * @return void
     */
    private static void addClientCertificate(OkHttpClient.Builder okHttpClientBuilder) {
        SSLContext sslContext = getSSLContextForClientCertificate(getApplicationContext().getResources().openRawResource(R.raw.myaccount_coxbusiness_com));
        if (sslContext != null) {
            okHttpClientBuilder.sslSocketFactory(sslContext.getSocketFactory(), trustManager);
            okHttpClientBuilder.hostnameVerifier((hostname, session) -> true);
        }
    }

    private static X509TrustManager getTrustManager() {
        X509TrustManager trustManager = null;
        if (PROD.contentEquals(BUILD_TYPE)) { // If client certificate is only available for prod environment.
            try {
                TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
                trustManagerFactory.init((KeyStore) null);
                TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();
                if (trustManagers[0] instanceof X509TrustManager) {
                    trustManager = (X509TrustManager) trustManagers[0];
                }
            } catch (GeneralSecurityException e) {
                throw new AssertionError();
            }
        } else { // For non prod environment build. This section is useful only if you don't have client certificate for lower environment.
            trustManager = new X509TrustManager() {
                @Override
                public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

                }

                @Override
                public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

                }

                @Override
                public X509Certificate[] getAcceptedIssuers() {
                    return new X509Certificate[0];
                }
            };
        }
        return trustManager;
    }
}
</code>```

Body:
I have a .p12 certificate file, and I use the SSL Converter to convert it to a .pem certificate file. Then I use that pem certificate file in my android code like this:

OkHttpClient okHttpClient = new OkHttpClient();
        try {
            CertificateFactory cf = CertificateFactory.getInstance("X.509");
            InputStream instream = context.getResources().openRawResource(R.raw.pem_certificate);
            Certificate ca;
            ca = cf.generateCertificate(instream);
            KeyStore kStore = KeyStore.getInstance(KeyStore.getDefaultType());
            kStore.load(null, null);
            kStore.setCertificateEntry("ca", ca);
            TrustManagerFactory tmf = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
            tmf.init(kStore);
            SSLContext sslContext = SSLContext.getInstance("TLS");
            sslContext.init(null, tmf.getTrustManagers(), null);
            okHttpClient.setSslSocketFactory(sslContext.getSocketFactory());
        } catch (CertificateException
                | KeyStoreException
                | NoSuchAlgorithmException
                | IOException
                | KeyManagementException e) {
            e.printStackTrace();
        }

        baseURL = endpoint;
        RestAdapter restAdapter = new RestAdapter.Builder()
                .setEndpoint(baseURL)
                .setClient(new OkClient(okHttpClient))
                .build();

        service = restAdapter.create(DishService.class);

But this code doesn't work. It was failed at the line "ca = cf.generateCertificate(instream);" with CertificateException message.

Body:
Step1: Put your security certificate inside raw directory like src/main/res/raw/client_certificate.cer

Step2: Create a SSLContext object with this certificate like

SSLContext sslContext = createCertificate(getApplicationContext().getResources().openRawResource(R.raw.client_certificate));

Step3: Add the created SSLContext object to your OkHttpClient builder.

addClientCertificate(okHttpClientBuilder);

Step4: Set the OkHttpClient builder to retrofit client like.

retrofitBuilder.client(okHttpClientBuilder.build());

Here is complete implementation for all above mentioned steps in RetrofitManager.java file.

RetrofitManager.java

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;

import java.io.IOException;
import java.io.InputStream;
import java.security.GeneralSecurityException;
import java.security.KeyManagementException;
import java.security.KeyStore;
import java.security.KeyStoreException;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.CertificateFactory;
import java.security.cert.X509Certificate;
import java.util.concurrent.TimeUnit;

import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.TrustManagerFactory;
import javax.net.ssl.X509TrustManager;
import java.security.cert.Certificate;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import okhttp3.logging.HttpLoggingInterceptor;
import retrofit2.Retrofit;
import retrofit2.adapter.rxjava.RxJavaCallAdapterFactory;
import retrofit2.converter.gson.GsonConverterFactory;

/**
 * Created by Rajoo Kannaujiya on 02/16/2020.
 */

public class RetrofitManager {
    private static final String URL = "https://stackoverflow.com"; // Fetch url from config file.
    private static final String TLS = "TLS";
    private static final String SSL = "SSL";
    private static final String CA = "ca";
    private static final String X_509 = "X.509";
    private static final String PROD = "prod";
    private static X509TrustManager trustManager = null;
    private static final String CONTENT_TYPE = "Content-Type";
    private static final String CONTENT_TYPE_VALUE = "application/json";
    public static final String BUILD_TYPE = "prod"; // Fetch build type from config file (Whether it is a dev, qa, stage or prod build).

    public static Retrofit getRetrofitForAPIUrl() {
        return getRetrofit(URL);
    }

    private static Retrofit getRetrofit(String url) {
        Gson gson = new GsonBuilder().setLenient().create();

        Retrofit.Builder retrofitBuilder = new Retrofit.Builder()
                .addCallAdapterFactory(RxJavaCallAdapterFactory.create())
                .addConverterFactory(GsonConverterFactory.create(gson))
                .baseUrl(url);

        HttpLoggingInterceptor logging = new HttpLoggingInterceptor();
        logging.level(HttpLoggingInterceptor.Level.BODY);

        OkHttpClient.Builder okHttpClientBuilder = new OkHttpClient.Builder();
        okHttpClientBuilder.addInterceptor(chain -> {
            Request origReq = chain.request();
            Request.Builder requestBuilder = origReq.newBuilder()
                    .addHeader(CONTENT_TYPE, CONTENT_TYPE_VALUE);

            Request request = requestBuilder.build();
            return chain.proceed(request);
        });

        addClientCertificate(okHttpClientBuilder);

        OkHttpClient.Builder okHttpBuilder = okHttpClientBuilder
                .connectTimeout(10, TimeUnit.SECONDS)
                .writeTimeout(20, TimeUnit.SECONDS)
                .readTimeout(30, TimeUnit.SECONDS)
                .addInterceptor(logging);
        retrofitBuilder.client(okHttpBuilder.build());

        return retrofitBuilder.build();
    }

    /**
     * Method to create SSLContext object for the client certificate and return it.
     *
     * @param clientCertificate
     * @return SSLContext Object
     */
    private static SSLContext getSSLContextForClientCertificate(InputStream clientCertificate) {
        SSLContext sslContext = null;
        // If given client certificate is for production environment only, Then check put check for build type.
        if (PROD.contentEquals(BUILD_TYPE)) {
            // Creating X.509 certificate factory instance
            CertificateFactory cf = null;
            try {
                cf = CertificateFactory.getInstance(X_509);
            } catch (CertificateException e) {
                e.printStackTrace();
            }
            // Generating client certificate
            Certificate ca = null;
            try {
                try {
                    if (cf != null) {
                        ca = cf.generateCertificate(clientCertificate);
                    }
                } catch (CertificateException e) {
                    e.printStackTrace();
                }
            } finally {
                try {
                    clientCertificate.close();
                } catch (IOException e) {
                    e.printStackTrace();
                }
            }
            // Creating a KeyStore containing our trusted CAs
            String keyStoreType = KeyStore.getDefaultType();
            KeyStore keyStore = null;
            try {
                keyStore = KeyStore.getInstance(keyStoreType);
            } catch (KeyStoreException e) {
                e.printStackTrace();
            }
            if (keyStore != null) {
                try {
                    keyStore.load(null, null);
                } catch (CertificateException | IOException | NoSuchAlgorithmException e) {
                    e.printStackTrace();
                }
                try {
                    keyStore.setCertificateEntry(CA, ca);
                } catch (KeyStoreException e) {
                    e.printStackTrace();
                }
            }
            // Creating a TrustManager that trusts the CAs in our KeyStore
            String tmfAlgorithm = TrustManagerFactory.getDefaultAlgorithm();
            TrustManagerFactory tmf = null;
            try {
                tmf = TrustManagerFactory.getInstance(tmfAlgorithm);
            } catch (NoSuchAlgorithmException e) {
                e.printStackTrace();
            }
            try {
                if (tmf != null) {
                    tmf.init(keyStore);
                }
            } catch (KeyStoreException e) {
                e.printStackTrace();
            }
            // Creating an SSLContext instance that uses our TrustManager
            try {
                sslContext = SSLContext.getInstance(TLS);
            } catch (NoSuchAlgorithmException e) {
                e.printStackTrace();
            }
            try {
                if (sslContext != null && tmf != null) {
                    sslContext.init(null, tmf.getTrustManagers(), null);
                }
            } catch (KeyManagementException e) {
                e.printStackTrace();
            }
        } else { // For non prod environment build. This section is useful only if you don't have client certificate for lower environment.
            try {
                sslContext = SSLContext.getInstance(SSL);
                sslContext.init(null, new TrustManager[]{getTrustManager()}, new java.security.SecureRandom());
            } catch (NoSuchAlgorithmException | KeyManagementException e) {
                e.printStackTrace();
            }
        }
        trustManager = getTrustManager();
        return sslContext;
    }

    /**
     * Method to add client certificate to OkHttpClient.Builder object.
     *
     * @param okHttpClientBuilder
     * @return void
     */
    private static void addClientCertificate(OkHttpClient.Builder okHttpClientBuilder) {
        SSLContext sslContext = getSSLContextForClientCertificate(getApplicationContext().getResources().openRawResource(R.raw.myaccount_coxbusiness_com));
        if (sslContext != null) {
            okHttpClientBuilder.sslSocketFactory(sslContext.getSocketFactory(), trustManager);
            okHttpClientBuilder.hostnameVerifier((hostname, session) -> true);
        }
    }

    private static X509TrustManager getTrustManager() {
        X509TrustManager trustManager = null;
        if (PROD.contentEquals(BUILD_TYPE)) { // If client certificate is only available for prod environment.
            try {
                TrustManagerFactory trustManagerFactory = TrustManagerFactory.getInstance(TrustManagerFactory.getDefaultAlgorithm());
                trustManagerFactory.init((KeyStore) null);
                TrustManager[] trustManagers = trustManagerFactory.getTrustManagers();
                if (trustManagers[0] instanceof X509TrustManager) {
                    trustManager = (X509TrustManager) trustManagers[0];
                }
            } catch (GeneralSecurityException e) {
                throw new AssertionError();
            }
        } else { // For non prod environment build. This section is useful only if you don't have client certificate for lower environment.
            trustManager = new X509TrustManager() {
                @Override
                public void checkClientTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

                }

                @Override
                public void checkServerTrusted(X509Certificate[] x509Certificates, String s) throws CertificateException {

                }

                @Override
                public X509Certificate[] getAcceptedIssuers() {
                    return new X509Certificate[0];
                }
            };
        }
        return trustManager;
    }
}


来源: stackexchange

问:
talend tsystem multiline commands
I have command to move files between s3 folders. I am getting bucket name from context variable.
placed the command in array line

"aws s3 mv s3://"+context.bucket+"/Egress/test1/abc/xyz.dat s3://"+context.bucket+"/Egress/test1/abc/archive/archive_xyz.dat"

The command fetches the bucket name from context variable, but shows no file or direcrtory error=2.
I think it is due to (") at begin and end.
Is there any way to solve.

please help

答:
You probably want to use an array command.
Using /bin/bash or cmd

then your command.

ContentLicense: CC BY-SA 4.0

Title: talend tsystem multiline commands


问:
Understanding GraphQL
I started experimenting with the GraphQL wp api.

I am querying the menus. As for the documentation, the query is very long

I would expect that querying

```<code>{
    menus
}
</code>```

only would bring about all the data nested in menus, it does not.

Why is this? What is the way to getting all nested data in an object as to see what's in there?

Thank you for your time

答:
The rule is that every "leaf" fields in a GraphQL query should be a Scalar something like Int , Boolean , String etc.  So if the meuns field in the root Query type is a Scalar , it is a valid query and will return you something.

If not , you have to continue navigating the Menu type and pick the fields that you want to include in the GraphQL query such as :

```<code>{
   menus {
     id
     createdDate
  }
}
</code>```

There is no wildcard that can represent all fields in current GraphQL spec.You have to explicitly declare all fields you want to select in the query.By looking at the GraphQL schema, you can know the available fields for each type. One of the tips is to rely on the GraphQL introspection system .It basically means that you can use some of the GraphQL client such as Altair, Graphiql, or GraphQL Playground etc. which most of them will have some auto-suggest function that will guide you to compose a query by suggesting you what fields are available to be included for a type .

P.S. A similar analogy to SQL is that there is no select * from foo , you have to explicitly define the columns that you want to select in the select clause such as select id,name,address from foo.

Title: Understanding GraphQL

Body:
I started experimenting with the GraphQL wp api.

I am querying the menus. As for the documentation, the query is very long

I would expect that querying

{
    menus
}

only would bring about all the data nested in menus, it does not.

Why is this? What is the way to getting all nested data in an object as to see what's in there?

Thank you for your time

Body:
The rule is that every "leaf" fields in a GraphQL query should be a Scalar something like Int , Boolean , String etc.  So if the meuns field in the root Query type is a Scalar , it is a valid query and will return you something.

If not , you have to continue navigating the Menu type and pick the fields that you want to include in the GraphQL query such as :

{
   menus {
     id
     createdDate
  }
}

There is no wildcard that can represent all fields in current GraphQL spec.You have to explicitly declare all fields you want to select in the query.By looking at the GraphQL schema, you can know the available fields for each type. One of the tips is to rely on the GraphQL introspection system .It basically means that you can use some of the GraphQL client such as Altair, Graphiql, or GraphQL Playground etc. which most of them will have some auto-suggest function that will guide you to compose a query by suggesting you what fields are available to be included for a type .

P.S. A similar analogy to SQL is that there is no select * from foo , you have to explicitly define the columns that you want to select in the select clause such as select id,name,address from foo.


问:
How do you update objects in a document's array (nested updating)
Assume we have the following collection, which I have few questions about:

```<code>{
    "_id" : ObjectId("4faaba123412d654fe83hg876"),
    "user_id" : 123456,
    "total" : 100,
    "items" : [
            {
                    "item_name" : "my_item_one",
                    "price" : 20
            },
            {
                    "item_name" : "my_item_two",
                    "price" : 50
            },
            {
                    "item_name" : "my_item_three",
                    "price" : 30
            }
    ]
}
</code>```

I want to increase the price for "item_name":"my_item_two" and if it doesn't exists, it should be appended to the "items" array.

How can I update two fields at the same time? For example, increase the price for "my_item_three" and at the same time increase the "total" (with the same value).

I prefer to do this on the MongoDB side, otherwise I have to load the document in client-side (Python) and construct the updated document and replace it with the existing one in MongoDB.

This is what I have tried and works fine if the object exists:

```<code>db.test_invoice.update({user_id : 123456 , "items.item_name":"my_item_one"} , {$inc: {"items.$.price": 10}})
</code>```

However, if the key doesn't exist, it does nothing.
Also, it only updates the nested object. There is no way with this command to update the "total" field as well.

答:
We can use $set operator to update the nested array inside object filed update the value

```<code>db.getCollection('geolocations').update(
   {
       "_id" : ObjectId("5bd3013ac714ea4959f80115"),
       "geolocation.country" : "United States of America"
   },
   { $set:
       {
           "geolocation.$.country" : "USA"
       }
    },
   false,
   true
);
</code>```

Body:
Assume we have the following collection, which I have few questions about:

{
    "_id" : ObjectId("4faaba123412d654fe83hg876"),
    "user_id" : 123456,
    "total" : 100,
    "items" : [
            {
                    "item_name" : "my_item_one",
                    "price" : 20
            },
            {
                    "item_name" : "my_item_two",
                    "price" : 50
            },
            {
                    "item_name" : "my_item_three",
                    "price" : 30
            }
    ]
}

I want to increase the price for "item_name":"my_item_two" and if it doesn't exists, it should be appended to the "items" array.

How can I update two fields at the same time? For example, increase the price for "my_item_three" and at the same time increase the "total" (with the same value).

I prefer to do this on the MongoDB side, otherwise I have to load the document in client-side (Python) and construct the updated document and replace it with the existing one in MongoDB.

This is what I have tried and works fine if the object exists:

db.test_invoice.update({user_id : 123456 , "items.item_name":"my_item_one"} , {$inc: {"items.$.price": 10}})

However, if the key doesn't exist, it does nothing.
Also, it only updates the nested object. There is no way with this command to update the "total" field as well.

Body:
We can use $set operator to update the nested array inside object filed update the value

db.getCollection('geolocations').update(
   {
       "_id" : ObjectId("5bd3013ac714ea4959f80115"),
       "geolocation.country" : "United States of America"
   },
   { $set:
       {
           "geolocation.$.country" : "USA"
       }
    },
   false,
   true
);


问:
My regex correctly doesn't accept a string online, but preg_match() returns true
I have this regular expression /^(string@)([^#\s\\\\]|(\\\d{3}))*$/.

It's supposed to accept strings that start with string@, then all characters but #, \, or any whitespace characters can follow. \ can only appear when it's followed by 3 digits.

When I test this regex with the string string@test\07test on regex101.com and phpliveregex.com, no matches are found (correctly), but preg_match() returns true for the same string.

What am I doing wrong?

Thank you

UPDATE:

```<code>$regex = "/^(string@)([^#\s\\\\]|(\\\d{3}))*$/";

preg_match($regex, 'string@test\07test'); # this returns false, as it should

preg_match($regex, "string@test\07test"); # this returns true
</code>```

The test string is taken from a file using fgets() and stored in a variable, though, so I can't pass it to preg_match() and put apostrophes around it, I need to pass the variable ($string, for example), and preg_match($regex, $string) incorrectly returns true again.

答:
First things first:

'\07' is a string of length 3 beginning with a backslash character.

"\07" is a string of length 1 consisting of ASCII x'07'

```<code>Interactive shell

php > echo strlen('\07');
3
php > echo strlen("\07");
1
php >
</code>```

See PHP Strings

Regex:

```<code>/^string@(?:[^#\s\\]|\\(?=\d{3}))*$/
</code>```

See Regex Demo

^ Matches start of string.

string@ Matches string@.

[^#\s\\] Matches any character other than #, white space or \.

| or

\\(?=\d{3}) Matches \ if followed by 3 digits.

(?: expression 3 through 5)* matched 0 or more times.

$ Matches the end of string.

Code:

```<code><?php
$tests = [
    'abcd',
    'string@ test',
    'string@#test',
    'string@\07test',
    'string@\075test',
    'string@test'
];

$regex = '/^string@(?:[^#\s\\\\]|\\\\(?=\d{3}))*$/';
foreach ($tests as $test) {
    if (preg_match($regex, $test)) {
        echo "$test\n";
    }
}
</code>```

Prints:

```<code>string@\075test
string@test
</code>```

Body:
I have this regular expression /^(string@)([^#\s\\\\]|(\\\d{3}))*$/.

It's supposed to accept strings that start with string@, then all characters but #, \, or any whitespace characters can follow. \ can only appear when it's followed by 3 digits.

When I test this regex with the string string@test\07test on regex101.com and phpliveregex.com, no matches are found (correctly), but preg_match() returns true for the same string.

What am I doing wrong?

Thank you

UPDATE:

$regex = "/^(string@)([^#\s\\\\]|(\\\d{3}))*$/";

preg_match($regex, 'string@test\07test'); # this returns false, as it should

preg_match($regex, "string@test\07test"); # this returns true

The test string is taken from a file using fgets() and stored in a variable, though, so I can't pass it to preg_match() and put apostrophes around it, I need to pass the variable ($string, for example), and preg_match($regex, $string) incorrectly returns true again.

Body:
First things first:

'\07' is a string of length 3 beginning with a backslash character.

"\07" is a string of length 1 consisting of ASCII x'07'

Interactive shell

php > echo strlen('\07');
3
php > echo strlen("\07");
1
php >

See PHP Strings

Regex:

/^string@(?:[^#\s\\]|\\(?=\d{3}))*$/

See Regex Demo

^ Matches start of string.

string@ Matches string@.

[^#\s\\] Matches any character other than #, white space or \.

| or

\\(?=\d{3}) Matches \ if followed by 3 digits.

(?: expression 3 through 5)* matched 0 or more times.

$ Matches the end of string.

Code:

<?php
$tests = [
    'abcd',
    'string@ test',
    'string@#test',
    'string@\07test',
    'string@\075test',
    'string@test'
];

$regex = '/^string@(?:[^#\s\\\\]|\\\\(?=\d{3}))*$/';
foreach ($tests as $test) {
    if (preg_match($regex, $test)) {
        echo "$test\n";
    }
}

Prints:

string@\075test
string@test


问:
TypeScript compiler is not emitting imported modules
I have the project (on GitHub), which uses the Project References functionality of TypeScript compiler, so it have the root tsconfig.json file and the tsconfig.json file of subfolder named shared.

Shared subfolder have 2 files:

User.ts

Point.ts

The main index.ts file imports the User module, like

import User from './shared/User';
```

<p>Transpiled files is placed under the <code>target</code> folder. The issue I met is that as result of <code>tsc</code> the <code>target</code> folder has <code>Point</code> .d.ts and .js files (<strong>not imported</strong> by <code>index.ts</code>) and <strong>doesn't</strong> contains the <code>User</code> files (imported by <code>index.ts</code>).</p>

<p>The output of <code>tsc -b --listEmittedFiles</code>:</p>

```TSFILE: /home/user/Projects/test/target/index.js
TSFILE: /home/user/Projects/test/target/index.d.ts
TSFILE: /home/user/Projects/test/target/shared/Point.js
TSFILE: /home/user/Projects/test/target/shared/Point.d.ts
TSFILE: /home/user/Projects/test/target/shared/User.js
TSFILE: /home/user/Projects/test/target/shared/User.d.ts
```

<p>If I remove the <code>import</code> expression from <code>index.ts</code>, the target folder will have both <code>User</code> and <code>Point</code> scripts, as expected.</p>

<p>My question is: Is it bug of <code>tsc</code> or I'm doing something wrong?</p>

<p>Tested on both <code>3.3</code> and <code>3.4</code> versions of <code>tsc</code>, with same result.</p>

<p>The root <code>tsconfig.json</code>:</p>

```{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "declaration": true,
    "outDir": "./target",
    "rootDir": "./source",
    "strict": true,
    "esModuleInterop": true
  },
  "references": [
    {
      "path": "./source/shared"
    }
  ]
}
```

<p>The <code>tsconfig.json</code> of <code>shared</code> folder:</p>

```{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "declaration": true,
    "composite": true,
    "strict": true,
    "esModuleInterop": true
  }
}
```

答:
Finally, I got the solution by myself!

tscconfig.json files of subprojects should point output to the target/ directory of main project!

So, let say, you have general project with source/ directory defined as rootDir and target/ directory as outDir.

In the source directory you have two subprojects: core and geometry (as it shown here).

So, both of them should have outDir set to the main target/ directory, like that: ../../target/core, ../../target/geometry.

Or, use configuration inheritance with "extends" property. In this case the config files of subprojects may be rewritten like that (example):

```<code>{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "composite": true
  }
}
</code>```

Body:
I have the project (on GitHub), which uses the Project References functionality of TypeScript compiler, so it have the root tsconfig.json file and the tsconfig.json file of subfolder named shared.

Shared subfolder have 2 files:

User.ts

Point.ts

The main index.ts file imports the User module, like

import User from './shared/User';

Transpiled files is placed under the target folder. The issue I met is that as result of tsc the target folder has Point .d.ts and .js files (not imported by index.ts) and doesn't contains the User files (imported by index.ts).

The output of tsc -b --listEmittedFiles:

TSFILE: /home/user/Projects/test/target/index.js
TSFILE: /home/user/Projects/test/target/index.d.ts
TSFILE: /home/user/Projects/test/target/shared/Point.js
TSFILE: /home/user/Projects/test/target/shared/Point.d.ts
TSFILE: /home/user/Projects/test/target/shared/User.js
TSFILE: /home/user/Projects/test/target/shared/User.d.ts

If I remove the import expression from index.ts, the target folder will have both User and Point scripts, as expected.

My question is: Is it bug of tsc or I'm doing something wrong?

Tested on both 3.3 and 3.4 versions of tsc, with same result.

The root tsconfig.json:

{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "declaration": true,
    "outDir": "./target",
    "rootDir": "./source",
    "strict": true,
    "esModuleInterop": true
  },
  "references": [
    {
      "path": "./source/shared"
    }
  ]
}

The tsconfig.json of shared folder:

{
  "compilerOptions": {
    "target": "es5",
    "module": "commonjs",
    "declaration": true,
    "composite": true,
    "strict": true,
    "esModuleInterop": true
  }
}

Body:
Finally, I got the solution by myself!

tscconfig.json files of subprojects should point output to the target/ directory of main project!

So, let say, you have general project with source/ directory defined as rootDir and target/ directory as outDir.

In the source directory you have two subprojects: core and geometry (as it shown here).

So, both of them should have outDir set to the main target/ directory, like that: ../../target/core, ../../target/geometry.

Or, use configuration inheritance with "extends" property. In this case the config files of subprojects may be rewritten like that (example):

{
  "extends": "../../tsconfig.json",
  "compilerOptions": {
    "composite": true
  }
}


问:
Pandas get date difference between 2 ISO date
I have 2 date strings: 2020-02-12T16:02:51Z and 2017-03-08T18:16:02-05:00, and I'd like to get the difference in days that includes partial day difference (for example, a difference of 1 day 12 hours 00 minutes 00 seconds will be 1.5 days).

Here is what I have:

```<code>import pandas as pd

date1 = pd.to_datetime("2020-02-12T16:02:51Z", utc=True)
date2 = pd.to_datetime("2017-03-08T18:16:02-05:00", utc=True)
diff = date1 - date2

diff.days
>>> 1070
</code>```

I expect it should be 1070.<some decimal digits>. Because diff is Timedelta('1070 days 16:46:49')

What did I do wrong? I am using Python 3.8.1 and pandas 1.0.1

答:
Timedelta.days represents the number of days in the delta. So, in your case 1070.

 However, you have different options to get the results in fractional form.

```<code>>>> diff = date1 - date2
>>> diff.days + diff.seconds/86400
1070.6991782407408
>>> diff.total_seconds()/86400
1070.6991782407408
</code>```

Body:
I have 2 date strings: 2020-02-12T16:02:51Z and 2017-03-08T18:16:02-05:00, and I'd like to get the difference in days that includes partial day difference (for example, a difference of 1 day 12 hours 00 minutes 00 seconds will be 1.5 days).

Here is what I have:

import pandas as pd

date1 = pd.to_datetime("2020-02-12T16:02:51Z", utc=True)
date2 = pd.to_datetime("2017-03-08T18:16:02-05:00", utc=True)
diff = date1 - date2

diff.days
>>> 1070

I expect it should be 1070.<some decimal digits>. Because diff is Timedelta('1070 days 16:46:49')

What did I do wrong? I am using Python 3.8.1 and pandas 1.0.1

Body:
Timedelta.days represents the number of days in the delta. So, in your case 1070.

 However, you have different options to get the results in fractional form.

>>> diff = date1 - date2
>>> diff.days + diff.seconds/86400
1070.6991782407408
>>> diff.total_seconds()/86400
1070.6991782407408


问:
Select matching raws in table with Minizinc
I have a table of Projects with 4 features (name, value, cost, people). I want to describe a model in Minizinc that selects the projects that maximize the total value but with a total budget of 255 and I can choose almost 9 projects between 20.

I write a data file: data.dzn

```<code>%       name       value    cost  people
data = [
      | PRJ0,      6000,    35,    5
      | PRJ1,      4000,    34,    3
      | PRJ2,      1000,    26,    4
      ...
      | PRJ20,     1200,    18,    2
      |];

PRJ = {PRJ0,...,PRJ20};
FEATURE = {name, value, budget, personnel};
max_budget = 225;
max_prj=9;
</code>```

So my constraints are:

```<code>choose_project <= 9 /\ budget<=255 s.t solve maximize tot_value;
</code>```

How can I select a non-arbitrary number (1 to 9) of decision variables among projects raw in the  table?
Until now this is my code: invest.mzn

```<code>include "data.dzn";
int: max_budget;  %255
int: max_prj;     %9
enum FEATURE;
enum PRJ;
array[PRJ,FEATURE] of int: data;
constraint ...
...
solve maximize tot_value;
</code>```

答:
You can declare an array of Boolean variables, say selected_projects, that encodes whether or not a project PRJ_k is selected or not.

Then you can simply count how many projects in this array are being selected at the same time.

Example:

```<code>enum FEATURE = {name, value, budget, personnel};
enum PRJ = { PRJ0, PRJ1 };

array[PRJ,FEATURE] of int: data =
     [| PRJ0,      6000,    35,    5
      | PRJ1,      4000,    34,    3
      |];

array[PRJ] of var bool: selected_projects;

var int: tot_value;

% The total number of selected projects must be in [1, 9]
constraint let {
        var int: tot_selected = sum(prj in PRJ) ( selected_projects[prj] )
    } in
        1 <= tot_selected /\ tot_selected <= 9;

constraint tot_value = sum(prj in PRJ) ( selected_projects[prj] * data[prj, value] );

% ...
% encoding of budget and personnel constraints
% ...

solve maximize tot_value;
</code>```

Body:
I have a table of Projects with 4 features (name, value, cost, people). I want to describe a model in Minizinc that selects the projects that maximize the total value but with a total budget of 255 and I can choose almost 9 projects between 20.

I write a data file: data.dzn

%       name       value    cost  people
data = [
      | PRJ0,      6000,    35,    5
      | PRJ1,      4000,    34,    3
      | PRJ2,      1000,    26,    4
      ...
      | PRJ20,     1200,    18,    2
      |];

PRJ = {PRJ0,...,PRJ20};
FEATURE = {name, value, budget, personnel};
max_budget = 225;
max_prj=9;

So my constraints are:

choose_project <= 9 /\ budget<=255 s.t solve maximize tot_value;

How can I select a non-arbitrary number (1 to 9) of decision variables among projects raw in the  table?
Until now this is my code: invest.mzn

include "data.dzn";
int: max_budget;  %255
int: max_prj;     %9
enum FEATURE;
enum PRJ;
array[PRJ,FEATURE] of int: data;
constraint ...
...
solve maximize tot_value;

Body:
You can declare an array of Boolean variables, say selected_projects, that encodes whether or not a project PRJ_k is selected or not.

Then you can simply count how many projects in this array are being selected at the same time.

Example:

enum FEATURE = {name, value, budget, personnel};
enum PRJ = { PRJ0, PRJ1 };

array[PRJ,FEATURE] of int: data =
     [| PRJ0,      6000,    35,    5
      | PRJ1,      4000,    34,    3
      |];

array[PRJ] of var bool: selected_projects;

var int: tot_value;

% The total number of selected projects must be in [1, 9]
constraint let {
        var int: tot_selected = sum(prj in PRJ) ( selected_projects[prj] )
    } in
        1 <= tot_selected /\ tot_selected <= 9;

constraint tot_value = sum(prj in PRJ) ( selected_projects[prj] * data[prj, value] );

% ...
% encoding of budget and personnel constraints
% ...

solve maximize tot_value;


问:
How to insert a value into a text field under a div using Watir
I used set  to insert a value in a text field under a div.

Below is the approach that I've used without success. This is the only way I was able to identify the element. When I tried to identify text field by name was not recognized.

```<code>@browser.div(:evaluation, :id => "evaluation_form_attributes").text_field(:id => "evaluation_form_name")
@browser.set('Teacher Evaluation Form')
</code>```

The following error was displayed:

```<code>undefined method `set' for #<Watir::IE:0x4dd9348>
</code>```

This is the HTML:

```<code>div id="evaluation_form_attributes"
</code>```

Evaluation name:

```<code>input id="evaluation_form_name" type="text" size="50" name="evaluation_form[name]" maxlength="30"
</code>```

答:
Try this:</p>

```<code>browser.text_field(:id => "evaluation_form_name").set 'Teacher Evaluation Form'
</code>```

ContentLicense: CC BY-SA 3.0

Body:
I used set  to insert a value in a text field under a div.

Below is the approach that I've used without success. This is the only way I was able to identify the element. When I tried to identify text field by name was not recognized.

@browser.div(:evaluation, :id => "evaluation_form_attributes").text_field(:id => "evaluation_form_name")
@browser.set('Teacher Evaluation Form')

The following error was displayed:

undefined method `set' for #<Watir::IE:0x4dd9348>

This is the HTML:

div id="evaluation_form_attributes"

Evaluation name:

input id="evaluation_form_name" type="text" size="50" name="evaluation_form[name]" maxlength="30"

Body:
Try this:

browser.text_field(:id => "evaluation_form_name").set 'Teacher Evaluation Form'


问:
AKButton and gesture tap conflict
I have a pop up UIView as a control window. Contains sliders and buttons from the AudioKitUI framework.
I have a touch tap gesture set up to close the Pop up view but of course it also closes the UIview when I tap a button. Is it possible to ignore the tap gesture when the AKButton is pressed?

Everything is enclosed in its own class structure.
This is the gesture code at the top of my Overide Init.

```<code>override init(frame: CGRect) {
        super.init(frame: frame)

        self.addGestureRecognizer(UITapGestureRecognizer(target: self, action: #selector(animateout)))
</code>```

and the code for my AKButton.

```<code>   // Dynamics on/off button
        let dynamicsButton = AKButton(title: "") { button in
            if boxDynamicStatus[boxBloqDoubleTapped] == false {
                button.title = "FLOATING"
                boxDynamicStatus[boxBloqDoubleTapped] = true
                boxBloqArray[boxBloqDoubleTapped].physicsBody?.isDynamic = true
            } else {
                boxDynamicStatus[boxBloqDoubleTapped] = false
                button.title = "FIXED"
                boxBloqArray[boxBloqDoubleTapped].physicsBody?.isDynamic = false
            }
        }
        // Set button text from box array
        if boxDynamicStatus[boxBloqDoubleTapped] == false {
            dynamicsButton.title = "FIXED"
        } else {
            dynamicsButton.title = "FLOATING"
        }
</code>```

I don't get this issue with the sliders I suppose because of the moving gesture rather than it being a tap.

All the buttons/sliders are held within a container

```<code>fileprivate let container: UIView = {
        let v = UIView()
        v.translatesAutoresizingMaskIntoConstraints = false
        v.backgroundColor = UIColor.black.withAlphaComponent(0.3)
        v.layer.cornerRadius = 5

        return v

    }()
</code>```

can the gesture be programmed to not register within the container margins?

答:
I think this may be the effectively the same question as here: Prevent click through view in swift

So, it could be solved with the z-index ordering of your views.  If you provided a working version of your code, or some playground version, I could test, but this is the best I can suggest with the information given.

Title: AKButton and gesture tap conflict

Body:
I have a pop up UIView as a control window. Contains sliders and buttons from the AudioKitUI framework.
I have a touch tap gesture set up to close the Pop up view but of course it also closes the UIview when I tap a button. Is it possible to ignore the tap gesture when the AKButton is pressed?

Everything is enclosed in its own class structure.
This is the gesture code at the top of my Overide Init.

override init(frame: CGRect) {
        super.init(frame: frame)

        self.addGestureRecognizer(UITapGestureRecognizer(target: self, action: #selector(animateout)))

and the code for my AKButton.

   // Dynamics on/off button
        let dynamicsButton = AKButton(title: "") { button in
            if boxDynamicStatus[boxBloqDoubleTapped] == false {
                button.title = "FLOATING"
                boxDynamicStatus[boxBloqDoubleTapped] = true
                boxBloqArray[boxBloqDoubleTapped].physicsBody?.isDynamic = true
            } else {
                boxDynamicStatus[boxBloqDoubleTapped] = false
                button.title = "FIXED"
                boxBloqArray[boxBloqDoubleTapped].physicsBody?.isDynamic = false
            }
        }
        // Set button text from box array
        if boxDynamicStatus[boxBloqDoubleTapped] == false {
            dynamicsButton.title = "FIXED"
        } else {
            dynamicsButton.title = "FLOATING"
        }

I don't get this issue with the sliders I suppose because of the moving gesture rather than it being a tap.

All the buttons/sliders are held within a container

fileprivate let container: UIView = {
        let v = UIView()
        v.translatesAutoresizingMaskIntoConstraints = false
        v.backgroundColor = UIColor.black.withAlphaComponent(0.3)
        v.layer.cornerRadius = 5

        return v

    }()

can the gesture be programmed to not register within the container margins?


问:
Using ASCII for board co-ordinates in Java
I'm pretty new to Java and learning it as a hobby mainly after school to kill my free time productively. I find it really interesting and picking it up relatively pain free however I'm a bit stuck trying to implement a basic chess program that can be played via the command line. Ideally, I'd like to print out the board initially with just Kings and Queens on both sides and get them moving forwards, backwards and diagonally. (Once I get the hang of this I would try adding all other pieces, but at first I would like to just start as simply as possible). For simplicity I will just be using the standard 8x8 playing board.

I've already created a main game loop which uses a command line input of 1 to switch players and 2 to exit the game but I am stuck when it comes to printing out positions and having them change during the game. Firstly I would like to print out the starting positions of both kings and queens as a string (e.g ["D1","E1","D8","E8"]) for the player to see. I imagine the best way to do this is by using the ASCII index but I'm not really sure where to go from this, I tried the code below I know it's not correct and I don't know what to change...

```<code>    int num[] = {65, 66, 67, 68, 69, 70, 71, 72};
    String start_Pos =null;
    for(int i = 4; i < 6; i++){
        start_Pos[i] = (Character.toString((char)i) + "1");
    }

    int num[] = {65, 66, 67, 68, 69, 70, 71, 72};
    String start_Pos =null;
    for(int i = 4; i < 6; i++){
        start_Pos1[i] = (Character.toString((char)i) + "8");
    }
        System.out.println(start_Pos + start_Pos1);
</code>```

I have also tried coding the board set-up but this is literally just printing out the starting positions of the pieces and therefore will not change when a player makes a move - ideally it should. An example would be the starting positions are shown on the board like so:

Photo (PS I know QK should be swapped on one side so they're not opposite each other, my bad!

But after an input of "D1 D3" (first coordinates indicating what piece and the second indicating final position)from player 1 the board changes to reflect this. Is this possible without having to recompile the entire code after each turn? (Maybe a stupid question...).

Any help would be greatly appreciated. I find learning through making small games like these a lot more interesting and rewarding so if anyone is able to help me implement this I would be very thankful.

答:
Java is an Object Oriented language, so it is best to use classes and object instances.

Of course, with a chessboard, you'll still want to perform calculations on the x, y coordinates. Computers are just better with numbers, they have problems interpreting things. The chess notation is mainly of use to us humans.

So here is a class that can be used to parse and interpret chess positions. Note that you should keep this class immutable and simply use a new object instance rather than changing the x or y field.

```<code>public final class ChessPosition {
    // use constants so you understand what the 8 is in your code
    private static final int BOARD_SIZE = 8;

    // zero based indices, to be used in a 2D array
    private final int x;
    private final int y;

    public ChessPosition(int x, int y) {
        // guards checks, so that the object doesn't enter an invalid state
        if (x < 0 || x >= BOARD_SIZE || y < 0 || y >= BOARD_SIZE) {
            throw new IllegalArgumentException("Invalid position");
        }

        this.x = x;
        this.y = y;
    }

    public ChessPosition(String chessNotation) {
        // accept upper and lowercase, but output only uppercase
        String chessNotationUpper = chessNotation.toUpperCase();
        // use a regular expression for this guard
        if (!chessNotationUpper.matches("[A-H][0-7]")) {
            throw new IllegalArgumentException("Invalid position");
        }

        // can be done in one statement, but we're not in a hurry
        char xc = chessNotationUpper.charAt(0);
        // chars are also numbers, so we can just use subtraction with another char
        x = xc - 'A';

        char yc = chessNotation.charAt(1);
        // note that before calculation, they are converted to int by Java
        y = yc - '1';
    }

    public int getX() {
        return x;
    }

    public int getY() {
        return y;
    }

    public String getChessNotation() {
        // perform the reverse and create a string out of it
        // using StringBuilder#append(char c) is another option, but this is easier
        return new String(new char[] {
            // Java converts 'A' to int first (value 0x00000041)
            (char) (x + 'A'),
            (char) (y + '1')
        });
    }

    // this will be helpfully displayed in your debugger, so implement toString()!
    @Override
    public String toString() {
        return getChessNotation();
    }
}
</code>```

Now probably you want to also create a Board class with a backing 2D array of ChessPiece[][] and perform board.set(WHITE_KING, new ChessPosition("E1")) or something similar.

Body:
I'm pretty new to Java and learning it as a hobby mainly after school to kill my free time productively. I find it really interesting and picking it up relatively pain free however I'm a bit stuck trying to implement a basic chess program that can be played via the command line. Ideally, I'd like to print out the board initially with just Kings and Queens on both sides and get them moving forwards, backwards and diagonally. (Once I get the hang of this I would try adding all other pieces, but at first I would like to just start as simply as possible). For simplicity I will just be using the standard 8x8 playing board.

I've already created a main game loop which uses a command line input of 1 to switch players and 2 to exit the game but I am stuck when it comes to printing out positions and having them change during the game. Firstly I would like to print out the starting positions of both kings and queens as a string (e.g ["D1","E1","D8","E8"]) for the player to see. I imagine the best way to do this is by using the ASCII index but I'm not really sure where to go from this, I tried the code below I know it's not correct and I don't know what to change...

    int num[] = {65, 66, 67, 68, 69, 70, 71, 72};
    String start_Pos =null;
    for(int i = 4; i < 6; i++){
        start_Pos[i] = (Character.toString((char)i) + "1");
    }

    int num[] = {65, 66, 67, 68, 69, 70, 71, 72};
    String start_Pos =null;
    for(int i = 4; i < 6; i++){
        start_Pos1[i] = (Character.toString((char)i) + "8");
    }
        System.out.println(start_Pos + start_Pos1);

I have also tried coding the board set-up but this is literally just printing out the starting positions of the pieces and therefore will not change when a player makes a move - ideally it should. An example would be the starting positions are shown on the board like so:

Photo (PS I know QK should be swapped on one side so they're not opposite each other, my bad!

But after an input of "D1 D3" (first coordinates indicating what piece and the second indicating final position)from player 1 the board changes to reflect this. Is this possible without having to recompile the entire code after each turn? (Maybe a stupid question...).

Any help would be greatly appreciated. I find learning through making small games like these a lot more interesting and rewarding so if anyone is able to help me implement this I would be very thankful.

Body:
Java is an Object Oriented language, so it is best to use classes and object instances.

Of course, with a chessboard, you'll still want to perform calculations on the x, y coordinates. Computers are just better with numbers, they have problems interpreting things. The chess notation is mainly of use to us humans.

So here is a class that can be used to parse and interpret chess positions. Note that you should keep this class immutable and simply use a new object instance rather than changing the x or y field.

public final class ChessPosition {
    // use constants so you understand what the 8 is in your code
    private static final int BOARD_SIZE = 8;

    // zero based indices, to be used in a 2D array
    private final int x;
    private final int y;

    public ChessPosition(int x, int y) {
        // guards checks, so that the object doesn't enter an invalid state
        if (x < 0 || x >= BOARD_SIZE || y < 0 || y >= BOARD_SIZE) {
            throw new IllegalArgumentException("Invalid position");
        }

        this.x = x;
        this.y = y;
    }

    public ChessPosition(String chessNotation) {
        // accept upper and lowercase, but output only uppercase
        String chessNotationUpper = chessNotation.toUpperCase();
        // use a regular expression for this guard
        if (!chessNotationUpper.matches("[A-H][0-7]")) {
            throw new IllegalArgumentException("Invalid position");
        }

        // can be done in one statement, but we're not in a hurry
        char xc = chessNotationUpper.charAt(0);
        // chars are also numbers, so we can just use subtraction with another char
        x = xc - 'A';

        char yc = chessNotation.charAt(1);
        // note that before calculation, they are converted to int by Java
        y = yc - '1';
    }

    public int getX() {
        return x;
    }

    public int getY() {
        return y;
    }

    public String getChessNotation() {
        // perform the reverse and create a string out of it
        // using StringBuilder#append(char c) is another option, but this is easier
        return new String(new char[] {
            // Java converts 'A' to int first (value 0x00000041)
            (char) (x + 'A'),
            (char) (y + '1')
        });
    }

    // this will be helpfully displayed in your debugger, so implement toString()!
    @Override
    public String toString() {
        return getChessNotation();
    }
}

Now probably you want to also create a Board class with a backing 2D array of ChessPiece[][] and perform board.set(WHITE_KING, new ChessPosition("E1")) or something similar.


问:
How to resolve target redefinition problem in diamond subproject dependency?
There are two libs libA and libB depending on libBase and including it into build process with

add_subdirectory(../libBase build/libBase)

Everything works when they are build separately. But when project prj wants to include everything into its build process:

```<code>add_subdirectory(../libA build/libA)
add_subdirectory(../libB build/libB)
</code>```

Then cmake complaints about target redefinition:

```<code>add_library cannot create target "libBase" because another target with the same name already exists.
</code>```

How to make such project (building all dependences in one step) correct? Is it achievable with add_subdirectory or sth else should be used to add dependencies?

答: Found it: include_guard() since cmake 3.10</p>

Body:
There are two libs libA and libB depending on libBase and including it into build process with

add_subdirectory(../libBase build/libBase)

Everything works when they are build separately. But when project prj wants to include everything into its build process:

add_subdirectory(../libA build/libA)
add_subdirectory(../libB build/libB)

Then cmake complaints about target redefinition:

add_library cannot create target "libBase" because another target with the same name already exists.

How to make such project (building all dependences in one step) correct? Is it achievable with add_subdirectory or sth else should be used to add dependencies?


问:
maven-bundle-plugin fails with "Invalid class file module-info.class"
After adding a dependency that contains a JPMS module, the maven-bundle-plugin (version 3.3.0) fails with:

```<code>[INFO] --- maven-bundle-plugin:3.3.0:bundle (default-bundle) @ my-bundle ---
[ERROR] Bundle myGroup:my-bundle:bundle:1.0 :
        Exception: java.lang.ArrayIndexOutOfBoundsException: 19
[ERROR] Bundle myGroup:my-bundle:bundle:1.0 :
        Invalid class file module-info.class (java.lang.ArrayIndexOutOfBoundsException: 19)
</code>```

It seems that the plugin tries (and fails) to analyze the module-info.class, which should be irrelevant in an OSGi context.

答: I've been having the same issue. After a lot of playing, I found out that it was the uber-jar that was the root cause. YMMV.</p>

Body:
After adding a dependency that contains a JPMS module, the maven-bundle-plugin (version 3.3.0) fails with:

[INFO] --- maven-bundle-plugin:3.3.0:bundle (default-bundle) @ my-bundle ---
[ERROR] Bundle myGroup:my-bundle:bundle:1.0 :
        Exception: java.lang.ArrayIndexOutOfBoundsException: 19
[ERROR] Bundle myGroup:my-bundle:bundle:1.0 :
        Invalid class file module-info.class (java.lang.ArrayIndexOutOfBoundsException: 19)

It seems that the plugin tries (and fails) to analyze the module-info.class, which should be irrelevant in an OSGi context.


问:
Why do the bullet and bell characters sometimes do not cause a beep sound when they are sent to stdout?
I found out that when you send the bullet and bell characters to stdout the speakers emit a beep sound. I tried it on a computer running x86-64 Windows. It worked. Then I tried it on another computer with the same specs at the first and no beep. I think it must be a hardware issue because both computers had the same software running on them. I have not experimented with additional computers or OSs. I did observe, when testing it on the second computer, that there was a delay when the the bell or bullet characters were sent to stdout. So the only missing feature is the sound itself.
What is causing this discrepancy?</p>

答:
It appears that the bell character needs to be sent to STDOUT (the Perl file handle for standard out) to hear the Beep.
For example if you run a Perl script named "test.pl" having
print "\a BEEP\r\n";
using the command
Perl test.pl
it should sound the beep.

However , if you re-direct STDOUT to a file with the command
perl test.pl > testPerl.txt
NO sound will be heard.
This was observed on PC/Windows 10 and Mac/OS X using a bash terminal.


问:
How dask array chunking works for three or more dimension?
Consider the following code snippet where I am trying to create a dask array with asymmetric but repeated blocks of size (3, 4, 5):

```<code>import numpy as np
import dask.array as da
a = np.random.randint(0, 9, (3, 12, 10))
d = da.from_array(a, chunks=(3, 4, 5))
</code>```

The above snippet does not throw any error / warning. But When I try to do the following:

```<code>r = d.map_blocks(np.sum)
out = r.compute()

</code>```

It throws the error below:

```<code>python3.7/site-packages/dask/array/core.py in <listcomp>(.0)
   4099
   4100     while isinstance(arrays, (list, tuple)):
-> 4101         result.append(tuple([shape(deepfirst(a))[dim] for a in arrays]))
   4102         arrays = arrays[0]
   4103         dim += 1

IndexError: tuple index out of range
</code>```

What am I doing wrong?

答:
If you check map_block docs here. Under chunks parameter description, it said:

"If not provided the resulting array is assumed to have the same block structure as the first input array."

So your sum function should be something like this:

```<code>def compute_block_sum(block):
    return np.array([[np.sum(block)]])[:,None]
</code>```

Then you do

```<code>d.map_blocks(compute_block_sum).compute()
</code>```

Body:
Consider the following code snippet where I am trying to create a dask array with asymmetric but repeated blocks of size (3, 4, 5):

import numpy as np
import dask.array as da
a = np.random.randint(0, 9, (3, 12, 10))
d = da.from_array(a, chunks=(3, 4, 5))

The above snippet does not throw any error / warning. But When I try to do the following:

r = d.map_blocks(np.sum)
out = r.compute()

It throws the error below:

python3.7/site-packages/dask/array/core.py in <listcomp>(.0)
   4099
   4100     while isinstance(arrays, (list, tuple)):
-> 4101         result.append(tuple([shape(deepfirst(a))[dim] for a in arrays]))
   4102         arrays = arrays[0]
   4103         dim += 1

IndexError: tuple index out of range

What am I doing wrong?

Body:
If you check map_block docs here. Under chunks parameter description, it said:

"If not provided the resulting array is assumed to have the same block structure as the first input array."

So your sum function should be something like this:

def compute_block_sum(block):
    return np.array([[np.sum(block)]])[:,None]

Then you do

d.map_blocks(compute_block_sum).compute()


问:
Hide and show a div using input value using JavaScript
Someone help me with the below code I want to hide and show a div using input value using JavaScript.

        <form name="myform">

<input type="text" id="ref3" value="">

                Show me

        </form>

    ```

答:
Few notes: you didn't name your handler function, you can just pick any name but remember it so that you can use it once you bind the event to you DOM element

Always move the script tag down because javascript will run before the HTML render and it will not attach or catch DOM element.

I just included the same HTML format so that you know how to run it locally, but other answers provided a better format.

        <form name="myform">

<input type="text" id="ref3" onkeyup="inputChanged()" value="2" defaultvalue="2">

                Show me

        </form>

    ```

Body:
Few notes: you didn't name your handler function, you can just pick any name but remember it so that you can use it once you bind the event to you DOM element

Always move the script tag down because javascript will run before the HTML render and it will not attach or catch DOM element.

I just included the same HTML format so that you know how to run it locally, but other answers provided a better format.

<html>
    <head>

    </head>

    <body>
        <form name="myform">
            <div><input type="text" id="ref3" onkeyup="inputChanged()" value="2" defaultvalue="2"></div>
            <div id="hotapp3" style="visibility:hidden">
                <label>Show me</label>
            </div>
        </form>
        <script>
    function inputChanged(event){
      let value = document.getElementById('ref3').value;
        if (value == 2) {
          document.getElementById('hotapp3').style.visibility = 'visible';
        } else {
          document.getElementById('hotapp3').style.visibility = 'hidden';
        }
    }
    inputChanged();
    </script>
    </body>

    </html>


问:
Can't allocate 2-4 Gb of RAM with new[]/ C++/Linux/ x86_64
For this easy test, and the linux box with 4Gb or RAM, 0byte of swap and CPU in x86_64 mode, I can't allocate more than 1 Gb of array.

Source:

```<code>#include <cstdio>
int main()
{
 for(int i=0;i<33;i++) {
  char*a=new char[1<<i];
  *a=1;
  delete[]a;
  printf("%d\n",i);
  fflush(stdout);
 }
}
</code>```

Run:

```<code>$ file test
test: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV)
$ ./test
...
24
25
26
27
28
29
30
terminate called after throwing an instance of 'std::bad_alloc'
  what():  St9bad_alloc
Aborted
</code>```

There is no ulimit for memory:

```<code>virtual memory          (kbytes, -v) unlimited
data seg size           (kbytes, -d) unlimited
</code>```

Why the error?

Glibc is 2.3.4, kernel is 2.6.9

UPDATE: Compiler is gcc4.1

Thanks! The test definitely has a error, 1ull<<i gives me up to 31 (2gb). This error was unintended. But the real failed code is

```<code> for(j=0;j<2;j++)
  for(i=0;i<25;i++)
   some_array[j][i] = new int[1<<24];
</code>```

so there is no sign overflow in the real code.

Size of int is 4 byte:

```<code>$ echo 'main(){return sizeof(int);}'| gcc -x c - && ./a.out; echo $?
4
</code>```

the every request will be for 1<<24 * 4 = 1<<26; total memory required is 2*25*(1<<26) 3355443200 bytes + 50*sizeof(pointer) for some_array + 50*(size of new[] overhead).

答:
A naked constant in C is an int. A signed int.
So 1 << 31 is -2147483648.
because
1<<31 = 0x10000000 = -2147483648

Try (size_t)1 << i

Body:
For this easy test, and the linux box with 4Gb or RAM, 0byte of swap and CPU in x86_64 mode, I can't allocate more than 1 Gb of array.

Source:

#include <cstdio>
int main()
{
 for(int i=0;i<33;i++) {
  char*a=new char[1<<i];
  *a=1;
  delete[]a;
  printf("%d\n",i);
  fflush(stdout);
 }
}

Run:

$ file test
test: ELF 64-bit LSB executable, AMD x86-64, version 1 (SYSV)
$ ./test
...
24
25
26
27
28
29
30
terminate called after throwing an instance of 'std::bad_alloc'
  what():  St9bad_alloc
Aborted

There is no ulimit for memory:

virtual memory          (kbytes, -v) unlimited
data seg size           (kbytes, -d) unlimited

Why the error?

Glibc is 2.3.4, kernel is 2.6.9

UPDATE: Compiler is gcc4.1

Thanks! The test definitely has a error, 1ull<<i gives me up to 31 (2gb). This error was unintended. But the real failed code is

 for(j=0;j<2;j++)
  for(i=0;i<25;i++)
   some_array[j][i] = new int[1<<24];

so there is no sign overflow in the real code.

Size of int is 4 byte:

$ echo 'main(){return sizeof(int);}'| gcc -x c - && ./a.out; echo $?
4

the every request will be for 1<<24 * 4 = 1<<26; total memory required is 2*25*(1<<26) 3355443200 bytes + 50*sizeof(pointer) for some_array + 50*(size of new[] overhead).


答:
You are not doing anything wrong. pandas.Timedelta.days returns full days in the timedelta object. You can find fractional days with using diff.value (which returns nanoseconds), such as:

```<code>import pandas as pd

date1 = pd.to_datetime("2020-02-12T16:02:51Z", utc=True)
date2 = pd.to_datetime("2017-03-08T18:16:02-05:00", utc=True)
diff = date1 - date2

# nano seconds to days
diff.value / 8.64e+13
>>> 1070.6991782407408
</code>```

Title: Pandas get date difference between 2 ISO date

Body:
You are not doing anything wrong. pandas.Timedelta.days returns full days in the timedelta object. You can find fractional days with using diff.value (which returns nanoseconds), such as:

import pandas as pd

date1 = pd.to_datetime("2020-02-12T16:02:51Z", utc=True)
date2 = pd.to_datetime("2017-03-08T18:16:02-05:00", utc=True)
diff = date1 - date2

# nano seconds to days
diff.value / 8.64e+13
>>> 1070.6991782407408


问:
Merge multiple tables having different columns
I have 4 tables each tables has different number of columns as listed below.

```<code>tableA - 34
tableB - 47
tableC - 26
tableD - 16
</code>```

Every table has a common column called id, now i need to perform a union the problem is since the columns are not equal length and entirely different i can't do a union.

Based on id only i can get the details from every table, so how should i approach this.

What is the optimized way to solve this, tried full join but that takes too much time.

Tried so far

```<code>SELECT * FROM tableA FULL JOIN
              tableB FULL JOIN
              tableC FULL JOIN
              tableD
         USING (id)
         WHERE tableA.id = 123 OR
               tableB.id = 123 OR
               tableC.id = 123 OR
               tableD.id = 123
</code>```

答:
Snowflake does have a declared limitation in use of Set operators (such as UNION):

When using these operators:

Make sure that each query selects the same number of columns.

[...]

However, since the column names are well known, it is possible to come up with a superset of all unique column names required in the final result and project them explicitly from each query.

There's not enough information in the question on how many columns overlap (47 unique columns?), or if they are all different (46 + 33 + 25 + 15 = 119 unique columns?). The answer to this would determine the amount of effort required to write out each query, as it would involve adapting a query from the following form:

SELECT * FROM t1

Into an explicit form with dummy columns defined with acceptable defaults that match the data type on tables where they are present:

SELECT
  present_col1,
  NULL AS absent_col2,
  0.0 AS absent_col3,
  present_col4,
  [...]
FROM
  t1
```

You can also use some meta programming with stored procedures to "generate" such an altered query by inspecting independent result's column names using the Statement::getColumnCount(), Statement::getColumnName(), etc. APIs and forming a superset union version with default/empty values.

Body:
I have 4 tables each tables has different number of columns as listed below.

tableA - 34
tableB - 47
tableC - 26
tableD - 16

Every table has a common column called id, now i need to perform a union the problem is since the columns are not equal length and entirely different i can't do a union.

Based on id only i can get the details from every table, so how should i approach this.

What is the optimized way to solve this, tried full join but that takes too much time.

Tried so far

SELECT * FROM tableA FULL JOIN
              tableB FULL JOIN
              tableC FULL JOIN
              tableD
         USING (id)
         WHERE tableA.id = 123 OR
               tableB.id = 123 OR
               tableC.id = 123 OR
               tableD.id = 123

Body:
Snowflake does have a declared limitation in use of Set operators (such as UNION):

When using these operators:

Make sure that each query selects the same number of columns.

[...]

However, since the column names are well known, it is possible to come up with a superset of all unique column names required in the final result and project them explicitly from each query.

There's not enough information in the question on how many columns overlap (47 unique columns?), or if they are all different (46 + 33 + 25 + 15 = 119 unique columns?). The answer to this would determine the amount of effort required to write out each query, as it would involve adapting a query from the following form:

SELECT * FROM t1

Into an explicit form with dummy columns defined with acceptable defaults that match the data type on tables where they are present:

SELECT
  present_col1,
  NULL AS absent_col2,
  0.0 AS absent_col3,
  present_col4,
  [...]
FROM
  t1

You can also use some meta programming with stored procedures to "generate" such an altered query by inspecting independent result's column names using the Statement::getColumnCount(), Statement::getColumnName(), etc. APIs and forming a superset union version with default/empty values.


问:
Replace extra spaces in string with a single space in Java, without regex
I want to basically to do this:

```<code>String secondString = firstString.trim().replaceAll(" +", " ");
</code>```

Or replace all the multiple spaces in my string, with just a single space. However, is it possibly to do this without regex like I am doing here?

Thanks!

答:
However, is it possibly to do this without regex like I am doing here?

Yes. The regular expression is clear and easy to read, so I would probably use that; but you could use a nested loop, a StringBuilder and Character.isWhitespace(char) like

```<code>StringBuilder sb = new StringBuilder();
for (int i = 0; i < firstString.length(); i++) {
    sb.append(firstString.charAt(i));
    if (Character.isWhitespace(firstString.charAt(i))) {
        // The current character is white space, skip characters until
        // the next character is not.
        while (i + 1 < firstString.length()
                    && Character.isWhitespace(firstString.charAt(i + 1))) {
            i++;
        }
    }
}
String secondString = sb.toString();
</code>```

Note: This is a rare example of a nested loop that is O(N); the inner loop modifies the same counter as the outer loop.

Body:
I want to basically to do this:

String secondString = firstString.trim().replaceAll(" +", " ");

Or replace all the multiple spaces in my string, with just a single space. However, is it possibly to do this without regex like I am doing here?

Thanks!

Body:
However, is it possibly to do this without regex like I am doing here?

Yes. The regular expression is clear and easy to read, so I would probably use that; but you could use a nested loop, a StringBuilder and Character.isWhitespace(char) like

StringBuilder sb = new StringBuilder();
for (int i = 0; i < firstString.length(); i++) {
    sb.append(firstString.charAt(i));
    if (Character.isWhitespace(firstString.charAt(i))) {
        // The current character is white space, skip characters until
        // the next character is not.
        while (i + 1 < firstString.length()
                    && Character.isWhitespace(firstString.charAt(i + 1))) {
            i++;
        }
    }
}
String secondString = sb.toString();

Note: This is a rare example of a nested loop that is O(N); the inner loop modifies the same counter as the outer loop.


问:
Block someone access from my software if he is using 32bit PC
How does softwares block access when they identify a 32bit PC, and how can I perform this with Python ?</p>

答: I haven't used it, but the cpuinfo module looks like it should work. Maybe try installing that module from pip and then test cpuinfo.get_cpu_info()["bits"] == 32


问:
Improve Efficiency of Pandas Transformation Process
Right now I've got a process to extract a large amount of data (~1.5m rows) from a URL that comes in an unorganized way that I need to subsequently regorganize. The current process works flawlessly but it is extremely memory heavy and inefficient so I was looking for help.

The data that I receive comes in the following structure: (notice after exit 8 there are another 5 columns of Na's and None signifying end of current SCP

```<code>['C/A','UNIT','SCP','DATE1','TIME1','DESC1','ENTRIES1','EXITS1','DATE2','TIME2','ESC2',\
 'ENTRIES2','EXITS2','DATE3','TIME3','DESC3','ENTRIES3','EXITS3','DATE4','TIME4','DESC4',\
 'ENTRIES4','EXITS4','DATE5','TIME5','DESC5','ENTRIES5','EXITS5','DATE6','TIME6','DESC6',\
 'ENTRIES6','EXITS6','DATE7','TIME7','DESC7','ENTRIES7','EXITS7','DATE8','TIME8','DESC8',\
 'ENTRIES8','EXITS8']
</code>```

My goal is to have it reorganized like such:

```<code>['c/a','unit','scp','date','time','description','entries','exit']
</code>```

Example of Raw Output:

```<code>     C/A        UNIT    SCP     DATE1   TIME1   DESC1   ENTRIES1    EXITS1  DATE2   TIME2   ESC2    ENTRIES2    EXITS2  DATE3   TIME3   DESC3   ENTRIES3    EXITS3  DATE4   TIME4   DESC4   ENTRIES4    EXITS4  DATE5   TIME5   DESC5   ENTRIES5    EXITS5  DATE6   TIME6   DESC6   ENTRIES6    EXITS6  DATE7   TIME7   DESC7   ENTRIES7    EXITS7  DATE8   TIME8   DESC8   ENTRIES8    EXITS8
0   A002    R051    02-00-00    04-20-13    00:00:00    REGULAR 4084276 1405308 04-20-13    04:00:00    REGULAR 4084308.0   1405312.0   04-20-13    08:00:00    REGULAR 4084332.0   1405348.0   04-20-13    12:00:00    REGULAR 4084429.0   1405441.0   04-20-13    16:00:00    REGULAR 4084714.0   1405494.0   04-20-13    20:00:00    REGULAR 4085107.0   1405550.0   04-21-13    00:00:00    REGULAR 4085286.0   1405578.0   04-21-13    04:00:00    REGULAR 4085317.0   1405582.0
1   A002    R051    02-00-00    04-21-13    08:00:00    REGULAR 4085336 1405603 04-21-13    12:00:00    REGULAR 4085421.0   1405673.0   04-21-13    16:00:00    REGULAR 4085543.0   1405725.0   04-21-13    20:00:00    REGULAR 4085543.0   1405781.0   04-22-13    00:00:00    REGULAR 4085669.0   1405820.0   04-22-13    04:00:00    REGULAR 4085684.0   1405825.0   04-22-13    08:00:00    REGULAR 4085715.0   1405929.0   04-22-13    12:00:00    REGULAR 4085878.0   1406175.0
2   A002    R051    02-00-00    04-22-13    16:00:00    REGULAR 4086116 1406242 04-22-13    20:00:00    REGULAR 4086986.0   1406310.0   04-23-13    00:00:00    REGULAR 4087164.0   1406335.0   04-23-13    04:00:00    REGULAR 4087172.0   1406339.0   04-23-13    08:00:00    REGULAR 4087214.0   1406441.0   04-23-13    12:00:00    REGULAR 4087390.0   1406685.0   04-23-13    16:00:00    REGULAR 4087738.0   1406741.0   04-23-13    20:00:00    REGULAR 4088682.0   1406813.0
3   A002    R051    02-00-00    04-24-13    00:00:00    REGULAR 4088879 1406839 04-24-13    04:00:00    REGULAR 4088890.0   1406845.0   04-24-13    08:00:00    REGULAR 4088934.0   1406951.0   04-24-13    12:00:00    REGULAR 4089105.0   1407209.0   04-24-13    16:00:00    REGULAR 4089378.0   1407269.0   04-24-13    20:00:00    REGULAR 4090319.0   1407336.0   04-25-13    00:00:00    REGULAR 4090535.0   1407365.0   04-25-13    04:00:00    REGULAR 4090550.0   1407370.0
4   A002    R051    02-00-00    04-25-13    08:00:00    REGULAR 4090589 1407469 04-25-13    08:57:03    DOOR OPEN   4090629.0   1407591.0   04-25-13    08:58:01    LOGON   4090629.0   1407591.0   04-25-13    09:01:08    LGF-MAN 4090629.0   1407591.0   04-25-13    09:01:53    LOGON   4090629.0   1407591.0   04-25-13    09:02:02    DOOR CLOSE  4090629.0   1407591.0   04-25-13    09:02:04    DOOR OPEN   4090629.0   1407591.0   04-25-13    09:02:31    DOOR CLOSE  4090629.0   1407591.0
5   A002    R051    02-00-00    04-25-13    09:02:32    DOOR OPEN   4090629 1407591 04-25-13    09:07:21    LOGON   4090629.0   1407591.0   04-25-13    09:12:12    LGF-MAN 4090642.0   1407592.0   04-25-13    09:12:20    DOOR CLOSE  4090642.0   1407592.0   04-25-13    12:00:00    REGULAR 4090743.0   1407723.0   04-25-13    16:00:00    REGULAR 4091064.0   1407793.0   04-25-13    20:00:00    REGULAR 4092044.0   1407840.0   04-26-13    00:00:00    REGULAR 4092314.0   1407859.0
6   A002    R051    02-00-00    04-26-13    04:00:00    REGULAR 4092325 1407861 04-26-13    08:00:00    REGULAR 4092363.0   1407958.0   04-26-13    12:00:00    REGULAR 4092541.0   1408225.0   04-26-13    16:00:00    REGULAR 4092837.0   1408285.0   04-26-13    20:00:00    REGULAR 4093823.0   1408341.0   None    None    None    NaN NaN None    None    None    NaN NaN None    None    None    NaN NaN
</code>```

My current inefficient function looks like this:

```<code>def cleanData(dataFrame):

    tempDf = dataFrame

    tempColName = ['date','time','description','entries','exit','c/a','unit', 'scp']
    finalColName = ['c/a','unit','scp','date','time','description','entries','exit']

    tempDf1 = tempDf.iloc[:,:8]
    tempDf1.dropna(inplace=True)
    tempDf1.columns = finalColName

    tempDf2 = tempDf.iloc[:,8:13]
    tempDf2['c/a'] = tempDf['C/A']
    tempDf2['unit'] = tempDf['UNIT']
    tempDf2['scp'] = tempDf['SCP']
    tempDf2.dropna(inplace=True)
    tempDf2.columns = tempColName

    tempDf3 = tempDf.iloc[:,13:18]
    tempDf3['c/a'] = tempDf['C/A']
    tempDf3['unit'] = tempDf['UNIT']
    tempDf3['scp'] = tempDf['SCP']
    tempDf3.dropna(inplace=True)
    tempDf3.columns = tempColName

    tempDf4 = tempDf.iloc[:,18:23]
    tempDf4['c/a'] = tempDf['C/A']
    tempDf4['unit'] = tempDf['UNIT']
    tempDf4['scp'] = tempDf['SCP']
    tempDf4.dropna(inplace=True)
    tempDf4.columns = tempColName

    tempDf5 = tempDf.iloc[:,23:28]
    tempDf5['c/a'] = tempDf['C/A']
    tempDf5['unit'] = tempDf['UNIT']
    tempDf5['scp'] = tempDf['SCP']
    tempDf5.dropna(inplace=True)
    tempDf5.columns = tempColName

    tempDf6 = tempDf.iloc[:,28:33]
    tempDf6['c/a'] = tempDf['C/A']
    tempDf6['unit'] = tempDf['UNIT']
    tempDf6['scp'] = tempDf['SCP']
    tempDf6.dropna(inplace=True)
    tempDf6.columns = tempColName

    tempDf7 = tempDf.iloc[:,33:38]
    tempDf7['c/a'] = tempDf['C/A']
    tempDf7['unit'] = tempDf['UNIT']
    tempDf7['scp'] = tempDf['SCP']
    tempDf7.dropna(inplace=True)
    tempDf7.columns = tempColName

    tempDf8 = tempDf.iloc[:,38:43]
    tempDf8['c/a'] = tempDf['C/A']
    tempDf8['unit'] = tempDf['UNIT']
    tempDf8['scp'] = tempDf['SCP']
    tempDf8.dropna(inplace=True)
    tempDf8.columns = tempColName

    placeHolderDf = pd.concat([tempDf2,tempDf3,tempDf4,tempDf5,tempDf6,tempDf7,tempDf8])
    placeHolderDf = placeHolderDf[['c/a','unit','scp','date','time','description','entries','exit']]
    fullData = pd.concat([tempDf1,placeHolderDf])
    fullData['date'] = pd.to_datetime(fullData['date'])

    return fullData.reset_index(drop=True)
</code>```

with a correct final output like:

```<code>    c/a     unit    scp        date         time    description entries exit
0   A002    R051    02-00-00    2013-04-20  00:00:00    REGULAR 4084276 1405308
1   A002    R051    02-00-00    2013-04-21  08:00:00    REGULAR 4085336 1405603
2   A002    R051    02-00-00    2013-04-22  16:00:00    REGULAR 4086116 1406242
3   A002    R051    02-00-00    2013-04-24  00:00:00    REGULAR 4088879 1406839
4   A002    R051    02-00-00    2013-04-25  08:00:00    REGULAR 4090589 1407469
</code>```

Any help is immensely appreaciated.

答:
Hope this suit your requirements

pd.wide_to_long will automatically group the column name.

```<code>list_melt =  ['DATE','TIME','DESC','ENTRIES','EXITS']
list_id = ['C/A','UNIT','SCP']
# pd.wide_to_long id has to be unique
df['count'] = df.groupby(list_id)[list_id].cumcount()
df_melt = pd.wide_to_long(df, stubnames = list_melt, i = ['C/A','UNIT','SCP','count'], j = 'value').reset_index()
df_melt.drop(columns = ['count', 'value'], axis = 1, inplace = True)
df_melt.columns = ['c/a','unit','scp','date','time','description','entries','exit']
</code>```

This is a proxy output:

```<code>list_cols = ['C/A','UNIT','SCP','DATE1','TIME1','DESC1','ENTRIES1','EXITS1','DATE2','TIME2','DESC2',\
 'ENTRIES2','EXITS2','DATE3','TIME3','DESC3','ENTRIES3','EXITS3']

df = pd.DataFrame([[1] * 8 + [1,2,3,4,5] + [6,7,8,9,10]], columns = list_cols)

df_melt
  c/a   unit    scp date    time description entries exit
0   1     1       1    1    1             1    1      1
1   1     1       1    1    2             3    4      5
2   1     1       1    6    7             8    9      10
</code>```

Body:
Right now I've got a process to extract a large amount of data (~1.5m rows) from a URL that comes in an unorganized way that I need to subsequently regorganize. The current process works flawlessly but it is extremely memory heavy and inefficient so I was looking for help.

The data that I receive comes in the following structure: (notice after exit 8 there are another 5 columns of Na's and None signifying end of current SCP

['C/A','UNIT','SCP','DATE1','TIME1','DESC1','ENTRIES1','EXITS1','DATE2','TIME2','ESC2',\
 'ENTRIES2','EXITS2','DATE3','TIME3','DESC3','ENTRIES3','EXITS3','DATE4','TIME4','DESC4',\
 'ENTRIES4','EXITS4','DATE5','TIME5','DESC5','ENTRIES5','EXITS5','DATE6','TIME6','DESC6',\
 'ENTRIES6','EXITS6','DATE7','TIME7','DESC7','ENTRIES7','EXITS7','DATE8','TIME8','DESC8',\
 'ENTRIES8','EXITS8']

My goal is to have it reorganized like such:

['c/a','unit','scp','date','time','description','entries','exit']

Example of Raw Output:

     C/A        UNIT    SCP     DATE1   TIME1   DESC1   ENTRIES1    EXITS1  DATE2   TIME2   ESC2    ENTRIES2    EXITS2  DATE3   TIME3   DESC3   ENTRIES3    EXITS3  DATE4   TIME4   DESC4   ENTRIES4    EXITS4  DATE5   TIME5   DESC5   ENTRIES5    EXITS5  DATE6   TIME6   DESC6   ENTRIES6    EXITS6  DATE7   TIME7   DESC7   ENTRIES7    EXITS7  DATE8   TIME8   DESC8   ENTRIES8    EXITS8
0   A002    R051    02-00-00    04-20-13    00:00:00    REGULAR 4084276 1405308 04-20-13    04:00:00    REGULAR 4084308.0   1405312.0   04-20-13    08:00:00    REGULAR 4084332.0   1405348.0   04-20-13    12:00:00    REGULAR 4084429.0   1405441.0   04-20-13    16:00:00    REGULAR 4084714.0   1405494.0   04-20-13    20:00:00    REGULAR 4085107.0   1405550.0   04-21-13    00:00:00    REGULAR 4085286.0   1405578.0   04-21-13    04:00:00    REGULAR 4085317.0   1405582.0
1   A002    R051    02-00-00    04-21-13    08:00:00    REGULAR 4085336 1405603 04-21-13    12:00:00    REGULAR 4085421.0   1405673.0   04-21-13    16:00:00    REGULAR 4085543.0   1405725.0   04-21-13    20:00:00    REGULAR 4085543.0   1405781.0   04-22-13    00:00:00    REGULAR 4085669.0   1405820.0   04-22-13    04:00:00    REGULAR 4085684.0   1405825.0   04-22-13    08:00:00    REGULAR 4085715.0   1405929.0   04-22-13    12:00:00    REGULAR 4085878.0   1406175.0
2   A002    R051    02-00-00    04-22-13    16:00:00    REGULAR 4086116 1406242 04-22-13    20:00:00    REGULAR 4086986.0   1406310.0   04-23-13    00:00:00    REGULAR 4087164.0   1406335.0   04-23-13    04:00:00    REGULAR 4087172.0   1406339.0   04-23-13    08:00:00    REGULAR 4087214.0   1406441.0   04-23-13    12:00:00    REGULAR 4087390.0   1406685.0   04-23-13    16:00:00    REGULAR 4087738.0   1406741.0   04-23-13    20:00:00    REGULAR 4088682.0   1406813.0
3   A002    R051    02-00-00    04-24-13    00:00:00    REGULAR 4088879 1406839 04-24-13    04:00:00    REGULAR 4088890.0   1406845.0   04-24-13    08:00:00    REGULAR 4088934.0   1406951.0   04-24-13    12:00:00    REGULAR 4089105.0   1407209.0   04-24-13    16:00:00    REGULAR 4089378.0   1407269.0   04-24-13    20:00:00    REGULAR 4090319.0   1407336.0   04-25-13    00:00:00    REGULAR 4090535.0   1407365.0   04-25-13    04:00:00    REGULAR 4090550.0   1407370.0
4   A002    R051    02-00-00    04-25-13    08:00:00    REGULAR 4090589 1407469 04-25-13    08:57:03    DOOR OPEN   4090629.0   1407591.0   04-25-13    08:58:01    LOGON   4090629.0   1407591.0   04-25-13    09:01:08    LGF-MAN 4090629.0   1407591.0   04-25-13    09:01:53    LOGON   4090629.0   1407591.0   04-25-13    09:02:02    DOOR CLOSE  4090629.0   1407591.0   04-25-13    09:02:04    DOOR OPEN   4090629.0   1407591.0   04-25-13    09:02:31    DOOR CLOSE  4090629.0   1407591.0
5   A002    R051    02-00-00    04-25-13    09:02:32    DOOR OPEN   4090629 1407591 04-25-13    09:07:21    LOGON   4090629.0   1407591.0   04-25-13    09:12:12    LGF-MAN 4090642.0   1407592.0   04-25-13    09:12:20    DOOR CLOSE  4090642.0   1407592.0   04-25-13    12:00:00    REGULAR 4090743.0   1407723.0   04-25-13    16:00:00    REGULAR 4091064.0   1407793.0   04-25-13    20:00:00    REGULAR 4092044.0   1407840.0   04-26-13    00:00:00    REGULAR 4092314.0   1407859.0
6   A002    R051    02-00-00    04-26-13    04:00:00    REGULAR 4092325 1407861 04-26-13    08:00:00    REGULAR 4092363.0   1407958.0   04-26-13    12:00:00    REGULAR 4092541.0   1408225.0   04-26-13    16:00:00    REGULAR 4092837.0   1408285.0   04-26-13    20:00:00    REGULAR 4093823.0   1408341.0   None    None    None    NaN NaN None    None    None    NaN NaN None    None    None    NaN NaN

My current inefficient function looks like this:

def cleanData(dataFrame):

    tempDf = dataFrame

    tempColName = ['date','time','description','entries','exit','c/a','unit', 'scp']
    finalColName = ['c/a','unit','scp','date','time','description','entries','exit']

    tempDf1 = tempDf.iloc[:,:8]
    tempDf1.dropna(inplace=True)
    tempDf1.columns = finalColName

    tempDf2 = tempDf.iloc[:,8:13]
    tempDf2['c/a'] = tempDf['C/A']
    tempDf2['unit'] = tempDf['UNIT']
    tempDf2['scp'] = tempDf['SCP']
    tempDf2.dropna(inplace=True)
    tempDf2.columns = tempColName

    tempDf3 = tempDf.iloc[:,13:18]
    tempDf3['c/a'] = tempDf['C/A']
    tempDf3['unit'] = tempDf['UNIT']
    tempDf3['scp'] = tempDf['SCP']
    tempDf3.dropna(inplace=True)
    tempDf3.columns = tempColName

    tempDf4 = tempDf.iloc[:,18:23]
    tempDf4['c/a'] = tempDf['C/A']
    tempDf4['unit'] = tempDf['UNIT']
    tempDf4['scp'] = tempDf['SCP']
    tempDf4.dropna(inplace=True)
    tempDf4.columns = tempColName

    tempDf5 = tempDf.iloc[:,23:28]
    tempDf5['c/a'] = tempDf['C/A']
    tempDf5['unit'] = tempDf['UNIT']
    tempDf5['scp'] = tempDf['SCP']
    tempDf5.dropna(inplace=True)
    tempDf5.columns = tempColName

    tempDf6 = tempDf.iloc[:,28:33]
    tempDf6['c/a'] = tempDf['C/A']
    tempDf6['unit'] = tempDf['UNIT']
    tempDf6['scp'] = tempDf['SCP']
    tempDf6.dropna(inplace=True)
    tempDf6.columns = tempColName

    tempDf7 = tempDf.iloc[:,33:38]
    tempDf7['c/a'] = tempDf['C/A']
    tempDf7['unit'] = tempDf['UNIT']
    tempDf7['scp'] = tempDf['SCP']
    tempDf7.dropna(inplace=True)
    tempDf7.columns = tempColName

    tempDf8 = tempDf.iloc[:,38:43]
    tempDf8['c/a'] = tempDf['C/A']
    tempDf8['unit'] = tempDf['UNIT']
    tempDf8['scp'] = tempDf['SCP']
    tempDf8.dropna(inplace=True)
    tempDf8.columns = tempColName

    placeHolderDf = pd.concat([tempDf2,tempDf3,tempDf4,tempDf5,tempDf6,tempDf7,tempDf8])
    placeHolderDf = placeHolderDf[['c/a','unit','scp','date','time','description','entries','exit']]
    fullData = pd.concat([tempDf1,placeHolderDf])
    fullData['date'] = pd.to_datetime(fullData['date'])

    return fullData.reset_index(drop=True)

with a correct final output like:

    c/a     unit    scp        date         time    description entries exit
0   A002    R051    02-00-00    2013-04-20  00:00:00    REGULAR 4084276 1405308
1   A002    R051    02-00-00    2013-04-21  08:00:00    REGULAR 4085336 1405603
2   A002    R051    02-00-00    2013-04-22  16:00:00    REGULAR 4086116 1406242
3   A002    R051    02-00-00    2013-04-24  00:00:00    REGULAR 4088879 1406839
4   A002    R051    02-00-00    2013-04-25  08:00:00    REGULAR 4090589 1407469

Any help is immensely appreaciated.

Body:
Hope this suit your requirements

pd.wide_to_long will automatically group the column name.

list_melt =  ['DATE','TIME','DESC','ENTRIES','EXITS']
list_id = ['C/A','UNIT','SCP']
# pd.wide_to_long id has to be unique
df['count'] = df.groupby(list_id)[list_id].cumcount()
df_melt = pd.wide_to_long(df, stubnames = list_melt, i = ['C/A','UNIT','SCP','count'], j = 'value').reset_index()
df_melt.drop(columns = ['count', 'value'], axis = 1, inplace = True)
df_melt.columns = ['c/a','unit','scp','date','time','description','entries','exit']

This is a proxy output:

list_cols = ['C/A','UNIT','SCP','DATE1','TIME1','DESC1','ENTRIES1','EXITS1','DATE2','TIME2','DESC2',\
 'ENTRIES2','EXITS2','DATE3','TIME3','DESC3','ENTRIES3','EXITS3']

df = pd.DataFrame([[1] * 8 + [1,2,3,4,5] + [6,7,8,9,10]], columns = list_cols)

df_melt
  c/a   unit    scp date    time description entries exit
0   1     1       1    1    1             1    1      1
1   1     1       1    1    2             3    4      5
2   1     1       1    6    7             8    9      10