问:
Error using boost serialization with binary archive
I get the following error while reading from boost::archive::binary_iarchive into my variable:

```<code>test-serialization(9285,0x11c62fdc0) malloc: can't allocate region
*** mach_vm_map(size=18014398509486080) failed (error code=3)
test-serialization(9285,0x11c62fdc0) malloc: *** set a breakpoint in malloc_error_break to debug
</code>```

My serialization and deserialization code are:

template<class Archive>
void save(Archive & archive, const helib::PubKey & pubkey, const unsigned int version){
  BOOST_TEST_MESSAGE("inside save_construct_data");
  archive << &(pubkey.context);
  archive << pubkey.skBounds;
  archive << pubkey.keySwitching;
  archive << pubkey.keySwitchMap;
  archive << pubkey.KS_strategy;
  archive << pubkey.recryptKeyID;
}

template<class Archive>
void load_construct_data(Archive & archive, helib::PubKey * pubkey, const unsigned int version){
  helib::Context * context = new helib::Context(2,3,1); //random numbers since there is no default constructor
  BOOST_TEST_MESSAGE("deserializing context");
  archive >> context;
  std::vector<double> skBounds;
  std::vector<helib::KeySwitch> keySwitching;
  std::vector<std::vector<long>> keySwitchMap;
  NTL::Vec<long> KS_strategy;
  long recryptKeyID;
  BOOST_TEST_MESSAGE("deserializing skbounds");
  archive >> skBounds;
  BOOST_TEST_MESSAGE("deserializing keyswitching");
  archive >> keySwitching;
  BOOST_TEST_MESSAGE("deserializing keyswitchmap");
  archive >> keySwitchMap;
  BOOST_TEST_MESSAGE("deserializing KS_strategy");
  archive >> KS_strategy;
  BOOST_TEST_MESSAGE("deserializing recryptKeyID");
  archive >> recryptKeyID;
  BOOST_TEST_MESSAGE("new pubkey");
  ::new(pubkey)helib::PubKey(*context);
  //TODO: complete
}

template<class Archive>
void serialize(Archive & archive, helib::PubKey & pubkey, const unsigned int version){
  split_free(archive, pubkey, version);
}

template<class Archive>
void load(Archive & archive, helib::PubKey & pubkey, const unsigned int version){
}

```

<p>The test that calls the code is the following:</p>

<pre class="lang-cpp prettyprint-override"><code>BOOST_AUTO_TEST_CASE(serialization_pubkey)
{
  auto context = helibTestContext();
  helib::SecKey secret_key(context);
  secret_key.GenSecKey();
  // Compute key-switching matrices that we need
  helib::addSome1DMatrices(secret_key);
  // Set the secret key (upcast: SecKey is a subclass of PubKey)
  const helib::PubKey& original_pubkey = secret_key;

  std::string filename = "pubkey.serialized";

  std::ofstream os(filename, std::ios::binary);
  {
    boost::archive::binary_oarchive oarchive(os);
    oarchive << original_pubkey;
  }

  helib::PubKey * restored_pubkey = new helib::PubKey(helib::Context(2,3,1));
  {
    std::ifstream ifs(filename, std::ios::binary);
    boost::archive::binary_iarchive iarchive(ifs);
    BOOST_TEST_CHECKPOINT("calling deserialization");
    iarchive >> restored_pubkey;
    BOOST_TEST_CHECKPOINT("done with deserialization");

    //tests ommitted
  }
}
</code>```

Considerations:

Serialization works both fine with boost::archive::text_oarchive and boost::archive::binary_oarchive. They create a file of 46M and 21M respectively (big, I know).

Deserialization with boost::archive::text_iarchive basically stopped at the execution of archive >> keySwitching; The process gets automatically killed. This is in fact the biggest part of the archive.

I decided to try with boost::archive::binary_iarchive since the file is half the size, but I get the error shown at the beginning. The error happens while executing the first read from the archive: archive >> context;.

The asymmetry between input and output (save and load_construct_data) is because I could not find another way to avoid the implementation of the serialization of a derived class of helib::PubKey. Using a pointer to helib::PubKey was giving me compilation errors asking for the serialization of the derived class. If there is some other way I'm all ears.

Thank you for your help.

UPDATE:

I am implementing deserialization for some classes in the cryptographic library HElib because I need to send ciphertext over the wire. One of these classes is helib::PubKey. I'm using the boost serialization library for the implementation. I have created a gist to provide a reprex as suggested in the comments. There are 3 files:

serialization.hpp, it contains the serialiation implementation. Unfortunately, helib::PubKey depends on many other classes making the file rather long. All the other classes have unit tests that pass. Furthermore, I had to make a tiny modification to the class with the goal of serializing it. I made public the private members.

test-serialization.cpp, it contains the unit test.

Makefile. Running make creates the executable test-serialization.

答:
vector<bool> strikes again

It's actually allocating for 0x1fffffffff20000 bits (that's 144 petabits) on my test box. That's coming directly from IndexSet::resize().

Now I have serious questions about HElib using std::vector<bool> here (it seems they would be far better served with something like boost::icl::interval_set<>).

Well. That was a wild goose chase (that IndexSet serialization can be much improved). However, the real problem is that you had Undefined Behaviour because you don't deserialize the same type as you serialize.

You serialize a PubKey, but attempt to deserialize as PubKey*. Uhoh.

Now beyond that, there's quite a bit of problems:

You had to modify the library to make private members public. This can easily violate ODR (making the class layout incompatible).

You seem to treat the context as a "dynamic" resource, which will engage Object Tracking. This could be a viable approach. BUT. You'll have to think about ownership.

It seems like you didn't do that yet. For example, the line in load_construct_data for DoublCRT is a definite memory-leak:

```<code>helib::Context * context = new helib::Context(2,3,1);
</code>```

You never use it nor ever free it. In fact, you simply overwrite it with the deserialized instance, which may or may not be owned. Catch-22

Exactly the same happens in load_construct_data for PubKey.

worse, in save_construct_data you completely gratuitously copy context objects for each DoubleCRT in each SecKey:

```<code> auto context = polynomial->getContext();
 archive << &context;
</code>```

Because you fake it out as pointer-serialization, again (obviously useless) object tracking kicks in, just meaning you serialize redundant Context copies which will will be all be leaked un deserialization.

I'd be tempted to assume the context instances in both would always be the same? Why not serialize the context(s) separately anyways?

In fact I went and analyzed the HElib source code to check these assumptions. It turns out I was correct. Nothing ever constructs a context outside

```<code>std::unique_ptr<Context> buildContextFromBinary(std::istream& str);
std::unique_ptr<Context> buildContextFromAscii(std::istream& str);
</code>```

As you can see, they return owned pointers. You should have been using them. Perhaps even with the built-in serialization, that I practically stumble over here.

Time To Regroup

I'd use the serialization code from HElib (because, why reinvent the wheel and make a ton of bugs doing so?). If you insist on integration with Boost Serialization, you can have your cake and eat it:

```<code>template <class Archive> void save(Archive& archive, const helib::PubKey& pubkey, unsigned) {
    using V = std::vector<char>;
    using D = iostreams::back_insert_device<V>;
    V data;
    {
        D dev(data);
        iostreams::stream_buffer<D> sbuf(dev);
        std::ostream os(&sbuf); // expose as std::ostream
        helib::writePubKeyBinary(os, pubkey);
    }
    archive << data;
}

template <class Archive> void load(Archive& archive, helib::PubKey& pubkey, unsigned) {
    std::vector<char> data;
    archive >> data;
    using S = iostreams::array_source;
    S source(data.data(), data.size());
    iostreams::stream_buffer<S> sbuf(source);
    {
        std::istream is(&sbuf); // expose as std::istream
        helib::readPubKeyBinary(is, pubkey);
    }
}
</code>```

That's all. 24 lines of code. And it's gonna be tested and maintained by the library authors. You can't beat that (clearly). I've modified the tests a bit so we don't abuse private details anymore.

Cleaning Up The Code

By separating out a helper to deal with the blob writing, we can implement different helib types in a very similar way:

```<code>namespace helib { // leverage ADL
    template <class A> void save(A& ar, const Context& o, unsigned) {
        Blob data = to_blob(o, writeContextBinary);
        ar << data;
    }
    template <class A> void load(A& ar, Context& o, unsigned) {
        Blob data;
        ar >> data;
        from_blob(data, o, readContextBinary);
    }
    template <class A> void save(A& ar, const PubKey& o, unsigned) {
        Blob data = to_blob(o, writePubKeyBinary);
        ar << data;
    }
    template <class A> void load(A& ar, PubKey& o, unsigned) {
        Blob data;
        ar >> data;
        from_blob(data, o, readPubKeyBinary);
    }
}
</code>```

This is elegance to me.

FULL LISTING

I have cloned a new gist https://gist.github.com/sehe/ba82a0329e4ec586363eb82d3f3b9326 that includes the following change-sets:

```<code>0079c07 Make it compile locally
b3b2cf1 Squelch the warnings
011b589 Endof investigations, regroup time

f4d79a6 Reimplemented using HElib binary IO
a403e97 Bitwise reproducible outputs
</code>```

Only the last two commits contains changes related to the actual fixes.

I'll list the full code here too for posterity. There are a number of subtle reorganizations and ditto comments in the test code. You'd do well to read through them carefully to see whether you understand them and the implications suit your needs. I left comments describing why the test assertions are what they are to help.

File serialization.hpp

```<code>#ifndef EVOTING_SERIALIZATION_H
#define EVOTING_SERIALIZATION_H

#define BOOST_TEST_MODULE main
#include <helib/helib.h>
#include <boost/serialization/split_free.hpp>
#include <boost/serialization/vector.hpp>
#include <boost/iostreams/stream_buffer.hpp>
#include <boost/iostreams/device/back_inserter.hpp>
#include <boost/iostreams/device/array.hpp>

namespace /* file-static */ {
    using Blob = std::vector<char>;

    template <typename T, typename F>
    Blob to_blob(const T& object, F writer) {
        using D = boost::iostreams::back_insert_device<Blob>;
        Blob data;
        {
            D dev(data);
            boost::iostreams::stream_buffer<D> sbuf(dev);
            std::ostream os(&sbuf); // expose as std::ostream
            writer(os, object);
        }
        return data;
    }

    template <typename T, typename F>
    void from_blob(Blob const& data, T& object, F reader) {
        boost::iostreams::stream_buffer<boost::iostreams::array_source>
            sbuf(data.data(), data.size());
        std::istream is(&sbuf); // expose as std::istream
        reader(is, object);
    }
}

namespace helib { // leverage ADL
    template <class A> void save(A& ar, const Context& o, unsigned) {
        Blob data = to_blob(o, writeContextBinary);
        ar << data;
    }
    template <class A> void load(A& ar, Context& o, unsigned) {
        Blob data;
        ar >> data;
        from_blob(data, o, readContextBinary);
    }
    template <class A> void save(A& ar, const PubKey& o, unsigned) {
        Blob data = to_blob(o, writePubKeyBinary);
        ar << data;
    }
    template <class A> void load(A& ar, PubKey& o, unsigned) {
        Blob data;
        ar >> data;
        from_blob(data, o, readPubKeyBinary);
    }
}

BOOST_SERIALIZATION_SPLIT_FREE(helib::Context)
BOOST_SERIALIZATION_SPLIT_FREE(helib::PubKey)
#endif //EVOTING_SERIALIZATION_H
</code>```

File test-serialization.cpp

```<code>#define BOOST_TEST_MODULE main
#include <boost/test/included/unit_test.hpp>
#include <helib/helib.h>
#include <fstream>
#include "serialization.hpp"
#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/archive/binary_iarchive.hpp>

helib::Context helibTestMinimalContext(){
  // Plaintext prime modulus
  unsigned long p = 4999;
  // Cyclotomic polynomial - defines phi(m)
  unsigned long m = 32109;
  // Hensel lifting (default = 1)
  unsigned long r = 1;
  return helib::Context(m, p, r);
}

helib::Context helibTestContext(){
  auto context = helibTestMinimalContext();

  // Number of bits of the modulus chain
  unsigned long bits = 300;
  // Number of columns of Key-Switching matix (default = 2 or 3)
  unsigned long c = 2;

  // Modify the context, adding primes to the modulus chain
  buildModChain(context, bits, c);
  return context;
}

BOOST_AUTO_TEST_CASE(serialization_pubkey) {
    auto context = helibTestContext();
    helib::SecKey secret_key(context);
    secret_key.GenSecKey();
    // Compute key-switching matrices that we need
    helib::addSome1DMatrices(secret_key);
    // Set the secret key (upcast: SecKey is a subclass of PubKey)
    const helib::PubKey& original_pubkey = secret_key;

    std::string const filename = "pubkey.serialized";

    {
        std::ofstream os(filename, std::ios::binary);
        boost::archive::binary_oarchive oarchive(os);
        oarchive << context << original_pubkey;
    }
    {
        // just checking reproducible output
        std::ofstream os(filename + ".2", std::ios::binary);
        boost::archive::binary_oarchive oarchive(os);
        oarchive << context << original_pubkey;
    }

    // reading back to independent instances of Context/PubKey
    {
        // NOTE: if you start from something rogue, it will fail with PAlgebra mismatch.
        helib::Context surrogate = helibTestMinimalContext();

        std::ifstream ifs(filename, std::ios::binary);
        boost::archive::binary_iarchive iarchive(ifs);
        iarchive >> surrogate;

        // we CAN test that the contexts end up matching
        BOOST_TEST((context == surrogate));

        helib::SecKey independent(surrogate);
        helib::PubKey& indep_pk = independent;
        iarchive >> indep_pk;
        // private again, as it should be, but to understand the relation:
        // BOOST_TEST((&independent.context == &surrogate));

        // The library's operator== compares the reference, so it would say "not equal"
        BOOST_TEST((indep_pk != original_pubkey));
        {
            // just checking reproducible output
            std::ofstream os(filename + ".3", std::ios::binary);
            boost::archive::binary_oarchive oarchive(os);
            oarchive << surrogate << indep_pk;
        }
    }

    // doing it the other way (sharing the context):
    {
        helib::PubKey restored_pubkey(context);
        {
            std::ifstream ifs(filename, std::ios::binary);
            boost::archive::binary_iarchive iarchive(ifs);
            iarchive >> context >> restored_pubkey;
        }
        // now `operator==` confirms equality
        BOOST_TEST((restored_pubkey == original_pubkey));

        {
            // just checking reproducible output
            std::ofstream os(filename + ".4", std::ios::binary);
            boost::archive::binary_oarchive oarchive(os);
            oarchive << context << restored_pubkey;
        }
    }
}
</code>```

TEST OUTPUT

```<code>time ./test-serialization -l all -r detailed
Running 1 test case...
Entering test module "main"
test-serialization.cpp(34): Entering test case "serialization_pubkey"
test-serialization.cpp(61): info: check (context == surrogate) has passed
test-serialization.cpp(70): info: check (indep_pk != original_pubkey) has passed
test-serialization.cpp(82): info: check (restored_pubkey == original_pubkey) has passed
test-serialization.cpp(34): Leaving test case "serialization_pubkey"; testing time: 36385217us
Leaving test module "main"; testing time: 36385273us

Test module "main" has passed with:
  1 test case out of 1 passed
  3 assertions out of 3 passed

  Test case "serialization_pubkey" has passed with:
    3 assertions out of 3 passed

real    0m36,698s
user    0m35,558s
sys     0m0,850s
</code>```

Bitwise Reproducible Outputs

On repeated serialization it appears that indeed the output is bitwise identical, which may be an important property:

```<code>sha256sum pubkey.serialized*
66b95adbd996b100bff58774e066e7a309e70dff7cbbe08b5c77b9fa0f63c97f  pubkey.serialized
66b95adbd996b100bff58774e066e7a309e70dff7cbbe08b5c77b9fa0f63c97f  pubkey.serialized.2
66b95adbd996b100bff58774e066e7a309e70dff7cbbe08b5c77b9fa0f63c97f  pubkey.serialized.3
66b95adbd996b100bff58774e066e7a309e70dff7cbbe08b5c77b9fa0f63c97f  pubkey.serialized.4
</code>```

Note that it is (obviously) not identical across runs (because it generates different key material).

Side Quest (The Wild Goose Chase)

One way to improve the IndexSet serialization code manually is to also use vector<bool>:

```<code> auto context = polynomial->getContext();
 archive << &context;
</code>```0

Better idea would be to use dynamic_bitset (for which I happen to have contributed the serialization code (see How to serialize boost::dynamic_bitset?)):

```<code> auto context = polynomial->getContext();
 archive << &context;
</code>```1

Of course, you would likely have to do similar things for IndexMap.

Body:
I get the following error while reading from boost::archive::binary_iarchive into my variable:

test-serialization(9285,0x11c62fdc0) malloc: can't allocate region
*** mach_vm_map(size=18014398509486080) failed (error code=3)
test-serialization(9285,0x11c62fdc0) malloc: *** set a breakpoint in malloc_error_break to debug

My serialization and deserialization code are:

template<class Archive>
void save(Archive & archive, const helib::PubKey & pubkey, const unsigned int version){
  BOOST_TEST_MESSAGE("inside save_construct_data");
  archive << &(pubkey.context);
  archive << pubkey.skBounds;
  archive << pubkey.keySwitching;
  archive << pubkey.keySwitchMap;
  archive << pubkey.KS_strategy;
  archive << pubkey.recryptKeyID;
}

template<class Archive>
void load_construct_data(Archive & archive, helib::PubKey * pubkey, const unsigned int version){
  helib::Context * context = new helib::Context(2,3,1); //random numbers since there is no default constructor
  BOOST_TEST_MESSAGE("deserializing context");
  archive >> context;
  std::vector<double> skBounds;
  std::vector<helib::KeySwitch> keySwitching;
  std::vector<std::vector<long>> keySwitchMap;
  NTL::Vec<long> KS_strategy;
  long recryptKeyID;
  BOOST_TEST_MESSAGE("deserializing skbounds");
  archive >> skBounds;
  BOOST_TEST_MESSAGE("deserializing keyswitching");
  archive >> keySwitching;
  BOOST_TEST_MESSAGE("deserializing keyswitchmap");
  archive >> keySwitchMap;
  BOOST_TEST_MESSAGE("deserializing KS_strategy");
  archive >> KS_strategy;
  BOOST_TEST_MESSAGE("deserializing recryptKeyID");
  archive >> recryptKeyID;
  BOOST_TEST_MESSAGE("new pubkey");
  ::new(pubkey)helib::PubKey(*context);
  //TODO: complete
}

template<class Archive>
void serialize(Archive & archive, helib::PubKey & pubkey, const unsigned int version){
  split_free(archive, pubkey, version);
}

template<class Archive>
void load(Archive & archive, helib::PubKey & pubkey, const unsigned int version){
}

The test that calls the code is the following:

BOOST_AUTO_TEST_CASE(serialization_pubkey)
{
  auto context = helibTestContext();
  helib::SecKey secret_key(context);
  secret_key.GenSecKey();
  // Compute key-switching matrices that we need
  helib::addSome1DMatrices(secret_key);
  // Set the secret key (upcast: SecKey is a subclass of PubKey)
  const helib::PubKey& original_pubkey = secret_key;

  std::string filename = "pubkey.serialized";

  std::ofstream os(filename, std::ios::binary);
  {
    boost::archive::binary_oarchive oarchive(os);
    oarchive << original_pubkey;
  }

  helib::PubKey * restored_pubkey = new helib::PubKey(helib::Context(2,3,1));
  {
    std::ifstream ifs(filename, std::ios::binary);
    boost::archive::binary_iarchive iarchive(ifs);
    BOOST_TEST_CHECKPOINT("calling deserialization");
    iarchive >> restored_pubkey;
    BOOST_TEST_CHECKPOINT("done with deserialization");

    //tests ommitted
  }
}

Considerations:

Serialization works both fine with boost::archive::text_oarchive and boost::archive::binary_oarchive. They create a file of 46M and 21M respectively (big, I know).

Deserialization with boost::archive::text_iarchive basically stopped at the execution of archive >> keySwitching; The process gets automatically killed. This is in fact the biggest part of the archive.

I decided to try with boost::archive::binary_iarchive since the file is half the size, but I get the error shown at the beginning. The error happens while executing the first read from the archive: archive >> context;.

The asymmetry between input and output (save and load_construct_data) is because I could not find another way to avoid the implementation of the serialization of a derived class of helib::PubKey. Using a pointer to helib::PubKey was giving me compilation errors asking for the serialization of the derived class. If there is some other way I'm all ears.

Thank you for your help.

UPDATE:

I am implementing deserialization for some classes in the cryptographic library HElib because I need to send ciphertext over the wire. One of these classes is helib::PubKey. I'm using the boost serialization library for the implementation. I have created a gist to provide a reprex as suggested in the comments. There are 3 files:

serialization.hpp, it contains the serialiation implementation. Unfortunately, helib::PubKey depends on many other classes making the file rather long. All the other classes have unit tests that pass. Furthermore, I had to make a tiny modification to the class with the goal of serializing it. I made public the private members.

test-serialization.cpp, it contains the unit test.

Makefile. Running make creates the executable test-serialization.

Body:
vector<bool> strikes again

It's actually allocating for 0x1fffffffff20000 bits (that's 144 petabits) on my test box. That's coming directly from IndexSet::resize().

Now I have serious questions about HElib using std::vector<bool> here (it seems they would be far better served with something like boost::icl::interval_set<>).

Well. That was a wild goose chase (that IndexSet serialization can be much improved). However, the real problem is that you had Undefined Behaviour because you don't deserialize the same type as you serialize.

You serialize a PubKey, but attempt to deserialize as PubKey*. Uhoh.

Now beyond that, there's quite a bit of problems:

You had to modify the library to make private members public. This can easily violate ODR (making the class layout incompatible).

You seem to treat the context as a "dynamic" resource, which will engage Object Tracking. This could be a viable approach. BUT. You'll have to think about ownership.

It seems like you didn't do that yet. For example, the line in load_construct_data for DoublCRT is a definite memory-leak:

helib::Context * context = new helib::Context(2,3,1);

You never use it nor ever free it. In fact, you simply overwrite it with the deserialized instance, which may or may not be owned. Catch-22

Exactly the same happens in load_construct_data for PubKey.

worse, in save_construct_data you completely gratuitously copy context objects for each DoubleCRT in each SecKey:

 auto context = polynomial->getContext();
 archive << &context;

Because you fake it out as pointer-serialization, again (obviously useless) object tracking kicks in, just meaning you serialize redundant Context copies which will will be all be leaked un deserialization.

I'd be tempted to assume the context instances in both would always be the same? Why not serialize the context(s) separately anyways?

In fact I went and analyzed the HElib source code to check these assumptions. It turns out I was correct. Nothing ever constructs a context outside

std::unique_ptr<Context> buildContextFromBinary(std::istream& str);
std::unique_ptr<Context> buildContextFromAscii(std::istream& str);

As you can see, they return owned pointers. You should have been using them. Perhaps even with the built-in serialization, that I practically stumble over here.

Time To Regroup

I'd use the serialization code from HElib (because, why reinvent the wheel and make a ton of bugs doing so?). If you insist on integration with Boost Serialization, you can have your cake and eat it:

template <class Archive> void save(Archive& archive, const helib::PubKey& pubkey, unsigned) {
    using V = std::vector<char>;
    using D = iostreams::back_insert_device<V>;
    V data;
    {
        D dev(data);
        iostreams::stream_buffer<D> sbuf(dev);
        std::ostream os(&sbuf); // expose as std::ostream
        helib::writePubKeyBinary(os, pubkey);
    }
    archive << data;
}

template <class Archive> void load(Archive& archive, helib::PubKey& pubkey, unsigned) {
    std::vector<char> data;
    archive >> data;
    using S = iostreams::array_source;
    S source(data.data(), data.size());
    iostreams::stream_buffer<S> sbuf(source);
    {
        std::istream is(&sbuf); // expose as std::istream
        helib::readPubKeyBinary(is, pubkey);
    }
}

That's all. 24 lines of code. And it's gonna be tested and maintained by the library authors. You can't beat that (clearly). I've modified the tests a bit so we don't abuse private details anymore.

Cleaning Up The Code

By separating out a helper to deal with the blob writing, we can implement different helib types in a very similar way:

namespace helib { // leverage ADL
    template <class A> void save(A& ar, const Context& o, unsigned) {
        Blob data = to_blob(o, writeContextBinary);
        ar << data;
    }
    template <class A> void load(A& ar, Context& o, unsigned) {
        Blob data;
        ar >> data;
        from_blob(data, o, readContextBinary);
    }
    template <class A> void save(A& ar, const PubKey& o, unsigned) {
        Blob data = to_blob(o, writePubKeyBinary);
        ar << data;
    }
    template <class A> void load(A& ar, PubKey& o, unsigned) {
        Blob data;
        ar >> data;
        from_blob(data, o, readPubKeyBinary);
    }
}

This is elegance to me.

FULL LISTING

I have cloned a new gist https://gist.github.com/sehe/ba82a0329e4ec586363eb82d3f3b9326 that includes the following change-sets:

0079c07 Make it compile locally
b3b2cf1 Squelch the warnings
011b589 Endof investigations, regroup time

f4d79a6 Reimplemented using HElib binary IO
a403e97 Bitwise reproducible outputs

Only the last two commits contains changes related to the actual fixes.

I'll list the full code here too for posterity. There are a number of subtle reorganizations and ditto comments in the test code. You'd do well to read through them carefully to see whether you understand them and the implications suit your needs. I left comments describing why the test assertions are what they are to help.

File serialization.hpp

#ifndef EVOTING_SERIALIZATION_H
#define EVOTING_SERIALIZATION_H

#define BOOST_TEST_MODULE main
#include <helib/helib.h>
#include <boost/serialization/split_free.hpp>
#include <boost/serialization/vector.hpp>
#include <boost/iostreams/stream_buffer.hpp>
#include <boost/iostreams/device/back_inserter.hpp>
#include <boost/iostreams/device/array.hpp>

namespace /* file-static */ {
    using Blob = std::vector<char>;

    template <typename T, typename F>
    Blob to_blob(const T& object, F writer) {
        using D = boost::iostreams::back_insert_device<Blob>;
        Blob data;
        {
            D dev(data);
            boost::iostreams::stream_buffer<D> sbuf(dev);
            std::ostream os(&sbuf); // expose as std::ostream
            writer(os, object);
        }
        return data;
    }

    template <typename T, typename F>
    void from_blob(Blob const& data, T& object, F reader) {
        boost::iostreams::stream_buffer<boost::iostreams::array_source>
            sbuf(data.data(), data.size());
        std::istream is(&sbuf); // expose as std::istream
        reader(is, object);
    }
}

namespace helib { // leverage ADL
    template <class A> void save(A& ar, const Context& o, unsigned) {
        Blob data = to_blob(o, writeContextBinary);
        ar << data;
    }
    template <class A> void load(A& ar, Context& o, unsigned) {
        Blob data;
        ar >> data;
        from_blob(data, o, readContextBinary);
    }
    template <class A> void save(A& ar, const PubKey& o, unsigned) {
        Blob data = to_blob(o, writePubKeyBinary);
        ar << data;
    }
    template <class A> void load(A& ar, PubKey& o, unsigned) {
        Blob data;
        ar >> data;
        from_blob(data, o, readPubKeyBinary);
    }
}

BOOST_SERIALIZATION_SPLIT_FREE(helib::Context)
BOOST_SERIALIZATION_SPLIT_FREE(helib::PubKey)
#endif //EVOTING_SERIALIZATION_H

File test-serialization.cpp

#define BOOST_TEST_MODULE main
#include <boost/test/included/unit_test.hpp>
#include <helib/helib.h>
#include <fstream>
#include "serialization.hpp"
#include <boost/archive/text_oarchive.hpp>
#include <boost/archive/text_iarchive.hpp>
#include <boost/archive/binary_oarchive.hpp>
#include <boost/archive/binary_iarchive.hpp>

helib::Context helibTestMinimalContext(){
  // Plaintext prime modulus
  unsigned long p = 4999;
  // Cyclotomic polynomial - defines phi(m)
  unsigned long m = 32109;
  // Hensel lifting (default = 1)
  unsigned long r = 1;
  return helib::Context(m, p, r);
}

helib::Context helibTestContext(){
  auto context = helibTestMinimalContext();

  // Number of bits of the modulus chain
  unsigned long bits = 300;
  // Number of columns of Key-Switching matix (default = 2 or 3)
  unsigned long c = 2;

  // Modify the context, adding primes to the modulus chain
  buildModChain(context, bits, c);
  return context;
}

BOOST_AUTO_TEST_CASE(serialization_pubkey) {
    auto context = helibTestContext();
    helib::SecKey secret_key(context);
    secret_key.GenSecKey();
    // Compute key-switching matrices that we need
    helib::addSome1DMatrices(secret_key);
    // Set the secret key (upcast: SecKey is a subclass of PubKey)
    const helib::PubKey& original_pubkey = secret_key;

    std::string const filename = "pubkey.serialized";

    {
        std::ofstream os(filename, std::ios::binary);
        boost::archive::binary_oarchive oarchive(os);
        oarchive << context << original_pubkey;
    }
    {
        // just checking reproducible output
        std::ofstream os(filename + ".2", std::ios::binary);
        boost::archive::binary_oarchive oarchive(os);
        oarchive << context << original_pubkey;
    }

    // reading back to independent instances of Context/PubKey
    {
        // NOTE: if you start from something rogue, it will fail with PAlgebra mismatch.
        helib::Context surrogate = helibTestMinimalContext();

        std::ifstream ifs(filename, std::ios::binary);
        boost::archive::binary_iarchive iarchive(ifs);
        iarchive >> surrogate;

        // we CAN test that the contexts end up matching
        BOOST_TEST((context == surrogate));

        helib::SecKey independent(surrogate);
        helib::PubKey& indep_pk = independent;
        iarchive >> indep_pk;
        // private again, as it should be, but to understand the relation:
        // BOOST_TEST((&independent.context == &surrogate));

        // The library's operator== compares the reference, so it would say "not equal"
        BOOST_TEST((indep_pk != original_pubkey));
        {
            // just checking reproducible output
            std::ofstream os(filename + ".3", std::ios::binary);
            boost::archive::binary_oarchive oarchive(os);
            oarchive << surrogate << indep_pk;
        }
    }

    // doing it the other way (sharing the context):
    {
        helib::PubKey restored_pubkey(context);
        {
            std::ifstream ifs(filename, std::ios::binary);
            boost::archive::binary_iarchive iarchive(ifs);
            iarchive >> context >> restored_pubkey;
        }
        // now `operator==` confirms equality
        BOOST_TEST((restored_pubkey == original_pubkey));

        {
            // just checking reproducible output
            std::ofstream os(filename + ".4", std::ios::binary);
            boost::archive::binary_oarchive oarchive(os);
            oarchive << context << restored_pubkey;
        }
    }
}

TEST OUTPUT

time ./test-serialization -l all -r detailed
Running 1 test case...
Entering test module "main"
test-serialization.cpp(34): Entering test case "serialization_pubkey"
test-serialization.cpp(61): info: check (context == surrogate) has passed
test-serialization.cpp(70): info: check (indep_pk != original_pubkey) has passed
test-serialization.cpp(82): info: check (restored_pubkey == original_pubkey) has passed
test-serialization.cpp(34): Leaving test case "serialization_pubkey"; testing time: 36385217us
Leaving test module "main"; testing time: 36385273us

Test module "main" has passed with:
  1 test case out of 1 passed
  3 assertions out of 3 passed

  Test case "serialization_pubkey" has passed with:
    3 assertions out of 3 passed

real    0m36,698s
user    0m35,558s
sys     0m0,850s

Bitwise Reproducible Outputs

On repeated serialization it appears that indeed the output is bitwise identical, which may be an important property:

sha256sum pubkey.serialized*
66b95adbd996b100bff58774e066e7a309e70dff7cbbe08b5c77b9fa0f63c97f  pubkey.serialized
66b95adbd996b100bff58774e066e7a309e70dff7cbbe08b5c77b9fa0f63c97f  pubkey.serialized.2
66b95adbd996b100bff58774e066e7a309e70dff7cbbe08b5c77b9fa0f63c97f  pubkey.serialized.3
66b95adbd996b100bff58774e066e7a309e70dff7cbbe08b5c77b9fa0f63c97f  pubkey.serialized.4

Note that it is (obviously) not identical across runs (because it generates different key material).

Side Quest (The Wild Goose Chase)

One way to improve the IndexSet serialization code manually is to also use vector<bool>:

template<class Archive>
    void save(Archive & archive, const helib::IndexSet & index_set, const unsigned int version){
        std::vector<bool> elements;
        elements.resize(index_set.last()-index_set.first()+1);
        for (auto n : index_set)
            elements[n-index_set.first()] = true;
        archive << index_set.first() << elements;
    }

template<class Archive>
    void load(Archive & archive, helib::IndexSet & index_set, const unsigned int version){
        long first_ = 0;
        std::vector<bool> elements;
        archive >> first_ >> elements;
        index_set.clear();
        for (size_t n = 0; n < elements.size(); ++n) {
            if (elements[n])
                index_set.insert(n+first_);
        }
    }

Better idea would be to use dynamic_bitset (for which I happen to have contributed the serialization code (see How to serialize boost::dynamic_bitset?)):

template<class Archive>
    void save(Archive & archive, const helib::IndexSet & index_set, const unsigned int version){
        boost::dynamic_bitset<> elements;
        elements.resize(index_set.last()-index_set.first()+1);
        for (auto n : index_set)
            elements.set(n-index_set.first());
        archive << index_set.first() << elements;
    }

template<class Archive>
    void load(Archive & archive, helib::IndexSet & index_set, const unsigned int version) {
        long first_ = 0;
        boost::dynamic_bitset<> elements;
        archive >> first_ >> elements;
        index_set.clear();
        for (size_t n = elements.find_first(); n != -1; n = elements.find_next(n))
            index_set.insert(n+first_);
    }

Of course, you would likely have to do similar things for IndexMap.


来源: stackexchange

问:
Getting exception while executing Azure function locally when trying to connect to local database

Have installed nuget for System.Data.SqlClient 4.8.1. Getting below error on running

[5/28/2020 4:09:39 PM] Executed 'Function1' (Failed, Id=990d4cbf-bd32-4a77-ab52-8b5fb1dce252)
[5/28/2020 4:09:39 PM] System.Private.CoreLib: Exception while executing function: Function1. FunctionAppDemo: Could not load file or assembly 'System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621). System.Private.CoreLib: Could not load file or assembly 'System.Data.SqlClient, Version=4.6.1.1, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.

Even when I downgrade to Version=4.6.1.1, I get similar error with version 4.5.1

答:
I didn't reproduce your issue on my side. I am using .NET Core 2.0 and System.Data.SqlClient(4.8.1).

Please go to Dependencies to check if System.Data.SqlClient is there.

If the dependency is there, you can refer to the solutions here.

ContentLicense: CC BY-SA 4.0


问:
Nlohmann not installing to /usr/local/include
My vmImage is 'ubuntu-latest'.

When my build comes to nlohmann I get the following error:

```<code>-- Up-to-date: /usr/local/include
CMake Error at libraries/nlohmann_json/cmake_install.cmake:41 (file):
  file INSTALL cannot set permissions on "/usr/local/include": Operation not
  permitted.
Call Stack (most recent call first):
  libraries/cmake_install.cmake:42 (include)
  cmake_install.cmake:42 (include)
</code>```

It works fine locally and also on the Windows and MacOS vmImage pipelines, so I am assuming this is some type of permissions issue/setting with DevOps?

The yaml file is as follows:

```<code># Starter pipeline
trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
  displayName: 'install python 3.x'
  inputs:
    versionSpec: '3.x'
    addToPath: true
    architecture: 'x64'

- task: CmdLine@2
  displayName: 'install opengl'
  inputs:
    script: 'sudo apt-get -y install freeglut3 freeglut3-dev libglew1.5 libglew1.5-dev libglu1-mesa libglu1-mesa-dev libgl1-mesa-glx libgl1-mesa-dev'
    failOnStderr: true

- task: CmdLine@2
  displayName: 'install sdl'
  inputs:
    script: 'sudo apt-get install -y libsdl2-dev libsdl2-image-dev libsdl2-ttf-dev libsdl2-net-dev'
    failOnStderr: true

- task: CmdLine@2
  displayName: 'update google gpg key'
  inputs:
    script: 'wget -q -O - curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -'
    failOnStderr: false

- task: CmdLine@2
  displayName: 'install gcc 10'
  inputs:
    script: 'sudo apt-get update && sudo add-apt-repository ppa:ubuntu-toolchain-r/test && sudo apt-get update && sudo apt-get -y install gcc-10 g++-10'
    failOnStderr: true

- task: PythonScript@0
  displayName: 'run build.py'
  inputs:
    scriptSource: 'filePath'
    scriptPath: '$(Build.SourcesDirectory)/build.py'
    failOnStderr: false
</code>```

答:
In the end, I disabled the install step of nlohmann json by doing the following before I added the subdirectory:

```<code>set(JSON_Install OFF CACHE INTERNAL "")
</code>```

Body:
My vmImage is 'ubuntu-latest'.

When my build comes to nlohmann I get the following error:

-- Up-to-date: /usr/local/include
CMake Error at libraries/nlohmann_json/cmake_install.cmake:41 (file):
  file INSTALL cannot set permissions on "/usr/local/include": Operation not
  permitted.
Call Stack (most recent call first):
  libraries/cmake_install.cmake:42 (include)
  cmake_install.cmake:42 (include)

It works fine locally and also on the Windows and MacOS vmImage pipelines, so I am assuming this is some type of permissions issue/setting with DevOps?

The yaml file is as follows:

# Starter pipeline
trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
- task: UsePythonVersion@0
  displayName: 'install python 3.x'
  inputs:
    versionSpec: '3.x'
    addToPath: true
    architecture: 'x64'

- task: CmdLine@2
  displayName: 'install opengl'
  inputs:
    script: 'sudo apt-get -y install freeglut3 freeglut3-dev libglew1.5 libglew1.5-dev libglu1-mesa libglu1-mesa-dev libgl1-mesa-glx libgl1-mesa-dev'
    failOnStderr: true

- task: CmdLine@2
  displayName: 'install sdl'
  inputs:
    script: 'sudo apt-get install -y libsdl2-dev libsdl2-image-dev libsdl2-ttf-dev libsdl2-net-dev'
    failOnStderr: true

- task: CmdLine@2
  displayName: 'update google gpg key'
  inputs:
    script: 'wget -q -O - curl https://bazel.build/bazel-release.pub.gpg | sudo apt-key add -'
    failOnStderr: false

- task: CmdLine@2
  displayName: 'install gcc 10'
  inputs:
    script: 'sudo apt-get update && sudo add-apt-repository ppa:ubuntu-toolchain-r/test && sudo apt-get update && sudo apt-get -y install gcc-10 g++-10'
    failOnStderr: true

- task: PythonScript@0
  displayName: 'run build.py'
  inputs:
    scriptSource: 'filePath'
    scriptPath: '$(Build.SourcesDirectory)/build.py'
    failOnStderr: false

Body:
In the end, I disabled the install step of nlohmann json by doing the following before I added the subdirectory:

set(JSON_Install OFF CACHE INTERNAL "")


问:
How to skip a few password protected excel files when processing a large number of excel workbooks in c#
Is it possible to skip password protected Excel files in c#?

I dont know the passwords to those files, and because all these files are supposed to be processed automatically, the program needs to be able to skip all the files that ask password, as the prompt asking for user input for the password will interrupt the program.

This is my code for opening the Excel file:

```<code>Excel.Workbook workbook = app.Workbooks.Open(fullFileName, ReadOnly: false, Password: "");
</code>```

答:
you can use EPPlus, with it it's relativly easy, only problem is to know is exception raised because protection or something else, here is the code :</p>

```<code>public bool IsXlsxPasswordProtected(string fileName)
{
  bool encrypted = false;
  FileStream fs = new FileStream(fileName, FileMode.Open);
  ExcelPackage pack = new ExcelPackage();
  try
  {
    pack.Load(fs);
  }
  catch (Exception ex)
  {
    /// maybe there is better way to know if exception is because file is protected
    /// with password, idealy EPP should raise dedicated exception type
    /// e.g. PasswordProtectedException
    if (ex.InnerException != null && ex.InnerException is FileFormatException)
      encrypted = true;
    else
      throw;
  }
  return encrypted;
}
</code>```

ContentLicense: CC BY-SA 3.0

Body:
Is it possible to skip password protected Excel files in c#?

I dont know the passwords to those files, and because all these files are supposed to be processed automatically, the program needs to be able to skip all the files that ask password, as the prompt asking for user input for the password will interrupt the program.

This is my code for opening the Excel file:

Excel.Workbook workbook = app.Workbooks.Open(fullFileName, ReadOnly: false, Password: "");

Body:
you can use EPPlus, with it it's relativly easy, only problem is to know is exception raised because protection or something else, here is the code :

public bool IsXlsxPasswordProtected(string fileName)
{
  bool encrypted = false;
  FileStream fs = new FileStream(fileName, FileMode.Open);
  ExcelPackage pack = new ExcelPackage();
  try
  {
    pack.Load(fs);
  }
  catch (Exception ex)
  {
    /// maybe there is better way to know if exception is because file is protected
    /// with password, idealy EPP should raise dedicated exception type
    /// e.g. PasswordProtectedException
    if (ex.InnerException != null && ex.InnerException is FileFormatException)
      encrypted = true;
    else
      throw;
  }
  return encrypted;
}


问:
How come declaring something as a double data type would lead the ide to think it is a integer?
I am very confused on how a declared double data type would lead to the intellij ide to believe I was requiring a int to declare. Can someone give me some insight on this?

Here is the code in a picture:
enter image description here

答:
The value inside the square brackets is for defining the length of the array, therefore it needs an int.
If you want cholo to be in the array then use</p>

```<code>double[] yu = new double[] {cholo};
</code>```

Body:
The value inside the square brackets is for defining the length of the array, therefore it needs an int.
If you want cholo to be in the array then use

double[] yu = new double[] {cholo};


问:
Error with pyspark in local when I execute pytest in VS Code from Git Bash
I have the Error with pyspark in local when I execute pytest in VS Code from Git Bash.

If I execute pytest from gitbash console:

I try to debug my code when I create the spark Dataframe in Visual code, show me:

Java gateway process exited before sending its port number

I have configured all environment of variables in my PC:

I have seen this error can be my variable Java is not configured really well, but I have checked it is properly.

答:
It's like the version mismatch causes this. As '_PYSPARK_DRIVER_CALLBACK_HOST' has been removed during version 0.10.7(here) to 2.3, and gets back from version 2.3.1. So, you should check your Spark version, as the SPARK_HOME points to the right one(at least 2.3.1).

from here


问:
Why custom Middleware usage position affect its calling?
Why does this work as expected:

```<code>    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        app.UseRequestTimeoutMiddleware();
        app.UseExceptionHandlingMiddleware();

        app.UseHttpsRedirection();

        app.UseRouting();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }
</code>```

but this, does not:

```<code>    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        app.UseHttpsRedirection();

        app.UseRouting();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });

        app.UseRequestTimeoutMiddleware();
        app.UseExceptionHandlingMiddleware();
    }
</code>```

On the second case, the InvokeAsync is never called, that's what I mean by doesn't work.

答:
It has to do with the way ASP.NET Core middleware ordering works. Microsoft has some great documentation here regarding middleware ordering, but for the sake of these docs changing I'll copy some of it to here.

Review the middleware execution order below:

You can see the order in which ASP.NET Core executes through the middleware pipeline, with the endpoint routing being the last thing in the execution order. By placing your custom middleware after the endpoint routing, it's essentially being short-circuited at that point and never being hit (as it traverses back up the execution chain).

From the link above, the docs state:

The order is critical for security, performance, and functionality.

To the love of some and hate of others, the convention is a bit of a gotcha when it comes to expected behavior.

Body:
Why does this work as expected:

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        app.UseRequestTimeoutMiddleware();
        app.UseExceptionHandlingMiddleware();

        app.UseHttpsRedirection();

        app.UseRouting();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });
    }

but this, does not:

    public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
    {
        app.UseHttpsRedirection();

        app.UseRouting();

        app.UseEndpoints(endpoints =>
        {
            endpoints.MapControllers();
        });

        app.UseRequestTimeoutMiddleware();
        app.UseExceptionHandlingMiddleware();
    }

On the second case, the InvokeAsync is never called, that's what I mean by doesn't work.


问:
Including one list component and excluding all other components repeatedly in R
I have a list of values. I would like to use the first value in one portion of a function and use the remaining values in a separate part of the function. Then, I would like to use the second value from the list in one portion of the function and use the remaining values (including the first value) in another part of the function.

The following is not real code since I have excluded some function inputs, but it shows what I am trying to do:

```<code>group <- c(1, 3, 4, 5, 9) #the complete group
#It is important to note that these values are not continuous. They are distinct values.

#include 1st unit of the list group as treatment and then use
#the remaining values as controls

controls_not2 <- c(3, 4, 5, 9)
df1 <- dataprep(treatment = 1, controls = controls_not1)
gaps.plot(dataprep = df1)

#use the 2nd unit from the list as the treatment and use
#the remaining values as controls

controls_not2 <- c(1, 4, 5, 9)
df2 <- dataprep(treatment = 3, controls = controls_not3)
gaps.plot(dataprep = df3)
</code>```

In the full list, I have about 50 values. Is there a way for me to iterate over all these values in the way I have shown above?

Here is some additional information.

Packages used:

```<code>library(tidyverse)
library(synth)
</code>```

Here is an example of what the full code looks like to create a single graph.

treatment.identifier is equivalent to what I have as treatment in the simple example above. controls.identifier is equivalent to what I have as controls in the example above.

```<code>dataprep.out1 <- dataprep(foo = as.data.frame(df),
                         predictors = predictors1,
                         predictors.op = "mean",
                         special.predictors = NULL,
                         dependent = "logfatalitiespercapita",
                         unit.variable = "state",
                         time.variable = "year",
                     treatment.identifier = "99",
                     controls.identifier = controlstates,
                     time.predictors.prior = c(1981:1985),
                     time.optimize.ssr = c(1981:1985), #check on these years
                     time.plot = 1981:2003,
                     unit.names.variable = "statenames")

#identifying weights that create the best possible synthetic control unit for the treated
synth.out1 <- synth(dataprep.out1)

#graph
spec1 <- gaps.plot(dataprep.res = dataprep.out1, synth.res = synth.out1, Xlab = c("Year"), Ylab = c("Gap"))
</code>```

答:
To store the results in a list, this ignores what the values are and uses 1:50 as the index

```<code>library(purrr)
result <- map(1:50, ~{ #replace 50 with the length of your list
df <- dataprep(treatment = group[[.x]], controls = group[[-.x]])
 gaps.plot(dataprep = df)})
</code>```

This is a similar solution inspired by the solution above.

```<code>newfunction <- function(x) {
    df <- dataprep(treatment = group[x], controls = group[-x])
     gaps.plot(dataprep = df)
}

graphs <- lapply(group, newfunction)
</code>```

Body:
I have a list of values. I would like to use the first value in one portion of a function and use the remaining values in a separate part of the function. Then, I would like to use the second value from the list in one portion of the function and use the remaining values (including the first value) in another part of the function.

The following is not real code since I have excluded some function inputs, but it shows what I am trying to do:

group <- c(1, 3, 4, 5, 9) #the complete group
#It is important to note that these values are not continuous. They are distinct values.

#include 1st unit of the list group as treatment and then use
#the remaining values as controls

controls_not2 <- c(3, 4, 5, 9)
df1 <- dataprep(treatment = 1, controls = controls_not1)
gaps.plot(dataprep = df1)

#use the 2nd unit from the list as the treatment and use
#the remaining values as controls

controls_not2 <- c(1, 4, 5, 9)
df2 <- dataprep(treatment = 3, controls = controls_not3)
gaps.plot(dataprep = df3)

In the full list, I have about 50 values. Is there a way for me to iterate over all these values in the way I have shown above?

Here is some additional information.

Packages used:

library(tidyverse)
library(synth)

Here is an example of what the full code looks like to create a single graph.

treatment.identifier is equivalent to what I have as treatment in the simple example above. controls.identifier is equivalent to what I have as controls in the example above.

dataprep.out1 <- dataprep(foo = as.data.frame(df),
                         predictors = predictors1,
                         predictors.op = "mean",
                         special.predictors = NULL,
                         dependent = "logfatalitiespercapita",
                         unit.variable = "state",
                         time.variable = "year",
                     treatment.identifier = "99",
                     controls.identifier = controlstates,
                     time.predictors.prior = c(1981:1985),
                     time.optimize.ssr = c(1981:1985), #check on these years
                     time.plot = 1981:2003,
                     unit.names.variable = "statenames")

#identifying weights that create the best possible synthetic control unit for the treated
synth.out1 <- synth(dataprep.out1)

#graph
spec1 <- gaps.plot(dataprep.res = dataprep.out1, synth.res = synth.out1, Xlab = c("Year"), Ylab = c("Gap"))

Body:
To store the results in a list, this ignores what the values are and uses 1:50 as the index

library(purrr)
result <- map(1:50, ~{ #replace 50 with the length of your list
df <- dataprep(treatment = group[[.x]], controls = group[[-.x]])
 gaps.plot(dataprep = df)})

This is a similar solution inspired by the solution above.

newfunction <- function(x) {
    df <- dataprep(treatment = group[x], controls = group[-x])
     gaps.plot(dataprep = df)
}

graphs <- lapply(group, newfunction)


问:
PHP Cookie management
here is what my client requirements are:

1> There is a common web page that lets each of the customer login. On successful login, a cookie is created and the customer is redirected to a "Waiting area" page. I need to create a cookie that would last for 10 secs and keep refreshing the time until the customer has closed the window.

2> The admin page will list all the customers who have not yet closed the "Waiting area" page and are waiting on the admin to move forward.

Since I am new to session and cookie management, I am not sure exactly what needs to be done here i.e. use session_start or setcookie ? Since the admin needs a list of all customers waiting, I guess all these entries need to go in a database table? And whenever a customer closes the page, the entry should be deleted from the database.

Is my understanding correct? If anyone has better suggestions, please let me know.

答:
If you make it database related, this means you will need to update table every second. If you decide to go with that, you can create a simple table which holds session ids, end_time.

To avoid the every second connection to the db, I would suggest when admin clicks on let's say viewSessions.php on top of that script to have a single query which deletes all sessions which should be deleted - DELETE FROM my_sessions WHERE end_time<=(DATE_SUB(now(), INTERVAL 10 SECOND));

Title: PHP Cookie management


答:
You have pass the size of the double array which is integer, that's why you're getting that error.
This is the shortcut syntax to create and initialize an array of double</p>

```<code>double[] values = {1,2,3,4,};
</code>```

Title: How come declaring something as a double data type would lead the ide to think it is a integer?

Body:
I am very confused on how a declared double data type would lead to the intellij ide to believe I was requiring a int to declare. Can someone give me some insight on this?

Here is the code in a picture:
enter image description here

Body:
You have pass the size of the double array which is integer, that's why you're getting that error.
This is the shortcut syntax to create and initialize an array of double

double[] values = {1,2,3,4,};


问:
How to execute a file with multiple queries in Godror?
I would like to execute an sql-file with the godror Oracle driver for Go. My problem is that godror does not accept more than 1 query in the file. Do you have any idea how I can solve this?

```<code>db2, _ := sql.Open("godror", databaseURL)

c, ioErr := ioutil.ReadFile(sqlFile)

_, err := db2.Exec(string(c))

defer db2.Close()
</code>```

With this code this query works:

```<code>create TABLE books3 (title VARCHAR2(100))
</code>```

But this doesn't:

```<code>create TABLE books3 (title VARCHAR2(100));
create TABLE books4 (title VARCHAR2(100));
create TABLE books6 (title VARCHAR2(100));
</code>```

答:
You have to write your own file parser to extract the statements from the file, and execute them one at a time. It's easier if you use a slightly stricter syntax than SQL*Plus files can have, e.g. if you always use / as the terminator for both SQL & PL/SQL, and you make sure the file doesn't contain SQL*Plus statements (like COLUMN).  One related parser is in Python at https://github.com/oracle/python-cx_Oracle/blob/7.3/samples/SampleEnv.py#L124-L157

A better way is probably to store the strings in your Go code.

Body:
I would like to execute an sql-file with the godror Oracle driver for Go. My problem is that godror does not accept more than 1 query in the file. Do you have any idea how I can solve this?

db2, _ := sql.Open("godror", databaseURL)

c, ioErr := ioutil.ReadFile(sqlFile)

_, err := db2.Exec(string(c))

defer db2.Close()

With this code this query works:

create TABLE books3 (title VARCHAR2(100))

But this doesn't:

create TABLE books3 (title VARCHAR2(100));
create TABLE books4 (title VARCHAR2(100));
create TABLE books6 (title VARCHAR2(100));


问:
Converting daily data to summed/averaged monthly data with a specific layout format
I've looked through similar past questions but have yet to find something specific to what I'm looking for.

I have daily data, that I would like to convert to average/sum monthly data. With the final product being a dataframe with months in the column and years in the rows Example.

I've managed to get the monthly average of my dataset using:

```<code>library(xts)
ts <- xts(data$tmax, as.Date(data$date, "%Y-%m-%d"))
ts_m = apply.monthly(ts, mean)

    data$Date   data$tmax
1   1951-01-01  3.22777778
2   1951-01-02  6.48888889
3   1951-01-03  10.52777778
4   1951-01-04  1.92777778
5   1951-01-05  1.30000000
6   1951-01-06  0.10000000
7   1951-01-07  -6.72777778
8   1951-01-08  -4.48888889
9   1951-01-09  -0.83888889
10  1951-02-01  -9.92777778
11  1951-02-02  -11.60000000
12  1951-02-03  -8.61111111
13  1951-02-04  -1.40000000
... ...        ...
</code>```

The code above gives me an xts with the averages:

```<code>Y-M-D       Tmax_avg
1951-01-09  1.279630
1951-02-12  -3.548611
</code>```

But I can't figure out out to convert the layout of the xts (or if I have to convert the xts) so that it looks like this (months running down, and years running across):

```<code>   1951   1952  1953
01 1.27   ...
02 -3.54  ...
...
12 ...    ...
</code>```

Thanks in advance!

答:
We can extract the 'year' and the 'Date' and then use xtabs

```<code>Year <- format(as.Date(index(tsm)), '%Y')
Month <- format(as.Date(index(tsm)), '%m')
df1 <- data.frame(Year, Month, tmax = tsm[,1])
xtabs(tmax ~ Month + Year, df1)
</code>```

Body:
I've looked through similar past questions but have yet to find something specific to what I'm looking for.

I have daily data, that I would like to convert to average/sum monthly data. With the final product being a dataframe with months in the column and years in the rows Example.

I've managed to get the monthly average of my dataset using:

library(xts)
ts <- xts(data$tmax, as.Date(data$date, "%Y-%m-%d"))
ts_m = apply.monthly(ts, mean)

    data$Date   data$tmax
1   1951-01-01  3.22777778
2   1951-01-02  6.48888889
3   1951-01-03  10.52777778
4   1951-01-04  1.92777778
5   1951-01-05  1.30000000
6   1951-01-06  0.10000000
7   1951-01-07  -6.72777778
8   1951-01-08  -4.48888889
9   1951-01-09  -0.83888889
10  1951-02-01  -9.92777778
11  1951-02-02  -11.60000000
12  1951-02-03  -8.61111111
13  1951-02-04  -1.40000000
... ...        ...

The code above gives me an xts with the averages:

Y-M-D       Tmax_avg
1951-01-09  1.279630
1951-02-12  -3.548611

But I can't figure out out to convert the layout of the xts (or if I have to convert the xts) so that it looks like this (months running down, and years running across):

   1951   1952  1953
01 1.27   ...
02 -3.54  ...
...
12 ...    ...

Thanks in advance!

Body:
We can extract the 'year' and the 'Date' and then use xtabs

Year <- format(as.Date(index(tsm)), '%Y')
Month <- format(as.Date(index(tsm)), '%m')
df1 <- data.frame(Year, Month, tmax = tsm[,1])
xtabs(tmax ~ Month + Year, df1)


问:
Best way to find Order of Time series from matrix of groups
Is there a standard way of doing this?

I basically have some users, who performed actions together and split off as a group. We don't know the order of the events, but can infer them:

```<code>                      A B C D E
 WentToMall           1 1 1 0 0
 DroveToTheMovies     1 0 0 0 0
 AteLunchTogether     1 1 0 0 0
 BoughtClothes        1 1 0 0 1
 BoughtElectronics    1 1 0 0 0
</code>```

The rule is they can't converge together after.
So the time series would look like:

Time 0 is always all of them together, then the largest 'grouping' is splitting off into
'WentToMall' where we get A,B,C and D,E split off.

From there, it looks like AB split off from C, and AB proceed to 'AteLunchTogether, BoughtClothes, BoughtElectronics'. Sometime during 'boughtclothes', it looks like E split off from D.
Finally, A and B split off at the end as A 'Drove to the movies'.

If possible, I'd like to also show this visually, maybe with nodes showing the number of events separating the split (which would look like):

```<code>ABCDE   ---> ABC --> AB  ->A
 |             |      |->B
 |             |
 |             |--> C
 |
 |
 |---> DE --> D
       |-->E
</code>```

A problem that comes up is sometimes you get time points which are 'difficult to asses' or appear contradictory, and don't fit in based on the minimal amount of columns. I'm not sure what to do about those either. I am given 'weights' for the actions, so I could decide based on that, or I guess generate all versions of the graph.

I was thinking maybe of recursion to do a search through, or similar?

答:
edit: the latest file is here

The process is through recursion. Pandas is useful in your scenario, though there might be a more efficient ways to do this.

We search from the further nodes. In your case, these would be A and E nodes. How we know these are the furthest nodes? Just count 0 and 1 values of all rows. Then get sum of 0 and 1 values. Also, sort values by 0. For first case, it should be like that:

```<code>                    0   1
DroveToTheMovies    4.0 1.0
AteLunchTogether    3.0 2.0
BoughtElectronics   3.0 2.0
WentToMall          2.0 3.0
BoughtClothes       2.0 3.0
FirstCase           0.0 5.0
</code>```

This means there is 1 person drove to the movies. You see the pattern. There are people joining this person later on. In first case, there are 5 people which we began with. But there is a problem. How we know whether the previous person was in the group? Let's say X drove to the movies. Now we check for ate lunch. Say Y and Z joined the group, but not X. For this case, we will check if the latest group is in the new group. So until we reach first case, we add all the event to an array. So now we have a branch.

Assume there were people not in group case. In this case, we stored this odd behavior as well. Then, we go from there now. In first case our beginning node was A; not it is B using the same technique. So the process will be repeated again.

My final results were like that:

```<code>    0                   1
0   DroveToTheMovies    Index(['A'], dtype='object')
1   AteLunchTogether    Index(['A', 'B'], dtype='object')
2   BoughtElectronics   Index(['A', 'B'], dtype='object')
3   WentToMall          Index(['A', 'B', 'C'], dtype='object')
4   FirstCase           Index(['A', 'B', 'C', 'D', 'E'], dtype='object')
5   BoughtClothes       Index(['E'], dtype='object')
6   FirstCase           Index(['D', 'E'], dtype='object')
</code>```

There are two FirstCase. But you need to process these two FirstCase values and know that, this D-E group is from the first FirstCase group, then E went for bought clothes. D is unknown, therefore could be assigned as something else. And there you have it.

First branch:

```<code>ABCDE   ---> ABC --> AB  ->A
               |      |->B
               |
               |--> C
</code>```

Second branch:

```<code>(first case)---> DE --> D
                  |-->E
</code>```

All you have to do is now find who left branches. For first branch it is B, C, and D-E. These are easy to calculate from now on. Hope it'll help you. The code is here, and I suggest to to debug the code in order to get the whole idea more clear:

```<code>import pandas as pd

df = pd.DataFrame(
   [[1, 1, 1, 0, 0],
    [1, 0, 0, 0, 0],
    [1, 1, 0, 0, 0],
    [1, 1, 0, 0, 1],
    [1, 1, 0, 0, 0]], columns=list("ABCDE"))

df.index = ['WentToMall', 'DroveToTheMovies', 'AteLunchTogether', 'BoughtClothes', 'BoughtElectronics']

first_case = pd.DataFrame(
   [[1, 1, 1, 1, 1]], columns=list("ABCDE"), index=['FirstCase'])

all_case = pd.concat([first_case, df])

def case_finder(all_case):
    df_case = all_case.apply(lambda x: x.value_counts(), axis=1).fillna(0)
    df_case = df_case.loc[df_case[1] != 0]
    return df_case.sort_values(by=1)

def check_together(x):
    x = df.iloc[x]
    activity = all_case.loc[x.name]
    does_activity = activity.loc[activity == 1]
    return activity.name, does_activity.index

def check_in(pre, now):
    return pre.isin(now).all()

def check_odd(i):
    act = check_together(i)[0]
    who = check_together(i)[1][~check_together(i)[1].isin(check_together(i-1)[1])]
    return act, who

df = case_finder(all_case)
total = all_case.shape[0]

all_acts = []
last_stable = []
while True:
    for i in range(total):
        act, ind = check_together(i)
        if ind.size == 1:
            print("Initiliazed!")
            all_acts.append([act, ind])
            pass
        else:
            p_act, p_ind = check_together(i-1)
            if check_in(p_ind, ind) == True:
                print("So a new person joins us!")
                all_acts.append([act, ind])
            else:
                print("This is weird. We'll check later!")
                # act, who = check_odd(i)
                last_stable.append([i, p_ind])
                continue
        if act == 'FirstCase':
            break

    if len(last_stable) == 0:
        print("Process done!")
        break
    else:
        print("Update cases!")
        ls_ind = last_stable[0]
        all_case = all_case.drop(last_stable[0][1], axis=1)
        total = all_case.shape[0]
        df = case_finder(all_case)
        last_stable = last_stable[1:]

print(all_acts)

x = pd.DataFrame(all_acts)
</code>```

Body:
Is there a standard way of doing this?

I basically have some users, who performed actions together and split off as a group. We don't know the order of the events, but can infer them:

                      A B C D E
 WentToMall           1 1 1 0 0
 DroveToTheMovies     1 0 0 0 0
 AteLunchTogether     1 1 0 0 0
 BoughtClothes        1 1 0 0 1
 BoughtElectronics    1 1 0 0 0

The rule is they can't converge together after.
So the time series would look like:

Time 0 is always all of them together, then the largest 'grouping' is splitting off into
'WentToMall' where we get A,B,C and D,E split off.

From there, it looks like AB split off from C, and AB proceed to 'AteLunchTogether, BoughtClothes, BoughtElectronics'. Sometime during 'boughtclothes', it looks like E split off from D.
Finally, A and B split off at the end as A 'Drove to the movies'.

If possible, I'd like to also show this visually, maybe with nodes showing the number of events separating the split (which would look like):

ABCDE   ---> ABC --> AB  ->A
 |             |      |->B
 |             |
 |             |--> C
 |
 |
 |---> DE --> D
       |-->E

A problem that comes up is sometimes you get time points which are 'difficult to asses' or appear contradictory, and don't fit in based on the minimal amount of columns. I'm not sure what to do about those either. I am given 'weights' for the actions, so I could decide based on that, or I guess generate all versions of the graph.

I was thinking maybe of recursion to do a search through, or similar?

Body:
edit: the latest file is here

The process is through recursion. Pandas is useful in your scenario, though there might be a more efficient ways to do this.

We search from the further nodes. In your case, these would be A and E nodes. How we know these are the furthest nodes? Just count 0 and 1 values of all rows. Then get sum of 0 and 1 values. Also, sort values by 0. For first case, it should be like that:

                    0   1
DroveToTheMovies    4.0 1.0
AteLunchTogether    3.0 2.0
BoughtElectronics   3.0 2.0
WentToMall          2.0 3.0
BoughtClothes       2.0 3.0
FirstCase           0.0 5.0

This means there is 1 person drove to the movies. You see the pattern. There are people joining this person later on. In first case, there are 5 people which we began with. But there is a problem. How we know whether the previous person was in the group? Let's say X drove to the movies. Now we check for ate lunch. Say Y and Z joined the group, but not X. For this case, we will check if the latest group is in the new group. So until we reach first case, we add all the event to an array. So now we have a branch.

Assume there were people not in group case. In this case, we stored this odd behavior as well. Then, we go from there now. In first case our beginning node was A; not it is B using the same technique. So the process will be repeated again.

My final results were like that:

    0                   1
0   DroveToTheMovies    Index(['A'], dtype='object')
1   AteLunchTogether    Index(['A', 'B'], dtype='object')
2   BoughtElectronics   Index(['A', 'B'], dtype='object')
3   WentToMall          Index(['A', 'B', 'C'], dtype='object')
4   FirstCase           Index(['A', 'B', 'C', 'D', 'E'], dtype='object')
5   BoughtClothes       Index(['E'], dtype='object')
6   FirstCase           Index(['D', 'E'], dtype='object')

There are two FirstCase. But you need to process these two FirstCase values and know that, this D-E group is from the first FirstCase group, then E went for bought clothes. D is unknown, therefore could be assigned as something else. And there you have it.

First branch:

ABCDE   ---> ABC --> AB  ->A
               |      |->B
               |
               |--> C

Second branch:

(first case)---> DE --> D
                  |-->E

All you have to do is now find who left branches. For first branch it is B, C, and D-E. These are easy to calculate from now on. Hope it'll help you. The code is here, and I suggest to to debug the code in order to get the whole idea more clear:

import pandas as pd

df = pd.DataFrame(
   [[1, 1, 1, 0, 0],
    [1, 0, 0, 0, 0],
    [1, 1, 0, 0, 0],
    [1, 1, 0, 0, 1],
    [1, 1, 0, 0, 0]], columns=list("ABCDE"))

df.index = ['WentToMall', 'DroveToTheMovies', 'AteLunchTogether', 'BoughtClothes', 'BoughtElectronics']

first_case = pd.DataFrame(
   [[1, 1, 1, 1, 1]], columns=list("ABCDE"), index=['FirstCase'])

all_case = pd.concat([first_case, df])

def case_finder(all_case):
    df_case = all_case.apply(lambda x: x.value_counts(), axis=1).fillna(0)
    df_case = df_case.loc[df_case[1] != 0]
    return df_case.sort_values(by=1)

def check_together(x):
    x = df.iloc[x]
    activity = all_case.loc[x.name]
    does_activity = activity.loc[activity == 1]
    return activity.name, does_activity.index

def check_in(pre, now):
    return pre.isin(now).all()

def check_odd(i):
    act = check_together(i)[0]
    who = check_together(i)[1][~check_together(i)[1].isin(check_together(i-1)[1])]
    return act, who

df = case_finder(all_case)
total = all_case.shape[0]

all_acts = []
last_stable = []
while True:
    for i in range(total):
        act, ind = check_together(i)
        if ind.size == 1:
            print("Initiliazed!")
            all_acts.append([act, ind])
            pass
        else:
            p_act, p_ind = check_together(i-1)
            if check_in(p_ind, ind) == True:
                print("So a new person joins us!")
                all_acts.append([act, ind])
            else:
                print("This is weird. We'll check later!")
                # act, who = check_odd(i)
                last_stable.append([i, p_ind])
                continue
        if act == 'FirstCase':
            break

    if len(last_stable) == 0:
        print("Process done!")
        break
    else:
        print("Update cases!")
        ls_ind = last_stable[0]
        all_case = all_case.drop(last_stable[0][1], axis=1)
        total = all_case.shape[0]
        df = case_finder(all_case)
        last_stable = last_stable[1:]

print(all_acts)

x = pd.DataFrame(all_acts)


问:
How to let a webworker do multiple tasks simultaneously?
I am trying to let a Web-Worker manage its state, meanwhile serving multiple async requests.

worker.ts file

```<code>let a =0; //this is my worker's state

let worker=self as unknown as Worker;

worker.onmessage =(e)=>{
    console.log("Rec msg", e.data);

    if(e.data === "+1"){
        setTimeout(()=>{
            a=a+1;
            worker.postMessage(a);
        },3000);
    }else if(e.data=== "+2"){
        setTimeout(()=>{
            a=a+2;
            worker.postMessage(a);
        },1000)
    }
}
</code>```

And this is my main file: main.ts

```<code>let w =new Worker("./worker.ts", {type: "module"})

let wf =async (op: string)=>{
    w.postMessage(op);
    return new Promise<any>((res,rej)=>{
        w.onmessage=res;
    });
}

(async()=>{
    let f1 = await wf("+1");
    console.log("f1",f1.data);
})();

(async()=>{
    let f2 = await wf("+2");
    console.log("f2",f2.data);
})()
</code>```

Only f2 is returned , and f1 is lost.
I have used timeouts to simulate say some async task done by worker themselves.

How do I receive both f1 and f2?

答:
Your problem is that you are trying to take an event based API and use it as a Promise based one, but events may fire multiple times, while Promise should resolve only once.

The communication between the Worker and the main thread works by sending and receiving messages, but there is by default no one-to-one relation between these messages. Both ends of the communication (ports) will simply stack incoming messages, and handle them sequentially, when they'll get time.

In your code, the main thread's worker.onmessage handler of f1 has been overwritten by the second call f2 synchronously (one microtask later, but that's still synchronous for our matter).

You could attach your event using the addEventListener method, at least this way it wouldn't be overwritten. But even then, when the first message event will fire on worker, both handlers will think it's there own message that did arrive, while in fact it was the one of f2. so that's not what you need...

What you need is to set up a protocol of communication which would allow both ends to identify each task. You could for instance wrap all your tasks' data with an object containing a .UIID member, be sure both ends wraps their message this way, and then from main thread check that UUID to resolve the appropriate Promise.

But that can become a bit complicated to implement and to use.

My personal favorite way is to create a new MessageChannel per task. If you don't know this API, I invite you to read this answer of mine explaining the basics.

Since we are sure the only one message that will come through this MessageChannel is the response from the Worker to the one task we sent to it, we can await it just like a Promise.

All we have to do, is to make sure that in the Worker thread we respond through the transferred port instead of the global scope.

const url = getWorkerURL();
const worker = new Worker(url)

const workerFunc = (op) => {
  // we create a new MessageChannel
  const channel = new MessageChannel();
  // we transfer one of its ports to the Worker thread
  worker.postMessage(op, [channel.port1]);

  return new Promise((res,rej) => {
    // we listen for a message from the remaining port of our MessageChannel
    channel.port2.onmessage = (evt) => res(evt.data);
  });
}

(async () => {
  const f1 = await workerFunc("+1");
  console.log("f1", f1);
})();

(async () => {
  const f2 = await workerFunc("+2");
  console.log("f2", f2);
})()

// SO only
function getWorkerURL() {
  const elem = document.querySelector( '[type="worker-script"]' );
  const script = elem.textContent;
  const blob = new Blob( [script], { type: "text/javascript" } );
  return URL.createObjectURL( blob );
}```
<pre class="snippet-code-html lang-html prettyprint-override"><code><script type="worker-script">
let a = 0;
const worker = self;

worker.onmessage = (evt) => {
  const port = evt.ports[0]; // this is where we will respond
  if (evt.data === "+1") {
    setTimeout(() => {
      a = a + 1;
      // we respond through the 'port'
      port.postMessage(a);
    }, 3000);
  }
  else if (evt.data === "+2") {
    setTimeout(() => {
      a = a + 2;
      // we respond through the 'port'
      port.postMessage(a);
    }, 1000)
  }
};
</script></code>```

Body:
I am trying to let a Web-Worker manage its state, meanwhile serving multiple async requests.

worker.ts file

let a =0; //this is my worker's state

let worker=self as unknown as Worker;

worker.onmessage =(e)=>{
    console.log("Rec msg", e.data);

    if(e.data === "+1"){
        setTimeout(()=>{
            a=a+1;
            worker.postMessage(a);
        },3000);
    }else if(e.data=== "+2"){
        setTimeout(()=>{
            a=a+2;
            worker.postMessage(a);
        },1000)
    }
}

And this is my main file: main.ts

let w =new Worker("./worker.ts", {type: "module"})

let wf =async (op: string)=>{
    w.postMessage(op);
    return new Promise<any>((res,rej)=>{
        w.onmessage=res;
    });
}

(async()=>{
    let f1 = await wf("+1");
    console.log("f1",f1.data);
})();

(async()=>{
    let f2 = await wf("+2");
    console.log("f2",f2.data);
})()

Only f2 is returned , and f1 is lost.
I have used timeouts to simulate say some async task done by worker themselves.

How do I receive both f1 and f2?

Body:
Your problem is that you are trying to take an event based API and use it as a Promise based one, but events may fire multiple times, while Promise should resolve only once.

The communication between the Worker and the main thread works by sending and receiving messages, but there is by default no one-to-one relation between these messages. Both ends of the communication (ports) will simply stack incoming messages, and handle them sequentially, when they'll get time.

In your code, the main thread's worker.onmessage handler of f1 has been overwritten by the second call f2 synchronously (one microtask later, but that's still synchronous for our matter).

You could attach your event using the addEventListener method, at least this way it wouldn't be overwritten. But even then, when the first message event will fire on worker, both handlers will think it's there own message that did arrive, while in fact it was the one of f2. so that's not what you need...

What you need is to set up a protocol of communication which would allow both ends to identify each task. You could for instance wrap all your tasks' data with an object containing a .UIID member, be sure both ends wraps their message this way, and then from main thread check that UUID to resolve the appropriate Promise.

But that can become a bit complicated to implement and to use.

My personal favorite way is to create a new MessageChannel per task. If you don't know this API, I invite you to read this answer of mine explaining the basics.

Since we are sure the only one message that will come through this MessageChannel is the response from the Worker to the one task we sent to it, we can await it just like a Promise.

All we have to do, is to make sure that in the Worker thread we respond through the transferred port instead of the global scope.

const url = getWorkerURL();
const worker = new Worker(url)

const workerFunc = (op) => {
  // we create a new MessageChannel
  const channel = new MessageChannel();
  // we transfer one of its ports to the Worker thread
  worker.postMessage(op, [channel.port1]);

  return new Promise((res,rej) => {
    // we listen for a message from the remaining port of our MessageChannel
    channel.port2.onmessage = (evt) => res(evt.data);
  });
}

(async () => {
  const f1 = await workerFunc("+1");
  console.log("f1", f1);
})();

(async () => {
  const f2 = await workerFunc("+2");
  console.log("f2", f2);
})()

// SO only
function getWorkerURL() {
  const elem = document.querySelector( '[type="worker-script"]' );
  const script = elem.textContent;
  const blob = new Blob( [script], { type: "text/javascript" } );
  return URL.createObjectURL( blob );
}

<script type="worker-script">
let a = 0;
const worker = self;

worker.onmessage = (evt) => {
  const port = evt.ports[0]; // this is where we will respond
  if (evt.data === "+1") {
    setTimeout(() => {
      a = a + 1;
      // we respond through the 'port'
      port.postMessage(a);
    }, 3000);
  }
  else if (evt.data === "+2") {
    setTimeout(() => {
      a = a + 2;
      // we respond through the 'port'
      port.postMessage(a);
    }, 1000)
  }
};
</script>


问:
How to filter out problems in VSCode based on text?
I'm using VSCode 1.41.0 on Windows 10 x64.

In my code (which is in a research language you've likely never heard of), there is a certain class of warning message I get a lot of in the Problems panel, and I want to ignore those messages. The text of the message varies a bit across instances, but it always contains the word "duplicate". How can I filter out problems containing "duplicate"? The text filter that already exists keeps problems containing the specified text, and I want the opposite.

I tried !duplicate but it didn't seem to do anything.

答: Not the answer you are looking for, but negative filters currently only work with glob patterns, not with text. This is a known issue with the filter feature of the problem panel in vscode. You can check the progress here: https://github.com/microsoft/vscode/issues/62531</p>


问:
How can I combine these two async functions?
I'm looping thru a teams array getting team data and players for each team...

I have 2 async functions that pull the same data but I can't figure out how to combine them.

As it works currently, each loop adds the teamPlayers array to a players array in the first async function. On completion I have 1450 player objects in my players array. The second async function adds the team object to a teams array. On completion I have 32 team objects in my teams array.

I've tried changing the first async function to get data and then have the dispatch return a value of results.teamPlayers but I end up with a players array with 32 arrays each with x number of player objects.

```<code>async function loadRosters(endPoints, teams, authKey) {
    axios.defaults.headers.common['Authorization'] = authKey;

    const requests = teams.map(teamId => {
        try {
            return axios.get(endPoints.roster + teamId).then(r => r.data.teamPlayers);
        } catch (e) {
            console.log(e);
        }
        return [];
    });

    const results = (await Promise.all(requests)).flat();
    dispatch({ type: 'SET_PLAYERS', value: results });

    return results;
}

async function loadTeamData(endPoints, teams, authKey) {
    axios.defaults.headers.common['Authorization'] = authKey;

    const requests = teams.map(teamId => {
        try {
            return axios.get(endPoints.roster + teamId).then(r => r.data.team);
        } catch (e) {
            //
        }
        return [];
    });

    const results = (await Promise.all(requests)).flat();
    dispatch({ type: 'SET_TEAM_LOGOS', value: results });

    return results;
}
</code>```

Here is the structure of the data I'm getting:

```<code>{
  "team": {
    "teamId": "string",
    "abbr": "string",
    "logo": "string"
  },
  "teamPlayers": [
    {
      "firstName": "string",
      "lastName": "string",
      "esbId": "string",
      "position": "string",
      "jerseyNumber": 0,
      "teamAbbr": "string",
      "headshot": "string"
    }
  ]
}
</code>```

It may be simpler if I get the team.logo and add it to each player in teamPlayers.

答:
I figured it out but please tell me if you think there is a more efficient solution.</p>

```<code>async function loadRosters(endPoints, teams, authKey) {
    axios.defaults.headers.common['Authorization'] = authKey;

    const requests = teams.map(teamId => {
        try {
            return axios.get(endPoints.roster + teamId).then(r => r.data);
        } catch (e) {
            //
        }
        return [];
    });

    const results = (await Promise.all(requests)).flat();
    dispatch({ type: 'SET_PLAYERS', value: results.map(r => r.teamPlayers).flat(Infinity) });
    dispatch({ type: 'SET_TEAM_LOGOS', value: results.map(r => r.team) });

    return results;
}
</code>```

Body:
I'm looping thru a teams array getting team data and players for each team...

I have 2 async functions that pull the same data but I can't figure out how to combine them.

As it works currently, each loop adds the teamPlayers array to a players array in the first async function. On completion I have 1450 player objects in my players array. The second async function adds the team object to a teams array. On completion I have 32 team objects in my teams array.

I've tried changing the first async function to get data and then have the dispatch return a value of results.teamPlayers but I end up with a players array with 32 arrays each with x number of player objects.

async function loadRosters(endPoints, teams, authKey) {
    axios.defaults.headers.common['Authorization'] = authKey;

    const requests = teams.map(teamId => {
        try {
            return axios.get(endPoints.roster + teamId).then(r => r.data.teamPlayers);
        } catch (e) {
            console.log(e);
        }
        return [];
    });

    const results = (await Promise.all(requests)).flat();
    dispatch({ type: 'SET_PLAYERS', value: results });

    return results;
}

async function loadTeamData(endPoints, teams, authKey) {
    axios.defaults.headers.common['Authorization'] = authKey;

    const requests = teams.map(teamId => {
        try {
            return axios.get(endPoints.roster + teamId).then(r => r.data.team);
        } catch (e) {
            //
        }
        return [];
    });

    const results = (await Promise.all(requests)).flat();
    dispatch({ type: 'SET_TEAM_LOGOS', value: results });

    return results;
}

Here is the structure of the data I'm getting:

{
  "team": {
    "teamId": "string",
    "abbr": "string",
    "logo": "string"
  },
  "teamPlayers": [
    {
      "firstName": "string",
      "lastName": "string",
      "esbId": "string",
      "position": "string",
      "jerseyNumber": 0,
      "teamAbbr": "string",
      "headshot": "string"
    }
  ]
}

It may be simpler if I get the team.logo and add it to each player in teamPlayers.

Body:
I figured it out but please tell me if you think there is a more efficient solution.

async function loadRosters(endPoints, teams, authKey) {
    axios.defaults.headers.common['Authorization'] = authKey;

    const requests = teams.map(teamId => {
        try {
            return axios.get(endPoints.roster + teamId).then(r => r.data);
        } catch (e) {
            //
        }
        return [];
    });

    const results = (await Promise.all(requests)).flat();
    dispatch({ type: 'SET_PLAYERS', value: results.map(r => r.teamPlayers).flat(Infinity) });
    dispatch({ type: 'SET_TEAM_LOGOS', value: results.map(r => r.team) });

    return results;
}


问:
Heroku Maven github deploy
I have some issue in heroku deploy</p>

```<code>2020-05-29T00:59:40.015516+00:00 app[web.1]: Error: Unable to access jarfile target/barbershop-0.0.1-SNAPSHOT.jar
2020-05-29T00:59:40.082046+00:00 heroku[web.1]: Process exited with status 1
2020-05-29T00:59:40.135944+00:00 heroku[web.1]: State changed from starting to crashed
2020-05-29T00:59:44.102979+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=powerful-taiga-48681.herokuapp.com request_id=759de18f-db81-47ca-8108-452945494b3d fwd="179.180.253.131" dyno= connect= service= status=503 bytes= protocol=https
</code>```

答:
Error: Unable to access jarfile target/barbershop-0.0.1-SNAPSHOT.jar

Did you check if target/barbershop-0.0.1-SNAPSHOT.jar is in your file system?

Title: Heroku Maven github deploy

Body:
I have some issue in heroku deploy

2020-05-29T00:59:40.015516+00:00 app[web.1]: Error: Unable to access jarfile target/barbershop-0.0.1-SNAPSHOT.jar
2020-05-29T00:59:40.082046+00:00 heroku[web.1]: Process exited with status 1
2020-05-29T00:59:40.135944+00:00 heroku[web.1]: State changed from starting to crashed
2020-05-29T00:59:44.102979+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=powerful-taiga-48681.herokuapp.com request_id=759de18f-db81-47ca-8108-452945494b3d fwd="179.180.253.131" dyno= connect= service= status=503 bytes= protocol=https


问:
Nextjs mix SSR and client data in components
I faced the situation when the same component shows mixed data when randomisation involved, html get jumbled data partially from SSR and partially from client rendering.

Here is the code :

```<code>const Component = (props) => {
    const rand = Math.random();
    console.log('==========================', rand);

    return <a href={rand}>{rand}</a>
}
</code>```

The result is following.

SSR :

```<code>========================== 0.30408232064749563
</code>```

Client rendering :

```<code>========================== 0.6842738761932372
</code>```

Result HTML :

```<code><a href="0.30408232064749563">0.6842738761932372</a>
</code>```

So the a tag get old SSR value in href while text value get updated.

答:
If you want to make the data identical in SSR and CSR, you should create Math.random() in getInitialProps and pass it by props.

```<code>const Component = props => {
  console.log(props.rand)
}

Component.getInitialProps = async () => {
  const rand = Math.random();
  console.log(rand);

  return {
    rand
  }
}
</code>```

Body:
I faced the situation when the same component shows mixed data when randomisation involved, html get jumbled data partially from SSR and partially from client rendering.

Here is the code :

const Component = (props) => {
    const rand = Math.random();
    console.log('==========================', rand);

    return {rand}
}

The result is following.

SSR :

========================== 0.30408232064749563

Client rendering :

========================== 0.6842738761932372

Result HTML :

0.6842738761932372

So the a tag get old SSR value in href while text value get updated.

Body:
If you want to make the data identical in SSR and CSR, you should create Math.random() in getInitialProps and pass it by props.

const Component = props => {
  console.log(props.rand)
}

Component.getInitialProps = async () => {
  const rand = Math.random();
  console.log(rand);

  return {
    rand
  }
}


问:
How to post assignment .io on github?
guys I'm supposed to upload assignment for this online course..
I followed the tutorial along exactly but something is not working properly.

https://zwc1625.github.io/coursera-web24/ is the where my repo is published and it's working fine.

but,
https://zwc1625.github.io/coursera-web24/mod_2/
where mod_2 is the assignment folder containing 1 html and 1 css file,
when I try to type in this address in the url 404 comes up

My local drive is up to date with the github repo

Help. New and very frustrated with github, I'm spending so much time with this and making no progress.

答:
Github will serve a few specifically named files automatically, like 'index.html', 'readme.md' or 'readme'. Most servers will look in the specified folder (https://zwc1625.github.io/coursera-web24/mod_2/) for files named: 'index' or 'default' with a specific extension like 'html', 'asp', 'xhtml'.

As the file in your folder 'mod_2' is called 'module2HTML.html' it will not get served automatically. Consequently, if you want to open that file in your browser, you will need to name it specifically in the URL you enter in the browser addressbar: https://zwc1625.github.io/coursera-web24/mod_2/module2HTML.html

Furthermore, when you did name your file 'index.html' it may be possible that there is a time lag between your 'commit' and Github being able to serve your page. Some patience may be required....

Title: How to post assignment .io on github?


问:
How can i convert UTF-8 text to octal in Python 3.7?
For example if i had to convert text to hex i can use

```<code>    import binascii
    x = input()
    print(str(binascii.b2a_hex(x.encode("utf-8")))[2:-2])
</code>```

But I cant do that for octal.

答:
It's a great question, my mind went to something like this</p>

```<code>x=[chr(x) for x in range(97,123)]
y=[oct(ord(i)) for i in x]
print(dict(zip(x,y)))
</code>```

Body:
For example if i had to convert text to hex i can use

    import binascii
    x = input()
    print(str(binascii.b2a_hex(x.encode("utf-8")))[2:-2])

But I cant do that for octal.

Body:
It's a great question, my mind went to something like this

x=[chr(x) for x in range(97,123)]
y=[oct(ord(i)) for i in x]
print(dict(zip(x,y)))


问:
How to dynamically add Nodes to VBox in ScalaFX
My ScalaFX app has a Button that adds a GridPane to a VBox. I tried directly updating the VBox children in Button.onAction but that causes the Button to become non-responsive after rendering the first added GridPane. I am now trying the following code:

```<code>
import scalafx.scene.{Node, Scene}
import scalafx.application.JFXApp
import scalafx.geometry.Insets
import scalafx.scene.control.{Button, Label}
import scalafx.scene.layout.{GridPane, VBox}
import scalafx.Includes._
import scalafx.collections.ObservableBuffer

object Gui extends JFXApp {

  stage = new JFXApp.PrimaryStage {
    title = "DSP Lab"
    scene = new Scene(320, 300) {

      val bufferOfNodes = new ObservableBuffer[Node]

      val vBox = new VBox {
        children = bufferOfNodes // assign ObservableBuffer to children
      }

      val addGridPaneButton = new Button {
        text = "Add GridPane to VBox"
        onAction = _ => {
          bufferOfNodes += getNewGridPane // update the ObservableBuffer
        }
      }

      content = List(addGridPaneButton, vBox)
    }
  }

  stage.setX(100)
  stage.setY(100)

  def getNewGridPane: GridPane = new GridPane {
    vgap = 5
    hgap = 10
    padding = Insets(20)

    val sampleFrequencyLabel = new Label("Sample Frequency:")
    val sampleFrequencyField = new DoubleField
    val amplitudeLabel = new Label("Amplitude:")
    val amplitudeField = new DoubleField
    add(sampleFrequencyLabel, 0, 0)
    add(sampleFrequencyField, 1, 0)
    add(amplitudeLabel, 0, 1)
    add(amplitudeField, 1, 1)
  }
}
</code>```

By assigning VBox.children to the ObservableBuffer[Node] the Button stays responsive and onAction adds the GridPane to bufferOfNodes but the GridPanes seem to never get added to the VBox and they are never rendered.

Is there a way to do this?

答:
This was asked and answered on ScalaFX-Users mailing list:
https://groups.google.com/forum/?utm_medium=email&utm_source=footer&pli=1#!topic/scalafx-users/3IzGAnflUnU</p>

Body:
My ScalaFX app has a Button that adds a GridPane to a VBox. I tried directly updating the VBox children in Button.onAction but that causes the Button to become non-responsive after rendering the first added GridPane. I am now trying the following code:

import scalafx.scene.{Node, Scene}
import scalafx.application.JFXApp
import scalafx.geometry.Insets
import scalafx.scene.control.{Button, Label}
import scalafx.scene.layout.{GridPane, VBox}
import scalafx.Includes._
import scalafx.collections.ObservableBuffer

object Gui extends JFXApp {

  stage = new JFXApp.PrimaryStage {
    title = "DSP Lab"
    scene = new Scene(320, 300) {

      val bufferOfNodes = new ObservableBuffer[Node]

      val vBox = new VBox {
        children = bufferOfNodes // assign ObservableBuffer to children
      }

      val addGridPaneButton = new Button {
        text = "Add GridPane to VBox"
        onAction = _ => {
          bufferOfNodes += getNewGridPane // update the ObservableBuffer
        }
      }

      content = List(addGridPaneButton, vBox)
    }
  }

  stage.setX(100)
  stage.setY(100)

  def getNewGridPane: GridPane = new GridPane {
    vgap = 5
    hgap = 10
    padding = Insets(20)

    val sampleFrequencyLabel = new Label("Sample Frequency:")
    val sampleFrequencyField = new DoubleField
    val amplitudeLabel = new Label("Amplitude:")
    val amplitudeField = new DoubleField
    add(sampleFrequencyLabel, 0, 0)
    add(sampleFrequencyField, 1, 0)
    add(amplitudeLabel, 0, 1)
    add(amplitudeField, 1, 1)
  }
}

By assigning VBox.children to the ObservableBuffer[Node] the Button stays responsive and onAction adds the GridPane to bufferOfNodes but the GridPanes seem to never get added to the VBox and they are never rendered.

Is there a way to do this?