问:
Event Sourcing with DynamoDB and Exactly Once Delivery, should I sort by AggregateId?
Hi friends of big data!

I am planning on implementing an event sourcing model with DynamoDB and Elixir for an internal simple messaging system. The purpose of this particular table in DynamoDB will be to log a meaningful history of events of messages (the payload will be stored elsewhere).

Here is my initial approach...

```<code>Domain/TeamId: Primary partition key, splits events by domain (different teams .etc.). This will avoid creating too many partitions.
EventId: Snowflake (RangeKey)
Timestamp: SomeTimestamp
ProfileId: String
MessageId: String
Type: String
AggregateId: String (LSI)
</code>```

The EventId will represent the version, and the AggregateId will initially represent the aggregate of messages per profile using profileId_<a_profile_id>_messageId_<a_message_id>. I'll fire events to the table like whether a message is read or deleted. I'll then use an aggregateId as the range key with = semantics to aggregate all of the events. This is to support a kind of running history of messages per profile.

The only side effect I can think of is there will be no guarantees on the order of events when querying for a particular aggregate. But if an aggregate is a reduce over all events, does this really matter?

Using this I plan to support exactly once delivery semantics by checking to see if the eventId has been updated for a particular team, and if it has check the aggregate again to make sure we're not adding two MessageRead events (we only want to track whether a message has been read once and when it was read). If somehow superfluous events end up, I'll handle this on the aggregate model.

Can I clarify whether this seems like an acceptable approach? It's difficult to come up with an exact title because this is a relatively nuanced and complex issue. Elixir doesn't exactly lend itself to perfect CQRS implementations, but I want to make sure I capture enough data to implement a canonical event sourcing model partitioned for our use case.

Please be advised I don't have control over the database technology or language!

Due to this constraint the idea of the aggregate key is just to make it easier to query less data, however I have to sacrifice sort order if the aggregateId isn't the partition key  because there's just nothing to range on, makes me dubious!

答:
This response is 1.5 years late. Hopefully you are enjoying your successful system, but I'll post this answer for future searchers.

Firstly, regarding naming of TeamId or Domain. I would go as generic as possible for this name. A successful event-sourcing system will be used for cases which you've never previously considered.  This is why most event-sourcing installations name this field "StreamId". I'll refer to it as this.

Monotonically increasing IDs or Versions in a Stream (or Domain, in your case) are important for any consumers of your events. If you have multiple writers to your event-stream, this becomes difficult.  Snowflake can generate monotonically increasing IDs, but only on a single process. So if you want monotonicity, then you need some sort of consensus.

Using this I plan to support exactly once delivery semantics by
  checking to see if the eventId has been updated for a particular team,
  and if it has check the aggregate again to make sure we're not adding
  two MessageRead events (we only want to track whether a message has
  been read once and when it was read).

Excellent. Presumably you're using Conditional Writes in dynamo to achieve this effect. If you are, then you are already achieving all of the consistency that you need to set those EventId values to anything that you want.  I would argue that you may as well set the ID's to increment by exactly 1 for every new record for a particular Stream.

Having increments of exactly 1 for a stream are important for 2 reasons:

Order probably does matter. Not for all cases, but for most
applications, you do need to process events in order.  If you're
counting, or performing other commutative operations, then maybe
not.  However, most business applications can't be expressed in
order-independent operationts. Again, if you're going through the
hassle of building this system, you might as well take some simple
steps to make it useful for a variety of cases.

Missed messages indicator.  If all of your services/actors are only reading from the dynamo table, this is less important.  However, if you have a distributed system with some sort of message delivery, then each actor can confidently build their own aggregations and projections if they can be sure that they are processing all messages (and only once)

Body:
Hi friends of big data!

I am planning on implementing an event sourcing model with DynamoDB and Elixir for an internal simple messaging system. The purpose of this particular table in DynamoDB will be to log a meaningful history of events of messages (the payload will be stored elsewhere).

Here is my initial approach...

Domain/TeamId: Primary partition key, splits events by domain (different teams .etc.). This will avoid creating too many partitions.
EventId: Snowflake (RangeKey)
Timestamp: SomeTimestamp
ProfileId: String
MessageId: String
Type: String
AggregateId: String (LSI)

The EventId will represent the version, and the AggregateId will initially represent the aggregate of messages per profile using profileId_<a_profile_id>_messageId_<a_message_id>. I'll fire events to the table like whether a message is read or deleted. I'll then use an aggregateId as the range key with = semantics to aggregate all of the events. This is to support a kind of running history of messages per profile.

The only side effect I can think of is there will be no guarantees on the order of events when querying for a particular aggregate. But if an aggregate is a reduce over all events, does this really matter?

Using this I plan to support exactly once delivery semantics by checking to see if the eventId has been updated for a particular team, and if it has check the aggregate again to make sure we're not adding two MessageRead events (we only want to track whether a message has been read once and when it was read). If somehow superfluous events end up, I'll handle this on the aggregate model.

Can I clarify whether this seems like an acceptable approach? It's difficult to come up with an exact title because this is a relatively nuanced and complex issue. Elixir doesn't exactly lend itself to perfect CQRS implementations, but I want to make sure I capture enough data to implement a canonical event sourcing model partitioned for our use case.

Please be advised I don't have control over the database technology or language!

Due to this constraint the idea of the aggregate key is just to make it easier to query less data, however I have to sacrifice sort order if the aggregateId isn't the partition key  because there's just nothing to range on, makes me dubious!


来源: stackexchange

问:
PyTorch model saving error: "Can't pickle local object"
When I try to save the PyTorch model with this piece of code:

```<code>checkpoint = {'model': Net(), 'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')
</code>```

I get the following error:

```<code>    E:\PROGRAM FILES\Anaconda\envs\staj_projesi\lib\site-packages\torch\serialization.py:251: UserWarning: Couldn't retrieve source code for container of type Net. It won't be checked for correctness upon loading.
...

      "type " + obj.__name__ + ". It won't be checked "
    Can't pickle local object 'trainModel.<locals>.Net'
</code>```

When I try to save the PyTorch model with this piece of code:

```<code>checkpoint = {'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')
</code>```

I don't don't get any errors, but I want to save the ANN class. How can I solve this problem? Also, I could save the model with the first structure in the other projects before

答:
You can't! torch.save is saving the objects state_dict() only.

When you use the following:

```<code>checkpoint = {'model': Net(), 'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')
</code>```

You are trying to save the model itself, but this data is saved in the model.state_dict() and when loading a model with the state_dict you should first initiate a model object.

This is exactly the reason why the second method works properly:

```<code>checkpoint = {'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')
</code>```

I would suggest reading the pytorch docs of how to properly save\load a model in the following link:
https://pytorch.org/tutorials/beginner/saving_loading_models.html

ContentLicense: CC BY-SA 4.0

Body:
When I try to save the PyTorch model with this piece of code:

checkpoint = {'model': Net(), 'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')

I get the following error:

    E:\PROGRAM FILES\Anaconda\envs\staj_projesi\lib\site-packages\torch\serialization.py:251: UserWarning: Couldn't retrieve source code for container of type Net. It won't be checked for correctness upon loading.
...

      "type " + obj.__name__ + ". It won't be checked "
    Can't pickle local object 'trainModel.<locals>.Net'

When I try to save the PyTorch model with this piece of code:

checkpoint = {'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')

I don't don't get any errors, but I want to save the ANN class. How can I solve this problem? Also, I could save the model with the first structure in the other projects before

Body:
You can't! torch.save is saving the objects state_dict() only.

When you use the following:

checkpoint = {'model': Net(), 'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')

You are trying to save the model itself, but this data is saved in the model.state_dict() and when loading a model with the state_dict you should first initiate a model object.

This is exactly the reason why the second method works properly:

checkpoint = {'state_dict': model.state_dict(),'optimizer' :optimizer.state_dict()}
torch.save(checkpoint, 'Checkpoint.pth')

I would suggest reading the pytorch docs of how to properly save\load a model in the following link:
https://pytorch.org/tutorials/beginner/saving_loading_models.html


问:
How do I use random numbers to generate truly random results? ABS(CHECKSUM(NewID))) is not working
Each time I run a query I need different results.

If I

```<code>SELECT AccountNumber
FROM TABLE
</code>```

with no other specifications, the table returns the records in order of account number by default (i have no control over this).

My solution (I thought) was to

```<code>SELECT ABS(CHECKSUM(NewID())) [RandomNumber], AccountNumber
INTO #tempTABLE
FROM TABLE
ORDER BY [RandomNumber]
</code>```

So a whole new table gets built ordered by the randomly generated numbers. Then I could just

```<code>SELECT TOP 1 AccountNumber
FROM #tempTABLE
</code>```

What I'm finding is there is a very high probability that the lowest random number will correlate with the lowest accountnumber. In short, this random number strategy is giving the same results very frequently. I need the results of my query to be a different, randomly selected account number each time i run it.

答:
You should add the order by to your (2nd) select query:</p>

```<code>SELECT TOP 1 AccountNumber
FROM #tempTABLE
ORDER BY [RandomNumber]
</code>```

Body:
Each time I run a query I need different results.

If I

SELECT AccountNumber
FROM TABLE

with no other specifications, the table returns the records in order of account number by default (i have no control over this).

My solution (I thought) was to

SELECT ABS(CHECKSUM(NewID())) [RandomNumber], AccountNumber
INTO #tempTABLE
FROM TABLE
ORDER BY [RandomNumber]

So a whole new table gets built ordered by the randomly generated numbers. Then I could just

SELECT TOP 1 AccountNumber
FROM #tempTABLE

What I'm finding is there is a very high probability that the lowest random number will correlate with the lowest accountnumber. In short, this random number strategy is giving the same results very frequently. I need the results of my query to be a different, randomly selected account number each time i run it.

Body:
You should add the order by to your (2nd) select query:

SELECT TOP 1 AccountNumber
FROM #tempTABLE
ORDER BY [RandomNumber]


问:
Keras fit and datagenerator return "Error when checking input: expected input_1 to have 3 dimensions, but got array with shape (None, 1)"
I am trying to create a generator using a dataframe stored in pickle. The targets are 3 columns whith their name starting by "target". The features are stored in the "input" column. Each cell of "input" are np.array. Their dimensions are 51x7 (7 is the number of channels).
(I am trying to just focus on the datagenerator for now)

```<code># General Libraries
import os
import math
import pickle
import numpy as np
import pandas as pd
from numpy import random

# Tensorflow and Keras
import tensorflow as tf
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Model
from tensorflow.keras.utils import Sequence
from tensorflow.keras.layers import Input, Conv1D, Dense, Dropout, MaxPooling1D, Flatten

# Project Libraries
import config

class DataGenerator(Sequence):
    def __init__(self, path_experiment, batch_size, mode='train', validation_ratio=0.1):
        self.path_pickle = os.path.join(path_experiment, 'dataframe.pkl')
        self.dataframe = self.load_dataframe()
        self.batch_size = batch_size
        self.mode = mode
        self.indexes = None

        if mode == 'validation':
            self.dataframe = self.dataframe[-int(len(self.dataframe) * validation_ratio):].copy()
        else:
            self.dataframe = self.dataframe[:-int(len(self.dataframe) * validation_ratio)].copy()

        self.name_targets = [col for col in self.dataframe if col.startswith('target')]
        self.dataframe[self.name_targets] = np.log(self.dataframe[self.name_targets])

        self.targets = self.dataframe[self.name_targets].values
        self.inputs = self.dataframe['input'].values

    def load_dataframe(self):
        with open(self.path_pickle, 'rb') as f:
            return pickle.load(f)

    def __len__(self):
        return int(math.ceil(len(self.dataframe) / float(self.batch_size)))

    def on_epoch_end(self):
        self.indexes = range(len(self.dataframe))
        if self.mode == 'train':
            # Shuffles indexes after each epoch if in training mode
            self.indexes = random.sample(self.indexes, k=len(self.indexes))

    def get_batch_labels(self, idx):
        # Fetch a batch of labels
        return self.targets[idx * self.batch_size: (idx + 1) * self.batch_size]

    def get_batch_features(self, idx):
        # Fetch a batch of inputs
        return self.inputs[idx * self.batch_size: (idx + 1) * self.batch_size]

    def __getitem__(self, idx):
        batch_x = self.get_batch_features(idx)
        batch_y = self.get_batch_labels(idx)
        # print(batch_x[0].shape)  # Display (51,7)
        # print(batch_x[0].shape)  # Display (3,)
        return batch_x, batch_y

path = os.path.join(config.path_model_input, 'experiment_2', '2')
train_generator = DataGenerator(path, batch_size=32, mode='train')
validation_generator = DataGenerator(path, batch_size=32, mode='validation')

input_shape = (51, 7)
i = Input(shape=input_shape)
x = Flatten()(i)
x = Dense(10, activation='relu')(x)
output = Dense(3, activation='linear')(x)
model = Model(inputs=i, outputs=output)

def RMSE(label, prediction):
    return tf.sqrt(tf.losses.mean_squared_error(label, prediction))

optimizer = Adam(lr=0.0001)
model.compile(optimizer=optimizer, loss=RMSE)
model.fit(train_generator, epochs=100, verbose=1, use_multiprocessing=False, workers=1)
</code>```

When I am trying to run this code I got this error :

```<code>---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-6dde9c267440> in <module>
      7 optimizer = Adam(lr=0.0001)
      8 model.compile(optimizer=optimizer, loss=RMSE)
----> 9 model.fit(train_generator, epochs=100, verbose=1, use_multiprocessing=False, workers=1)#, validation_data=validation_generator)

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    817         max_queue_size=max_queue_size,
    818         workers=workers,
--> 819         use_multiprocessing=use_multiprocessing)
    820
    821   def evaluate(self,

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    233           max_queue_size=max_queue_size,
    234           workers=workers,
--> 235           use_multiprocessing=use_multiprocessing)
    236
    237       total_samples = _get_total_number_of_samples(training_data_adapter)

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in _process_training_inputs(model, x, y, batch_size, epochs, sample_weights, class_weights, steps_per_epoch, validation_split, validation_data, validation_steps, shuffle, distribution_strategy, max_queue_size, workers, use_multiprocessing)
    591         max_queue_size=max_queue_size,
    592         workers=workers,
--> 593         use_multiprocessing=use_multiprocessing)
    594     val_adapter = None
    595     if validation_data:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in _process_inputs(model, mode, x, y, batch_size, epochs, sample_weights, class_weights, shuffle, steps, distribution_strategy, max_queue_size, workers, use_multiprocessing)
    704       max_queue_size=max_queue_size,
    705       workers=workers,
--> 706       use_multiprocessing=use_multiprocessing)
    707
    708   return adapter

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py in __init__(self, x, y, sample_weights, standardize_function, shuffle, workers, use_multiprocessing, max_queue_size, **kwargs)
    950         use_multiprocessing=use_multiprocessing,
    951         max_queue_size=max_queue_size,
--> 952         **kwargs)
    953
    954   @staticmethod

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py in __init__(self, x, y, sample_weights, standardize_function, workers, use_multiprocessing, max_queue_size, **kwargs)
    765
    766     if standardize_function is not None:
--> 767       dataset = standardize_function(dataset)
    768
    769     if kwargs.get("shuffle", False) and self.get_size() is not None:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in standardize_function(dataset)
    682           return x, y
    683         return x, y, sample_weights
--> 684       return dataset.map(map_fn, num_parallel_calls=dataset_ops.AUTOTUNE)
    685
    686   if mode == ModeKeys.PREDICT:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in map(self, map_func, num_parallel_calls)
   1589     else:
   1590       return ParallelMapDataset(
-> 1591           self, map_func, num_parallel_calls, preserve_cardinality=True)
   1592
   1593   def flat_map(self, map_func):

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in __init__(self, input_dataset, map_func, num_parallel_calls, use_inter_op_parallelism, preserve_cardinality, use_legacy_function)
   3924         self._transformation_name(),
   3925         dataset=input_dataset,
-> 3926         use_legacy_function=use_legacy_function)
   3927     self._num_parallel_calls = ops.convert_to_tensor(
   3928         num_parallel_calls, dtype=dtypes.int32, name="num_parallel_calls")

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in __init__(self, func, transformation_name, dataset, input_classes, input_shapes, input_types, input_structure, add_to_graph, use_legacy_function, defun_kwargs)
   3145       with tracking.resource_tracker_scope(resource_tracker):
   3146         # TODO(b/141462134): Switch to using garbage collection.
-> 3147         self._function = wrapper_fn._get_concrete_function_internal()
   3148
   3149         if add_to_graph:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _get_concrete_function_internal(self, *args, **kwargs)
   2393     """Bypasses error checking when getting a graph function."""
   2394     graph_function = self._get_concrete_function_internal_garbage_collected(
-> 2395         *args, **kwargs)
   2396     # We're returning this concrete function to someone, and they may keep a
   2397     # reference to the FuncGraph without keeping a reference to the

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2387       args, kwargs = None, None
   2388     with self._lock:
-> 2389       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2390     return graph_function
   2391

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _maybe_define_function(self, args, kwargs)
   2701
   2702       self._function_cache.missed.add(call_context_key)
-> 2703       graph_function = self._create_graph_function(args, kwargs)
   2704       self._function_cache.primary[cache_key] = graph_function
   2705       return graph_function, args, kwargs

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2591             arg_names=arg_names,
   2592             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2593             capture_by_value=self._capture_by_value),
   2594         self._function_attributes,
   2595         # Tell the ConcreteFunction to clean up its graph once it goes out of

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\framework\func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    976                                           converted_func)
    977
--> 978       func_outputs = python_func(*func_args, **func_kwargs)
    979
    980       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in wrapper_fn(*args)
   3138           attributes=defun_kwargs)
   3139       def wrapper_fn(*args):  # pylint: disable=missing-docstring
-> 3140         ret = _wrapper_helper(*args)
   3141         ret = structure.to_tensor_list(self._output_structure, ret)
   3142         return [ops.convert_to_tensor(t) for t in ret]

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in _wrapper_helper(*args)
   3080         nested_args = (nested_args,)
   3081
-> 3082       ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
   3083       # If `func` returns a list of tensors, `nest.flatten()` and
   3084       # `ops.convert_to_tensor()` would conspire to attempt to stack

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\autograph\impl\api.py in wrapper(*args, **kwargs)
    235       except Exception as e:  # pylint:disable=broad-except
    236         if hasattr(e, 'ag_error_metadata'):
--> 237           raise e.ag_error_metadata.to_exception(e)
    238         else:
    239           raise

ValueError: in converted code:

    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py:677 map_fn
        batch_size=None)
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training.py:2410 _standardize_tensors
        exception_prefix='input')
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_utils.py:573 standardize_input_data
        'with shape ' + str(data_shape))

    ValueError: Error when checking input: expected input_1 to have 3 dimensions, but got array with shape (None, 1)
</code>```

I passed quite some time on this issue and I can't seem to find the reason of this error, I checked several time all the dimensions of the inputs/outputs and I can't see the mistake that I made.
Any help would be much appreciated !

EDIT:
print(self.inputs.shape) and print(self.targets.shape) give (1101,) and (1101, 3) respectively.

I tried to do self.dataframe['input']=self.dataframe['input'].apply(lambda x: np.array(x)) before taking the values, but it still return (1101,) instead of (1101,57,3)

I also checked the shape of all the array in the column input:

```<code>train_generator.dataframe['input_shape']=train_generator.dataframe['input'].apply(lambda x: x.shape)
train_generator.dataframe['input_shape'].unique()
</code>```

It returns : array([(51, 7)], dtype=object)
I am not sure If it is normal that it returns "object" as type of the column

答:
The solution is in this thread: (for the details)

Convert pandas column of numpy arrays to numpy array of higher dimension

Turns out what I originally wanted to do with dataframe is not quite possible. Instead I used :
self.inputs = np.stack(self.dataframe['input'].to_numpy())

Body:
I am trying to create a generator using a dataframe stored in pickle. The targets are 3 columns whith their name starting by "target". The features are stored in the "input" column. Each cell of "input" are np.array. Their dimensions are 51x7 (7 is the number of channels).
(I am trying to just focus on the datagenerator for now)

# General Libraries
import os
import math
import pickle
import numpy as np
import pandas as pd
from numpy import random

# Tensorflow and Keras
import tensorflow as tf
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.models import Model
from tensorflow.keras.utils import Sequence
from tensorflow.keras.layers import Input, Conv1D, Dense, Dropout, MaxPooling1D, Flatten

# Project Libraries
import config

class DataGenerator(Sequence):
    def __init__(self, path_experiment, batch_size, mode='train', validation_ratio=0.1):
        self.path_pickle = os.path.join(path_experiment, 'dataframe.pkl')
        self.dataframe = self.load_dataframe()
        self.batch_size = batch_size
        self.mode = mode
        self.indexes = None

        if mode == 'validation':
            self.dataframe = self.dataframe[-int(len(self.dataframe) * validation_ratio):].copy()
        else:
            self.dataframe = self.dataframe[:-int(len(self.dataframe) * validation_ratio)].copy()

        self.name_targets = [col for col in self.dataframe if col.startswith('target')]
        self.dataframe[self.name_targets] = np.log(self.dataframe[self.name_targets])

        self.targets = self.dataframe[self.name_targets].values
        self.inputs = self.dataframe['input'].values

    def load_dataframe(self):
        with open(self.path_pickle, 'rb') as f:
            return pickle.load(f)

    def __len__(self):
        return int(math.ceil(len(self.dataframe) / float(self.batch_size)))

    def on_epoch_end(self):
        self.indexes = range(len(self.dataframe))
        if self.mode == 'train':
            # Shuffles indexes after each epoch if in training mode
            self.indexes = random.sample(self.indexes, k=len(self.indexes))

    def get_batch_labels(self, idx):
        # Fetch a batch of labels
        return self.targets[idx * self.batch_size: (idx + 1) * self.batch_size]

    def get_batch_features(self, idx):
        # Fetch a batch of inputs
        return self.inputs[idx * self.batch_size: (idx + 1) * self.batch_size]

    def __getitem__(self, idx):
        batch_x = self.get_batch_features(idx)
        batch_y = self.get_batch_labels(idx)
        # print(batch_x[0].shape)  # Display (51,7)
        # print(batch_x[0].shape)  # Display (3,)
        return batch_x, batch_y

path = os.path.join(config.path_model_input, 'experiment_2', '2')
train_generator = DataGenerator(path, batch_size=32, mode='train')
validation_generator = DataGenerator(path, batch_size=32, mode='validation')

input_shape = (51, 7)
i = Input(shape=input_shape)
x = Flatten()(i)
x = Dense(10, activation='relu')(x)
output = Dense(3, activation='linear')(x)
model = Model(inputs=i, outputs=output)

def RMSE(label, prediction):
    return tf.sqrt(tf.losses.mean_squared_error(label, prediction))

optimizer = Adam(lr=0.0001)
model.compile(optimizer=optimizer, loss=RMSE)
model.fit(train_generator, epochs=100, verbose=1, use_multiprocessing=False, workers=1)

When I am trying to run this code I got this error :

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-6dde9c267440> in <module>
      7 optimizer = Adam(lr=0.0001)
      8 model.compile(optimizer=optimizer, loss=RMSE)
----> 9 model.fit(train_generator, epochs=100, verbose=1, use_multiprocessing=False, workers=1)#, validation_data=validation_generator)

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    817         max_queue_size=max_queue_size,
    818         workers=workers,
--> 819         use_multiprocessing=use_multiprocessing)
    820
    821   def evaluate(self,

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in fit(self, model, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
    233           max_queue_size=max_queue_size,
    234           workers=workers,
--> 235           use_multiprocessing=use_multiprocessing)
    236
    237       total_samples = _get_total_number_of_samples(training_data_adapter)

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in _process_training_inputs(model, x, y, batch_size, epochs, sample_weights, class_weights, steps_per_epoch, validation_split, validation_data, validation_steps, shuffle, distribution_strategy, max_queue_size, workers, use_multiprocessing)
    591         max_queue_size=max_queue_size,
    592         workers=workers,
--> 593         use_multiprocessing=use_multiprocessing)
    594     val_adapter = None
    595     if validation_data:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in _process_inputs(model, mode, x, y, batch_size, epochs, sample_weights, class_weights, shuffle, steps, distribution_strategy, max_queue_size, workers, use_multiprocessing)
    704       max_queue_size=max_queue_size,
    705       workers=workers,
--> 706       use_multiprocessing=use_multiprocessing)
    707
    708   return adapter

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py in __init__(self, x, y, sample_weights, standardize_function, shuffle, workers, use_multiprocessing, max_queue_size, **kwargs)
    950         use_multiprocessing=use_multiprocessing,
    951         max_queue_size=max_queue_size,
--> 952         **kwargs)
    953
    954   @staticmethod

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\data_adapter.py in __init__(self, x, y, sample_weights, standardize_function, workers, use_multiprocessing, max_queue_size, **kwargs)
    765
    766     if standardize_function is not None:
--> 767       dataset = standardize_function(dataset)
    768
    769     if kwargs.get("shuffle", False) and self.get_size() is not None:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py in standardize_function(dataset)
    682           return x, y
    683         return x, y, sample_weights
--> 684       return dataset.map(map_fn, num_parallel_calls=dataset_ops.AUTOTUNE)
    685
    686   if mode == ModeKeys.PREDICT:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in map(self, map_func, num_parallel_calls)
   1589     else:
   1590       return ParallelMapDataset(
-> 1591           self, map_func, num_parallel_calls, preserve_cardinality=True)
   1592
   1593   def flat_map(self, map_func):

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in __init__(self, input_dataset, map_func, num_parallel_calls, use_inter_op_parallelism, preserve_cardinality, use_legacy_function)
   3924         self._transformation_name(),
   3925         dataset=input_dataset,
-> 3926         use_legacy_function=use_legacy_function)
   3927     self._num_parallel_calls = ops.convert_to_tensor(
   3928         num_parallel_calls, dtype=dtypes.int32, name="num_parallel_calls")

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in __init__(self, func, transformation_name, dataset, input_classes, input_shapes, input_types, input_structure, add_to_graph, use_legacy_function, defun_kwargs)
   3145       with tracking.resource_tracker_scope(resource_tracker):
   3146         # TODO(b/141462134): Switch to using garbage collection.
-> 3147         self._function = wrapper_fn._get_concrete_function_internal()
   3148
   3149         if add_to_graph:

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _get_concrete_function_internal(self, *args, **kwargs)
   2393     """Bypasses error checking when getting a graph function."""
   2394     graph_function = self._get_concrete_function_internal_garbage_collected(
-> 2395         *args, **kwargs)
   2396     # We're returning this concrete function to someone, and they may keep a
   2397     # reference to the FuncGraph without keeping a reference to the

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs)
   2387       args, kwargs = None, None
   2388     with self._lock:
-> 2389       graph_function, _, _ = self._maybe_define_function(args, kwargs)
   2390     return graph_function
   2391

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _maybe_define_function(self, args, kwargs)
   2701
   2702       self._function_cache.missed.add(call_context_key)
-> 2703       graph_function = self._create_graph_function(args, kwargs)
   2704       self._function_cache.primary[cache_key] = graph_function
   2705       return graph_function, args, kwargs

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\eager\function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes)
   2591             arg_names=arg_names,
   2592             override_flat_arg_shapes=override_flat_arg_shapes,
-> 2593             capture_by_value=self._capture_by_value),
   2594         self._function_attributes,
   2595         # Tell the ConcreteFunction to clean up its graph once it goes out of

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\framework\func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes)
    976                                           converted_func)
    977
--> 978       func_outputs = python_func(*func_args, **func_kwargs)
    979
    980       # invariant: `func_outputs` contains only Tensors, CompositeTensors,

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in wrapper_fn(*args)
   3138           attributes=defun_kwargs)
   3139       def wrapper_fn(*args):  # pylint: disable=missing-docstring
-> 3140         ret = _wrapper_helper(*args)
   3141         ret = structure.to_tensor_list(self._output_structure, ret)
   3142         return [ops.convert_to_tensor(t) for t in ret]

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\data\ops\dataset_ops.py in _wrapper_helper(*args)
   3080         nested_args = (nested_args,)
   3081
-> 3082       ret = autograph.tf_convert(func, ag_ctx)(*nested_args)
   3083       # If `func` returns a list of tensors, `nest.flatten()` and
   3084       # `ops.convert_to_tensor()` would conspire to attempt to stack

C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\autograph\impl\api.py in wrapper(*args, **kwargs)
    235       except Exception as e:  # pylint:disable=broad-except
    236         if hasattr(e, 'ag_error_metadata'):
--> 237           raise e.ag_error_metadata.to_exception(e)
    238         else:
    239           raise

ValueError: in converted code:

    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_v2.py:677 map_fn
        batch_size=None)
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training.py:2410 _standardize_tensors
        exception_prefix='input')
    C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\keras\engine\training_utils.py:573 standardize_input_data
        'with shape ' + str(data_shape))

    ValueError: Error when checking input: expected input_1 to have 3 dimensions, but got array with shape (None, 1)

I passed quite some time on this issue and I can't seem to find the reason of this error, I checked several time all the dimensions of the inputs/outputs and I can't see the mistake that I made.
Any help would be much appreciated !

EDIT:
print(self.inputs.shape) and print(self.targets.shape) give (1101,) and (1101, 3) respectively.

I tried to do self.dataframe['input']=self.dataframe['input'].apply(lambda x: np.array(x)) before taking the values, but it still return (1101,) instead of (1101,57,3)

I also checked the shape of all the array in the column input:

train_generator.dataframe['input_shape']=train_generator.dataframe['input'].apply(lambda x: x.shape)
train_generator.dataframe['input_shape'].unique()

It returns : array([(51, 7)], dtype=object)
I am not sure If it is normal that it returns "object" as type of the column


问:
Best way to go about creating in-house analytics for my Rails 3 app?
I have a Rails 3 app that I'm looking to create in-house analytics for. The items I need to track are impressions (and unique impressions), clicks that come from those impressions, and conversions that come from those clicks. And these are all user-specific so each user can see how many impressions, clicks, and conversions they've received.

What is the best way to go about this? Should I create a separate rails app and call it with pixels? Or should I include all the analytics code in the same app?

Also, are there any analytics platforms already out there that I can customize to meet my needs?

Thanks!

Tim

答:
Before you start re-inventing the wheel, Google Analytics provide a developer API (via OAuth, among other choices) that may provide you with the ability to do what you need (provide each user with a view of their own data).

http://code.google.com/apis/analytics/docs/gdata/home.html

Building your own, while it may seem like an initially basic thing to do, could have serious performance implications further down the line, and Google provide a very detailed view of the the data.

If you really want to write your own, I would strongly urge you not to hit the database for each request you want to track.  Keep the data in Redis, or one of the alternatives and periodically persist it to the database via a background task.

ContentLicense: CC BY-SA 3.0


问:
How to get return value from call to perl sub from bash script?
I am calling a perl script from a bash script. I want to obtain the return value that the perl script returns in the bash script in order to act on it. When I have the following, the output is a blank line to the console when I expect it to be "good".

bashThing.sh

```<code>#!/bin/bash

ARG="valid"
VAL=$(perl -I. -MperlThing -e "perlThing::isValid ${ARG}")
echo $VAL
</code>```

perlThing.pm

```<code>#! /usr/bin/perl -w
use strict;

package perlThing;

sub isValid
{
  my $arg = shift;
  if($arg == "valid")
  {
    return "good";
  }
  else
  {
    return "bad";
  }
}

1;
</code>```

答:
You didn't have Perl warnings enabled completely.  You would have seen several warnings.  I enabled warnings both in bash (using -w on the Perl one-liner), and in Perl with use warnings;.  The shebang line is ignored in the .pm file since it is not being executed as a script.

isValid returned a string, but you were ignoring the returned value.  To fix that, use print in the one-liner.

You also needed to pass the value to isValid properly.

Lastly, you need to use eq instead of == for string comparison.

bash:

```<code>#!/bin/bash

ARG="valid"
VAL=$(perl -w -I. -MperlThing -e "print perlThing::isValid(q(${ARG}))")
echo $VAL
</code>```

Perl:

```<code>use strict;
use warnings;

package perlThing;

sub isValid
{
  my $arg = shift;
  if($arg eq "valid")
  {
    return "good";
  }
  else
  {
    return "bad";
  }
}

1;
</code>```

Body:
I am calling a perl script from a bash script. I want to obtain the return value that the perl script returns in the bash script in order to act on it. When I have the following, the output is a blank line to the console when I expect it to be "good".

bashThing.sh

#!/bin/bash

ARG="valid"
VAL=$(perl -I. -MperlThing -e "perlThing::isValid ${ARG}")
echo $VAL

perlThing.pm

#! /usr/bin/perl -w
use strict;

package perlThing;

sub isValid
{
  my $arg = shift;
  if($arg == "valid")
  {
    return "good";
  }
  else
  {
    return "bad";
  }
}

1;

Body:
You didn't have Perl warnings enabled completely.  You would have seen several warnings.  I enabled warnings both in bash (using -w on the Perl one-liner), and in Perl with use warnings;.  The shebang line is ignored in the .pm file since it is not being executed as a script.

isValid returned a string, but you were ignoring the returned value.  To fix that, use print in the one-liner.

You also needed to pass the value to isValid properly.

Lastly, you need to use eq instead of == for string comparison.

bash:

#!/bin/bash

ARG="valid"
VAL=$(perl -w -I. -MperlThing -e "print perlThing::isValid(q(${ARG}))")
echo $VAL

Perl:

use strict;
use warnings;

package perlThing;

sub isValid
{
  my $arg = shift;
  if($arg eq "valid")
  {
    return "good";
  }
  else
  {
    return "bad";
  }
}

1;


问:
(errno: 150 "Foreign key constraint is incorrectly formed") in many-to-many relationship in Laravel
I'm try to create many-to-many that link with customer and shop in laravel but stuck in this error (errno: 150 "Foreign key constraint is incorrectly formed") and still, not figure it out.

Here my customers table

```<code>Schema::create('customers', function (Blueprint $table) {
  $table->bigIncrements('id');
  $table->unsignedBigInteger('shop_id');
  $table->string('name');
  $table->string('email');
  $table->string('phone');
  $table->timestamps();

  $table->foreign('shop_id')->references('id')->on('shop');
});
</code>```

Here my shops table

```<code>Schema::create('shops', function (Blueprint $table) {
  $table->bigIncrements('id');
  $table->unsignedBigInteger('customer_id');
  $table->string('name');
  $table->timestamps();

  $table->foreign('customer_id')->references('id')->on('customer');
});
</code>```

My Shop Model

```<code>protected $fillable = ['name'];

public function customer()
{
  return $this->belongsToMany(\App\Customer::class);
}
</code>```

My Customer Model

```<code>protected $fillable = ['name', 'email', 'phone'];

public function shop()
{
  return $this->belongsToMany(\App\Shop::class);
}
</code>```

Any Help? Thanks in advances....

答:
The relation of many-to-many need third table pivot table which you miss it.

for do that create new migration.

php artisan make:migration create_customer_shop

there is no need create model for pivot table

then you Schema pivot table something like this.

Pivot table

```<code>Schema::create('cutomer_shop', function (Blueprint $table) {
  $table->increments('id');
  $table->integer('customer_id')->unsigned();
  $table->integer('shop_id')->unsigned();
  $table->timestamps();

  $table->foreign('customer_id')->references('id')->on('customers');
  $table->foreign('shop_id')->references('id')->on('shops');
});
</code>```

The two table shops and customers doesn't have any direct relations they have relation by just pivot table.

Note: make sure all three table id type increments('id') and all foreign key are $table->integer('shop_id')->unsigned(); otherwise give you incorrect format error.

Body:
I'm try to create many-to-many that link with customer and shop in laravel but stuck in this error (errno: 150 "Foreign key constraint is incorrectly formed") and still, not figure it out.

Here my customers table

Schema::create('customers', function (Blueprint $table) {
  $table->bigIncrements('id');
  $table->unsignedBigInteger('shop_id');
  $table->string('name');
  $table->string('email');
  $table->string('phone');
  $table->timestamps();

  $table->foreign('shop_id')->references('id')->on('shop');
});

Here my shops table

Schema::create('shops', function (Blueprint $table) {
  $table->bigIncrements('id');
  $table->unsignedBigInteger('customer_id');
  $table->string('name');
  $table->timestamps();

  $table->foreign('customer_id')->references('id')->on('customer');
});

My Shop Model

protected $fillable = ['name'];

public function customer()
{
  return $this->belongsToMany(\App\Customer::class);
}

My Customer Model

protected $fillable = ['name', 'email', 'phone'];

public function shop()
{
  return $this->belongsToMany(\App\Shop::class);
}

Any Help? Thanks in advances....

Body:
The relation of many-to-many need third table pivot table which you miss it.

for do that create new migration.

php artisan make:migration create_customer_shop

there is no need create model for pivot table

then you Schema pivot table something like this.

Pivot table

Schema::create('cutomer_shop', function (Blueprint $table) {
  $table->increments('id');
  $table->integer('customer_id')->unsigned();
  $table->integer('shop_id')->unsigned();
  $table->timestamps();

  $table->foreign('customer_id')->references('id')->on('customers');
  $table->foreign('shop_id')->references('id')->on('shops');
});

The two table shops and customers doesn't have any direct relations they have relation by just pivot table.

Note: make sure all three table id type increments('id') and all foreign key are $table->integer('shop_id')->unsigned(); otherwise give you incorrect format error.


问:
Select some rows from a XML file with Python
I have an XML that i want to export some rows, base in attribute 1 content.

My File is like that:

```<code><Root>
  <First_Level_Node>
    <Element_Name>
      <attribute1>1</attribute1>
      <attribute2>2</attribute2>
      <attribute3>3</attribute3>
      <attribute4>4</attribute4>
      <attribute5>5</attribute5>
      <attribute6>6</attribute6>
    </Element_Name>
    <Element_Name>
      <attribute1>41</attribute1>
      <attribute2></attribute2>
      <attribute3>NO</attribute3>
      <attribute4>42</attribute4>
      <attribute5>random value</attribute5>
      <attribute6>18th Jun 2014  07:09:18 GMT</attribute6>
    </Element_Name>
    <Element_Name>
      <attribute1>42</attribute1>
      <attribute2></attribute2>
      <attribute3>NO</attribute3>
      <attribute4>42</attribute4>
      <attribute5>random</attribute5>
      <attribute6>23rd Jul 2014  02:47:10 GMT</attribute6>
    </Element_Name>
    <Element_Name>
      <attribute1>43</attribute1>
      <attribute2></attribute2>
      <attribute3>NO</attribute3>
      <attribute4>42</attribute4>
      <attribute5>random</attribute5>
      <attribute6>08th Nov 2014  23:53:31 GMT</attribute6>
    </Element_Name>
  </First_Level_Node>
</Root>
</code>```

And based on the value of the attribute 1, I want to get all the rows from that element.
For example, I want attribute 1 = 42 or attribute 1 = 43, so the output will be like:

```<code><Root>
  <First_Level_Node>
    <Element_Name>
      <attribute1>42</attribute1>
      <attribute2></attribute2>
      <attribute3>NO</attribute3>
      <attribute4>42</attribute4>
      <attribute5>random</attribute5>
      <attribute6>23rd Jul 2014  02:47:10 GMT</attribute6>
    </Element_Name>
    <Element_Name>
      <attribute1>43</attribute1>
      <attribute2></attribute2>
      <attribute3>NO</attribute3>
      <attribute4>42</attribute4>
      <attribute5>random</attribute5>
      <attribute6>08th Nov 2014  23:53:31 GMT</attribute6>
    </Element_Name>
  </First_Level_Node>
</Root>
</code>```

I want to use Python for this.

I tried to use ElementTree, but was unsuccessful. That's what I tried so far

```<code>with open("output.xml", 'r') as f3:
        root = ET.parse(f3)

    tree=root.getroot()
    with open("list_xxx", 'r') as f2:
        for line in f2:
            line = line.rstrip()
            print (line)
            #get all subnet nodes containing certain ManagedElementId
            subnet_path = './/*[attribute1="{0}"]'
            subnet_path = subnet_path.format(line)
            for subnet in tree.findall(subnet_path):
                #reconstruct subnet node:
                parent = ET.Element(subnet.tag, attrib=subnet.attrib)
                #path to find all ManagedElement containing certain ManagedElementId
                content_path = ".//*[attribute1='{0}']".format(line)
                print (content_path)

                #append all ManagedElement found to the new subnet:
                for content in subnet.findall(content_path):
                    parent.append(content)
                #print new subnet:
                print (ET.tostring(parent))
</code>```

And list xxx has the values of attribute 1 that I am looking for:

```<code>41
42
</code>```

答:
Using lxml and xpath should make it much simpler:

```<code>from lxml import etree

dat = """[your xml snippet above]"""
doc = etree.fromstring(dat)

targets = doc.xpath('//Element_Name[not(attribute1="42" or attribute1="43")]')
#the xpath expression above selects all ElementName nodes which do NOT meet your requirements
for target in targets:
    target.getparent().remove(target) #once an unnecessary element is selected, you go up to the parent and remove the unnecessary element
print(etree.tostring(doc).decode())
</code>```

Output is your desire output.

Body:
I have an XML that i want to export some rows, base in attribute 1 content.

My File is like that:

<Root>
  <First_Level_Node>
    <Element_Name>
      <attribute1>1</attribute1>
      <attribute2>2</attribute2>
      <attribute3>3</attribute3>
      <attribute4>4</attribute4>
      <attribute5>5</attribute5>
      <attribute6>6</attribute6>
    </Element_Name>
    <Element_Name>
      <attribute1>41</attribute1>
      <attribute2></attribute2>
      <attribute3>NO</attribute3>
      <attribute4>42</attribute4>
      <attribute5>random value</attribute5>
      <attribute6>18th Jun 2014  07:09:18 GMT</attribute6>
    </Element_Name>
    <Element_Name>
      <attribute1>42</attribute1>
      <attribute2></attribute2>
      <attribute3>NO</attribute3>
      <attribute4>42</attribute4>
      <attribute5>random</attribute5>
      <attribute6>23rd Jul 2014  02:47:10 GMT</attribute6>
    </Element_Name>
    <Element_Name>
      <attribute1>43</attribute1>
      <attribute2></attribute2>
      <attribute3>NO</attribute3>
      <attribute4>42</attribute4>
      <attribute5>random</attribute5>
      <attribute6>08th Nov 2014  23:53:31 GMT</attribute6>
    </Element_Name>
  </First_Level_Node>
</Root>

And based on the value of the attribute 1, I want to get all the rows from that element.
For example, I want attribute 1 = 42 or attribute 1 = 43, so the output will be like:

<Root>
  <First_Level_Node>
    <Element_Name>
      <attribute1>42</attribute1>
      <attribute2></attribute2>
      <attribute3>NO</attribute3>
      <attribute4>42</attribute4>
      <attribute5>random</attribute5>
      <attribute6>23rd Jul 2014  02:47:10 GMT</attribute6>
    </Element_Name>
    <Element_Name>
      <attribute1>43</attribute1>
      <attribute2></attribute2>
      <attribute3>NO</attribute3>
      <attribute4>42</attribute4>
      <attribute5>random</attribute5>
      <attribute6>08th Nov 2014  23:53:31 GMT</attribute6>
    </Element_Name>
  </First_Level_Node>
</Root>

I want to use Python for this.

I tried to use ElementTree, but was unsuccessful. That's what I tried so far

with open("output.xml", 'r') as f3:
        root = ET.parse(f3)

    tree=root.getroot()
    with open("list_xxx", 'r') as f2:
        for line in f2:
            line = line.rstrip()
            print (line)
            #get all subnet nodes containing certain ManagedElementId
            subnet_path = './/*[attribute1="{0}"]'
            subnet_path = subnet_path.format(line)
            for subnet in tree.findall(subnet_path):
                #reconstruct subnet node:
                parent = ET.Element(subnet.tag, attrib=subnet.attrib)
                #path to find all ManagedElement containing certain ManagedElementId
                content_path = ".//*[attribute1='{0}']".format(line)
                print (content_path)

                #append all ManagedElement found to the new subnet:
                for content in subnet.findall(content_path):
                    parent.append(content)
                #print new subnet:
                print (ET.tostring(parent))

And list xxx has the values of attribute 1 that I am looking for:

41
42

Body:
Using lxml and xpath should make it much simpler:

from lxml import etree

dat = """[your xml snippet above]"""
doc = etree.fromstring(dat)

targets = doc.xpath('//Element_Name[not(attribute1="42" or attribute1="43")]')
#the xpath expression above selects all ElementName nodes which do NOT meet your requirements
for target in targets:
    target.getparent().remove(target) #once an unnecessary element is selected, you go up to the parent and remove the unnecessary element
print(etree.tostring(doc).decode())

Output is your desire output.


问:
Fabric Debug Craslytic Reports : Signup, build Id missing, apply plugin : io.fabric
We have programming an Android app and try to implement Crashlytics to our app.

We have different types of problem .
Version we used :
Android studio version :
3.3

Gradle version :
classpath 'com.android.tools.build:gradle:3.3.1'

Plugin :
Fabric for Android studio v4.3.0

Implementation :
implementation('com.crashlytics.sdk.android:crashlytics:2.9.9') { transitive = true }
implementation('io.fabric.sdk.android:fabric:1.4.0@aar') { transitive = true }

gradle-wrapper.properties :
 distributionUrl=https://services.gradle.org/distributions/gradle-5.2.1-all.zip

First Problem :

When we implement Crashlytics, you know that developers have 3 steps. We can not skip 2,3.steps. Because we have not compiled our application yet. We had two main errors:

Error 1:

This app relies on Crashlytics. Please sign up for access at https://fabric.io/sign_up,

Error 2 :

```<code>E/CrashlyticsCore: The Crashlytics build ID is missing.This occurs when
Crashlytics tooling is absent from your app's build configuration.
Please review Crashlytics onboarding instructions and ensure you have a valid Crashlytics account.
</code>```

So, we had to be disabled "Debug Mod" to complete implementation of Crashlytic.

```<code>Crashlytics crashlyticsKit = new Crashlytics.Builder()
        .core(new CrashlyticsCore.Builder().disabled(BuildConfig.DEBUG).build())
        .build();

Fabric.with(this, crashlyticsKit);  // Tod from Fabric suggested in stackoverflow
</code>```

And implementation was completed.

But we do not want to do that. Because, when we have a crash, it does not  any report to Crashlytics or Firebase. We also want to have debug mod’s crashes.

When we removed -> ....disabled(BuildConfig.DEBUG)  -  it shows again : Error 1, Error 2.

Second Problem :

In gradle ; apply plugin : ‘io.fabric’, we made the comment line, when we remove comments line,  we have errors below :

Error 3 :

```<code>WARNING: API 'variant.getExternalNativeBuildTasks()' is obsolete and has been replaced with 'variant.getExternalNativeBuildProviders()'.
It will be removed at the end of 2019.
For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
To determine what is calling variant.getExternalNativeBuildTasks(), use -Pandroid.debug.obsoleteApi=true on the command line to display a stack trace.
Affected Modules: app
</code>```

When we searched it, this error related to new android studio gradle. So we needed to make comment line “apply plugin: fabric.io” again.

There is no good solution about that.

To run application we can not remove:

new CrashlyticsCore.Builder().disabled(BuildConfig.DEBUG).build() : (due to Error1  Error2)

So we try to enable report different ways :

What have we try to add until here?

1. Enable in gradle :

```<code>buildTypes {
    debug {
        manifestPlaceholders = [crashlyticsEnabled: true]
        minifyEnabled false
        proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
    }
    release {
        minifyEnabled false
        proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        manifestPlaceholders = [crashlyticsEnabled: false]

    }
</code>```

2.    Enable in Manifest

```<code><meta-data
    android:name="firebase_crashlytics_collection_enabled"
    android:value="true" />
</code>```

3.Enable in ADB

```<code>  adb shell setprop log.tag.Fabric DEBUG
  adb shell setprop log.tag.CrashlyticsCore DEBUG
</code>```

But still, Crashlytics or Firebase does not get any Debug Crash reports.

We have expecting your solutions.

答:
In my case I was missing Crashlytics plugin:

```<code>apply plugin: 'com.google.firebase.crashlytics'
</code>```

Put that line at the top of your module's build.gradle file e.g.

```<code>apply plugin: 'com.android.application'
apply plugin: 'com.google.firebase.crashlytics'
</code>```

Body:
We have programming an Android app and try to implement Crashlytics to our app.

We have different types of problem .
Version we used :
Android studio version :
3.3

Gradle version :
classpath 'com.android.tools.build:gradle:3.3.1'

Plugin :
Fabric for Android studio v4.3.0

Implementation :
implementation('com.crashlytics.sdk.android:crashlytics:2.9.9') { transitive = true }
implementation('io.fabric.sdk.android:fabric:1.4.0@aar') { transitive = true }

gradle-wrapper.properties :
 distributionUrl=https://services.gradle.org/distributions/gradle-5.2.1-all.zip

First Problem :

When we implement Crashlytics, you know that developers have 3 steps. We can not skip 2,3.steps. Because we have not compiled our application yet. We had two main errors:

Error 1:

This app relies on Crashlytics. Please sign up for access at https://fabric.io/sign_up,

Error 2 :

E/CrashlyticsCore: The Crashlytics build ID is missing.This occurs when
Crashlytics tooling is absent from your app's build configuration.
Please review Crashlytics onboarding instructions and ensure you have a valid Crashlytics account.

So, we had to be disabled "Debug Mod" to complete implementation of Crashlytic.

Crashlytics crashlyticsKit = new Crashlytics.Builder()
        .core(new CrashlyticsCore.Builder().disabled(BuildConfig.DEBUG).build())
        .build();

Fabric.with(this, crashlyticsKit);  // Tod from Fabric suggested in stackoverflow

And implementation was completed.

But we do not want to do that. Because, when we have a crash, it does not  any report to Crashlytics or Firebase. We also want to have debug mod’s crashes.

When we removed -> ....disabled(BuildConfig.DEBUG)  -  it shows again : Error 1, Error 2.

Second Problem :

In gradle ; apply plugin : ‘io.fabric’, we made the comment line, when we remove comments line,  we have errors below :

Error 3 :

WARNING: API 'variant.getExternalNativeBuildTasks()' is obsolete and has been replaced with 'variant.getExternalNativeBuildProviders()'.
It will be removed at the end of 2019.
For more information, see https://d.android.com/r/tools/task-configuration-avoidance.
To determine what is calling variant.getExternalNativeBuildTasks(), use -Pandroid.debug.obsoleteApi=true on the command line to display a stack trace.
Affected Modules: app

When we searched it, this error related to new android studio gradle. So we needed to make comment line “apply plugin: fabric.io” again.

There is no good solution about that.

To run application we can not remove:

new CrashlyticsCore.Builder().disabled(BuildConfig.DEBUG).build() : (due to Error1  Error2)

So we try to enable report different ways :

What have we try to add until here?

1. Enable in gradle :

buildTypes {
    debug {
        manifestPlaceholders = [crashlyticsEnabled: true]
        minifyEnabled false
        proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
    }
    release {
        minifyEnabled false
        proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        manifestPlaceholders = [crashlyticsEnabled: false]

    }

2.    Enable in Manifest

<meta-data
    android:name="firebase_crashlytics_collection_enabled"
    android:value="true" />

3.Enable in ADB

  adb shell setprop log.tag.Fabric DEBUG
  adb shell setprop log.tag.CrashlyticsCore DEBUG

But still, Crashlytics or Firebase does not get any Debug Crash reports.

We have expecting your solutions.

Body:
In my case I was missing Crashlytics plugin:

apply plugin: 'com.google.firebase.crashlytics'

Put that line at the top of your module's build.gradle file e.g.

apply plugin: 'com.android.application'
apply plugin: 'com.google.firebase.crashlytics'


问:
Vue this.$refs is undefined during mounted() in vue2-dropzone component
I am currently trying to make sure the my componentvue2dropzone is disabled on the initial load of the application, then i began to notice this error in the console when I am trying to make the app mobile responsive.

[Vue warn]: Error in mounted hook: "TypeError: this.$refs.vue2dropzone is undefined"

followed by a

TypeError: "this.$refs.vue2dropzone is undefined"

I have a noticed this problem occurring in Vue that seems to be a common issue based on my research, such that when you have v-if or v-else-if in a component child modifying to show something, the lifecycle will be called again.

A snippet of my code for vue2dropzone component just includes

In my typescript file

```<code>private mounted() {
  this.$refs.vue2dropzone.disable();
}
</code>```

In my vue file

```<code>    <div v-if="some condition">
     <form>
       <vue2Dropzone
       ref="vue2dropzone"
       id="dropzone"
       class="vue-dropzone"
       ></vue2Dropzone>
     </form>
    </div>
    <div v-else-if="some other condition">
     <form>
       <vue2Dropzone
       ref="vue2dropzone"
       id="dropzone"
       class="vue-dropzone"
       ></vue2Dropzone>
     </form>
   </div>
</code>```

I am beginning to wonder if the v-show being before the first v-if doesn't let recall the lifecycle because when I am on the web app, it is disabled but when i switch to mobile responsiveness it shows up as undefined.

Are there any solutions to solve this? I have seen v-show being used and this.$nextTick but it really didn't help me in my case. Should i put this in another part of the lifecycle besides mounted()?

答:
ok so I have found a method to resolve this. It seems as the vue app is loading it is quite possible that since the v-if is there the condition is not fulfilled until other asynchronous processes have fully loaded which more than likely has past the mounted lifecycle.

In order to solve this we can use a native feature of vue2-dropzone which is vdropzone-mounted so whenever it is being toggled or loads, the function runs.

e.g.

Create a function for using the disable function of vue2-dropzone

```<code>public disableDropZoneBox() {
  this.$refs.vue2dropzone.disable();
}
</code>```

in vue file

```<code> <vue2Dropzone
 ref="vue2dropzone"
 id="dropzone"
 class="vue-dropzone"
 @vdropzone-mounted="disableDropZoneBox"
 ></vue2Dropzone>
</code>```

Body:
I am currently trying to make sure the my componentvue2dropzone is disabled on the initial load of the application, then i began to notice this error in the console when I am trying to make the app mobile responsive.

[Vue warn]: Error in mounted hook: "TypeError: this.$refs.vue2dropzone is undefined"

followed by a

TypeError: "this.$refs.vue2dropzone is undefined"

I have a noticed this problem occurring in Vue that seems to be a common issue based on my research, such that when you have v-if or v-else-if in a component child modifying to show something, the lifecycle will be called again.

A snippet of my code for vue2dropzone component just includes

In my typescript file

private mounted() {
  this.$refs.vue2dropzone.disable();
}

In my vue file

     <form>
       <vue2Dropzone
       ref="vue2dropzone"
       id="dropzone"
       class="vue-dropzone"
       ></vue2Dropzone>
     </form>

     <form>
       <vue2Dropzone
       ref="vue2dropzone"
       id="dropzone"
       class="vue-dropzone"
       ></vue2Dropzone>
     </form>

I am beginning to wonder if the v-show being before the first v-if doesn't let recall the lifecycle because when I am on the web app, it is disabled but when i switch to mobile responsiveness it shows up as undefined.

Are there any solutions to solve this? I have seen v-show being used and this.$nextTick but it really didn't help me in my case. Should i put this in another part of the lifecycle besides mounted()?

Body:
ok so I have found a method to resolve this. It seems as the vue app is loading it is quite possible that since the v-if is there the condition is not fulfilled until other asynchronous processes have fully loaded which more than likely has past the mounted lifecycle.

In order to solve this we can use a native feature of vue2-dropzone which is vdropzone-mounted so whenever it is being toggled or loads, the function runs.

e.g.

Create a function for using the disable function of vue2-dropzone

public disableDropZoneBox() {
  this.$refs.vue2dropzone.disable();
}

in vue file

 <vue2Dropzone
 ref="vue2dropzone"
 id="dropzone"
 class="vue-dropzone"
 @vdropzone-mounted="disableDropZoneBox"
 ></vue2Dropzone>


问:
How to get info on a process's virtual memory mappings in C (MacOS)
I want to get information on the virtual memory layout of a process (stack, libc, heap, and code) for my current process in C (MacOS).

I read that in Linux you can do this by looking at /proc/<pid>/maps and that you can do this on the MacOS command line via the vmmap command line utility.

How can I do this in C? If possible I would like to get all pages in the virtual memory space pertaining to my process.

答:
You can use the mach_vm_region_info APIs to query this information from C. Here is an example from Jonathan Levin's book MacOS and iOS Internals:

```<code>#include <mach/vm_map.h>
#include <stdio.h>

#include <mach-o/dyld_images.h>

/**
 * vmmap(1) clone for OS X and iOS
 * -------------------------------
 *
 * This is a simple example of using the mach_vm_region_info APIs in order to
 * obtain a process' (technically, a task's) virtual memory address space, in a
 * manner akin to /proc/[pid]/maps on Linux.
 *
 * The process is simple - get the task port, then call mach_vm_region_info until
 * you've exhausted the address space (in iOS this happens around 0x40000000,
 * where the commpage is). On iOS 6, for some peculiar reason the task port is
 * invalidated after each call, so the quick workaround here solves the problem
 * by regetting the port. The actual mach error code to check for is in the header
 * files, though the code simply tries regetting.
 *
 * N.B - For this code to work, you MUST provide the entitlements to allow
 * task-for-pid to work, else you'll fail with error 5. The entitlements are in
 * the output in Chapter 3, but for those of you who haven't bought the book, it would be:
 *
--- Cut here

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>com.apple.springboard.debugapplications</key>
        <true/>
        <key>get-task-allow</key>
        <true/>
        <key>proc_info-allow</key>
        <true/>
        <key>task_for_pid-allow</key>
        <true/>
        <key>run-unsigned-code</key>
        <true/>
</dict>
</plist>

--- Ok, enough :-)
 *
 *  so - copy the above XML to a file, say, "ent.xml", and be sure to run "ldid -Sent.xml vmmap"
 *  before trying to run this. You can download the binary (already thus signed) if you're lazy
 *  (and trust me, because you *will* need root on your i-Device for this)
 *
 *  As the book clearly states, once you have the task port, the world is your oyster. You can
 *  control the entire virtual memory space, reading and writing it as you please. Stay tuned
 *  for the corrupt tool (which will be provided soon in binary form)
 *
 */

int g_pid = 0; // required in iOS 6 (read below)

 /* 03/08/13 - Added List of Mach-O images: */

struct dyld_image_info *g_dii = NULL;
int     g_imageCount;

unsigned char *
readProcessMemory (int pid, mach_vm_address_t addr, mach_msg_type_number_t *size)
{
    // Helper function to read process memory (a la Win32 API of same name)
    // To make it easier for inclusion elsewhere, it takes a pid, and
    // does the task_for_pid by itself. Given that iOS invalidates task ports
    // after use, it's actually a good idea, since we'd need to reget anyway

    task_t  t;
    task_for_pid(mach_task_self(),pid, &t);
        mach_msg_type_number_t  dataCnt = size;
        vm_offset_t readMem;

    // Use vm_read, rather than mach_vm_read, since the latter is different
    // in iOS.

        kern_return_t kr = vm_read(t,        // vm_map_t target_task,
                     addr,     // mach_vm_address_t address,
                     *size,     // mach_vm_size_t size
                     &readMem,     //vm_offset_t *data,
                     size);     // mach_msg_type_number_t *dataCnt

        if (kr) {
                // DANG..
                fprintf (stderr, "Unable to read target task's memory @%p - kr 0x%x\n" , addr, kr);
                 return NULL;
                }

    return ( (unsigned char *) readMem);

}

kern_return_t mach_vm_read (vm_map_t, mach_vm_address_t, mach_vm_size_t, vm_offset_t *, mach_msg_type_number_t *);
void
findListOfBinaries(task_t  t, mach_vm_address_t    addr, int size)
{

        kern_return_t kr;
        mach_msg_type_number_t  dataCnt = size;

    unsigned char *readData = readProcessMemory (g_pid, addr, &dataCnt);

    int machsig = 0xfeedface;

    // Checking only 0xfeedfa is a simple way to catch both 64-bit (facf) and 32-bit (face) headers
        // Machine endianness is automatically taken care of, too..

    if (readData && memcmp (readData + 1, ((unsigned char *) &machsig) + 1 , 3) == 0)
    {
        // This is a Mach header
        int i = 0;

        // A MUCH better way would be to iterate through the LC and find the name of dyld
        // but this would require my machlib.c (closed source) and really get the same result.
        // This works because on both iOS and OS X dyld is at /usr/lib.

        for (i = 0; i <dataCnt; i++)
        {
            if (memcmp(readData+i, "lib/dyld", 8) == 0)
            {
                unsigned int dyld_all_image_infos_offset ;
                int imageCount = 0;

                memcpy (&dyld_all_image_infos_offset, readData+DYLD_ALL_IMAGE_INFOS_OFFSET_OFFSET, sizeof (unsigned int));

                 struct dyld_all_image_infos *dyldaii ;
                 // Safeguard: should check that dyld_all_image_infos_offset is < size..

                if (dyld_all_image_infos_offset > size)
                {
                    // This is to be expected, since the dyld_all_image_infos is in a data region

                    //printf ("Offset %x is greater than region size : %x\n", dyld_all_image_infos_offset, size);
                    dataCnt = sizeof(dyld_all_image_infos);
                    readData = readProcessMemory (g_pid, addr + dyld_all_image_infos_offset , &dataCnt);

                    if (!readData) { return;}
                    dyldaii = (struct dyld_all_image_infos *) readData;

                }
                else
                {
                    dyldaii = (struct dyld_all_image_infos *) (readData +dyld_all_image_infos_offset);
                }

                printf ("Version: %d, %d images at offset %p\n",
                        dyldaii->version, dyldaii->infoArrayCount, dyldaii->infoArray);

                // Go to dyldaii->infoArray address

                imageCount = dyldaii->infoArrayCount;
                dataCnt = imageCount * sizeof(struct dyld_image_info);
                g_dii = (struct dyld_image_info *) malloc (dataCnt);
                g_imageCount = imageCount;
                readData = readProcessMemory(g_pid, dyldaii->infoArray, &dataCnt);
                if (!readData) { return;}

                struct dyld_image_info *dii = (struct dyld_image_info *) readData;

                // We don't need i anymore, anyway
                for (i = 0; i < imageCount; i++)
                    {
                        dataCnt = 1024;
                        char *imageName = readProcessMemory (g_pid, dii[i].imageFilePath, &dataCnt);
                        if (imageName) g_dii[i].imageFilePath = strdup(imageName);
                        else g_dii[i].imageFilePath = NULL;
                        g_dii[i].imageLoadAddress = dii[i].imageLoadAddress;
                    }

                break;
            }

        }
    }

}

 /* End 03/08/13 */
char *
behavior_to_text (vm_behavior_t b)
{

  switch (b)
    {
        case VM_BEHAVIOR_DEFAULT: return("default");
        case VM_BEHAVIOR_RANDOM:  return("random");
        case VM_BEHAVIOR_SEQUENTIAL: return("fwd-seq");
        case VM_BEHAVIOR_RSEQNTL: return("rev-seq");
        case VM_BEHAVIOR_WILLNEED: return("will-need");
        case VM_BEHAVIOR_DONTNEED: return("will-need");
        case VM_BEHAVIOR_FREE: return("free-nowb");
        case VM_BEHAVIOR_ZERO_WIRED_PAGES: return("zero-wire");
        case VM_BEHAVIOR_REUSABLE: return("reusable");
        case VM_BEHAVIOR_REUSE: return("reuse");
        case VM_BEHAVIOR_CAN_REUSE: return("canreuse");
        default: return ("?");
    }

}
char *
protection_bits_to_rwx (vm_prot_t p)
{

  // previous version of this somehow lost the "p&", always returning rwx..
  static char returned[4];

  returned[0] = (p &VM_PROT_READ    ? 'r' : '-');
  returned[1] = (p &VM_PROT_WRITE   ? 'w' : '-');
  returned[2] = (p & VM_PROT_EXECUTE ? 'x' : '-');
  returned[3] = '\0';

 // memory leak here. No biggy
  return (strdup(returned));

}

const char *
unparse_inheritance (vm_inherit_t i)
{
  switch (i)
    {
    case VM_INHERIT_SHARE:
      return "share";
    case VM_INHERIT_COPY:
      return "copy";
    case VM_INHERIT_NONE:
      return "none";
    default:
      return "???";
    }
}

macosx_debug_regions (task_t task, mach_vm_address_t address, int max)
{
  kern_return_t kret;

  mach_vm_address_t prev_address;
  /* @TODO: warning - potential overflow here - gotta fix this.. */
  vm_region_basic_info_data_t prev_info,info;
  mach_vm_size_t size, prev_size;

  mach_port_t object_name;
  mach_msg_type_number_t count;

  int nsubregions = 0;
  int num_printed = 0;

  count = VM_REGION_BASIC_INFO_COUNT_64;
  kret = mach_vm_region (task, &address, &size, VM_REGION_BASIC_INFO,
             (vm_region_info_t) &info, &count, &object_name);

  if (kret != KERN_SUCCESS)
    {
      printf ("mach_vm_region: Error %d - %s", kret, mach_error_string(kret));
      return;
    }
  memcpy (&prev_info, &info, sizeof (vm_region_basic_info_data_t));
  prev_address = address;
  prev_size = size;
  nsubregions = 1;

  for (;;)
    {
      int print = 0;
      int done = 0;

      address = prev_address + prev_size;

      /* Check to see if address space has wrapped around. */
      if (address == 0)
    {
        print = done = 1;
    }

      if (!done)
        {
          // Even on iOS, we use VM_REGION_BASIC_INFO_COUNT_64. This works.

          count = VM_REGION_BASIC_INFO_COUNT_64;

          kret =
            mach_vm_region (task, &address, &size, VM_REGION_BASIC_INFO,
                          (vm_region_info_t) &info, &count, &object_name);

          if (kret != KERN_SUCCESS)
            {
        /* iOS 6 workaround - attempt to reget the task port to avoiD */
        /* "(ipc/send) invalid destination port" (1000003 or something) */
        task_for_pid(mach_task_self(),g_pid, &task);

        kret =
            mach_vm_region (task, &address, &size, VM_REGION_BASIC_INFO,
                              (vm_region_info_t) &info, &count, &object_name);

        }
       if (kret != KERN_SUCCESS)
    {
        fprintf (stderr,"mach_vm_region failed for address %p - Error: %x\n", address,(kret));
              size = 0;
    if (address >= 0x4000000) return;
              print = done = 1;
            }
        }

      if (address != prev_address + prev_size)
        print = 1;

      if ((info.protection != prev_info.protection)
          || (info.max_protection != prev_info.max_protection)
          || (info.inheritance != prev_info.inheritance)
          || (info.shared != prev_info.reserved)
          || (info.reserved != prev_info.reserved))
        print = 1;

      if (print)
        {
      int   print_size;
      char *print_size_unit;
          if (num_printed == 0)

            printf ("Region ");
          else
            printf ("   ... ");

       findListOfBinaries(task, prev_address, prev_size);
      /* Quick hack to show size of segment, which GDB does not */
      print_size = prev_size;
      if (print_size > 1024) { print_size /= 1024; print_size_unit = "K"; }
      if (print_size > 1024) { print_size /= 1024; print_size_unit = "M"; }
      if (print_size > 1024) { print_size /= 1024; print_size_unit = "G"; }
      /* End Quick hack */
          printf (" %p-%p [%d%s](%s/%s; %s, %s, %s) %s",
                           (prev_address),
                           (prev_address + prev_size),
               print_size,
               print_size_unit,
                           protection_bits_to_rwx (prev_info.protection),
                           protection_bits_to_rwx (prev_info.max_protection),
                           unparse_inheritance (prev_info.inheritance),
                           prev_info.shared ? "shared" : "private",
                           prev_info.reserved ? "reserved" : "not-reserved",
               behavior_to_text (prev_info.behavior));

          if (nsubregions > 1)
            printf (" (%d sub-regions)", nsubregions);

          printf ("\n");

          prev_address = address;
          prev_size = size;
          memcpy (&prev_info, &info, sizeof (vm_region_basic_info_data_t));
          nsubregions = 1;

          num_printed++;
        }
      else
        {
          prev_size += size;
          nsubregions++;
        }

      if ((max > 0) && (num_printed >= max))
    {
     printf ("Max %d num_printed %d\n", max, num_printed);
        done = 1;
    }

      if (done)
        break;
    }
}

void
main(int argc, char **argv)
{

    struct vm_region_basic_info vmr;
    kern_return_t   rc;
    mach_port_t task;

    mach_vm_size_t  size = 8;
    vm_region_info_t    info = (vm_region_info_t) malloc(10000);
    mach_msg_type_number_t  info_count;
    mach_port_t     object_name;
    mach_vm_address_t   addr =1;
    int pid;

        if (!argv[1]) { printf ("Usage: %s <PID>\n"); exit (1);}
    pid = atoi(argv[1]);
    g_pid = pid; // req for iOS 6
    rc = task_for_pid(mach_task_self(),pid, &task);

    if (rc) { fprintf (stderr, "task_for_pid() failed with error %d - %s\n", rc, mach_error_string(rc)); exit(1); }
    printf ("RC %d - Task: %d\n",rc, task);

    macosx_debug_regions (task, addr, 1000);

    int i ;

    for ( i = 0; i < g_imageCount; i++)
    {
        printf("Image: %s loaded @%p\n",
            g_dii[i].imageFilePath, g_dii[i].imageLoadAddress);
    }
    printf("Done\n");

}
</code>```

I also found Julia Evans' blog post about recreating vmmap in Rust interesting and relevant to your question.

Another way you could do this is to call system with vmmap PID > file as the argument and then read the process mapping information from the file after system finishes.

Body:
You can use the mach_vm_region_info APIs to query this information from C. Here is an example from Jonathan Levin's book MacOS and iOS Internals:

#include <mach/vm_map.h>
#include <stdio.h>

#include <mach-o/dyld_images.h>

/**
 * vmmap(1) clone for OS X and iOS
 * -------------------------------
 *
 * This is a simple example of using the mach_vm_region_info APIs in order to
 * obtain a process' (technically, a task's) virtual memory address space, in a
 * manner akin to /proc/[pid]/maps on Linux.
 *
 * The process is simple - get the task port, then call mach_vm_region_info until
 * you've exhausted the address space (in iOS this happens around 0x40000000,
 * where the commpage is). On iOS 6, for some peculiar reason the task port is
 * invalidated after each call, so the quick workaround here solves the problem
 * by regetting the port. The actual mach error code to check for is in the header
 * files, though the code simply tries regetting.
 *
 * N.B - For this code to work, you MUST provide the entitlements to allow
 * task-for-pid to work, else you'll fail with error 5. The entitlements are in
 * the output in Chapter 3, but for those of you who haven't bought the book, it would be:
 *
--- Cut here

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
        <key>com.apple.springboard.debugapplications</key>
        <true/>
        <key>get-task-allow</key>
        <true/>
        <key>proc_info-allow</key>
        <true/>
        <key>task_for_pid-allow</key>
        <true/>
        <key>run-unsigned-code</key>
        <true/>
</dict>
</plist>

--- Ok, enough :-)
 *
 *  so - copy the above XML to a file, say, "ent.xml", and be sure to run "ldid -Sent.xml vmmap"
 *  before trying to run this. You can download the binary (already thus signed) if you're lazy
 *  (and trust me, because you *will* need root on your i-Device for this)
 *
 *  As the book clearly states, once you have the task port, the world is your oyster. You can
 *  control the entire virtual memory space, reading and writing it as you please. Stay tuned
 *  for the corrupt tool (which will be provided soon in binary form)
 *
 */

int g_pid = 0; // required in iOS 6 (read below)

 /* 03/08/13 - Added List of Mach-O images: */

struct dyld_image_info *g_dii = NULL;
int     g_imageCount;

unsigned char *
readProcessMemory (int pid, mach_vm_address_t addr, mach_msg_type_number_t *size)
{
    // Helper function to read process memory (a la Win32 API of same name)
    // To make it easier for inclusion elsewhere, it takes a pid, and
    // does the task_for_pid by itself. Given that iOS invalidates task ports
    // after use, it's actually a good idea, since we'd need to reget anyway

    task_t  t;
    task_for_pid(mach_task_self(),pid, &t);
        mach_msg_type_number_t  dataCnt = size;
        vm_offset_t readMem;

    // Use vm_read, rather than mach_vm_read, since the latter is different
    // in iOS.

        kern_return_t kr = vm_read(t,        // vm_map_t target_task,
                     addr,     // mach_vm_address_t address,
                     *size,     // mach_vm_size_t size
                     &readMem,     //vm_offset_t *data,
                     size);     // mach_msg_type_number_t *dataCnt

        if (kr) {
                // DANG..
                fprintf (stderr, "Unable to read target task's memory @%p - kr 0x%x\n" , addr, kr);
                 return NULL;
                }

    return ( (unsigned char *) readMem);

}

kern_return_t mach_vm_read (vm_map_t, mach_vm_address_t, mach_vm_size_t, vm_offset_t *, mach_msg_type_number_t *);
void
findListOfBinaries(task_t  t, mach_vm_address_t    addr, int size)
{

        kern_return_t kr;
        mach_msg_type_number_t  dataCnt = size;

    unsigned char *readData = readProcessMemory (g_pid, addr, &dataCnt);

    int machsig = 0xfeedface;

    // Checking only 0xfeedfa is a simple way to catch both 64-bit (facf) and 32-bit (face) headers
        // Machine endianness is automatically taken care of, too..

    if (readData && memcmp (readData + 1, ((unsigned char *) &machsig) + 1 , 3) == 0)
    {
        // This is a Mach header
        int i = 0;

        // A MUCH better way would be to iterate through the LC and find the name of dyld
        // but this would require my machlib.c (closed source) and really get the same result.
        // This works because on both iOS and OS X dyld is at /usr/lib.

        for (i = 0; i <dataCnt; i++)
        {
            if (memcmp(readData+i, "lib/dyld", 8) == 0)
            {
                unsigned int dyld_all_image_infos_offset ;
                int imageCount = 0;

                memcpy (&dyld_all_image_infos_offset, readData+DYLD_ALL_IMAGE_INFOS_OFFSET_OFFSET, sizeof (unsigned int));

                 struct dyld_all_image_infos *dyldaii ;
                 // Safeguard: should check that dyld_all_image_infos_offset is < size..

                if (dyld_all_image_infos_offset > size)
                {
                    // This is to be expected, since the dyld_all_image_infos is in a data region

                    //printf ("Offset %x is greater than region size : %x\n", dyld_all_image_infos_offset, size);
                    dataCnt = sizeof(dyld_all_image_infos);
                    readData = readProcessMemory (g_pid, addr + dyld_all_image_infos_offset , &dataCnt);

                    if (!readData) { return;}
                    dyldaii = (struct dyld_all_image_infos *) readData;

                }
                else
                {
                    dyldaii = (struct dyld_all_image_infos *) (readData +dyld_all_image_infos_offset);
                }

                printf ("Version: %d, %d images at offset %p\n",
                        dyldaii->version, dyldaii->infoArrayCount, dyldaii->infoArray);

                // Go to dyldaii->infoArray address

                imageCount = dyldaii->infoArrayCount;
                dataCnt = imageCount * sizeof(struct dyld_image_info);
                g_dii = (struct dyld_image_info *) malloc (dataCnt);
                g_imageCount = imageCount;
                readData = readProcessMemory(g_pid, dyldaii->infoArray, &dataCnt);
                if (!readData) { return;}

                struct dyld_image_info *dii = (struct dyld_image_info *) readData;

                // We don't need i anymore, anyway
                for (i = 0; i < imageCount; i++)
                    {
                        dataCnt = 1024;
                        char *imageName = readProcessMemory (g_pid, dii[i].imageFilePath, &dataCnt);
                        if (imageName) g_dii[i].imageFilePath = strdup(imageName);
                        else g_dii[i].imageFilePath = NULL;
                        g_dii[i].imageLoadAddress = dii[i].imageLoadAddress;
                    }

                break;
            }

        }
    }

}

 /* End 03/08/13 */
char *
behavior_to_text (vm_behavior_t b)
{

  switch (b)
    {
        case VM_BEHAVIOR_DEFAULT: return("default");
        case VM_BEHAVIOR_RANDOM:  return("random");
        case VM_BEHAVIOR_SEQUENTIAL: return("fwd-seq");
        case VM_BEHAVIOR_RSEQNTL: return("rev-seq");
        case VM_BEHAVIOR_WILLNEED: return("will-need");
        case VM_BEHAVIOR_DONTNEED: return("will-need");
        case VM_BEHAVIOR_FREE: return("free-nowb");
        case VM_BEHAVIOR_ZERO_WIRED_PAGES: return("zero-wire");
        case VM_BEHAVIOR_REUSABLE: return("reusable");
        case VM_BEHAVIOR_REUSE: return("reuse");
        case VM_BEHAVIOR_CAN_REUSE: return("canreuse");
        default: return ("?");
    }

}
char *
protection_bits_to_rwx (vm_prot_t p)
{

  // previous version of this somehow lost the "p&", always returning rwx..
  static char returned[4];

  returned[0] = (p &VM_PROT_READ    ? 'r' : '-');
  returned[1] = (p &VM_PROT_WRITE   ? 'w' : '-');
  returned[2] = (p & VM_PROT_EXECUTE ? 'x' : '-');
  returned[3] = '\0';

 // memory leak here. No biggy
  return (strdup(returned));

}

const char *
unparse_inheritance (vm_inherit_t i)
{
  switch (i)
    {
    case VM_INHERIT_SHARE:
      return "share";
    case VM_INHERIT_COPY:
      return "copy";
    case VM_INHERIT_NONE:
      return "none";
    default:
      return "???";
    }
}

macosx_debug_regions (task_t task, mach_vm_address_t address, int max)
{
  kern_return_t kret;

  mach_vm_address_t prev_address;
  /* @TODO: warning - potential overflow here - gotta fix this.. */
  vm_region_basic_info_data_t prev_info,info;
  mach_vm_size_t size, prev_size;

  mach_port_t object_name;
  mach_msg_type_number_t count;

  int nsubregions = 0;
  int num_printed = 0;

  count = VM_REGION_BASIC_INFO_COUNT_64;
  kret = mach_vm_region (task, &address, &size, VM_REGION_BASIC_INFO,
             (vm_region_info_t) &info, &count, &object_name);

  if (kret != KERN_SUCCESS)
    {
      printf ("mach_vm_region: Error %d - %s", kret, mach_error_string(kret));
      return;
    }
  memcpy (&prev_info, &info, sizeof (vm_region_basic_info_data_t));
  prev_address = address;
  prev_size = size;
  nsubregions = 1;

  for (;;)
    {
      int print = 0;
      int done = 0;

      address = prev_address + prev_size;

      /* Check to see if address space has wrapped around. */
      if (address == 0)
    {
        print = done = 1;
    }

      if (!done)
        {
          // Even on iOS, we use VM_REGION_BASIC_INFO_COUNT_64. This works.

          count = VM_REGION_BASIC_INFO_COUNT_64;

          kret =
            mach_vm_region (task, &address, &size, VM_REGION_BASIC_INFO,
                          (vm_region_info_t) &info, &count, &object_name);

          if (kret != KERN_SUCCESS)
            {
        /* iOS 6 workaround - attempt to reget the task port to avoiD */
        /* "(ipc/send) invalid destination port" (1000003 or something) */
        task_for_pid(mach_task_self(),g_pid, &task);

        kret =
            mach_vm_region (task, &address, &size, VM_REGION_BASIC_INFO,
                              (vm_region_info_t) &info, &count, &object_name);

        }
       if (kret != KERN_SUCCESS)
    {
        fprintf (stderr,"mach_vm_region failed for address %p - Error: %x\n", address,(kret));
              size = 0;
    if (address >= 0x4000000) return;
              print = done = 1;
            }
        }

      if (address != prev_address + prev_size)
        print = 1;

      if ((info.protection != prev_info.protection)
          || (info.max_protection != prev_info.max_protection)
          || (info.inheritance != prev_info.inheritance)
          || (info.shared != prev_info.reserved)
          || (info.reserved != prev_info.reserved))
        print = 1;

      if (print)
        {
      int   print_size;
      char *print_size_unit;
          if (num_printed == 0)

            printf ("Region ");
          else
            printf ("   ... ");

       findListOfBinaries(task, prev_address, prev_size);
      /* Quick hack to show size of segment, which GDB does not */
      print_size = prev_size;
      if (print_size > 1024) { print_size /= 1024; print_size_unit = "K"; }
      if (print_size > 1024) { print_size /= 1024; print_size_unit = "M"; }
      if (print_size > 1024) { print_size /= 1024; print_size_unit = "G"; }
      /* End Quick hack */
          printf (" %p-%p [%d%s](%s/%s; %s, %s, %s) %s",
                           (prev_address),
                           (prev_address + prev_size),
               print_size,
               print_size_unit,
                           protection_bits_to_rwx (prev_info.protection),
                           protection_bits_to_rwx (prev_info.max_protection),
                           unparse_inheritance (prev_info.inheritance),
                           prev_info.shared ? "shared" : "private",
                           prev_info.reserved ? "reserved" : "not-reserved",
               behavior_to_text (prev_info.behavior));

          if (nsubregions > 1)
            printf (" (%d sub-regions)", nsubregions);

          printf ("\n");

          prev_address = address;
          prev_size = size;
          memcpy (&prev_info, &info, sizeof (vm_region_basic_info_data_t));
          nsubregions = 1;

          num_printed++;
        }
      else
        {
          prev_size += size;
          nsubregions++;
        }

      if ((max > 0) && (num_printed >= max))
    {
     printf ("Max %d num_printed %d\n", max, num_printed);
        done = 1;
    }

      if (done)
        break;
    }
}

void
main(int argc, char **argv)
{

    struct vm_region_basic_info vmr;
    kern_return_t   rc;
    mach_port_t task;

    mach_vm_size_t  size = 8;
    vm_region_info_t    info = (vm_region_info_t) malloc(10000);
    mach_msg_type_number_t  info_count;
    mach_port_t     object_name;
    mach_vm_address_t   addr =1;
    int pid;

        if (!argv[1]) { printf ("Usage: %s <PID>\n"); exit (1);}
    pid = atoi(argv[1]);
    g_pid = pid; // req for iOS 6
    rc = task_for_pid(mach_task_self(),pid, &task);

    if (rc) { fprintf (stderr, "task_for_pid() failed with error %d - %s\n", rc, mach_error_string(rc)); exit(1); }
    printf ("RC %d - Task: %d\n",rc, task);

    macosx_debug_regions (task, addr, 1000);

    int i ;

    for ( i = 0; i < g_imageCount; i++)
    {
        printf("Image: %s loaded @%p\n",
            g_dii[i].imageFilePath, g_dii[i].imageLoadAddress);
    }
    printf("Done\n");

}

I also found Julia Evans' blog post about recreating vmmap in Rust interesting and relevant to your question.

Another way you could do this is to call system with vmmap PID > file as the argument and then read the process mapping information from the file after system finishes.