resource\_exhausted with no description and overloaded nodes
hey all!

just a few questions around resource exhausted. we are using the daml on vmware ledger. key features of this ledger being a topology with participant nodes, where there can also exist participant nodes which are identical replicas (for fault-tolerance purposes). these participant nodes are…
i suspect that the resource\_exhausted message may have originated from a back-pressuring committer node ([code](https://github.com/digital-asset/daml/blob/926fb5997a795e7f80443a9a3ba7898b1568d158/ledger/participant-integration-api/src/main/scala/platform/apiserver/services/apisubmissionservice.scala#l155-l157)). if that’s the case, i believe the approach should be the same: change the configuration if you need and make sure to handle those errors with the appropriate retry strategy.
Here is a sample of traversing a simple xml structure

 
```
    Dim myxml = <root>
                    <level id="1">
                        <prop n="PROPERTY" v="VALUE"/>
                    </level>
                    <level id="2">
                        <prop n="PROPERTY" v="VALUE"/>
                    </level>
                </root>

    For Each level In myxml.Elements
        Debug.WriteLine("id=" & level.@id.ToString)
        For Each prop In level.Elements
            Debug.WriteLine("n=" & prop.@n.ToString & "  v=" & prop.@v.ToString)
        Next
    Next

``` Dapper just wraps raw ADO.NET; raw ADO.NET doesn't offer a facility for this, therefore dapper does not. What you want is `SqlBulkCopy`. You could *also* use a table-valued-parameter, but this really feels like a `SqlBulkCopy` job.


In a pinch, you can *use* dapper here - `Execute` will unroll an `IEnumerable<T>` into a series of commands about `T` - but it will be lots of commands; and unless you explicitly enable async-pipelining, it will suffer from latency per-command (the pipelined mode avoids this, but it will still be n commands). But `SqlBulkCopy` will be *much* more efficient.


If the input data is an `IEnumerable<T>`, you might want to use `ObjectReader` from FastMember; for example:



```
IEnumerable<SomeType> data = ...
using(var bcp = new SqlBulkCopy(connection))
using(var reader = ObjectReader.Create(data, "Id", "Name", "Description"))
{
    bcp.DestinationTableName = "SomeTable";
    bcp.WriteToServer(reader);
}

``` do you have transport rules in place that are used for moderation?  sound like one is in place and isn't properly configured. 
emc > organization configuration > hub transport > transport rules From the explore section you can see the resourcce which are linked with existing table(profile etc).

To see a resource which are not linked , you can select resource from the properties section of query activity

![1402857_pastedImage_0.png](/t5/image/serverpage/image-id/9513i93F570CE23A95886/image-size/large?v=v2&px=999 "1402857_pastedImage_0.png")




[View solution in original post](/t5/adobe-campaign-standard/query-with-custom-resources-in-acs/m-p/246365#M8688) A couple of notes to get started:



> 
> .apply(literal\_eval)
> 
> 
> 


Wouldn't this be better as `map`?



> 
> I then re-create the Dask DF:
> 
> 
> ddf = dd.from\_pandas(ddf\_out, npartitions=nCores)
> 
> 
> 


`ddf_out` was already a dask dataframe, I don't know why you had to do this.



> 
> The columns in result seem to match those in metadir.
> 
> 
> 


The value of `result.columns` is taken from the meta you provided, no computing happens until you ask for it (dask is lazy in most operations). Does the ValueError exception not provide further information?


Here is a full example



```
x = ({'F9_07_PZ_COMP_DIRECT': '0',
  'F9_07_PZ_DIRTRSTKEY_NAME': 'DEBRA MEALY',
  'F9_07_PZ_COMP_OTHER': '0',
  'F9_07_PZ_COMP_RELATED': '0',
  'F9_07_PZ_TITLE': 'CHAIR PERSON',
  'F9_07_PZ_AVE_HOURS_WEEK': '1.00',
  'F9_07_PC_TRUSTEE_INDIVIDUAL': 'X'},
 {'F9_07_PZ_COMP_DIRECT': '0',
  'F9_07_PZ_DIRTRSTKEY_NAME': 'HELEN GORDON',
  'F9_07_PZ_COMP_OTHER': '0',
  'F9_07_PZ_COMP_RELATED': '0',
  'F9_07_PZ_TITLE': 'VICE CHAIR',
  'F9_07_PZ_AVE_HOURS_WEEK': '1.00',
  'F9_07_PC_TRUSTEE_INDIVIDUAL': 'X'})
df = pd.DataFrame({'a': x})
d = dd.from_pandas(df, 1)
meta = pd.DataFrame(columns=['F9_07_PZ_COMP_DIRECT', 
       'F9_07_PZ_DIRTRSTKEY_NAME',
       'F9_07_PZ_COMP_OTHER', 'F9_07_PZ_COMP_RELATED', 'F9_07_PZ_TITLE',
       'F9_07_PZ_AVE_HOURS_WEEK', 'F9_07_PC_TRUSTEE_INDIVIDUAL'], dtype="O")
d.map_partitions(lambda df: df.a.apply(pd.Series), meta=meta).compute()

```

How did I know what `meta` to use? I applied the function to the pandas dataframe - you might use a small piece of the dataframe to do this.


Some additional notes:


* it is an anti-pattern to load the data with pandas, pass to dask workers and then gather the whole result back to a pandas (in memory) dataframe, you are very unlikely to see a speedup this way and may incur a lot of overhead. You are better off loading with something like `dd.read_csv`, and aggregating or writing with dask functions too. Only `compute()` on something that will be small or not return anything (because it involves writing the output). The official examples do not use from\_pandas.
* string and dict processing are python methods, and so hold the interpreter lock (GIL) of any python function: threads will not actually run in parallel. To get parallelism, you need to run in processes, which is most easily achieved using <https://docs.dask.org/en/latest/setup/single-distributed.html>
* the distributed scheduler also gives you access to the dashboard, which has a lot of useful information to diagnose how your system is running. You can also configure a lot about its behaviour, in case you have firewall rules you need to obey. You could look at LINQ to XML. I use it all the time to read XML files and parse them.



<http://msdn.microsoft.com/en-us/library/bb387098.aspx>



If this is for an assignment, I'll leave the actual DAL building up to you, but LINQ to XML will provide you the mechanisms for reading the XML file and generating objects you require. see [creating feedback loops](https://vvvv.org/documentation/creating-feedback-loops).

it’s only the subtype of a datatype. don’t worry about that.

fyi: create an [asstring (value)](https://vvvv.org/documentation/asstring-(value)) and check its sub type for possible subtypes of the datatype “value” performance has always been an issue with dynamo, when working with large data sets, mainly due to the functional way in which a dynamo graph operates. you have to remember that data is never mutated. instead, it is always copied between nodes.

when you input something into a node, dynamo’s backend… > 
> how can I detect when these threads are done with their job and dispose them of their resources?
> 
> 
> 


I think the `ThreadState` property will tell you want you need to know. I just posted an example of this over in a similar thread about JTable [here](/software-development/java/threads/373078/1605358#post1605358). Perhaps it will help.