Dask to Flatten Dictionary Column




I am new to Dask and am looking to find a way to flatten a dictionary column in a PANDAS dataframe. Here is a screenshot of the first row of a 16 million-row dataframe:


[![screenshot of first two rows of data](https://i.stack.imgur.com/xLIuC.png)](https://i.stack.imgur.com/xLIuC.png)


And here is a sample of the text from three rows: 



```
{{u'F9_07_PZ_COMP_DIRECT': u'0', u'F9_07_PZ_DIRTRSTKEY_NAME': u'DEBRA MEALY', u'F9_07_PZ_COMP_OTHER': u'0', u'F9_07_PZ_COMP_RELATED': u'0', u'F9_07_PZ_TITLE': u'CHAIR PERSON', u'F9_07_PZ_AVE_HOURS_WEEK': u'1.00', u'F9_07_PC_TRUSTEE_INDIVIDUAL': u'X'}, {u'F9_07_PZ_COMP_DIRECT': u'0', u'F9_07_PZ_DIRTRSTKEY_NAME': u'HELEN GORDON', u'F9_07_PZ_COMP_OTHER': u'0', u'F9_07_PZ_COMP_RELATED': u'0', u'F9_07_PZ_TITLE': u'VICE CHAIR', u'F9_07_PZ_AVE_HOURS_WEEK': u'1.00', u'F9_07_PC_TRUSTEE_INDIVIDUAL': u'X'}, {'F9_07_PC_HIGH_COMP_EMPLOYEE': 'X', 'F9_07_PZ_DIRTRSTKEY_NAME': 'ROB S KHANUJA', 'F9_07_PZ_COMP_OTHER': '14902', 'F9_07_PZ_COMP_RELATED': '0', 'F9_07_PZ_TITLE': 'EXEC. DIR. OPERATIONS', 'F9_07_PZ_AVE_HOURS_WEEK': '40.00', 'F9_07_PZ_COMP_DIRECT': '133173'}}

```

I would normally flatten the *Form990PartVIISectionAGrp* column with the following code:



```
    df = pd.concat([df.drop(['Form990PartVIISectionAGrp'], axis=1), df['Form990PartVIISectionAGrp'].swifter.apply(pd.Series)], axis=1)

```

I'm looking to do this in Dask but am getting the following error: "ValueError: The columns in the computed data do not match the columns in the provided metadata."


I am using Python 2.7. I import the relevant packages



```
    from dask import dataframe as dd
    from dask.multiprocessing import get
    from multiprocessing import cpu_count
    nCores = cpu_count()

```

To test the code, I created a random sample of the data:



```
    dfs = df.sample(1000)

```

And then generate the Dask dataframe:



```
    ddf = dd.from_pandas(dfs, npartitions=nCores)

```

The column is currently in string format so I convert it to a dictionary. Normally, I would just write one line of code:



```
dfs['Form990PartVIISectionAGrp'] = dfs['Form990PartVIISectionAGrp'].apply(literal_eval) 

```

But I've instead tried to do it here in a more 'Dask-like' form so I write the following function and then apply it:



```
    def make_dict(dfs):
        dfs['Form990PartVIISectionAGrp'] = dfs['Form990PartVIISectionAGrp'].apply(literal_eval)   
        return dfs

```


```
    ddf_out = ddf.map_partitions(make_dict, meta=dfs[:0]).compute()

```

This works -- it returns a PANDAS dataframe where the Form990PartVIISectionAGrp column is in dictionary format (it's not any faster than the non-Dask apply, however). 


[![ddf_out](https://i.stack.imgur.com/huxKZ.png)](https://i.stack.imgur.com/huxKZ.png)


I then re-create the Dask DF:



```
    ddf = dd.from_pandas(ddf_out, npartitions=nCores)

```

And write a function to flatten the column:



```
    def flatten(ddf_out):
        ddf_out = pd.concat([ddf_out.drop(['Form990PartVIISectionAGrp'], axis=1), ddf_out['Form990PartVIISectionAGrp'].apply(pd.Series)], axis=1)
        #ddf_out = ddf_out['Form990PartVIISectionAGrp'].apply(pd.Series)
    return ddf_out

```

If I then run this code:



```
    result = ddf.map_partitions(flatten)

```

I get the following output, where the column has not been flattened:


[![result](https://i.stack.imgur.com/rT7uI.png)](https://i.stack.imgur.com/rT7uI.png)


I was also getting errors about missing meta data, and given that the above did not help parse the dictionary column, so I created a list of the columns produced by the plain Python flattening of the columns and used that to create a dictionary of the columns and data types:



```
metadir = {u'BusinessName': 'O', u'F9_07_PC_FORMER': 'O', u'F9_07_PC_HIGH_COMP_EMPLOYEE': 'O',
       u'F9_07_PC_KEY_EMPLOYEE': 'O', u'F9_07_PC_OFFICER': 'O',
       u'F9_07_PC_TRUSTEE_INDIVIDUAL': 'O', u'F9_07_PC_TRUSTEE_INSTITUTIONAL': 'O',
       u'F9_07_PZ_AVE_HOURS_WEEK': 'O', u'F9_07_PZ_AVE_HOURS_WEEK_RELATED': 'O',
       u'F9_07_PZ_COMP_DIRECT': 'O', u'F9_07_PZ_COMP_OTHER': 'O',
       u'F9_07_PZ_COMP_RELATED': 'O', u'F9_07_PZ_DIRTRSTKEY_NAME': 'O',
       u'F9_07_PZ_TITLE': 'O', u'NameBusiness': 'O', u'URL': 'O'}

```

I then apply the flatten function with this meta data:



```
    result = ddf.map_partitions(flatten, meta=metadir)

```

I get the following output as result:


[![result](https://i.stack.imgur.com/BiWz8.png)](https://i.stack.imgur.com/BiWz8.png)


Running result.columns yields this:


[![result.columns](https://i.stack.imgur.com/KiBlf.png)](https://i.stack.imgur.com/KiBlf.png)


Where this fails is in running compute(), where I get the following error message: "ValueError: The columns in the computed data do not match the columns in the provided metadata." I get the same error whether I write:



```
result.compute()

```

or 



```
result.compute(meta=metadir)

```

I'm not sure what I'm doing wrong here. The columns in *result* seem to match those in *metadir*. Any suggestions would be greatly appreciated.


UPDATE:
Here is my stab at updating the flatten function.



```
    meta = pd.DataFrame(columns=['URL', 'F9_07_PC_TRUSTEE_INDIVIDUAL',
     'F9_07_PZ_DIRTRSTKEY_NAME',
     'F9_07_PZ_COMP_OTHER',
     'F9_07_PZ_COMP_RELATED',
     'F9_07_PZ_TITLE',
     'F9_07_PZ_AVE_HOURS_WEEK',
     'F9_07_PZ_COMP_DIRECT',
     'F9_07_PZ_AVE_HOURS_WEEK_RELATED',
     'F9_07_PC_OFFICER',
     'F9_07_PC_HIGH_COMP_EMPLOYEE',
     'BusinessName',
     'F9_07_PC_KEY_EMPLOYEE',
     'F9_07_PC_TRUSTEE_INSTITUTIONAL',
     'NameBusiness',
     'F9_07_PC_FORMER'], dtype="O")

    def flatten(ddf_out):
        ddf_out = pd.concat([df.drop(['Form990PartVIISectionAGrp'], axis=1), df['Form990PartVIISectionAGrp'].apply(pd.Series)], axis=1)
        for m in meta:
            if m not in ddf_out:
                df[m] = '' 
        return ddf_out

```

Then I run:



```
result = ddf.map_partitions(flatten, meta=meta).compute()

```
A couple of notes to get started:



> 
> .apply(literal\_eval)
> 
> 
> 


Wouldn't this be better as `map`?



> 
> I then re-create the Dask DF:
> 
> 
> ddf = dd.from\_pandas(ddf\_out, npartitions=nCores)
> 
> 
> 


`ddf_out` was already a dask dataframe, I don't know why you had to do this.



> 
> The columns in result seem to match those in metadir.
> 
> 
> 


The value of `result.columns` is taken from the meta you provided, no computing happens until you ask for it (dask is lazy in most operations). Does the ValueError exception not provide further information?


Here is a full example



```
x = ({'F9_07_PZ_COMP_DIRECT': '0',
  'F9_07_PZ_DIRTRSTKEY_NAME': 'DEBRA MEALY',
  'F9_07_PZ_COMP_OTHER': '0',
  'F9_07_PZ_COMP_RELATED': '0',
  'F9_07_PZ_TITLE': 'CHAIR PERSON',
  'F9_07_PZ_AVE_HOURS_WEEK': '1.00',
  'F9_07_PC_TRUSTEE_INDIVIDUAL': 'X'},
 {'F9_07_PZ_COMP_DIRECT': '0',
  'F9_07_PZ_DIRTRSTKEY_NAME': 'HELEN GORDON',
  'F9_07_PZ_COMP_OTHER': '0',
  'F9_07_PZ_COMP_RELATED': '0',
  'F9_07_PZ_TITLE': 'VICE CHAIR',
  'F9_07_PZ_AVE_HOURS_WEEK': '1.00',
  'F9_07_PC_TRUSTEE_INDIVIDUAL': 'X'})
df = pd.DataFrame({'a': x})
d = dd.from_pandas(df, 1)
meta = pd.DataFrame(columns=['F9_07_PZ_COMP_DIRECT', 
       'F9_07_PZ_DIRTRSTKEY_NAME',
       'F9_07_PZ_COMP_OTHER', 'F9_07_PZ_COMP_RELATED', 'F9_07_PZ_TITLE',
       'F9_07_PZ_AVE_HOURS_WEEK', 'F9_07_PC_TRUSTEE_INDIVIDUAL'], dtype="O")
d.map_partitions(lambda df: df.a.apply(pd.Series), meta=meta).compute()

```

How did I know what `meta` to use? I applied the function to the pandas dataframe - you might use a small piece of the dataframe to do this.


Some additional notes:


* it is an anti-pattern to load the data with pandas, pass to dask workers and then gather the whole result back to a pandas (in memory) dataframe, you are very unlikely to see a speedup this way and may incur a lot of overhead. You are better off loading with something like `dd.read_csv`, and aggregating or writing with dask functions too. Only `compute()` on something that will be small or not return anything (because it involves writing the output). The official examples do not use from\_pandas.
* string and dict processing are python methods, and so hold the interpreter lock (GIL) of any python function: threads will not actually run in parallel. To get parallelism, you need to run in processes, which is most easily achieved using <https://docs.dask.org/en/latest/setup/single-distributed.html>
* the distributed scheduler also gives you access to the dashboard, which has a lot of useful information to diagnose how your system is running. You can also configure a lot about its behaviour, in case you have firewall rules you need to obey.
Simply using `df.mean()` will Do The Right Thing(tm) with respect to NaNs:



```
>>> df
                 A      B
DATE                     
2013-05-01  473077  71333
2013-05-02   35131  62441
2013-05-03     727  27381
2013-05-04     481   1206
2013-05-05     226   1733
2013-05-06     NaN   4064
2013-05-07     NaN  41151
2013-05-08     NaN   8144
2013-05-09     NaN     23
2013-05-10     NaN     10
>>> df.mean(axis=1)
DATE
2013-05-01    272205.0
2013-05-02     48786.0
2013-05-03     14054.0
2013-05-04       843.5
2013-05-05       979.5
2013-05-06      4064.0
2013-05-07     41151.0
2013-05-08      8144.0
2013-05-09        23.0
2013-05-10        10.0
dtype: float64

```

You can use `df[["A", "B"]].mean(axis=1)` if there are other columns to ignore. Resolved:  

  

FORM 1.  

  

Copy Code

```
private void btnModificaContractul\_Click(object sender, EventArgs e)
        {


            string[] IDProdusContractPrelugit = new string[dataProduseContract.Rows.Count];
            string[] IDProdusContractRetras = new string[dataProduseContract.Rows.Count];

            for (int i = 0; i < dataProduseContract.Rows.Count; i++)
            {

                if (Convert.ToBoolean(dataProduseContract.Rows[i].Cells[0].Value) == false)
                {
                    IDProdusContractPrelugit[i] = dataProduseContract.Rows[i].Cells[5].Value.ToString();
                }
                else
                {
                    IDProdusContractRetras[i] = dataProduseContract.Rows[i].Cells[5].Value.ToString();
                }
            }

           

            try
            {
                frmModificaContract ModificareContract = new frmModificaContract(IDProdusContractPrelugit, IDProdusContractRetras);
                ModificareContract.ShowDialog();
            }
            catch { MessageBox.Show("Trebuie să selectaţi un contract din listă pentru a edita.", "Datele furnizate sunt incomplete."); }
           
        }
```
  

  

FORM 2.  

  

Copy Code

```
string[] IDProdusContractPrelugit;
        string[] IDProdusContractRetras;

        public frmModificaContract(string[] IDProdusContractPrelugit, string[] IDProdusContractRetras)
        {
            InitializeComponent();
            this.IDProdusContractPrelugit = IDProdusContractPrelugit;
            this.IDProdusContractRetras = IDProdusContractRetras;
            AfisareProduseContract();
        }


        void AfisareProduseContract()
        {

            foreach (string ContractPrelungit in IDProdusContractPrelugit)
            {
                if (ContractPrelungit != null)
                {
                    Program.Connection.CommandText = "select * from ContractItems WHERE ContractItemId=@ContractItemId";
                    Program.Connection.AddParameter("@ContractItemId", ContractPrelungit);

                    DataTable Table = new DataTable();
                    Program.Connection.FillDataTable(Table, true);


                    for (int i = 0; i < Table.Rows.Count; i++)
                    {
                        if (String.IsNullOrWhiteSpace(Table.Rows[i]["DeletedId"].ToString()))
                        {
                            bool validare;
                            validare = true;

                            dataProduseContract.Rows.Add(validare, Table.Rows[i]["Name"].ToString(), Table.Rows[i]["Quantity"].ToString() + " " + Table.Rows[i]["QuantityUnit"].ToString(), Table.Rows[i]["Payment"].ToString(), Table.Rows[i]["Description"].ToString(), Table.Rows[i]["ContractItemId"].ToString());
                        }
                        else
                        {
                            bool validarescos;
                            validarescos = false;
                            dataProduseContract.Rows.Add(validarescos, Table.Rows[i]["Name"].ToString(), Table.Rows[i]["Quantity"].ToString() + " " + Table.Rows[i]["QuantityUnit"].ToString(), Table.Rows[i]["Payment"].ToString(), Table.Rows[i]["Description"].ToString(), Table.Rows[i]["ContractItemId"].ToString());
                            dataProduseContract.Enabled = false;
                        }

                    }
                }
            }

            
        }
```
  

  

The problem is allwais i have Null value in array. am rezolvat , multumesc , am "furat" sursa html si css din pluginul " enhanced user info panel "   o sa las mai jos si sursele pentru cei ce vor sa faca ceva de genul 
 


 forums  /  font  /  topics  /  postcontainer
 


{{$idfield = $comment::$databasecolumnid;}}
{{$itemclasssafe = str\_replace( '\\', '\_', mb\_substr( $comment::$itemclass, 4 ) );}}
{{if $comment->isignored()}}
 <div class='ipscomment ipscomment\_ignored ipstype\_light' id='elignorecomment\_{$comment->$idfield}' data-ignorecommentid='elcomment\_{$comment->$idfield}' data-ignoreuserid='{$comment->author()->member\_id}'>
 {lang="ignoring\_content" sprintf="$comment->author()->name"} <a href='#elignorecomment\_{$comment->$idfield}\_menu' data-ipsmenu data-ipsmenu-menuid='elignorecomment\_{$comment->$idfield}\_menu' data-ipsmenu-appendto='#elignorecomment\_{$comment->$idfield}' data-action="ignoreoptions" title='{lang="see\_post\_ignore\_options"}' class='ipstype\_blendlinks'>{lang="options"} <i class='fa fa-caret-down'></i></a>
 <ul class='ipsmenu ipshide' id='elignorecomment\_{$comment->$idfield}\_menu'>
 <li class='ipsmenu\_item ipsjs\_show' data-ipsmenuvalue='showpost'><a href='#'>{lang="show\_this\_post"}</a></li>
 <li class='ipsmenu\_sep ipsjs\_show'><hr></li>
 <li class='ipsmenu\_item' data-ipsmenuvalue='stopignoring'><a href='{url="app=core&module=system&controller=ignore&do=remove&id={$comment->author()->member\_id}" seotemplate="ignore"}'>{lang="stop\_ignoring\_posts\_by" sprintf="$comment->author()->name"}</a></li>
 <li class='ipsmenu\_item'><a href='{url="app=core&module=members&controller=ignore" seotemplate="ignore"}'>{lang="change\_ignore\_preferences"}</a></li>
 </ul>
 </div>
{{endif}}

<a id='comment-{$comment->$idfield}'></a>
<article itemscope {{if $item->isquestion() && !$comment->new\_topic}}itemprop="suggestedanswer {{if $comment->post\_bwoptions['best\_answer']}}acceptedanswer{{endif}}" itemtype="http://schema.org/answer"{{else}}itemtype="http://schema.org/comment"{{endif}} id='elcomment\_{$comment->$idfield}' class='cpost ipsbox {{if $otherclasses}}{$otherclasses}{{endif}} ipscomment {{if settings.reputation\_highlight and $comment->reputation() >= settings.reputation\_highlight}}ipscomment\_popular{{endif}} ipscomment\_parent ipsclearfix ipsclear ipscolumns ipscolumns\_nospacing ipscolumns\_collapsephone {{if $comment->isignored()}}ipshide{{endif}} {{if $comment->hidden()}}ipsmoderated{{endif}}'>
 {{if $item->isquestion() and !$comment->new\_topic}}
 {template="postrating" group="topics" app="forums" params="$item, $comment, $votes"}
 {{endif}}
 <aside class='ipscomment\_author cauthorpane ipscolumn ipscolumn\_medium'>
 <h3 class='ipstype\_sectionhead cauthorpane\_author ipstype\_blendlinks ipstype\_break' itemprop="creator" itemscope itemtype="http://schema.org/person"><strong itemprop="name">{$comment->author()->link( $comment->warningref() )|raw}</strong> <span class='ipsresponsive\_showphone ipsresponsive\_inline'>&nbsp;&nbsp;{template="reputationbadge" group="global" app="core" params="$comment->author()"}</span></h3>
 <ul class='cauthorpane\_info ipslist\_reset'>
 
 {{if $comment->author()->member\_title && $comment->author()->member\_id}}
 
 {{elseif $comment->author()->rank['title'] && $comment->author()->member\_id}}
 <li class='ipsresponsive\_hidephone'>{$comment->author()->rank['title']}</li>
 {{endif}} 
 <!-- {{if $comment->author()->rank['image'] && $comment->author()->member\_id}}
 <li class='ipsresponsive\_hidephone'>{$comment->author()->rank['image']|raw}</li>
 {{endif}} -->
 
 <li class='cauthorpane\_photo'>
 {{if $comment->author()->pp\_main\_photo}}<a href='{url="app=core&module=members&controller=profile&id={$comment->author()->member\_id}-{$comment->author()->name}" seotemplate="profile" seotitle="$comment->author()->name"}' data-ipshover data-ipshover-target='{url="app=core&module=members&controller=profile&id={$comment->author()->member\_id}-{$comment->author()->name}&do=hovercard" seotemplate="profile" seotitle="$comment->author()->name"}' title="{lang="view\_user\_profile" sprintf="$comment->author()->name"}"><span class='ipsuserphoto ipsuserphoto\_variable'><img src="{setting="base\_url"}uploads/{$comment->author()->pp\_main\_photo}" /></span></a>{{else}}<a href='{url="app=core&module=members&controller=profile&id={$comment->author()->member\_id}-{$comment->author()->name}" seotemplate="profile" seotitle="$comment->author()->name"}' data-ipshover data-ipshover-target='{url="app=core&module=members&controller=profile&id={$comment->author()->member\_id}-{$comment->author()->name}&do=hovercard" seotemplate="profile" seotitle="$comment->author()->name"}' title="{lang="view\_user\_profile" sprintf="$comment->author()->name"}"><span class='ipsuserphoto ipsuserphoto\_variable'><img src="{resource="default\_photo.png" app="core" location="global"}" /></a>{{endif}}
 </li>

 <li class='group\_badge'>
 <div class='group\_name'>{expression="\ips\member\group::load( $comment->author()->member\_group\_id )->formattedname" raw="true"}</div>
 <!-- <div class='member\_title'>
 {{if $comment->author()->member\_title && $comment->author()->member\_id}}
 {$comment->author()->member\_title}
 {{elseif $comment->author()->rank['title'] && $comment->author()->member\_id}}
 {$comment->author()->rank['title']}
 {{endif}}
 </div> -->
 </li>
 </ul>
 {{if $comment->author()->rank['image'] && $comment->author()->member\_id}}
 
 {{endif}}
 
 {{if $comment->author()->member\_id}}
 <ul class='dpxutprofile'>
 <li class="ipsresponsive\_hidephone ipstype\_break">
 <div class='stat\_name'><i class="fa {setting="euip\_fafauserrank"}" aria-hidden="true"></i> {lang="euip\_membertitle"}:</div>
 <div class='stat\_value'>{$comment->author()->member\_title}</div>
 </li>
 
 <li class='euip\_border'>
 <div class='stat\_name'><i class="fa {setting="euip\_fafausername"}" aria-hidden="true"></i> {lang="euip\_username"}:</div>
 <div class='stat\_value'> <strong itemprop="name">{$comment->author()->link( $comment->warningref() )|raw}</strong></div>
 </li>
 <li class='euip\_border {{if \ips\settings::i()->euip\_usergrouphidephone == 1}}ipsresponsive\_hidephone{{endif}}'>
 <div class='stat\_name'><i class="fa {setting="euip\_fafausergroup"}" aria-hidden="true"></i> {lang="euip\_usergroup"}:</div>
 <div class='stat\_value'>{expression="\ips\member\group::load( $comment->author()->member\_group\_id )->formattedname" raw="true"}</div>
 </li>
 <li> 
 <div class='stat\_name'>
 <i class="fa fa-comments-o" aria-hidden="true"></i> posts 
 </div>
 <div class='stat\_value'> 
 {$comment->author()->member\_posts}
 </div>
 </li>
 
 <li class='euip\_border {{if \ips\settings::i()->euip\_userrephidephone == 1}}ipsresponsive\_hidephone{{endif}}'>
 <div class='stat\_name'><i class="fa {setting="euip\_fafauserrep"}" aria-hidden="true"></i> {lang="euip\_totalrep"}:</div>
 <div class='stat\_value'>{template="reputationbadge" group="global" app="core" params="$comment->author()"}</div>
 </li>
 <li> 
 <div class='stat\_name'>
 <i class="fa fa-exclamation-triangle" aria-hidden="true"></i> warn level
 </div>
 <div class='stat\_value'> 
 {$comment->author()->warn\_level}
 </div>
 </li>
 
 <li class='euip\_border {{if \ips\settings::i()->euip\_birthdayhidephone == 1}}ipsresponsive\_hidephone{{endif}}'>
 <div class='stat\_name'><i class="fa {setting="euip\_fafabirthday"}" aria-hidden="true"></i> {lang="euip\_age"}:</div>
 <div class='stat\_value'>{$comment->author()->age()}</div>
 </li>
 
 <li class='euip\_border {{if \ips\settings::i()->euip\_useridhidephone == 1}}ipsresponsive\_hidephone{{endif}}'>
 <div class='stat\_name'><i class="fa {setting="euip\_fafauserid"}" aria-hidden="true"></i> {lang="euip\_memberid"}:</div>
 <div class='stat\_value'>{$comment->author()->member\_id}</div>
 </li>
 
 <li class='euip\_border {{if \ips\settings::i()->euip\_lastactivityhidephone == 1}}ipsresponsive\_hidephone{{endif}}'>
 <div class='stat\_name'><i class="fa {setting="euip\_fafalastactivity"}" aria-hidden="true"></i> {lang="euip\_lastactivity"}:</div>
 <div class='stat\_value'>{{if $comment->author()->last\_activity}}{datetime="$comment->author()->last\_activity"}{{else}}{lang="never"}{{endif}}</div>
 </li>
 
 <li class='euip\_border {{if \ips\settings::i()->euip\_onlinestatushidephone == 1}}ipsresponsive\_hidephone{{endif}}'>
 <div class='stat\_name'><i class="fa {setting="euip\_fafaonlinestatus"}" aria-hidden="true"></i> {lang="euip\_onlinestatus"}:</div>
 <div class='stat\_value'>{{if $comment->author()->isonline()}}<span style='color:green' class='metric'>{lang="euip\_online"}</span>{{else}}<span style='color:red'>{lang="euip\_offline"}</span>{{endif}}</div>
 </li> 
 
 
 <li class='euip\_border {{if \ips\settings::i()->euip\_joindatehidephone == 1}}ipsresponsive\_hidephone{{endif}}'>
 <div class='stat\_name'><i class="fa {setting="euip\_fafauserjoindate"}" aria-hidden="true"></i> {lang="euip\_joined"}:</div>
 <div class='stat\_value'>{datetime="$comment->author()->joined" dateonly="true"}</div>
 </li>
 <li class='euip\_border {{if \ips\settings::i()->euip\_birthdayhidephone == 1}}ipsresponsive\_hidephone{{endif}}'>
 <div class='stat\_name'> <i class="fa {setting="euip\_fafabirthday"}" aria-hidden="true"></i> {lang="euip\_bday"}:</div>
 <div class='stat\_value'> {$comment->author()->birthday}</div>
 </li>
 <li class='ipsresponsive\_hidephone'>{$comment->author()->rank['image']|raw}</li>
 </ul>
 {{endif}}
 </aside>
 <div class='ipscolumn ipscolumn\_fluid'>
 {template="post" group="topics" app="forums" params="$item, $comment, $item::$formlangprefix . 'comment', $item::$application, $item::$module, $itemclasssafe"}
 </div>
</article>


 in custom.css se adauga urmatorul cod :
 


ul.dpxutprofile > li {
 display: flex;
 width: 90%;
 justify-content: space-between;
 margin: 5px auto;
 font-size: 10px;
 border-bottom: 1px dashed #444;
}
ul.dpxutprofile {
 margin: 0;
 padding: 0;
 list-style: none;
 margin-top: 15px;
}

ul.dpxutprofile > li > .stat\_name {
 display: inline-block;
 text-align: left;
 color: #525252;
 cursor: default;
 float: left;
 letter-spacing: 1px;
}

ul.dpxutprofile > li > .stat\_name > i {
 font-size: 14px;
 position: relative;
 top: 1px;
 margin-right: 5px;
}
ul.dpxutprofile > li > .stat\_value {
 display: inline-block;
 text-align: left;
 color: #828282;
 float: right;
}

.group\_badge {
 background: #2a2a2a;
 background: -webkit-linear-gradient(90deg,#23f88a transparent);
 background: -o-linear-gradient(90deg,#23f88a transparent);
 background: -moz-linear-gradient(90deg,#2a2a2a, transparent);
 background: linear-gradient(90deg,#2a2a2a, transparent);
 margin: 0 auto;
 margin-bottom: 15px;
 margin-top: 15px;
 width: 80%;
 height: 35px;
 line-height: 35px;
 border-radius: 30px;
}

.group\_name {
 position: relative;
 top: 0;
 transition: all .2s;
 cursor: default;
}


 rezultatul You are mixing [expressions](https://docs.microsoft.com/en-us/dotnet/api/system.linq.expressions.expression?view=netframework-4.7.1) with [lambda expressions](https://docs.microsoft.com/en-us/dotnet/api/system.linq.expressions.lambdaexpression?view=netframework-4.7.1). There are many posts showing how you can combine lambda expressions, but the essential part is to compose expressions from lambda expression [bodies](https://docs.microsoft.com/en-us/dotnet/api/system.linq.expressions.lambdaexpression.body?view=netframework-4.7.1) and rebind the [parameters](https://docs.microsoft.com/en-us/dotnet/api/system.linq.expressions.lambdaexpression.parameters?view=netframework-4.7.1).


The later is usually achieved by a custom [`ExpressionVisitor`](https://docs.microsoft.com/en-us/dotnet/api/system.linq.expressions.expressionvisitor?view=netframework-4.7.1) like this:



```
using System.Linq.Expressions;

public static class ExpressionExtensions
{
    public static Expression ReplaceParameter(this Expression expression, ParameterExpression source, Expression target)
    {
        return new ParameterReplacer { Source = source, Target = target }.Visit(expression);
    }

    class ParameterReplacer : ExpressionVisitor
    {
        public ParameterExpression Source;
        public Expression Target;
        protected override Expression VisitParameter(ParameterExpression node)
        {
            return node == Source ? Target : base.VisitParameter(node);
        }
    }
}

```

Now regarding the EF Core combined query filters. 


Using dictionaries and expression lists seems overcomplicated for what are you doing. Since [`IMutableEntityType`](https://docs.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.metadata.imutableentitytype?view=efcore-2.1) provides read/write access to the [`QueryFilter`](https://docs.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.metadata.imutableentitytype.queryfilter?view=efcore-2.1#Microsoft_EntityFrameworkCore_Metadata_IMutableEntityType_QueryFilter), the same can be achieved with a small set of a custom extension methods.


All they go inside a class like this:



```
using System;
using System.Linq.Expressions;
using Microsoft.EntityFrameworkCore.Metadata;
using Microsoft.EntityFrameworkCore.Metadata.Builders;

public static class QueryFilterExtensions
{
}

```

First method:



```
public static void AddQueryFilter(this IMutableEntityType target, LambdaExpression filter)
{
    if (target.QueryFilter == null)
        target.QueryFilter = filter;
    else
    {
        var parameter = target.QueryFilter.Parameters[0];
        var left = target.QueryFilter.Body;
        var right = filter.Body.ReplaceParameter(filter.Parameters[0], parameter);
        var body = Expression.AndAlso(left, right);
        target.QueryFilter = Expression.Lambda(body, parameter);
    }
}

```

This is a non generic method which combines the exiting filter with passed filter using `AndAlso` (C# `&&`) operator and shows the aforementioned lambda expression combining principles.


However it's not so useful directly, like inside your entity type configuration loop (it can, but requires you to build manually the lambda expression instead of letting C# compiler do that). So here comes the second method:



```
public static void AddQueryFilter<T>(this IMutableEntityType target, Expression<Func<T, bool>> filter)
{
    LambdaExpression targetFilter = filter;
    if (target.ClrType != typeof(T))
    {
        var parameter = Expression.Parameter(target.ClrType, "e");
        var body = filter.Body.ReplaceParameter(filter.Parameters[0], parameter);
        targetFilter = Expression.Lambda(body, parameter);
    }
    target.AddQueryFilter(targetFilter);
}

```

It's a generic method - not quite type-safe, but allows you to use a compile time lambda expression and bind it to the actual entity type as follows:



```
foreach (var entityType in modelBuilder.Model.GetEntityTypes())
{
    if (typeof(IDeleted).IsAssignableFrom(entityType.ClrType))
        entityType.AddQueryFilter<IDeleted>(e => e.DateTimeDeleted == null);
}

```

Looks better, isn't it :)


The last custom extension method is complement to (replacement of) the standard EF Core generic [`HasQueryFilter`](https://docs.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.metadata.builders.entitytypebuilder-1.hasqueryfilter?view=efcore-2.1#Microsoft_EntityFrameworkCore_Metadata_Builders_EntityTypeBuilder_1_HasQueryFilter_System_Linq_Expressions_Expression_System_Func__0_System_Boolean___) method:



```
public static EntityTypeBuilder<TEntity> AddQueryFilter<TEntity>(this EntityTypeBuilder<TEntity> target, Expression<Func<TEntity, bool>> filter)
    where TEntity : class
{
    target.Metadata.AddQueryFilter(filter);
    return target;
}

```

and allows you to replace



```
this.AddToDict<DbMyEntity>(t => t.Name == null || t.Name == "Something");

```

with the more convenient



```
modelBuilder.Entity<DbMyEntity>()
    .AddQueryFilter(t => t.Name == null || t.Name == "Something");

```

**Update (EF Core 3.0):** `QueryFilter` property has been replaced with `GetQueryFilter` and `SetQueryFilter` extension methods. A few things to try.


1. Use [Variable Tables](http://odetocode.com/Articles/365.aspx) instead -> maybe the import wizard prefers that?
2. Name your return fields.


Try using the following stored proc (untested .. just thinking out loud...)



```
CREATE PROCEDURE dbo.Foo
AS

    SET NOCOUNT ON

    DECLARE @ResultTable TABLE (SomeId INTEGER)

    INSERT INTO @ResultTable
    SELECT DISTINCT Id AS Identity -- Or u can rename this field to anything...
    FROM SomeExistingTableWhichHasAnIdentityField

GO

```

Try that and see if the wizard refreshes, now.


-- 


 Attempt #2 :)


Ok .. when the EF designer/wizard/whatever fails to figure out EXACTLY what my stored proc is suppose to be returning, I usually do the following :-


1. Make sure the stored procedure doesn't exist at all in the EF designer/context, etc. (You have a clean starting point)
2. Open up your stored procedure and `/* /*` comment out *EVERYTHING* after the procedure definition.


eg..



```
ALTER PROCEDURE dbo.Foo
(
    Bar1 INT,
    Bar2 TINYINT,
    ... // whatever u have as your optional input arguments //
)
AS
    SET NOCOUNT ON

    /* 
    .... every thing in here is commented out 
    */
GO

```

Now ...
 3. Add a forced fake return in the stored proc, which (more or less) just *defines* the output structure/fields.


eg..



```
ALTER PROCEDURE dbo.Foo
(
    Bar1 INT,
    Bar2 TINYINT,
    ... // whatever u have as your optional input arguments //
)
AS
    SET NOCOUNT ON

    SELECT 1 AS Id, 1 AS UserId, 1 AS SomeOtherId, 
        CAST('AAA' AS NVARCHAR(350)) AS Name,
        -- etc etc etc..
    /* 
    .... every thing in here is commented out 
    */
GO

```

[and then](http://www.youtube.com/watch?v=K7luMp6lb9M) ...


4. Add this stored proc to your EF designer/wizard/etc... Now the correct fields should be 'determined' by the designer. AWESOME. Yes .. the values are all hardcoded .. but that's ok (so far).
5. Once your happy that EF is now updated right, go back to your stored proc, and remove all hardcoded SELECT (which we did in the above step). Now we remove the comments which we commented out the entire real code. So you should have your original stored proc, back.


... and now EF is updated and doesn't know we've changed the plumbing of your stored proc.


win :)


does this work for ya? hello [@saurabhgupta](/u/saurabhgupta) :wave:

i’m afraid that the last updated column isn’t supported by our api at this time. this is because it’s calculated client side. you can see a list of these column types here - <https://monday.com/developers/v2#column-values-section-auto-calculated-column>

although the api do… Hi JimCreech,


It is possible to achieve a similar result in Dataiku by going through the following steps:


* run a [Window](https://knowledge.dataiku.com/latest/courses/visual-recipes/window.html) recipe using id as partitioning column, event as ordering column, and compute the lagdiff values on days
* run a [Prepare](https://knowledge.dataiku.com/latest/courses/basics/prepare-data/concept-prepare-recipe.html) recipe which will:
	+ run the assertions on the specific conditions you mentioned and create a dummy\_tmp column we'll reuse at the final step
	+ filter out the records where event == 01 (since we won't need it for the aggregation)
* run a [Groupby](https://knowledge.dataiku.com/latest/courses/basics/group-data/group-summary.html) recipe to aggregate at the id level and get the minimum of the dummy\_tmp column (which should be 1 if all conditions you described are verified, 0 otherwise).


You will find attached a sample DSS project with an example of implementation of those steps using the small example data mentioned in your post. If you are working on a regular basis with time series data, you may find this [tutorial](https://knowledge.dataiku.com/latest/courses/visual-recipes/window/window.html) on the Window recipe capabilities useful.


Hope this helps !


Best,


Harizo




[View solution in original post](/t5/General-Discussion/Using-Lag-and-Lead-in-Dataiku/m-p/19308#M1146) Should be able to do something like this:


UPDATE table\_name  

SET new\_column = (calculated value)  

WHERE new\_column is NULL thanks i was able to use something similar. i have just seen online that we’re not able to extract column values for calculated columns? is there any other way of extracting this data?

query = """ query { boards (ids: %s limit: 20, page: 1) { id name columns … Over the years, I've had several versions of DW -- some were more stable than others.  I can honestly say, the FTP has never been a big problem for me.   If it ever becomes glitchy, I close DW and clear my cache or restore preferences as that fixes most  problems. 

**[Unusual behavior in Dreamweaver? Try restoring preferences.](https://helpx.adobe.com/dreamweaver/kb/restore-preferences-dreamweaver.html "https://helpx.adobe.com/dreamweaver/kb/restore-preferences-dreamweaver.html")** 

I mainly work with dynamic sites but I don't have more than 2,000 files per site folder.    Each site folder resides on my primary hard drive within my local testing server's default web directory (htdocs or www).   

![CC-testingServer1.jpg](/legacyfs/online/1550323_CC-testingServer1.jpg)

I use Ctrl+Shift+U to upload an open document to server.  To be quite honest, it happens so fast, I don't even see the background process panel.    If I'm uploading many files, I expand my files panel so I can compare contents of local site with remote server to ensure that the last modified timestamps are consistent.

![CC-FilesPanel-expanded.jpg](/legacyfs/online/1550322_CC-FilesPanel-expanded.jpg)

Finally, I had to turn off Real-Time backups and anti-virus scanning while I work.  When those processes were running in the background, everything came to a crawl. 

Hope this helps.



 Nancy O'Shea, ACP  

Alt-Web Design & Publishing