问:
GridView doesn't scroll down to virtualized items
In my application, I need to select the newly created document(note) when I go back to library. After library item is selected, the Library must be scrolled to the selected item.

My library's OnLoaded method:

```<code>private async void OnLoaded(object sender, RoutedEventArgs e)
{
   await this.ViewModel.InitializeAsync();

   // CollectionViewSource of my GridView being filled
   ViewModel.CollectionChanging = true;
   GroupInfoCVS.Source = ViewModel.GroupsCollection;
   ViewModel.CollectionChanging = false;

   // Loading Last selected item - THIS CHANGES SELECTION
   ViewModel.LoadLastSelection();
}
</code>```

After I call the LoadLastSelection method, selection is changed successfuly (I've tested). This is the method that is called after that (in our GridView's extended control):

```<code>private void OnSelectionChanged(object sender, SelectionChangedEventArgs e)
{
    this.SelectedItemsCount = this.SelectedItems.Count;

    var newlySelectedItems = e.AddedItems;
    if (newlySelectedItems.Any())
    {
        var item = newlySelectedItems.Last();
        ScrollTo(item);
    }
}

private void ScrollTo(object item)
{
    UpdateLayout();
    var itemUI = (FrameworkElement)this.ContainerFromItem(item);

    if (itemUI != null)
    {
        _scrollViewer.UpdateLayout();
        _scrollViewer.ChangeView(null, itemUI.ActualOffset.Y - itemUI.ActualHeight, null, false);
    }
}
</code>```

This also works for the most part. When itemUI is not null, the method scrolls successfully to the required item. The problems start when the items start to overflow the screen size. When items are completely hidden from the screen, they are virtualized. That means that ContainerFromItem returns null, so I can't take the offset properties. Keep in mind that this actually occurs before Library's OnLoaded method is finished.

Please, help me with some alternative to get such properties or other methods of scrolling, which will help me scroll successfully.

I've read a lot and tried using Dispatcher.RunAsync and ScrollIntoView methods, but I couldn't manage to produce any scrolling behavior. If you point me how to use them successfully, that would be a nice help too.

Here's what I've read (and tried):

 ItemContainerGenerator.ContainerFromItem() returns null?

How to Know When a FrameworkElement Has Been Totally Rendered?

Is there a "All Children loaded" event in WPF

Let ListView scroll to selected item

Thanks in advance!

IMPORTANT: If you don't want to read all the conversation within the official answer, please read the solution in short here:

TemplatedControl's style had changed ScrollViewer's name from "ScrollViewer" to "LibraryScrollViewer" and that rendered ScrollIntoView method useless.

答:
For GridView, the best way to achieve your needs is to call GridView.ScrollIntoView.

But you seem to have made similar attempts, and it does not to be successful, then the following points may help you:

1. Don't use GridView as a child element of ScrollViewer.

In your code, I see that you are calling the method of ScrollViewer.ChangeView to adjust the view scrolling, so it is speculated that you may put the GridView in the ScrollViewer, which is not recommended.

Because there is a ScrollViewer inside the GridView, and its ScrollIntoView method is to change the scroll area of the internal ScrollViewer. When there is a ScrollViewer outside, the ScrollViewer inside the GridView will lose the scrolling ability, thus making the ScrollIntoView method invalid.

2. Implement the Equals method of the data class.

If your data class is not a simple type (such as String, Int32, etc.), then implementing the Equals method of the data class will help the GridView to find the corresponding item.

Thanks.

Body:
In my application, I need to select the newly created document(note) when I go back to library. After library item is selected, the Library must be scrolled to the selected item.

My library's OnLoaded method:

private async void OnLoaded(object sender, RoutedEventArgs e)
{
   await this.ViewModel.InitializeAsync();

   // CollectionViewSource of my GridView being filled
   ViewModel.CollectionChanging = true;
   GroupInfoCVS.Source = ViewModel.GroupsCollection;
   ViewModel.CollectionChanging = false;

   // Loading Last selected item - THIS CHANGES SELECTION
   ViewModel.LoadLastSelection();
}

After I call the LoadLastSelection method, selection is changed successfuly (I've tested). This is the method that is called after that (in our GridView's extended control):

private void OnSelectionChanged(object sender, SelectionChangedEventArgs e)
{
    this.SelectedItemsCount = this.SelectedItems.Count;

    var newlySelectedItems = e.AddedItems;
    if (newlySelectedItems.Any())
    {
        var item = newlySelectedItems.Last();
        ScrollTo(item);
    }
}

private void ScrollTo(object item)
{
    UpdateLayout();
    var itemUI = (FrameworkElement)this.ContainerFromItem(item);

    if (itemUI != null)
    {
        _scrollViewer.UpdateLayout();
        _scrollViewer.ChangeView(null, itemUI.ActualOffset.Y - itemUI.ActualHeight, null, false);
    }
}

This also works for the most part. When itemUI is not null, the method scrolls successfully to the required item. The problems start when the items start to overflow the screen size. When items are completely hidden from the screen, they are virtualized. That means that ContainerFromItem returns null, so I can't take the offset properties. Keep in mind that this actually occurs before Library's OnLoaded method is finished.

Please, help me with some alternative to get such properties or other methods of scrolling, which will help me scroll successfully.

I've read a lot and tried using Dispatcher.RunAsync and ScrollIntoView methods, but I couldn't manage to produce any scrolling behavior. If you point me how to use them successfully, that would be a nice help too.

Here's what I've read (and tried):

 ItemContainerGenerator.ContainerFromItem() returns null?

How to Know When a FrameworkElement Has Been Totally Rendered?

Is there a "All Children loaded" event in WPF

Let ListView scroll to selected item

Thanks in advance!

IMPORTANT: If you don't want to read all the conversation within the official answer, please read the solution in short here:

TemplatedControl's style had changed ScrollViewer's name from "ScrollViewer" to "LibraryScrollViewer" and that rendered ScrollIntoView method useless.


来源: stackexchange

问:
Script Bash - cp Target is not a Directory - Using variables
I know that this type of questions maybe were duplicated. But in my case, I'm use a variable that contains the destination directory and the error says, that is not a valid directory.

cp target var2 is not a directory

Now, this is my code, I'm trying to copy all images from the folder images (contained in Desktop) to the folder photos (In desktop too and empty). I use GIT Bash for Windows 10

```<code>function pause(){

    read -p "$*"
}

echo "Help: Use the bar / to separate the directory names"
echo ""

echo "Enter the directory location"
read var1

echo "Enter the destinity directory location"
read var2

for var1 in "var2"
do
    cp * var1 var2
done

echo ""
pause "Press [ENTER] to continue..."
</code>```

What's my mistake? Thanks!

When i execute the script, the directory I use is:

C:/Users/Mauri/Desktop/images

And the var2 is:

C:/Users/Mauri/Desktop/photos

With this code into the for:

cp -r * $var1 $var2

Was moved all archives in the desktop to photos folder, It's not my objetive, I want to move only the images contained on the image folder in the desktop to the empty folder photos in the desktop too.

答:
Instead of your for loop, it would be great if we could run cp "$var1/"*.{jpg,png,gif} "$var2". That would be an easy way to copy all of the files with extension jpg, png, or gif from $var1 into $var2. Unfortunately, that command fails if there happen to be no files with one of the given extensions, for example if we have several jpg files but no gifs.

So we have to make things a little more complicated, and our answer is the find command, as shown below. If you want to add more image file types, just add another set of -o -name '*.EXT' within the parentheses, where EXT is the file extension you want.

```<code>function pause(){

    read -p "$*"
}

echo "Help: Use the bar / to separate the directory names"
echo ""

echo "Enter the source directory"
read source

echo "Enter the destination directory"
read dest

find $source \( -name '*.jpg' -o -name '*.gif' -o -name '*.jpeg' -o -name '*.png' \) -exec cp -prv {} $dest \;

echo ""
pause "Press [ENTER] to continue..."
</code>```

The -prv flag on cp means preserve attributes, copy recursively, and use verbose mode (print what has been copied).

ContentLicense: CC BY-SA 4.0

Body:
I know that this type of questions maybe were duplicated. But in my case, I'm use a variable that contains the destination directory and the error says, that is not a valid directory.

cp target var2 is not a directory

Now, this is my code, I'm trying to copy all images from the folder images (contained in Desktop) to the folder photos (In desktop too and empty). I use GIT Bash for Windows 10

function pause(){

    read -p "$*"
}

echo "Help: Use the bar / to separate the directory names"
echo ""

echo "Enter the directory location"
read var1

echo "Enter the destinity directory location"
read var2

for var1 in "var2"
do
    cp * var1 var2
done

echo ""
pause "Press [ENTER] to continue..."

What's my mistake? Thanks!

When i execute the script, the directory I use is:

C:/Users/Mauri/Desktop/images

And the var2 is:

C:/Users/Mauri/Desktop/photos

With this code into the for:

cp -r * $var1 $var2

Was moved all archives in the desktop to photos folder, It's not my objetive, I want to move only the images contained on the image folder in the desktop to the empty folder photos in the desktop too.

Body:
Instead of your for loop, it would be great if we could run cp "$var1/"*.{jpg,png,gif} "$var2". That would be an easy way to copy all of the files with extension jpg, png, or gif from $var1 into $var2. Unfortunately, that command fails if there happen to be no files with one of the given extensions, for example if we have several jpg files but no gifs.

So we have to make things a little more complicated, and our answer is the find command, as shown below. If you want to add more image file types, just add another set of -o -name '*.EXT' within the parentheses, where EXT is the file extension you want.

function pause(){

    read -p "$*"
}

echo "Help: Use the bar / to separate the directory names"
echo ""

echo "Enter the source directory"
read source

echo "Enter the destination directory"
read dest

find $source \( -name '*.jpg' -o -name '*.gif' -o -name '*.jpeg' -o -name '*.png' \) -exec cp -prv {} $dest \;

echo ""
pause "Press [ENTER] to continue..."

The -prv flag on cp means preserve attributes, copy recursively, and use verbose mode (print what has been copied).


问:
Which Path should I use in my website? Related path absolute path?
I am building website for my client. I was wondering which path should I use for my website? which path is more secure? also correct me if I am totally wrong or if you have other reason for safety concerns for website! Thanks in advance.

For example...

```<code><script src="https://example.com/admin/js/myjavascript.js"></script>
or
<script src="/js/myjavascript.js"></script>

<img src="https://swadtiffin.ca/wp-content/uploads/myimg.jpg" alt="Loading...!">
or
<img src="/wp-content/uploads/myimg.jpg" alt="Loading...!">

$http({
  method: 'POST',
  url: '/wp-content/themes/astra-child/php/get_all_food_combos.php',
  data: $.param({ 'data': "test" }),
  headers: { 'Content-Type': 'application/x-www-form-urlencoded' }
  }).then(function (resp) {
  console.log(resp.data); //all items
  });
  }, function (error) {
     console.log(error);
  });

or

$http({
  method: 'POST',
  url: 'https://example.com/wp-content/themes/astra-child/php/get_combos.php',
  data: $.param({ 'data': "test" }),
  headers: { 'Content-Type': 'application/x-www-form-urlencoded' }
  }).then(function (resp) {
  console.log(resp.data); //all items
  });
  }, function (error) {
     console.log(error);
  });
</code>```

答:
As Anukul mentioned in this case both are okay.

The URI Paths generally do not effect the security of a an application. It is a matter of preference and following standards.

There are a couple of things to consider when determining paths though:

Is there sensitive data?

If so don't use path parameters or query
    parameters to send this sensitive data. If possible use a request
    payload in a POST with SSL/TLS.

GET methods are generally logged in server logs.

Would this be okay in this circumstance with the parameters?

Body:
I am building website for my client. I was wondering which path should I use for my website? which path is more secure? also correct me if I am totally wrong or if you have other reason for safety concerns for website! Thanks in advance.

For example...

or

or

$http({
  method: 'POST',
  url: '/wp-content/themes/astra-child/php/get_all_food_combos.php',
  data: $.param({ 'data': "test" }),
  headers: { 'Content-Type': 'application/x-www-form-urlencoded' }
  }).then(function (resp) {
  console.log(resp.data); //all items
  });
  }, function (error) {
     console.log(error);
  });

or

$http({
  method: 'POST',
  url: 'https://example.com/wp-content/themes/astra-child/php/get_combos.php',
  data: $.param({ 'data': "test" }),
  headers: { 'Content-Type': 'application/x-www-form-urlencoded' }
  }).then(function (resp) {
  console.log(resp.data); //all items
  });
  }, function (error) {
     console.log(error);
  });


问:
Webpack raw-loader errors out when require markdown file
raw-loader errors out when attempting to require any .md file.

Adding raw loader to import markdown files:

```<code>  test: /\.md$/i,
      use: [{
        loader: 'raw-loader',
        options: {
          esModule: false
        }
      }],
</code>```

In .js file, require the markdown file..

```<code>return require(postPath)
// postPath is '../posts/awards.md'
</code>```

```<code>Error: Cannot find module '../posts/awards.md'
at webpackEmptyContext (eval at <path to file>)
....
</code>```

the path to markdown file is the relative path: /posts/awards.md

If I change awards.md to awards.json it works. So maybe it is an issue with raw-loader looking for a export in awards.md and not find one, thus erroring out? Isn't the point of esModule: false to instruct Webpack to NOT treat it as module?

答:
Seems like you're having the same problem as this person.

Quoting an answer:

Webpack performs a static analyse at build time.

It doesn't try to infer variables which that import(test) could be anything, hence the failure.

It is also the case for import(path+"a.js").

If you need truly dynamic imports, you have to restrict it to a known path:

import("./locales/" + locale + ".js")

I re-created your problem and sure enough:

```<code>function test(mod) {
    return require(mod)
}

console.log(test("./test.md"))
</code>```

Doesn't work. HOWEVER this works:

```<code>function test(mod) {
    return require("./" + mod)
}

console.log(test("test.md"))
</code>```

So it should suffice to change your code to this:

```<code>return require("../" + postPath + ".md")
</code>```

And change postPath to:

```<code>// postPath is 'posts/awards'
</code>```

Body:
raw-loader errors out when attempting to require any .md file.

Adding raw loader to import markdown files:

  test: /\.md$/i,
      use: [{
        loader: 'raw-loader',
        options: {
          esModule: false
        }
      }],

In .js file, require the markdown file..

return require(postPath)
// postPath is '../posts/awards.md'

Error: Cannot find module '../posts/awards.md'
at webpackEmptyContext (eval at <path to file>)
....

the path to markdown file is the relative path: /posts/awards.md

If I change awards.md to awards.json it works. So maybe it is an issue with raw-loader looking for a export in awards.md and not find one, thus erroring out? Isn't the point of esModule: false to instruct Webpack to NOT treat it as module?

Body:
Seems like you're having the same problem as this person.

Quoting an answer:

Webpack performs a static analyse at build time.

It doesn't try to infer variables which that import(test) could be anything, hence the failure.

It is also the case for import(path+"a.js").

If you need truly dynamic imports, you have to restrict it to a known path:

import("./locales/" + locale + ".js")

I re-created your problem and sure enough:

function test(mod) {
    return require(mod)
}

console.log(test("./test.md"))

Doesn't work. HOWEVER this works:

function test(mod) {
    return require("./" + mod)
}

console.log(test("test.md"))

So it should suffice to change your code to this:

return require("../" + postPath + ".md")

And change postPath to:

// postPath is 'posts/awards'


问:
How to plot multiple lines with different X indices
I have 3 data frames as so:

Range is the index second Column is Number

print(df1)
Range
1.27     2386.0
0.93     5598.0
1.27     3607.0
1.29     2262.0
0.94    12227.0

print(df2)
Range
1.26     6410.0
1.27     5688.0
1.25     7329.0
0.93     7757.0
1.26     2118.0
1.26     5772.0

print(df2)
Range
0.92     3368.0
1.26     4935.0
1.28     4749.0
0.94    13716.0
0.92     8478.0
1.27     7997.0
0.92    12459.0
0.92     5805.0
0.92     7842.0
1.26     8316.0
1.27    11069.0
1.27    10011.0
```

<p>How can I plot a singe graph with the x-axis using the range and the y-axis using the second column?</p>

<p>I tried: </p>

<pre class="lang-py prettyprint-override"><code>df=pd.concat([df1,df2,df3], ignore_index=True, axis=1)
ax=sns.lineplot(data=df,  markers=True, dashes=False)
ax.set_title("Range vs Number")
ax.set(xlabel="Range (m)", ylabel = "Number")
plt.legend(loc='upper right', labels=['df1','df2','df3'])
plt.show()
</code>```

But go the following error:

```<code>ValueError: cannot reindex from a duplicate axis
</code>```

答:
Test DataFrames and Imports

Tested in python 3.11, pandas 1.5.2, matplotlib 3.6.2, seaborn 0.12.1

import pandas as pd
import matplotlib.pyplot as plt

d1 = {'Range': [0.93, 0.94, 1.27, 1.29],
      'Number': [5598.0, 12227.0, 3607.0, 2262.0]}
df1 = pd.DataFrame(d1).set_index('Range')

d2 = {'Range': [1.26, 1.27, 1.25, 0.93, 1.26, 1.26],
      'Number': [6410.0, 5688.0, 7329.0, 7757.0, 2118.0, 5772.0]}
df2 = pd.DataFrame(d2).set_index('Range')

d3 = {'Range': [0.92, 1.26, 1.28, 0.94, 0.92, 1.27, 0.92, 0.92, 0.92, 1.26, 1.27, 1.27],
      'Number': [3368.0, 4935.0, 4749.0, 13716.0, 8478.0, 7997.0, 12459.0, 5805.0, 7842.0, 8316.0, 11069.0, 10011.0]}
df3 = pd.DataFrame(d3).set_index('Range')
```
<h2>Combine DataFrames without an Identifying Column</h2>
<pre class="lang-py prettyprint-override"><code>df = pd.concat([df1, df2, df3])
</code>```

estimator='mean': default

As shown in the OP. When there are multiple points for a single index, the 'mean' is plotted and a confidence interval is shown.

ax = sns.lineplot(data=df, markers=True)
```
<p><img src="https://i.stack.imgur.com/fvAXE.png" alt="enter image description here" /></p>
<h3><code>estimator=None</code></h3>
<ul>
<li>A single line with all the points shown</li>
</ul>
<pre class="lang-py prettyprint-override"><code>ax = sns.lineplot(data=df, markers=True, estimator=None)
</code>```

Plot Each DataFrame in a Loop

matplotlib.pyplot.plot

for i, df in enumerate([df1, df2, df3], 1):
    df = df.sort_index()  # the index must be sorted
    plt.plot(df.index, df['Number'], label=f'df{i}', marker='.')
plt.legend()
plt.show()
```
<p><img src="https://i.stack.imgur.com/u3HFm.png" alt="enter image description here" /></p>
<h3><code>pandas.DataFrame.plot</code></h3>
<ul>
<li>Each DataFrame can be plotted directly with <code>df.plot</code> instead of <code>plt.plot</code>, but this option requires a little more setup.
<ul>
<li>Create an <code>ax</code> to plot each DataFrame to, or multiple figures will be created.</li>
<li>The legend labels must be updated after creating the plot.</li>
</ul>
</li>
</ul>
<pre class="lang-py prettyprint-override"><code>fig, ax = plt.subplots()
for i, df in enumerate([df1, df2, df3], 1):
    df = df.sort_index()
    df.plot(marker='.', ax=ax)

ax.legend(labels=[f'df{i}' for i in range(1, 4)])
plt.show()
</code>```

Combine DataFrames with an Identifying Column

# create a single dataframe with an identifying column
df = pd.concat([df.assign(id=f'df{i}') for i, df in enumerate([df1, df2, df3], 1)])

# reset the index
df = df.reset_index()

# plot using hue to separate the lines
ax = sns.lineplot(data=df, x='Range', y='Number', hue='id', estimator=None, marker='o', hue_order=['df1', 'df2', 'df3'])
```
<ul>
<li>Note the difference with <code>sns.lineplot</code> comparted to <code>plt.plot</code> in the loop. <code>seaborn</code> internally sorts the values for the x-axis, and the y-axis.
<ul>
<li><code>df = df.sort_values(['Range', 'Number'])</code></li>
</ul>
</li>
</ul>
<p><img src="https://i.stack.imgur.com/zMFDn.png" alt="enter image description here" /></p>
<ul>
<li>Adjusting the loop to reset the index, <strong>and</strong> sort by the <code>'Range'</code> (the <code>index</code>) and <code>'Number'</code>.</li>
</ul>
<pre class="lang-py prettyprint-override"><code>for i, df in enumerate([df1, df2, df3], 1):
    df = df.reset_index().sort_values(['Range', 'Number'])
    plt.plot('Range', 'Number', data=df, label=f'df{i}', marker='.')
plt.legend()
plt.show()
</code>```

Body:
I have 3 data frames as so:

Range is the index second Column is Number

print(df1)
Range
1.27     2386.0
0.93     5598.0
1.27     3607.0
1.29     2262.0
0.94    12227.0

print(df2)
Range
1.26     6410.0
1.27     5688.0
1.25     7329.0
0.93     7757.0
1.26     2118.0
1.26     5772.0

print(df2)
Range
0.92     3368.0
1.26     4935.0
1.28     4749.0
0.94    13716.0
0.92     8478.0
1.27     7997.0
0.92    12459.0
0.92     5805.0
0.92     7842.0
1.26     8316.0
1.27    11069.0
1.27    10011.0

How can I plot a singe graph with the x-axis using the range and the y-axis using the second column?

I tried:

df=pd.concat([df1,df2,df3], ignore_index=True, axis=1)
ax=sns.lineplot(data=df,  markers=True, dashes=False)
ax.set_title("Range vs Number")
ax.set(xlabel="Range (m)", ylabel = "Number")
plt.legend(loc='upper right', labels=['df1','df2','df3'])
plt.show()

But go the following error:

ValueError: cannot reindex from a duplicate axis

Body:
Test DataFrames and Imports

Tested in python 3.11, pandas 1.5.2, matplotlib 3.6.2, seaborn 0.12.1

import pandas as pd
import matplotlib.pyplot as plt

d1 = {'Range': [0.93, 0.94, 1.27, 1.29],
      'Number': [5598.0, 12227.0, 3607.0, 2262.0]}
df1 = pd.DataFrame(d1).set_index('Range')

d2 = {'Range': [1.26, 1.27, 1.25, 0.93, 1.26, 1.26],
      'Number': [6410.0, 5688.0, 7329.0, 7757.0, 2118.0, 5772.0]}
df2 = pd.DataFrame(d2).set_index('Range')

d3 = {'Range': [0.92, 1.26, 1.28, 0.94, 0.92, 1.27, 0.92, 0.92, 0.92, 1.26, 1.27, 1.27],
      'Number': [3368.0, 4935.0, 4749.0, 13716.0, 8478.0, 7997.0, 12459.0, 5805.0, 7842.0, 8316.0, 11069.0, 10011.0]}
df3 = pd.DataFrame(d3).set_index('Range')

Combine DataFrames without an Identifying Column

df = pd.concat([df1, df2, df3])

estimator='mean': default

As shown in the OP. When there are multiple points for a single index, the 'mean' is plotted and a confidence interval is shown.

ax = sns.lineplot(data=df, markers=True)

estimator=None

A single line with all the points shown

ax = sns.lineplot(data=df, markers=True, estimator=None)

Plot Each DataFrame in a Loop

matplotlib.pyplot.plot

for i, df in enumerate([df1, df2, df3], 1):
    df = df.sort_index()  # the index must be sorted
    plt.plot(df.index, df['Number'], label=f'df{i}', marker='.')
plt.legend()
plt.show()

pandas.DataFrame.plot

Each DataFrame can be plotted directly with df.plot instead of plt.plot, but this option requires a little more setup.

Create an ax to plot each DataFrame to, or multiple figures will be created.

The legend labels must be updated after creating the plot.

fig, ax = plt.subplots()
for i, df in enumerate([df1, df2, df3], 1):
    df = df.sort_index()
    df.plot(marker='.', ax=ax)

ax.legend(labels=[f'df{i}' for i in range(1, 4)])
plt.show()

Combine DataFrames with an Identifying Column

# create a single dataframe with an identifying column
df = pd.concat([df.assign(id=f'df{i}') for i, df in enumerate([df1, df2, df3], 1)])

# reset the index
df = df.reset_index()

# plot using hue to separate the lines
ax = sns.lineplot(data=df, x='Range', y='Number', hue='id', estimator=None, marker='o', hue_order=['df1', 'df2', 'df3'])

Note the difference with sns.lineplot comparted to plt.plot in the loop. seaborn internally sorts the values for the x-axis, and the y-axis.

df = df.sort_values(['Range', 'Number'])

Adjusting the loop to reset the index, and sort by the 'Range' (the index) and 'Number'.

for i, df in enumerate([df1, df2, df3], 1):
    df = df.reset_index().sort_values(['Range', 'Number'])
    plt.plot('Range', 'Number', data=df, label=f'df{i}', marker='.')
plt.legend()
plt.show()


问:
Detect cycles in a genealogy graph during a Depth-first search
I'm loading horse genealogical data recursively.
For some wrong sets of data my recursion never stops... and that is because  there are cycles in the data.

How can I detect those cycles to stop recurring?

I thought of while recurring maintain a hashTable with all "visited" horses.
But that will find some false positives, because a horse CAN be twice in a tree.

What cannot happen is that a horse appear as a father or grandfather or great grandfather of ITSELF.

答:
You're dealing with a directed acyclic graph, not a tree. There should not be any cycles, as a horse's descendant cannot also be its ancestor.

Knowing this, you should apply code techniques specific to directed acyclic graphs.

OwnerDisplayName: Romias

ContentLicense: CC BY-SA 3.0

ContentLicense: CC BY-SA 2.5


问:
Finding a element with a fake attribute that was added
I added a fake attribute to all my elements on the page, for example myId=100, each element has a different fake id.

Now i need to find an element according to the fake id, is this possible?

I tried doing

```<code>$("#myFrame").contents().find('a').each(function () {
    if ($(this).attr['myid'] === 100) {
        $(this).hide();
    }
});
</code>```

Any better ideas?

thank you

答:
Instead of adding a custom attribute, do it the HTML5 way:

```<code><span data-id="blahblah">Foo</span>

alert($(this).data("id")); // blahblah
</code>```

See http://api.jquery.com/data/

Body:
I added a fake attribute to all my elements on the page, for example myId=100, each element has a different fake id.

Now i need to find an element according to the fake id, is this possible?

I tried doing

$("#myFrame").contents().find('a').each(function () {
    if ($(this).attr['myid'] === 100) {
        $(this).hide();
    }
});

Any better ideas?

thank you

Body:
Instead of adding a custom attribute, do it the HTML5 way:

<span data-id="blahblah">Foo</span>

alert($(this).data("id")); // blahblah

See http://api.jquery.com/data/


问:
Can't find method in module
I am learning Rust and trying to figure out why I can't call a method from a module I have written. The code is very simple as it's a learning exercise and I have read this and this but I still don't know what the problem is. I have also tried using the code from this question almost exactly however the error persists.

The code for main.rs and helper.rs are below.

main.rs:

```<code>mod helpers;

fn main() {
    let h = helpers::new("TestString");
    println!("{}", h.get_name().to_string());
    helpers::test_func();
}
</code>```

helpers.rs

```<code>struct Helper{
    name: String
}

impl Helper {
    pub fn new(name: &String) -> Helper {
        let helper = Helper{ name: name.to_string()};
        return helper;
    }
}

impl Helper {
    pub fn get_name(&self) -> &String {
        return &self.name;
    }
}

pub fn test_func(){
    println!("test_func was called.");
}
</code>```

The error I am getting is:

```<code>error[E0425]: cannot find function `new` in module `helpers`
 --> src/main.rs:4:30
  |
4 |     let h = helpers::new("TestString");
  |                              ^^^ not found in `helpers`

error: aborting due to previous error
</code>```

I know the file is being read because if you commend out lines 4 & 5 from main.rs and just run helpers::test_func() it works fine however I don't know why it can't find the new method.

Any help would be very much appreciated.

答:
First thing is the Helper struct needs to be exposed from the module. You can do this with the pub keyword.

```<code>pub struct Helper { ...
</code>```

The other thing is that helpers is the name of the module, but the new method is associated with the Helper struct.
You can import the struct with

```<code>mod helpers;
use helpers::Helper;
</code>```

and then call it with

```<code>Helper::new(&String::from(""))
</code>```

side note:
You can change the parameter type on new from &String to &strto avoid having to make a new String object and saving a pointless copy.

```<code>pub fn new(name: &str) -> Helper {
    Helper{
        name: name.to_string()
    }
}
</code>```

then call it with

```<code>Helper::new("")
</code>```

Title: Can't find method in module

Body:
I am learning Rust and trying to figure out why I can't call a method from a module I have written. The code is very simple as it's a learning exercise and I have read this and this but I still don't know what the problem is. I have also tried using the code from this question almost exactly however the error persists.

The code for main.rs and helper.rs are below.

main.rs:

mod helpers;

fn main() {
    let h = helpers::new("TestString");
    println!("{}", h.get_name().to_string());
    helpers::test_func();
}

helpers.rs

struct Helper{
    name: String
}

impl Helper {
    pub fn new(name: &String) -> Helper {
        let helper = Helper{ name: name.to_string()};
        return helper;
    }
}

impl Helper {
    pub fn get_name(&self) -> &String {
        return &self.name;
    }
}

pub fn test_func(){
    println!("test_func was called.");
}

The error I am getting is:

error[E0425]: cannot find function `new` in module `helpers`
 --> src/main.rs:4:30
  |
4 |     let h = helpers::new("TestString");
  |                              ^^^ not found in `helpers`

error: aborting due to previous error

I know the file is being read because if you commend out lines 4 & 5 from main.rs and just run helpers::test_func() it works fine however I don't know why it can't find the new method.

Any help would be very much appreciated.

Body:
First thing is the Helper struct needs to be exposed from the module. You can do this with the pub keyword.

pub struct Helper { ...

The other thing is that helpers is the name of the module, but the new method is associated with the Helper struct.
You can import the struct with

mod helpers;
use helpers::Helper;

and then call it with

Helper::new(&String::from(""))

side note:
You can change the parameter type on new from &String to &strto avoid having to make a new String object and saving a pointless copy.

pub fn new(name: &str) -> Helper {
    Helper{
        name: name.to_string()
    }
}

then call it with

Helper::new("")


问:
What's the recommended approach for building a complete data frame (feature values + names) after using ColumTransformer/ FeatureUnion?
I have seen this topic showing many times on the Internet but never have a seen a complete, comprehensive solution that would work across the board for all use cases with the current library versions of sklearn. Could somebody please try to explain how that should be achieved using the following example?

In this example I'm using the following dataset

data = pd.read_csv('heart.csv')

# Preparing individual pipelines for numerical and categorical features
pipe_numeric = Pipeline(steps=[
    ('impute_num', SimpleImputer(
        missing_values = np.nan,
        strategy = 'median',
        copy = False,
        add_indicator = True)
    )
])

pipe_categorical = Pipeline(steps=[
    ('impute_cat', SimpleImputer(
        missing_values = np.nan,
        strategy = 'constant',
        fill_value = 99999,
        copy = False)
    ),
    ('one_hot', OneHotEncoder(handle_unknown='ignore'))
])

# Combining them into a transformer
transformer_union = ColumnTransformer([
    ('feat_numeric', pipe_numeric, ['age']),
    ('feat_categorical', pipe_categorical, ['cp']),
], remainder = 'passthrough')

# Fitting the transformer
transformer_union.fit(data)

# We can then apply and get the data in the following way
transformer_union.transform(data)

# And it has the following shape
transformer_union.transform(data).shape
```

<p>Now comes the main question: how to efficiently combine the output numpy array with the new column names that resulted from all the transformations? This example, even though would require quite some work, is still relatively simple, but this can get severely more complicated with bigger pipelines.</p>

<pre class="lang-py prettyprint-override"><code># Transformers object
transformers = transformer_union.named_transformers_

# Categorical features (from transformer)
transformers['feat_categorical'].named_steps['one_hot'].get_feature_names()

# Numerical features (from transformer) - no names are available?
transformers['feat_numeric'].named_steps['impute_num']

# All the other columns that were not transformed - no names are available?
transformers['remainder']
</code>```

I've checked all kind of different examples and there doesn't seem to be any silver bullet for this:

sklearn doesn't support this natively - there's no way to get an aligned vector of column names that could be easily combined with the array into a new DF, but perhaps I'm mistaken - could anyone point me to a resource if that's the case?

Some people were implementing their custom transformers/ pipelines, but this gets a bit hectic when you want to build large pipelines

Are there any other sklearn-related packages that alleviate that issue?

I'm a little bit surprised by how sklearn manages that - in R in the tidymodels ecosystem (it's still under development, but nevertheless),  this is handled very easily with the prep and bake methods. I would imagine it could somehow be done similarly.

Inspecting the final output in its entirety is vital to the data science work - could anyone advise on the best path?

答:
The sklearn devs are working on this; discussion spans several SLEPs and many Issues.  There is already some progress, with some transformers implementing get_features_names and others having internal attributes tracking column names when the input was a pandas dataframe.  ColumnTransformer does have a get_feature_names, but Pipeline does not, so that it would fail on your example.

The most complete current solution seems to be sklearn-pandas:

https://github.com/scikit-learn-contrib/sklearn-pandas

Another interesting approach is hidden away inside eli5.  In their explain_weights, they have a generic function transform_feature_names.  It has a few specialized dispatches, but otherwise tries to call get_feature_names; most notably, there is a dispatch for Pipeline.  Unfortunately, currently this will fail on a ColumnTransformer with a Pipeline as a transformer; see https://stackoverflow.com/a/62124484/10495893 for an example and a potential workaround.

Body:
I have seen this topic showing many times on the Internet but never have a seen a complete, comprehensive solution that would work across the board for all use cases with the current library versions of sklearn. Could somebody please try to explain how that should be achieved using the following example?

In this example I'm using the following dataset

data = pd.read_csv('heart.csv')

# Preparing individual pipelines for numerical and categorical features
pipe_numeric = Pipeline(steps=[
    ('impute_num', SimpleImputer(
        missing_values = np.nan,
        strategy = 'median',
        copy = False,
        add_indicator = True)
    )
])

pipe_categorical = Pipeline(steps=[
    ('impute_cat', SimpleImputer(
        missing_values = np.nan,
        strategy = 'constant',
        fill_value = 99999,
        copy = False)
    ),
    ('one_hot', OneHotEncoder(handle_unknown='ignore'))
])

# Combining them into a transformer
transformer_union = ColumnTransformer([
    ('feat_numeric', pipe_numeric, ['age']),
    ('feat_categorical', pipe_categorical, ['cp']),
], remainder = 'passthrough')

# Fitting the transformer
transformer_union.fit(data)

# We can then apply and get the data in the following way
transformer_union.transform(data)

# And it has the following shape
transformer_union.transform(data).shape

Now comes the main question: how to efficiently combine the output numpy array with the new column names that resulted from all the transformations? This example, even though would require quite some work, is still relatively simple, but this can get severely more complicated with bigger pipelines.

# Transformers object
transformers = transformer_union.named_transformers_

# Categorical features (from transformer)
transformers['feat_categorical'].named_steps['one_hot'].get_feature_names()

# Numerical features (from transformer) - no names are available?
transformers['feat_numeric'].named_steps['impute_num']

# All the other columns that were not transformed - no names are available?
transformers['remainder']

I've checked all kind of different examples and there doesn't seem to be any silver bullet for this:

sklearn doesn't support this natively - there's no way to get an aligned vector of column names that could be easily combined with the array into a new DF, but perhaps I'm mistaken - could anyone point me to a resource if that's the case?

Some people were implementing their custom transformers/ pipelines, but this gets a bit hectic when you want to build large pipelines

Are there any other sklearn-related packages that alleviate that issue?

I'm a little bit surprised by how sklearn manages that - in R in the tidymodels ecosystem (it's still under development, but nevertheless),  this is handled very easily with the prep and bake methods. I would imagine it could somehow be done similarly.

Inspecting the final output in its entirety is vital to the data science work - could anyone advise on the best path?


问:
Node.js not executing nested for loop correctly as it is asynchronous
```<code>let data = {};
sql.query("SELECT * FROM courses WHERE name = ?", [name], function(err, course) {
  if (err) result(null, err);
  else {
    data.id = course[0].id;
    data.course = course[0].name;
    data.terms = []

    sql.query("SELECT * FROM terms WHERE course_id = ?", [course[0].id], function(err, term) {
      if (err) result(null, err);
      else {
        for (let i = 0; i < term.length; i++) {
          data.terms.push({
            id: term[i].id,
            name: term[i].name,
            modules: []
          });
          sql.query("SELECT * FROM modules WHERE term_id = ?", [term[i].id], function(err, module) {
            if (err) result(null, err);
            else {
              for (let j = 0; j < module.length; j++) {
                data.terms[i].modules.push({
                  id: module[j].id,
                  name: module[j].name,
                  topics: []
                });
                sql.query("SELECT * FROM topics WHERE module_id = ?", [module[j].id], function(err, topic) {
                  if (err) result(null, err);
                  else {
                    for (let k = 0; k < topic.length; k++) {
                      data.terms[i].modules[j].topics.push({
                        id: topic[k].id,
                        name: topic[k].name
                      });
                    }
                    if (i === term.length - 1) result(null, data);
                  }
                });
              }
            }
          });
        }
      }
    });
  }
});
</code>```

答:
I suggest you to use Bluebird package. Here is the link to npm page https://www.npmjs.com/package/bluebird Have you tried it before?

Also you can see the oficial docs page here http://bluebirdjs.com/docs/getting-started.html

Here is an snippet from some project that I was working on recently. I'll paste it here as an example.

Inside a while loop Bluebird iterates over a collection of players and wait (one second) before moving to the next array item:

Don't forget to import bluebird first: const Promise = require('bluebird');

```<code>while(someCondition) {
  await Promise.each(players, async (p) => {
    const team = this.getTeamByPlayer(p);
    logger.info(`[${team.name}] ${p.nombre} is making a move`);
    p.jugar();
    await this.wait(1000);
  })
 }
</code>```

this.wait(n)is a class method that implements an async timeout

As you can see I'm using async/await over the iteration and inside the loop as well.

Bluebird also has a Promisify method that can help you to write code in a "async/await" way making your code clearest!

Let me know if you have any question!

Hope it helps!

Lucas.-

Body:
I suggest you to use Bluebird package. Here is the link to npm page https://www.npmjs.com/package/bluebird Have you tried it before?

Also you can see the oficial docs page here http://bluebirdjs.com/docs/getting-started.html

Here is an snippet from some project that I was working on recently. I'll paste it here as an example.

Inside a while loop Bluebird iterates over a collection of players and wait (one second) before moving to the next array item:

Don't forget to import bluebird first: const Promise = require('bluebird');

while(someCondition) {
  await Promise.each(players, async (p) => {
    const team = this.getTeamByPlayer(p);
    logger.info(`[${team.name}] ${p.nombre} is making a move`);
    p.jugar();
    await this.wait(1000);
  })
 }

this.wait(n)is a class method that implements an async timeout

As you can see I'm using async/await over the iteration and inside the loop as well.

Bluebird also has a Promisify method that can help you to write code in a "async/await" way making your code clearest!

Let me know if you have any question!

Hope it helps!

Lucas.-


问:
async bcrypt post request returns no response
I am trying to compare password in order to login user, but i keep getting "this request has no response."

this is my post route.

```<code>router.route("/login/:username1").post(async (req, res) => {
  const userexist = User.find(username=>userexist.username=req.body.username1)
      if(userexist == null)
      {
        return res.status(400).json("Cannot find user");
      }
      try
      {
        if(await bcrypt.compare(req.body.password, userexist.password))
        {
          res.json("success");
        }
        else
        {
          res.json("Not Allowed");
        }
      } catch {
        res.status(500).json();
      }
});
</code>```

this is my post method from my login page.

```<code>onSubmit(e) {
    e.preventDefault();
    const isValid = this.validate();
    const username1=this.state.username;
    if (isValid) {
      const user = {
        username: this.state.username,
        password: this.state.password,
      };

      axios
      .post("http://localhost:5000/users/login/"+ username1, user)
      .then((res) => console.log(res.data));
      this.setState(initialState);
    }
  }
</code>```

答:
Make few corrections and see how you go:

use await in your controller

With below line of code, you are sending user object. So in your controller, use req.body.username (not req.body.username1)

axios.post("http://localhost:5000/users/login/"+ username1, user)
```

<ul>
<li>make correction to your <code>find</code> function as shown in the code snippet</li>
<li>also check your cors and make sure that client is adhering server cors rules</li>
</ul>

```router.route("/login/:username1").post(async (req, res) => {
    const userexist = await User.find(user=>user.username==req.body.username) //<---- make correction to find method ... use await and use correct req body object i.e. username not username1
        if(userexist == null)
        {
          return res.status(400).json("Cannot find user");
        }
        try
        {
          if(await bcrypt.compare(req.body.password, userexist.password))
          {
            res.send("success"); //<----- as best practice, while sending strings just use res.send('success')... res.json is not reqd
          }
          else
          {
            res.send("Not Allowed"); //<----- for strings, it is better to use res.send instead of res.json
          }
        } catch {
          res.status(500).json(); //<----- remember to pass meaningful error message back to client
        }
  });
```

Body:
I am trying to compare password in order to login user, but i keep getting "this request has no response."

this is my post route.

router.route("/login/:username1").post(async (req, res) => {
  const userexist = User.find(username=>userexist.username=req.body.username1)
      if(userexist == null)
      {
        return res.status(400).json("Cannot find user");
      }
      try
      {
        if(await bcrypt.compare(req.body.password, userexist.password))
        {
          res.json("success");
        }
        else
        {
          res.json("Not Allowed");
        }
      } catch {
        res.status(500).json();
      }
});

this is my post method from my login page.

onSubmit(e) {
    e.preventDefault();
    const isValid = this.validate();
    const username1=this.state.username;
    if (isValid) {
      const user = {
        username: this.state.username,
        password: this.state.password,
      };

      axios
      .post("http://localhost:5000/users/login/"+ username1, user)
      .then((res) => console.log(res.data));
      this.setState(initialState);
    }
  }

Body:
Make few corrections and see how you go:

use await in your controller

With below line of code, you are sending user object. So in your controller, use req.body.username (not req.body.username1)

axios.post("http://localhost:5000/users/login/"+ username1, user)

make correction to your find function as shown in the code snippet

also check your cors and make sure that client is adhering server cors rules

router.route("/login/:username1").post(async (req, res) => {
    const userexist = await User.find(user=>user.username==req.body.username) //<---- make correction to find method ... use await and use correct req body object i.e. username not username1
        if(userexist == null)
        {
          return res.status(400).json("Cannot find user");
        }
        try
        {
          if(await bcrypt.compare(req.body.password, userexist.password))
          {
            res.send("success"); //<----- as best practice, while sending strings just use res.send('success')... res.json is not reqd
          }
          else
          {
            res.send("Not Allowed"); //<----- for strings, it is better to use res.send instead of res.json
          }
        } catch {
          res.status(500).json(); //<----- remember to pass meaningful error message back to client
        }
  });


问:
Adonis ace migration on Docker - SyntaxError: Unexpected identifier
I'm getting this error "SyntaxError: Unexpected identifier" when I try to run node ace migration:run --force

```<code>/app/startup.sh:3
. node ace migration:run --force
       ^^^

SyntaxError: Unexpected identifier
    at wrapSafe (internal/modules/cjs/loader.js:1054:16)
    at Module._compile (internal/modules/cjs/loader.js:1102:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)
    at Module.load (internal/modules/cjs/loader.js:986:32)
    at Function.Module._load (internal/modules/cjs/loader.js:879:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47
</code>```

Dockerfile

```<code>FROM node:12 AS bizi
COPY . /app
WORKDIR /app
RUN ["npm", "i"]
</code>```

I start the container with a startup.sh shell script, this is where the error occurs:

#!/bin/sh
cd /app
node ace migration:run --force
adonis serve
```

<p>Build and start the container:</p>

<pre class="lang-sh prettyprint-override"><code>docker build -t myapp .
docker run -p 8080:80 myapp startup.sh
</code>```

Here are all my dependencies for my package.json file:

```<code>  "dependencies": {
    "@adonisjs/ace": "^5.0.8",
    "@adonisjs/ally": "^2.1.3",
    "@adonisjs/auth": "^3.0.7",
    "@adonisjs/bodyparser": "^2.0.5",
    "@adonisjs/cors": "^1.0.7",
    "@adonisjs/fold": "^4.0.9",
    "@adonisjs/framework": "^5.0.13",
    "@adonisjs/generic-exceptions": "^3.0.1",
    "@adonisjs/ignitor": "^2.0.8",
    "@adonisjs/lucid": "^6.1.3",
    "@adonisjs/mail": "^3.0.9",
    "@adonisjs/validator": "^5.0.6",
    "@azure/service-bus": "^1.1.7",
    "axios": "^0.19.0",
    "cpf-check": "^3.0.0",
    "firebase-admin": "^8.3.0",
    "moment": "^2.24.0",
    "order-id": "^1.1.0",
    "pg": "^7.12.0",
    "require-dir": "^1.2.0",
    "uuid": "^3.3.2"
  }
</code>```

If I put the ace commands within the Dockerfile they seem to be OK.

Perhaps it is a linux configuration that I'm missing?

```<code>FROM node:12 AS bizi
COPY . /app
WORKDIR /app
RUN ["npm", "i"]

# Added ace commands
RUN node ace
RUN node ace migration:run --force
</code>```

This is the docker build output with the ace commands inside, the problem seems to happen only when I run the shell.

```<code>Step 5/6 : RUN node ace --help
 ---> Running in f9bd319e8031
Usage:
  command [arguments] [options]

Global Options:
  --env                  Set NODE_ENV before running the commands
  --no-ansi              Disable colored output
  --version              output the version number

Available Commands:
  seed                   Seed database using seed files
 connect
  connect:fridge-queue   Tell something helpful about this command
 make
  make:validator         Make route validator
 migration
  migration:refresh      Refresh migrations by performing rollback and then running from start
  migration:reset        Rollback migration to the first batch
  migration:rollback     Rollback migration to latest batch or to a specific batch number
  migration:run          Run all pending migrations
  migration:status       Check migrations current status
</code>```

I was getting a similar error with @adonisjs/cli when trying to run the migration.

答:
I switched from the node image to ubuntu and configured the node installation myself to solve the problem.

```<code>FROM ubuntu:latest
USER root

RUN apt-get update
RUN apt -y upgrade
RUN apt-get -y install curl gnupg
RUN curl -sL https://deb.nodesource.com/setup_12.x  | bash -
RUN apt-get -y install nodejs

COPY . /app
WORKDIR /app
RUN npm i -g @adonisjs/cli
RUN npm i
``
</code>```

Body:
I'm getting this error "SyntaxError: Unexpected identifier" when I try to run node ace migration:run --force

/app/startup.sh:3
. node ace migration:run --force
       ^^^

SyntaxError: Unexpected identifier
    at wrapSafe (internal/modules/cjs/loader.js:1054:16)
    at Module._compile (internal/modules/cjs/loader.js:1102:27)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1158:10)
    at Module.load (internal/modules/cjs/loader.js:986:32)
    at Function.Module._load (internal/modules/cjs/loader.js:879:14)
    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
    at internal/main/run_main_module.js:17:47

Dockerfile

FROM node:12 AS bizi
COPY . /app
WORKDIR /app
RUN ["npm", "i"]

I start the container with a startup.sh shell script, this is where the error occurs:

#!/bin/sh
cd /app
node ace migration:run --force
adonis serve

Build and start the container:

docker build -t myapp .
docker run -p 8080:80 myapp startup.sh

Here are all my dependencies for my package.json file:

  "dependencies": {
    "@adonisjs/ace": "^5.0.8",
    "@adonisjs/ally": "^2.1.3",
    "@adonisjs/auth": "^3.0.7",
    "@adonisjs/bodyparser": "^2.0.5",
    "@adonisjs/cors": "^1.0.7",
    "@adonisjs/fold": "^4.0.9",
    "@adonisjs/framework": "^5.0.13",
    "@adonisjs/generic-exceptions": "^3.0.1",
    "@adonisjs/ignitor": "^2.0.8",
    "@adonisjs/lucid": "^6.1.3",
    "@adonisjs/mail": "^3.0.9",
    "@adonisjs/validator": "^5.0.6",
    "@azure/service-bus": "^1.1.7",
    "axios": "^0.19.0",
    "cpf-check": "^3.0.0",
    "firebase-admin": "^8.3.0",
    "moment": "^2.24.0",
    "order-id": "^1.1.0",
    "pg": "^7.12.0",
    "require-dir": "^1.2.0",
    "uuid": "^3.3.2"
  }

If I put the ace commands within the Dockerfile they seem to be OK.

Perhaps it is a linux configuration that I'm missing?

FROM node:12 AS bizi
COPY . /app
WORKDIR /app
RUN ["npm", "i"]

# Added ace commands
RUN node ace
RUN node ace migration:run --force

This is the docker build output with the ace commands inside, the problem seems to happen only when I run the shell.

Step 5/6 : RUN node ace --help
 ---> Running in f9bd319e8031
Usage:
  command [arguments] [options]

Global Options:
  --env                  Set NODE_ENV before running the commands
  --no-ansi              Disable colored output
  --version              output the version number

Available Commands:
  seed                   Seed database using seed files
 connect
  connect:fridge-queue   Tell something helpful about this command
 make
  make:validator         Make route validator
 migration
  migration:refresh      Refresh migrations by performing rollback and then running from start
  migration:reset        Rollback migration to the first batch
  migration:rollback     Rollback migration to latest batch or to a specific batch number
  migration:run          Run all pending migrations
  migration:status       Check migrations current status

I was getting a similar error with @adonisjs/cli when trying to run the migration.

Body:
I switched from the node image to ubuntu and configured the node installation myself to solve the problem.

FROM ubuntu:latest
USER root

RUN apt-get update
RUN apt -y upgrade
RUN apt-get -y install curl gnupg
RUN curl -sL https://deb.nodesource.com/setup_12.x  | bash -
RUN apt-get -y install nodejs

COPY . /app
WORKDIR /app
RUN npm i -g @adonisjs/cli
RUN npm i
``


问:
Accessing key property of React component

I map through an array of locations

I want to access the key of the component via onClick

This is my console.logconst addTo = (element) => {console.log(element);};

```<code>return (
    <>
      {data.map((item) => {

        return (
          <Marker key={Math.random()} position={item.fields.location}>
            <Popup>
              {item.fields.name} <br></br>
               <button
                onClick={() => {
                  addTo(item.fields.name);
                }}
              >Add
              </button>
            </Popup>
          </Marker>
        );
      })}
    </>
  );
}
</code>```

答:
From the React docs:

Keys serve as a hint to React but they don’t get passed to your components. If you need the same value in your component, pass it explicitly as a prop with a different name:

Therefore you need to either store the key as a variable or pass it as another prop name

return (
    <>
      {data.map((item, index) => {
        const key = index;
        return (
          <Marker key={key} position={item.fields.location}>
            <Popup>
              {item.fields.name}

               <button
                onClick={() => {
                  addTo(item.fields.name);
                  console.log(key)
                }}
              >Add
              </button>
            </Popup>
          </Marker>
        );
      })}
    </>
  );
}
```

(Also you should consider using another key other than Math.random, as that changes with every render of the component)

If you want to access the element itself, you should consider using an attribute (i.e. data-id={index}) or better yet a ref. This answer explains how to create multiple refs

Body:
I map through an array of locations

I want to access the key of the component via onClick

This is my console.logconst addTo = (element) => {console.log(element);};

return (
    <>
      {data.map((item) => {

        return (
          <Marker key={Math.random()} position={item.fields.location}>
            <Popup>
              {item.fields.name}

               <button
                onClick={() => {
                  addTo(item.fields.name);
                }}
              >Add
              </button>
            </Popup>
          </Marker>
        );
      })}
    </>
  );
}

Body:
From the React docs:

Keys serve as a hint to React but they don’t get passed to your components. If you need the same value in your component, pass it explicitly as a prop with a different name:

Therefore you need to either store the key as a variable or pass it as another prop name

return (
    <>
      {data.map((item, index) => {
        const key = index;
        return (
          <Marker key={key} position={item.fields.location}>
            <Popup>
              {item.fields.name} <br></br>
               <button
                onClick={() => {
                  addTo(item.fields.name);
                  console.log(key)
                }}
              >Add
              </button>
            </Popup>
          </Marker>
        );
      })}
    </>
  );
}

(Also you should consider using another key other than Math.random, as that changes with every render of the component)

If you want to access the element itself, you should consider using an attribute (i.e. data-id={index}) or better yet a ref. This answer explains how to create multiple refs


问:
Pyspark: K means clustering error at model fittting
While running K means clustering using pyspark, I am using the following lines of code to find the optimal K value. But there is some error constantly popping up at the model fitting line.

The preprocessing stages included removing NAs and label encoding,

```<code>from pyspark.sql.functions import when,col

#Encode column "Potential" from dataframe df
high = list(range(86,101))
middle = list(range (71, 86))
low = list(range(56, 71))
very_low = list(range(45,56))

#checking for NULL
df = df.where(col("Potential").isNotNull())

#encoding the column potential
df1 = df.withColumn("Potential_Grading", when(col("Potential").isin(high), "high").
                    when(col("Potential").isin(middle), "Middle").
                    when(col("Potential").isin(low), "Low").
                    when(col("Potential").isin(very_low),"Verylow"))
df_kmeans = df1.select("ID","Height(CM)","Weight(KG)", "Crossing", "Finishing", "HeadingAccuracy", "ShortPassing",
          "Volleys", "Dribbling", "Curve", "FKAccuracy", "LongPassing", "BallControl", "Acceleration",
          "SprintSpeed", "Agility", "Reactions", "Balance", "ShotPower", "Jumping", "Stamina", "Strength",
          "LongShots", "Aggression", "Interceptions", "Vision", "Penalties", "Composure", "Positioning",
          "Marking", "StandingTackle", "SlidingTackle","Potential_Grading")

FEATURES_COL1 = ['Height(CM)', 'Weight(KG)',
                      'Crossing', 'Finishing', 'HeadingAccuracy',
                      'ShortPassing', 'Volleys', 'Dribbling', 'Curve',
                      'FKAccuracy', 'LongPassing', 'BallControl',
                      'Acceleration', 'SprintSpeed', 'Agility',
                      'Reactions', 'Balance', 'ShotPower', 'Jumping',
                      'Stamina', 'Strength', 'LongShots', 'Aggression',
                      'Interceptions', 'Vision', 'Penalties','Positioning',
                      'Composure', 'Marking', 'StandingTackle', 'SlidingTackle']

from pyspark.ml.clustering import KMeans
from pyspark.ml.feature import VectorAssembler

#Vector assembler is a transformer which takes all of the columns specified under FEATURES_COL and combines them into a new vector column.
vecAssembler = VectorAssembler(inputCols=FEATURES_COL1, outputCol="features")

#df_kmeans_1 = vecAssembler.transform(df_kmeans).select('ID','features')
df_kmeans_1 = vecAssembler.transform(df_kmeans).select('ID','features')
</code>```

The lines of code for fitting K means

```<code>cost = np.zeros(21)

    from pyspark.ml.clustering import KMeans

for k in range(2,21):

      # Trains a k-means model.
      kmeans = KMeans().setK(k).setFeaturesCol("features").setSeed(1)
      model1 = kmeans.fit(df_kmeans_1)

      #Evaluate clustering by computing Within Set Sum of Squared Errors.
      cost[k] = model1.computeCost(df_kmeans_1)
</code>```

I am getting an error at the model fitting line of code:

```<code>    ---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-15-2840daf75485> in <module>()
      8   # Trains a k-means model.
      9   kmeans = KMeans().setK(k).setFeaturesCol("features").setSeed(1)
---> 10   model1 = kmeans.fit(df_kmeans_1)
     11
     12   #Evaluate clustering by computing Within Set Sum of Squared Errors.

5 frames
/content/spark-2.4.0-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o346.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 15.0 failed 1 times, most recent failure: Lost task 0.0 in stage 15.0 (TID 18, localhost, executor driver): org.apache.spark.SparkException: Failed to execute user defined function($anonfun$4: (struct<Height(CM):double,Weight(KG):double,Crossing_double_VectorAssembler_f596de6f26ae:double,Finishing_double_VectorAssembler_f596de6f26ae:double,HeadingAccuracy_double_VectorAssembler_f596de6f26ae:double,ShortPassing_double_VectorAssembler_f596de6f26ae:double,Volleys_double_VectorAssembler_f596de6f26ae:double,Dribbling_double_VectorAssembler_f596de6f26ae:double,Curve_double_VectorAssembler_f596de6f26ae:double,FKAccuracy_double_VectorAssembler_f596de6f26ae:double,LongPassing_double_VectorAssembler_f596de6f26ae:double,BallControl_double_VectorAssembler_f596de6f26ae:double,Acceleration_double_VectorAssembler_f596de6f26ae:double,SprintSpeed_double_VectorAssembler_f596de6f26ae:double,Agility_double_VectorAssembler_f596de6f26ae:double,Reactions_double_VectorAssembler_f596de6f26ae:double,Balance_double_VectorAssembler_f596de6f26ae:double,ShotPower_double_VectorAssembler_f596de6f26ae:double,Jumping_double_VectorAssembler_f596de6f26ae:double,Stamina_double_VectorAssembler_f596de6f26ae:double,Strength_double_VectorAssembler_f596de6f26ae:double,LongShots_double_VectorAssembler_f596de6f26ae:double,Aggression_double_VectorAssembler_f596de6f26ae:double,Interceptions_double_VectorAssembler_f596de6f26ae:double,V...
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:619)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:220)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:298)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:882)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
    at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Encountered null while assembling a row with handleInvalid = "keep". Consider
removing nulls from dataset or using handleInvalid = "keep" or "skip".
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:287)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:255)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
    at org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:255)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$4.apply(VectorAssembler.scala:144)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$4.apply(VectorAssembler.scala:143)
    ... 29 more

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1887)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1875)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1874)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1874)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2108)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2057)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2046)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1168)
    at org.apache.spark.rdd.RDD$$anonfun$takeSample$1.apply(RDD.scala:572)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
    at org.apache.spark.rdd.RDD.takeSample(RDD.scala:561)
    at org.apache.spark.mllib.clustering.KMeans.initKMeansParallel(KMeans.scala:386)
    at org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:282)
    at org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:251)
    at org.apache.spark.ml.clustering.KMeans$$anonfun$fit$1.apply(KMeans.scala:362)
    at org.apache.spark.ml.clustering.KMeans$$anonfun$fit$1.apply(KMeans.scala:340)
    at org.apache.spark.ml.util.Instrumentation$$anonfun$11.apply(Instrumentation.scala:183)
    at scala.util.Try$.apply(Try.scala:192)
    at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:183)
    at org.apache.spark.ml.clustering.KMeans.fit(KMeans.scala:340)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Failed to execute user defined function($anonfun$4: (struct<Height(CM):double,Weight(KG):double,Crossing_double_VectorAssembler_f596de6f26ae:double,Finishing_double_VectorAssembler_f596de6f26ae:double,HeadingAccuracy_double_VectorAssembler_f596de6f26ae:double,ShortPassing_double_VectorAssembler_f596de6f26ae:double,Volleys_double_VectorAssembler_f596de6f26ae:double,Dribbling_double_VectorAssembler_f596de6f26ae:double,Curve_double_VectorAssembler_f596de6f26ae:double,FKAccuracy_double_VectorAssembler_f596de6f26ae:double,LongPassing_double_VectorAssembler_f596de6f26ae:double,BallControl_double_VectorAssembler_f596de6f26ae:double,Acceleration_double_VectorAssembler_f596de6f26ae:double,SprintSpeed_double_VectorAssembler_f596de6f26ae:double,Agility_double_VectorAssembler_f596de6f26ae:double,Reactions_double_VectorAssembler_f596de6f26ae:double,Balance_double_VectorAssembler_f596de6f26ae:double,ShotPower_double_VectorAssembler_f596de6f26ae:double,Jumping_double_VectorAssembler_f596de6f26ae:double,Stamina_double_VectorAssembler_f596de6f26ae:double,Strength_double_VectorAssembler_f596de6f26ae:double,LongShots_double_VectorAssembler_f596de6f26ae:double,Aggression_double_VectorAssembler_f596de6f26ae:double,Interceptions_double_VectorAssembler_f596de6f26ae:double,Vision_double_VectorAssembler_f596de6f26ae:double,Penalties_double_VectorAssembler_f596de6f26ae:double,Positioning_double_VectorAssembler_f596de6f26ae:double,Composure_double_VectorAs...
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:619)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:220)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:298)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:882)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
    at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more
Caused by: org.apache.spark.SparkException: Encountered null while assembling a row with handleInvalid = "keep". Consider
removing nulls from dataset or using handleInvalid = "keep" or "skip".
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:287)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:255)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
    at org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:255)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$4.apply(VectorAssembler.scala:144)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$4.apply(VectorAssembler.scala:143)
    ... 29 more
</code>```

Can someone please help me figure out the error.

答:
From your log:

```<code>Caused by: org.apache.spark.SparkException: Encountered null while assembling a row with handleInvalid = "keep". Consider
removing nulls from dataset or using handleInvalid = "keep" or "skip".
</code>```

You have null value in assembling row.
vecAssembler = VectorAssembler(inputCols=FEATURES_COL1, outputCol="features")

Please fill in null value in your data first, or use handleInvalid parameter of vector assembler, see Official Doc for detail.

Body:
While running K means clustering using pyspark, I am using the following lines of code to find the optimal K value. But there is some error constantly popping up at the model fitting line.

The preprocessing stages included removing NAs and label encoding,

from pyspark.sql.functions import when,col

#Encode column "Potential" from dataframe df
high = list(range(86,101))
middle = list(range (71, 86))
low = list(range(56, 71))
very_low = list(range(45,56))

#checking for NULL
df = df.where(col("Potential").isNotNull())

#encoding the column potential
df1 = df.withColumn("Potential_Grading", when(col("Potential").isin(high), "high").
                    when(col("Potential").isin(middle), "Middle").
                    when(col("Potential").isin(low), "Low").
                    when(col("Potential").isin(very_low),"Verylow"))
df_kmeans = df1.select("ID","Height(CM)","Weight(KG)", "Crossing", "Finishing", "HeadingAccuracy", "ShortPassing",
          "Volleys", "Dribbling", "Curve", "FKAccuracy", "LongPassing", "BallControl", "Acceleration",
          "SprintSpeed", "Agility", "Reactions", "Balance", "ShotPower", "Jumping", "Stamina", "Strength",
          "LongShots", "Aggression", "Interceptions", "Vision", "Penalties", "Composure", "Positioning",
          "Marking", "StandingTackle", "SlidingTackle","Potential_Grading")

FEATURES_COL1 = ['Height(CM)', 'Weight(KG)',
                      'Crossing', 'Finishing', 'HeadingAccuracy',
                      'ShortPassing', 'Volleys', 'Dribbling', 'Curve',
                      'FKAccuracy', 'LongPassing', 'BallControl',
                      'Acceleration', 'SprintSpeed', 'Agility',
                      'Reactions', 'Balance', 'ShotPower', 'Jumping',
                      'Stamina', 'Strength', 'LongShots', 'Aggression',
                      'Interceptions', 'Vision', 'Penalties','Positioning',
                      'Composure', 'Marking', 'StandingTackle', 'SlidingTackle']

from pyspark.ml.clustering import KMeans
from pyspark.ml.feature import VectorAssembler

#Vector assembler is a transformer which takes all of the columns specified under FEATURES_COL and combines them into a new vector column.
vecAssembler = VectorAssembler(inputCols=FEATURES_COL1, outputCol="features")

#df_kmeans_1 = vecAssembler.transform(df_kmeans).select('ID','features')
df_kmeans_1 = vecAssembler.transform(df_kmeans).select('ID','features')

The lines of code for fitting K means

cost = np.zeros(21)

    from pyspark.ml.clustering import KMeans

for k in range(2,21):

      # Trains a k-means model.
      kmeans = KMeans().setK(k).setFeaturesCol("features").setSeed(1)
      model1 = kmeans.fit(df_kmeans_1)

      #Evaluate clustering by computing Within Set Sum of Squared Errors.
      cost[k] = model1.computeCost(df_kmeans_1)

I am getting an error at the model fitting line of code:

    ---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-15-2840daf75485> in <module>()
      8   # Trains a k-means model.
      9   kmeans = KMeans().setK(k).setFeaturesCol("features").setSeed(1)
---> 10   model1 = kmeans.fit(df_kmeans_1)
     11
     12   #Evaluate clustering by computing Within Set Sum of Squared Errors.

5 frames
/content/spark-2.4.0-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    326                 raise Py4JJavaError(
    327                     "An error occurred while calling {0}{1}{2}.\n".
--> 328                     format(target_id, ".", name), value)
    329             else:
    330                 raise Py4JError(

Py4JJavaError: An error occurred while calling o346.fit.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 15.0 failed 1 times, most recent failure: Lost task 0.0 in stage 15.0 (TID 18, localhost, executor driver): org.apache.spark.SparkException: Failed to execute user defined function($anonfun$4: (struct<Height(CM):double,Weight(KG):double,Crossing_double_VectorAssembler_f596de6f26ae:double,Finishing_double_VectorAssembler_f596de6f26ae:double,HeadingAccuracy_double_VectorAssembler_f596de6f26ae:double,ShortPassing_double_VectorAssembler_f596de6f26ae:double,Volleys_double_VectorAssembler_f596de6f26ae:double,Dribbling_double_VectorAssembler_f596de6f26ae:double,Curve_double_VectorAssembler_f596de6f26ae:double,FKAccuracy_double_VectorAssembler_f596de6f26ae:double,LongPassing_double_VectorAssembler_f596de6f26ae:double,BallControl_double_VectorAssembler_f596de6f26ae:double,Acceleration_double_VectorAssembler_f596de6f26ae:double,SprintSpeed_double_VectorAssembler_f596de6f26ae:double,Agility_double_VectorAssembler_f596de6f26ae:double,Reactions_double_VectorAssembler_f596de6f26ae:double,Balance_double_VectorAssembler_f596de6f26ae:double,ShotPower_double_VectorAssembler_f596de6f26ae:double,Jumping_double_VectorAssembler_f596de6f26ae:double,Stamina_double_VectorAssembler_f596de6f26ae:double,Strength_double_VectorAssembler_f596de6f26ae:double,LongShots_double_VectorAssembler_f596de6f26ae:double,Aggression_double_VectorAssembler_f596de6f26ae:double,Interceptions_double_VectorAssembler_f596de6f26ae:double,V...
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:619)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:220)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:298)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:882)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
    at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Encountered null while assembling a row with handleInvalid = "keep". Consider
removing nulls from dataset or using handleInvalid = "keep" or "skip".
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:287)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:255)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
    at org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:255)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$4.apply(VectorAssembler.scala:144)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$4.apply(VectorAssembler.scala:143)
    ... 29 more

Driver stacktrace:
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1887)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1875)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1874)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1874)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:926)
    at scala.Option.foreach(Option.scala:257)
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:926)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2108)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2057)
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2046)
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:737)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2061)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2082)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2101)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:2126)
    at org.apache.spark.rdd.RDD.count(RDD.scala:1168)
    at org.apache.spark.rdd.RDD$$anonfun$takeSample$1.apply(RDD.scala:572)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:363)
    at org.apache.spark.rdd.RDD.takeSample(RDD.scala:561)
    at org.apache.spark.mllib.clustering.KMeans.initKMeansParallel(KMeans.scala:386)
    at org.apache.spark.mllib.clustering.KMeans.runAlgorithm(KMeans.scala:282)
    at org.apache.spark.mllib.clustering.KMeans.run(KMeans.scala:251)
    at org.apache.spark.ml.clustering.KMeans$$anonfun$fit$1.apply(KMeans.scala:362)
    at org.apache.spark.ml.clustering.KMeans$$anonfun$fit$1.apply(KMeans.scala:340)
    at org.apache.spark.ml.util.Instrumentation$$anonfun$11.apply(Instrumentation.scala:183)
    at scala.util.Try$.apply(Try.scala:192)
    at org.apache.spark.ml.util.Instrumentation$.instrumented(Instrumentation.scala:183)
    at org.apache.spark.ml.clustering.KMeans.fit(KMeans.scala:340)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:282)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:238)
    at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Failed to execute user defined function($anonfun$4: (struct<Height(CM):double,Weight(KG):double,Crossing_double_VectorAssembler_f596de6f26ae:double,Finishing_double_VectorAssembler_f596de6f26ae:double,HeadingAccuracy_double_VectorAssembler_f596de6f26ae:double,ShortPassing_double_VectorAssembler_f596de6f26ae:double,Volleys_double_VectorAssembler_f596de6f26ae:double,Dribbling_double_VectorAssembler_f596de6f26ae:double,Curve_double_VectorAssembler_f596de6f26ae:double,FKAccuracy_double_VectorAssembler_f596de6f26ae:double,LongPassing_double_VectorAssembler_f596de6f26ae:double,BallControl_double_VectorAssembler_f596de6f26ae:double,Acceleration_double_VectorAssembler_f596de6f26ae:double,SprintSpeed_double_VectorAssembler_f596de6f26ae:double,Agility_double_VectorAssembler_f596de6f26ae:double,Reactions_double_VectorAssembler_f596de6f26ae:double,Balance_double_VectorAssembler_f596de6f26ae:double,ShotPower_double_VectorAssembler_f596de6f26ae:double,Jumping_double_VectorAssembler_f596de6f26ae:double,Stamina_double_VectorAssembler_f596de6f26ae:double,Strength_double_VectorAssembler_f596de6f26ae:double,LongShots_double_VectorAssembler_f596de6f26ae:double,Aggression_double_VectorAssembler_f596de6f26ae:double,Interceptions_double_VectorAssembler_f596de6f26ae:double,Vision_double_VectorAssembler_f596de6f26ae:double,Penalties_double_VectorAssembler_f596de6f26ae:double,Positioning_double_VectorAssembler_f596de6f26ae:double,Composure_double_VectorAs...
    at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
    at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
    at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$11$$anon$1.hasNext(WholeStageCodegenExec.scala:619)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
    at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:220)
    at org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:298)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1165)
    at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:1091)
    at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1156)
    at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:882)
    at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:335)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:286)
    at org.apache.spark.rdd.ZippedPartitionsRDD2.compute(ZippedPartitionsRDD.scala:89)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:121)
    at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:402)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1360)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:408)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    ... 1 more
Caused by: org.apache.spark.SparkException: Encountered null while assembling a row with handleInvalid = "keep". Consider
removing nulls from dataset or using handleInvalid = "keep" or "skip".
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:287)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:255)
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
    at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35)
    at org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:255)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$4.apply(VectorAssembler.scala:144)
    at org.apache.spark.ml.feature.VectorAssembler$$anonfun$4.apply(VectorAssembler.scala:143)
    ... 29 more

Can someone please help me figure out the error.

Body:
From your log:

Caused by: org.apache.spark.SparkException: Encountered null while assembling a row with handleInvalid = "keep". Consider
removing nulls from dataset or using handleInvalid = "keep" or "skip".

You have null value in assembling row.
vecAssembler = VectorAssembler(inputCols=FEATURES_COL1, outputCol="features")

Please fill in null value in your data first, or use handleInvalid parameter of vector assembler, see Official Doc for detail.


问:
Convert repetitive pattern into JSON file with python
Hope you are doing fine,

I have a data file(containing 1000s of a structured pattern of data), like below

```<code>PARTNER="ABC"
  ADDRESS1="ABC Country INN"
  DEPARTMENT="ABC Department"
  CONTACT_PERSON="HR"
  TELEPHONE="+91.90.XX XX X XXX"
  FAX="+01.XX.XX XX XX XX"
  EMAIL=""

PARTNER="DEF"
  ADDRESS1="DEF Malaysia"
  DEPARTMENT=""
  CONTACT_PERSON=""
  TELEPHONE="(YYY)YYYYY"
  FAX="(001)YYYYYYYY"
  EMAIL=""

PARTNER="GEH-LOP"
  ADDRESS1="GEH LOP Street"
  DEPARTMENT="HR"
  CONTACT_PERSON="Adam"
  TELEPHONE="+91.ZZ.ZZ.ZZZZ"
  FAX="+91.ZZ.ZZ.ZZZ"
  EMAIL=""
</code>```

I tried to convert the datafile(partner.txt) to JSON with below code:

Created empty dictionaries dict1 and dict2

Reading the data file line by line

used this if not line.isspace() to make sure the linefeed is read is written in dictionary dict1

When linebreak(empty line appears) appended the content of dict1 to dict2 using dict2.update(dict1)

```<code>import json

dict1 = {}
dict2 ={}

with open("partner.txt", "r") as fh:
    out_file = open("test1.json", "w")
    for line in fh:
        if not line.isspace():
            command, description = line.strip().split("=")
            dict1[command] = description.strip('"')

        else:
            dict2.update(dict1)
            print("space found")
    json.dump(dict2,out_file,indent=1)

out_file.close()

print("json file created")

</code>```

But this code creates a json(test1.json) with only the single block of PARTNER

```<code>{
 "PARTNER": "DEF",
 "ADDRESS1": "DEF Malaysia",
 "DEPARTMENT": "",
 "CONTACT_PERSON": "",
 "TELEPHONE": "(YYY)YYYYY",
 "FAX": "(001)YYYYYYYY",
 "EMAIL": ""
}
</code>```

Expected Output
I tried looking up a lot but couldn't find a way:-

```<code>{
"data":[
{
 "PARTNER": "ABC",
 "ADDRESS1": "ABC Country INN",
 "DEPARTMENT": "ABC Department",
 "CONTACT_PERSON": "HR",
 "TELEPHONE": "+91.90.XX XX X XXX",
 "FAX": "+01.XX.XX XX XX XX",
 "EMAIL": ""
},
{
 "PARTNER": "DEF",
 "ADDRESS1": "DEF Malaysia",
 "DEPARTMENT": "",
 "CONTACT_PERSON": "",
 "TELEPHONE": "(YYY)YYYYY",
 "FAX": "(001)YYYYYYYY",
 "EMAIL": ""
},
{
 "PARTNER": "GEH-LOP",
 "ADDRESS1": "GEH LOP Street",
 "DEPARTMENT": "HR",
 "CONTACT_PERSON": "Adam",
 "TELEPHONE": "+91.ZZ.ZZ.ZZZZ",
 "FAX": "+91.ZZ.ZZ.ZZZ",
 "EMAIL": ""
}
]
}
</code>```

答:
You need to append the dict to a list of dictionaries, not use update, as it overwrites the keys that are always the same:</p>

```<code>import json
dict1 = {}
data = []

with open("partner.txt", "r") as fh:
    out_file = open("test1.json", "w")
    for line in fh:
        if not line.isspace():
            command, description = line.strip().split("=")
            dict1[command] = description.strip('"')
        else:
            data.append(dict1)
            dict1 = {}                             # set it to new dict
            print("space found")
    output = {'data': data}
    json.dump(output, out_file, indent=1)

out_file.close()

print("json file created")
</code>```

Body:
Hope you are doing fine,

I have a data file(containing 1000s of a structured pattern of data), like below

PARTNER="ABC"
  ADDRESS1="ABC Country INN"
  DEPARTMENT="ABC Department"
  CONTACT_PERSON="HR"
  TELEPHONE="+91.90.XX XX X XXX"
  FAX="+01.XX.XX XX XX XX"
  EMAIL=""

PARTNER="DEF"
  ADDRESS1="DEF Malaysia"
  DEPARTMENT=""
  CONTACT_PERSON=""
  TELEPHONE="(YYY)YYYYY"
  FAX="(001)YYYYYYYY"
  EMAIL=""

PARTNER="GEH-LOP"
  ADDRESS1="GEH LOP Street"
  DEPARTMENT="HR"
  CONTACT_PERSON="Adam"
  TELEPHONE="+91.ZZ.ZZ.ZZZZ"
  FAX="+91.ZZ.ZZ.ZZZ"
  EMAIL=""

I tried to convert the datafile(partner.txt) to JSON with below code:

Created empty dictionaries dict1 and dict2

Reading the data file line by line

used this if not line.isspace() to make sure the linefeed is read is written in dictionary dict1

When linebreak(empty line appears) appended the content of dict1 to dict2 using dict2.update(dict1)

import json

dict1 = {}
dict2 ={}

with open("partner.txt", "r") as fh:
    out_file = open("test1.json", "w")
    for line in fh:
        if not line.isspace():
            command, description = line.strip().split("=")
            dict1[command] = description.strip('"')

        else:
            dict2.update(dict1)
            print("space found")
    json.dump(dict2,out_file,indent=1)

out_file.close()

print("json file created")

But this code creates a json(test1.json) with only the single block of PARTNER

{
 "PARTNER": "DEF",
 "ADDRESS1": "DEF Malaysia",
 "DEPARTMENT": "",
 "CONTACT_PERSON": "",
 "TELEPHONE": "(YYY)YYYYY",
 "FAX": "(001)YYYYYYYY",
 "EMAIL": ""
}

Expected Output
I tried looking up a lot but couldn't find a way:-

{
"data":[
{
 "PARTNER": "ABC",
 "ADDRESS1": "ABC Country INN",
 "DEPARTMENT": "ABC Department",
 "CONTACT_PERSON": "HR",
 "TELEPHONE": "+91.90.XX XX X XXX",
 "FAX": "+01.XX.XX XX XX XX",
 "EMAIL": ""
},
{
 "PARTNER": "DEF",
 "ADDRESS1": "DEF Malaysia",
 "DEPARTMENT": "",
 "CONTACT_PERSON": "",
 "TELEPHONE": "(YYY)YYYYY",
 "FAX": "(001)YYYYYYYY",
 "EMAIL": ""
},
{
 "PARTNER": "GEH-LOP",
 "ADDRESS1": "GEH LOP Street",
 "DEPARTMENT": "HR",
 "CONTACT_PERSON": "Adam",
 "TELEPHONE": "+91.ZZ.ZZ.ZZZZ",
 "FAX": "+91.ZZ.ZZ.ZZZ",
 "EMAIL": ""
}
]
}

Body:
You need to append the dict to a list of dictionaries, not use update, as it overwrites the keys that are always the same:

import json
dict1 = {}
data = []

with open("partner.txt", "r") as fh:
    out_file = open("test1.json", "w")
    for line in fh:
        if not line.isspace():
            command, description = line.strip().split("=")
            dict1[command] = description.strip('"')
        else:
            data.append(dict1)
            dict1 = {}                             # set it to new dict
            print("space found")
    output = {'data': data}
    json.dump(output, out_file, indent=1)

out_file.close()

print("json file created")


问:
How does the spring framework map properties to a Java Object?
I am trying to do something very similar to what spring @ConfigurationProperties does to map properties to a Java POJO, but at runtime. I am hoping to be able to reuse the spring framework code that does this for spring properties.

I looked through a bunch of spring's code but I am having trouble isolating the code that does this specific functionality. I can write my own algorithm to do this, but it feels like a waste since I know this is existing functionality in the spring framework and I have spring dependencies in my project already.

答: After some more digging, I found the following class: ConfigurationPropertiesBinder, which binds properties from various PropertySources to a Bindable object. I was able to extract enough information from this bean to create my own version to do the conversion I was looking for.