问:
git +jenkins fetch issue
when i checkout code from git repository it gives error like :  </p>

```<code>Started by user anonymous
Checkout:workspace / /Users/shriram/.jenkins/jobs/github1/workspace - hudson.remoting.LocalChannel@46347456
Using strategy: Default
Last Built Revision: Revision b0a93e1344ac7ce65fbfc34fb2d3b5a8b0a72e05 (origin/master)
Checkout:workspace / /Users/shriram/.jenkins/jobs/github1/workspace - hudson.remoting.LocalChannel@46347456
Fetching changes from 1 remote Git repository
Fetching upstream changes from ssh:///shriarm@17.209.10.99 : 22/~/jengittest.git
ERROR: Problem fetching from origin / origin - could be unavailable. Continuing anyway
ERROR:  (Underlying report) : Error performing command: /opt/local/bin/git fetch -t ssh:///shriram@17.209.10.99 : 22/~/jengittest.git +refs/heads/*:refs/remotes/origin/*
Command "/opt/local/bin/git fetch -t ssh:///shriram@17.209.10.99 : 22/~/jengittest.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: Host key verification failed.
fatal: The remote end hung up unexpectedly
</code>```

答:
The error message hints at the solution: The host key for 17.209.10.99 is unknown or invalid.

Delete the line with that host from your .ssh/known_hosts file. After that, do a manual connect to that host using ssh shriarm@17.209.10.99 and accept the host key again.

Title: git +jenkins fetch issue

Body:
when i checkout code from git repository it gives error like :

Started by user anonymous
Checkout:workspace / /Users/shriram/.jenkins/jobs/github1/workspace - hudson.remoting.LocalChannel@46347456
Using strategy: Default
Last Built Revision: Revision b0a93e1344ac7ce65fbfc34fb2d3b5a8b0a72e05 (origin/master)
Checkout:workspace / /Users/shriram/.jenkins/jobs/github1/workspace - hudson.remoting.LocalChannel@46347456
Fetching changes from 1 remote Git repository
Fetching upstream changes from ssh:///shriarm@17.209.10.99 : 22/~/jengittest.git
ERROR: Problem fetching from origin / origin - could be unavailable. Continuing anyway
ERROR:  (Underlying report) : Error performing command: /opt/local/bin/git fetch -t ssh:///shriram@17.209.10.99 : 22/~/jengittest.git +refs/heads/*:refs/remotes/origin/*
Command "/opt/local/bin/git fetch -t ssh:///shriram@17.209.10.99 : 22/~/jengittest.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: Host key verification failed.
fatal: The remote end hung up unexpectedly


来源: stackexchange

问:
Django add multiple Users to lobby model
I want to add Users to my players field in the lobby but i dont know how.
Every user should be in just one lobby.

User Model

```<code>class GeoUser(AbstractUser):
        user_score = models.IntegerField(default=0)
        email = models.EmailField(unique=True)
</code>```

Lobby Model

```<code>class Lobby(models.Model):
    lobby_code = models.CharField(
        max_length=18, unique=True)

    players = # here i want to add multiple users that should be in the lobby
</code>```

Thanks in advance

答:
I believe what you're looking for is the ManyToManyField, which you would implement by:

```<code>players = models.ManyToManyField(GeoUser)
</code>```

You do have the constraint that a user should be part of a maximum of 1 lobby. I guess you could technically iterate over all of the lobbies every time and check if the user is currently in any lobby. If you plan to have a lot of lobbies, then it would probably make sense to save the user's lobby in the User model.

In my opinion, it's much better to create separate model responsible for maintaining the relation between the user and the lobby (see Django documentation for the through parameter). Now you would be able to enforce uniqueness using UniqueConstraint and also save other metadata such as the timestamp of when they joined.

ContentLicense: CC BY-SA 4.0

Body:
I want to add Users to my players field in the lobby but i dont know how.
Every user should be in just one lobby.

User Model

class GeoUser(AbstractUser):
        user_score = models.IntegerField(default=0)
        email = models.EmailField(unique=True)

Lobby Model

class Lobby(models.Model):
    lobby_code = models.CharField(
        max_length=18, unique=True)

    players = # here i want to add multiple users that should be in the lobby

Thanks in advance

Body:
I believe what you're looking for is the ManyToManyField, which you would implement by:

players = models.ManyToManyField(GeoUser)

You do have the constraint that a user should be part of a maximum of 1 lobby. I guess you could technically iterate over all of the lobbies every time and check if the user is currently in any lobby. If you plan to have a lot of lobbies, then it would probably make sense to save the user's lobby in the User model.

In my opinion, it's much better to create separate model responsible for maintaining the relation between the user and the lobby (see Django documentation for the through parameter). Now you would be able to enforce uniqueness using UniqueConstraint and also save other metadata such as the timestamp of when they joined.


问:
COUNTIFS w/ Multiple Criteria in Same Column
I'm trying to count instances of letters (like letters C through Z, excluding RR) within the same column if they are listed next to a name. Here's my sample Google Sheet that you can edit.

I'm trying to insert the formula in cell F3 that is highlighted yellow. So far I have...

```<code>=arrayformula(SUM(COUNTIFS(A2:A,{"C","D","E","F","G"},C2:C,E3:E)))
</code>```

It seems like what I have should work, but it's giving me a #VALUE error, saying, "ARRAY arguments to COUNTIFS are of different size".

It seems like REGEXMATCH could be used inside the COUNTIF formula to make it easier to restrict the search to the range of letters I need, but not sure how to construct the formula.

Thanks for your help!

UPDATE
This formula below works but only for column A. I actually need to specify a different range of letters to be counted in column B and totaled in column K.

```<code>=QUERY(A2:C,"select C,count(A) where A matches 'C|D|E|F|G|H|I|J|K' group by C label count(A)''", 0)
</code>```

Seems like this post almost answers it.

答:
This formula should work for you if you paste it in cells F3 to F7.
I currently have cells P1 to P26 (you'd probably want to hide this column) occupied with the letters of the alphabet in order, making it so that you can select P3:P26 (C-Z) to put into your first condition for your COUNTIFS.

```<code>=arrayformula(SUM(COUNTIFS(A2:A, $P$3:$P$26,C2:C,E3)))
</code>```

You'll have to implement these formulas in the other places that you want them too, but it shouldn't be hard to change this formula to work in the other places as well.

Body:
I'm trying to count instances of letters (like letters C through Z, excluding RR) within the same column if they are listed next to a name. Here's my sample Google Sheet that you can edit.

I'm trying to insert the formula in cell F3 that is highlighted yellow. So far I have...

=arrayformula(SUM(COUNTIFS(A2:A,{"C","D","E","F","G"},C2:C,E3:E)))

It seems like what I have should work, but it's giving me a #VALUE error, saying, "ARRAY arguments to COUNTIFS are of different size".

It seems like REGEXMATCH could be used inside the COUNTIF formula to make it easier to restrict the search to the range of letters I need, but not sure how to construct the formula.

Thanks for your help!

UPDATE
This formula below works but only for column A. I actually need to specify a different range of letters to be counted in column B and totaled in column K.

=QUERY(A2:C,"select C,count(A) where A matches 'C|D|E|F|G|H|I|J|K' group by C label count(A)''", 0)

Seems like this post almost answers it.

Body:
This formula should work for you if you paste it in cells F3 to F7.
I currently have cells P1 to P26 (you'd probably want to hide this column) occupied with the letters of the alphabet in order, making it so that you can select P3:P26 (C-Z) to put into your first condition for your COUNTIFS.

=arrayformula(SUM(COUNTIFS(A2:A, $P$3:$P$26,C2:C,E3)))

You'll have to implement these formulas in the other places that you want them too, but it shouldn't be hard to change this formula to work in the other places as well.


问:
How to create a consistent plotly color map with dynamic input?
I have some dataframes I want to plot with a variable number of Fruits. When plotting, each dataframe will always have the same Fruit types, even though some may have a value of 0 in their respective row.

df1
Day  Value Fruit
1    5     Banana
1    3     Pear
2    4     Banana
2    2     Pear
3    3     Banana
3    3     Pear

df2  # note all pears 0'd
Day  Value Fruit
1    5     Banana
1    0     Pear
2    4     Banana
2    0     Pear
3    3     Banana
3    0     Pear

# AKA
df1.Fruit.unique() == df2.Fruit.unique()
```
<p>Plotting code:</p>
<pre class="lang-py prettyprint-override"><code>import dash_bootstrap_components as dbc
from plotly import express as px

bar_charts = []
for df in dfs:
  # to get a stacked bar chart
  fruit_data = df.groupby(["Day", "Fruit"]).Value.sum().to_frame().reset_index()
  bar_charts.append(
    dcc.Graph(
      figure=px.bar(
        fruit_data,
        x="Day",
        y="Value",
        color="Fruit",
      ),
    )
  )
return dbc.Col(bar_charts)
</code>```

Plotly seems to be assigning differnt colors to the Fruit between dataframes based on the value of the Fruit. I want the Fruits to always be the same color across graphs regardless of value (e.g. Banana always yellow, Pear always green). HOWEVER, I don't know what Fruits are available until runtime, so I can't just hard code a color map ahead of time.

How do I tell plotly to always color the Fruit the same?

Im sure this is some silly little option im missing.

答:
To make sure that the same colors are assigned to the same Fruit, you can make a color map using fruit_data['Fruit'].unique() and any list of colors like ['yellow', 'green'] or a longer list px.colors.qualitative.Alphabet for color_discrete_map in px.bar like this:

```<code>colordict = {f:px.colors.qualitative.Alphabet[i] for i, f in enumerate(fruit_data.Fruit.unique())}
colordict = {f:['yellow', 'green'][i] for i, f in enumerate(fruit_data.Fruit.unique())}
</code>```

Plot:

And if you're working with a large number of variables and worry about running out of colors, just incorporate the approach described in Plotly: How to increase the number of colors to assure unique colors for all lines like this:

```<code>n_colors = len(fruit_data.Fruit.unique())
colorscale = colors = px.colors.sample_colorscale("viridis", [n/(n_colors -1) for n in range(n_colors)])
colordict = {f:colorscale[i] for i, f in enumerate(fruit_data.Fruit.unique())}
</code>```

Complete code:

```<code>import io
import plotly.express as px
import pandas as pd

# data
fruit_data = pd.read_csv(io.StringIO("""Day  Value Fruit
                                1    5     Banana
                                1    3     Pear
                                2    4     Banana
                                2    2     Pear
                                3    3     Banana
                                3    3     Pear"""),sep= '\\s+')

# color assignment:
# colordict = {f:px.colors.qualitative.Alphabet[i] for i, f in enumerate(fruit_data.Fruit.unique())}
colordict = {f:['yellow', 'green'][i] for i, f in enumerate(fruit_data.Fruit.unique())}

# color assignment for a large number of variables:
n_colors = len(fruit_data.Fruit.unique())
colorscale = colors = px.colors.sample_colorscale("viridis", [n/(n_colors -1) for n in range(n_colors)])
colordict = {f:colorscale[i] for i, f in enumerate(fruit_data.Fruit.unique())}

# plotly figure
fig = px.bar(
        fruit_data,
        x="Day",
        y="Value",
        color = 'Fruit',
        color_discrete_map = colordict
      )
fig.show()
</code>```

Body:
I have some dataframes I want to plot with a variable number of Fruits. When plotting, each dataframe will always have the same Fruit types, even though some may have a value of 0 in their respective row.

df1
Day  Value Fruit
1    5     Banana
1    3     Pear
2    4     Banana
2    2     Pear
3    3     Banana
3    3     Pear

df2  # note all pears 0'd
Day  Value Fruit
1    5     Banana
1    0     Pear
2    4     Banana
2    0     Pear
3    3     Banana
3    0     Pear

# AKA
df1.Fruit.unique() == df2.Fruit.unique()

Plotting code:

import dash_bootstrap_components as dbc
from plotly import express as px

bar_charts = []
for df in dfs:
  # to get a stacked bar chart
  fruit_data = df.groupby(["Day", "Fruit"]).Value.sum().to_frame().reset_index()
  bar_charts.append(
    dcc.Graph(
      figure=px.bar(
        fruit_data,
        x="Day",
        y="Value",
        color="Fruit",
      ),
    )
  )
return dbc.Col(bar_charts)

Plotly seems to be assigning differnt colors to the Fruit between dataframes based on the value of the Fruit. I want the Fruits to always be the same color across graphs regardless of value (e.g. Banana always yellow, Pear always green). HOWEVER, I don't know what Fruits are available until runtime, so I can't just hard code a color map ahead of time.

How do I tell plotly to always color the Fruit the same?

Im sure this is some silly little option im missing.

Body:
To make sure that the same colors are assigned to the same Fruit, you can make a color map using fruit_data['Fruit'].unique() and any list of colors like ['yellow', 'green'] or a longer list px.colors.qualitative.Alphabet for color_discrete_map in px.bar like this:

colordict = {f:px.colors.qualitative.Alphabet[i] for i, f in enumerate(fruit_data.Fruit.unique())}
colordict = {f:['yellow', 'green'][i] for i, f in enumerate(fruit_data.Fruit.unique())}

Plot:

And if you're working with a large number of variables and worry about running out of colors, just incorporate the approach described in Plotly: How to increase the number of colors to assure unique colors for all lines like this:

n_colors = len(fruit_data.Fruit.unique())
colorscale = colors = px.colors.sample_colorscale("viridis", [n/(n_colors -1) for n in range(n_colors)])
colordict = {f:colorscale[i] for i, f in enumerate(fruit_data.Fruit.unique())}

Complete code:

import io
import plotly.express as px
import pandas as pd

# data
fruit_data = pd.read_csv(io.StringIO("""Day  Value Fruit
                                1    5     Banana
                                1    3     Pear
                                2    4     Banana
                                2    2     Pear
                                3    3     Banana
                                3    3     Pear"""),sep= '\\s+')

# color assignment:
# colordict = {f:px.colors.qualitative.Alphabet[i] for i, f in enumerate(fruit_data.Fruit.unique())}
colordict = {f:['yellow', 'green'][i] for i, f in enumerate(fruit_data.Fruit.unique())}

# color assignment for a large number of variables:
n_colors = len(fruit_data.Fruit.unique())
colorscale = colors = px.colors.sample_colorscale("viridis", [n/(n_colors -1) for n in range(n_colors)])
colordict = {f:colorscale[i] for i, f in enumerate(fruit_data.Fruit.unique())}

# plotly figure
fig = px.bar(
        fruit_data,
        x="Day",
        y="Value",
        color = 'Fruit',
        color_discrete_map = colordict
      )
fig.show()


问:
Use a custom deserializer only on certain fields?
With gson, is it possible to use a custom deserializer / serializer only on certain fields? The user guide shows how to register an adapter for an entire type, not for specific fields. The reason why I want this is because I parse a custom date format and store it in a long member field (as a Unix timestamp), so I don't want to register a type adapter for all Long fields.

Is there a way to do this?

答:
I also store Date values as long in my objects for easy defensive copies. I also desired a way to override only the date fields when serializing my object and not having to write out all the fields in the process. This is the solution I came up with. Not sure it is the optimal way to handle this, but it seems to perform just fine.

The DateUtil class is a custom class used here to get a Date parsed as a String.

```<code>public final class Person {
  private final String firstName;
  private final String lastName;
  private final long birthDate;

  private Person(String firstName, String lastName, Date birthDate) {
    this.firstName = firstName;
    this.lastName = lastName;
    this.birthDate = birthDate.getTime();
  }

  public static Person getInstance(String firstName, String lastName, Date birthDate) {
    return new Person(firstName, lastName, birthDate);
  }

  public String toJson() {
    return new GsonBuilder().registerTypeAdapter(Person.class, new PersonSerializer()).create().toJson(this);
  }

  public static class PersonSerializer implements JsonSerializer<Person> {
    @Override
    public JsonElement serialize(Person person, Type type, JsonSerializationContext context) {
      JsonElement personJson = new Gson().toJsonTree(person);
      personJson.getAsJsonObject().add("birthDate", new JsonPrimitive(DateUtil.getFormattedDate(new Date(policy.birthDate), DateFormat.USA_DATE)));
      return personJson;
    }
  }
}
</code>```

When the class is serialized, the birthDate field is returned as a formatted String instead of the long value.

ContentLicense: CC BY-SA 3.0

Body:
I also store Date values as long in my objects for easy defensive copies. I also desired a way to override only the date fields when serializing my object and not having to write out all the fields in the process. This is the solution I came up with. Not sure it is the optimal way to handle this, but it seems to perform just fine.

The DateUtil class is a custom class used here to get a Date parsed as a String.

public final class Person {
  private final String firstName;
  private final String lastName;
  private final long birthDate;

  private Person(String firstName, String lastName, Date birthDate) {
    this.firstName = firstName;
    this.lastName = lastName;
    this.birthDate = birthDate.getTime();
  }

  public static Person getInstance(String firstName, String lastName, Date birthDate) {
    return new Person(firstName, lastName, birthDate);
  }

  public String toJson() {
    return new GsonBuilder().registerTypeAdapter(Person.class, new PersonSerializer()).create().toJson(this);
  }

  public static class PersonSerializer implements JsonSerializer<Person> {
    @Override
    public JsonElement serialize(Person person, Type type, JsonSerializationContext context) {
      JsonElement personJson = new Gson().toJsonTree(person);
      personJson.getAsJsonObject().add("birthDate", new JsonPrimitive(DateUtil.getFormattedDate(new Date(policy.birthDate), DateFormat.USA_DATE)));
      return personJson;
    }
  }
}

When the class is serialized, the birthDate field is returned as a formatted String instead of the long value.


问:
Loading Robot Resource files from Python package
I have a number of teams working in Robot in separate repositories, and want to make available a set of common Python and Robot code to avoid duplication. I've pulled out the .py and .robot files that I want to package, have built the package (making sure that the .robot files are present in the .tar.gz), and have published it to our local artifactory. When I go to use these files in Robot tests, however, I can successfully use the .py files with the

Library  package.module

syntax.

However using Robot resources with the

Resource  package.resource.robot

syntax fails. Is there some way to include and utilize .robot resources in a python package?

答:
I got it to work as such:</p>
```<code>Library         package.pythonmodule
Resource        package/robotfile.robot
</code>```

Body:
I got it to work as such:

Library         package.pythonmodule
Resource        package/robotfile.robot


问:
How can I update/save a Property Change for Child Item from a Hierarchical List View of Items
See the following app screens:

Content View Screen:

Content View with hierarchical list children rows disclosed:

Parent Row Detail View:

Child Row Detail View:

Referencing the above views, here are the steps I do and the resulting problem I’m trying to solve:

Launch the app.

From the Functions (Content View) presented at launch, see that there is one item listed in a list view (1.0 Move Vessel)

Click the yellow (my app accent color) disclosure arrow at the right of the list item.

Two subordinate child list rows appear under the parent list item, 1.1 Move Position and 1.2 Hold Position.

When I tap the parent item (1.0 Move Vessel) in the hierarchy list, I'm successfully able to navigate to a detail view for that tapped item.

Edit the description of the 1.0 Move Vessel item (defaults to test) of the tapped item properties in the detail view using a TextEditor view.

Click yellow Save button at top left of detail view. The app navigates back to the parent Functions (Content View).

Click on the parent 1.0 Move Vessel row again.

See that description was successfully saved and now displayed from the change made in Step 5 and 6.

Repeat steps 5 through 8 again for 1.1 Move Position list row.

See that the edit/change made to the description was not saved and the default test1 description is displayed instead (not what is wanted).

Repeat steps 5 through 8 again for 1.2 Hold Position list row.

See that the edit/change made to the description was not saved and the default test2 description is displayed instead (not what is wanted).

I think I may have a problem in my save code logic and I'm trying to investigate.

Here are the swift files for the Detail View, the View Model, and the Model (I’ve not included the content view code because that code is working ok with the detail view. Again, I think the problem is in my save button and function call code for updating the view model.
NOTE: sorry that I can’t seem to figure out how to get all the code for a file contiguous in the code view. I seem to have some closing braces that don’t appear in the code view. I think you can still follow the code.

```<code>struct FunctionDetailView: View {
    @State var vesselFunction: VesselFunction
    @State var vesselFunctionDescription: String
    @Environment(\.presentationMode) var presentationMode
    @EnvironmentObject var functionViewModel : FunctionViewModel

    var body: some View {

        NavigationView {

            Form {
                Text("Enter description below")
                TextEditor(text: $vesselFunctionDescription)
                    .frame(height: 200)
                    .toolbar {
                        Button {
                            //print(vesselFunction)

                            vesselFunction.funcDescription = vesselFunctionDescription

                            //print(vesselFunction)

                            functionViewModel.updateVesselFunction(vesselFunction: vesselFunction)

                            //print(vesselFunction)

                            presentationMode.wrappedValue.dismiss()
                        } label: {
                            Text("Save")
                        }

                    }
            }
            .padding()
            .navigationTitle(vesselFunction.name)
            .navigationBarTitleDisplayMode(.inline)
        }
    }
}

struct FunctionDetailView_Previews: PreviewProvider {
    static var previews: some View {
        FunctionDetailView(vesselFunction: VesselFunction(id: UUID(), name: "x.x Verb Noun", funcDescription: "Description", children: nil), vesselFunctionDescription: "placeholder")
            .environmentObject(FunctionViewModel())
            .preferredColorScheme(.dark)
    }
}

</code>```

FunctionViewModel.swift

```<code>@MainActor class FunctionViewModel: ObservableObject {

    @Published private(set) var decomp : [VesselFunction] = [
        VesselFunction(id: UUID(), name: "1.0 Move Vessel", funcDescription: "test", children: [
            VesselFunction(id: UUID(), name: "1.1 Move Position", funcDescription: "test1", children: nil),
            VesselFunction(id: UUID(), name: "1.2 Hold Position", funcDescription: "test2", children: nil)
        ])
    ]

    func updateVesselFunction(vesselFunction: VesselFunction) {
        /*
         if let index = decomp.firstIndex(where: { (existingVesselFunction) -> Bool in
         return existingVesselFunction.id == vesselFunction.id
         }) {
         //run this code
         }
         */
        // cleaner version of above
        if let index = decomp.firstIndex(where: { $0.id == vesselFunction.id }) {
            decomp[index] = vesselFunction.updateCompletion()
        }
        /*
         else {
         for item in decomp {
         if item.children != nil {
         if let index = item.children?.firstIndex(where: { $0.id == vesselFunction.id }) {
         item.children![index] = vesselFunction.updateCompletion()
         }
         }
         }
         } */
    }
}
</code>```

FunctionModel.swift

```<code>struct VesselFunction: Identifiable {
    let id : UUID
    let name : String
    var funcDescription : String
    var children : [VesselFunction]?

    init(id: UUID, name: String, funcDescription: String, children: [VesselFunction]?) {
        self.id = id
        self.name = name
        self.funcDescription = funcDescription
        self.children = children
    }

    func updateCompletion() -> VesselFunction {
        return VesselFunction(id: id, name: name, funcDescription: funcDescription, children: children)
    }
}
</code>```

As you can see from the else and for-in loop code commented out at the bottom of the FunctionViewModel code, I was trying to see if I needed to do something like this code to access the children VesselFunction array entries of the decomp published property. With the if let index code that is not commented out, the save function works but only for the top-level decomp array VesselFunction elements, not the nested children arrays elements.

Any help would be appreciated so all decomp array elements, both parent and nested children, can be updated when the TextEditor field is changed and the Save button is pressed in the FunctionDetailView.
NOTE: I am only showing a 1 level deep nested array of children for the decomp property. I actually want to have multiple (at least 3) level of children arrays, so if you have any ideas how to make an updateVesselFunction function work for multiple children array elements, I would appreciate it.

答: In the main View use ForEach($model.items) { $item in so you get a write access to the model item. In the detail View change the @State to @Binding.

Body:
See the following app screens:

Content View Screen:

Content View with hierarchical list children rows disclosed:

Parent Row Detail View:

Child Row Detail View:

Referencing the above views, here are the steps I do and the resulting problem I’m trying to solve:

Launch the app.

From the Functions (Content View) presented at launch, see that there is one item listed in a list view (1.0 Move Vessel)

Click the yellow (my app accent color) disclosure arrow at the right of the list item.

Two subordinate child list rows appear under the parent list item, 1.1 Move Position and 1.2 Hold Position.

When I tap the parent item (1.0 Move Vessel) in the hierarchy list, I'm successfully able to navigate to a detail view for that tapped item.

Edit the description of the 1.0 Move Vessel item (defaults to test) of the tapped item properties in the detail view using a TextEditor view.

Click yellow Save button at top left of detail view. The app navigates back to the parent Functions (Content View).

Click on the parent 1.0 Move Vessel row again.

See that description was successfully saved and now displayed from the change made in Step 5 and 6.

Repeat steps 5 through 8 again for 1.1 Move Position list row.

See that the edit/change made to the description was not saved and the default test1 description is displayed instead (not what is wanted).

Repeat steps 5 through 8 again for 1.2 Hold Position list row.

See that the edit/change made to the description was not saved and the default test2 description is displayed instead (not what is wanted).

I think I may have a problem in my save code logic and I'm trying to investigate.

Here are the swift files for the Detail View, the View Model, and the Model (I’ve not included the content view code because that code is working ok with the detail view. Again, I think the problem is in my save button and function call code for updating the view model.
NOTE: sorry that I can’t seem to figure out how to get all the code for a file contiguous in the code view. I seem to have some closing braces that don’t appear in the code view. I think you can still follow the code.

struct FunctionDetailView: View {
    @State var vesselFunction: VesselFunction
    @State var vesselFunctionDescription: String
    @Environment(\.presentationMode) var presentationMode
    @EnvironmentObject var functionViewModel : FunctionViewModel

    var body: some View {

        NavigationView {

            Form {
                Text("Enter description below")
                TextEditor(text: $vesselFunctionDescription)
                    .frame(height: 200)
                    .toolbar {
                        Button {
                            //print(vesselFunction)

                            vesselFunction.funcDescription = vesselFunctionDescription

                            //print(vesselFunction)

                            functionViewModel.updateVesselFunction(vesselFunction: vesselFunction)

                            //print(vesselFunction)

                            presentationMode.wrappedValue.dismiss()
                        } label: {
                            Text("Save")
                        }

                    }
            }
            .padding()
            .navigationTitle(vesselFunction.name)
            .navigationBarTitleDisplayMode(.inline)
        }
    }
}

struct FunctionDetailView_Previews: PreviewProvider {
    static var previews: some View {
        FunctionDetailView(vesselFunction: VesselFunction(id: UUID(), name: "x.x Verb Noun", funcDescription: "Description", children: nil), vesselFunctionDescription: "placeholder")
            .environmentObject(FunctionViewModel())
            .preferredColorScheme(.dark)
    }
}

FunctionViewModel.swift

@MainActor class FunctionViewModel: ObservableObject {

    @Published private(set) var decomp : [VesselFunction] = [
        VesselFunction(id: UUID(), name: "1.0 Move Vessel", funcDescription: "test", children: [
            VesselFunction(id: UUID(), name: "1.1 Move Position", funcDescription: "test1", children: nil),
            VesselFunction(id: UUID(), name: "1.2 Hold Position", funcDescription: "test2", children: nil)
        ])
    ]

    func updateVesselFunction(vesselFunction: VesselFunction) {
        /*
         if let index = decomp.firstIndex(where: { (existingVesselFunction) -> Bool in
         return existingVesselFunction.id == vesselFunction.id
         }) {
         //run this code
         }
         */
        // cleaner version of above
        if let index = decomp.firstIndex(where: { $0.id == vesselFunction.id }) {
            decomp[index] = vesselFunction.updateCompletion()
        }
        /*
         else {
         for item in decomp {
         if item.children != nil {
         if let index = item.children?.firstIndex(where: { $0.id == vesselFunction.id }) {
         item.children![index] = vesselFunction.updateCompletion()
         }
         }
         }
         } */
    }
}

FunctionModel.swift

struct VesselFunction: Identifiable {
    let id : UUID
    let name : String
    var funcDescription : String
    var children : [VesselFunction]?

    init(id: UUID, name: String, funcDescription: String, children: [VesselFunction]?) {
        self.id = id
        self.name = name
        self.funcDescription = funcDescription
        self.children = children
    }

    func updateCompletion() -> VesselFunction {
        return VesselFunction(id: id, name: name, funcDescription: funcDescription, children: children)
    }
}

As you can see from the else and for-in loop code commented out at the bottom of the FunctionViewModel code, I was trying to see if I needed to do something like this code to access the children VesselFunction array entries of the decomp published property. With the if let index code that is not commented out, the save function works but only for the top-level decomp array VesselFunction elements, not the nested children arrays elements.

Any help would be appreciated so all decomp array elements, both parent and nested children, can be updated when the TextEditor field is changed and the Save button is pressed in the FunctionDetailView.
NOTE: I am only showing a 1 level deep nested array of children for the decomp property. I actually want to have multiple (at least 3) level of children arrays, so if you have any ideas how to make an updateVesselFunction function work for multiple children array elements, I would appreciate it.


问:
Retrieving custom claims Firebase/React immediately after authentication
I want to request custom claims as soon as my React app loads, and the user has been authorized.

Here is my code:

```<code>  const [user, setUser] = useState({})
  const [claims, setClaims] = useState({})
  const [isCustomClaimRequested, setIsCustomClaimRequested] = useState(false)

  onAuthStateChanged(auth, (currentUser) => {
    setUser(currentUser)
    if(currentUser) {
      localStorage.setItem("signedIn", "true")
      if ( !isCustomClaimRequested ) {
        setIsCustomClaimRequested(true)
        auth.currentUser.getIdTokenResult().then((idTokenResult) => {
          setClaims(idTokenResult.claims)
          console.log("userCustomClaims", claims)
        })
      }
    } else {
      localStorage.removeItem("signedIn")
    }
  })
</code>```

Currently, when I run this code, the console logs:

```<code>userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
</code>```

Which means that it is calling getIDTokenResult() 8 times. But it does work to retrieve the custom claims.

How can I make the request for the custom claims only once, after the user is authenticated?

答:
I just moved it to useEffect and that fixed it:

```<code>  useEffect(() => {
    if (isCustomClaimRequested) {
      auth.currentUser.getIdTokenResult()
        .then((tokenResult) => {
          if (tokenResult.claims) {
            setClaims(tokenResult.claims)
          }
        })

    }
  }, [isCustomClaimRequested])
</code>```

Body:
I want to request custom claims as soon as my React app loads, and the user has been authorized.

Here is my code:

  const [user, setUser] = useState({})
  const [claims, setClaims] = useState({})
  const [isCustomClaimRequested, setIsCustomClaimRequested] = useState(false)

  onAuthStateChanged(auth, (currentUser) => {
    setUser(currentUser)
    if(currentUser) {
      localStorage.setItem("signedIn", "true")
      if ( !isCustomClaimRequested ) {
        setIsCustomClaimRequested(true)
        auth.currentUser.getIdTokenResult().then((idTokenResult) => {
          setClaims(idTokenResult.claims)
          console.log("userCustomClaims", claims)
        })
      }
    } else {
      localStorage.removeItem("signedIn")
    }
  })

Currently, when I run this code, the console logs:

userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}
userCustomClaims  {}

Which means that it is calling getIDTokenResult() 8 times. But it does work to retrieve the custom claims.

How can I make the request for the custom claims only once, after the user is authenticated?

Body:
I just moved it to useEffect and that fixed it:

  useEffect(() => {
    if (isCustomClaimRequested) {
      auth.currentUser.getIdTokenResult()
        .then((tokenResult) => {
          if (tokenResult.claims) {
            setClaims(tokenResult.claims)
          }
        })

    }
  }, [isCustomClaimRequested])


问:
hdfs distcp failing to copy from hdfs to s3
We have a snowball configured in our inhouse staging node with end-point http://10.91.16.213:8080. It all works properly, I can even list files in this snowball via s3 cli command

```<code>aws s3 ls my-bucket/data/ --endpoint-url=http://10.91.16.213:8080`
</code>```

Now I am trying to use the hadoop distcp command to copy data from hdfs to s3 snowball. first I tested hadoop distcp command to copy some files to actual s3 test-bucket in my aws account,

```<code>hadoop distcp \
-Dfs.s3a.fast.upload=true \
-Dfs.s3a.access.key=AKIAUPWDYDZTSGWUWJWN \
-Dfs.s3a.secret.key=<my-secret> \
hdfs://path/to/data/ \
s3a://test-bucket-anum/
</code>```

The above command executes fine and starts the copying job in hadoop cluster. Now in order to copy to my in-house snowballs, all I have to do is change the endpoint. This is what I try;

```<code>hadoop distcp \
-Dfs.s3a.endpoint=http://10.91.16.213:8080 \
-Dfs.s3a.fast.upload=true \
-Dfs.s3a.access.key=AKIACEMGMYDQNJXGQ2DEOBXG42SQCFR2ZJFTDED3HX3KLVTLOIN6AH3FSDHUF \
-Dfs.s3a.secret.key=<snowball-secret>  \
hdfs://path/to/data/ \
s3a://my-bucket/
</code>```

The above command fails with following error;

```<code>20/09/02 19:20:22 INFO s3a.S3AFileSystem: Caught an AmazonClientException, which means the client encountered a serious internal problem while trying to communicate with S3, such as not being able to access the network.
20/09/02 19:20:22 INFO s3a.S3AFileSystem: Error Message: {}com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200, Response Text: OK
com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200, Response Text: OK
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:738)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:399)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:604)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:962)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:217)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:116)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:430)
Caused by: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:150)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:279)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:75)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:72)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:712)
    ... 12 more
Caused by: java.lang.RuntimeException: Invalid value for IsTruncated field:
true
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler.endElement(XmlResponsesSaxParser.java:647)
    at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
    at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:141)
    ... 18 more
20/09/02 19:20:22 ERROR tools.DistCp: Invalid arguments:
com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200, Response Text: OK
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:738)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:399)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:604)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:962)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:217)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:116)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:430)
Caused by: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:150)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:279)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:75)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:72)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:712)
    ... 12 more
Caused by: java.lang.RuntimeException: Invalid value for IsTruncated field:
true
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler.endElement(XmlResponsesSaxParser.java:647)
    at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
    at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:141)
    ... 18 more
Invalid arguments: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200, Response Text: OK
usage: distcp OPTIONS [source_path...] <target_path>
              OPTIONS
 -append                Reuse existing data in target files and append new
                        data to them if possible
 -async                 Should distcp execution be blocking
 -atomic                Commit all changes or none
 -bandwidth <arg>       Specify bandwidth per map in MB
 -delete                Delete from target, files missing in source
 -diff <arg>            Use snapshot diff report to identify the
                        difference between source and target
 -f <arg>               List of files that need to be copied
 -filelimit <arg>       (Deprecated!) Limit number of files copied to <= n
 -i                     Ignore failures during copy
 -log <arg>             Folder on DFS where distcp execution logs are
                        saved
 -m <arg>               Max number of concurrent maps to use for copy
 -mapredSslConf <arg>   Configuration for ssl config file, to use with
                        hftps://
 -overwrite             Choose to overwrite target files unconditionally,
                        even if they exist.
 -p <arg>               preserve status (rbugpcaxt)(replication,
                        block-size, user, group, permission,
                        checksum-type, ACL, XATTR, timestamps). If -p is
                        specified with no <arg>, then preserves
                        replication, block size, user, group, permission,
                        checksum type and timestamps. raw.* xattrs are
                        preserved when both the source and destination
                        paths are in the /.reserved/raw hierarchy (HDFS
                        only). raw.* xattrpreservation is independent of
                        the -p flag. Refer to the DistCp documentation for
                        more details.
 -sizelimit <arg>       (Deprecated!) Limit number of files copied to <= n
                        bytes
 -skipcrccheck          Whether to skip CRC checks between source and
                        target paths.
 -strategy <arg>        Copy strategy to use. Default is dividing work
                        based on file sizes
 -tmp <arg>             Intermediate work path to be used for atomic
                        commit
 -update                Update target, copying only missingfiles or
                        directories
</code>```

Here are some other hadoop configurations I tried aswell, with no luck.

-Dfs.s3a.connection.ssl.enabled=false : since my endpoint is http.

-Dfs.s3a.region=eu-west-1

Am I missing something ?

UPDATE:

As the error message also includes Invalid arguments:  I suppose I might be providing some invalid chars into the args, so I tried writing these options in /etc/hadoop/conf/core-site.xml as follows;

```<code><property>
 <name>fs.s3a.endpoint</name>
 <value>http://10.91.16.213:8080</value>
</property>

<property>
 <name>fs.s3a.fast.upload</name>
 <value>true</value>
</property>

<property>
 <name>fs.s3a.access.key</name>
 <value>AKIACEMGMYDQNJXGQ2DEOBXG42SQCFR2ZJFTDED3HX3KLVTLOIN6AH3FSDHUF</value>
</property>

<property>
 <name>fs.s3a.secret.key</name>
 <value><snowball-secret></value>
</property>
</code>```

but got same error message :(

UPDATE 2:

After reading this, it looks like s3 xml parsing problem while doing ListObjects. AWS client has this option .withEncodingType("url");, but couldn't find anything similar for hadoop distcp.

答:
the s3a connector does not support working with an S3 snowball device -and will not do so unless and until someone sits down to implement all of HADOOP-14710.

nobody is doing that as of february 2022. if you or anyone else reading this page want the feature

check the JIRA to see if it is done, if it is, use a version of hadoop with the feature.

if it's not fixed at all -this is your chance to contribute to the community

Body:
We have a snowball configured in our inhouse staging node with end-point http://10.91.16.213:8080. It all works properly, I can even list files in this snowball via s3 cli command

aws s3 ls my-bucket/data/ --endpoint-url=http://10.91.16.213:8080`

Now I am trying to use the hadoop distcp command to copy data from hdfs to s3 snowball. first I tested hadoop distcp command to copy some files to actual s3 test-bucket in my aws account,

hadoop distcp \
-Dfs.s3a.fast.upload=true \
-Dfs.s3a.access.key=AKIAUPWDYDZTSGWUWJWN \
-Dfs.s3a.secret.key=<my-secret> \
hdfs://path/to/data/ \
s3a://test-bucket-anum/

The above command executes fine and starts the copying job in hadoop cluster. Now in order to copy to my in-house snowballs, all I have to do is change the endpoint. This is what I try;

hadoop distcp \
-Dfs.s3a.endpoint=http://10.91.16.213:8080 \
-Dfs.s3a.fast.upload=true \
-Dfs.s3a.access.key=AKIACEMGMYDQNJXGQ2DEOBXG42SQCFR2ZJFTDED3HX3KLVTLOIN6AH3FSDHUF \
-Dfs.s3a.secret.key=<snowball-secret>  \
hdfs://path/to/data/ \
s3a://my-bucket/

The above command fails with following error;

20/09/02 19:20:22 INFO s3a.S3AFileSystem: Caught an AmazonClientException, which means the client encountered a serious internal problem while trying to communicate with S3, such as not being able to access the network.
20/09/02 19:20:22 INFO s3a.S3AFileSystem: Error Message: {}com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200, Response Text: OK
com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200, Response Text: OK
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:738)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:399)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:604)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:962)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:217)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:116)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:430)
Caused by: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:150)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:279)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:75)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:72)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:712)
    ... 12 more
Caused by: java.lang.RuntimeException: Invalid value for IsTruncated field:
true
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler.endElement(XmlResponsesSaxParser.java:647)
    at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
    at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:141)
    ... 18 more
20/09/02 19:20:22 ERROR tools.DistCp: Invalid arguments:
com.amazonaws.AmazonClientException: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200, Response Text: OK
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:738)
    at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:399)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3480)
    at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:604)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:962)
    at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:77)
    at org.apache.hadoop.fs.FileSystem.exists(FileSystem.java:1424)
    at org.apache.hadoop.tools.DistCp.setTargetPathExists(DistCp.java:217)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:116)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:430)
Caused by: com.amazonaws.AmazonClientException: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:150)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:279)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:75)
    at com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:72)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
    at com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
    at com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:712)
    ... 12 more
Caused by: java.lang.RuntimeException: Invalid value for IsTruncated field:
true
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler.endElement(XmlResponsesSaxParser.java:647)
    at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
    at org.apache.xerces.impl.XMLNSDocumentScannerImpl.scanEndElement(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
    at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
    at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
    at com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseXmlInputStream(XmlResponsesSaxParser.java:141)
    ... 18 more
Invalid arguments: Unable to unmarshall response (Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler). Response Code: 200, Response Text: OK
usage: distcp OPTIONS [source_path...] <target_path>
              OPTIONS
 -append                Reuse existing data in target files and append new
                        data to them if possible
 -async                 Should distcp execution be blocking
 -atomic                Commit all changes or none
 -bandwidth <arg>       Specify bandwidth per map in MB
 -delete                Delete from target, files missing in source
 -diff <arg>            Use snapshot diff report to identify the
                        difference between source and target
 -f <arg>               List of files that need to be copied
 -filelimit <arg>       (Deprecated!) Limit number of files copied to <= n
 -i                     Ignore failures during copy
 -log <arg>             Folder on DFS where distcp execution logs are
                        saved
 -m <arg>               Max number of concurrent maps to use for copy
 -mapredSslConf <arg>   Configuration for ssl config file, to use with
                        hftps://
 -overwrite             Choose to overwrite target files unconditionally,
                        even if they exist.
 -p <arg>               preserve status (rbugpcaxt)(replication,
                        block-size, user, group, permission,
                        checksum-type, ACL, XATTR, timestamps). If -p is
                        specified with no <arg>, then preserves
                        replication, block size, user, group, permission,
                        checksum type and timestamps. raw.* xattrs are
                        preserved when both the source and destination
                        paths are in the /.reserved/raw hierarchy (HDFS
                        only). raw.* xattrpreservation is independent of
                        the -p flag. Refer to the DistCp documentation for
                        more details.
 -sizelimit <arg>       (Deprecated!) Limit number of files copied to <= n
                        bytes
 -skipcrccheck          Whether to skip CRC checks between source and
                        target paths.
 -strategy <arg>        Copy strategy to use. Default is dividing work
                        based on file sizes
 -tmp <arg>             Intermediate work path to be used for atomic
                        commit
 -update                Update target, copying only missingfiles or
                        directories

Here are some other hadoop configurations I tried aswell, with no luck.

-Dfs.s3a.connection.ssl.enabled=false : since my endpoint is http.

-Dfs.s3a.region=eu-west-1

Am I missing something ?

UPDATE:

As the error message also includes Invalid arguments:  I suppose I might be providing some invalid chars into the args, so I tried writing these options in /etc/hadoop/conf/core-site.xml as follows;

<property>
 <name>fs.s3a.endpoint</name>
 <value>http://10.91.16.213:8080</value>
</property>

<property>
 <name>fs.s3a.fast.upload</name>
 <value>true</value>
</property>

<property>
 <name>fs.s3a.access.key</name>
 <value>AKIACEMGMYDQNJXGQ2DEOBXG42SQCFR2ZJFTDED3HX3KLVTLOIN6AH3FSDHUF</value>
</property>

<property>
 <name>fs.s3a.secret.key</name>
 <value><snowball-secret></value>
</property>

but got same error message :(

UPDATE 2:

After reading this, it looks like s3 xml parsing problem while doing ListObjects. AWS client has this option .withEncodingType("url");, but couldn't find anything similar for hadoop distcp.


问:
React state change best practices: useEffect vs callback
I have a question regarding best practices when using useEffect and when to use it to handle state changes.

For example let's say I have a button that will send an email on the 5th click (pseudo code as I didn't really test to see if it works). What's better according to React best practices.

Here I am using useEffect to handle the count state change:

```<code>import React, { useState, useEffect } from 'react'
export default () => {
    const [count, setCount] = useState(0)
    useEffect(() => {
        if (count === 5) {
            sendEmail()
            setCount(0)
        }
    }, [count])

    return (
        <div>
            <input
                type="button"
                onClick={() => {
                    setCount((count) => count + 1)
                }}
            ></input>
        </div>
    )
}
</code>```

Here is the same functionality with a callback:

```<code>import React, { useState } from 'react'
export default () => {
    const [count, setCount] = useState(0)

    return (
        <div>
            <input
                type="button"
                onClick={() => {
                    if(count + 1 === 5) {
                        sendEmail()
                        setCount(0)
                        return
                    }
                    setCount(count=>count+1)
                }}
            ></input>
        </div>
    )
}
</code>```

The argument for using useEffect, at least as I see it, is to react to state changes. However if I want to check other state in that useEffect block or other dependencies, then I have to add it to the dependency array and it's in danger of re-running even though I'm only interested in running it in response to the count changing (one piece of state). In this example it isn't too egregious but in more advanced situations it seems that you have to add a bunch of conditionals in the useEffect block if you only want it to react to one piece of state change. if(x,y,z) {run count logic}

The argument for the callback is that it's readable and will only be called on the button being pressed however it is more imperative than declarative and not as REACTive as useEffect. Furthermore with the setCount being async it might create a race condition (not in this case but in more advanced cases).

Also I can use useEffect to run code when a component has first rendered but if it has logic that touches dependencies I have to add them to the dependency array according to my linter and Dan Abramov but then I run the chance of running it multiple times unless I add a bunch of conditionals in the body of useEffect (as I mentioned above). In short I may want to run the useEffect on one piece of state change and not when any of the dependencies change.

I hope that makes sense.

An example of the useEffect issue with conditionals is if I want to send the email if some localState variable === "whatever". Once I add the localState variable into the dep array, then it will run whenever it changes even if I only want to run it when specific state (count) changes. The only way around this I see is adding logic on the useEffect to exit early but it starts to get complicated and error prone.

答:
In your kind of situation, it's better to just call sendmail within your click function (option 2) to avoid unnecessary rerendering.

With regards to question of "race condition", instead of using setCount(0), you can always use a ref to store the count. You can use ref to store the count since you do not need to trigger a rerender when your count changes.

Otherwise, instead of resetting to 0, you can always do

```<code>if ((count % 5) === 0) {
 sendmail()
}
</code>```

Body:
I have a question regarding best practices when using useEffect and when to use it to handle state changes.

For example let's say I have a button that will send an email on the 5th click (pseudo code as I didn't really test to see if it works). What's better according to React best practices.

Here I am using useEffect to handle the count state change:

import React, { useState, useEffect } from 'react'
export default () => {
    const [count, setCount] = useState(0)
    useEffect(() => {
        if (count === 5) {
            sendEmail()
            setCount(0)
        }
    }, [count])

    return (

            <input
                type="button"
                onClick={() => {
                    setCount((count) => count + 1)
                }}
            ></input>

    )
}

Here is the same functionality with a callback:

import React, { useState } from 'react'
export default () => {
    const [count, setCount] = useState(0)

    return (

            <input
                type="button"
                onClick={() => {
                    if(count + 1 === 5) {
                        sendEmail()
                        setCount(0)
                        return
                    }
                    setCount(count=>count+1)
                }}
            ></input>

    )
}

The argument for using useEffect, at least as I see it, is to react to state changes. However if I want to check other state in that useEffect block or other dependencies, then I have to add it to the dependency array and it's in danger of re-running even though I'm only interested in running it in response to the count changing (one piece of state). In this example it isn't too egregious but in more advanced situations it seems that you have to add a bunch of conditionals in the useEffect block if you only want it to react to one piece of state change. if(x,y,z) {run count logic}

The argument for the callback is that it's readable and will only be called on the button being pressed however it is more imperative than declarative and not as REACTive as useEffect. Furthermore with the setCount being async it might create a race condition (not in this case but in more advanced cases).

Also I can use useEffect to run code when a component has first rendered but if it has logic that touches dependencies I have to add them to the dependency array according to my linter and Dan Abramov but then I run the chance of running it multiple times unless I add a bunch of conditionals in the body of useEffect (as I mentioned above). In short I may want to run the useEffect on one piece of state change and not when any of the dependencies change.

I hope that makes sense.

An example of the useEffect issue with conditionals is if I want to send the email if some localState variable === "whatever". Once I add the localState variable into the dep array, then it will run whenever it changes even if I only want to run it when specific state (count) changes. The only way around this I see is adding logic on the useEffect to exit early but it starts to get complicated and error prone.

Body:
In your kind of situation, it's better to just call sendmail within your click function (option 2) to avoid unnecessary rerendering.

With regards to question of "race condition", instead of using setCount(0), you can always use a ref to store the count. You can use ref to store the count since you do not need to trigger a rerender when your count changes.

Otherwise, instead of resetting to 0, you can always do

if ((count % 5) === 0) {
 sendmail()
}


问:
Beautiful soup returns none even when there is an element
Trying to filter the products name list using the header tags, but it always returns none.

source : https://www.tendercuts.in/chicken

code :

```<code>import requests
from bs4 import BeautifulSoup
def ExtractData(url):
response = requests.get(url=url).content
soup = BeautifulSoup(response, 'lxml')
header = soup.find("mat-card-header", {"class": "mat-card-header ng-tns- c9-188"})
print(header)
ExtractData(url="https://www.tendercuts.in/chicken")
</code>```

答:
This is the most common problem with web scraping: most websites use JavaScript to change or add to the content on the page after loading the initial page. Whatever the JavaScript is supposed to change or load isn't on the page after the initial request.

The same is true for your code. If you look at the actual HTML (not in a browser, in your code), you'll find that it has many fields that angular.js code will be filling in later.

You'll need to load your page using a package like selenium, which uses a browser driver to load the page, execute the JavaScript and make the result available to you. (it does a lot more, like allowing you to navigate the site by clicking it, filling out fields, etc.)

selenium is a complex library with many options, but you can get started with:

```<code>pip install selenium
</code>```

And by downloading a browser driver like Gecko Driver or ChromeDriver

And then something like this will work:

```<code>from selenium import webdriver
from selenium.webdriver.chrome.service import Service  # for Chrome
from selenium.webdriver.firefox.service import Service  # for Firefox

service = Service('/path/to/driver')

service.start()

driver = webdriver.Remote(service.service_url)

driver.get('https://www.tendercuts.in/chicken');

# do something with what driver loaded here

driver.quit()
</code>```

You could just bs4 your way through driver.page_source, but since you now have selenium anyway, you could also look into the ways selenium allows you to find and select elements, like using the built-in XPath functions.

Body:
Trying to filter the products name list using the header tags, but it always returns none.

source : https://www.tendercuts.in/chicken

code :

import requests
from bs4 import BeautifulSoup
def ExtractData(url):
response = requests.get(url=url).content
soup = BeautifulSoup(response, 'lxml')
header = soup.find("mat-card-header", {"class": "mat-card-header ng-tns- c9-188"})
print(header)
ExtractData(url="https://www.tendercuts.in/chicken")

Body:
This is the most common problem with web scraping: most websites use JavaScript to change or add to the content on the page after loading the initial page. Whatever the JavaScript is supposed to change or load isn't on the page after the initial request.

The same is true for your code. If you look at the actual HTML (not in a browser, in your code), you'll find that it has many fields that angular.js code will be filling in later.

You'll need to load your page using a package like selenium, which uses a browser driver to load the page, execute the JavaScript and make the result available to you. (it does a lot more, like allowing you to navigate the site by clicking it, filling out fields, etc.)

selenium is a complex library with many options, but you can get started with:

pip install selenium

And by downloading a browser driver like Gecko Driver or ChromeDriver

And then something like this will work:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service  # for Chrome
from selenium.webdriver.firefox.service import Service  # for Firefox

service = Service('/path/to/driver')

service.start()

driver = webdriver.Remote(service.service_url)

driver.get('https://www.tendercuts.in/chicken');

# do something with what driver loaded here

driver.quit()

You could just bs4 your way through driver.page_source, but since you now have selenium anyway, you could also look into the ways selenium allows you to find and select elements, like using the built-in XPath functions.


问:
KeyError: 'values' while deleting part of dictionary
I have a dictionary of dictionaries (see bottom of post), and as you can see, the unpacked_conversation_dict[convers]["question"]["values"] exist for every entry:

```<code>print("length of dict is: ", len(unpacked_conversation_dict))
for convers in unpacked_conversation_dict:
    print("In %s" % convers, unpacked_conversation_dict[convers]["question"]["values"])
</code>```

Output:

```<code>length of dict is: 9
In conversation1 {'Young': 0.3, 'Adult': 0.3, '50+': 0.3}
In conversation2 {'Young': 0.8, 'Adult': 0.8, '50+': 0.8}
In conversation3 {'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
In conversation4 {'Young': -0.5, 'Adult': -0.5, '50+': -0.5}
In conversation5 {'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
In conversation6 {'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
In conversation7 {'Young': 0.3, 'Adult': 0.3, '50+': 0.3}
In conversation8 {'Young': 0.8, 'Adult': 0.8, '50+': 0.8}
In conversation9 {'Young': -0.3, 'Adult': -0.3, '50+': -0.3}
</code>```

The error occurs when this code is run:

```<code>for convers in unpacked_conversation_dict:
    print(convers)
    print(unpacked_conversation_dict[convers]["question"]["values"])
    del unpacked_conversation_dict[convers]["question"]["values"]
    del unpacked_conversation_dict[convers]["question"]["question_type"]
</code>```

Output:

```<code>conversation1
{'Young': 0.3, 'Adult': 0.3, '50+': 0.3}
conversation2
{'Young': 0.8, 'Adult': 0.8, '50+': 0.8}
conversation3
{'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
conversation4
{'Young': -0.5, 'Adult': -0.5, '50+': -0.5}
conversation5
{'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
conversation6
{'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
conversation7

KeyError: 'values'
</code>```

Dictionary of dictionaries:

```<code>{'conversation1': {'user1': {'age': 'Young',
   'gender': 'M',
   'hobbies': ['social media', 'games']},
  'user2': {'age': 'Adult',
   'gender': 'F',
   'hobbies': ['wine', 'social media', 'business']},
  'question': {'question_number': 7,
   'question_type': 'music',
   'values': {'Young': 0.3, 'Adult': 0.3, '50+': 0.3}}},
 'conversation2': {'user1': {'age': 'Adult',
   'gender': 'M',
   'hobbies': ['cars', 'social media', 'business']},
  'user2': {'age': 'Young', 'gender': 'M', 'hobbies': ['business', 'music']},
  'question': {'question_number': 10,
   'question_type': 'games',
   'values': {'Young': 0.8, 'Adult': 0.8, '50+': 0.8}}},
 'conversation3': {'user1': {'age': 'Young',
   'gender': 'M',
   'hobbies': ['social media', 'games']},
  'user2': {'age': '50+', 'gender': 'M', 'hobbies': ['cars', 'business']},
  'question': {'question_number': 4,
   'question_type': 'games',
   'values': {'Young': 0.5, 'Adult': 0.5, '50+': 0.5}}},
 'conversation4': {'user1': {'age': '50+',
   'gender': 'F',
   'hobbies': ['wine', 'games', 'business']},
  'user2': {'age': '50+', 'gender': 'M', 'hobbies': ['cars', 'wine', 'music']},
  'question': {'question_number': 14,
   'question_type': 'business',
   'values': {'Young': -0.5, 'Adult': -0.5, '50+': -0.5}}},

...

 'conversation9': {'user1': {'age': 'Adult',
   'gender': 'F',
   'hobbies': ['wine', 'social media', 'business']},
  'user2': {'age': 'Young',
   'gender': 'F',
   'hobbies': ['social media', 'music']},
  'question': {'question_number': 6,
   'question_type': 'business',
   'values': {'Young': -0.3, 'Adult': -0.3, '50+': -0.3}}}}

</code>```

It's been hard to make a minimal reproducible example for this, please excuse this.

答:
You probably have the same dict value object for two different keys.

You go through each conversation and modify its value dict. When you hit conversionation7, its dict value is the same dict (by identity, not by value) as one of the previous, which has already been modified.

example:

value = {'foo': 1}
conversations = {
    'conversation1': value,
    'conversation2': value,
}

for key, value in conversations.items():
    del value['foo']

# raises `KeyError: 'foo'`
```
<p>Based on your print output, it looks like you'll find:</p>
<pre class="lang-py prettyprint-override"><code># "is" will check for identity, not value equality
unpacked_conversation_dict['conversionation1'] is unpacked_conversation_dict['conversionation7']
</code>```

Just to close it out, some general advice: I don't know what you're ultimately using this logic for, but code is generally easier to reason about by:

reducing the amount of state change & mutation being done (stateless functions)

localizing when/where mutation is done (encapsulating state in local scope/objects)

You can perhaps ask a new question about how to accomplish the above by providing context on "what you're ultimately using this for".

Body:
I have a dictionary of dictionaries (see bottom of post), and as you can see, the unpacked_conversation_dict[convers]["question"]["values"] exist for every entry:

print("length of dict is: ", len(unpacked_conversation_dict))
for convers in unpacked_conversation_dict:
    print("In %s" % convers, unpacked_conversation_dict[convers]["question"]["values"])

Output:

length of dict is: 9
In conversation1 {'Young': 0.3, 'Adult': 0.3, '50+': 0.3}
In conversation2 {'Young': 0.8, 'Adult': 0.8, '50+': 0.8}
In conversation3 {'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
In conversation4 {'Young': -0.5, 'Adult': -0.5, '50+': -0.5}
In conversation5 {'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
In conversation6 {'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
In conversation7 {'Young': 0.3, 'Adult': 0.3, '50+': 0.3}
In conversation8 {'Young': 0.8, 'Adult': 0.8, '50+': 0.8}
In conversation9 {'Young': -0.3, 'Adult': -0.3, '50+': -0.3}

The error occurs when this code is run:

for convers in unpacked_conversation_dict:
    print(convers)
    print(unpacked_conversation_dict[convers]["question"]["values"])
    del unpacked_conversation_dict[convers]["question"]["values"]
    del unpacked_conversation_dict[convers]["question"]["question_type"]

Output:

conversation1
{'Young': 0.3, 'Adult': 0.3, '50+': 0.3}
conversation2
{'Young': 0.8, 'Adult': 0.8, '50+': 0.8}
conversation3
{'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
conversation4
{'Young': -0.5, 'Adult': -0.5, '50+': -0.5}
conversation5
{'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
conversation6
{'Young': 0.5, 'Adult': 0.5, '50+': 0.5}
conversation7

KeyError: 'values'

Dictionary of dictionaries:

{'conversation1': {'user1': {'age': 'Young',
   'gender': 'M',
   'hobbies': ['social media', 'games']},
  'user2': {'age': 'Adult',
   'gender': 'F',
   'hobbies': ['wine', 'social media', 'business']},
  'question': {'question_number': 7,
   'question_type': 'music',
   'values': {'Young': 0.3, 'Adult': 0.3, '50+': 0.3}}},
 'conversation2': {'user1': {'age': 'Adult',
   'gender': 'M',
   'hobbies': ['cars', 'social media', 'business']},
  'user2': {'age': 'Young', 'gender': 'M', 'hobbies': ['business', 'music']},
  'question': {'question_number': 10,
   'question_type': 'games',
   'values': {'Young': 0.8, 'Adult': 0.8, '50+': 0.8}}},
 'conversation3': {'user1': {'age': 'Young',
   'gender': 'M',
   'hobbies': ['social media', 'games']},
  'user2': {'age': '50+', 'gender': 'M', 'hobbies': ['cars', 'business']},
  'question': {'question_number': 4,
   'question_type': 'games',
   'values': {'Young': 0.5, 'Adult': 0.5, '50+': 0.5}}},
 'conversation4': {'user1': {'age': '50+',
   'gender': 'F',
   'hobbies': ['wine', 'games', 'business']},
  'user2': {'age': '50+', 'gender': 'M', 'hobbies': ['cars', 'wine', 'music']},
  'question': {'question_number': 14,
   'question_type': 'business',
   'values': {'Young': -0.5, 'Adult': -0.5, '50+': -0.5}}},

...

 'conversation9': {'user1': {'age': 'Adult',
   'gender': 'F',
   'hobbies': ['wine', 'social media', 'business']},
  'user2': {'age': 'Young',
   'gender': 'F',
   'hobbies': ['social media', 'music']},
  'question': {'question_number': 6,
   'question_type': 'business',
   'values': {'Young': -0.3, 'Adult': -0.3, '50+': -0.3}}}}

It's been hard to make a minimal reproducible example for this, please excuse this.

Body:
You probably have the same dict value object for two different keys.

You go through each conversation and modify its value dict. When you hit conversionation7, its dict value is the same dict (by identity, not by value) as one of the previous, which has already been modified.

example:

value = {'foo': 1}
conversations = {
    'conversation1': value,
    'conversation2': value,
}

for key, value in conversations.items():
    del value['foo']

# raises `KeyError: 'foo'`

Based on your print output, it looks like you'll find:

# "is" will check for identity, not value equality
unpacked_conversation_dict['conversionation1'] is unpacked_conversation_dict['conversionation7']

Just to close it out, some general advice: I don't know what you're ultimately using this logic for, but code is generally easier to reason about by:

reducing the amount of state change & mutation being done (stateless functions)

localizing when/where mutation is done (encapsulating state in local scope/objects)

You can perhaps ask a new question about how to accomplish the above by providing context on "what you're ultimately using this for".


问:
Can anyone define the Windows PE Checksum Algorithm?
I would like to implement this in C#

I have looked here:
http://www.codeproject.com/KB/cpp/PEChecksum.aspx

And am aware of the ImageHlp.dll MapFileAndCheckSum function.

However, for various reasons, I would like to implement this myself.

The best I have found is here:
http://forum.sysinternals.com/optional-header-checksum-calculation_topic24214.html

But, I don't understand the explanation. Can anyone clarify how the checksum is calculated?

Thanks!

Update

I from the code example, I do not understand what this means, and how to translate it into C#

```<code>sum -= sum < low 16 bits of CheckSum in file // 16-bit borrow
sum -= low 16 bits of CheckSum in file
sum -= sum < high 16 bits of CheckSum in file
sum -= high 16 bits of CheckSum in file
</code>```

Update #2

Thanks, came across some Python code that does similar too here

```<code>    def generate_checksum(self):

    # This will make sure that the data representing the PE image
    # is updated with any changes that might have been made by
    # assigning values to header fields as those are not automatically
    # updated upon assignment.
    #
    self.__data__ = self.write()

    # Get the offset to the CheckSum field in the OptionalHeader
    #
    checksum_offset = self.OPTIONAL_HEADER.__file_offset__ + 0x40 # 64

    checksum = 0

    # Verify the data is dword-aligned. Add padding if needed
    #
    remainder = len(self.__data__) % 4
    data = self.__data__ + ( '\0' * ((4-remainder) * ( remainder != 0 )) )

    for i in range( len( data ) / 4 ):

        # Skip the checksum field
        #
        if i == checksum_offset / 4:
            continue

        dword = struct.unpack('I', data[ i*4 : i*4+4 ])[0]
        checksum = (checksum & 0xffffffff) + dword + (checksum>>32)
        if checksum > 2**32:
            checksum = (checksum & 0xffffffff) + (checksum >> 32)

    checksum = (checksum & 0xffff) + (checksum >> 16)
    checksum = (checksum) + (checksum >> 16)
    checksum = checksum & 0xffff

    # The length is the one of the original data, not the padded one
    #
    return checksum + len(self.__data__)
</code>```

However, it's still not working for me - here is my conversion of this code:

```<code>using System;
using System.IO;

namespace CheckSumTest
{
    class Program
    {
        static void Main(string[] args)
        {
            var data = File.ReadAllBytes(@"c:\Windows\notepad.exe");

            var PEStart = BitConverter.ToInt32(data, 0x3c);
            var PECoffStart = PEStart + 4;
            var PEOptionalStart = PECoffStart + 20;
            var PECheckSum = PEOptionalStart + 64;
            var checkSumInFile = BitConverter.ToInt32(data, PECheckSum);
            Console.WriteLine(string.Format("{0:x}", checkSumInFile));

            long checksum = 0;

            var remainder = data.Length % 4;
            if (remainder > 0)
            {
                Array.Resize(ref data, data.Length + (4 - remainder));
            }

            var top = Math.Pow(2, 32);

            for (int i = 0; i < data.Length / 4; i++)
            {
                if (i == PECheckSum / 4)
                {
                    continue;
                }
                var dword = BitConverter.ToInt32(data, i * 4);
                checksum = (checksum & 0xffffffff) + dword + (checksum >> 32);
                if (checksum > top)
                {
                    checksum = (checksum & 0xffffffff) + (checksum >> 32);
                }
            }

            checksum = (checksum & 0xffff) + (checksum >> 16);
            checksum = (checksum) + (checksum >> 16);
            checksum = checksum & 0xffff;

            checksum += (uint)data.Length;
            Console.WriteLine(string.Format("{0:x}", checksum));

            Console.ReadKey();
        }
    }
}
</code>```

Can anyone tell me where I'm being stupid?

答:
The CheckSum field is 32 bits long and is calculated as follows

1. Add all dwords (32 bit pieces) of the entire file to a sum

Add all dwords of the entire file not including the CheckSum field itself, including all headers and all of the contents, to a dword. If the dword overflows, add the overflowed bit back to the first bit (2^0) of the dword.
If the file is not entirely divisible into dwords (4 bit pieces) see 2.

The best way I know to realize this is by using the GNU C Compilers Integer Overflow Builtin function __builtin_uadd_overflow.
In the original ChkSum function documented by Jeffrey Walton the sum
was calculated by performing an add (%esi),%eax where

esi contains the base address of the file and eax is 0 and adding the rest of the file like this

```<code>adc 0x4(%esi),%eax
adc 0x8(%esi),%eax
adc 0xc(%esi),%eax
adc 0x10(%esi),%eax
...
adc $0x0,%eax
</code>```

The first add adds the first dword ignoring any carry flag. The next dwords

are added by the adc instruction which does the same thing as add but

adds any carry flag that was set before executing the instruction in addition

to the summand. The last adc $0x0,%eax adds only the last carry flag if it

was set and cannot be discarded.

Please keep in mind that the dword of CheckSum field itself should not be added.

2. Add the remainder to the sum if there is one

If the file is not entirely divisible into dwords, add the remainder as a

zero-padded dword. For example: say your file is 15 bytes long and looks like this

0E 1F BA 0E | 00 B4 09 CD | 21 B8 01 4C | CD 21 54

You need to add the remainder as 0x005421CD to the sum. My system is a

little-endian system. I do not know if the checksum would change because of the

this order of the bytes on big-endian systems, or you would just simulate this

behaviour.

I do this by rounding up the buffer_size to the next bytecount divisible by 4

without remainder or put differently: the next whole dword count represented

in bytes. Then I allocate with calloc because it initializes the memory block

with all zeros.

```<code>if(buffer_size%4)
  {buffer_size+=4-(buffer_size%4);
...
calloc(buffer_size,1)
</code>```

3. Add the lower word (16 bit piece) and the higher word of the sum together.

sum=(sum&0xffff)+(sum>>16);

4. Add the new higher word once again

sum+=(sum>>16);

5. Only keep the lower word

sum&=0xffff;

6. Add the number of bytes in the file to the sum

return(sum+size);

This is how I wrote it. It is not C#, but C. off_t size is the number of bytes in the file. uint32_t *base is a pointer to the file loaded into memory. The block of memory should be padded with zeros at the end to the next bytecount divisible by 4.

```<code>uint32_t pe_header_checksum(uint32_t *base,off_t size)
  {uint32_t sum=0;
  off_t i;
  for(i=0;i<(size/4);i++)
    {if(i==0x36)
      {continue;}
    sum+=__builtin_uadd_overflow(base[i],sum,&sum);}
  if(size%4)
    {sum+=base[i];}
  sum=(sum&0xffff)+(sum>>16);
  sum+=(sum>>16);
  sum&=0xffff;
  return(sum+size);}
</code>```

If you want you can see the code in action and read a little bit more here.

Body:
I would like to implement this in C#

I have looked here:
http://www.codeproject.com/KB/cpp/PEChecksum.aspx

And am aware of the ImageHlp.dll MapFileAndCheckSum function.

However, for various reasons, I would like to implement this myself.

The best I have found is here:
http://forum.sysinternals.com/optional-header-checksum-calculation_topic24214.html

But, I don't understand the explanation. Can anyone clarify how the checksum is calculated?

Thanks!

Update

I from the code example, I do not understand what this means, and how to translate it into C#

sum -= sum < low 16 bits of CheckSum in file // 16-bit borrow
sum -= low 16 bits of CheckSum in file
sum -= sum < high 16 bits of CheckSum in file
sum -= high 16 bits of CheckSum in file

Update #2

Thanks, came across some Python code that does similar too here

    def generate_checksum(self):

    # This will make sure that the data representing the PE image
    # is updated with any changes that might have been made by
    # assigning values to header fields as those are not automatically
    # updated upon assignment.
    #
    self.__data__ = self.write()

    # Get the offset to the CheckSum field in the OptionalHeader
    #
    checksum_offset = self.OPTIONAL_HEADER.__file_offset__ + 0x40 # 64

    checksum = 0

    # Verify the data is dword-aligned. Add padding if needed
    #
    remainder = len(self.__data__) % 4
    data = self.__data__ + ( '\0' * ((4-remainder) * ( remainder != 0 )) )

    for i in range( len( data ) / 4 ):

        # Skip the checksum field
        #
        if i == checksum_offset / 4:
            continue

        dword = struct.unpack('I', data[ i*4 : i*4+4 ])[0]
        checksum = (checksum & 0xffffffff) + dword + (checksum>>32)
        if checksum > 2**32:
            checksum = (checksum & 0xffffffff) + (checksum >> 32)

    checksum = (checksum & 0xffff) + (checksum >> 16)
    checksum = (checksum) + (checksum >> 16)
    checksum = checksum & 0xffff

    # The length is the one of the original data, not the padded one
    #
    return checksum + len(self.__data__)

However, it's still not working for me - here is my conversion of this code:

using System;
using System.IO;

namespace CheckSumTest
{
    class Program
    {
        static void Main(string[] args)
        {
            var data = File.ReadAllBytes(@"c:\Windows\notepad.exe");

            var PEStart = BitConverter.ToInt32(data, 0x3c);
            var PECoffStart = PEStart + 4;
            var PEOptionalStart = PECoffStart + 20;
            var PECheckSum = PEOptionalStart + 64;
            var checkSumInFile = BitConverter.ToInt32(data, PECheckSum);
            Console.WriteLine(string.Format("{0:x}", checkSumInFile));

            long checksum = 0;

            var remainder = data.Length % 4;
            if (remainder > 0)
            {
                Array.Resize(ref data, data.Length + (4 - remainder));
            }

            var top = Math.Pow(2, 32);

            for (int i = 0; i < data.Length / 4; i++)
            {
                if (i == PECheckSum / 4)
                {
                    continue;
                }
                var dword = BitConverter.ToInt32(data, i * 4);
                checksum = (checksum & 0xffffffff) + dword + (checksum >> 32);
                if (checksum > top)
                {
                    checksum = (checksum & 0xffffffff) + (checksum >> 32);
                }
            }

            checksum = (checksum & 0xffff) + (checksum >> 16);
            checksum = (checksum) + (checksum >> 16);
            checksum = checksum & 0xffff;

            checksum += (uint)data.Length;
            Console.WriteLine(string.Format("{0:x}", checksum));

            Console.ReadKey();
        }
    }
}

Can anyone tell me where I'm being stupid?

Body:
The CheckSum field is 32 bits long and is calculated as follows

1. Add all dwords (32 bit pieces) of the entire file to a sum

Add all dwords of the entire file not including the CheckSum field itself, including all headers and all of the contents, to a dword. If the dword overflows, add the overflowed bit back to the first bit (2^0) of the dword.
If the file is not entirely divisible into dwords (4 bit pieces) see 2.

The best way I know to realize this is by using the GNU C Compilers Integer Overflow Builtin function __builtin_uadd_overflow.
In the original ChkSum function documented by Jeffrey Walton the sum
was calculated by performing an add (%esi),%eax where

esi contains the base address of the file and eax is 0 and adding the rest of the file like this

adc 0x4(%esi),%eax
adc 0x8(%esi),%eax
adc 0xc(%esi),%eax
adc 0x10(%esi),%eax
...
adc $0x0,%eax

The first add adds the first dword ignoring any carry flag. The next dwords

are added by the adc instruction which does the same thing as add but

adds any carry flag that was set before executing the instruction in addition

to the summand. The last adc $0x0,%eax adds only the last carry flag if it

was set and cannot be discarded.

Please keep in mind that the dword of CheckSum field itself should not be added.

2. Add the remainder to the sum if there is one

If the file is not entirely divisible into dwords, add the remainder as a

zero-padded dword. For example: say your file is 15 bytes long and looks like this

0E 1F BA 0E | 00 B4 09 CD | 21 B8 01 4C | CD 21 54

You need to add the remainder as 0x005421CD to the sum. My system is a

little-endian system. I do not know if the checksum would change because of the

this order of the bytes on big-endian systems, or you would just simulate this

behaviour.

I do this by rounding up the buffer_size to the next bytecount divisible by 4

without remainder or put differently: the next whole dword count represented

in bytes. Then I allocate with calloc because it initializes the memory block

with all zeros.

if(buffer_size%4)
  {buffer_size+=4-(buffer_size%4);
...
calloc(buffer_size,1)

3. Add the lower word (16 bit piece) and the higher word of the sum together.

sum=(sum&0xffff)+(sum>>16);

4. Add the new higher word once again

sum+=(sum>>16);

5. Only keep the lower word

sum&=0xffff;

6. Add the number of bytes in the file to the sum

return(sum+size);

This is how I wrote it. It is not C#, but C. off_t size is the number of bytes in the file. uint32_t *base is a pointer to the file loaded into memory. The block of memory should be padded with zeros at the end to the next bytecount divisible by 4.

uint32_t pe_header_checksum(uint32_t *base,off_t size)
  {uint32_t sum=0;
  off_t i;
  for(i=0;i<(size/4);i++)
    {if(i==0x36)
      {continue;}
    sum+=__builtin_uadd_overflow(base[i],sum,&sum);}
  if(size%4)
    {sum+=base[i];}
  sum=(sum&0xffff)+(sum>>16);
  sum+=(sum>>16);
  sum&=0xffff;
  return(sum+size);}

If you want you can see the code in action and read a little bit more here.


问:
Conditional regex to use with sed having trouble
Hi I am having trouble to understand why this regex:

```<code>(candy)?(?(1)A| is false)
</code>```

for this String: A candy is true is false is not matching the A, and instead is matching  is false

What I'm really trying to do is, for example, if I have a Windows in this string:

```<code>Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) is false.
</code>```

I want to substitute all this string to just Windows using sed command in Bash. If the string doesn't contain Windows, it's supposed to just replace with -.

The bash command I'm trying to use is:

```<code>sed -i 's/(?(?=Windows)("Mozilla.*)|(?!))/Windows/' accesslog.txt
</code>```

答:
First of all, sed's POSIX regex flavor does not support conditional constructs. What you want to do has nothing to do with regex conditional constructs as all they do is match further text depending on the previous group match, or text immediately before or after the current location.

You want something like

#!/bin/bash
s='Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) is false.
some other line'
sed 's/.*\(Windows\).*/\1/; t; s/.*/-/' <<< "$s"
```
<p>See the online demo. Output:</p>
<pre class="lang-none prettyprint-override"><code>Windows
-
</code>```

The 's/.*\(Windows\).*/\1/; t; s/.*/-/' command means:

s/.*\(Windows\).*/\1/ - replace the whole line that contains Windows (captured into Group 1 with a capturing group) with the Group 1 value

t - jump to the end of the command  upon successful substitution

s/.*/-/ - else, replace the whole line with -.

Body:
Hi I am having trouble to understand why this regex:

(candy)?(?(1)A| is false)

for this String: A candy is true is false is not matching the A, and instead is matching  is false

What I'm really trying to do is, for example, if I have a Windows in this string:

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) is false.

I want to substitute all this string to just Windows using sed command in Bash. If the string doesn't contain Windows, it's supposed to just replace with -.

The bash command I'm trying to use is:

sed -i 's/(?(?=Windows)("Mozilla.*)|(?!))/Windows/' accesslog.txt

Body:
First of all, sed's POSIX regex flavor does not support conditional constructs. What you want to do has nothing to do with regex conditional constructs as all they do is match further text depending on the previous group match, or text immediately before or after the current location.

You want something like

#!/bin/bash
s='Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) is false.
some other line'
sed 's/.*\(Windows\).*/\1/; t; s/.*/-/' <<< "$s"

See the online demo. Output:

Windows
-

The 's/.*\(Windows\).*/\1/; t; s/.*/-/' command means:

s/.*\(Windows\).*/\1/ - replace the whole line that contains Windows (captured into Group 1 with a capturing group) with the Group 1 value

t - jump to the end of the command  upon successful substitution

s/.*/-/ - else, replace the whole line with -.


问:
VS code can't execute Hello World program (Error)
I just started to learn Python on Visual Studio Code, I was about to write my first Hello world program but it gives me this error:

```<code>print("Hello", end=" ")
print("World!")

print("Hello", end=" ")
                  ^
SyntaxError: invalid syntax
</code>```

Am I doing something wrong? Please help me, thanks for the help!

I'm on a MacBook Pro M1 Max

答: You are using python 2.x version. end= only works for python 3.x

Body:
I just started to learn Python on Visual Studio Code, I was about to write my first Hello world program but it gives me this error:

print("Hello", end=" ")
print("World!")

print("Hello", end=" ")
                  ^
SyntaxError: invalid syntax

Am I doing something wrong? Please help me, thanks for the help!

I'm on a MacBook Pro M1 Max