问:
Taguchi Method Programming Example
I've been asked to research some programming related to the "Taguchi Method", especially as it relates to Multi-variant testing. This is one of the first subjects I've tried to research that I've found zero, nada, zilch, code examples for, especially considering its mathematical basis.

I've found some books describing the math involved but it looks like I'm going to be doing some math brush up unless I can find some code examples I can relate to.

Is this one of those rare things that once you work out the programming, it's so valuable that no one shares? Or do I just fail at Taguchi + google?

答:
Taguchi designs are the same thing as covering arrays. The basic idea is that if you have F data "fields" and every one can have N different values, it is possible to construct NF different test cases. A covering array is basically a set of test cases that together cover all possible pairwise combinations of two field values, and the idea is to generate as small one as possible. E.g. if F=3 and N=3, you have 27 possible test cases, but it is enough to have nine test cases if you aim for pairwise coverage:

```<code>    Field A | Field B | Field C
    ---------------------------
          1         1         1
          1         2         2
          1         3         3
          2         1         2
          2         2         3
          2         3         1
          3         1         3
          3         2         1
          3         3         2
</code>```

In this table, you can choose any two fields and any two values and you can always find a row that contains the chosen values for the chosen fields.

Generating Taguchi designs in general is a difficult combinatorial problem.

You can generate Taguchi designs by various methods:

Branch and bound

Stochastic search (e.g. tabu search or simulated annealing)

Greedy search

Specific mathematical constructions for some specific structures

Title: Taguchi Method Programming Example

Body:
Taguchi designs are the same thing as covering arrays. The basic idea is that if you have F data "fields" and every one can have N different values, it is possible to construct NF different test cases. A covering array is basically a set of test cases that together cover all possible pairwise combinations of two field values, and the idea is to generate as small one as possible. E.g. if F=3 and N=3, you have 27 possible test cases, but it is enough to have nine test cases if you aim for pairwise coverage:

    Field A | Field B | Field C
    ---------------------------
          1         1         1
          1         2         2
          1         3         3
          2         1         2
          2         2         3
          2         3         1
          3         1         3
          3         2         1
          3         3         2

In this table, you can choose any two fields and any two values and you can always find a row that contains the chosen values for the chosen fields.

Generating Taguchi designs in general is a difficult combinatorial problem.

You can generate Taguchi designs by various methods:

Branch and bound

Stochastic search (e.g. tabu search or simulated annealing)

Greedy search

Specific mathematical constructions for some specific structures


来源: stackexchange

问:
Put time limit on Jest test?
I'm running a set of tests with Jest to demonstrate Big O using two different methods for Fibonacci.

```<code>const fastFib = require('./../fastFib');
const slowFib = require('./../slowFib');

test('Fast way of getting Fibonacci of 44', () => {
  expect(fastFib(44)).toBe(701408733);
});

test('Slow way of getting Fibonacci of 44', () => {
  expect(slowFib(44)).toBe(701408733);
});
</code>```

I'm wondering if there is a way to specify the maximum length of a test? I saw you can pass a third variable for an async timeout but it doesn't seem to have any effect on normal functions:

```<code>test('Slow way of getting Fibonacci of 44', () => {
  expect(slowFib(44)).toBe(701408733);
}, 5000);
</code>```

Is there a way I can specify the maximum execution time for a function with Jest?

I will share slowFib.js for reference:

```<code>function fib(n) {
  return (n<=1) ? n : fib(n - 1) + fib(n - 2);
}

module.exports = fib;
</code>```

答:
In your test file you can set
jest.setTimeout(5000); which overrides the default timeout for a test case in jest

ContentLicense: CC BY-SA 4.0

Title: Put time limit on Jest test?

Body:
I'm running a set of tests with Jest to demonstrate Big O using two different methods for Fibonacci.

const fastFib = require('./../fastFib');
const slowFib = require('./../slowFib');

test('Fast way of getting Fibonacci of 44', () => {
  expect(fastFib(44)).toBe(701408733);
});

test('Slow way of getting Fibonacci of 44', () => {
  expect(slowFib(44)).toBe(701408733);
});

I'm wondering if there is a way to specify the maximum length of a test? I saw you can pass a third variable for an async timeout but it doesn't seem to have any effect on normal functions:

test('Slow way of getting Fibonacci of 44', () => {
  expect(slowFib(44)).toBe(701408733);
}, 5000);

Is there a way I can specify the maximum execution time for a function with Jest?

I will share slowFib.js for reference:

function fib(n) {
  return (n<=1) ? n : fib(n - 1) + fib(n - 2);
}

module.exports = fib;


问:
Why impala-jdbc throws exception when casting from BigDecimal to DECIMAL?
I'm writing to a Kudu table using impala-jdbc 2.6.4.1005.

I got this error when inserting a BigDecimal with value 7896163500 to DECIMAL(20,2).

```<code>[Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: Possible loss of precision for target table 'ST.RFQ_SPOT'.
Expression '7896163500' (type: BIGINT) would need to be cast to DECIMAL(20,2) for column 'req_amount'
</code>```

However, the value I'm inserting would fit also in a DECIMAL(12,2).
I cannot understand why I get this error from the driver.

If I define the column as DECIMAL(30,2) it works correctly.

In addition, if I try to manually make the cast, I get the result as expected:

```<code>select cast (cast(7896163500 as BIGINT) as DECIMAL(12,2))
1   7896163500.00
</code>```

答:
This is an expected event.

In Impala, by default - If you specify a value such as 7896163500 to go into a DECIMAL column, Impala checks if the column has enough precision to represent the largest value of that integer type, and raises an error if not.

Here 7896163500 is a BIGINT and BIGINT's maximum storage is 8 Bytes. Since we have 2 fractional digits here, you see the below error,

```<code>Expression '7896163500' (type: BIGINT) would need to be cast to DECIMAL(20,2) for column 'req_amount'
</code>```

Therefore, use an expression like (cast(7896163500 as BIGINT) as DECIMAL(12,2)) for DECIMAL columns as you have already done to insert the details. Hope this helps!

For more: https://www.cloudera.com/documentation/enterprise/5-6-x/topics/impala_decimal.html

Body:
I'm writing to a Kudu table using impala-jdbc 2.6.4.1005.

I got this error when inserting a BigDecimal with value 7896163500 to DECIMAL(20,2).

[Cloudera][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AnalysisException: Possible loss of precision for target table 'ST.RFQ_SPOT'.
Expression '7896163500' (type: BIGINT) would need to be cast to DECIMAL(20,2) for column 'req_amount'

However, the value I'm inserting would fit also in a DECIMAL(12,2).
I cannot understand why I get this error from the driver.

If I define the column as DECIMAL(30,2) it works correctly.

In addition, if I try to manually make the cast, I get the result as expected:

select cast (cast(7896163500 as BIGINT) as DECIMAL(12,2))
1   7896163500.00

Body:
This is an expected event.

In Impala, by default - If you specify a value such as 7896163500 to go into a DECIMAL column, Impala checks if the column has enough precision to represent the largest value of that integer type, and raises an error if not.

Here 7896163500 is a BIGINT and BIGINT's maximum storage is 8 Bytes. Since we have 2 fractional digits here, you see the below error,

Expression '7896163500' (type: BIGINT) would need to be cast to DECIMAL(20,2) for column 'req_amount'

Therefore, use an expression like (cast(7896163500 as BIGINT) as DECIMAL(12,2)) for DECIMAL columns as you have already done to insert the details. Hope this helps!

For more: https://www.cloudera.com/documentation/enterprise/5-6-x/topics/impala_decimal.html


问:
Binding two tableviews together but due to content row heights are different that causes index mismatch
I'm working on javafx application and I needed fixed column in tableview  that's why I have binded two tables together and get it working but one table have only one column which contains only indexes and another table  has multiple columns and many data.

If any row has only one line data then rows of both tables having same height but in different case row has multiple lines then indexes are mismatched. I'm sharing my test code:

```<code>import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.Set;
import javafx.application.Application;
import javafx.application.Platform;
import javafx.beans.property.IntegerProperty;
import javafx.beans.property.SimpleIntegerProperty;
import javafx.beans.property.SimpleStringProperty;
import javafx.beans.property.StringProperty;
import javafx.collections.FXCollections;
import javafx.collections.ObservableList;
import javafx.geometry.Orientation;
import javafx.scene.Node;
import javafx.scene.Scene;
import javafx.scene.control.ScrollBar;
import javafx.scene.control.TableColumn;
import javafx.scene.control.TableRow;
import javafx.scene.control.TableView;
import javafx.scene.layout.BorderPane;
import javafx.scene.layout.HBox;
import javafx.scene.layout.Priority;
import javafx.scene.layout.Region;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;

/**
 *
 * @author nitin
 */
public class NewClass extends Application {

@Override
public void start(Stage primaryStage) throws Exception {
    primaryStage.setTitle("Fix Column");
    Scene scene = new Scene(getView());
    primaryStage.setScene(scene);
    primaryStage.show();
}

private BorderPane getView() {
    BorderPane borderPane = new BorderPane();
    List<Student> students = Student.getList();

    TableView<Student> fixedColumnTable = new TableView<>();
    fixedColumnTable.setItems(FXCollections.observableArrayList(students));
    TableColumn<Student, Number> possionNumber = new TableColumn<>("#");
    possionNumber.setCellValueFactory(value -> value.getValue().getIdProperty());
    fixedColumnTable.getColumns().add(possionNumber);
    fixedColumnTable.setColumnResizePolicy(fixedColumnTable.CONSTRAINED_RESIZE_POLICY);
    fixedColumnTable.setPrefWidth(60);
    VBox vBox = new VBox();
    vBox.getChildren().add(fixedColumnTable);
    VBox.setVgrow(fixedColumnTable, Priority.ALWAYS);
    HBox.setHgrow(fixedColumnTable, Priority.ALWAYS);
    borderPane.setLeft(vBox);
    Region region = new Region();

    TableView<Student> table = new TableView<>();
    table.setItems(FXCollections.observableArrayList(students));

    table.setRowFactory(tv -> new TableRow<Student>() {
        @Override
        public void updateItem(Student item, boolean empty) {
            setRowHight(fixedColumnTable, table);
        }
    });

    TableColumn<Student, String> nameCol = new TableColumn<>("Name");
    nameCol.setCellValueFactory(value -> value.getValue().getNameProperty());

    TableColumn<Student, Number> ageCol = new TableColumn<>("Age");
    ageCol.setCellValueFactory(value -> value.getValue().getAgeProperty());

    TableColumn<Student, String> courseCol = new TableColumn<>("Course");
    courseCol.setCellValueFactory(value -> value.getValue().getCourseProperty());

    table.getColumns().addAll(nameCol, ageCol, courseCol);
    borderPane.setCenter(table);

    Platform.runLater(() -> {
        ScrollBar fixColtableVerticalBar = (ScrollBar) fixedColumnTable.lookup(".scroll-bar:vertical");
        fixColtableVerticalBar.setPrefWidth(0);
        fixColtableVerticalBar.setMaxWidth(0);
        fixColtableVerticalBar.setVisible(false);
        fixColtableVerticalBar.setOpacity(1);
        ScrollBar tableVerticalBar = (ScrollBar) table.lookup(".scroll-bar:vertical");
        tableVerticalBar.valueProperty().bindBidirectional(fixColtableVerticalBar.valueProperty());

        ObservableList<Node> list = vBox.getChildren();
        Set<Node> scrollBars = table.lookupAll(".scroll-bar");
        Optional<Node> horizontalScrollBar = scrollBars.stream()
                .filter(node
                        -> ((ScrollBar) node).getOrientation().equals(Orientation.HORIZONTAL))
                .findAny();
        horizontalScrollBar.ifPresent(node
                -> node.visibleProperty().addListener((observable, oldValue, newValue) -> {
                    if (newValue && !list.contains(region)) {
                        list.add(region);
                        Platform.runLater(() -> region.setMinHeight(((ScrollBar) node).getHeight()));
                    } else {
                        list.remove(region);
                    }
                })
        );

    });

    return borderPane;
}

private void setRowHight(TableView t1, TableView t2) {
    List<Node> leftTableRows = new ArrayList(t1.lookupAll(".table-row-cell"));
    List<Node> rightTableRows = new ArrayList(t2.lookupAll(".table-row-cell"));
    if (leftTableRows.size() == rightTableRows.size()) {
        for (int i = 0; i < leftTableRows.size(); i++) {
            TableRow leftTableRow = (TableRow) leftTableRows.get(i);
            TableRow leftRow = (TableRow) leftTableRows.get(i);
            TableRow rightRow = (TableRow) rightTableRows.get(i);
            leftRow.setPrefHeight(rightRow.getHeight());
        }
    }
}

public static void main(String[] args) {
    launch(args);
}
}

class Student {

private int id;
private String name;
private int age;
private String course;

public Student(int id, String name, int age, String course) {
    this.id = id;
    this.name = name;
    this.age = age;
    this.course = course;
}

public int getId() {
    return id;
}

public IntegerProperty getIdProperty() {
    return new SimpleIntegerProperty(id);
}

public String getName() {
    return name;
}

public StringProperty getNameProperty() {
    return name != null ? new SimpleStringProperty(name) : null;
}

public int getAge() {
    return age;
}

public IntegerProperty getAgeProperty() {
    return new SimpleIntegerProperty(age);
}

public String getCourse() {
    return course;
}

public StringProperty getCourseProperty() {
    return course != null ? new SimpleStringProperty(course) : null;
}

public static List<Student> getList() {
    List<Student> list = new ArrayList<>();
    list.add(new Student(0, "A", 18, "Java \n sfsf\n sdfsdf\ndsfsfsd : "));
    for (int i = 1; i <= 100; i++) {
        list.add(new Student(i, "A" + i, 18, "Java : " + i));
    }
    return list;
}
}
</code>```

答:
You need to get each row of each table. Then set the left height based on the right height.

```<code>private void setRowHight(TableView t1, TableView t2) {
    List<Node> leftTableRows = new ArrayList(t1.lookupAll(".table-row-cell"));
    List<Node> rightTableRows = new ArrayList(t2.lookupAll(".table-row-cell"));

    for (Node tableRow : rightTableRows) {
        TableRow<?> r = (TableRow<?>) tableRow;
        for (Node tableRowLeft : leftTableRows) {
            TableRow<?> l = (TableRow<?>) tableRowLeft;
                if (r.getIndex() == l.getIndex()) {
                    l.setPrefHeight(r.getHeight());
                    break;
                }
            }
    }
}
</code>```

Body:
I'm working on javafx application and I needed fixed column in tableview  that's why I have binded two tables together and get it working but one table have only one column which contains only indexes and another table  has multiple columns and many data.

If any row has only one line data then rows of both tables having same height but in different case row has multiple lines then indexes are mismatched. I'm sharing my test code:

import java.util.ArrayList;
import java.util.List;
import java.util.Optional;
import java.util.Set;
import javafx.application.Application;
import javafx.application.Platform;
import javafx.beans.property.IntegerProperty;
import javafx.beans.property.SimpleIntegerProperty;
import javafx.beans.property.SimpleStringProperty;
import javafx.beans.property.StringProperty;
import javafx.collections.FXCollections;
import javafx.collections.ObservableList;
import javafx.geometry.Orientation;
import javafx.scene.Node;
import javafx.scene.Scene;
import javafx.scene.control.ScrollBar;
import javafx.scene.control.TableColumn;
import javafx.scene.control.TableRow;
import javafx.scene.control.TableView;
import javafx.scene.layout.BorderPane;
import javafx.scene.layout.HBox;
import javafx.scene.layout.Priority;
import javafx.scene.layout.Region;
import javafx.scene.layout.VBox;
import javafx.stage.Stage;

/**
 *
 * @author nitin
 */
public class NewClass extends Application {

@Override
public void start(Stage primaryStage) throws Exception {
    primaryStage.setTitle("Fix Column");
    Scene scene = new Scene(getView());
    primaryStage.setScene(scene);
    primaryStage.show();
}

private BorderPane getView() {
    BorderPane borderPane = new BorderPane();
    List<Student> students = Student.getList();

    TableView<Student> fixedColumnTable = new TableView<>();
    fixedColumnTable.setItems(FXCollections.observableArrayList(students));
    TableColumn<Student, Number> possionNumber = new TableColumn<>("#");
    possionNumber.setCellValueFactory(value -> value.getValue().getIdProperty());
    fixedColumnTable.getColumns().add(possionNumber);
    fixedColumnTable.setColumnResizePolicy(fixedColumnTable.CONSTRAINED_RESIZE_POLICY);
    fixedColumnTable.setPrefWidth(60);
    VBox vBox = new VBox();
    vBox.getChildren().add(fixedColumnTable);
    VBox.setVgrow(fixedColumnTable, Priority.ALWAYS);
    HBox.setHgrow(fixedColumnTable, Priority.ALWAYS);
    borderPane.setLeft(vBox);
    Region region = new Region();

    TableView<Student> table = new TableView<>();
    table.setItems(FXCollections.observableArrayList(students));

    table.setRowFactory(tv -> new TableRow<Student>() {
        @Override
        public void updateItem(Student item, boolean empty) {
            setRowHight(fixedColumnTable, table);
        }
    });

    TableColumn<Student, String> nameCol = new TableColumn<>("Name");
    nameCol.setCellValueFactory(value -> value.getValue().getNameProperty());

    TableColumn<Student, Number> ageCol = new TableColumn<>("Age");
    ageCol.setCellValueFactory(value -> value.getValue().getAgeProperty());

    TableColumn<Student, String> courseCol = new TableColumn<>("Course");
    courseCol.setCellValueFactory(value -> value.getValue().getCourseProperty());

    table.getColumns().addAll(nameCol, ageCol, courseCol);
    borderPane.setCenter(table);

    Platform.runLater(() -> {
        ScrollBar fixColtableVerticalBar = (ScrollBar) fixedColumnTable.lookup(".scroll-bar:vertical");
        fixColtableVerticalBar.setPrefWidth(0);
        fixColtableVerticalBar.setMaxWidth(0);
        fixColtableVerticalBar.setVisible(false);
        fixColtableVerticalBar.setOpacity(1);
        ScrollBar tableVerticalBar = (ScrollBar) table.lookup(".scroll-bar:vertical");
        tableVerticalBar.valueProperty().bindBidirectional(fixColtableVerticalBar.valueProperty());

        ObservableList<Node> list = vBox.getChildren();
        Set<Node> scrollBars = table.lookupAll(".scroll-bar");
        Optional<Node> horizontalScrollBar = scrollBars.stream()
                .filter(node
                        -> ((ScrollBar) node).getOrientation().equals(Orientation.HORIZONTAL))
                .findAny();
        horizontalScrollBar.ifPresent(node
                -> node.visibleProperty().addListener((observable, oldValue, newValue) -> {
                    if (newValue && !list.contains(region)) {
                        list.add(region);
                        Platform.runLater(() -> region.setMinHeight(((ScrollBar) node).getHeight()));
                    } else {
                        list.remove(region);
                    }
                })
        );

    });

    return borderPane;
}

private void setRowHight(TableView t1, TableView t2) {
    List<Node> leftTableRows = new ArrayList(t1.lookupAll(".table-row-cell"));
    List<Node> rightTableRows = new ArrayList(t2.lookupAll(".table-row-cell"));
    if (leftTableRows.size() == rightTableRows.size()) {
        for (int i = 0; i < leftTableRows.size(); i++) {
            TableRow leftTableRow = (TableRow) leftTableRows.get(i);
            TableRow leftRow = (TableRow) leftTableRows.get(i);
            TableRow rightRow = (TableRow) rightTableRows.get(i);
            leftRow.setPrefHeight(rightRow.getHeight());
        }
    }
}

public static void main(String[] args) {
    launch(args);
}
}

class Student {

private int id;
private String name;
private int age;
private String course;

public Student(int id, String name, int age, String course) {
    this.id = id;
    this.name = name;
    this.age = age;
    this.course = course;
}

public int getId() {
    return id;
}

public IntegerProperty getIdProperty() {
    return new SimpleIntegerProperty(id);
}

public String getName() {
    return name;
}

public StringProperty getNameProperty() {
    return name != null ? new SimpleStringProperty(name) : null;
}

public int getAge() {
    return age;
}

public IntegerProperty getAgeProperty() {
    return new SimpleIntegerProperty(age);
}

public String getCourse() {
    return course;
}

public StringProperty getCourseProperty() {
    return course != null ? new SimpleStringProperty(course) : null;
}

public static List<Student> getList() {
    List<Student> list = new ArrayList<>();
    list.add(new Student(0, "A", 18, "Java \n sfsf\n sdfsdf\ndsfsfsd : "));
    for (int i = 1; i <= 100; i++) {
        list.add(new Student(i, "A" + i, 18, "Java : " + i));
    }
    return list;
}
}

Body:
You need to get each row of each table. Then set the left height based on the right height.

private void setRowHight(TableView t1, TableView t2) {
    List<Node> leftTableRows = new ArrayList(t1.lookupAll(".table-row-cell"));
    List<Node> rightTableRows = new ArrayList(t2.lookupAll(".table-row-cell"));

    for (Node tableRow : rightTableRows) {
        TableRow<?> r = (TableRow<?>) tableRow;
        for (Node tableRowLeft : leftTableRows) {
            TableRow<?> l = (TableRow<?>) tableRowLeft;
                if (r.getIndex() == l.getIndex()) {
                    l.setPrefHeight(r.getHeight());
                    break;
                }
            }
    }
}


问:
In WebStorm, is there a way to collapse all the elements at my current level
We have a common structure in our code where we have several related classes in the same file with several methods in each. What I would like to achieve is to fold all the methods in the class I'm in so that I could get a good overview of the "moving parts" in this class.

I know there is the Collapse Recursively method, but the problem with that is that it fold all the inside parts of the methods, so that when I want to dig in, I can't.

So my question is, is there a way (complex as it may be) to collapse all the elements at my current level (with a keyboard shortcut)?

答:
I found this question while attempting to find an answer myself. The two existing answers ignored parts of the original question. This question is very similar to this other question, but with the explicitly stated requirement that the code within the method blocks should not be folded.

The other question has a better answer:

Code -> Folding -> Expand all to level -> 1

The naming on this is a little weird because it says "Expand" when you're really trying to collapse all method bodies. But if the method bodies are already expanded, this will collapse them. The shortcut for this is ctrlshiftNumPad *, 1. If you find that shortcut combination cumbersome or don't have a num pad, you can create your own shortcut by going to Preferences -> Keymap.

This seems to me like the perfect solution. It won't expand or collapse any blocks at other levels. So if you manually collapse a block within a method body, this will leave it collapsed.

If that's not your preference, and you want it to expand all blocks within the method bodies while collapsing the method bodies themselves, you could create a macro that runs Code -> Folding -> Expand All followed by Code -> Folding -> Expand all to level -> 1, and then map your keyboard shortcut to that macro.

ContentLicense: CC BY-SA 3.0


问:
Cannot run 'scrapy crawl quotes'
Cannot get scrapy tutorial to work.

Am trying to learn scrapy but can't get even the tutorial to run. I have tried to run this in python 3.7 & 3.5.5 with the same results

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"

```<code>def start_requests(self):
    urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]
    for url in urls:
        yield scrapy.Request(url=url, callback=self.parse)

def parse(self, response):
    page = response.url.split("/")[-2]
    filename = 'quotes-%s.html' % page
    with open(filename, 'wb') as f:
        f.write(response.body)
    self.log('Saved file %s' % filename)
</code>```

This appears to run OK. At least it throws no errors.

When I run "scrapy crawl quotes" in Anaconda prompt window, I get this:

```<code>"hed) C:\Users\userOne\python script files\scrapy\tutorial>scrapy crawl
 quotes
 2019-01-23 18:34:27 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot:
 tutorial)
 2019-01-23 18:34:27 [scrapy.utils.log] INFO: Versions: lxml 4.2.3.0, libxml2
 2.9.5, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python
 3.5.5 | packaged by conda-forge | (default, Jul 24 2018, 01:52:17) [MSC
 v.1900 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.0.2p  14 Aug 2018),
 cryptography 2.3.1, Platform Windows-10-10.0.17134-SP0
 Traceback (most recent call last):
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\spiderloader.py", line 69, in load
     return self._spiders[spider_name]
 KeyError: 'quotes'

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "C:\Users\userOne\Anaconda3\envs\hed\Scripts\scrapy-script.py", line
 10, in <module>
     sys.exit(execute())
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\cmdline.py", line 150, in execute
     _run_print_help(parser, _run_command, cmd, args, opts)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\cmdline.py", line 90, in _run_print_help
     func(*a, **kw)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\cmdline.py", line 157, in _run_command
     cmd.run(args, opts)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\commands\crawl.py", line 57, in run
     self.crawler_process.crawl(spname, **opts.spargs)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\crawler.py", line 170, in crawl
     crawler = self.create_crawler(crawler_or_spidercls)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\crawler.py", line 198, in create_crawler
     return self._create_crawler(crawler_or_spidercls)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\crawler.py", line 202, in _create_crawler
     spidercls = self.spider_loader.load(spidercls)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\spiderloader.py", line 71, in load
     raise KeyError("Spider not found: {}".format(spider_name))
 KeyError: 'Spider not found: quotes'
</code>```

"

The output should be similar to this:

```<code>"016-12-16 21:24:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:24:05 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: None)
2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-1.html
2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-2.html
2016-12-16 21:24:05 [scrapy.core.engine] INFO: Closing spider (finished)"
</code>```

Thank in advance for any help you can give.

答:
I believe I found the answer. The tutorial doesn't mention one step that is only mentioned in your command line after you create the project via

```<code>scrapy startproject tutorial
</code>```

The output for that command, besides creating your tutorial project, is

```<code>You can start your first spider with:
cd tutorial
scrapy genspider example example.com
</code>```

For the tutorial to work, you need to enter

```<code>scrapy genspider quotes quotes.toscrape.com
</code>```

Title: Cannot run 'scrapy crawl quotes'

Body:
Cannot get scrapy tutorial to work.

Am trying to learn scrapy but can't get even the tutorial to run. I have tried to run this in python 3.7 & 3.5.5 with the same results

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"

def start_requests(self):
    urls = [
        'http://quotes.toscrape.com/page/1/',
        'http://quotes.toscrape.com/page/2/',
    ]
    for url in urls:
        yield scrapy.Request(url=url, callback=self.parse)

def parse(self, response):
    page = response.url.split("/")[-2]
    filename = 'quotes-%s.html' % page
    with open(filename, 'wb') as f:
        f.write(response.body)
    self.log('Saved file %s' % filename)

This appears to run OK. At least it throws no errors.

When I run "scrapy crawl quotes" in Anaconda prompt window, I get this:

"hed) C:\Users\userOne\python script files\scrapy\tutorial>scrapy crawl
 quotes
 2019-01-23 18:34:27 [scrapy.utils.log] INFO: Scrapy 1.5.1 started (bot:
 tutorial)
 2019-01-23 18:34:27 [scrapy.utils.log] INFO: Versions: lxml 4.2.3.0, libxml2
 2.9.5, cssselect 1.0.3, parsel 1.5.0, w3lib 1.19.0, Twisted 18.7.0, Python
 3.5.5 | packaged by conda-forge | (default, Jul 24 2018, 01:52:17) [MSC
 v.1900 64 bit (AMD64)], pyOpenSSL 18.0.0 (OpenSSL 1.0.2p  14 Aug 2018),
 cryptography 2.3.1, Platform Windows-10-10.0.17134-SP0
 Traceback (most recent call last):
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\spiderloader.py", line 69, in load
     return self._spiders[spider_name]
 KeyError: 'quotes'

 During handling of the above exception, another exception occurred:

 Traceback (most recent call last):
   File "C:\Users\userOne\Anaconda3\envs\hed\Scripts\scrapy-script.py", line
 10, in <module>
     sys.exit(execute())
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\cmdline.py", line 150, in execute
     _run_print_help(parser, _run_command, cmd, args, opts)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\cmdline.py", line 90, in _run_print_help
     func(*a, **kw)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\cmdline.py", line 157, in _run_command
     cmd.run(args, opts)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\commands\crawl.py", line 57, in run
     self.crawler_process.crawl(spname, **opts.spargs)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\crawler.py", line 170, in crawl
     crawler = self.create_crawler(crawler_or_spidercls)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\crawler.py", line 198, in create_crawler
     return self._create_crawler(crawler_or_spidercls)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\crawler.py", line 202, in _create_crawler
     spidercls = self.spider_loader.load(spidercls)
   File "C:\Users\userOne\Anaconda3\envs\hed\lib\site- packages\scrapy\spiderloader.py", line 71, in load
     raise KeyError("Spider not found: {}".format(spider_name))
 KeyError: 'Spider not found: quotes'

"

The output should be similar to this:

"016-12-16 21:24:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2016-12-16 21:24:05 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) <GET http://quotes.toscrape.com/robots.txt> (referer: None)
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/1/> (referer: None)
2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://quotes.toscrape.com/page/2/> (referer: None)
2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-1.html
2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-2.html
2016-12-16 21:24:05 [scrapy.core.engine] INFO: Closing spider (finished)"

Thank in advance for any help you can give.

Body:
I believe I found the answer. The tutorial doesn't mention one step that is only mentioned in your command line after you create the project via

scrapy startproject tutorial

The output for that command, besides creating your tutorial project, is

You can start your first spider with:
cd tutorial
scrapy genspider example example.com

For the tutorial to work, you need to enter

scrapy genspider quotes quotes.toscrape.com


问:
How can I ignore a parameter in a URL?
My app decodes URLs and I've noticed that when it hits an "&", the process stops and ignores the rest of the URL. It appears that URLComponents is breaking up the URL into pieces - my required parameter and the one that it believes to be a parameter.

```<code> hxxp://www.g.com/url?u=http://a.com/test&d=another
  - scheme : "hxxp"
  - host : "www.g.com"
  - path : "/url"
  ▿ queryItems : 2 elements
    ▿ 0 : u=http://a.com/test
      - name : "u"
      ▿ value : Optional<String>
        - some : "http://a.com/test"
    ▿ 1 : d=another
      - name : "d"
      ▿ value : Optional<String>
        - some : "another"
</code>```

What I need from above is the full URL after "u=".

Here is the initial call to the function that decodes the string:

```<code>if (fullDecode.contains("?u=")) {
            initialDecode = getQueryStringParameter(urlToAttempt: fullDecode, param: "u")!
</code>```

Here is the function that returns the decoded string:

```<code>func getQueryStringParameter(urlToAttempt: String, param: String) -> String? {
        guard let urlDecoded = URLComponents(string: urlToAttempt) else { return nil }
        print(urlDecoded)
        return urlDecoded.queryItems?.first(where: { $0.name == param })?.value
    }
</code>```

For this code, if the url was "http://www.g.com/url?u=http://a.com/test&d=another", it would return "http://a.com/test" - everything else is stripped away. I'd like it to return "http://a.com/test&d=another"

Is there a way that I'm able to do this with URLComponents, or do I need to do custom code to support this?

Update
If I change the url that's being passed to " hxxp://www.g.com/url?u=http://a.com/test%26d=another", URLComponents returns the full url back with the &d=another intact. I'm now going to try to percent encode the special character before sending to the function and see if that fixes the issue.

Second Update

Here is a heavily modified type of link I need to decode:

```<code>hxxps://ue.pt.com/v2/url?u=https-3A__oe.le.com_-3Fauthkey-3D-2521IB-2DCRV-2DqQ88-26cid-3DBACDF2ED353D-26id-3DBACDF2EFB61D353D-2521549-26parId-3Droot-26o-3DOneUp&d=DwMGaQ&c=WNpQK9bFmT89misLWAzsd66s44iGV-VujF_o4whjrfc&r=Ej_UhLznQMBqt3H3IYBQkjyx4xqdnS9mLiYA&m=HOBrLfxamFr4PYdACIR-A49th_oIe3MW69N7X-E&s=bXWSJ8gaSbKSlNuIf30S7Qsa6RcMKA-EOvP577XUyq0&e=
</code>```

These links are generated, which is why I have no control of how the link is built. I need to be able to decode this link to a legible URL. I have dictionaries for the special obfuscation like "-3A__" is "://" Where the above fails is where you see &d=DwM... It's not encoded, and that's where URLComponents is failing:

```<code>https://oe.le.com/?authkey=!IB-CRV-qQ88&cid=BACDF2ED353D&id=BACDF2EFB6353D!549&parId=root&o=OneUp
</code>```

Does this help?

答:
I wanted to post what I did to correct this, but all of the answers that were submitted helped lead me down this road. The code that sends to the function looks like this:

```<code>if (fullDecode.contains("?u=")) {
    fullDecode = fullDecode.replacingOccurrences(of: "&", with: "%26")
    initialDecode = getQueryStringParameter(urlToAttempt: fullDecode, param: "u")!
</code>```

If the uitextview fullDecode contains "?u=", then I know there is text in the view. The text should be a fully obfuscated URL, and if it contains an "&", I manually convert it to %26. This is sent to the URLComponents function BEFORE I do my other conversions using my custom dictionary.

```<code>   func getQueryStringParameter(urlToAttempt: String, param: String) -> String? {
        guard let urlDecoded = URLComponents(string: urlToAttempt) else { return nil }
        return urlDecoded.queryItems?.first(where: { $0.name == param })?.value
    }
</code>```

I was really overthinking this one.

Title: How can I ignore a parameter in a URL?

Body:
My app decodes URLs and I've noticed that when it hits an "&", the process stops and ignores the rest of the URL. It appears that URLComponents is breaking up the URL into pieces - my required parameter and the one that it believes to be a parameter.

 hxxp://www.g.com/url?u=http://a.com/test&d=another
  - scheme : "hxxp"
  - host : "www.g.com"
  - path : "/url"
  ▿ queryItems : 2 elements
    ▿ 0 : u=http://a.com/test
      - name : "u"
      ▿ value : Optional<String>
        - some : "http://a.com/test"
    ▿ 1 : d=another
      - name : "d"
      ▿ value : Optional<String>
        - some : "another"

What I need from above is the full URL after "u=".

Here is the initial call to the function that decodes the string:

if (fullDecode.contains("?u=")) {
            initialDecode = getQueryStringParameter(urlToAttempt: fullDecode, param: "u")!

Here is the function that returns the decoded string:

func getQueryStringParameter(urlToAttempt: String, param: String) -> String? {
        guard let urlDecoded = URLComponents(string: urlToAttempt) else { return nil }
        print(urlDecoded)
        return urlDecoded.queryItems?.first(where: { $0.name == param })?.value
    }

For this code, if the url was "http://www.g.com/url?u=http://a.com/test&d=another", it would return "http://a.com/test" - everything else is stripped away. I'd like it to return "http://a.com/test&d=another"

Is there a way that I'm able to do this with URLComponents, or do I need to do custom code to support this?

Update
If I change the url that's being passed to " hxxp://www.g.com/url?u=http://a.com/test%26d=another", URLComponents returns the full url back with the &d=another intact. I'm now going to try to percent encode the special character before sending to the function and see if that fixes the issue.

Second Update

Here is a heavily modified type of link I need to decode:

hxxps://ue.pt.com/v2/url?u=https-3A__oe.le.com_-3Fauthkey-3D-2521IB-2DCRV-2DqQ88-26cid-3DBACDF2ED353D-26id-3DBACDF2EFB61D353D-2521549-26parId-3Droot-26o-3DOneUp&d=DwMGaQ&c=WNpQK9bFmT89misLWAzsd66s44iGV-VujF_o4whjrfc&r=Ej_UhLznQMBqt3H3IYBQkjyx4xqdnS9mLiYA&m=HOBrLfxamFr4PYdACIR-A49th_oIe3MW69N7X-E&s=bXWSJ8gaSbKSlNuIf30S7Qsa6RcMKA-EOvP577XUyq0&e=

These links are generated, which is why I have no control of how the link is built. I need to be able to decode this link to a legible URL. I have dictionaries for the special obfuscation like "-3A__" is "://" Where the above fails is where you see &d=DwM... It's not encoded, and that's where URLComponents is failing:

https://oe.le.com/?authkey=!IB-CRV-qQ88&cid=BACDF2ED353D&id=BACDF2EFB6353D!549&parId=root&o=OneUp

Does this help?

Body:
I wanted to post what I did to correct this, but all of the answers that were submitted helped lead me down this road. The code that sends to the function looks like this:

if (fullDecode.contains("?u=")) {
    fullDecode = fullDecode.replacingOccurrences(of: "&", with: "%26")
    initialDecode = getQueryStringParameter(urlToAttempt: fullDecode, param: "u")!

If the uitextview fullDecode contains "?u=", then I know there is text in the view. The text should be a fully obfuscated URL, and if it contains an "&", I manually convert it to %26. This is sent to the URLComponents function BEFORE I do my other conversions using my custom dictionary.

   func getQueryStringParameter(urlToAttempt: String, param: String) -> String? {
        guard let urlDecoded = URLComponents(string: urlToAttempt) else { return nil }
        return urlDecoded.queryItems?.first(where: { $0.name == param })?.value
    }

I was really overthinking this one.


问:
Deploying React + Express App to Kubernetes. How to structure it with docker?
So I've been struggling with how to deploy a dockerized application. The app consists of a react frontend, and an express API.
My docker-compose.yml for the development environment looks like the following:

```<code>version: '3'
services:
  # Express Container
  backend:
    build: ./backend
    expose:
      - ${BACKEND_PORT}
    env_file:
      - ./.env
    environment:
      - PORT=${BACKEND_PORT}
    ports:
      - ${BACKEND_PORT}:${BACKEND_PORT}
    volumes:
      - ./backend:/backend
    command: npm run devstart
    links:
      - mongo
  # React Container
  frontend:
    build: './frontend'
    expose:
      - ${REACT_APP_PORT}
    env_file:
      - ./.env
    environment:
      - REACT_APP_BACKEND_PORT=${BACKEND_PORT}
    ports:
      - ${REACT_APP_PORT}:${REACT_APP_PORT}
    volumes:
      - ./frontend/src:/frontend/src
      - ./frontend/public:/frontend/public
    links:
      - backend
    command: npm start
  mongo:
    image: mongo
    ports:
      - "27017:27017"
</code>```

But I've been struggling on how to structure it for production.

I've seen that there is basically 3 options:

Deploy the frontend and backend separately to different servers. In this case, the react frontend would be on some web hosting service, and the express backend would be hosted on kubernetes

Have the express application serve out the react application

Have the applications separate but use NGINX to proxy API requests to the express app

I was thinking I would go with option 3, because it would keep development and production environments quite similar. (Please tell me if this is bad structure, this application is expected to receive a lot of traffic.)

Should I maybe forget docker-compose and create a multistage dockerfile that uses multistage builds to copy over frontend and backend code? That way I can deploy a single Docker container?

My folder structure looks like the following:

```<code>app/
  .env
  docker-compose.yml
  docker-compose.prod.yml
  .gitignore
  frontend/
    Dockerfile
    ... react stuff
  backend
    Dockerfile
    .. express stuff
</code>```

Am I going about this all wrong? How have you deployed your applications with docker-compose to production (preferably on kubernetes).

I can find tons of stuff about how to get this stuff running in development, but I'm lost when it comes to direction for deploying this type of stack.

答: I like option 1 more than 3 since it's keeping the frontend and the backend separate.  A huge advantage is that you can host the frontend on something like AWS S3 with a CloudFront CDN.  So all the static content is distributed to edge servers around the world.  This will get the heavy items (images, large js libraries, css etc) to your end users very quickly and make your application function much faster.</p>

Body:
So I've been struggling with how to deploy a dockerized application. The app consists of a react frontend, and an express API.
My docker-compose.yml for the development environment looks like the following:

version: '3'
services:
  # Express Container
  backend:
    build: ./backend
    expose:
      - ${BACKEND_PORT}
    env_file:
      - ./.env
    environment:
      - PORT=${BACKEND_PORT}
    ports:
      - ${BACKEND_PORT}:${BACKEND_PORT}
    volumes:
      - ./backend:/backend
    command: npm run devstart
    links:
      - mongo
  # React Container
  frontend:
    build: './frontend'
    expose:
      - ${REACT_APP_PORT}
    env_file:
      - ./.env
    environment:
      - REACT_APP_BACKEND_PORT=${BACKEND_PORT}
    ports:
      - ${REACT_APP_PORT}:${REACT_APP_PORT}
    volumes:
      - ./frontend/src:/frontend/src
      - ./frontend/public:/frontend/public
    links:
      - backend
    command: npm start
  mongo:
    image: mongo
    ports:
      - "27017:27017"

But I've been struggling on how to structure it for production.

I've seen that there is basically 3 options:

Deploy the frontend and backend separately to different servers. In this case, the react frontend would be on some web hosting service, and the express backend would be hosted on kubernetes

Have the express application serve out the react application

Have the applications separate but use NGINX to proxy API requests to the express app

I was thinking I would go with option 3, because it would keep development and production environments quite similar. (Please tell me if this is bad structure, this application is expected to receive a lot of traffic.)

Should I maybe forget docker-compose and create a multistage dockerfile that uses multistage builds to copy over frontend and backend code? That way I can deploy a single Docker container?

My folder structure looks like the following:

app/
  .env
  docker-compose.yml
  docker-compose.prod.yml
  .gitignore
  frontend/
    Dockerfile
    ... react stuff
  backend
    Dockerfile
    .. express stuff

Am I going about this all wrong? How have you deployed your applications with docker-compose to production (preferably on kubernetes).

I can find tons of stuff about how to get this stuff running in development, but I'm lost when it comes to direction for deploying this type of stack.


问:
Invert of a shift operation - determine shift count used to generate value
I have code that generates a value from an initial value of 1 by using a left shift. At a different point in the code, I know the value generated and need to calculate the shift count used to generate the value.

In other words I need to determine the shift count used to generate the result of a shift operation by performing an operation on the value generated.

Is it possible to invert a shift operation of a base two variable without using a dedicated function? I am looking for the value of a shift count variable y when the shifted value, x, and result of the shift operation, z, are known.

Example:

```<code>x << y = z
1 << 6 = 64

z = 64 and x is 1 but what is y???  // how to calculate y, the shift count?
</code>```

There are many solutions including log(64)/log(2), but I would have to use math.h. I am looking for some sort of bitwise operation that is fast and doesn't require a function.

EDIT: Thank you for your answers! My question is answered, there is no way to this with a simple operation, my compiler does not have CTZ. BR

答:
There is no single operation or function in standard C that computes the number of trailing zero bits in an integer.  You can perform such a computation by checking one bit at a time via a loop, as other answers have suggested.  If you need to do it a lot, though, then you might want a more efficient alternative, such as this:

```<code>int trailing_zero_bits(uint64_t x) {
    uint64_t bits = ~(uint64_t)0;
    int rval = 0;

    for (int shift = 32; shift; shift >>= 1) {
        bits >>= shift;
        if (!(x & bits)) {
            rval += shift;
            x >>= shift;
        }
    }

    return rval + !x;  // The !x adds 1 if x is zero at this point
}
</code>```

That loop will iterate exactly 6 times, vs. up to 63 times for the 64-bit bit-at-a-time alternative.

Of course, there may be system- or environment-specific alternatives that are even more efficient.

Body:
I have code that generates a value from an initial value of 1 by using a left shift. At a different point in the code, I know the value generated and need to calculate the shift count used to generate the value.

In other words I need to determine the shift count used to generate the result of a shift operation by performing an operation on the value generated.

Is it possible to invert a shift operation of a base two variable without using a dedicated function? I am looking for the value of a shift count variable y when the shifted value, x, and result of the shift operation, z, are known.

Example:

x << y = z
1 << 6 = 64

z = 64 and x is 1 but what is y???  // how to calculate y, the shift count?

There are many solutions including log(64)/log(2), but I would have to use math.h. I am looking for some sort of bitwise operation that is fast and doesn't require a function.

EDIT: Thank you for your answers! My question is answered, there is no way to this with a simple operation, my compiler does not have CTZ. BR

Body:
There is no single operation or function in standard C that computes the number of trailing zero bits in an integer.  You can perform such a computation by checking one bit at a time via a loop, as other answers have suggested.  If you need to do it a lot, though, then you might want a more efficient alternative, such as this:

int trailing_zero_bits(uint64_t x) {
    uint64_t bits = ~(uint64_t)0;
    int rval = 0;

    for (int shift = 32; shift; shift >>= 1) {
        bits >>= shift;
        if (!(x & bits)) {
            rval += shift;
            x >>= shift;
        }
    }

    return rval + !x;  // The !x adds 1 if x is zero at this point
}

That loop will iterate exactly 6 times, vs. up to 63 times for the 64-bit bit-at-a-time alternative.

Of course, there may be system- or environment-specific alternatives that are even more efficient.


问:
What are the ways of Key-Value extraction from unstructured text?
I'm trying to figure out what are the ways (and which of them the best one) of extraction of Values for predefined Keys in the unstructured text?

Input:

The doctor prescribed me a drug called favipiravir.

His name is Yury.

Ilya has already told me about that.

The weather is cold today.

I am taking a medicine called nazivin.

Key list: ['drug', 'name', 'weather']

Output:

['drug=favipiravir', 'drug=nazivin', 'name=Yury', 'weather=cold']

So, as you can see, in the 3d sentence there is no explicit key 'name' and therefore no value extracted (I think there is the difference with NER). At the same time, 'drug' and 'medicine' are synonyms and we should treat 'medicine' as 'drug' key and extract the value also.

And the next question, what if the key set will be mutable?
Should I use as a base regexp approach because of predefined Keys or there is a way to implement it with supervised learning/NN? (but in this case how to deal with mutable keys?)

答:
You can use a parser to tag words. Your problem is similar to Named Entity Recognition (NER). A lot of libraries, like NLTK in Python, have POS taggers available. You can try those. They are generally trained to identify names, locations, etc. Depending on the type of words you need, you may need to train the parser. So you'll need some labeled data also.
 Check out this link:
https://nlp.stanford.edu/software/CRF-NER.html


问:
How does the value event work on updates with Firebase listeners?
Do event listeners guarantee that all data ever written to a path will be delivered to the client eventually?
For instance if I have a game client that pushes moves to the same path one after the other will the listening client receive all updates?
What would happen in this situation: client A pushes move 1 to game1/user1/move_data which client B is listening on; client A then immediately pushes another move updating the value at game1/user1/move_data.
Will the listening client be guaranteed to receive all moves pushed?

Currently I have a system that creates a new path per move and then I am calling single listeners on each move as each client reaches that move in it's state. It doesn't seem efficient as if the client A receives the most recent move that client B has made then client A begins listening on a path that doesn't exist yet.

The below quotes are from this link: https://firebase.google.com/docs/database/admin/retrieve-data

"The value event is used to read a static snapshot of the contents at a given database path, as they existed at the time of the read event. It is triggered once with the initial data and again every time the data changes. The event callback is passed a snapshot containing all data at that location, including child data. In the code example above, value returned all of the blog posts in your app. Everytime a new blog post is added, the callback function will return all of the posts."

The part about as they existed at the time of the read event causes me to think that if a listener is on a path then the client will receive all values ever on that path eventually.

There is also this line from the guarantees section which I am struggling to decipher:

"Value events are always triggered last and are guaranteed to contain updates from any other events which occurred before that snapshot was taken."

I am working with a language that does not have a Google based sdk and am asking this question, so I can further assess Firebases' suitability for my uses.

答:
Firebase Realtime Database performs state synchronization. If a client is listening to data in a location, it will receive the state of that data. If there are changes in the data, it will receive the latest state of that data.

...if I have a game client that pushes moves to the same path one after the other will the listening client receive all updates?

If there are multiple updates before the Firebase server has a chance to send the state to a listener, it may skip some intermediate values. So there is no guarantee that your client will see every state change, there is just a guarantee that it will eventually see the latest state.

If you want to ensure that all clients (can) see all state changes, you should store the state changes themselves in the database.


问:
Deployment Pools - Access Denied - username needs Manage permissions to perform the action
I'm attempting under "Organization Settings" to either create or rename a Deployment Pool, but get the message...

Access denied. USERNAME needs the Manage permissions to perform the action. For more information, contact the Azure DevOps Server administrator.

I've tried to find a "Manage" permission, but can't find it anywhere. Message is very vague.

Any help greatly received - Thanks

答:
Even though you are one of PCA member, if you don’t have permission or correct role of deployment pool, you will also could not manage the deployment pools.

Contact your org owner or anyone who is administrator of deployment pool. Ask them add you and assign suitable role to you(for roles you can refer previous pic)

Assigned steps:  Organization setting—>Deployment pools —>Security—>Add, and then input your account and select the role.


问:
OleDB selecting multiple CSVs with first row as field names in C#
So my code looks like this currently, which is all fine and good

```<code>String q = "SELECT * FROM "+"test.csv";
        try
        {
            OleDbConnection cn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0; Data Source=C:\\;Extended Properties=\"Text;HDR=No;FMT=Delimited\"");
            OleDbDataAdapter da = new OleDbDataAdapter();
            DataSet ds = new DataSet();
            OleDbCommand cd = new OleDbCommand(q, cn);
            cn.Open();
            da.SelectCommand = cd;
            ds.Clear();
            da.Fill(ds, "CSV");
            dataGridView1.DataSource = ds.Tables[0];
            cn.Close();
            }
        catch (Exception ex)
        {
            MessageBox.Show("There was an issue: " + ex.Message);
        }
</code>```

However, I would like to take the first row in my CSV and use that as the field names, so I can do something like

```<code>"select a.ID, a.SOMETHING_ELSE from test.csv a"
where the CSV looks something like
ID,SOMETHING_ELSE,DONT_WANT_THIS
1,blah,I don't want to see this
</code>```

Yet, from what I see, about all examples on the web show nothing other than select *, and I would like my datagrid to show the headers as the field names selected, instead of F1, F2....

Is this implemented anywhere?

答:
Change your connection string to include HDR=Yes (instead of HDR=No):

```<code>OleDbConnection cn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0; Data Source=C:\\;Extended Properties=\"Text;HDR=Yes;FMT=Delimited\"");
</code>```

For OleDB, the HDR property in the connection string indicates whether or not the first row contains the column names.

Body:
So my code looks like this currently, which is all fine and good

String q = "SELECT * FROM "+"test.csv";
        try
        {
            OleDbConnection cn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0; Data Source=C:\\;Extended Properties=\"Text;HDR=No;FMT=Delimited\"");
            OleDbDataAdapter da = new OleDbDataAdapter();
            DataSet ds = new DataSet();
            OleDbCommand cd = new OleDbCommand(q, cn);
            cn.Open();
            da.SelectCommand = cd;
            ds.Clear();
            da.Fill(ds, "CSV");
            dataGridView1.DataSource = ds.Tables[0];
            cn.Close();
            }
        catch (Exception ex)
        {
            MessageBox.Show("There was an issue: " + ex.Message);
        }

However, I would like to take the first row in my CSV and use that as the field names, so I can do something like

"select a.ID, a.SOMETHING_ELSE from test.csv a"
where the CSV looks something like
ID,SOMETHING_ELSE,DONT_WANT_THIS
1,blah,I don't want to see this

Yet, from what I see, about all examples on the web show nothing other than select *, and I would like my datagrid to show the headers as the field names selected, instead of F1, F2....

Is this implemented anywhere?

Body:
Change your connection string to include HDR=Yes (instead of HDR=No):

OleDbConnection cn = new OleDbConnection("Provider=Microsoft.Jet.OLEDB.4.0; Data Source=C:\\;Extended Properties=\"Text;HDR=Yes;FMT=Delimited\"");

For OleDB, the HDR property in the connection string indicates whether or not the first row contains the column names.


问:
Use case containing the presenter or returning data?
Considering the Clean Architecture definition, and especially the little flow diagram describing relationships between a controller, a use case interactor, and a presenter, I'm not sure if I correctly understand what the "Use Case Output Port" should be.

Clean architecture, like ports/adapters architecture, distinguishes between primary ports (methods) and secondary ports (interfaces to be implemented by adapters). Following the communication flow, I expect the "Use Case Input Port" to be a primary port (thus, just a method), and the "Use Case Output Port" an interface to be implemented, perhaps a constructor argument taking the actual adapter, so that the interactor can use it.

To make a code example, this could be the controller code:

```<code>Presenter presenter = new Presenter();
Repository repository = new Repository();
UseCase useCase = new UseCase(presenter, repository);
useCase->doSomething();
</code>```

The presenter interface:

```<code>// Use Case Output Port
interface Presenter
{
    public void present(Data data);
}
</code>```

Finally, the interactor itself:

```<code>class UseCase
{
    private Repository repository;
    private Presenter presenter;

    public UseCase(Repository repository, Presenter presenter)
    {
        this.repository = repository;
        this.presenter = presenter;
    }

    // Use Case Input Port
    public void doSomething()
    {
        Data data = this.repository.getData();
        this.presenter.present(data);
    }
}
</code>```

This interpretation seems to be confirmed by the aforementioned diagram itself, where the relation between the controller and the input port is represented by a solid arrow with a "sharp" head (UML for "association", meaning "has a", where the controller "has a" use case), while the relation between the presenter and the output port is represented by a solid arrow with a "white" head (UML for "inheritance", which is not the one for "implementation", but probably that's the meaning anyway).

However, my problem with this approach is that the use case must take care of the presentation itself. Now, I see that the purpose of the Presenter interface is to be abstract enough to represent several different types of presenters (GUI, Web, CLI, etc.), and that it really just means "output", which is something a use case might very well have, but still I'm not totally confident with it.

Now, looking around the Web for applications of the clean architecture, I seem to only find people interpreting the output port as a method returning some DTO. This would be something like:

```<code>Repository repository = new Repository();
UseCase useCase = new UseCase(repository);
Data data = useCase.getData();
Presenter presenter = new Presenter();
presenter.present(data);

// I'm omitting the changes to the classes, which are fairly obvious
</code>```

This is attractive because we're moving the responsibility of "calling" the presentation out of the use case, so the use case doesn't concern itself with knowing what to do with the data anymore, rather just with providing the data. Also, in this case we're still not breaking the dependency rule, because the use case still doesn't know anything about the outer layer.

However, the use case doesn't control the moment when the actual presentation is performed anymore (which may be useful, for example to do additional stuff at that moment, like logging, or to abort it altogether if necessary). Also, notice that we lost the Use Case Input Port, because now the controller is only using the getData() method (which is our new output port). Furthermore, it looks to me that we're breaking the "tell, don't ask" principle here, because we're asking the interactor for some data to do something with it, rather than telling it to do the actual thing in the first place.

So, is any of these two alternatives the "correct" interpretation of the Use Case Output Port according to the Clean Architecture? Are they both viable?

In this answer to another question, Robert Martin describes exactly a use case where the interactor calls the presenter upon a read request. No mention of MVC, MVVC, etc. can be found, so I guess that the Clean Architecture doesn't play very well with MVC in general?

Clicking on the map causes either the placePinController to be invoked. It gathers the location of the click, and any other contextual data, constructs a placePinRequest data structure and passes it to the PlacePinInteractor which checks the location of the pin, validates it if necessary, create a Place entity to record the pin, constructs a EditPlaceReponse object and passes it to the EditPlacePresenter which brings up the place editor screen.

A possible explanation would be that the application logic that traditionally would go into the controller, here is moved to the interactor, because we don't want any application logic to leak outside the application layer. Thus, here the model is not calling the presenter, because the interactor is not the model, but rather the actual implementation of the controller. The model is just the data structure being passed around. This seems to be confirmed by:

The software in this layer is a set of adapters that convert data from the format most convenient for the use cases and entities, to the format most convenient for some external agency such as the Database or the Web.

From the original article, talking about Interface Adapters. Since the controller must just be a thin adapter converting one data format to another, it must not contain any application logic, which is thus moved to the interactor.

答:
In a discussion related to your question, Uncle Bob explains the purpose of the presenter in his Clean Architecture:

Given this code sample:

namespace Some\Controller;

class UserController extends Controller {
    public function registerAction() {
        // Build the Request object
        $request = new RegisterRequest();
        $request->name = $this->getRequest()->get('username');
        $request->pass = $this->getRequest()->get('password');

        // Build the Interactor
        $usecase = new RegisterUser();

        // Execute the Interactors method and retrieve the response
        $response = $usecase->register($request);

        // Pass the result to the view
        $this->render(
            '/user/registration/template.html.twig',
            array('id' =>  $response->getId()
        );
    }
}
```

<p>Uncle Bob said this:</p>

<blockquote>
  <p>"<strong>The purpose of the presenter is to decouple the use cases from the format of the UI.</strong>  In your example, the $response variable is created by the interactor, but is used by the view.  This couples the interactor to the view.  For example, let's say that one of the fields in the $response object is a date.  That field would be a binary date object that could be rendered in many different date formats.  The wants a very specific date format, perhaps DD/MM/YYYY.  Whose responsibility is it to create the format?  If the interactor creates that format, then it knows too much about the View.  But if the view takes the binary date object then it knows too much about the interactor.
  <br /><br />
  "The presenter's job is to take the data from the response object and format it for the View.  <strong>Neither the view nor the interactor know about each other's formats.</strong>"
  <br /><br />
  --- Uncle Bob</p>
</blockquote>

<p>Given that answer of Uncle Bob, I think <strong>it doesn't matter</strong> that much whether we do <strong>option #1</strong> (let interactor use presenter)...</p>

<pre class="lang-cs prettyprint-override"><code>class UseCase
{
    private Presenter presenter;
    private Repository repository;

    public UseCase(Repository repository, Presenter presenter)
    {
        this.presenter = presenter;
        this.repository = repository;
    }

    public void Execute(Request request)
    {
        ...
        Response response = new Response() {...}
        this.presenter.Show(response);
    }
}
</code>```

... or we do option #2 (let interactor return response, create a presenter inside the controller, then pass response to presenter)...

class Controller
{
    public void ExecuteUseCase(Data data)
    {
        Request request = ...
        UseCase useCase = new UseCase(repository);
        Response response = useCase.Execute(request);
        Presenter presenter = new Presenter();
        presenter.Show(response);
    }
}
```

<hr>

<p><strong>Personally, I prefer option #1</strong> because I want to be able control <strong>inside the <code>interactor</code></strong> <em>when</em> to show data and error messages, like this example below:</p>

<pre class="lang-cs prettyprint-override"><code>class UseCase
{
    private Presenter presenter;
    private Repository repository;

    public UseCase(Repository repository, Presenter presenter)
    {
        this.presenter = presenter;
        this.repository = repository;
    }

    public void Execute(Request request)
    {
        if (<invalid request>)
        {
            this.presenter.ShowError("...");
            return;
        }

        if (<there is another error>)
        {
            this.presenter.ShowError("another error...");
            return;
        }

        ...
        Response response = new Response() {...}
        this.presenter.Show(response);
    }
}
</code>```

... I want to be able to do these if/else that are related to presentation inside the interactor and not outside the interactor.

If on the other hand we do option #2, we would have to store the error message(s) in the response object, return that response object from the interactor to the controller, and make the controller parse the response object...

class UseCase
{
    public Response Execute(Request request)
    {
        Response response = new Response();
        if (<invalid request>)
        {
            response.AddError("...");
        }

        if (<there is another error>)
        {
            response.AddError("another error...");
        }

        if (response.HasNoErrors)
        {
            response.Whatever = ...
        }

        ...
        return response;
    }
}
```

<pre class="lang-cs prettyprint-override"><code>class Controller
{
    private UseCase useCase;

    public Controller(UseCase useCase)
    {
        this.useCase = useCase;
    }

    public void ExecuteUseCase(Data data)
    {
        Request request = new Request()
        {
            Whatever = data.whatever,
        };
        Response response = useCase.Execute(request);
        Presenter presenter = new Presenter();
        if (response.ErrorMessages.Count > 0)
        {
            if (response.ErrorMessages.Contains(<invalid request>))
            {
                presenter.ShowError("...");
            }
            else if (response.ErrorMessages.Contains("another error")
            {
                presenter.ShowError("another error...");
            }
        }
        else
        {
            presenter.Show(response);
        }
    }
}
</code>```

I don't like parsing response data for errors inside the controller because if we do that we are doing redundant work --- if we change something in the interactor, we also have to change something in the controller.

Also, if we later decide to reuse our interactor to present data using the console, for example, we have to remember to copy-paste all those if/else in the controller of our console app.

```<code>// in the controller for our console app
if (response.ErrorMessages.Count > 0)
{
    if (response.ErrorMessages.Contains(<invalid request>))
    {
        presenterForConsole.ShowError("...");
    }
    else if (response.ErrorMessages.Contains("another error")
    {
        presenterForConsole.ShowError("another error...");
    }
}
else
{
    presenterForConsole.Present(response);
}
</code>```

If we use option #1 we will have this if/else only in one place: the interactor.

If you are using ASP.NET MVC (or other similar MVC frameworks), option #2 is the easier way to go.

But we can still do option #1 in that kind of environment. Here is an example of doing option #1 in ASP.NET MVC:

(Notice that we need to have public IActionResult Result in the presenter of our ASP.NET MVC app)

class UseCase
{
    private Repository repository;

    public UseCase(Repository repository)
    {
        this.repository = repository;
    }

    public void Execute(Request request, Presenter presenter)
    {
        if (<invalid request>)
        {
            this.presenter.ShowError("...");
            return;
        }

        if (<there is another error>)
        {
            this.presenter.ShowError("another error...");
            return;
        }

        ...
        Response response = new Response() {...}
        this.presenter.Show(response);
    }
}
```

<pre class="lang-cs prettyprint-override"><code>// controller for ASP.NET app

class AspNetController
{
    private UseCase useCase;

    public AspNetController(UseCase useCase)
    {
        this.useCase = useCase;
    }

    [HttpPost("dosomething")]
    public void ExecuteUseCase(Data data)
    {
        Request request = new Request()
        {
            Whatever = data.whatever,
        };
        var presenter = new AspNetPresenter();
        useCase.Execute(request, presenter);
        return presenter.Result;
    }
}
</code>```

// presenter for ASP.NET app

public class AspNetPresenter
{
    public IActionResult Result { get; private set; }

    public AspNetPresenter(...)
    {
    }

    public async void Show(Response response)
    {
        Result = new OkObjectResult(new { });
    }

    public void ShowError(string errorMessage)
    {
        Result = new BadRequestObjectResult(errorMessage);
    }
}
```

<p><strong>(Notice that we need to have <code>public IActionResult Result</code> in the presenter of our ASP.NET MVC app)</strong></p>

<p>If we decide to create another app for the console, we can reuse the <code>UseCase</code> above and create just the <code>Controller</code> and <code>Presenter</code> for the console:</p>

<pre class="lang-cs prettyprint-override"><code>// controller for console app

class ConsoleController
{
    public void ExecuteUseCase(Data data)
    {
        Request request = new Request()
        {
            Whatever = data.whatever,
        };
        var presenter = new ConsolePresenter();
        useCase.Execute(request, presenter);
    }
}
</code>```

// presenter for console app

public class ConsolePresenter
{
    public ConsolePresenter(...)
    {
    }

    public async void Show(Response response)
    {
        // write response to console
    }

    public void ShowError(string errorMessage)
    {
        Console.WriteLine("Error: " + errorMessage);
    }
}
```

(Notice that we DO NOT HAVE public IActionResult Result in the presenter of our console app)

Body:
Considering the Clean Architecture definition, and especially the little flow diagram describing relationships between a controller, a use case interactor, and a presenter, I'm not sure if I correctly understand what the "Use Case Output Port" should be.

Clean architecture, like ports/adapters architecture, distinguishes between primary ports (methods) and secondary ports (interfaces to be implemented by adapters). Following the communication flow, I expect the "Use Case Input Port" to be a primary port (thus, just a method), and the "Use Case Output Port" an interface to be implemented, perhaps a constructor argument taking the actual adapter, so that the interactor can use it.

To make a code example, this could be the controller code:

Presenter presenter = new Presenter();
Repository repository = new Repository();
UseCase useCase = new UseCase(presenter, repository);
useCase->doSomething();

The presenter interface:

// Use Case Output Port
interface Presenter
{
    public void present(Data data);
}

Finally, the interactor itself:

class UseCase
{
    private Repository repository;
    private Presenter presenter;

    public UseCase(Repository repository, Presenter presenter)
    {
        this.repository = repository;
        this.presenter = presenter;
    }

    // Use Case Input Port
    public void doSomething()
    {
        Data data = this.repository.getData();
        this.presenter.present(data);
    }
}

This interpretation seems to be confirmed by the aforementioned diagram itself, where the relation between the controller and the input port is represented by a solid arrow with a "sharp" head (UML for "association", meaning "has a", where the controller "has a" use case), while the relation between the presenter and the output port is represented by a solid arrow with a "white" head (UML for "inheritance", which is not the one for "implementation", but probably that's the meaning anyway).

However, my problem with this approach is that the use case must take care of the presentation itself. Now, I see that the purpose of the Presenter interface is to be abstract enough to represent several different types of presenters (GUI, Web, CLI, etc.), and that it really just means "output", which is something a use case might very well have, but still I'm not totally confident with it.

Now, looking around the Web for applications of the clean architecture, I seem to only find people interpreting the output port as a method returning some DTO. This would be something like:

Repository repository = new Repository();
UseCase useCase = new UseCase(repository);
Data data = useCase.getData();
Presenter presenter = new Presenter();
presenter.present(data);

// I'm omitting the changes to the classes, which are fairly obvious

This is attractive because we're moving the responsibility of "calling" the presentation out of the use case, so the use case doesn't concern itself with knowing what to do with the data anymore, rather just with providing the data. Also, in this case we're still not breaking the dependency rule, because the use case still doesn't know anything about the outer layer.

However, the use case doesn't control the moment when the actual presentation is performed anymore (which may be useful, for example to do additional stuff at that moment, like logging, or to abort it altogether if necessary). Also, notice that we lost the Use Case Input Port, because now the controller is only using the getData() method (which is our new output port). Furthermore, it looks to me that we're breaking the "tell, don't ask" principle here, because we're asking the interactor for some data to do something with it, rather than telling it to do the actual thing in the first place.

So, is any of these two alternatives the "correct" interpretation of the Use Case Output Port according to the Clean Architecture? Are they both viable?

In this answer to another question, Robert Martin describes exactly a use case where the interactor calls the presenter upon a read request. No mention of MVC, MVVC, etc. can be found, so I guess that the Clean Architecture doesn't play very well with MVC in general?

Clicking on the map causes either the placePinController to be invoked. It gathers the location of the click, and any other contextual data, constructs a placePinRequest data structure and passes it to the PlacePinInteractor which checks the location of the pin, validates it if necessary, create a Place entity to record the pin, constructs a EditPlaceReponse object and passes it to the EditPlacePresenter which brings up the place editor screen.

A possible explanation would be that the application logic that traditionally would go into the controller, here is moved to the interactor, because we don't want any application logic to leak outside the application layer. Thus, here the model is not calling the presenter, because the interactor is not the model, but rather the actual implementation of the controller. The model is just the data structure being passed around. This seems to be confirmed by:

The software in this layer is a set of adapters that convert data from the format most convenient for the use cases and entities, to the format most convenient for some external agency such as the Database or the Web.

From the original article, talking about Interface Adapters. Since the controller must just be a thin adapter converting one data format to another, it must not contain any application logic, which is thus moved to the interactor.

Body:
In a discussion related to your question, Uncle Bob explains the purpose of the presenter in his Clean Architecture:

Given this code sample:

namespace Some\Controller;

class UserController extends Controller {
    public function registerAction() {
        // Build the Request object
        $request = new RegisterRequest();
        $request->name = $this->getRequest()->get('username');
        $request->pass = $this->getRequest()->get('password');

        // Build the Interactor
        $usecase = new RegisterUser();

        // Execute the Interactors method and retrieve the response
        $response = $usecase->register($request);

        // Pass the result to the view
        $this->render(
            '/user/registration/template.html.twig',
            array('id' =>  $response->getId()
        );
    }
}

Uncle Bob said this:

"The purpose of the presenter is to decouple the use cases from the format of the UI.  In your example, the $response variable is created by the interactor, but is used by the view.  This couples the interactor to the view.  For example, let's say that one of the fields in the $response object is a date.  That field would be a binary date object that could be rendered in many different date formats.  The wants a very specific date format, perhaps DD/MM/YYYY.  Whose responsibility is it to create the format?  If the interactor creates that format, then it knows too much about the View.  But if the view takes the binary date object then it knows too much about the interactor.

  "The presenter's job is to take the data from the response object and format it for the View.  Neither the view nor the interactor know about each other's formats."

  --- Uncle Bob

Given that answer of Uncle Bob, I think it doesn't matter that much whether we do option #1 (let interactor use presenter)...

class UseCase
{
    private Presenter presenter;
    private Repository repository;

    public UseCase(Repository repository, Presenter presenter)
    {
        this.presenter = presenter;
        this.repository = repository;
    }

    public void Execute(Request request)
    {
        ...
        Response response = new Response() {...}
        this.presenter.Show(response);
    }
}

... or we do option #2 (let interactor return response, create a presenter inside the controller, then pass response to presenter)...

class Controller
{
    public void ExecuteUseCase(Data data)
    {
        Request request = ...
        UseCase useCase = new UseCase(repository);
        Response response = useCase.Execute(request);
        Presenter presenter = new Presenter();
        presenter.Show(response);
    }
}

Personally, I prefer option #1 because I want to be able control inside the interactor when to show data and error messages, like this example below:

class UseCase
{
    private Presenter presenter;
    private Repository repository;

    public UseCase(Repository repository, Presenter presenter)
    {
        this.presenter = presenter;
        this.repository = repository;
    }

    public void Execute(Request request)
    {
        if (<invalid request>)
        {
            this.presenter.ShowError("...");
            return;
        }

        if (<there is another error>)
        {
            this.presenter.ShowError("another error...");
            return;
        }

        ...
        Response response = new Response() {...}
        this.presenter.Show(response);
    }
}

... I want to be able to do these if/else that are related to presentation inside the interactor and not outside the interactor.

If on the other hand we do option #2, we would have to store the error message(s) in the response object, return that response object from the interactor to the controller, and make the controller parse the response object...

class UseCase
{
    public Response Execute(Request request)
    {
        Response response = new Response();
        if (<invalid request>)
        {
            response.AddError("...");
        }

        if (<there is another error>)
        {
            response.AddError("another error...");
        }

        if (response.HasNoErrors)
        {
            response.Whatever = ...
        }

        ...
        return response;
    }
}

class Controller
{
    private UseCase useCase;

    public Controller(UseCase useCase)
    {
        this.useCase = useCase;
    }

    public void ExecuteUseCase(Data data)
    {
        Request request = new Request()
        {
            Whatever = data.whatever,
        };
        Response response = useCase.Execute(request);
        Presenter presenter = new Presenter();
        if (response.ErrorMessages.Count > 0)
        {
            if (response.ErrorMessages.Contains(<invalid request>))
            {
                presenter.ShowError("...");
            }
            else if (response.ErrorMessages.Contains("another error")
            {
                presenter.ShowError("another error...");
            }
        }
        else
        {
            presenter.Show(response);
        }
    }
}

I don't like parsing response data for errors inside the controller because if we do that we are doing redundant work --- if we change something in the interactor, we also have to change something in the controller.

Also, if we later decide to reuse our interactor to present data using the console, for example, we have to remember to copy-paste all those if/else in the controller of our console app.

// in the controller for our console app
if (response.ErrorMessages.Count > 0)
{
    if (response.ErrorMessages.Contains(<invalid request>))
    {
        presenterForConsole.ShowError("...");
    }
    else if (response.ErrorMessages.Contains("another error")
    {
        presenterForConsole.ShowError("another error...");
    }
}
else
{
    presenterForConsole.Present(response);
}

If we use option #1 we will have this if/else only in one place: the interactor.

If you are using ASP.NET MVC (or other similar MVC frameworks), option #2 is the easier way to go.

But we can still do option #1 in that kind of environment. Here is an example of doing option #1 in ASP.NET MVC:

(Notice that we need to have public IActionResult Result in the presenter of our ASP.NET MVC app)

class UseCase
{
    private Repository repository;

    public UseCase(Repository repository)
    {
        this.repository = repository;
    }

    public void Execute(Request request, Presenter presenter)
    {
        if (<invalid request>)
        {
            this.presenter.ShowError("...");
            return;
        }

        if (<there is another error>)
        {
            this.presenter.ShowError("another error...");
            return;
        }

        ...
        Response response = new Response() {...}
        this.presenter.Show(response);
    }
}

// controller for ASP.NET app

class AspNetController
{
    private UseCase useCase;

    public AspNetController(UseCase useCase)
    {
        this.useCase = useCase;
    }

    [HttpPost("dosomething")]
    public void ExecuteUseCase(Data data)
    {
        Request request = new Request()
        {
            Whatever = data.whatever,
        };
        var presenter = new AspNetPresenter();
        useCase.Execute(request, presenter);
        return presenter.Result;
    }
}

// presenter for ASP.NET app

public class AspNetPresenter
{
    public IActionResult Result { get; private set; }

    public AspNetPresenter(...)
    {
    }

    public async void Show(Response response)
    {
        Result = new OkObjectResult(new { });
    }

    public void ShowError(string errorMessage)
    {
        Result = new BadRequestObjectResult(errorMessage);
    }
}

(Notice that we need to have public IActionResult Result in the presenter of our ASP.NET MVC app)

If we decide to create another app for the console, we can reuse the UseCase above and create just the Controller and Presenter for the console:

// controller for console app

class ConsoleController
{
    public void ExecuteUseCase(Data data)
    {
        Request request = new Request()
        {
            Whatever = data.whatever,
        };
        var presenter = new ConsolePresenter();
        useCase.Execute(request, presenter);
    }
}

// presenter for console app

public class ConsolePresenter
{
    public ConsolePresenter(...)
    {
    }

    public async void Show(Response response)
    {
        // write response to console
    }

    public void ShowError(string errorMessage)
    {
        Console.WriteLine("Error: " + errorMessage);
    }
}

(Notice that we DO NOT HAVE public IActionResult Result in the presenter of our console app)


问:
What is the best way to get the result through Java8 function?
I need to filter elements and then sort based on certain column. Post that I would need to find the unique entries based on combination of columns. Since it is file processing, pipe(|) is used as delimiter to denote the column value.

```<code>String s1= "12|Thor|Asgaurd|1000000|Avenger|Active"
String s2= "234|Iron man|New York|9999999|Avenger|Active"
String s3= "420|Loki|Asgaurd|||Inactive"
String s4= "12|Thor|Asgaurd Bank|1000000|Avenger HQ|Active"
</code>```

Data first needs to be filtered based on the Active/Inactive status. Then it needs to be sorted based on 4th column. Lastly, the uniqueness needs to be maintained by combining column 1,2,3.

Expected Output =

```<code>"234|Iron man|New York|9999999|Avenger|Active"
"12|Thor|Asgaurd|1000000|Avenger|Active"
</code>```

答:
First of all you should create an Object which represents your String data. Something like this:

```<code>public class MyObject {
    private int id;
    private String name;
    private String location;
    private Integer value;
    private String category;
    private String state;

    public MyObject(String entry) {
        String[] parts = entry.split("\\|");
        if (parts.length != 6) {
            throw new IllegalArgumentException("entry has not 6 parts");
        }
        id = Integer.parseInt(parts[0]);
        name = parts[1];
        location = parts[2];
        try {
            value = Integer.parseInt(parts[3]);
        } catch (NumberFormatException ignored) {
        }
        category = parts[4];
        state = parts[5];
    }

    // getters

    @Override
    public String toString() {
        return String.join("|", String.valueOf(id), name, location, String.valueOf(value), category, state);
    }
}
</code>```

With this you can create a Stream of objects from your Strings and to the filter, sort and distinct operations afterwards:

```<code>Collection<MyObject> result = Stream.of(s1, s2, s3, s4)
        .map(MyObject::new)
        .filter(o -> "Active".equals(o.getState()))
        .sorted(Comparator.comparing(MyObject::getValue).reversed())
        .collect(Collectors.toMap(o -> Arrays.asList(o.getId(), o.getName()),
                Function.identity(), (o1, o2) -> o1, LinkedHashMap::new))
        .values();

result.forEach(System.out::println);
</code>```

After the map operation you filter the values by state and sort them by column 4 (value in my case). At the end you collect all the values in a map for the distinct operation. Add all values you need distinction for to the Arrays.asList(). As values the map takes all the original values (Function.identity()). For duplicates we keep the first value ((o1, o2) -> o1) and we are using a LinkedHashMap to keep the order of the items. At the end wee use only the values of the map.

If you need a List instead of a Collection use new ArrayList(result).

The result will be this:

```<code>234|Iron man|New York|9999999|Avenger|Active
12|Thor|Asgaurd|1000000|Avenger|Active
</code>```

Body:
I need to filter elements and then sort based on certain column. Post that I would need to find the unique entries based on combination of columns. Since it is file processing, pipe(|) is used as delimiter to denote the column value.

String s1= "12|Thor|Asgaurd|1000000|Avenger|Active"
String s2= "234|Iron man|New York|9999999|Avenger|Active"
String s3= "420|Loki|Asgaurd|||Inactive"
String s4= "12|Thor|Asgaurd Bank|1000000|Avenger HQ|Active"

Data first needs to be filtered based on the Active/Inactive status. Then it needs to be sorted based on 4th column. Lastly, the uniqueness needs to be maintained by combining column 1,2,3.

Expected Output =

"234|Iron man|New York|9999999|Avenger|Active"
"12|Thor|Asgaurd|1000000|Avenger|Active"

Body:
First of all you should create an Object which represents your String data. Something like this:

public class MyObject {
    private int id;
    private String name;
    private String location;
    private Integer value;
    private String category;
    private String state;

    public MyObject(String entry) {
        String[] parts = entry.split("\\|");
        if (parts.length != 6) {
            throw new IllegalArgumentException("entry has not 6 parts");
        }
        id = Integer.parseInt(parts[0]);
        name = parts[1];
        location = parts[2];
        try {
            value = Integer.parseInt(parts[3]);
        } catch (NumberFormatException ignored) {
        }
        category = parts[4];
        state = parts[5];
    }

    // getters

    @Override
    public String toString() {
        return String.join("|", String.valueOf(id), name, location, String.valueOf(value), category, state);
    }
}

With this you can create a Stream of objects from your Strings and to the filter, sort and distinct operations afterwards:

Collection<MyObject> result = Stream.of(s1, s2, s3, s4)
        .map(MyObject::new)
        .filter(o -> "Active".equals(o.getState()))
        .sorted(Comparator.comparing(MyObject::getValue).reversed())
        .collect(Collectors.toMap(o -> Arrays.asList(o.getId(), o.getName()),
                Function.identity(), (o1, o2) -> o1, LinkedHashMap::new))
        .values();

result.forEach(System.out::println);

After the map operation you filter the values by state and sort them by column 4 (value in my case). At the end you collect all the values in a map for the distinct operation. Add all values you need distinction for to the Arrays.asList(). As values the map takes all the original values (Function.identity()). For duplicates we keep the first value ((o1, o2) -> o1) and we are using a LinkedHashMap to keep the order of the items. At the end wee use only the values of the map.

If you need a List instead of a Collection use new ArrayList(result).

The result will be this:

234|Iron man|New York|9999999|Avenger|Active
12|Thor|Asgaurd|1000000|Avenger|Active


问:
For loop iterates less times than I expected in Python
I would expect the following loop to iterate six times, instead it iterates three with python3. I don't understand this behavior.
I understand that the list changes as I delete elements but I don't get how that affects the for loop condition.
Why is the loop iterating less than six times?</p>

```<code>a = [1, 2, 3, 4, 5, 6]
for elem in a:
        del a[0]
        print(a)
</code>```

答:
The list iterator in CPython works by iterating over the positions of the list. You can think of it working like this:

```<code>def list_iter(items: list):
    index = 0
    while True:
        yield items[index]
        index += 1
</code>```

In other words, iteration provides the item at 0, then 1, then 2, and so on. There is no pre-fetching of items - an item is looked up from the list when needed.

As you delete the first item on every step, the list is shortened by 1 on each step. Since you start with a list of 6 items, on the third iteration it is down to 3 items - meaning the fourth iteration fails to look up an item. Thus, your iteration finishes after three steps.

You can see this when printing also the current element in each loop. To visualise the effect, use enumerate to get the index of the iteration. Notice that it advances by one index, but the values are also shifted for a total offset of two:

```<code>>>> a = [1, 2, 3, 4, 5, 6]
... for idx, elem in enumerate(a):
...     print(elem, 'from', a)
...     print('      ', '   '*idx, '^')
...     del a[0]
...
1 from [1, 2, 3, 4, 5, 6]
        ^
3 from [2, 3, 4, 5, 6]
           ^
5 from [3, 4, 5, 6]
              ^
</code>```

It is generally not well-defined to modify a container while iterating over it. You should iterate over a copy instead:

```<code>a = [1, 2, 3, 4, 5, 6]
for elem in a.copy():
    del a[0]
    print(a)
</code>```

Body:
I would expect the following loop to iterate six times, instead it iterates three with python3. I don't understand this behavior.
I understand that the list changes as I delete elements but I don't get how that affects the for loop condition.
Why is the loop iterating less than six times?

a = [1, 2, 3, 4, 5, 6]
for elem in a:
        del a[0]
        print(a)

Body:
The list iterator in CPython works by iterating over the positions of the list. You can think of it working like this:

def list_iter(items: list):
    index = 0
    while True:
        yield items[index]
        index += 1

In other words, iteration provides the item at 0, then 1, then 2, and so on. There is no pre-fetching of items - an item is looked up from the list when needed.

As you delete the first item on every step, the list is shortened by 1 on each step. Since you start with a list of 6 items, on the third iteration it is down to 3 items - meaning the fourth iteration fails to look up an item. Thus, your iteration finishes after three steps.

You can see this when printing also the current element in each loop. To visualise the effect, use enumerate to get the index of the iteration. Notice that it advances by one index, but the values are also shifted for a total offset of two:

>>> a = [1, 2, 3, 4, 5, 6]
... for idx, elem in enumerate(a):
...     print(elem, 'from', a)
...     print('      ', '   '*idx, '^')
...     del a[0]
...
1 from [1, 2, 3, 4, 5, 6]
        ^
3 from [2, 3, 4, 5, 6]
           ^
5 from [3, 4, 5, 6]
              ^

It is generally not well-defined to modify a container while iterating over it. You should iterate over a copy instead:

a = [1, 2, 3, 4, 5, 6]
for elem in a.copy():
    del a[0]
    print(a)