问:
Can we use global symbols in Typescript Declaration files?
I'm trying to implement something like this but I'm not sure it's possible.  I think Typescript only allows unique symbols, not global ones.  Is this correct?

Is there a better way to achieve using global symbols?

```<code>// sample.d.ts
const mySymbol = Symbol.for('internal.symbol')

interface Sample{
    [mySymbol]: string
    a: number
    b: number
}

// sample.js
class SampleClass implements Sample {
    [mySymbol]: string
    a: number
    b: number

    constructor(a: number, b: number){
        this.a = a;
        this.b = b;
        this[mySymbol] = `${a}-${b}`
    }
}

let mySample = new SampleClass(1, 2)

</code>```

Is there a way to accomplish this?  mySymbol can (and ideally will) be a global symbol that will be used by other objects as well so it can be defined separately if that can be accomplished.

答:
Here is how I was able to accomplish this.

```<code>// misc.ts
export const mySymbol = Symbol.for('internal.symbol')

// sample.d.ts
import {mySymbol} from './misc'

export as namespace Sample
export = Sample
interface Sample{
    [mySymbol]: string
    a: number
    b: number
}

// sample.js
class SampleClass implements Sample {
    [mySymbol]: string
    a: number
    b: number

    constructor(a: number, b: number){
        this.a = a;
        this.b = b;
        this[mySymbol] = `${a}-${b}`
    }
}

let mySample = new SampleClass(1, 2)
</code>```

Once mySymbol is imported into the declaration file it turns into a module.  Hence it needs to be specifically exported with the export = Sample and export as namespace Sample.  See sample module.d.ts.

Body:
I'm trying to implement something like this but I'm not sure it's possible.  I think Typescript only allows unique symbols, not global ones.  Is this correct?

Is there a better way to achieve using global symbols?

// sample.d.ts
const mySymbol = Symbol.for('internal.symbol')

interface Sample{
    [mySymbol]: string
    a: number
    b: number
}

// sample.js
class SampleClass implements Sample {
    [mySymbol]: string
    a: number
    b: number

    constructor(a: number, b: number){
        this.a = a;
        this.b = b;
        this[mySymbol] = `${a}-${b}`
    }
}

let mySample = new SampleClass(1, 2)

Is there a way to accomplish this?  mySymbol can (and ideally will) be a global symbol that will be used by other objects as well so it can be defined separately if that can be accomplished.

Body:
Here is how I was able to accomplish this.

// misc.ts
export const mySymbol = Symbol.for('internal.symbol')

// sample.d.ts
import {mySymbol} from './misc'

export as namespace Sample
export = Sample
interface Sample{
    [mySymbol]: string
    a: number
    b: number
}

// sample.js
class SampleClass implements Sample {
    [mySymbol]: string
    a: number
    b: number

    constructor(a: number, b: number){
        this.a = a;
        this.b = b;
        this[mySymbol] = `${a}-${b}`
    }
}

let mySample = new SampleClass(1, 2)

Once mySymbol is imported into the declaration file it turns into a module.  Hence it needs to be specifically exported with the export = Sample and export as namespace Sample.  See sample module.d.ts.


来源: stackexchange

问:
How to optimize a SQL query that aggregates data for individual days over a series of dates?
I'm trying to optimize the below SQL query that returns the count of all products with a specific status on any given day in a series of dates. On any given day, the latest status of the product should be the one given in the query (In the below query snippet it is '2') and there should not be any further status update(s) in that product before that date. The query returns the correct results but it is taking a longer time to run (~ 12 - 15 seconds) if the number of products in a specific store is large and queried for 30 days.

The query takes longer time to run if the number of days queried is more, i.e, it takes only 3 seconds if queried for 7 days, but I have to query for 30 days only.

```<code>select
statisticDate as date,
(
select
    count(*)
from
    product as p
join product_status_history as psh1 on
    p.id = psh1.product_id
    and psh1.id = (
    select
        min(id)
    from
        product_status_history
    where
        product_id = p.id
        and productstatus_id = 2
    group by
        product_id)
join product_status_history as psh2 on
    p.id = psh2.product_id
    and psh2.id = (
    select
        max(id)
    from
        product_status_history
    where
        product_id = p.id
    group by
        product_id)
where
    p.store_code = 'ABCD123'
    and ((psh2.productstatus_id = 2
    and cast(psh2.created_at as date) <= statisticDate)
    or (psh2.productstatus_id <> 2
    and cast(psh1.created_at as date) <= statisticDate
    and cast(psh2.created_at as date) > statisticDate))) as counter
from
    generate_series(current_date - 30, current_date + 1, '1 day') as statisticDate
order by
    statisticDate desc;
</code>```

Structure of the two tables is as follows

And the query returns results like this

```<code>Date       - Counter
2019-05-29 - 60
2019-05-28 - 60
2019-05-27 - 111
2019-05-26 - 123
2019-05-25 - 148
2019-05-24 - 234
2019-05-23 - 344
2019-05-22 - 434
2019-05-21 - 339
2019-05-20 - 256
2019-05-19 - 306
2019-05-18 - 392
2019-05-17 - 361
2019-05-16 - 480
2019-05-15 - 406
2019-05-14 - 203
2019-05-13 - 314
2019-05-12 - 396
2019-05-11 - 368
2019-05-10 - 484
2019-05-09 - 420
2019-05-08 - 234
2019-05-07 - 341
2019-05-06 - 204
2019-05-05 - 245
2019-05-04 - 306
2019-05-03 - 408
2019-05-02 - 342
2019-05-01 - 290
2019-04-30 - 272
2019-04-29 - 202
2019-04-28 - 241
</code>```

答:
"On any given day, the latest status of the product should be
  the one given in the query (In the below query snippet it is '2') and
  there should not be any further status update(s) in that product
  before that date."

This means for date X you only want to count products which have a single history record with created_at < given_date and productstatus_id = 2

Your query however is doing other checks.
If I only take into account your description you can use following query:

The count(psh.id)=1 guarantees there is only one history before the date to verify.
The max(psh.product_status_id)=2 verifies of this single history record has the correct status.

```<code>select statisticDate as date,
 (select count(*)
    from product as p
   where p.store_code = 'ABCD123'
     and exists (select max(product_status_id), count(psh.id)
                   from product_status_history psh
                  where psh.product_id = p.id
                    and psh.created_at < cast((statisticDate + interval '1 day') as text):: timestamp
                  having max(psh.product_status_id) = 2 and count(psh.id) = 1)
                )
from generate_series(current_date - 30, current_date + 1, '1 day') as statisticDate
order by statisticDate desc;
</code>```

I also did casting on the statisticdate instead of the psh.created_at because so a possible index on created_at could be used.

ContentLicense: CC BY-SA 4.0

Body:
I'm trying to optimize the below SQL query that returns the count of all products with a specific status on any given day in a series of dates. On any given day, the latest status of the product should be the one given in the query (In the below query snippet it is '2') and there should not be any further status update(s) in that product before that date. The query returns the correct results but it is taking a longer time to run (~ 12 - 15 seconds) if the number of products in a specific store is large and queried for 30 days.

The query takes longer time to run if the number of days queried is more, i.e, it takes only 3 seconds if queried for 7 days, but I have to query for 30 days only.

select
statisticDate as date,
(
select
    count(*)
from
    product as p
join product_status_history as psh1 on
    p.id = psh1.product_id
    and psh1.id = (
    select
        min(id)
    from
        product_status_history
    where
        product_id = p.id
        and productstatus_id = 2
    group by
        product_id)
join product_status_history as psh2 on
    p.id = psh2.product_id
    and psh2.id = (
    select
        max(id)
    from
        product_status_history
    where
        product_id = p.id
    group by
        product_id)
where
    p.store_code = 'ABCD123'
    and ((psh2.productstatus_id = 2
    and cast(psh2.created_at as date) <= statisticDate)
    or (psh2.productstatus_id <> 2
    and cast(psh1.created_at as date) <= statisticDate
    and cast(psh2.created_at as date) > statisticDate))) as counter
from
    generate_series(current_date - 30, current_date + 1, '1 day') as statisticDate
order by
    statisticDate desc;

Structure of the two tables is as follows

And the query returns results like this

Date       - Counter
2019-05-29 - 60
2019-05-28 - 60
2019-05-27 - 111
2019-05-26 - 123
2019-05-25 - 148
2019-05-24 - 234
2019-05-23 - 344
2019-05-22 - 434
2019-05-21 - 339
2019-05-20 - 256
2019-05-19 - 306
2019-05-18 - 392
2019-05-17 - 361
2019-05-16 - 480
2019-05-15 - 406
2019-05-14 - 203
2019-05-13 - 314
2019-05-12 - 396
2019-05-11 - 368
2019-05-10 - 484
2019-05-09 - 420
2019-05-08 - 234
2019-05-07 - 341
2019-05-06 - 204
2019-05-05 - 245
2019-05-04 - 306
2019-05-03 - 408
2019-05-02 - 342
2019-05-01 - 290
2019-04-30 - 272
2019-04-29 - 202
2019-04-28 - 241

Body:
"On any given day, the latest status of the product should be
  the one given in the query (In the below query snippet it is '2') and
  there should not be any further status update(s) in that product
  before that date."

This means for date X you only want to count products which have a single history record with created_at < given_date and productstatus_id = 2

Your query however is doing other checks.
If I only take into account your description you can use following query:

The count(psh.id)=1 guarantees there is only one history before the date to verify.
The max(psh.product_status_id)=2 verifies of this single history record has the correct status.

select statisticDate as date,
 (select count(*)
    from product as p
   where p.store_code = 'ABCD123'
     and exists (select max(product_status_id), count(psh.id)
                   from product_status_history psh
                  where psh.product_id = p.id
                    and psh.created_at < cast((statisticDate + interval '1 day') as text):: timestamp
                  having max(psh.product_status_id) = 2 and count(psh.id) = 1)
                )
from generate_series(current_date - 30, current_date + 1, '1 day') as statisticDate
order by statisticDate desc;

I also did casting on the statisticdate instead of the psh.created_at because so a possible index on created_at could be used.


问:
I do see in QM log:The call to initialize the User ID 'msgw' failed withReason 2035.If an MQCSP block was used, the User ID in the MQCSP block was
I do see below log in queue manager error log :
AMQ9557: Queue Manager User ID initialization failed for 'msgw'
EXPLANATION:
The call to initialize the User ID 'msgw' failed with CompCode 2 and Reason
2035. If an MQCSP block was used, the User ID in the MQCSP block was ''.
ACTION:
Correct the error and try again.

Did the same setup on QA environment, but there was no such error in the log.
Please advise.

I have installed MQ version 8 on rhel7 :
3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 15 17:36:42 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
when the application is trying to connect they are getting MQRC 2538.
When I check my logs I saw the above mentioned error.
All permissions are present on queue manager and queue level.

Application should be able to connect to the queue manager without any errors in queue manager log

答:
The error code usually means that the queue manager has been created to use the Object Authority Manager (OAM).  By default the queue manager is created this way and will attempt to authenticate each user against the credentials on the node where the MQ server is running.

We have other authentications methods in place before users can execute our applications, so we don't want any connecting user authenticated against the OS credentials where the MQ server is running.  Therefore we export this  environment variable:

export MQSNOAUT=yes

just prior to issuing the 'crtmqm' command to create the queue manager.

You will see these entries in you qm.ini file if the queue manager has been created with OAM enabled:
.

```<code>Service:
    Name=AuthorizationService
    EntryPoints=14
ServiceComponent:
    Service=AuthorizationService
    Name=MQSeries.UNIX.auth.service
    Module=amqzfu
    ComponentDataSize=0
</code>```

If these entries aren't there, then the queue manager was created with OAM disabled.

Body:
The error code usually means that the queue manager has been created to use the Object Authority Manager (OAM).  By default the queue manager is created this way and will attempt to authenticate each user against the credentials on the node where the MQ server is running.

We have other authentications methods in place before users can execute our applications, so we don't want any connecting user authenticated against the OS credentials where the MQ server is running.  Therefore we export this  environment variable:

export MQSNOAUT=yes

just prior to issuing the 'crtmqm' command to create the queue manager.

You will see these entries in you qm.ini file if the queue manager has been created with OAM enabled:
.

Service:
    Name=AuthorizationService
    EntryPoints=14
ServiceComponent:
    Service=AuthorizationService
    Name=MQSeries.UNIX.auth.service
    Module=amqzfu
    ComponentDataSize=0

If these entries aren't there, then the queue manager was created with OAM disabled.


问:
How to convert stacked columns with duplicate indices into multiple unique columns with pandas?
I'm working with a cryptocurrency time-series data-set which has all of the different currencies vertically stacked. It has 3 columns for the date, currency and price. There date ranges are also different for each currency.

i.e.

```<code>>>> df
        Currency  Date    Price
0       0x        2017-08-16  0.111725
1       0x        2017-08-17  0.211486
2       0x        2017-08-18  0.283789
3       0x        2017-08-19  0.511434
4       0x        2017-08-20  0.429522
...     ...       ...         ...
657311  zurcoin   2018-02-04  0.003254
657312  zurcoin   2018-02-05  0.002774
657313  zurcoin   2018-02-06  0.001986
657314  zurcoin   2018-02-09  0.002684
657315  zurcoin   2018-02-10  0.002325
</code>```

I need to instead have a column for each currency's price and the date as the index with only unique dates. There will be plenty of null values which I intend to replace with 0's.

i.e

```<code>date          0x_price  10mtoken_price  1337coin_price  ...
2017-08-16    1         4               (NaN)->0        ...
2017-08-17    2         5               (NaN)->0        ...
2017-08-18    3         6               7               ...
...           ...       ...             ...             ...
</code>```

I've tried to iterate over the dataframe with a groupby as shown:

```<code>df2 = pd.DataFrame()
df2["date"] = df["Date"].unique()
df2.set_index("date", inplace=True)

for currency, group in df.groupby("Currency"):
df2.loc[df2.index.isin(group.Date), f"{currency}_price"] = group["Price"]
</code>```

This returned the desired column names and shape but the dataframe was filled with NaN's.

i.e.

```<code>date          0x_price  10mtoken_price  1337coin_price  ...
2017-08-16    NaN       NaN             NaN             ...
2017-08-17    NaN       NaN             NaN             ...
2017-08-18    NaN       NaN             NaN             ...
...           ...       ...             ...             ...
</code>```

I also tried to achieve the same thing with df.join() as shown:

```<code>df2 = pd.DataFrame()
df2["date"] = df["Date"].unique()
df2.set_index("date", inplace=True)

for currency, group in df.groupby("Currency"):
df2 = df2.join(group.set_index("Date")[["Price"]].rename(columns={"Price": f"{currency}_price"}))
</code>```

This didn't get to finish executing before freezing up my computer. Perhaps it's inefficient and I'm working with around 650,000 entries?

I haven't been able to find the same type of problem here and I haven't been able to figure out a solution after checking the documentation. I've probably missed something but hopefully I've described the problem sufficiently. Thanks in advance.

Body:
I'm working with a cryptocurrency time-series data-set which has all of the different currencies vertically stacked. It has 3 columns for the date, currency and price. There date ranges are also different for each currency.

i.e.

>>> df
        Currency  Date    Price
0       0x        2017-08-16  0.111725
1       0x        2017-08-17  0.211486
2       0x        2017-08-18  0.283789
3       0x        2017-08-19  0.511434
4       0x        2017-08-20  0.429522
...     ...       ...         ...
657311  zurcoin   2018-02-04  0.003254
657312  zurcoin   2018-02-05  0.002774
657313  zurcoin   2018-02-06  0.001986
657314  zurcoin   2018-02-09  0.002684
657315  zurcoin   2018-02-10  0.002325

I need to instead have a column for each currency's price and the date as the index with only unique dates. There will be plenty of null values which I intend to replace with 0's.

i.e

date          0x_price  10mtoken_price  1337coin_price  ...
2017-08-16    1         4               (NaN)->0        ...
2017-08-17    2         5               (NaN)->0        ...
2017-08-18    3         6               7               ...
...           ...       ...             ...             ...

I've tried to iterate over the dataframe with a groupby as shown:

df2 = pd.DataFrame()
df2["date"] = df["Date"].unique()
df2.set_index("date", inplace=True)

for currency, group in df.groupby("Currency"):
df2.loc[df2.index.isin(group.Date), f"{currency}_price"] = group["Price"]

This returned the desired column names and shape but the dataframe was filled with NaN's.

i.e.

date          0x_price  10mtoken_price  1337coin_price  ...
2017-08-16    NaN       NaN             NaN             ...
2017-08-17    NaN       NaN             NaN             ...
2017-08-18    NaN       NaN             NaN             ...
...           ...       ...             ...             ...

I also tried to achieve the same thing with df.join() as shown:

df2 = pd.DataFrame()
df2["date"] = df["Date"].unique()
df2.set_index("date", inplace=True)

for currency, group in df.groupby("Currency"):
df2 = df2.join(group.set_index("Date")[["Price"]].rename(columns={"Price": f"{currency}_price"}))

This didn't get to finish executing before freezing up my computer. Perhaps it's inefficient and I'm working with around 650,000 entries?

I haven't been able to find the same type of problem here and I haven't been able to figure out a solution after checking the documentation. I've probably missed something but hopefully I've described the problem sufficiently. Thanks in advance.

Body:
Pandas pivot_table could help here. I would use:

resul = df.pivot_table(index=['Date'], columns=['Currency'], values=['Price']).fillna(0)

With your example data, it gives:

               Price
Currency          0x   zurcoin
Date
2017-08-16  0.111725  0.000000
2017-08-17  0.211486  0.000000
2017-08-18  0.283789  0.000000
2017-08-19  0.511434  0.000000
2017-08-20  0.429522  0.000000
2018-02-04  0.000000  0.003254
2018-02-05  0.000000  0.002774
2018-02-06  0.000000  0.001986
2018-02-09  0.000000  0.002684
2018-02-10  0.000000  0.002325


问:
How to remove security requirement on an apiconnect path?
By default apiconnect adds security to be required on all the paths. How can I allow a hit on a path without requiring authentication first?

I've tried unchecking the "Use API security definitions" in the Security section of on my User.count path, but when I use the browser on the URL https://localhost:4002/api/Users/count I get back

```<code>Error
401 Authorization Required
code: AUTHORIZATION_REQUIRED
Error: Authorization Required
at D:\PeteSoft\node\test3\node_modules\loopback\lib\application.js:433:21
at D:\PeteSoft\node\test3\node_modules\loopback\lib\model.js:359:7
at D:\PeteSoft\node\test3\node_modules\loopback\common\models\acl.js:536:16
at D:\PeteSoft\node\test3\node_modules\async\dist\async.js:3888:9
at D:\PeteSoft\node\test3\node_modules\async\dist\async.js:473:16
at iteratorCallback (D:\PeteSoft\node\test3\node_modules\async\dist\async.js:1064:13)
at D:\PeteSoft\node\test3\node_modules\async\dist\async.js:969:16
at D:\PeteSoft\node\test3\node_modules\async\dist\async.js:3885:13
at D:\PeteSoft\node\test3\node_modules\loopback\common\models\acl.js:518:17
at D:\PeteSoft\node\test3\node_modules\loopback\common\models\role.js:447:21
at process._tickCallback (internal/process/next_tick.js:61:11)
</code>```

How can I get the count returned instead of the Authorization Required error?

答:
If you are using the free version, you have to include the IBM client Id  (and IBM client secret is optional)..

However, the Authorization error can be catched, so if you properly configure the catch to capture that error, you could work withouth the IBM Client id.. (it is a bad practice, but might solve your problem).

Catch Auth Error

If you want to work with out those keys, you have to buy the product, and in the Design Tab, it will allow you to remove it...

Body:
By default apiconnect adds security to be required on all the paths. How can I allow a hit on a path without requiring authentication first?

I've tried unchecking the "Use API security definitions" in the Security section of on my User.count path, but when I use the browser on the URL https://localhost:4002/api/Users/count I get back

Error
401 Authorization Required
code: AUTHORIZATION_REQUIRED
Error: Authorization Required
at D:\PeteSoft\node\test3\node_modules\loopback\lib\application.js:433:21
at D:\PeteSoft\node\test3\node_modules\loopback\lib\model.js:359:7
at D:\PeteSoft\node\test3\node_modules\loopback\common\models\acl.js:536:16
at D:\PeteSoft\node\test3\node_modules\async\dist\async.js:3888:9
at D:\PeteSoft\node\test3\node_modules\async\dist\async.js:473:16
at iteratorCallback (D:\PeteSoft\node\test3\node_modules\async\dist\async.js:1064:13)
at D:\PeteSoft\node\test3\node_modules\async\dist\async.js:969:16
at D:\PeteSoft\node\test3\node_modules\async\dist\async.js:3885:13
at D:\PeteSoft\node\test3\node_modules\loopback\common\models\acl.js:518:17
at D:\PeteSoft\node\test3\node_modules\loopback\common\models\role.js:447:21
at process._tickCallback (internal/process/next_tick.js:61:11)

How can I get the count returned instead of the Authorization Required error?


问:
Using foreign keys when referencing aggregate roots by id
It is a best-practice to reference another aggregate root by ID and not by reference (see "Implementing Domain Driven Design" page 359 and following).

```<code>// direct reference
class AggregateRootA(val b: AggregateRootB)

// reference by ID (preferred)
class AggregateRootA(val b: AggregateRootBId)

class AggregateRootB(val id: AggregateRootBId)
class AggregateRootBId(val id: Long)
</code>```

The central argument for this kind of decoupling is that each aggregate root should be a transaction boundary.

I wonder whether it is a good or bad idea to use a foreign key relationship in the database when using reference by id in the code. This constraint would enforce consistency at the database level, since there couldn't be a database record of AggregateRootA without the referenced record of AggregateRootB. This is essentially what I want, since the object would be invalid otherwise.

Is there any downside of using a foreign key here apart from slightly more cumbersome testing?

答:
Is there any downside of using a foreign key here apart from slightly more cumbersome testing?

The constraint introduces some temporal coupling; the database won't allow the write unless the foreign key is actually available, which introduces a "happens-before" relationship between the two.

So if you have a real business concern that the bookkeeping about B has to happen before the bookkeeping linking A to B, then enforcing that constraint at the database is fine.

But that isn't a universal concern, and the foreign key constraint can be painful if the information doesn't always arrive in the natural order.

In short, measure carefully the costs of having the constraint with the benefits of having it.

Body:
It is a best-practice to reference another aggregate root by ID and not by reference (see "Implementing Domain Driven Design" page 359 and following).

// direct reference
class AggregateRootA(val b: AggregateRootB)

// reference by ID (preferred)
class AggregateRootA(val b: AggregateRootBId)

class AggregateRootB(val id: AggregateRootBId)
class AggregateRootBId(val id: Long)

The central argument for this kind of decoupling is that each aggregate root should be a transaction boundary.

I wonder whether it is a good or bad idea to use a foreign key relationship in the database when using reference by id in the code. This constraint would enforce consistency at the database level, since there couldn't be a database record of AggregateRootA without the referenced record of AggregateRootB. This is essentially what I want, since the object would be invalid otherwise.

Is there any downside of using a foreign key here apart from slightly more cumbersome testing?


问:
Calculate new column in pandas based on certain criteria
I am trying to multiple a two df based on certain values.

```<code>data = {'a':10,'b':20,'c':30} .
</code>```

df1:

```<code>id,tag,factor
1,a ,20
2,b ,22
3,c ,25
</code>```

my final result should look like something below:

```<code>id,factor,calc
1,20,200
2,22,440
3,24,750
</code>```

The steps I have tried but giving error.

df['calc'] = df['factor'] * data.get(df['tag'])

答:
In your case </p>

```<code>df1['calc']=df1.factor*df1.tag.map(data)
</code>```

<p>Or do something different </p>

```<code>df['calc']=df.factor*np.vectorize(data.get)(df.tag)#pd.Series(data).get(df.tag).values
</code>```

Body:
I am trying to multiple a two df based on certain values.

data = {'a':10,'b':20,'c':30} .

df1:

id,tag,factor
1,a ,20
2,b ,22
3,c ,25

my final result should look like something below:

id,factor,calc
1,20,200
2,22,440
3,24,750

The steps I have tried but giving error.

df['calc'] = df['factor'] * data.get(df['tag'])

Body:
In your case

df1['calc']=df1.factor*df1.tag.map(data)

Or do something different

df['calc']=df.factor*np.vectorize(data.get)(df.tag)#pd.Series(data).get(df.tag).values


问:
Retain information about requested URL when using curl::curl_fetch_multi
I'm using the following code to perform multiple simultaneous requests.

```<code>urls <- c("https://httpbin.org/status/301", "https://httpbin.org/status/302", "https://httpbin.org/status/200")

result <- list()

p <- curl::new_pool(total_con = 10, host_con = 5, multiplex = T)

cb <- function(res) {
  result <<- append(result, list(res))
  cat("requested URL: ", url, "last URL: ", res$url, "\n\n")
}

for (url in urls) {
  curl::curl_fetch_multi(url, done = cb, handle = curl::new_handle(failonerror = F, nobody = F, followlocation = T, ssl_verifypeer = 0), pool = p)
}

curl::multi_run(pool = p)
</code>```

As you can see, I would like to print to the console the requested URL and the URL, that finally answered with 200 ok.

The following is printed to the console:

```<code>requested URL:  https://httpbin.org/status/200 last URL:  https://httpbin.org/status/200

requested URL:  https://httpbin.org/status/200 last URL:  https://httpbin.org/get

requested URL:  https://httpbin.org/status/200 last URL:  https://httpbin.org/get
</code>```

The requested URL in the console output is always https://httpbin.org/status/200, because it's the last URL that used in the for-loop. So, that is the wrong way to do it.

How can I retain information about the initial requested URL when using curl_fetch_multi to use it after multi_run returned? That means it would be ideal if the requested URL would be added to the res-list to query it with something like cat("requested URL: ", res$requested_url, "last URL: ", res$url, "\n\n").

答:
I had a similar issue where I wanted to do asynchronous POST requests using curl_fetch_multi and check which requests succeeded and which failed. However, due to the structure of the POST statement (all fields are in the request body) there is no identifying information whatsoever in the response object. My solution was to generate custom callback functions which carried an identifier.

urls <- c("https://httpbin.org/status/301", "https://httpbin.org/status/302", "https://httpbin.org/status/200")

result <- list()

# create an identifier for each url
url.ids = paste0("request_", seq_along(urls))

# custom callback function generator: generate a unique function for each url
cb = function(id){
  function(res){
    result[[id]] <<- res
  }
}
# create the list of callback functions
cbfuns = lapply(url.ids, cb)

p <- curl::new_pool(total_con = 10, host_con = 5, multiplex = T)

for (i in seq_along(urls)) {
  curl::curl_fetch_multi(urls[i], done = cbfuns[[i]], handle = curl::new_handle(failonerror = F, nobody = F, followlocation = T, ssl_verifypeer = 0), pool = p)
}

curl::multi_run(pool = p)
```

<p>In this example, the custom callback functions are simply used to name the elements of <code>result</code>:</p>

<pre class="lang-r prettyprint-override"><code>names(result)
## [1] "request_3" "request_1" "request_2"
</code>```

which can then be used to tie each response back to the original request.

ContentLicense: CC BY-SA 3.0

Body:
I'm using the following code to perform multiple simultaneous requests.

urls <- c("https://httpbin.org/status/301", "https://httpbin.org/status/302", "https://httpbin.org/status/200")

result <- list()

p <- curl::new_pool(total_con = 10, host_con = 5, multiplex = T)

cb <- function(res) {
  result <<- append(result, list(res))
  cat("requested URL: ", url, "last URL: ", res$url, "\n\n")
}

for (url in urls) {
  curl::curl_fetch_multi(url, done = cb, handle = curl::new_handle(failonerror = F, nobody = F, followlocation = T, ssl_verifypeer = 0), pool = p)
}

curl::multi_run(pool = p)

As you can see, I would like to print to the console the requested URL and the URL, that finally answered with 200 ok.

The following is printed to the console:

requested URL:  https://httpbin.org/status/200 last URL:  https://httpbin.org/status/200

requested URL:  https://httpbin.org/status/200 last URL:  https://httpbin.org/get

requested URL:  https://httpbin.org/status/200 last URL:  https://httpbin.org/get

The requested URL in the console output is always https://httpbin.org/status/200, because it's the last URL that used in the for-loop. So, that is the wrong way to do it.

How can I retain information about the initial requested URL when using curl_fetch_multi to use it after multi_run returned? That means it would be ideal if the requested URL would be added to the res-list to query it with something like cat("requested URL: ", res$requested_url, "last URL: ", res$url, "\n\n").

Body:
I had a similar issue where I wanted to do asynchronous POST requests using curl_fetch_multi and check which requests succeeded and which failed. However, due to the structure of the POST statement (all fields are in the request body) there is no identifying information whatsoever in the response object. My solution was to generate custom callback functions which carried an identifier.

urls <- c("https://httpbin.org/status/301", "https://httpbin.org/status/302", "https://httpbin.org/status/200")

result <- list()

# create an identifier for each url
url.ids = paste0("request_", seq_along(urls))

# custom callback function generator: generate a unique function for each url
cb = function(id){
  function(res){
    result[[id]] <<- res
  }
}
# create the list of callback functions
cbfuns = lapply(url.ids, cb)

p <- curl::new_pool(total_con = 10, host_con = 5, multiplex = T)

for (i in seq_along(urls)) {
  curl::curl_fetch_multi(urls[i], done = cbfuns[[i]], handle = curl::new_handle(failonerror = F, nobody = F, followlocation = T, ssl_verifypeer = 0), pool = p)
}

curl::multi_run(pool = p)

In this example, the custom callback functions are simply used to name the elements of result:

names(result)
## [1] "request_3" "request_1" "request_2"

which can then be used to tie each response back to the original request.


答:
If you are sure that each date each currency has only 1 price, you can do this:

```<code>df.groupby(['Date', 'Currency'])['Price'].mean().unstack().fillna(0)
</code>```

the mean() function calculates average price, if multiple prices are in df for that ccy at that date.

Title: How to convert stacked columns with duplicate indices into multiple unique columns with pandas?

Body:
If you are sure that each date each currency has only 1 price, you can do this:

df.groupby(['Date', 'Currency'])['Price'].mean().unstack().fillna(0)

the mean() function calculates average price, if multiple prices are in df for that ccy at that date.


问:
How to remove ' ' in a list in python?
I'm still a noob when it comes to python. I have this list that contains strings. My problem is to remove ''(empty string) from the list.

I've tried to use .remove() in python but it still does not work.

For example, I have a list that contains next data:

```<code>a = ['', 'word', 'three', '', 'five']
</code>```

I just want the output to be:

```<code>a = ['word', 'three', 'five']
</code>```

答:
The shortest way using list generators:

[elem for elem in a if elem] (This will filter out Nones, 0 and Falses)

The good short way:

[elem for elem in a if elem != '']

Using filter built-in function with lambdas:

list(filter(lambda e: e != '', a))

The generic way - construct the list of indices to remove and then remove them from the original list:

```<code>indices_to_remove = []
for i, elem in enumerate(a):
    if elem == '':  # Allows to use a complicated logic
        indices_to_remove.append(i)
for i in sorted(indices_to_remove, reverse=True):
    del a[i]
</code>```

Title: How to remove ' ' in a list in python?

Body:
I'm still a noob when it comes to python. I have this list that contains strings. My problem is to remove ''(empty string) from the list.

I've tried to use .remove() in python but it still does not work.

For example, I have a list that contains next data:

a = ['', 'word', 'three', '', 'five']

I just want the output to be:

a = ['word', 'three', 'five']

Body:
The shortest way using list generators:

[elem for elem in a if elem] (This will filter out Nones, 0 and Falses)

The good short way:

[elem for elem in a if elem != '']

Using filter built-in function with lambdas:

list(filter(lambda e: e != '', a))

The generic way - construct the list of indices to remove and then remove them from the original list:

indices_to_remove = []
for i, elem in enumerate(a):
    if elem == '':  # Allows to use a complicated logic
        indices_to_remove.append(i)
for i in sorted(indices_to_remove, reverse=True):
    del a[i]


问:
How can i create a button with boolean and a specific id?
I am currently coding a component to display documentation, like qt doc (https://doc.qt.io/qt-5/qstring.html) and I am using CodeMirror to display a code example.

I am using Angular 2:

          {{ method?.name }}

        <button class="button" *ngIf="!showCode" (click)="showCloseExample()">Show example</button>
        <button class="button" *ngIf="showCode" (click)="showCloseExample()">Close example</button>

          <app-code-mirror [data]="method.documentation"></app-code-mirror>

```

I created a button to show or hide a div with a CodeMirror component inside but when I click this button, it opens every examples.
I think it's because it's on an ngFor.
When I click this button, I want to open only one div and not all.

答: You would have to either use array of showCode flags or add flag to method variable. Right now you are using single flag showCode to show/hide every element in *ngFor.

Body:
I am currently coding a component to display documentation, like qt doc (https://doc.qt.io/qt-5/qstring.html) and I am using CodeMirror to display a code example.

I am using Angular 2:

          {{ method?.name }}

        <button class="button" *ngIf="!showCode" (click)="showCloseExample()">Show example</button>
        <button class="button" *ngIf="showCode" (click)="showCloseExample()">Close example</button>

          <app-code-mirror [data]="method.documentation"></app-code-mirror>

I created a button to show or hide a div with a CodeMirror component inside but when I click this button, it opens every examples.
I think it's because it's on an ngFor.
When I click this button, I want to open only one div and not all.


问:
HttpClientErrorException on restTemplate.getForObject
I'm trying to get the data from "https://swapi.co/api/people/?format=json" and put it into a POJO using the RestTemplate.getForObject() method. I followed the following tutorial for consuming a RESTful webservice and adapted this for my own project: https://spring.io/guides/gs/consuming-rest/.

Upon following the tutorial somehow I got the code to work.. once.. I copied the exact code from the tutorial and I changed the source URL and the object to fit the data into. But when I try to create my own maven project, include all the needed dependencies I get a 403 forbidden Exception.

My code:

@RestController
public class CharacterListController {

    @RequestMapping("/")
    public SWData swData() {

        RestTemplate restTemplate = new RestTemplate();
        SWData swData = restTemplate.getForObject("https://swapi.co/api/people/?format=json", SWData.class);
        return swData;

    }
```

<pre class="lang-java prettyprint-override"><code>@JsonIgnoreProperties(ignoreUnknown = true)
public class SWData {

    String next;
    String results;

    public String getNext() {
        return next;
    }

    public void setNext(String next) {
        this.next = next;
    }

    public String getResults() {
        return results;
    }

    public void setResults(String results) {
        this.results = results;
    }

    @Override
    public String toString() {
        return "SWData{" +
                "next='" + next + '\'' +
                ", results='" + results + '\'' +
                '}';
    }
}
</code>```

Stacktrace:

```<code>2019-05-28 15:00:28.800 ERROR 3643 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden] with root cause

org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:83) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:778) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:736) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:670) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:311) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at io.github.BasPeter.SWAPI.rest.CharacterListController.swData(CharacterListController.java:16) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_211]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_211]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_211]
</code>```

The most frustrating part is that the code functioned at some point. It looked like it worked using the gradle stuff provided by the tutorial, but wont work when using my own maven pom.

Somehow my request is rejected by the server (?), but i don't know why. I think my code is practically the same as the code in the tutorial..

-BP

答:
You have some extra work to do :)

1- Change your SWData class so that it will mach the JSON document you recieve when you type the targeted url on your browser. So your class should look like the following:

```<code>@JsonIgnoreProperties(ignoreUnknown = true)
public class People {
    private long count;
    private String next;
    private String previous;
    private Infos results[];
// Constructors, getters and setters
// Extra: You can add the Lombock project to automatically generates, constructors // getters and setters
}
</code>```

And here is the Infos class:

```<code>    public class Infos {
        private String name;
        private long height;
        private long mass;
        private String hair_color;
        private String skin_color;
        private String eye_color;
        private String birth_year;
        private String gender;
        private String homeworld;
        private String films[];
        private String species[];
        private String vehicles[];
        private String starships[];
        private String created;
        private String edited;
        private String url;
// Constructors, getters and setters
        }
</code>```

2- Now, if you didn't notice, you're trying to access to an https url. So You have first to configure your RestTemplate in order to use SSL certificate for request. To do that, you should first add the Apache HttpClient dependancy to your pom.xml. Use the latest version for example:

```<code><!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.8</version>
</dependency>
</code>```

3- Once you have Apache HttpClient libraries in you project, you can now configure your RestTemplate. Here is how you do that:

```<code>// Imports
    @RestController
    public class CharacterListController {

        @RequestMapping("/")
        public SWData swData() {

            CloseableHttpClient httpClient = HttpClients.custom()
                    .setSSLHostnameVerifier(new NoopHostnameVerifier())
                    .build();
            HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
            requestFactory.setHttpClient(httpClient);
            RestTemplate restTemplate = new RestTemplate(requestFactory);
            SWData swData = restTemplate.getForObject("https://swapi.co/api/people/?format=json", SWData.class);
            return swData;

        }
    }
</code>```

4- That's all, you have only to run your Spring-Boot application now. Hope that will be helpful. Keep me updated with :)

Body:
I'm trying to get the data from "https://swapi.co/api/people/?format=json" and put it into a POJO using the RestTemplate.getForObject() method. I followed the following tutorial for consuming a RESTful webservice and adapted this for my own project: https://spring.io/guides/gs/consuming-rest/.

Upon following the tutorial somehow I got the code to work.. once.. I copied the exact code from the tutorial and I changed the source URL and the object to fit the data into. But when I try to create my own maven project, include all the needed dependencies I get a 403 forbidden Exception.

My code:

@RestController
public class CharacterListController {

    @RequestMapping("/")
    public SWData swData() {

        RestTemplate restTemplate = new RestTemplate();
        SWData swData = restTemplate.getForObject("https://swapi.co/api/people/?format=json", SWData.class);
        return swData;

    }

@JsonIgnoreProperties(ignoreUnknown = true)
public class SWData {

    String next;
    String results;

    public String getNext() {
        return next;
    }

    public void setNext(String next) {
        this.next = next;
    }

    public String getResults() {
        return results;
    }

    public void setResults(String results) {
        this.results = results;
    }

    @Override
    public String toString() {
        return "SWData{" +
                "next='" + next + '\'' +
                ", results='" + results + '\'' +
                '}';
    }
}

Stacktrace:

2019-05-28 15:00:28.800 ERROR 3643 --- [nio-8080-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet]    : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden] with root cause

org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden
    at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:83) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:122) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:102) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:778) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:736) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:670) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:311) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at io.github.BasPeter.SWAPI.rest.CharacterListController.swData(CharacterListController.java:16) ~[classes/:na]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_211]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_211]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_211]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_211]
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:634) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882) ~[spring-webmvc-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) ~[tomcat-embed-websocket-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) ~[spring-web-5.1.6.RELEASE.jar:5.1.6.RELEASE]
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:200) ~[tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:834) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1415) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_211]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_211]
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.17.jar:9.0.17]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_211]

The most frustrating part is that the code functioned at some point. It looked like it worked using the gradle stuff provided by the tutorial, but wont work when using my own maven pom.

Somehow my request is rejected by the server (?), but i don't know why. I think my code is practically the same as the code in the tutorial..

-BP

Body:
You have some extra work to do :)

1- Change your SWData class so that it will mach the JSON document you recieve when you type the targeted url on your browser. So your class should look like the following:

@JsonIgnoreProperties(ignoreUnknown = true)
public class People {
    private long count;
    private String next;
    private String previous;
    private Infos results[];
// Constructors, getters and setters
// Extra: You can add the Lombock project to automatically generates, constructors // getters and setters
}

And here is the Infos class:

    public class Infos {
        private String name;
        private long height;
        private long mass;
        private String hair_color;
        private String skin_color;
        private String eye_color;
        private String birth_year;
        private String gender;
        private String homeworld;
        private String films[];
        private String species[];
        private String vehicles[];
        private String starships[];
        private String created;
        private String edited;
        private String url;
// Constructors, getters and setters
        }

2- Now, if you didn't notice, you're trying to access to an https url. So You have first to configure your RestTemplate in order to use SSL certificate for request. To do that, you should first add the Apache HttpClient dependancy to your pom.xml. Use the latest version for example:

<!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.8</version>
</dependency>

3- Once you have Apache HttpClient libraries in you project, you can now configure your RestTemplate. Here is how you do that:

// Imports
    @RestController
    public class CharacterListController {

        @RequestMapping("/")
        public SWData swData() {

            CloseableHttpClient httpClient = HttpClients.custom()
                    .setSSLHostnameVerifier(new NoopHostnameVerifier())
                    .build();
            HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
            requestFactory.setHttpClient(httpClient);
            RestTemplate restTemplate = new RestTemplate(requestFactory);
            SWData swData = restTemplate.getForObject("https://swapi.co/api/people/?format=json", SWData.class);
            return swData;

        }
    }

4- That's all, you have only to run your Spring-Boot application now. Hope that will be helpful. Keep me updated with :)


问:
eBay Product scraper
I am very limited on VBA,

The Code is in a Module, the code has a sub process as well, so sorry if I post the code wrong

A) open IE

B) Subprocess gets the data.

The code works fine on ebay.com but NOT for ebay.co.uk - can't work out why, also it converts urls to hyperlinks

It only does the first page, I need it to go through an X amount of pages -  have a code but can't get it to work so have removed it.

Can the search query be run AFTER Ebay opens, so it opens, then search item is input to ebay and then code runs, or to run from a cell, IF its Cell A1 the data extracted needs to be pasted in A2 and below.

I have looked at elements for ebay.com and ebay.co.uk and they look the same to me, so can't work out why its not working as it works for 1 and not the other.

I did input the code for getting data from several pages it did not work. I know this code works as I have it for when I fetch urls from google

```<code>Public IE As New SHDocVw.InternetExplorer
Sub GetData()

    Dim HTMLdoc As MSHTml.HTMLDocument
    Dim othwb As Variant
    Dim objShellWindows As New SHDocVw.ShellWindows

    Set IE = CreateObject("internetexplorer.application")

        With IE
            .Visible = True
            '.Navigate "https://www.ebay.co.uk/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=jackets&_sacat=0"
            .Navigate "https://www.ebay.com/sch/i.html_from=R40&_nkw=ralph+lauren&_sacat=1059&LH_TitleDesc=0&_dmd=1&rt=nc"
            While .Busy Or .readyState <> 4: DoEvents: Wend

Set HTMLdoc = IE.document
                ProcessHTMLPage HTMLdoc

            .Quit
        End With

End Sub
code here

    enter

'''''' THIS IS THE SUB PROCESS '''''

Sub ProcessHTMLPage(HTMLPage As MSHTml.HTMLDocument)

    Dim HTMLItem As MSHTml.IHTMLElement
    Dim HTMLItems As MSHTml.IHTMLElementCollection
    Dim HTMLInput As MSHTml.IHTMLElement
    Dim rownum As Long

    rownum = 1

    Set HTMLItems = HTMLPage.getElementsByClassName("s-item__title")

    For Each HTMLItem In HTMLItems

            Cells(rownum, 1).Value = HTMLItem.innerText
            rownum = rownum + 1

    Next HTMLItem

    rownum = 1

    Set HTMLItems = HTMLPage.getElementsByClassName("s-item__price")

    For Each HTMLItem In HTMLItems

            Cells(rownum, 2).Value = HTMLItem.innerText
            rownum = rownum + 1

    Next HTMLItem

    rownum = 1

  Set HTMLItems = HTMLPage.getElementsByClassName("s-item__link")
    For Each HTMLItem In HTMLItems
             Cells(rownum, 3).Value = HTMLItem.href
            rownum = rownum + 1

    Next HTMLItem

'Converts each text hyperlink selected into a working hyperlink from C1 to 25000 rows
  Range("C1:C25000").Select
    For Each xCell In Selection
        ActiveSheet.Hyperlinks.Add Anchor:=xCell, Address:=xCell.Formula
    Next xCell
Range("C1").Select
End Sub
</code>```

Code for going to next page

```<code>pageNumber = 1
'i = 2
    If pageNumber >= 6 Then Exit Do 'the first 6 pages
    internetdata.getElementById("pnnext").click 'next web page
    Do While internet.Busy Or internet.readyState <> 4
        DoEvents
    Loop
    Set internetdata = internet.document
    pageNumber = pageNumber + 1
Loop
</code>```

Does not work on Ebay.co.uk -  NO RESULTS ARE EXTRACTED - Works fine in ebay.com

Need it to get data from X amount of pages and NOT just 1 page

Can the search query be run AFTER Ebay opens, so it opens, then search item is input to ebay and then code runs, or to run from a cell, IF its Cell A1 the data extracted needs to be pasted in A2 and below.

This is my code for google search, I have got it working so the search comes from cell A1, I am look for something like this, I am going to see if I can use the ebay code with this. As this also does the first 25 pages in google search

```<code>enter Sub webpage()

Dim ie As Object
Dim htmlDoc As Object
Dim nextPageElement As Object
Dim div As Object
Dim link As Object
Dim url As String
Dim pageNumber As Long
Dim i As Long

' Takes seach from A1 and places it into google
url = "https://www.google.co.uk/search?q=" & Replace(Worksheets("Sheet1").Range("A1").Value, " ", "+")

Set ie = CreateObject("InternetExplorer.Application")

With ie
    .Visible = True
    .navigate url
    Do While .Busy Or .readyState <> 4
        DoEvents
    Loop
End With

Application.Wait Now + TimeSerial(0, 0, 5)

Set htmlDoc = ie.document

pageNumber = 1
i = 2
Do
    For Each div In htmlDoc.getElementsByTagName("div")
        If div.getAttribute("class") = "r" Then
            Set link = div.getElementsByTagName("a")(0)
            Cells(i, 2).Value = link.getAttribute("href")
            i = i + 1
        End If
    Next div
    If pageNumber >= 25 Then Exit Do 'the first 25 pages
    Set nextPageElement = htmlDoc.getElementById("pnnext")
    If nextPageElement Is Nothing Then Exit Do

    ' Clicks web next page
    nextPageElement.Click 'next web page
    Do While ie.Busy Or ie.readyState <> 4
        DoEvents
    Loop
    Application.Wait Now + TimeSerial(0, 0, 5)
    Set htmlDoc = ie.document
    pageNumber = pageNumber + 1
Loop

MsgBox "All Done"

Set ie = Nothing
Set htmlDoc = Nothing
Set nextPageElement = Nothing
Set div = Nothing
Set link = Nothing
</code>```

End Sub
code here

答:
Question 1: Why does it work for one domain but not the other?

To answer question 1 (the other questions should be new posts) - the html is not the same at all. The classes which work for ebay.com are not found in ebay.co.uk; So, your loop over collections doesn't do anything because they are count 0 (or length 0 with nodeLists if using querySelectorAll). Instead, you need branched code. Set your selectors based on the url domain.

I have used css selectors as this is the easiest, and fastest way, to select the required elements whilst maintaining the flexibility of a code re-factor to reduce the lines of repeated code.

Side note:

If you are unsure about whether your selection method will work across different pages you can do at least two things:

Right click > inspect element > visually check the class names are the same for the elements you are attempting to compare. So, if you are looking at product names, are the class names in the html the same on both pages?

You can use the search facility of the browser > open element tab via F12 then press Ctrl+F to pull up search box > enter your class name from the first page into this box in the second page and hit enter. You can also enter css selectors here and some cases regex. You will get a hit count telling you how many matches found. You can keep pressing enter to cycle through matches and each match will be highlighted in the html above, so you can easily compare if matched results are what you expected.

click image to enlarge

img url:  https://i.stack.imgur.com/MWkEx.png

VBA:

```<code>Option Explicit

Public Sub GetData()
    Dim htmlDoc As MSHTML.HTMLDocument, ie As SHDocVw.InternetExplorer, ws As Worksheet

    Set ie = New SHDocVw.InternetExplorer
    Set htmlDoc = New MSHTML.HTMLDocument
    Set ws = ThisWorkbook.Worksheets("Sheet1")

    With ie
        .Visible = True
        '.Navigate2 "https://www.ebay.co.uk/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=jackets&_sacat=0"
        .Navigate2 "https://www.ebay.com/sch/i.html_from=R40&_nkw=ralph+lauren&_sacat=1059&LH_TitleDesc=0&_dmd=1&rt=nc"
        While .Busy Or .readyState <> 4: DoEvents: Wend

        Dim index As Long, HTMLItems As Object, rowNum As Long, xCell As Range
        Dim cssSelectors(), i As Long

        Select Case True
        Case InStr(.document.URL, "ebay.co.uk") > 0
            cssSelectors = Array(".gvtitle a", ".amt", ".gvtitle a")
        Case InStr(.document.URL, "ebay.com") > 0
            cssSelectors = Array(".s-item__title", ".s-item__price", ".s-item__link")
        End Select

        With ws
            For i = LBound(cssSelectors) To UBound(cssSelectors)
                rowNum = 1
                Set HTMLItems = ie.document.querySelectorAll(cssSelectors(i))

                For index = 0 To HTMLItems.length - 1
                    .Cells(rowNum, i + 1).Value = IIf(i = 2, HTMLItems.item(index).getAttribute("href"), HTMLItems.item(index).innerText)
                    rowNum = rowNum + 1
                Next
            Next
            For Each xCell In .Range("C1:C25000") '<= all these really?
                .Hyperlinks.Add Anchor:=xCell, Address:=xCell.Formula
            Next xCell
        End With
        .Quit
    End With
End Sub
</code>```

Title: eBay Product scraper

Body:
I am very limited on VBA,

The Code is in a Module, the code has a sub process as well, so sorry if I post the code wrong

A) open IE

B) Subprocess gets the data.

The code works fine on ebay.com but NOT for ebay.co.uk - can't work out why, also it converts urls to hyperlinks

It only does the first page, I need it to go through an X amount of pages -  have a code but can't get it to work so have removed it.

Can the search query be run AFTER Ebay opens, so it opens, then search item is input to ebay and then code runs, or to run from a cell, IF its Cell A1 the data extracted needs to be pasted in A2 and below.

I have looked at elements for ebay.com and ebay.co.uk and they look the same to me, so can't work out why its not working as it works for 1 and not the other.

I did input the code for getting data from several pages it did not work. I know this code works as I have it for when I fetch urls from google

Public IE As New SHDocVw.InternetExplorer
Sub GetData()

    Dim HTMLdoc As MSHTml.HTMLDocument
    Dim othwb As Variant
    Dim objShellWindows As New SHDocVw.ShellWindows

    Set IE = CreateObject("internetexplorer.application")

        With IE
            .Visible = True
            '.Navigate "https://www.ebay.co.uk/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=jackets&_sacat=0"
            .Navigate "https://www.ebay.com/sch/i.html_from=R40&_nkw=ralph+lauren&_sacat=1059&LH_TitleDesc=0&_dmd=1&rt=nc"
            While .Busy Or .readyState <> 4: DoEvents: Wend

Set HTMLdoc = IE.document
                ProcessHTMLPage HTMLdoc

            .Quit
        End With

End Sub
code here

    enter

'''''' THIS IS THE SUB PROCESS '''''

Sub ProcessHTMLPage(HTMLPage As MSHTml.HTMLDocument)

    Dim HTMLItem As MSHTml.IHTMLElement
    Dim HTMLItems As MSHTml.IHTMLElementCollection
    Dim HTMLInput As MSHTml.IHTMLElement
    Dim rownum As Long

    rownum = 1

    Set HTMLItems = HTMLPage.getElementsByClassName("s-item__title")

    For Each HTMLItem In HTMLItems

            Cells(rownum, 1).Value = HTMLItem.innerText
            rownum = rownum + 1

    Next HTMLItem

    rownum = 1

    Set HTMLItems = HTMLPage.getElementsByClassName("s-item__price")

    For Each HTMLItem In HTMLItems

            Cells(rownum, 2).Value = HTMLItem.innerText
            rownum = rownum + 1

    Next HTMLItem

    rownum = 1

  Set HTMLItems = HTMLPage.getElementsByClassName("s-item__link")
    For Each HTMLItem In HTMLItems
             Cells(rownum, 3).Value = HTMLItem.href
            rownum = rownum + 1

    Next HTMLItem

'Converts each text hyperlink selected into a working hyperlink from C1 to 25000 rows
  Range("C1:C25000").Select
    For Each xCell In Selection
        ActiveSheet.Hyperlinks.Add Anchor:=xCell, Address:=xCell.Formula
    Next xCell
Range("C1").Select
End Sub

Code for going to next page

pageNumber = 1
'i = 2
    If pageNumber >= 6 Then Exit Do 'the first 6 pages
    internetdata.getElementById("pnnext").click 'next web page
    Do While internet.Busy Or internet.readyState <> 4
        DoEvents
    Loop
    Set internetdata = internet.document
    pageNumber = pageNumber + 1
Loop

Does not work on Ebay.co.uk -  NO RESULTS ARE EXTRACTED - Works fine in ebay.com

Need it to get data from X amount of pages and NOT just 1 page

Can the search query be run AFTER Ebay opens, so it opens, then search item is input to ebay and then code runs, or to run from a cell, IF its Cell A1 the data extracted needs to be pasted in A2 and below.

This is my code for google search, I have got it working so the search comes from cell A1, I am look for something like this, I am going to see if I can use the ebay code with this. As this also does the first 25 pages in google search

enter Sub webpage()

Dim ie As Object
Dim htmlDoc As Object
Dim nextPageElement As Object
Dim div As Object
Dim link As Object
Dim url As String
Dim pageNumber As Long
Dim i As Long

' Takes seach from A1 and places it into google
url = "https://www.google.co.uk/search?q=" & Replace(Worksheets("Sheet1").Range("A1").Value, " ", "+")

Set ie = CreateObject("InternetExplorer.Application")

With ie
    .Visible = True
    .navigate url
    Do While .Busy Or .readyState <> 4
        DoEvents
    Loop
End With

Application.Wait Now + TimeSerial(0, 0, 5)

Set htmlDoc = ie.document

pageNumber = 1
i = 2
Do
    For Each div In htmlDoc.getElementsByTagName("div")
        If div.getAttribute("class") = "r" Then
            Set link = div.getElementsByTagName("a")(0)
            Cells(i, 2).Value = link.getAttribute("href")
            i = i + 1
        End If
    Next div
    If pageNumber >= 25 Then Exit Do 'the first 25 pages
    Set nextPageElement = htmlDoc.getElementById("pnnext")
    If nextPageElement Is Nothing Then Exit Do

    ' Clicks web next page
    nextPageElement.Click 'next web page
    Do While ie.Busy Or ie.readyState <> 4
        DoEvents
    Loop
    Application.Wait Now + TimeSerial(0, 0, 5)
    Set htmlDoc = ie.document
    pageNumber = pageNumber + 1
Loop

MsgBox "All Done"

Set ie = Nothing
Set htmlDoc = Nothing
Set nextPageElement = Nothing
Set div = Nothing
Set link = Nothing

End Sub
code here

Body:
Question 1: Why does it work for one domain but not the other?

To answer question 1 (the other questions should be new posts) - the html is not the same at all. The classes which work for ebay.com are not found in ebay.co.uk; So, your loop over collections doesn't do anything because they are count 0 (or length 0 with nodeLists if using querySelectorAll). Instead, you need branched code. Set your selectors based on the url domain.

I have used css selectors as this is the easiest, and fastest way, to select the required elements whilst maintaining the flexibility of a code re-factor to reduce the lines of repeated code.

Side note:

If you are unsure about whether your selection method will work across different pages you can do at least two things:

Right click > inspect element > visually check the class names are the same for the elements you are attempting to compare. So, if you are looking at product names, are the class names in the html the same on both pages?

You can use the search facility of the browser > open element tab via F12 then press Ctrl+F to pull up search box > enter your class name from the first page into this box in the second page and hit enter. You can also enter css selectors here and some cases regex. You will get a hit count telling you how many matches found. You can keep pressing enter to cycle through matches and each match will be highlighted in the html above, so you can easily compare if matched results are what you expected.

click image to enlarge

img url:  https://i.stack.imgur.com/MWkEx.png

VBA:

Option Explicit

Public Sub GetData()
    Dim htmlDoc As MSHTML.HTMLDocument, ie As SHDocVw.InternetExplorer, ws As Worksheet

    Set ie = New SHDocVw.InternetExplorer
    Set htmlDoc = New MSHTML.HTMLDocument
    Set ws = ThisWorkbook.Worksheets("Sheet1")

    With ie
        .Visible = True
        '.Navigate2 "https://www.ebay.co.uk/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=jackets&_sacat=0"
        .Navigate2 "https://www.ebay.com/sch/i.html_from=R40&_nkw=ralph+lauren&_sacat=1059&LH_TitleDesc=0&_dmd=1&rt=nc"
        While .Busy Or .readyState <> 4: DoEvents: Wend

        Dim index As Long, HTMLItems As Object, rowNum As Long, xCell As Range
        Dim cssSelectors(), i As Long

        Select Case True
        Case InStr(.document.URL, "ebay.co.uk") > 0
            cssSelectors = Array(".gvtitle a", ".amt", ".gvtitle a")
        Case InStr(.document.URL, "ebay.com") > 0
            cssSelectors = Array(".s-item__title", ".s-item__price", ".s-item__link")
        End Select

        With ws
            For i = LBound(cssSelectors) To UBound(cssSelectors)
                rowNum = 1
                Set HTMLItems = ie.document.querySelectorAll(cssSelectors(i))

                For index = 0 To HTMLItems.length - 1
                    .Cells(rowNum, i + 1).Value = IIf(i = 2, HTMLItems.item(index).getAttribute("href"), HTMLItems.item(index).innerText)
                    rowNum = rowNum + 1
                Next
            Next
            For Each xCell In .Range("C1:C25000") '<= all these really?
                .Hyperlinks.Add Anchor:=xCell, Address:=xCell.Formula
            Next xCell
        End With
        .Quit
    End With
End Sub


答:
You can just do like below. filter function does that by default.

```<code>a = ['', 'word', 'three', '', 'five']

print(list(filter(None, a)))
</code>```

Output:

```<code>['word', 'three', 'five']
</code>```

Title: How to remove ' ' in a list in python?

Body:
You can just do like below. filter function does that by default.

a = ['', 'word', 'three', '', 'five']

print(list(filter(None, a)))

Output:

['word', 'three', 'five']


问:
Hyper-V Manager no longer shows my virtual machines
I'm running Windows 10 64-bit Enterprise Mode. Have been using Hyper-V Manager with a couple VMs I created without issue for a few years now. Last night my desktop rebooted after security updates were applied. Today when I open Hyper-V, my VMs are no longer displayed, and my desktop no longer appears listed in the left pane beneath "Hyper-V Manager".

I've verified that the expected services related to Hyper-V are all running, and all the boxes are selected when I look at Hyper-V from Control Panel > Programs and Features > Turn Windows features on or off.

The only option I have in Hyper-V Manager is "Connect to server". When I try to connect to my local machine, I get an error saying to check that the Virtual Machine Management service is running. As stated, it is.

Not sure what else to try to get reconnected to my virtual machines.

答:
Looks like this was related to a network adapter issue on my host system. Once the networking team cleared that up, I was able to see my virtual machines again within the Hyper-V Manager.

I did have to delete the old virtual switch then create a new one, as the VMs would start up but not launch. Error was "External ethernet adapter is already bound to the Microsoft virtual switch protocol."

After creating the new switch, I had to attach it to each of my existing snapshots. After that, all was well.


问:
Why I am getting RuntimeError: generator raised StopIteration? and how to solve it?
I am making Bigrams of the tokens stored in list docToken.

```<code>print(docToken[520])
</code>```

Output: ['sleepy', 'account', 'just', 'man', 'tired', 'twitter', 'case',
  'romney', 'candidate', 'looks']

```<code>list(nltk.bigrams(docToken[520]))
</code>```

Output: [('sleepy', 'account'),  ('account', 'just'),  ('just', 'man'),
  ('man', 'tired'),  ('tired', 'twitter'),  ('twitter', 'case'),
  ('case', 'romney'),  ('romney', 'candidate'),  ('candidate', 'looks')]

and when i'm using nltk.bigrams(docToken[i]) in a loop i'm getting following error on the range>=1000:

```<code>bigram=[]
for i in range(5000):
    ls=list(nltk.bigrams(docToken[i]))
    for j in ls:
        bigram.append(list(j))
</code>```

it's working just fine when the range(500) in the first loop but when the Range is 1000 or more it is giving me following error:

```<code>StopIteration                             Traceback (most recent call last)
~\Anaconda3\lib\site-packages\nltk\util.py in ngrams(sequence, n, pad_left,
  pad_right, left_pad_symbol, right_pad_symbol)
        467     while n > 1:
    --> 468         history.append(next(sequence))
        469         n -= 1

StopIteration:

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-76-8982951528bd> in <module>()
      1 bigram=[]
      2 for i in range(5000):
----> 3     ls=list(nltk.bigrams(docToken[i]))
      4     for j in ls:
      5         bigram.append(list(j))

~\Anaconda3\lib\site-packages\nltk\util.py in bigrams(sequence, **kwargs)
    489     """
    490
--> 491     for item in ngrams(sequence, 2, **kwargs):
    492         yield item
    493

RuntimeError: generator raised StopIteration
</code>```

答:
I was not able to resolve this error. Not sure why nltk.bigrams(docToken[i]) is generating this but I was able to create bigrams by using the following code.

```<code>bigram={}
for i in range(size):
    ls=[]
    for j in range(len(docToken[i])-1):
        for k in range(j,len(docToken[i])-1):
            ls.append([docToken[i][j],docToken[i][k+1]])

    bigram[i]=ls
</code>```

Body:
I am making Bigrams of the tokens stored in list docToken.

print(docToken[520])

Output: ['sleepy', 'account', 'just', 'man', 'tired', 'twitter', 'case',
  'romney', 'candidate', 'looks']

list(nltk.bigrams(docToken[520]))

Output: [('sleepy', 'account'),  ('account', 'just'),  ('just', 'man'),
  ('man', 'tired'),  ('tired', 'twitter'),  ('twitter', 'case'),
  ('case', 'romney'),  ('romney', 'candidate'),  ('candidate', 'looks')]

and when i'm using nltk.bigrams(docToken[i]) in a loop i'm getting following error on the range>=1000:

bigram=[]
for i in range(5000):
    ls=list(nltk.bigrams(docToken[i]))
    for j in ls:
        bigram.append(list(j))

it's working just fine when the range(500) in the first loop but when the Range is 1000 or more it is giving me following error:

StopIteration                             Traceback (most recent call last)
~\Anaconda3\lib\site-packages\nltk\util.py in ngrams(sequence, n, pad_left,
  pad_right, left_pad_symbol, right_pad_symbol)
        467     while n > 1:
    --> 468         history.append(next(sequence))
        469         n -= 1

StopIteration:

The above exception was the direct cause of the following exception:

RuntimeError                              Traceback (most recent call last)
<ipython-input-76-8982951528bd> in <module>()
      1 bigram=[]
      2 for i in range(5000):
----> 3     ls=list(nltk.bigrams(docToken[i]))
      4     for j in ls:
      5         bigram.append(list(j))

~\Anaconda3\lib\site-packages\nltk\util.py in bigrams(sequence, **kwargs)
    489     """
    490
--> 491     for item in ngrams(sequence, 2, **kwargs):
    492         yield item
    493

RuntimeError: generator raised StopIteration

Body:
I was not able to resolve this error. Not sure why nltk.bigrams(docToken[i]) is generating this but I was able to create bigrams by using the following code.

bigram={}
for i in range(size):
    ls=[]
    for j in range(len(docToken[i])-1):
        for k in range(j,len(docToken[i])-1):
            ls.append([docToken[i][j],docToken[i][k+1]])

    bigram[i]=ls