问:
AWS Elastic Beanstalk - Failing to install requirements.txt on deployment
I have tried the similar problems' solutions on here but none seem to work. It seems that I get a memory error when installing tensorflow from requirements.txt. Does anyone know of a workaround? I believe that installing with --no-cache-dir would fix it but I can't figure out how to get EB to do that. Thank you.

Logs:

```<code>----------------------------------------
Collecting tensorflow==2.8.0
  Downloading tensorflow-2.8.0-cp38-cp38-manylinux2010_x86_64.whl (497.6 MB)

2022/02/05 22:08:17.264961 [ERROR] An error occurred during execution of command [app-deploy] - [InstallDependency]. Stop running the command. Error: fail to install dependencies with requirements.txt file with error Command /bin/sh -c /var/app/venv/staging-LQM1lest/bin/pip install -r requirements.txt failed with error exit status 2. Stderr:ERROR: Exception:
Traceback (most recent call last):
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
    status = run_func(*args)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 338, in run
    requirement_set = resolver.resolve(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 482, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 349, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 201, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 281, in __init__
    super().__init__(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 527, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 213, in unpack_url
    file = get_http_url(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 94, in get_http_url
    from_path, content_type = download(link, temp_dir.path)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/network/download.py", line 145, in __call__
    for chunk in chunks:
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/cli/progress_bars.py", line 144, in iter
    for x in it:
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks
    for chunk in response.raw.stream(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 65, in read
    self._close()
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 52, in _close
    self.__callback(self.__buf.getvalue())
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/cachecontrol/controller.py", line 309, in cache_response
    cache_url, self.serializer.dumps(request, response, body=body)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/cachecontrol/serialize.py", line 72, in dumps
    return b",".join([b"cc=4", msgpack.dumps(data, use_bin_type=True)])
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 960, in pack
    self._pack(obj)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 943, in _pack
    return self._pack_map_pairs(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 1045, in _pack_map_pairs
    self._pack(v, nest_limit - 1)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 943, in _pack
    return self._pack_map_pairs(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 1045, in _pack_map_pairs
    self._pack(v, nest_limit - 1)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 889, in _pack
    return self._buffer.write(obj)
MemoryError

2022/02/05 22:08:17.264976 [INFO] Executing cleanup logic
2022/02/05 22:08:17.265065 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[{"msg":"Instance deployment failed to install application dependencies. The deployment failed.","timestamp":1644098897,"severity":"ERROR"},{"msg":"Instance deployment failed. For details, see 'eb-engine.log'.","timestamp":1644098897,"severity":"ERROR"}]}]}
</code>```

Requirements.txt:

```<code>absl-py==1.0.0
asgiref==3.5.0
astunparse==1.6.3
awsebcli==3.20.3
backports.zoneinfo==0.2.1
botocore==1.23.49
cachetools==5.0.0
cement==2.8.2
certifi==2021.10.8
charset-normalizer==2.0.11
colorama==0.4.3
cycler==0.11.0
Django==4.0.2
django-crispy-forms==1.14.0
django-environ==0.8.1
flatbuffers==2.0
fonttools==4.29.1
future==0.16.0
gast==0.5.3
google-auth==2.6.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.43.0
h5py==3.6.0
idna==3.3
importlib-metadata==4.10.1
imutils==0.5.4
jmespath==0.10.0
keras==2.8.0
Keras-Preprocessing==1.1.2
kiwisolver==1.3.2
libclang==13.0.0
Markdown==3.3.6
matplotlib==3.5.1
numpy==1.22.2
oauthlib==3.2.0
opencv-python==4.5.5.62
opt-einsum==3.3.0
packaging==21.3
pathspec==0.9.0
Pillow==9.0.1
protobuf==3.19.4
psycopg2-binary==2.9.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.7
python-dateutil==2.8.2
PyYAML==5.4.1
requests==2.26.0
requests-oauthlib==1.3.1
rsa==4.8
semantic-version==2.8.5
six==1.14.0
sqlparse==0.4.2
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.8.0
tensorflow-io-gcs-filesystem==0.24.0
termcolor==1.1.0
tf-estimator-nightly==2.8.0.dev2021122109
typing_extensions==4.0.1
tzdata==2021.5
urllib3==1.26.8
wcwidth==0.1.9
Werkzeug==2.0.2
wrapt==1.13.3
zipp==3.7.0
</code>```

答: The error says MemoryError. You must upgrade your ec2 instance to something with more memory. tensorflow is very memory hungry application.

Body:
I have tried the similar problems' solutions on here but none seem to work. It seems that I get a memory error when installing tensorflow from requirements.txt. Does anyone know of a workaround? I believe that installing with --no-cache-dir would fix it but I can't figure out how to get EB to do that. Thank you.

Logs:

----------------------------------------
Collecting tensorflow==2.8.0
  Downloading tensorflow-2.8.0-cp38-cp38-manylinux2010_x86_64.whl (497.6 MB)

2022/02/05 22:08:17.264961 [ERROR] An error occurred during execution of command [app-deploy] - [InstallDependency]. Stop running the command. Error: fail to install dependencies with requirements.txt file with error Command /bin/sh -c /var/app/venv/staging-LQM1lest/bin/pip install -r requirements.txt failed with error exit status 2. Stderr:ERROR: Exception:
Traceback (most recent call last):
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
    status = run_func(*args)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
    return func(self, options, args)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 338, in run
    requirement_set = resolver.resolve(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve
    result = self._result = resolver.resolve(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 482, in resolve
    state = resolution.resolve(requirements, max_rounds=max_rounds)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 349, in resolve
    self._add_to_criteria(self.state.criteria, r, parent=None)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/resolvelib/resolvers.py", line 173, in _add_to_criteria
    if not criterion.candidates:
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/resolvelib/structs.py", line 151, in __bool__
    return bool(self._sequence)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 155, in __bool__
    return any(self)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 143, in <genexpr>
    return (c for c in iterator if id(c) not in self._incompatible_ids)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/found_candidates.py", line 47, in _iter_built
    candidate = func()
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 201, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 281, in __init__
    super().__init__(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 527, in _prepare_linked_requirement
    local_file = unpack_url(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 213, in unpack_url
    file = get_http_url(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 94, in get_http_url
    from_path, content_type = download(link, temp_dir.path)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/network/download.py", line 145, in __call__
    for chunk in chunks:
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/cli/progress_bars.py", line 144, in iter
    for x in it:
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_internal/network/utils.py", line 63, in response_chunks
    for chunk in response.raw.stream(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/urllib3/response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 65, in read
    self._close()
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/cachecontrol/filewrapper.py", line 52, in _close
    self.__callback(self.__buf.getvalue())
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/cachecontrol/controller.py", line 309, in cache_response
    cache_url, self.serializer.dumps(request, response, body=body)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/cachecontrol/serialize.py", line 72, in dumps
    return b",".join([b"cc=4", msgpack.dumps(data, use_bin_type=True)])
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/__init__.py", line 35, in packb
    return Packer(**kwargs).pack(o)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 960, in pack
    self._pack(obj)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 943, in _pack
    return self._pack_map_pairs(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 1045, in _pack_map_pairs
    self._pack(v, nest_limit - 1)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 943, in _pack
    return self._pack_map_pairs(
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 1045, in _pack_map_pairs
    self._pack(v, nest_limit - 1)
  File "/var/app/venv/staging-LQM1lest/lib/python3.8/site-packages/pip/_vendor/msgpack/fallback.py", line 889, in _pack
    return self._buffer.write(obj)
MemoryError

2022/02/05 22:08:17.264976 [INFO] Executing cleanup logic
2022/02/05 22:08:17.265065 [INFO] CommandService Response: {"status":"FAILURE","api_version":"1.0","results":[{"status":"FAILURE","msg":"Engine execution has encountered an error.","returncode":1,"events":[{"msg":"Instance deployment failed to install application dependencies. The deployment failed.","timestamp":1644098897,"severity":"ERROR"},{"msg":"Instance deployment failed. For details, see 'eb-engine.log'.","timestamp":1644098897,"severity":"ERROR"}]}]}

Requirements.txt:

absl-py==1.0.0
asgiref==3.5.0
astunparse==1.6.3
awsebcli==3.20.3
backports.zoneinfo==0.2.1
botocore==1.23.49
cachetools==5.0.0
cement==2.8.2
certifi==2021.10.8
charset-normalizer==2.0.11
colorama==0.4.3
cycler==0.11.0
Django==4.0.2
django-crispy-forms==1.14.0
django-environ==0.8.1
flatbuffers==2.0
fonttools==4.29.1
future==0.16.0
gast==0.5.3
google-auth==2.6.0
google-auth-oauthlib==0.4.6
google-pasta==0.2.0
grpcio==1.43.0
h5py==3.6.0
idna==3.3
importlib-metadata==4.10.1
imutils==0.5.4
jmespath==0.10.0
keras==2.8.0
Keras-Preprocessing==1.1.2
kiwisolver==1.3.2
libclang==13.0.0
Markdown==3.3.6
matplotlib==3.5.1
numpy==1.22.2
oauthlib==3.2.0
opencv-python==4.5.5.62
opt-einsum==3.3.0
packaging==21.3
pathspec==0.9.0
Pillow==9.0.1
protobuf==3.19.4
psycopg2-binary==2.9.3
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyparsing==3.0.7
python-dateutil==2.8.2
PyYAML==5.4.1
requests==2.26.0
requests-oauthlib==1.3.1
rsa==4.8
semantic-version==2.8.5
six==1.14.0
sqlparse==0.4.2
tensorboard==2.8.0
tensorboard-data-server==0.6.1
tensorboard-plugin-wit==1.8.1
tensorflow==2.8.0
tensorflow-io-gcs-filesystem==0.24.0
termcolor==1.1.0
tf-estimator-nightly==2.8.0.dev2021122109
typing_extensions==4.0.1
tzdata==2021.5
urllib3==1.26.8
wcwidth==0.1.9
Werkzeug==2.0.2
wrapt==1.13.3
zipp==3.7.0


来源: stackexchange

问:
Expanding a data frame based on column-values
I have a data frame that has a structure similar to the following:

```<code>df <- data.frame(sample_id = c("s1", "s2", "s3"),
                 group = c("1", "2", "3"),
                 frequency = c("4", "2", "2"))

print(df)

sample_id    group   frequency
  s1           1         4
  s2           2         2
  s3           3         2
</code>```

For further manipulations and compatibility with downstream functions, I am looking for a way to transform this data frame by expanding the number of rows based on the number indicated in the frequency column. The expected output should look like this:

```<code>sample_id    group   frequency
  s1           1         1
  s1           1         1
  s1           1         1
  s1           1         1
  s2           2         1
  s2           2         1
  s3           3         1
  s3           3         1
</code>```

I'd appreciate any help towards this (the real data set I have is huge, and I could not figure out an efficient way to do so).

答:
We could use uncount from tidyr package:

```<code>library(dplyr)
library(tidyr)

df %>%
  type.convert(as.is = TRUE) %>%
  uncount(frequency)
</code>```
```<code>  sample_id group
1        s1     1
2        s1     1
3        s1     1
4        s1     1
5        s2     2
6        s2     2
7        s3     3
8        s3     3
</code>```

ContentLicense: CC BY-SA 4.0

Body:
I have a data frame that has a structure similar to the following:

df <- data.frame(sample_id = c("s1", "s2", "s3"),
                 group = c("1", "2", "3"),
                 frequency = c("4", "2", "2"))

print(df)

sample_id    group   frequency
  s1           1         4
  s2           2         2
  s3           3         2

For further manipulations and compatibility with downstream functions, I am looking for a way to transform this data frame by expanding the number of rows based on the number indicated in the frequency column. The expected output should look like this:

sample_id    group   frequency
  s1           1         1
  s1           1         1
  s1           1         1
  s1           1         1
  s2           2         1
  s2           2         1
  s3           3         1
  s3           3         1

I'd appreciate any help towards this (the real data set I have is huge, and I could not figure out an efficient way to do so).

Body:
We could use uncount from tidyr package:

library(dplyr)
library(tidyr)

df %>%
  type.convert(as.is = TRUE) %>%
  uncount(frequency)

  sample_id group
1        s1     1
2        s1     1
3        s1     1
4        s1     1
5        s2     2
6        s2     2
7        s3     3
8        s3     3


问:
Reading quicken data files
Looking for an open source library, for C++, Java, C# or Python, for reading the data from Quicken .qdf files.

@Swati:  Quicken .qif format is for transfer only and is not kept up to date by the application  like the .qdf file is.

答:
Check out http://qif.codeplex.com/

You may want to check the license before use. Thanks

OwnerDisplayName: Bart

ContentLicense: CC BY-SA 2.5

Title: Reading quicken data files

ContentLicense: CC BY-SA 3.0


问:
How to distinguish the terms 'primitives' 'axioms' 'expressions' 'basic axioms' 'primitive terms' in description logic / OWL?
I have read several terms in the context of description logic / OWL Web Ontology Language, but it is hard for me to distinguish them. I could not find for any of them a clear definition in the context of the others, and authors use them often without clear definitions. Can someone help me?

The terms are in description logic / OWL:

'primitives'

'axioms'

'facts'

'expressions'

'basic axioms'

'complex axioms'

'primitive terms'

'triple'

'class description'

'class constructors'

'assertion' / 'asserted'

If someone could provide an example then distinguish them or show similarities between, would be very helpful. Thanks in advance!

答:
An axiom is any thing you know to be true in your domain, e.g. the fact that all the dogs are animals.

A particular kind of axiom is the assertion (or fact), which declares something about specific individuals (e.g. Dog(lessie) or Parent(bob,alice)).

A triple is a particular way of storing a statements (e.g. <ex:lessie rdf:type ex:Dog>, or <ex:bob Parent ex:alice>, or <ex:Dog rdfs:subclassof ex:Animal>).

The primitive terms are the entities, such as classes, properties, and individuals.

A primitive is a particular function used for building class expressions.

Anyway, some terms (like the adjectives "basic" and "complex") may have different meanings depending on the particular context.


问:
How does tf.keras.metrics.TopKCategoricalAccuracy differ from Precision@k?
Coming from recommender systems, precision@k is a popular metric.

precision@k = number of relevant predictions in top k / k

On the tensorflow docs for tf.keras.metrics.TopKCategoricalAccuracy it states

Computes how often targets are in the top K predictions.

https://www.tensorflow.org/api_docs/python/tf/keras/metrics/TopKCategoricalAccuracy

Which seems to be exactly the same as precision@k. Am I missing something or are they equivalent and it just comes down to TF/recommender terminology?

答:
No. you are totally right.
They both are the same metric.</p>


问:
How important is it to check for malloc failures?
Is always protecting mallocs important? By protecting I mean:

```<code>char *test_malloc = malloc(sizeof(char) * 10000);

if (!test_malloc)
  exit(EXIT_FAILURE);
</code>```

I mean, in electronic devices, I don't doubt it's essential.
But in programs I'm running on my own machine that I'm sure my allocation size will be positive and that the size will not be astronomical.
Some people say, "Ah imagine there’s not enough memory in your computer at this moment."

答:
The Standard makes no distinction between a number of kinds of implementations:

Those where a program a program could allocate as much memory as it malloc() will supply, without affecting system stability, and where a program that could behave usefully without needing the storage from the failed allocation may run normally.

Those where system memory exhaustion may cause programs to terminate unexpectedly even without malloc() ever indicating a failure, but where attempting to dereference a pointer returned via malloc() will either access a valid memory region or force program termination.

Those where malloc() might return null, and dereferencing a null pointer might have unwanted effects beyond abnormal program termination, but where programs might be unexpectedly terminated due to a lack of storage even when malloc() reports success.

If there's no way a program would be able to usefully continue operation if malloc() fails, it's probably best to allocate memory via wrapper function that calls malloc() and forcibly terminates the program if it returns null.

Body:
Is always protecting mallocs important? By protecting I mean:

char *test_malloc = malloc(sizeof(char) * 10000);

if (!test_malloc)
  exit(EXIT_FAILURE);

I mean, in electronic devices, I don't doubt it's essential.
But in programs I'm running on my own machine that I'm sure my allocation size will be positive and that the size will not be astronomical.
Some people say, "Ah imagine there’s not enough memory in your computer at this moment."


问:
View octal encoding for a Unicode string (in-browser or OSX tool)
There is an invisible character breaking my XML parser.

c&

The XML claims to be UTF-8, but when I try to use <c:import . . . charEncoding="UTF-8">

I get this friendly message:

ERROR: javax.servlet.jsp.JspException: java.io.CharConversionException: illegal utf8 encoding at (187)

I have been able to locate the source of the problem. It is an invisible character located between 'c' and '&'.

I would like to know more about this character, but it seems IntelliJ cannot show me hidden characters . . .

I think I saw a tool online that would convert Unicode characters to their octal values, but I can not find it again. If there is a tool I need to download that would be fine.

Any suggestions?

OK a friend told me about od so I gave that a try:

```<code>$ echo -n "c&" | od -c
0000000    c 357 273 277 357 273 277   &
0000010
</code>```

So it seems the problem is cause by the byte sequence 357 273 277

Do we know what that sequence is?

答:
In the table below, the dots represent the breaks between octal digits, and the dashes represent the breaks between hex digits.

```<code>Octal:      3    5   7   |  2    7   3  |  2    7   7
Binary:    11.10-1.111   | 10.11-1.011  | 10.11-1.111
Hex:         E     F     |   B     B    |   B     F
</code>```

This has the correct form for valid UTF-8.  The first nybble shows two continuation bytes, and the next two bytes are indeed continuation bytes.  The second nybble of the first byte, and the last 6 bits of each of the next two bytes form the data for the Unicode character.

```<code>Unicode Binary:  1111 1110 11.11 1111
Unicode Hex:      F     E    F    F
</code>```

Therefore, the character is U+FEFF, which is the BOM (byte-order mark) or ZWNBSP (zero-width non-breaking space).  It is aconventional to encode the BOM in UTF-8 (it isn't needed); it is doubly aconventional to encode two of them in a row; and it is triply aconventional for the BOM not to be the first character in the UTF-8 code stream.

See the Unicode FAQ on BOM for more information.

Body:
There is an invisible character breaking my XML parser.

c&

The XML claims to be UTF-8, but when I try to use <c:import . . . charEncoding="UTF-8">

I get this friendly message:

ERROR: javax.servlet.jsp.JspException: java.io.CharConversionException: illegal utf8 encoding at (187)

I have been able to locate the source of the problem. It is an invisible character located between 'c' and '&'.

I would like to know more about this character, but it seems IntelliJ cannot show me hidden characters . . .

I think I saw a tool online that would convert Unicode characters to their octal values, but I can not find it again. If there is a tool I need to download that would be fine.

Any suggestions?

OK a friend told me about od so I gave that a try:

$ echo -n "c&" | od -c
0000000    c 357 273 277 357 273 277   &
0000010

So it seems the problem is cause by the byte sequence 357 273 277

Do we know what that sequence is?

Body:
In the table below, the dots represent the breaks between octal digits, and the dashes represent the breaks between hex digits.

Octal:      3    5   7   |  2    7   3  |  2    7   7
Binary:    11.10-1.111   | 10.11-1.011  | 10.11-1.111
Hex:         E     F     |   B     B    |   B     F

This has the correct form for valid UTF-8.  The first nybble shows two continuation bytes, and the next two bytes are indeed continuation bytes.  The second nybble of the first byte, and the last 6 bits of each of the next two bytes form the data for the Unicode character.

Unicode Binary:  1111 1110 11.11 1111
Unicode Hex:      F     E    F    F

Therefore, the character is U+FEFF, which is the BOM (byte-order mark) or ZWNBSP (zero-width non-breaking space).  It is aconventional to encode the BOM in UTF-8 (it isn't needed); it is doubly aconventional to encode two of them in a row; and it is triply aconventional for the BOM not to be the first character in the UTF-8 code stream.

See the Unicode FAQ on BOM for more information.


问:
water effect wavy loading animation in flutter
How I can create a water effect wavy loading animation in flutter ?

I want something like this

答:
AnimationController / CustomPainter solution

One solution it is to use a CustomPainter for drawing the wave, and creating a widget with a AnimationController. Then defining a Tween which is animated with the value provided from the animation controller. In the example below the tween is responsible for moving the wave leftward one wavelength as this would effectively loop the animation.

Example:

import 'package:flutter/material.dart';

void main() => runApp(const MyApp());

class MyApp extends StatelessWidget {
  const MyApp({Key? key}) : super(key: key);

  @override
  Widget build(BuildContext context) => MaterialApp(
    home: Scaffold(
      backgroundColor: Colors.white,
      body: Column(
        mainAxisAlignment: MainAxisAlignment.end,
        children: const [
          WaveBackground()
        ],
      )
    ),
  );
}

class WaveBackground extends StatefulWidget {
  const WaveBackground({Key? key}) : super(key: key);

  @override
  State<WaveBackground> createState() => _WaveBackgroundState();
}

class _WaveBackgroundState extends State<WaveBackground> with TickerProviderStateMixin {
  late AnimationController _controller;

  @override
  void initState() {
    super.initState();
    _controller = AnimationController(
      duration: const Duration(milliseconds: 3000),
      vsync: this,
    );
    _controller.repeat();
  }

  @override
  void dispose() {
    _controller.dispose();
    super.dispose();
  }

  Widget _buildAnimation(BuildContext context, Widget? widget) {
    return SizedBox(
      width: MediaQuery.of(context).size.width,
      height: MediaQuery.of(context).size.height,
      child: CustomPaint(
        painter: WavePainter(
            controller: _controller,
            waves: 2,
            waveAmplitude: 25
        ),
      ),
    );
  }
  @override
  Widget build(BuildContext context) {
    return AnimatedBuilder(
      animation: _controller,
      builder: _buildAnimation,
    );
  }
}

class WavePainter extends CustomPainter {
  late final Animation<double> position;
  final Animation<double> controller;

  /// Number of waves to paint.
  final int waves;

  /// How high the wave should be.
  final double waveAmplitude;
  int get waveSegments => 2 * waves - 1;

  WavePainter({required this.controller, required this.waves, required this.waveAmplitude}) {
    position = Tween(begin: 0.0, end: 1.0)
        .chain(CurveTween(curve: Curves.linear))
        .animate(controller);
  }

  void drawWave(Path path, int wave, size) {
    double waveWidth = size.width / waveSegments;
    double waveMinHeight = size.height / 2;

    double x1 = wave * waveWidth + waveWidth/2;
    // Minimum and maximum height points of the waves.
    double y1 = waveMinHeight + (wave.isOdd ? waveAmplitude: -waveAmplitude);

    double x2 = x1 + waveWidth/2;
    double y2 = waveMinHeight;

    path.quadraticBezierTo(x1, y1, x2, y2);
    if (wave <= waveSegments) {
      drawWave(path, wave + 1, size);
    }
  }

  @override
  void paint(Canvas canvas, Size size) {
    Paint paint = Paint()
      ..color = Colors.lightBlue
      ..style = PaintingStyle.fill;

    // Draw the waves
    Path path = Path()
      ..moveTo(0, size.height / 2);
    drawWave(path, 0, size);

    // Draw lines to the bottom corners of the size/screen with account for one extra wave.
    double waveWidth = (size.width / waveSegments) * 2;
    path..lineTo(size.width + waveWidth, size.height)
      ..lineTo(0, size.height)
      ..lineTo(0, size.height / 2)
      ..close();

    // Animate sideways one wave length, so it repeats cleanly.
    Path shiftedPath = path.shift(Offset(-position.value * waveWidth , 0));

    canvas.drawPath(shiftedPath, paint);
  }

  @override
  bool shouldRepaint(CustomPainter oldDelegate) => true;
}
```


问:
Unable To Scrape url from page using Python and BeautifulSoup. Any ideas?
As the title suggests. I'm playing around with a Twitter bot that scrapes rss feeds and tweets the title of the article and a link.

For some reason when I run the below code it runs without errors but doesn't retrieve the url link. Any suggestions are gratefully recieved.

```<code>from bs4 import BeautifulSoup
import requests

url = "https://www.kdnuggets.com/feed"
resp = requests.get(url)
soup = BeautifulSoup(resp.content)
items = soup.findAll('item')
item = items[1]

print(item.title.text)
print(item.link.text)
</code>```

The title prints fine but the link is nowhere to be found. For reference, below is a copy of the html that is returned for this item.

```<code><item>
<title>An Overview of Logistic Regression</title>
<link/>https://www.kdnuggets.com/2022/02/overview-logistic-regression.html
                                        <comments>https://www.kdnuggets.com/2022/02/overview-logistic-regression.html#disqus_thread</comments>
<dc:creator><![CDATA[Matt Mayo Editor]]></dc:creator>
<pubdate>Fri, 04 Feb 2022 13:00:11 +0000</pubdate>
<category><![CDATA[2022 Feb Tutorials, Overviews]]></category>
<category><![CDATA[Machine Learning]]></category>
<guid ispermalink="false">https://www.kdnuggets.com/?p=137943</guid>
<description><![CDATA[Logistic regression is an extension of linear regression to solve classification problems. Read more on the specifics of this algorithm here.]]></description>
<wfw:commentrss>https://www.kdnuggets.com/2022/02/overview-logistic-regression.html/feed</wfw:commentrss>
<slash:comments>0</slash:comments>
</item>
</code>```

Thanks in advance.

答:
Try this?

```<code>import httplib2
from bs4 import BeautifulSoup, SoupStrainer

http = httplib2.Http()
status, response = http.request('http://www.nytimes.com')

for link in BeautifulSoup(response, parse_only=SoupStrainer('a')):
    if link.has_attr('href'):
        print(link['href'])
</code>```

https://www.crummy.com/software/BeautifulSoup/bs4/doc/

from here

Body:
As the title suggests. I'm playing around with a Twitter bot that scrapes rss feeds and tweets the title of the article and a link.

For some reason when I run the below code it runs without errors but doesn't retrieve the url link. Any suggestions are gratefully recieved.

from bs4 import BeautifulSoup
import requests

url = "https://www.kdnuggets.com/feed"
resp = requests.get(url)
soup = BeautifulSoup(resp.content)
items = soup.findAll('item')
item = items[1]

print(item.title.text)
print(item.link.text)

The title prints fine but the link is nowhere to be found. For reference, below is a copy of the html that is returned for this item.

<item>
<title>An Overview of Logistic Regression</title>
<link/>https://www.kdnuggets.com/2022/02/overview-logistic-regression.html
                                        <comments>https://www.kdnuggets.com/2022/02/overview-logistic-regression.html#disqus_thread</comments>
<dc:creator><![CDATA[Matt Mayo Editor]]></dc:creator>
<pubdate>Fri, 04 Feb 2022 13:00:11 +0000</pubdate>
<category><![CDATA[2022 Feb Tutorials, Overviews]]></category>
<category><![CDATA[Machine Learning]]></category>
<guid ispermalink="false">https://www.kdnuggets.com/?p=137943</guid>
<description><![CDATA[Logistic regression is an extension of linear regression to solve classification problems. Read more on the specifics of this algorithm here.]]></description>
<wfw:commentrss>https://www.kdnuggets.com/2022/02/overview-logistic-regression.html/feed</wfw:commentrss>
<slash:comments>0</slash:comments>
</item>

Thanks in advance.

Body:
Try this?

import httplib2
from bs4 import BeautifulSoup, SoupStrainer

http = httplib2.Http()
status, response = http.request('http://www.nytimes.com')

for link in BeautifulSoup(response, parse_only=SoupStrainer('a')):
    if link.has_attr('href'):
        print(link['href'])

https://www.crummy.com/software/BeautifulSoup/bs4/doc/

from here


问:
Python/Tkinter : ModuleNotFoundError: No module named '_tkinter'
This is my first post StackOverflow, I will try to make it as correct and complete as possible if you have any tips to improve my post I will gladly accept it.

I'm having trouble running code written in Python that uses Tkinter.

I will try to describe in detail my actions to facilitate the identification of the error.

I started a course at Coursera on DSP (Digital Signal Processing) where it is suggested to install a tool written in python (and a little bit of C). I'm using Arch Linux.

link on Github:
sms-tools repo

Using pyenv/virtualenv/virtualenvwrapper I created an environment with Python 3.7.5, as recommended in the "How to use" section of the repository.

I installed the required libraries in my environment by pip:

%pip install ipython numpy matplotlib scipy cython

I compiled some C functions in the "/sms-tools/software/models/utilFunctions_C"

directory with the following command:

%python compileModule.py build_ext --inplace

Finally, I run the models GUI in the directory "/sms-tools/software/models_interface"

%python models_GUI.py

and I get the following message:

```<code>Traceback (most recent call last):
  File "models_GUI.py", line 6, in <module>
    from Tkinter import *   ## notice capitalized T in Tkinter
ModuleNotFoundError: No module named 'Tkinter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "models_GUI.py", line 9, in <module>
    from tkinter import *   ## notice lowercase 't' in tkinter here
  File "~/.pyenv/versions/3.7.5/lib/python3.7/tkinter/__init__.py", line 36, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ModuleNotFoundError: No module named '_tkinter'
</code>```

I will now describe some of my attempts to solve the problem:

Looking at Tkinter section in Python Wiki I tried installing Tcl and Tk.

%sudo pacman -S tk

but it was already installed. after that I tried installing with pip:

%pip install tk

and

%pip install tkinter

and the error remains the same.

I also tried to create a symlink with this code:
%ln -s /usr/lib/python3.8/lib-dynload/_tkinter.cpython-38-x86_64-linux-gnu.so _tkinter.cpython-38-x86_64-linux-gnu.so

the symlink was created in the following folders:

~/.ve/Coursera_DSP/lib/python3.7/lib-dynload

and

.pyenv/versions/3.7.5/lib/python3.7/lib-dynload

But I still get the same error.

I appreciate it if anyone has any suggestions and I apologize for the language errors since English is not my mother tongue.

After an incessant search on the internet, I believe the problem is related to pyenv and TCL/TK.
I don't understand much about the subject but I suspect that in the creation of the environment by virtualenv python has lost the connection with TCL/TK. Does that make any sense?

答:
mac Monterey (M1)

```<code>brew install python-tk
</code>```

python 3.9.10

Body:
This is my first post StackOverflow, I will try to make it as correct and complete as possible if you have any tips to improve my post I will gladly accept it.

I'm having trouble running code written in Python that uses Tkinter.

I will try to describe in detail my actions to facilitate the identification of the error.

I started a course at Coursera on DSP (Digital Signal Processing) where it is suggested to install a tool written in python (and a little bit of C). I'm using Arch Linux.

link on Github:
sms-tools repo

Using pyenv/virtualenv/virtualenvwrapper I created an environment with Python 3.7.5, as recommended in the "How to use" section of the repository.

I installed the required libraries in my environment by pip:

%pip install ipython numpy matplotlib scipy cython

I compiled some C functions in the "/sms-tools/software/models/utilFunctions_C"

directory with the following command:

%python compileModule.py build_ext --inplace

Finally, I run the models GUI in the directory "/sms-tools/software/models_interface"

%python models_GUI.py

and I get the following message:

Traceback (most recent call last):
  File "models_GUI.py", line 6, in <module>
    from Tkinter import *   ## notice capitalized T in Tkinter
ModuleNotFoundError: No module named 'Tkinter'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "models_GUI.py", line 9, in <module>
    from tkinter import *   ## notice lowercase 't' in tkinter here
  File "~/.pyenv/versions/3.7.5/lib/python3.7/tkinter/__init__.py", line 36, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ModuleNotFoundError: No module named '_tkinter'

I will now describe some of my attempts to solve the problem:

Looking at Tkinter section in Python Wiki I tried installing Tcl and Tk.

%sudo pacman -S tk

but it was already installed. after that I tried installing with pip:

%pip install tk

and

%pip install tkinter

and the error remains the same.

I also tried to create a symlink with this code:
%ln -s /usr/lib/python3.8/lib-dynload/_tkinter.cpython-38-x86_64-linux-gnu.so _tkinter.cpython-38-x86_64-linux-gnu.so

the symlink was created in the following folders:

~/.ve/Coursera_DSP/lib/python3.7/lib-dynload

and

.pyenv/versions/3.7.5/lib/python3.7/lib-dynload

But I still get the same error.

I appreciate it if anyone has any suggestions and I apologize for the language errors since English is not my mother tongue.

After an incessant search on the internet, I believe the problem is related to pyenv and TCL/TK.
I don't understand much about the subject but I suspect that in the creation of the environment by virtualenv python has lost the connection with TCL/TK. Does that make any sense?

Body:
mac Monterey (M1)

brew install python-tk

python 3.9.10


问:
Concurrency multithreading with requests
I am trying to figure out how to create concurrent requests with multithreading whilst using the requests library. I want to grab the links, and total pages from a url's POST request.

However, I am iterating over a very large loop so it will take an awfully long time. What I have tried doesn't seem to make the requests concurrent nor does it produce an output.

Here's what I have tried:

```<code>#smaller subset of my data

df = {'links': ['https://www.theparking.eu/used-cars/#!/used-cars/%3Fid_pays%3D6%26id_marque%3D138%26id_modele%3D687',
'https://www.theparking.eu/used-cars/#!/used-cars/%3Fid_pays%3D6%26id_marque%3D138%26id_modele%3D492',
'https://www.theparking.eu/used-cars/#!/used-cars/%3Fid_pays%3D6%26id_marque%3D138%26id_modele%3D499',
'https://www.theparking.eu/used-cars/#!/used-cars/%3Fid_pays%3D6%26id_marque%3D138%26id_modele%3D702',
'https://www.theparking.eu/used-cars/#!/used-cars/%3Fid_pays%3D6%26id_marque%3D138%26id_modele%3D6143'],
 'make': [138.0,138.0,138.0,138.0,138.0],
 'model': [687.0,492.0,499.0,702.0,6143.0],
 'country_id': [6.0,6.0,6.0,6.0,6.0]}

import requests
from bs4 import BeautifulSoup
from multiprocessing.pool import ThreadPool
import threading
import gc

def get_links(url):
    headers = {
        'authority': 'www.theparking.eu',
        'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"',
        'accept': '*/*',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'x-requested-with': 'XMLHttpRequest',
        'sec-ch-ua-mobile': '?0',
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36',
        'sec-ch-ua-platform': '"macOS"',
        'origin': 'https://www.theparking.eu',
        'sec-fetch-site': 'same-origin',
        'sec-fetch-mode': 'cors',
        'sec-fetch-dest': 'empty',
        'referer': 'https://www.theparking.eu/used-cars/used-cars/',
        'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    }
    formal_data = defaultdict(list)
    for id_ in df['country_id']:
        for make in df['make']:
            for model in df['model']:
                data = {
                    'ajax': '{"tab_id":"t0","cur_page":1,"cur_trie":"distance","query":"","critere":{"id_pays":[%s],"id_marque":[%s], "id_modele":[%s]},"sliders":{"prix":{"id":"#range_prix","face":"prix","max_counter":983615,"min":"1","max":"400000"},"km":{"id":"#range_km","face":"km","max_counter":1071165,"min":"1","max":"500000"},"millesime":{"id":"#range_millesime","face":"millesime","max_counter":1163610,"min":"1900","max":"2022"}},"req_num":1,"nb_results":"11795660","current_location_distance":-1,"logged_in":false}' % (round(id_), round(make), round(model)),
                    'tabs': '["t0"]'
                            }
                response = requests.post(url, headers=headers, data=data)
                test = json.loads(response.text)
                pages = round(int(test['context']['nb_results'])/27)
                if pages != 0:
                    formal_data['total_pages'].append(pages)
                    formal_data['links'].append(url)
                    print(f'You are on this link:{url}')
    return formal_data
threadLocal = threading.local()

with ThreadPool(8) as pool:
    urls = df['links']
    pool.map(get_links, urls)
    # must be done before terminate is explicitly or implicitly called on the pool:
    del threadLocal
    gc.collect()
</code>```

答:
Note that a more modern approach to use requests asynchronously is to use other libraries, like requests-threads.

With your approach, you connect to various URL in parallel, but sequentially to each URL. Consequentially, you might no be taking full advantage of multithreading. Indeed, for a single URL in df['links'], you get the same results as a single thread. The easiest way to circumvent this is to use itertools.product, that makes an iterator of what would be otherwise a nested loop.

import requests
from concurrent.futures import ThreadPoolExecutor as ThreadPool
from itertools import product

#   ... snipped df definition ...

def get_links(packed_pars):
    url, id_, make, model = packed_pars
    headers = {
        'authority': 'www.theparking.eu',
        'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97","Chromium";v="97"',
        'accept': '*/*',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'x-requested-with': 'XMLHttpRequest',
        'sec-ch-ua-mobile': '?0',
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36',
        'sec-ch-ua-platform': '"macOS"',
        'origin': 'https://www.theparking.eu',
        'sec-fetch-site': 'same-origin',
        'sec-fetch-mode': 'cors',
        'sec-fetch-dest': 'empty',
        'referer': 'https://www.theparking.eu/used-cars/used-cars/',
        'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    }
    data = {
        'ajax': '{"tab_id":"t0","cur_page":1,"cur_trie":"distance","query":"","critere":{"id_pays":[%s],"id_marque":[%s], "id_modele":[%s]},"sliders":{"prix":{"id":"#range_prix","face":"prix","max_counter":983615,"min":"1","max":"400000"},"km":{"id":"#range_km","face":"km","max_counter":1071165,"min":"1","max":"500000"},"millesime":{"id":"#range_millesime","face":"millesime","max_counter":1163610,"min":"1900","max":"2022"}},"req_num":1,"nb_results":"11795660","current_location_distance":-1,"logged_in":false}' % (round(id_), round(make), round(model)),
        'tabs': '["t0"]'
    }
    response = requests.post(url, headers=headers, data=data)
    test = response.json()
    pages = round(int(test['context']['nb_results'])/27)
    if pages != 0:
        print(f'You are on this link:{url}, with {pages} pages')
    else:
        print("no pages")
    return url, pages

with ThreadPool(8) as pool:
    rv = pool.map(get_links, product(df['links'], df['country_id'], df['make'],
                                     df['model']))
    # This converts rv to the dict of the original post:
    formal_data = dict()
    filtered_list = [(url, pages) for url, pages in rv if pages]
    if filtered_list:
        formal_data['links'], formal_data['total_pages'] = zip(*filtered_list)
    else:  # Protect against empty answers
        formal_data['links'], formal_data['total_pages'] = [], []
```

As for why this is not producing any output: at last with the data provided in the question, test['context']['nb_results'] is 0 every time. It is very likely that your query returns zero items each time even with the full dataset.

Some other comments:

The use of m̀ultiprocessing.pool.ThreadPool is not recommended: you should switch to concurrent.futures.ThreadPoolExecutor.

You are not using threadLocal at all: it can be removed. I do not know what you would use it for.

You are importing threading but not using it.

request responses have a json method that parses the text right away: there is no need for importing json in this case.

It is very likely that you want to ceil instead of round the number of pages.

Since you are waiting for I/O, it is OK to use more threads than available cores.

Body:
I am trying to figure out how to create concurrent requests with multithreading whilst using the requests library. I want to grab the links, and total pages from a url's POST request.

However, I am iterating over a very large loop so it will take an awfully long time. What I have tried doesn't seem to make the requests concurrent nor does it produce an output.

Here's what I have tried:

#smaller subset of my data

df = {'links': ['https://www.theparking.eu/used-cars/#!/used-cars/%3Fid_pays%3D6%26id_marque%3D138%26id_modele%3D687',
'https://www.theparking.eu/used-cars/#!/used-cars/%3Fid_pays%3D6%26id_marque%3D138%26id_modele%3D492',
'https://www.theparking.eu/used-cars/#!/used-cars/%3Fid_pays%3D6%26id_marque%3D138%26id_modele%3D499',
'https://www.theparking.eu/used-cars/#!/used-cars/%3Fid_pays%3D6%26id_marque%3D138%26id_modele%3D702',
'https://www.theparking.eu/used-cars/#!/used-cars/%3Fid_pays%3D6%26id_marque%3D138%26id_modele%3D6143'],
 'make': [138.0,138.0,138.0,138.0,138.0],
 'model': [687.0,492.0,499.0,702.0,6143.0],
 'country_id': [6.0,6.0,6.0,6.0,6.0]}

import requests
from bs4 import BeautifulSoup
from multiprocessing.pool import ThreadPool
import threading
import gc

def get_links(url):
    headers = {
        'authority': 'www.theparking.eu',
        'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"',
        'accept': '*/*',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'x-requested-with': 'XMLHttpRequest',
        'sec-ch-ua-mobile': '?0',
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36',
        'sec-ch-ua-platform': '"macOS"',
        'origin': 'https://www.theparking.eu',
        'sec-fetch-site': 'same-origin',
        'sec-fetch-mode': 'cors',
        'sec-fetch-dest': 'empty',
        'referer': 'https://www.theparking.eu/used-cars/used-cars/',
        'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    }
    formal_data = defaultdict(list)
    for id_ in df['country_id']:
        for make in df['make']:
            for model in df['model']:
                data = {
                    'ajax': '{"tab_id":"t0","cur_page":1,"cur_trie":"distance","query":"","critere":{"id_pays":[%s],"id_marque":[%s], "id_modele":[%s]},"sliders":{"prix":{"id":"#range_prix","face":"prix","max_counter":983615,"min":"1","max":"400000"},"km":{"id":"#range_km","face":"km","max_counter":1071165,"min":"1","max":"500000"},"millesime":{"id":"#range_millesime","face":"millesime","max_counter":1163610,"min":"1900","max":"2022"}},"req_num":1,"nb_results":"11795660","current_location_distance":-1,"logged_in":false}' % (round(id_), round(make), round(model)),
                    'tabs': '["t0"]'
                            }
                response = requests.post(url, headers=headers, data=data)
                test = json.loads(response.text)
                pages = round(int(test['context']['nb_results'])/27)
                if pages != 0:
                    formal_data['total_pages'].append(pages)
                    formal_data['links'].append(url)
                    print(f'You are on this link:{url}')
    return formal_data
threadLocal = threading.local()

with ThreadPool(8) as pool:
    urls = df['links']
    pool.map(get_links, urls)
    # must be done before terminate is explicitly or implicitly called on the pool:
    del threadLocal
    gc.collect()

Body:
Note that a more modern approach to use requests asynchronously is to use other libraries, like requests-threads.

With your approach, you connect to various URL in parallel, but sequentially to each URL. Consequentially, you might no be taking full advantage of multithreading. Indeed, for a single URL in df['links'], you get the same results as a single thread. The easiest way to circumvent this is to use itertools.product, that makes an iterator of what would be otherwise a nested loop.

import requests
from concurrent.futures import ThreadPoolExecutor as ThreadPool
from itertools import product

#   ... snipped df definition ...

def get_links(packed_pars):
    url, id_, make, model = packed_pars
    headers = {
        'authority': 'www.theparking.eu',
        'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97","Chromium";v="97"',
        'accept': '*/*',
        'content-type': 'application/x-www-form-urlencoded; charset=UTF-8',
        'x-requested-with': 'XMLHttpRequest',
        'sec-ch-ua-mobile': '?0',
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.99 Safari/537.36',
        'sec-ch-ua-platform': '"macOS"',
        'origin': 'https://www.theparking.eu',
        'sec-fetch-site': 'same-origin',
        'sec-fetch-mode': 'cors',
        'sec-fetch-dest': 'empty',
        'referer': 'https://www.theparking.eu/used-cars/used-cars/',
        'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    }
    data = {
        'ajax': '{"tab_id":"t0","cur_page":1,"cur_trie":"distance","query":"","critere":{"id_pays":[%s],"id_marque":[%s], "id_modele":[%s]},"sliders":{"prix":{"id":"#range_prix","face":"prix","max_counter":983615,"min":"1","max":"400000"},"km":{"id":"#range_km","face":"km","max_counter":1071165,"min":"1","max":"500000"},"millesime":{"id":"#range_millesime","face":"millesime","max_counter":1163610,"min":"1900","max":"2022"}},"req_num":1,"nb_results":"11795660","current_location_distance":-1,"logged_in":false}' % (round(id_), round(make), round(model)),
        'tabs': '["t0"]'
    }
    response = requests.post(url, headers=headers, data=data)
    test = response.json()
    pages = round(int(test['context']['nb_results'])/27)
    if pages != 0:
        print(f'You are on this link:{url}, with {pages} pages')
    else:
        print("no pages")
    return url, pages

with ThreadPool(8) as pool:
    rv = pool.map(get_links, product(df['links'], df['country_id'], df['make'],
                                     df['model']))
    # This converts rv to the dict of the original post:
    formal_data = dict()
    filtered_list = [(url, pages) for url, pages in rv if pages]
    if filtered_list:
        formal_data['links'], formal_data['total_pages'] = zip(*filtered_list)
    else:  # Protect against empty answers
        formal_data['links'], formal_data['total_pages'] = [], []

As for why this is not producing any output: at last with the data provided in the question, test['context']['nb_results'] is 0 every time. It is very likely that your query returns zero items each time even with the full dataset.

Some other comments:

The use of m̀ultiprocessing.pool.ThreadPool is not recommended: you should switch to concurrent.futures.ThreadPoolExecutor.

You are not using threadLocal at all: it can be removed. I do not know what you would use it for.

You are importing threading but not using it.

request responses have a json method that parses the text right away: there is no need for importing json in this case.

It is very likely that you want to ceil instead of round the number of pages.

Since you are waiting for I/O, it is OK to use more threads than available cores.


问:
Access phone files via .NET
I have a OnePlus attached to a Windows machine via USB.

I can see the file system objects on it via File Explorer.

The phone is exposed via two locations, the D: drive, which is mounted as a CDFS and contains some OnePlus drivers.

The other, labeled ONEPLUS A6013 which contains the OS file system.

How do I access the volume labeled ONEPLUS A6013 from .NET?

There's no drive for it returned by the DriveInfo.GetDrives() and I can't reference it by the path in File Explorer?

This PC\ONEPLUS A6013\Internal shared storage\Documents
```

答:
I've found a solution here https://stackoverflow.com/a/60478271/4412472

A MediaDevices nuget package is required.

The code may look like this:

```<code>static void EnumerateOnePlusFiles()
{
    var deviceName = "ONEPLUS A5010";

    var devices = MediaDevice.GetDevices();
    var device = devices.First(d => d.FriendlyName == deviceName);

    device.Connect();

    var photoDir = device.GetDirectoryInfo(@"\Internal shared storage\DCIM\Camera");
    var files = photoDir.EnumerateFiles("*.*", SearchOption.AllDirectories);

    foreach (var file in files)
    {
        Console.WriteLine(file.FullName);
    }

    device.Disconnect();
    device.Dispose();
}
</code>```

Title: Access phone files via .NET

Body:
I've found a solution here https://stackoverflow.com/a/60478271/4412472

A MediaDevices nuget package is required.

The code may look like this:

static void EnumerateOnePlusFiles()
{
    var deviceName = "ONEPLUS A5010";

    var devices = MediaDevice.GetDevices();
    var device = devices.First(d => d.FriendlyName == deviceName);

    device.Connect();

    var photoDir = device.GetDirectoryInfo(@"\Internal shared storage\DCIM\Camera");
    var files = photoDir.EnumerateFiles("*.*", SearchOption.AllDirectories);

    foreach (var file in files)
    {
        Console.WriteLine(file.FullName);
    }

    device.Disconnect();
    device.Dispose();
}


问:
Efficient memory bandwidth use for streaming
I have an application that streams through 250 MB of data, applying a simple and fast neural-net threshold function to the data chunks (which are just 2 32-bit words each). Based on the result of the (very simple) compute, the chunk is unpredictably pushed into one of 64 bins.  So it's one big stream in and 64 shorter (variable length) streams out.

This is repeated many times with different detection functions.

The compute is memory bandwidth limited. I can tell this because there's no speed change even if I use a discriminant function that's much more computationally intensive.

What is the best way to structure the writes of the new streams to optimize my memory bandwidth? I am especially thinking that understanding cache use and cache line size may play a big role in this. Imagine the worst case where I have my 64 output streams and by bad luck, many map to the same cache line. Then when I write the next 64 bits of data to a stream, the CPU has to flush out a stale cache line to main memory, and load in the proper cache line. Each of those uses 64 BYTES of bandwidth... so my bandwidth limited application may be wasting 95% of the memory bandwidth (in this hypothetical worst case, though).

It's hard to even try to measure the effect, so designing ways around it is even more vague. Or am I even chasing a ghost bottleneck that somehow the hardware optimizes better than I could?

I'm using Core II x86 processors if that makes any difference.

Edit: Here's some example code. It streams through an array and copies its elements to various output arrays picked pseudo-randomly.  Running the same program with different numbers of destination bins gives different runtimes, even though the same amount of computation and memory reads and writes were done:

2 output streams: 13 secs

8 output streams: 13 secs

32 output streams: 19 secs

128 output streams: 29 seconds

512 output streams: 47 seconds

The difference between using 512 versus 2 output streams is 4X, (probably??) caused by cache line eviction overhead.

```<code>#include <stdio.h>
#include <stdlib.h>
#include <ctime>

int main()
{
  const int size=1<<19;
  int streambits=3;
  int streamcount=1UL<<streambits; // # of output bins
  int *instore=(int *)malloc(size*sizeof(int));
  int **outstore=(int **)malloc(streamcount*sizeof(int *));
  int **out=(int **)malloc(streamcount*sizeof(int));
  unsigned int seed=0;

  for (int j=0; j<size; j++) instore[j]=j;

  for (int i=0; i< streamcount; ++i)
    outstore[i]=(int *)malloc(size*sizeof(int));

  int startTime=time(NULL);
  for (int k=0; k<10000; k++) {
    for (int i=0; i<streamcount; i++) out[i]=outstore[i];
    int *in=instore;

    for (int j=0; j<size/2; j++) {
      seed=seed*0x1234567+0x7162521;
      int bin=seed>>(32-streambits); // pseudorandom destination bin
      *(out[bin]++)=*(in++);
      *(out[bin]++)=*(in++);
    }

  }
  int endTime=time(NULL);
  printf("Eval time=%ld\n", endTime-startTime);
}
</code>```

答:
As you're writing to the 64 output bins, you'll be using many different memory locations. If the bins are filled essentially at random, it means that you'll sometimes have two bins that couls share the same cache line. Not a big problem; the Core 2 L1 cache is 8-way associative. That means you'd get a problem only with the 9th cache line. With just 65 live memory references at any time (1 read/64 write), 8-way associative is OK.

The L2 cache is apparently 12-way associative (3/6MB total, so 12 isn't that weird a number). So even if you'd have collisions in L1, chances are pretty good you're still not hitting main memory.

However, if you don't like this, re-arrange the bins in memory. Instead of stroing each bin sequentially, interleave them. For bin 0, store chunks 0-15 at offsets 0-63, but store chunks 16-31 at offset 8192-8255. For bin 1, store chunks 0-15 at offsets 64-127, etcetera. This takes just a few bit shifts and masks, but the result is that a pair of bins share 8 cache lines.

Another possible way to speed up your code in this case is SSE4, especially in x64 mode. You'd get 16 registers x 128 bits, and you can optimize the read (MOVNTDQA) to limit cache pollution. I'm not sure if that will help a lot with the read speed, though - I'd expect the Core2 prefetcher to catch this. Reading sequential integers is the most simple kind of access possible, any prefetcher should optimize that.

OwnerDisplayName: Arno Setagaya

Body:
I have an application that streams through 250 MB of data, applying a simple and fast neural-net threshold function to the data chunks (which are just 2 32-bit words each). Based on the result of the (very simple) compute, the chunk is unpredictably pushed into one of 64 bins.  So it's one big stream in and 64 shorter (variable length) streams out.

This is repeated many times with different detection functions.

The compute is memory bandwidth limited. I can tell this because there's no speed change even if I use a discriminant function that's much more computationally intensive.

What is the best way to structure the writes of the new streams to optimize my memory bandwidth? I am especially thinking that understanding cache use and cache line size may play a big role in this. Imagine the worst case where I have my 64 output streams and by bad luck, many map to the same cache line. Then when I write the next 64 bits of data to a stream, the CPU has to flush out a stale cache line to main memory, and load in the proper cache line. Each of those uses 64 BYTES of bandwidth... so my bandwidth limited application may be wasting 95% of the memory bandwidth (in this hypothetical worst case, though).

It's hard to even try to measure the effect, so designing ways around it is even more vague. Or am I even chasing a ghost bottleneck that somehow the hardware optimizes better than I could?

I'm using Core II x86 processors if that makes any difference.

Edit: Here's some example code. It streams through an array and copies its elements to various output arrays picked pseudo-randomly.  Running the same program with different numbers of destination bins gives different runtimes, even though the same amount of computation and memory reads and writes were done:

2 output streams: 13 secs

8 output streams: 13 secs

32 output streams: 19 secs

128 output streams: 29 seconds

512 output streams: 47 seconds

The difference between using 512 versus 2 output streams is 4X, (probably??) caused by cache line eviction overhead.

#include <stdio.h>
#include <stdlib.h>
#include <ctime>

int main()
{
  const int size=1<<19;
  int streambits=3;
  int streamcount=1UL<<streambits; // # of output bins
  int *instore=(int *)malloc(size*sizeof(int));
  int **outstore=(int **)malloc(streamcount*sizeof(int *));
  int **out=(int **)malloc(streamcount*sizeof(int));
  unsigned int seed=0;

  for (int j=0; j<size; j++) instore[j]=j;

  for (int i=0; i< streamcount; ++i)
    outstore[i]=(int *)malloc(size*sizeof(int));

  int startTime=time(NULL);
  for (int k=0; k<10000; k++) {
    for (int i=0; i<streamcount; i++) out[i]=outstore[i];
    int *in=instore;

    for (int j=0; j<size/2; j++) {
      seed=seed*0x1234567+0x7162521;
      int bin=seed>>(32-streambits); // pseudorandom destination bin
      *(out[bin]++)=*(in++);
      *(out[bin]++)=*(in++);
    }

  }
  int endTime=time(NULL);
  printf("Eval time=%ld\n", endTime-startTime);
}

LastEditorDisplayName: Roger Pate

OwnerDisplayName: MSalters


问:
Creating an Image from a DIV
I've scoured Google and Stackoverflow and can't seem to come up with a definitive answer.

I'm wanting to create an image from a particular bit of content. So I've got a div which I've added images, text, background colours etc. to and I'm wanting to save out that DIV as an Image. I've read on GD and how to create each individual bit, like creating the canvas first, filling it with a colour, rendering the fonts etc...
What I'm wondering, is there not an easier way? It seems like a straight forward task but I appreciate it's not. Are there any libraries that you could recommend that will aid this? Rather than having to go through rendering each image, each bit of text and each colour on the canvas..

Thanks

答:
PHP has no idea what your page looks like.

That's stuff a browser / user-agent does (rendering HTML).

Check out: Website screenshots using PHP

Title: Creating an Image from a DIV


问:
Hide and Replace Div after a certain time
I am displaying a table in angular on the components html component. I noticed that the pull of the data and filtering the data takes some time and the table doesn't load right away. so I decided to add a loading spinner image however I want the loading spinner to disappear after about 5 secs and for the table to appear. I wrote some code to have the spinner disappear after 5 secs however I am running into the trouble of the loading spinner div still being on the page and having to scroll down to see the table in the overflow box. is there a way to make the loading spinner div complete disappear or maybe even swap the table div for the spinner div?

component.html

```<code><!-- Displaying Data from NCEI Data Pull -->
<h1>Preview Data</h1>
<div id="scrollable-area">

    <!-- Display Loading Circle -->
    <div class="loading hide">
        <div class="circle"></div>
    </div>

    <table class="table">

        <!-- Display Headers -->
        <th *ngFor = "let row of headers">
            {{row}}
        </th>

        <!-- Display Data -->
        <tr *ngFor = "let row of dataObj">
            <td *ngFor = "let column of row" class="rows">{{ column }}</td>
        </tr>
    </table>
</div>
</code>```

component.css

```<code>h1{
      text-align: center;
      color:rgb(0, 0, 0);
      padding: 15px 32px;
      text-decoration: none;
      margin: 4px 2px;

}

table{
    font-family: 'Trebuchet MS', Arial, Arial, Helvetica, sans-serif;
    border-collapse: separate;
    width: 100%;
    border-radius:6px;
}

td{
  padding: 8px;
  align-content: center;
  text-align: center;
  background-color: white;
}
th{
  padding: 8px;
  align-content: center;
  text-align: center;
  background-color: white;
  color: black;
  position: sticky;
  top: 0;
}

#scrollable-area {
  margin: auto;
  width: 80%;
  height: 400px;
  border: 2px solid #ccc;
  overflow-y: scroll;
  border-radius: 20px;
  border-left:solid black 9px;
  border-top:solid black 9px;
  border-bottom: solid black 1px;
  border-right: solid black 1px;
  background-color: rgb(172, 169, 169);
}
#scrollable-area::-webkit-scrollbar {
  width: 9px;               /* width of the entire scrollbar */
  height: 9px;
}
#scrollable-area::-webkit-scrollbar-corner{
  background: black;
  border-bottom-right-radius: 20px;
}
#scrollable-area::-webkit-scrollbar-track {
  background: black;
  border-radius: 50px;
  border: double black 1px;
}

#scrollable-area::-webkit-scrollbar-thumb {
  background-color: rgb(0, 0, 0);
  border-radius: 20px;
  border: 3px solid rgb(255, 255, 255);

}

.loading{
  padding: 8px;
  display: flex;
  justify-content: center;
  align-items: center;
  height: 100%;
}

.circle{
  content: "";
  width: 20px;
  height: 20px;
  border: 5px solid #dddddd;
  border-top-color: rgb(88, 118, 86);
  border-radius: 50%;
  animation: loading 1.5s ease infinite;
}

@keyframes loading{
  to {
    transform: rotate(1turn)}
}

.hide {
  -webkit-animation: seconds 1.0s forwards;
  -webkit-animation-iteration-count: 1;
  -webkit-animation-delay: 5s;
  animation: seconds 1.0s forwards;
  animation-iteration-count: 1;
  animation-delay: 5s;
  position: relative;
}
@-webkit-keyframes seconds {
  0% {
    opacity: 1;
  }
  100% {
    opacity: 0;
    left: -9999px;
  }
}
@keyframes seconds {
  0% {
    opacity: 1;
  }
  100% {
    opacity: 0;
    left: -9999px;
  }
}
</code>```

any advice would be appreciated.

Three screenshots of progress
[1]: https://i.stack.imgur.com/0iqeN.png
[2]: https://i.stack.imgur.com/QPgm1.png
[3]: https://i.stack.imgur.com/0u8gB.png

答:
You can use the *ngIf directive to display and hide the loader on the basis of your data received or not. I suppose you get your response in dataObj? You can either create a flag isLoading or just use your dataObj to check if data has arrived or not.

            {{row}}

            {{ column }}

```
<p>Alternatively, as I mentioned that you can create a flag and use that to determine when you want to show the loader or not. So with a flag, it would look something like this. Firstly let's create the flag in your <code>.ts</code> component:</p>
<pre class="lang-js prettyprint-override"><code>//Either initialize it with true or set its value to true when you begin the get call
isLoading: boolean = true;

//Wherever you getting the response just set isLoading to false
this.isLoading = false
</code>```

and your HTML would look like this

            {{row}}

            {{ column }}

```

Body:
I am displaying a table in angular on the components html component. I noticed that the pull of the data and filtering the data takes some time and the table doesn't load right away. so I decided to add a loading spinner image however I want the loading spinner to disappear after about 5 secs and for the table to appear. I wrote some code to have the spinner disappear after 5 secs however I am running into the trouble of the loading spinner div still being on the page and having to scroll down to see the table in the overflow box. is there a way to make the loading spinner div complete disappear or maybe even swap the table div for the spinner div?

component.html

Preview Data

            {{row}}

            {{ column }}

component.css

h1{
      text-align: center;
      color:rgb(0, 0, 0);
      padding: 15px 32px;
      text-decoration: none;
      margin: 4px 2px;

}

table{
    font-family: 'Trebuchet MS', Arial, Arial, Helvetica, sans-serif;
    border-collapse: separate;
    width: 100%;
    border-radius:6px;
}

td{
  padding: 8px;
  align-content: center;
  text-align: center;
  background-color: white;
}
th{
  padding: 8px;
  align-content: center;
  text-align: center;
  background-color: white;
  color: black;
  position: sticky;
  top: 0;
}

#scrollable-area {
  margin: auto;
  width: 80%;
  height: 400px;
  border: 2px solid #ccc;
  overflow-y: scroll;
  border-radius: 20px;
  border-left:solid black 9px;
  border-top:solid black 9px;
  border-bottom: solid black 1px;
  border-right: solid black 1px;
  background-color: rgb(172, 169, 169);
}
#scrollable-area::-webkit-scrollbar {
  width: 9px;               /* width of the entire scrollbar */
  height: 9px;
}
#scrollable-area::-webkit-scrollbar-corner{
  background: black;
  border-bottom-right-radius: 20px;
}
#scrollable-area::-webkit-scrollbar-track {
  background: black;
  border-radius: 50px;
  border: double black 1px;
}

#scrollable-area::-webkit-scrollbar-thumb {
  background-color: rgb(0, 0, 0);
  border-radius: 20px;
  border: 3px solid rgb(255, 255, 255);

}

.loading{
  padding: 8px;
  display: flex;
  justify-content: center;
  align-items: center;
  height: 100%;
}

.circle{
  content: "";
  width: 20px;
  height: 20px;
  border: 5px solid #dddddd;
  border-top-color: rgb(88, 118, 86);
  border-radius: 50%;
  animation: loading 1.5s ease infinite;
}

@keyframes loading{
  to {
    transform: rotate(1turn)}
}

.hide {
  -webkit-animation: seconds 1.0s forwards;
  -webkit-animation-iteration-count: 1;
  -webkit-animation-delay: 5s;
  animation: seconds 1.0s forwards;
  animation-iteration-count: 1;
  animation-delay: 5s;
  position: relative;
}
@-webkit-keyframes seconds {
  0% {
    opacity: 1;
  }
  100% {
    opacity: 0;
    left: -9999px;
  }
}
@keyframes seconds {
  0% {
    opacity: 1;
  }
  100% {
    opacity: 0;
    left: -9999px;
  }
}

any advice would be appreciated.

Three screenshots of progress
[1]: https://i.stack.imgur.com/0iqeN.png
[2]: https://i.stack.imgur.com/QPgm1.png
[3]: https://i.stack.imgur.com/0u8gB.png

Body:
You can use the *ngIf directive to display and hide the loader on the basis of your data received or not. I suppose you get your response in dataObj? You can either create a flag isLoading or just use your dataObj to check if data has arrived or not.

    <div *ngIf="!dataObj">
        <div class="circle"></div>
    </div>
    <table class="table" *ngIf="dataObj">
        <!-- Display Headers -->
        <th *ngFor = "let row of headers">
            {{row}}
        </th>
        <!-- Display Data -->
        <tr *ngFor = "let row of dataObj">
            <td *ngFor = "let column of row" class="rows">{{ column }}</td>
        </tr>
    </table>

Alternatively, as I mentioned that you can create a flag and use that to determine when you want to show the loader or not. So with a flag, it would look something like this. Firstly let's create the flag in your .ts component:

//Either initialize it with true or set its value to true when you begin the get call
isLoading: boolean = true;

//Wherever you getting the response just set isLoading to false
this.isLoading = false

and your HTML would look like this

    <div *ngIf="isLoading">
        <div class="circle"></div>
    </div>
    <table class="table" *ngIf="!isLoading">
        <!-- Display Headers -->
        <th *ngFor = "let row of headers">
            {{row}}
        </th>
        <!-- Display Data -->
        <tr *ngFor = "let row of dataObj">
            <td *ngFor = "let column of row" class="rows">{{ column }}</td>
        </tr>
    </table>


答:
You aren't getting item.link.text because it's empty - the link element is

```<code><link/>
</code>```

Try this method to get the text:

```<code>>>> item.link
<link/>
>>> item.link.findNext().text
'https://www.kdnuggets.com/2022/02/overview-logistic-regression.html#disqus_thread'
</code>```

You'll still need to strip off the #.... but that's straightforward to do

Title: Unable To Scrape url from page using Python and BeautifulSoup. Any ideas?

Body:
You aren't getting item.link.text because it's empty - the link element is

<link/>

Try this method to get the text:

>>> item.link
<link/>
>>> item.link.findNext().text
'https://www.kdnuggets.com/2022/02/overview-logistic-regression.html#disqus_thread'

You'll still need to strip off the #.... but that's straightforward to do


问:
for loop IR generation in Bison
Consider the following grammar rule:

```<code>forstmt: for openparentheses nexpr semicolon expr semicolon nexpr closeparentheses stmt {}

nexpr: expr { }
    | %empty { }

expr: .
      .
      .
      // something huge
      .
      .
      .
</code>```

It is a parser rul for a for loop like below (a usual C like for loop):

```<code>for(i=0; i<10; i++){
    Print("hello world");
}
</code>```

I got to generate IR for this C like for loop (forstmt).

The IR for expr is already written.

The point is that the last nexpr's IR should go after the stmt.

I know about mid-rule actions and I thought that somehow I could solve this using a stack, but my thoughts didn't lead to any conclusions.

Precisely is there a way to stop bison from generating IR for the last nexpr and make it generate after the stmt ?

In other words, how to make all action of the last nexpr go after the stmt ?

Has anyone had a problem like this ?

答:
It must be done by hand!

Bison doesn't have or even it shouldn't have anything for that!

My solution was to some how set a bit somewhere to hold the generated IR in code generator and the release it after the for loop's stmt.

Title: for loop IR generation in Bison

Body:
Consider the following grammar rule:

forstmt: for openparentheses nexpr semicolon expr semicolon nexpr closeparentheses stmt {}

nexpr: expr { }
    | %empty { }

expr: .
      .
      .
      // something huge
      .
      .
      .

It is a parser rul for a for loop like below (a usual C like for loop):

for(i=0; i<10; i++){
    Print("hello world");
}

I got to generate IR for this C like for loop (forstmt).

The IR for expr is already written.

The point is that the last nexpr's IR should go after the stmt.

I know about mid-rule actions and I thought that somehow I could solve this using a stack, but my thoughts didn't lead to any conclusions.

Precisely is there a way to stop bison from generating IR for the last nexpr and make it generate after the stmt ?

In other words, how to make all action of the last nexpr go after the stmt ?

Has anyone had a problem like this ?


问:
PHP8.1 - how do I get imageavif() to work?
I recently upgraded my PHP version to 8.1-fpm and also upgraded Debian to Bullseye. I read somewhere that Bullseye comes with AVIF support but when I try to use imageavif(), I get the following error:

```<code>PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function imageavif()
</code>```

I ran "php -i | grep AVIF" on my server and it returns:

```<code>ImageMagick supported formats => 3FR, 3G2, 3GP, AAI, AI, APNG, ART, ARW, AVI, AVIF, AVS, BGR, BGRA, BGRO, BIE, BMP, BMP2, BMP3, BRF, CAL, CALS, CANVAS, CAPTION, CIN, CIP, CLIP, CMYK, CMYKA, CR2, CR3, CRW, CUR, CUT, DATA, DCM, DCR, DCX, DDS, DFONT, DNG, DPX, DXT1, DXT5, EPDF, EPI, EPS, EPS2, EPS3, EPSF, EPSI, EPT, EPT2, EPT3, ERF, FAX, FILE, FITS, FRACTAL, FTP, FTS, G3, G4, GIF, GIF87, GRADIENT, GRAY, GRAYA, GROUP4, H, HALD, HDR, HEIC, HISTOGRAM, HRZ, HTM, HTML, HTTP, HTTPS, ICB, ICO, ICON, IIQ, INFO, INLINE, IPL, ISOBRL, ISOBRL6, J2C, J2K, JBG, JBIG, JNG, JNX, JP2, JPC, JPE, JPEG, JPG, JPM, JPS, JPT, JSON, K25, KDC, LABEL, M2V, M4V, MAC, MAGICK, MAP, MASK, MAT, MATTE, MEF, MIFF, MKV, MNG, MONO, MOV, MP4, MPC, MPG, MRW, MSL, MTV, MVG, NEF, NRW, NULL, ORF, OTB, OTF, PAL, PALM, PAM, PATTERN, PBM, PCD, PCDS, PCL, PCT, PCX, PDB, PDF, PDFA, PEF, PES, PFA, PFB, PFM, PGM, PGX, PICON, PICT, PIX, PJPEG, PLASMA, PNG, PNG00, PNG24, PNG32, PNG48, PNG64, PNG8, PNM, POCKETMOD, PPM, PREVIEW, PS, PS2, PS3, PSB, PSD, PTIF, PWP, RADIAL-GRADIENT, RAF, RAS, RAW, RGB, RGBA, RGBO, RGF, RLA, RLE, RMF, RW2, SCR, SCT, SFW, SGI, SHTML, SIX, SIXEL, SPARSE-COLOR, SR2, SRF, STEGANO, SUN, TEXT, TGA, THUMBNAIL, TIFF, TIFF64, TILE, TIM, TTC, TTF, TXT, UBRL, UBRL6, UIL, UYVY, VDA, VICAR, VID, VIDEO, VIFF, VIPS, VST, WBMP, WEBM, WEBP, WMV, WPG, X, X3F, XBM, XC, XCF, XPM, XPS, XV, XWD, YCbCr, YCbCrA, YUV
</code>```

How do I get imageavif() to work? Am I missing something?

Edit: I ran the function "print_r(gd_info());" and this is the resulting array:

```<code>Array
(
    [GD Version] => 2.3.0
    [FreeType Support] => 1
    [FreeType Linkage] => with freetype
    [GIF Read Support] => 1
    [GIF Create Support] => 1
    [JPEG Support] => 1
    [PNG Support] => 1
    [WBMP Support] => 1
    [XPM Support] => 1
    [XBM Support] => 1
    [WebP Support] => 1
    [BMP Support] => 1
    [AVIF Support] =>
    [TGA Read Support] => 1
    [JIS-mapped Japanese Font Support] =>
)
</code>```

Seems like I have GD but its AVIF support isn't set. How do I get it to support AVIFs?

Also, I am currently using imagewebp() and that function works just fine.

Edit: Thanks to Nico Haase for pointing me to the GitHub issue for Ondřej Surý's PHP repository - https://github.com/oerdnj/deb.sury.org/issues/1641

In the meantime, is there anything I can do to get imageavif() to work that wouldn't be too excessive or is it best to just wait?

答:
GD extension depends on the libavif package to provide the underlying AVIF codecs. GD extension requires libavif version 0.8.2 or higher.

Ubuntu/Debian

```<code>apt install libavif-dev
</code>```

RHEL/Fedora

```<code>dnf install libavif-devel
</code>```

Once the dependencies are installed, PHP can be compiled with AVIF support with the new --with-avif flag in the ./configure script.

```<code>./buildconf --force
./configure --enable-gd --with-avif
</code>```

Note that the ./configure command above is just an example; For production systems, make sure to enable other extensions as well.

Source: https://php.watch/versions/8.1/gd-avif

Body:
I recently upgraded my PHP version to 8.1-fpm and also upgraded Debian to Bullseye. I read somewhere that Bullseye comes with AVIF support but when I try to use imageavif(), I get the following error:

PHP message: PHP Fatal error:  Uncaught Error: Call to undefined function imageavif()

I ran "php -i | grep AVIF" on my server and it returns:

ImageMagick supported formats => 3FR, 3G2, 3GP, AAI, AI, APNG, ART, ARW, AVI, AVIF, AVS, BGR, BGRA, BGRO, BIE, BMP, BMP2, BMP3, BRF, CAL, CALS, CANVAS, CAPTION, CIN, CIP, CLIP, CMYK, CMYKA, CR2, CR3, CRW, CUR, CUT, DATA, DCM, DCR, DCX, DDS, DFONT, DNG, DPX, DXT1, DXT5, EPDF, EPI, EPS, EPS2, EPS3, EPSF, EPSI, EPT, EPT2, EPT3, ERF, FAX, FILE, FITS, FRACTAL, FTP, FTS, G3, G4, GIF, GIF87, GRADIENT, GRAY, GRAYA, GROUP4, H, HALD, HDR, HEIC, HISTOGRAM, HRZ, HTM, HTML, HTTP, HTTPS, ICB, ICO, ICON, IIQ, INFO, INLINE, IPL, ISOBRL, ISOBRL6, J2C, J2K, JBG, JBIG, JNG, JNX, JP2, JPC, JPE, JPEG, JPG, JPM, JPS, JPT, JSON, K25, KDC, LABEL, M2V, M4V, MAC, MAGICK, MAP, MASK, MAT, MATTE, MEF, MIFF, MKV, MNG, MONO, MOV, MP4, MPC, MPG, MRW, MSL, MTV, MVG, NEF, NRW, NULL, ORF, OTB, OTF, PAL, PALM, PAM, PATTERN, PBM, PCD, PCDS, PCL, PCT, PCX, PDB, PDF, PDFA, PEF, PES, PFA, PFB, PFM, PGM, PGX, PICON, PICT, PIX, PJPEG, PLASMA, PNG, PNG00, PNG24, PNG32, PNG48, PNG64, PNG8, PNM, POCKETMOD, PPM, PREVIEW, PS, PS2, PS3, PSB, PSD, PTIF, PWP, RADIAL-GRADIENT, RAF, RAS, RAW, RGB, RGBA, RGBO, RGF, RLA, RLE, RMF, RW2, SCR, SCT, SFW, SGI, SHTML, SIX, SIXEL, SPARSE-COLOR, SR2, SRF, STEGANO, SUN, TEXT, TGA, THUMBNAIL, TIFF, TIFF64, TILE, TIM, TTC, TTF, TXT, UBRL, UBRL6, UIL, UYVY, VDA, VICAR, VID, VIDEO, VIFF, VIPS, VST, WBMP, WEBM, WEBP, WMV, WPG, X, X3F, XBM, XC, XCF, XPM, XPS, XV, XWD, YCbCr, YCbCrA, YUV

How do I get imageavif() to work? Am I missing something?

Edit: I ran the function "print_r(gd_info());" and this is the resulting array:

Array
(
    [GD Version] => 2.3.0
    [FreeType Support] => 1
    [FreeType Linkage] => with freetype
    [GIF Read Support] => 1
    [GIF Create Support] => 1
    [JPEG Support] => 1
    [PNG Support] => 1
    [WBMP Support] => 1
    [XPM Support] => 1
    [XBM Support] => 1
    [WebP Support] => 1
    [BMP Support] => 1
    [AVIF Support] =>
    [TGA Read Support] => 1
    [JIS-mapped Japanese Font Support] =>
)

Seems like I have GD but its AVIF support isn't set. How do I get it to support AVIFs?

Also, I am currently using imagewebp() and that function works just fine.

Edit: Thanks to Nico Haase for pointing me to the GitHub issue for Ondřej Surý's PHP repository - https://github.com/oerdnj/deb.sury.org/issues/1641

In the meantime, is there anything I can do to get imageavif() to work that wouldn't be too excessive or is it best to just wait?

Body:
GD extension depends on the libavif package to provide the underlying AVIF codecs. GD extension requires libavif version 0.8.2 or higher.

Ubuntu/Debian

apt install libavif-dev

RHEL/Fedora

dnf install libavif-devel

Once the dependencies are installed, PHP can be compiled with AVIF support with the new --with-avif flag in the ./configure script.

./buildconf --force
./configure --enable-gd --with-avif

Note that the ./configure command above is just an example; For production systems, make sure to enable other extensions as well.

Source: https://php.watch/versions/8.1/gd-avif


问:
Python Pandas - how to add columns of filtered sum and calculate percentage weight
I would like to add two columns to a pandas df to show daily totals and weights. Existing df:

Date
Name
Value

2022-2-1
Apple
5

2022-2-1
Pear
3

2022-2-2
Apple
4

2022-2-2
Pear
4

2022-2-2
Banana
2

2022-2-3
Apple
5

2022-2-3
Banana
3

2022-2-4
Apple
4

2022-2-4
Banana
2

2022-2-4
Papaya
9

Desired output:

Date
Name
Value
Daily Total
Percentage Weight

2022-2-1
Apple
5
8
62.5%

2022-2-1
Pear
3
8
37.5%

2022-2-2
Apple
4
10
40%

2022-2-2
Pear
4
10
40%

2022-2-2
Banana
2
10
20%

2022-2-3
Apple
5
8
62.5%

2022-2-3
Banana
3
8
37.5%

2022-2-4
Apple
4
15
26.7%

2022-2-4
Banana
2
15
13.3%

2022-2-4
Papaya
9
15
60%

Appreciate any ideas/suggestions.

答:
Use groupby + transform('sum'):

```<code>df['Daily Total'] = df.groupby('Date')['Value'].transform('sum')
df['Percentage Weight'] = (df['Value'] / df['Daily Total'] * 100).round(1).map('{}%'.format)
</code>```

Output:

```<code>>>> df
       Date    Name  Value  Daily Total Percentage Weight
0  2022-2-1   Apple      5            8             62.5%
1  2022-2-1    Pear      3            8             37.5%
2  2022-2-2   Apple      4           10             40.0%
3  2022-2-2    Pear      4           10             40.0%
4  2022-2-2  Banana      2           10             20.0%
5  2022-2-3   Apple      5            8             62.5%
6  2022-2-3  Banana      3            8             37.5%
7  2022-2-4   Apple      4           15             26.7%
8  2022-2-4  Banana      2           15             13.3%
9  2022-2-4  Papaya      9           15             60.0%
</code>```

One-liner:

```<code>df = df.assign(DailyTotal=df.groupby('Date')['Value'].transform('sum')).pipe(lambda x: x.assign(PercentageWeight=df['Value'].div(df['Daily Total']).mul(100).round(1).map('{}%'.format)))
</code>```

Body:
Use groupby + transform('sum'):

df['Daily Total'] = df.groupby('Date')['Value'].transform('sum')
df['Percentage Weight'] = (df['Value'] / df['Daily Total'] * 100).round(1).map('{}%'.format)

Output:

>>> df
       Date    Name  Value  Daily Total Percentage Weight
0  2022-2-1   Apple      5            8             62.5%
1  2022-2-1    Pear      3            8             37.5%
2  2022-2-2   Apple      4           10             40.0%
3  2022-2-2    Pear      4           10             40.0%
4  2022-2-2  Banana      2           10             20.0%
5  2022-2-3   Apple      5            8             62.5%
6  2022-2-3  Banana      3            8             37.5%
7  2022-2-4   Apple      4           15             26.7%
8  2022-2-4  Banana      2           15             13.3%
9  2022-2-4  Papaya      9           15             60.0%

One-liner:

df = df.assign(DailyTotal=df.groupby('Date')['Value'].transform('sum')).pipe(lambda x: x.assign(PercentageWeight=df['Value'].div(df['Daily Total']).mul(100).round(1).map('{}%'.format)))


问:
passing variables from PHP page to jQuery modal window
Ok, so as the title implies, I'm opening a modal dialog and need to pass it various variables, one of which is the url of the page I'm starting from. Now this wouldn't be a problem if the jQuery that creates the dialog was in script tags in the head of a page as I could use PHP to echo the info and append it to the URL as a GET string, but the code is actually in an external .js file which is referenced in the head.

script.js:

```<code>var dialogVariables = {
    modal: true,
    autoOpen: false,
    width: 500,
    buttons: {
        Cancel: function() {
            $(this).dialog('close');
        } //end cancel button
    },//end button
    open: function() {
        $("#reportDialog").load("/report.php");}
    };
$("#reportDialog").dialog(dialogVariables);
</code>```

Any ideas as to how I can pass the information across?

答: Output the data into hidden form fields. Use the values in those form fields to construct the url.</p>

Body:
Ok, so as the title implies, I'm opening a modal dialog and need to pass it various variables, one of which is the url of the page I'm starting from. Now this wouldn't be a problem if the jQuery that creates the dialog was in script tags in the head of a page as I could use PHP to echo the info and append it to the URL as a GET string, but the code is actually in an external .js file which is referenced in the head.

script.js:

var dialogVariables = {
    modal: true,
    autoOpen: false,
    width: 500,
    buttons: {
        Cancel: function() {
            $(this).dialog('close');
        } //end cancel button
    },//end button
    open: function() {
        $("#reportDialog").load("/report.php");}
    };
$("#reportDialog").dialog(dialogVariables);

Any ideas as to how I can pass the information across?


问:
How to create a new character column removing text from another column in dataframe in r
I have the following data frame in R

```<code>East    24  Consumer    7/18/2018 0:00  Second Class    Pennsylvania    Chairs  USD71.37
East    29  Consumer    9/21/2016 0:00  Standard Class  Pennsylvania    Binders USD9.61
East    33  Consumer    9/21/2016 0:00  Standard Class  Pennsylvania    Binders USD6.85
East    199 Corporate   7/13/2018 0:00  Standard Class  Pennsylvania    Binders USD2.94
East    216 Corporate   1/9/2016 0:00   Standard Class  Ohio    Machines    USD1188
East    222 Consumer    12/27/2016 0:00 First Class Ohio    Binders USD24
East    224 Consumer    12/27/2016 0:00 First Class Ohio    Machines    USD252
East    331 Consumer    9/1/2017 0:00   First Class Pennsylvania    Binders USD28
East    333 Consumer    4/29/2017 0:00  Second Class    Pennsylvania    Binders USD40
East    334 Consumer    4/29/2017 0:00  Second Class    Pennsylvania    Binders USD8
</code>```

The last column is sales which is a character - my problem is I want to create a new column with sales numbers that has just the numbers  - please help if able!!!!!

答:
We could use parse_number function from readr package:

Here V10 is your last column name:

```<code>library(dplyr)
library(readr)

df %>%
  mutate(newCol = parse_number(V10))
</code>```
```<code>     V1  V2        V3         V4   V5       V6    V7           V8       V9      V10  newCol
1  East  24  Consumer  7/18/2018 0:00   Second Class Pennsylvania   Chairs USD71.37   71.37
2  East  29  Consumer  9/21/2016 0:00 Standard Class Pennsylvania  Binders  USD9.61    9.61
3  East  33  Consumer  9/21/2016 0:00 Standard Class Pennsylvania  Binders  USD6.85    6.85
4  East 199 Corporate  7/13/2018 0:00 Standard Class Pennsylvania  Binders  USD2.94    2.94
5  East 216 Corporate   1/9/2016 0:00 Standard Class         Ohio Machines  USD1188 1188.00
6  East 222  Consumer 12/27/2016 0:00    First Class         Ohio  Binders    USD24   24.00
7  East 224  Consumer 12/27/2016 0:00    First Class         Ohio Machines   USD252  252.00
8  East 331  Consumer   9/1/2017 0:00    First Class Pennsylvania  Binders    USD28   28.00
9  East 333  Consumer  4/29/2017 0:00   Second Class Pennsylvania  Binders    USD40   40.00
10 East 334  Consumer  4/29/2017 0:00   Second Class Pennsylvania  Binders     USD8    8.00
</code>```

Body:
I have the following data frame in R

East    24  Consumer    7/18/2018 0:00  Second Class    Pennsylvania    Chairs  USD71.37
East    29  Consumer    9/21/2016 0:00  Standard Class  Pennsylvania    Binders USD9.61
East    33  Consumer    9/21/2016 0:00  Standard Class  Pennsylvania    Binders USD6.85
East    199 Corporate   7/13/2018 0:00  Standard Class  Pennsylvania    Binders USD2.94
East    216 Corporate   1/9/2016 0:00   Standard Class  Ohio    Machines    USD1188
East    222 Consumer    12/27/2016 0:00 First Class Ohio    Binders USD24
East    224 Consumer    12/27/2016 0:00 First Class Ohio    Machines    USD252
East    331 Consumer    9/1/2017 0:00   First Class Pennsylvania    Binders USD28
East    333 Consumer    4/29/2017 0:00  Second Class    Pennsylvania    Binders USD40
East    334 Consumer    4/29/2017 0:00  Second Class    Pennsylvania    Binders USD8

The last column is sales which is a character - my problem is I want to create a new column with sales numbers that has just the numbers  - please help if able!!!!!

Body:
We could use parse_number function from readr package:

Here V10 is your last column name:

library(dplyr)
library(readr)

df %>%
  mutate(newCol = parse_number(V10))

     V1  V2        V3         V4   V5       V6    V7           V8       V9      V10  newCol
1  East  24  Consumer  7/18/2018 0:00   Second Class Pennsylvania   Chairs USD71.37   71.37
2  East  29  Consumer  9/21/2016 0:00 Standard Class Pennsylvania  Binders  USD9.61    9.61
3  East  33  Consumer  9/21/2016 0:00 Standard Class Pennsylvania  Binders  USD6.85    6.85
4  East 199 Corporate  7/13/2018 0:00 Standard Class Pennsylvania  Binders  USD2.94    2.94
5  East 216 Corporate   1/9/2016 0:00 Standard Class         Ohio Machines  USD1188 1188.00
6  East 222  Consumer 12/27/2016 0:00    First Class         Ohio  Binders    USD24   24.00
7  East 224  Consumer 12/27/2016 0:00    First Class         Ohio Machines   USD252  252.00
8  East 331  Consumer   9/1/2017 0:00    First Class Pennsylvania  Binders    USD28   28.00
9  East 333  Consumer  4/29/2017 0:00   Second Class Pennsylvania  Binders    USD40   40.00
10 East 334  Consumer  4/29/2017 0:00   Second Class Pennsylvania  Binders     USD8    8.00