问:
Does union of bit fields make any sense
I recently came across a third party code snippet inside our application that does not make any sense to me. What puzzled me first it that it has been in production for at least 10 years and seemed to work. It is basically a union of bit fields:

```<code>union
{
    unsigned longitude : 2;         //!< status of the longitude (offset: 0)
    unsigned latitude : 2;          //!< status of the latitude (offset: 2)
    unsigned xPosition : 2;         //!< status of the x position relative to starting position (offset: 4)
    unsigned yPosition : 2;         //!< status of the y position relative to starting position (offset: 6)
    // ... Many more 2-bit fields... Total 26 fields
    unsigned reserved : 12;
    unsigned long status[2];            //!< raw status data
} status;
</code>```

I am pretty sure that this is a bug and what the author really wanted to write was:

```<code>union
{
    struct
    {
        unsigned longitude : 2;         //!< status of the longitude (offset: 0)
        unsigned latitude : 2;          //!< status of the latitude (offset: 2)
        unsigned xPosition : 2;         //!< status of the x position relative to starting position (offset: 4)
        unsigned yPosition : 2;         //!< status of the y position relative to starting position (offset: 6)
        // ... Many more 2-bit fields... Total 26 fields
        unsigned reserved : 12;
    };
    unsigned status[2];         //!< raw status data (should not be long!)
} status;
</code>```

The reason it was kind of working is that only one of those 26 bit fields was actually used!
But this bug made raised a few questions:

Is the compiler required to use the same first 2 bits of the same unsigned for all fields (despite what the comments say)?

Is there any real usage for a union of bit fields? I cannot think of any situation where this would make sense.

If that doesn't make any sense, how come that no compilers we are using (neither Clang, GCC nor MSVC) were issuing any warning?

答:
I was curious so I coded this up.  Certainly this was a bug waiting to happen.

```<code>#include <stdio.h>
#include <string.h>
union
{
    unsigned longitude : 2;         //!< status of the longitude (offset: 0)
    unsigned latitude : 2;          //!< status of the latitude (offset: 2)
    unsigned xPosition : 2;         //!< status of the x position relative to starting position (offset: 4)
    unsigned yPosition : 2;         //!< status of the y position relative to starting position (offset: 6)
    // ... Many more 2-bit fields... Total 26 fields
    unsigned reserved : 12;
    unsigned long status[2];            //!< raw status data
} status;

int main( int argc, char **argv) {
  memset( &status, 0x00, sizeof(status) );
  status.longitude = 0x02;

  printf("Union Size = %lu\n", sizeof(status));
  printf("status.longitude = %x\n", status.longitude );
  printf("status.latitude = %x\n", status.latitude );
}
</code>```

Setting status.longitude = 0x02; effectively sets all the other variables in the ūnion to the same value.

Output is:

Union Size = 16

status.longitude = 2

status.latitude = 2

Title: Does union of bit fields make any sense

Body:
I recently came across a third party code snippet inside our application that does not make any sense to me. What puzzled me first it that it has been in production for at least 10 years and seemed to work. It is basically a union of bit fields:

union
{
    unsigned longitude : 2;         //!< status of the longitude (offset: 0)
    unsigned latitude : 2;          //!< status of the latitude (offset: 2)
    unsigned xPosition : 2;         //!< status of the x position relative to starting position (offset: 4)
    unsigned yPosition : 2;         //!< status of the y position relative to starting position (offset: 6)
    // ... Many more 2-bit fields... Total 26 fields
    unsigned reserved : 12;
    unsigned long status[2];            //!< raw status data
} status;

I am pretty sure that this is a bug and what the author really wanted to write was:

union
{
    struct
    {
        unsigned longitude : 2;         //!< status of the longitude (offset: 0)
        unsigned latitude : 2;          //!< status of the latitude (offset: 2)
        unsigned xPosition : 2;         //!< status of the x position relative to starting position (offset: 4)
        unsigned yPosition : 2;         //!< status of the y position relative to starting position (offset: 6)
        // ... Many more 2-bit fields... Total 26 fields
        unsigned reserved : 12;
    };
    unsigned status[2];         //!< raw status data (should not be long!)
} status;

The reason it was kind of working is that only one of those 26 bit fields was actually used!
But this bug made raised a few questions:

Is the compiler required to use the same first 2 bits of the same unsigned for all fields (despite what the comments say)?

Is there any real usage for a union of bit fields? I cannot think of any situation where this would make sense.

If that doesn't make any sense, how come that no compilers we are using (neither Clang, GCC nor MSVC) were issuing any warning?

Body:
I was curious so I coded this up.  Certainly this was a bug waiting to happen.

#include <stdio.h>
#include <string.h>
union
{
    unsigned longitude : 2;         //!< status of the longitude (offset: 0)
    unsigned latitude : 2;          //!< status of the latitude (offset: 2)
    unsigned xPosition : 2;         //!< status of the x position relative to starting position (offset: 4)
    unsigned yPosition : 2;         //!< status of the y position relative to starting position (offset: 6)
    // ... Many more 2-bit fields... Total 26 fields
    unsigned reserved : 12;
    unsigned long status[2];            //!< raw status data
} status;

int main( int argc, char **argv) {
  memset( &status, 0x00, sizeof(status) );
  status.longitude = 0x02;

  printf("Union Size = %lu\n", sizeof(status));
  printf("status.longitude = %x\n", status.longitude );
  printf("status.latitude = %x\n", status.latitude );
}

Setting status.longitude = 0x02; effectively sets all the other variables in the ūnion to the same value.

Output is:

Union Size = 16

status.longitude = 2

status.latitude = 2


来源: stackexchange

问:
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
Importing from pyxdameraulevenshtein gives the following error, I have

```<code>pyxdameraulevenshtein==1.5.3
pandas==1.1.4
scikit-learn==0.20.2.
</code>```

Numpy is 1.16.1.

Works well in Python 3.6, Issue in Python 3.7.

Has anyone been facing similar issues with Python 3.7 (3.7.9), docker image python:3.7-buster?

```<code>from pyxdameraulevenshtein import normalized_damerau_levenshtein_distance as norm_dl_dist
__init__.pxd:242: in init pyxdameraulevenshtein
    ???
E   ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject
</code>```

答: numpy version 1.22 solved it for me.</p>

ContentLicense: CC BY-SA 4.0

Body:
Importing from pyxdameraulevenshtein gives the following error, I have

pyxdameraulevenshtein==1.5.3
pandas==1.1.4
scikit-learn==0.20.2.

Numpy is 1.16.1.

Works well in Python 3.6, Issue in Python 3.7.

Has anyone been facing similar issues with Python 3.7 (3.7.9), docker image python:3.7-buster?

from pyxdameraulevenshtein import normalized_damerau_levenshtein_distance as norm_dl_dist
__init__.pxd:242: in init pyxdameraulevenshtein
    ???
E   ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

Body: numpy version 1.22 solved it for me.


问:
How to write a pandas dataframe to_json() to s3 in json format
I have an AWS lambda function which creates a data frame, I need to write this file to a S3 bucket.</p>
```<code>import pandas as pd
import boto3
import io

# code to get the df

destination = "output_" + str(datetime.datetime.now().strftime('%Y_%m_%d_%H_%M_%S')) + '.json'

df.to_json(destination) # this file should be written to S3 bucket

</code>```

答:
You can use following code as well</p>
```<code>#Creating Session using Boto3

session = boto3.Session(
aws_access_key_id='<key ID>',
aws_secret_access_key='<secret_key>'
)

#Create s3 session with boto3

s3 = session.resource('s3')

json_buffer = io.StringIO()

# Create dataframe and convert to pandas
df = spark.range(4).withColumn("organisation", lit("stackoverflow"))
df_p = df.toPandas()
df_p.to_json(json_buffer, orient='records')

#Create s3 object
object = s3.Object('<bucket-name>', '<JSON file name>')

#Put the object into bucket
result = object.put(Body=json_buffer.getvalue())
</code>```

Body:
I have an AWS lambda function which creates a data frame, I need to write this file to a S3 bucket.

import pandas as pd
import boto3
import io

# code to get the df

destination = "output_" + str(datetime.datetime.now().strftime('%Y_%m_%d_%H_%M_%S')) + '.json'

df.to_json(destination) # this file should be written to S3 bucket

Body:
You can use following code as well

#Creating Session using Boto3

session = boto3.Session(
aws_access_key_id='<key ID>',
aws_secret_access_key='<secret_key>'
)

#Create s3 session with boto3

s3 = session.resource('s3')

json_buffer = io.StringIO()

# Create dataframe and convert to pandas
df = spark.range(4).withColumn("organisation", lit("stackoverflow"))
df_p = df.toPandas()
df_p.to_json(json_buffer, orient='records')

#Create s3 object
object = s3.Object('<bucket-name>', '<JSON file name>')

#Put the object into bucket
result = object.put(Body=json_buffer.getvalue())


问:
APScheduler RuntimeError: No application found
I have a problem. I have IP addresses and with APScheduler I try to ping them every 10 seconds and update my database. For APScheduler I understand that I need to use with app.app_context() but the problem is that I don't know where to place it and wherever I have tried to place it, it raise: RuntimeError: No application found and fails to update the database

init.py:

```<code>from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from os import path

db = SQLAlchemy()
DB_NAME = "database.db"

def create_app():
    app = Flask(__name__)
    app.config['SECRET_KEY'] = 'hjshjhdjah kjshkjdhjs'
    app.config['SQLALCHEMY_DATABASE_URI'] = f'sqlite:///{DB_NAME}'

    db.init_app(app)

    from .views import views
    from .auth import auth

    app.register_blueprint(views, url_prefix='/')
    app.register_blueprint(auth, url_prefix='/')

    from .models import Servers

    create_database(app)

    return app

def create_database(app):
    if not path.exists('websiteflaskupdatedd/' + DB_NAME):
        db.create_all(app=app)
        print('Created Database!')
</code>```

auth.py:

```<code>from flask import Blueprint, render_template, request, flash, redirect, url_for, escape, session
from .models import Servers, ping
from . import db, create_app
from apscheduler.schedulers.background import BackgroundScheduler

@auth.route('/servers', methods = ['POST', 'GET'])
def servers():
    if request.method == 'POST':
        server_ip = request.form.get("server_ip")

        new_server = Servers(server_ip = server_ip)

        try:
            db.session.add(new_server)
            db.session.commit()
            return redirect("/servers")

        except:
            return "There was an error adding your server"

    else:
        if not Servers.query.get(1) == None:
            servers = Servers.query.order_by(Servers.date_created)
            return render_template('servers.html', servers=servers)

def update_up_status():
    with create_app().app_context():
        server_status_column = Servers.query.filter(Servers.server_status.in_(["0","1"]))
        for server in server_status_column:
            server.server_status = ping(server.server_ip)

    db.session.commit()

scheduler = BackgroundScheduler()

if not scheduler.running:
    scheduler.add_job(func=update_up_status, trigger="interval", seconds=10)
    scheduler.start()
</code>```

models.py:

```<code>from . import db
from datetime import datetime
import platform
import subprocess

def ping(host):

    cmd = ['ping', '-w', "1", "-n", '1', host]

    return subprocess.call(cmd) == 0

class Servers(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    server_ip = db.Column(db.String(12), nullable=False, unique=True)
    server_status = db.Column(db.String(12), nullable=False)
    date_created = db.Column(db.DateTime, default= datetime.now)
</code>```

答:
Looks like you don't have your application running.

I use this setup regularly (flask + apscheduler) and have a main.py file with the following (including the necessary module imports from your other files):

```<code>if __name__ == "__main__":

    scheduler = BackgroundScheduler()
    scheduler.add_job(func=update_up_status, trigger="interval", seconds=10)

    scheduler.start()

    app.run(host="0.0.0.0", debug=True)
</code>```

The background scheduler works within the background of another application and will not be blocked by the main thread which in this case is the flask application. As opposed to the blocking scheduler which will block the main thread and cannot be run inside of a another application.

Body:
I have a problem. I have IP addresses and with APScheduler I try to ping them every 10 seconds and update my database. For APScheduler I understand that I need to use with app.app_context() but the problem is that I don't know where to place it and wherever I have tried to place it, it raise: RuntimeError: No application found and fails to update the database

init.py:

from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from os import path

db = SQLAlchemy()
DB_NAME = "database.db"

def create_app():
    app = Flask(__name__)
    app.config['SECRET_KEY'] = 'hjshjhdjah kjshkjdhjs'
    app.config['SQLALCHEMY_DATABASE_URI'] = f'sqlite:///{DB_NAME}'

    db.init_app(app)

    from .views import views
    from .auth import auth

    app.register_blueprint(views, url_prefix='/')
    app.register_blueprint(auth, url_prefix='/')

    from .models import Servers

    create_database(app)

    return app

def create_database(app):
    if not path.exists('websiteflaskupdatedd/' + DB_NAME):
        db.create_all(app=app)
        print('Created Database!')

auth.py:

from flask import Blueprint, render_template, request, flash, redirect, url_for, escape, session
from .models import Servers, ping
from . import db, create_app
from apscheduler.schedulers.background import BackgroundScheduler

@auth.route('/servers', methods = ['POST', 'GET'])
def servers():
    if request.method == 'POST':
        server_ip = request.form.get("server_ip")

        new_server = Servers(server_ip = server_ip)

        try:
            db.session.add(new_server)
            db.session.commit()
            return redirect("/servers")

        except:
            return "There was an error adding your server"

    else:
        if not Servers.query.get(1) == None:
            servers = Servers.query.order_by(Servers.date_created)
            return render_template('servers.html', servers=servers)

def update_up_status():
    with create_app().app_context():
        server_status_column = Servers.query.filter(Servers.server_status.in_(["0","1"]))
        for server in server_status_column:
            server.server_status = ping(server.server_ip)

    db.session.commit()

scheduler = BackgroundScheduler()

if not scheduler.running:
    scheduler.add_job(func=update_up_status, trigger="interval", seconds=10)
    scheduler.start()

models.py:

from . import db
from datetime import datetime
import platform
import subprocess

def ping(host):

    cmd = ['ping', '-w', "1", "-n", '1', host]

    return subprocess.call(cmd) == 0

class Servers(db.Model):
    id = db.Column(db.Integer, primary_key=True)
    server_ip = db.Column(db.String(12), nullable=False, unique=True)
    server_status = db.Column(db.String(12), nullable=False)
    date_created = db.Column(db.DateTime, default= datetime.now)

Body:
Looks like you don't have your application running.

I use this setup regularly (flask + apscheduler) and have a main.py file with the following (including the necessary module imports from your other files):

if __name__ == "__main__":

    scheduler = BackgroundScheduler()
    scheduler.add_job(func=update_up_status, trigger="interval", seconds=10)

    scheduler.start()

    app.run(host="0.0.0.0", debug=True)

The background scheduler works within the background of another application and will not be blocked by the main thread which in this case is the flask application. As opposed to the blocking scheduler which will block the main thread and cannot be run inside of a another application.


问:
Spring Security - DB Authorization with a default authority?
In Spring security 3 - can I declaratively define a default authority to be assigned to all authenticated users that are authenticated using the jdbc authentication provider? Currently my webapp does not have the concept of roles (nor will it - you either are a user allowed to access your account or you are not) and I don't want to implement an authority table in my database as it would be redundant. Currently with the below declaration i get the following error message.

message [Table 'authorities' doesn't
  exist];

```<code><authentication-manager>
        <authentication-provider>
            <password-encoder hash="plaintext"/>
            <jdbc-user-service data-source-ref="dataSource"
                               users-by-username-query="select account_uid,hashedpassword,true from account where account_uid=?" />

        </authentication-provider>
    </authentication-manager>
</code>```

If this cannot be done declaratively - what would be the best approach be?

Thanks In Advance,

Ian.

答:
This can be done in a simple way:

authorities-by-username-query="select account_uid, 'default' from account where account_uid=?"

This query reuses your account table and gives each user a default authority.

ContentLicense: CC BY-SA 2.5

Body:
In Spring security 3 - can I declaratively define a default authority to be assigned to all authenticated users that are authenticated using the jdbc authentication provider? Currently my webapp does not have the concept of roles (nor will it - you either are a user allowed to access your account or you are not) and I don't want to implement an authority table in my database as it would be redundant. Currently with the below declaration i get the following error message.

message [Table 'authorities' doesn't
  exist];

<authentication-manager>
        <authentication-provider>
            <password-encoder hash="plaintext"/>
            <jdbc-user-service data-source-ref="dataSource"
                               users-by-username-query="select account_uid,hashedpassword,true from account where account_uid=?" />

        </authentication-provider>
    </authentication-manager>

If this cannot be done declaratively - what would be the best approach be?

Thanks In Advance,

Ian.

ContentLicense: CC BY-SA 3.0


问:
Checking for a number in the url
```<code> if(window.location.href.includes('part_category_tags[]=2')) {
   this.checkedCategories.push(2);
 }
</code>```

Needed: depending on what the number is next to the part_category_tags[]= parameter, push this number into the array.
also, it may be that such a parameter occurs twice like (Or not be at all):

```<code>&part_category_tags[]=3&part_category_tags[]=2
</code>```

Without vue-router

答:
You can just loop through the params.

const url = new URL('https://fake.domain.name/?fake=query&part_category_tags[]=3&part_category_tags[]=2'); // document.location instead of fake utl.

let params = url.searchParams;

params.forEach((v,q) => console.info(q, v)); // just output queries and values

params.forEach((v,q) => { if (q === 'part_category_tags[]' && v === '2') { console.info('Pushing the 2') } });
// params.forEach((v,q) => { if (q === 'part_category_tags[]' && v === '2') { this.checkedCategories.push(v); } });```
</div>
</div>
</p>
<blockquote>
<p>Updated with numberless code.</p>
</blockquote>
<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>// // use document.location instead of fake url.
const url = new URL('https://fake.domain.name/?fake=query&part_category_tags=3&part_category_tags=2');

let params = url.searchParams;

params.forEach((v,q) => { if (q === 'part_category_tags') { console.info(`Pushing the ${v}`) } });
// params.forEach((v,q) => { if (q === 'part_category_tags') { this.checkedCategories.push(v); } });</code>```

Title: Checking for a number in the url

Body:
 if(window.location.href.includes('part_category_tags[]=2')) {
   this.checkedCategories.push(2);
 }

Needed: depending on what the number is next to the part_category_tags[]= parameter, push this number into the array.
also, it may be that such a parameter occurs twice like (Or not be at all):

&part_category_tags[]=3&part_category_tags[]=2

Without vue-router

Body:
You can just loop through the params.

const url = new URL('https://fake.domain.name/?fake=query&part_category_tags[]=3&part_category_tags[]=2'); // document.location instead of fake utl.

let params = url.searchParams;

params.forEach((v,q) => console.info(q, v)); // just output queries and values

params.forEach((v,q) => { if (q === 'part_category_tags[]' && v === '2') { console.info('Pushing the 2') } });
// params.forEach((v,q) => { if (q === 'part_category_tags[]' && v === '2') { this.checkedCategories.push(v); } });

Updated with numberless code.

// // use document.location instead of fake url.
const url = new URL('https://fake.domain.name/?fake=query&part_category_tags=3&part_category_tags=2');

let params = url.searchParams;

params.forEach((v,q) => { if (q === 'part_category_tags') { console.info(`Pushing the ${v}`) } });
// params.forEach((v,q) => { if (q === 'part_category_tags') { this.checkedCategories.push(v); } });


问:
Control throughput per second
I have a class witch is responsible for sending data to a client and all others classes use this when they need to send data. Let's call it 'DataSender.class'.

Now the client is asking us to control the throughput to a maximum of 50 calls per second.

I need to create an algoritm on this class (if possible) to keep the number of calls in the current second and if it reaches the maximum of 50, hold the process either with a sleep or something and continue without losing data.
Maybe I have to implement a queue or something better than a simple sleep. I need sugestions or a direction to follow.

For the sake of simplicity, just imagine that everyone is using something like this and I cannot change how they call me now. post() return is syncronous but that maybe I can possibly change (not sure yet):

```<code>DataSender ds = new DataSender();
ds.setdata(mydata);
if (ds.post()) {
   //data send succesfull
}
</code>```

答:
If I am not mistaken, what you are looking for is throttling or rate limiting.

As Andrew S pointed out, you will need a Queue, to hold the extra requests and a sender algorithm.

The main point is that because you are not sending the data right away, the callers need to be aware that the data is not necessarily sent when their call returns. Usually senders will not be happy, if their call returns, they assume data is sent, and then the data is lost. There are many reasons why data can be lost in this scenario. As Andrew S pointed out, making senders aware that it will be an asynchronous send queue, maybe with confirmations upon successful send, will be a safer or proper approach.

You will need to decide on the size of the queue, you have to limit the size or you can run out of memory. And you need to decide what happens with the request when queue is full. Also what happens when the end point is not accessible (server down, network issues, solar flares), keep accepting data to queue or reject / throw exception.

Hint : if you have 50 requests limit, don't blast 50 requests and then sleep for 1 second. Figure out the interval between sends, send one request, make a short interval sleep.

Pro hint : if new data sent invalidates the data that was previously requested to be sent, but not sent yet, you can optimize the data sent by removing the invalidated data from the queue. This is called Conflation. Usual example is stock market prices. Say you got price of ACME 100 ten seconds ago and for whatever reason that data was not sent. If you get a new price for ACME 101 now, it is usually not useful to send the 100 price record, just send the 101 price.

Title: Control throughput per second

Body:
I have a class witch is responsible for sending data to a client and all others classes use this when they need to send data. Let's call it 'DataSender.class'.

Now the client is asking us to control the throughput to a maximum of 50 calls per second.

I need to create an algoritm on this class (if possible) to keep the number of calls in the current second and if it reaches the maximum of 50, hold the process either with a sleep or something and continue without losing data.
Maybe I have to implement a queue or something better than a simple sleep. I need sugestions or a direction to follow.

For the sake of simplicity, just imagine that everyone is using something like this and I cannot change how they call me now. post() return is syncronous but that maybe I can possibly change (not sure yet):

DataSender ds = new DataSender();
ds.setdata(mydata);
if (ds.post()) {
   //data send succesfull
}


问:
Opening certain files in certain folders in a HTML page
I am trying to learn a bit about web technologies therefore I am trying to create a catalogue for my files.
The situation is the following:

I have a folder with N sub folders;

in each of there sub folders there is an image with always the same name (ie: image.jpg)

in each of there sub folders there is also a certain swg file with always the same name (ie: test.swg)

I would like to create an HTML file which read all the sub folders and create a preview using image.jpg, and when one clicks on the preview test.swg should be launched (not in the browser if possible)

The HTML files should contains all these preview like a catalogue.

How can I do this? should I have a local web server which runs in my machine? is it possible to do this with non web page technologies?

Thank you!

答:
As far as i know Javascript & HTML doesn't have access to the filesystem as it's running on your browser and shouldn't be possible to go through the files iteratively because it would be some kind of breach in security.

If you ask me it's possible or not without a server, it should be possible but it is going to use other technology, for example:

Using a Command Line Interface in Linux or Windows based os you could write a shell script that iteratively will go through the files and folder path, and possibly create a JSON from it. From there the javascript could technically load that file like below.

```<code>    <script type="text/javascript" src="data.json"></script>
    <script type="text/javascript" src="javascript.js"></script>
</code>```

But do note that you should periodically run the shell script periodically with something like scheduler or refresh it manually.

If you want to do it the normal way you could use many different server side language, for example NodeJs, or PHP as I think both of them require only little configuration.

You could post follow up question if you've decided on which language you want to use.

Below is some reference that you can use to start working on reading the directories

NodeJS

Node.js fs.readdir recursive directory search

Get all files recursively in directories NodejS

PHP

List all the files and folders in a Directory with PHP recursive function

How to recursively iterate through files in PHP?

After reading the directories & Files you just need to pass the data to the "rendering" part, and use some javascript to invoke the .swg when the image is clicked

But I'm not really sure about the .swg file can be invoked to the desktop app directly or not you could do some research on it

Open online file with desktop applications?

Body:
As far as i know Javascript & HTML doesn't have access to the filesystem as it's running on your browser and shouldn't be possible to go through the files iteratively because it would be some kind of breach in security.

If you ask me it's possible or not without a server, it should be possible but it is going to use other technology, for example:

Using a Command Line Interface in Linux or Windows based os you could write a shell script that iteratively will go through the files and folder path, and possibly create a JSON from it. From there the javascript could technically load that file like below.

    <script type="text/javascript" src="data.json"></script>
    <script type="text/javascript" src="javascript.js"></script>

But do note that you should periodically run the shell script periodically with something like scheduler or refresh it manually.

If you want to do it the normal way you could use many different server side language, for example NodeJs, or PHP as I think both of them require only little configuration.

You could post follow up question if you've decided on which language you want to use.

Below is some reference that you can use to start working on reading the directories

NodeJS

Node.js fs.readdir recursive directory search

Get all files recursively in directories NodejS

PHP

List all the files and folders in a Directory with PHP recursive function

How to recursively iterate through files in PHP?

After reading the directories & Files you just need to pass the data to the "rendering" part, and use some javascript to invoke the .swg when the image is clicked

But I'm not really sure about the .swg file can be invoked to the desktop app directly or not you could do some research on it

Open online file with desktop applications?


问:
Can every recursive process be transformed into an iterative process?
I was reading the book, Structure and Interpretation of Computer Programs, where in it tells about the distinction between a recursive procedure and recursive process, and similarly between iterative procedure and iterative process. So, a recursive procedure could still generate an iterative process.

My question is: given a procedure which generates a recursive process, can you always write another procedure that achieves the same result but generates an iterative process?

The specific problem that I was trying to solve was to write a procedure which does an in-order traversal of a binary search tree but generates an iterative process. I know how you can use a stack to get an iterative procedure for this problem. However, that still generates a recursive process (correct me if I am wrong here).

Thanks,

Abhinav.

答:
Read this former SO post:

Design patterns for converting recursive algorithms to iterative ones

there are a lot of good answers there which may help you further.


问:
Disable Multipart Upload to S3 On Spark
I'm trying to write on a bucket that access is granted anonymously (policy allows our VPC).
For a small workload, it works fine, but for a big one, I get the following exception:

```<code>22/02/08 19:25:40 WARN TaskSetManager: Lost task 7.0 in stage 2.0 (TID 227) (172.20.64.7 executor 1): org.apache.spark.SparkException: Task failed while writing rows.
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:396)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:284)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
    at org.apache.spark.scheduler.Task.doRunTask(Task.scala:150)
    at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:119)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.Task.run(Task.scala:91)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:813)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1620)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:816)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:672)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.AccessDeniedException: cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet: initiate MultiPartUpload on cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads.  Please authenticate.; request: POST https://prd-ifood-data-lake-transient-groceries.bucket.vpce-08b663a29475cd9f4-wcex1383.s3.us-east-1.vpce.amazonaws.com cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet {key=[null]} Hadoop 2.7.4, aws-sdk-java/1.11.655 Linux/5.4.0-1063-azure OpenJDK_64-Bit_Server_VM/25.302-b08 java/1.8.0_302 scala/2.12.10 vendor/Azul_Systems,_Inc. com.amazonaws.services.s3.model.InitiateMultipartUploadRequest; Request ID: WFX4E5H30RYT9SXJ, Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=, Cloud Provider: Azure, Instance ID: da1fea5b08ef43f1b09adb89a162772c (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: WFX4E5H30RYT9SXJ; S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=), S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=:AccessDenied
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:248)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:132)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.initiateMultiPartUpload(WriteOperationHelper.java:215)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.<init>(S3ABlockOutputStream.java:579)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream$PageCRCVerifyingMultiPartUpload.<init>(PageCRCVerifyingS3ABlockOutputStream.java:153)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream.initMultipartUpload(PageCRCVerifyingS3ABlockOutputStream.java:113)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.uploadCurrentBlock(S3ABlockOutputStream.java:326)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.write(S3ABlockOutputStream.java:306)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream.write(PageCRCVerifyingS3ABlockOutputStream.java:18)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
    at java.io.DataOutputStream.write(DataOutputStream.java:107)
    at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
    at org.apache.parquet.hadoop.util.HadoopPositionOutputStream.write(HadoopPositionOutputStream.java:45)
    at org.apache.parquet.bytes.ConcatenatingByteArrayCollector.writeAllTo(ConcatenatingByteArrayCollector.java:46)
    at org.apache.parquet.hadoop.ParquetFileWriter.writeDataPages(ParquetFileWriter.java:536)
    at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writeToFileWriter(ColumnChunkPageWriteStore.java:246)
    at org.apache.parquet.hadoop.ColumnChunkPageWriteStore.flushToFileWriter(ColumnChunkPageWriteStore.java:316)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:202)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:127)
    at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:41)
    at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:58)
    at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:75)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$2(FileFormatWriter.scala:377)
    at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1654)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:383)
    ... 19 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads.  Please authenticate.; request: POST https://prd-ifood-data-lake-transient-groceries.bucket.vpce-08b663a29475cd9f4-wcex1383.s3.us-east-1.vpce.amazonaws.com cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet {key=[null]} Hadoop 2.7.4, aws-sdk-java/1.11.655 Linux/5.4.0-1063-azure OpenJDK_64-Bit_Server_VM/25.302-b08 java/1.8.0_302 scala/2.12.10 vendor/Azul_Systems,_Inc. com.amazonaws.services.s3.model.InitiateMultipartUploadRequest; Request ID: WFX4E5H30RYT9SXJ, Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=, Cloud Provider: Azure, Instance ID: da1fea5b08ef43f1b09adb89a162772c (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: WFX4E5H30RYT9SXJ; S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=), S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4926)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4872)
    at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3560)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.initiateMultipartUpload(S3AFileSystem.java:3641)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$initiateMultiPartUpload$0(WriteOperationHelper.java:216)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
    ... 48 more
</code>```

Since the problem is with multipart upload, I've tried to disable it. I've already tried:

set spark.hadoop.fs.s3.multipart.uploads.enabled to false

set spark.hadoop.fs.s3a.multipart.uploads.enabled to false

set spark.hadoop.fs.s3n.multipart.uploads.enabled to false

set spark.hadoop.fs.s3.multipart.threshold to a very very big value

set spark.hadoop.fs.s3a.multipart.threshold to a very very big value

set spark.hadoop.fs.s3n.multipart.threshold to a very very big value

Everything on cluster startup and nothing seems to work, resulting in the same error.
It is worthwhile to say that:

It is an Azure Databricks instance.

I'm using Pyspark.

There's a Security restriction on creating users (access through accessKey/secretKey) thus, the access through anonymous user.

Does anyone had a similar issue and disabled with success the multipart upload?
Cheers!

答:
it says "Anonymous users cannot initiate multipart uploads."

if you can grant this permission-do so. otherwise, play with the S3a options to set the size at which uploads go from single post to multipart. I think it is 'fs.s3a.multipart.threshold' ...set it something big like "1G" and make sure you've enough local storage to buffer all active uploads

note that the stack trace included 'PageCRCVerifyingS3ABlockOutputStream'. that's not in the apache s3a connector, so you will be getting whatever they have implemented-which may be different

Title: Disable Multipart Upload to S3 On Spark

Body:
I'm trying to write on a bucket that access is granted anonymously (policy allows our VPC).
For a small workload, it works fine, but for a big one, I get the following exception:

22/02/08 19:25:40 WARN TaskSetManager: Lost task 7.0 in stage 2.0 (TID 227) (172.20.64.7 executor 1): org.apache.spark.SparkException: Task failed while writing rows.
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:396)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$15(FileFormatWriter.scala:284)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$3(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.$anonfun$runTask$1(ResultTask.scala:75)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:55)
    at org.apache.spark.scheduler.Task.doRunTask(Task.scala:150)
    at org.apache.spark.scheduler.Task.$anonfun$run$1(Task.scala:119)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.scheduler.Task.run(Task.scala:91)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$13(Executor.scala:813)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1620)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:816)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at com.databricks.spark.util.ExecutorFrameProfiler$.record(ExecutorFrameProfiler.scala:110)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:672)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.nio.file.AccessDeniedException: cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet: initiate MultiPartUpload on cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads.  Please authenticate.; request: POST https://prd-ifood-data-lake-transient-groceries.bucket.vpce-08b663a29475cd9f4-wcex1383.s3.us-east-1.vpce.amazonaws.com cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet {key=[null]} Hadoop 2.7.4, aws-sdk-java/1.11.655 Linux/5.4.0-1063-azure OpenJDK_64-Bit_Server_VM/25.302-b08 java/1.8.0_302 scala/2.12.10 vendor/Azul_Systems,_Inc. com.amazonaws.services.s3.model.InitiateMultipartUploadRequest; Request ID: WFX4E5H30RYT9SXJ, Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=, Cloud Provider: Azure, Instance ID: da1fea5b08ef43f1b09adb89a162772c (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: WFX4E5H30RYT9SXJ; S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=), S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=:AccessDenied
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:248)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:111)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.retry(WriteOperationHelper.java:132)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.initiateMultiPartUpload(WriteOperationHelper.java:215)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream$MultiPartUpload.<init>(S3ABlockOutputStream.java:579)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream$PageCRCVerifyingMultiPartUpload.<init>(PageCRCVerifyingS3ABlockOutputStream.java:153)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream.initMultipartUpload(PageCRCVerifyingS3ABlockOutputStream.java:113)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.uploadCurrentBlock(S3ABlockOutputStream.java:326)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3ABlockOutputStream.write(S3ABlockOutputStream.java:306)
    at shaded.databricks.org.apache.hadoop.fs.s3a.PageCRCVerifyingS3ABlockOutputStream.write(PageCRCVerifyingS3ABlockOutputStream.java:18)
    at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58)
    at java.io.DataOutputStream.write(DataOutputStream.java:107)
    at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
    at org.apache.parquet.hadoop.util.HadoopPositionOutputStream.write(HadoopPositionOutputStream.java:45)
    at org.apache.parquet.bytes.ConcatenatingByteArrayCollector.writeAllTo(ConcatenatingByteArrayCollector.java:46)
    at org.apache.parquet.hadoop.ParquetFileWriter.writeDataPages(ParquetFileWriter.java:536)
    at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writeToFileWriter(ColumnChunkPageWriteStore.java:246)
    at org.apache.parquet.hadoop.ColumnChunkPageWriteStore.flushToFileWriter(ColumnChunkPageWriteStore.java:316)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:202)
    at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:127)
    at org.apache.parquet.hadoop.ParquetRecordWriter.close(ParquetRecordWriter.java:165)
    at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.close(ParquetOutputWriter.scala:41)
    at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.releaseResources(FileFormatDataWriter.scala:58)
    at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.commit(FileFormatDataWriter.scala:75)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$2(FileFormatWriter.scala:377)
    at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1654)
    at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:383)
    ... 19 more
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Anonymous users cannot initiate multipart uploads.  Please authenticate.; request: POST https://prd-ifood-data-lake-transient-groceries.bucket.vpce-08b663a29475cd9f4-wcex1383.s3.us-east-1.vpce.amazonaws.com cdc_streaming/sitemercado/data/groceries-sm.dbo.produto_integracao/part-00007-43d2b092-ea37-43c3-908a-a987df7a9a88-c000.snappy.parquet {key=[null]} Hadoop 2.7.4, aws-sdk-java/1.11.655 Linux/5.4.0-1063-azure OpenJDK_64-Bit_Server_VM/25.302-b08 java/1.8.0_302 scala/2.12.10 vendor/Azul_Systems,_Inc. com.amazonaws.services.s3.model.InitiateMultipartUploadRequest; Request ID: WFX4E5H30RYT9SXJ, Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=, Cloud Provider: Azure, Instance ID: da1fea5b08ef43f1b09adb89a162772c (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: WFX4E5H30RYT9SXJ; S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=), S3 Extended Request ID: JMxLdOn+T0y1tysF63mg31uvPRttI3wTC7xQAlTxxfpiSY6myfzKYWdmL4G8Jvr1vMNcWgKAos0=
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1712)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1367)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1113)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:770)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:744)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4926)
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4872)
    at com.amazonaws.services.s3.AmazonS3Client.initiateMultipartUpload(AmazonS3Client.java:3560)
    at shaded.databricks.org.apache.hadoop.fs.s3a.S3AFileSystem.initiateMultipartUpload(S3AFileSystem.java:3641)
    at shaded.databricks.org.apache.hadoop.fs.s3a.WriteOperationHelper.lambda$initiateMultiPartUpload$0(WriteOperationHelper.java:216)
    at shaded.databricks.org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
    ... 48 more

Since the problem is with multipart upload, I've tried to disable it. I've already tried:

set spark.hadoop.fs.s3.multipart.uploads.enabled to false

set spark.hadoop.fs.s3a.multipart.uploads.enabled to false

set spark.hadoop.fs.s3n.multipart.uploads.enabled to false

set spark.hadoop.fs.s3.multipart.threshold to a very very big value

set spark.hadoop.fs.s3a.multipart.threshold to a very very big value

set spark.hadoop.fs.s3n.multipart.threshold to a very very big value

Everything on cluster startup and nothing seems to work, resulting in the same error.
It is worthwhile to say that:

It is an Azure Databricks instance.

I'm using Pyspark.

There's a Security restriction on creating users (access through accessKey/secretKey) thus, the access through anonymous user.

Does anyone had a similar issue and disabled with success the multipart upload?
Cheers!


问:
How to produce a time serie with geom_bar COUNTING INSTANCES in categoric variable?
My data frame (s1) has this structure:

PERIOD         : Date, format: "2019-02-01" "2019-02-01" "2019-02-01" "2019-02-01" ...

OPERATION      : chr  "SALE" "SALE" "SALE" "SALE"…

My goal: creating a plot showing the number of units sold each month between 2019-2021

When I try to produce it by means of ggplot :

```<code>ggplot(s1, aes(x= PERIOD, y= factor(OPERATION)))+
  geom_bar(stat="count", width=0.7, fill="steelblue")+
  theme_minimal()
</code>```

I get this message:

Error: stat_count() can only have an x or y aesthetic

My questions:

How can I manage to transfor “SALE” which is a categoric variable to become numeric  in order to plot the time serie?

Can you envisage a better solution to produce the desired plot?

Thanks in advance for your advices

答:
How can I manage to transfor “SALE” which is a categoric variable to
become numeric in order to plot the time serie?

In this case, you can change the parameter to stat or just use only one aes. If you use the x axis, then there is no need to do the counting on y axis, it's automatically computed by the graph using absolute frequency of x.

```<code>df %>% ggplot(aes(x= PERIOD)) +
  geom_bar(stat="count", width=0.7, fill="steelblue") +
  theme_minimal()
</code>```

Can you envisage a better solution to produce the desired plot?

I believe here you aim a time series plot. I gave you a code for it. Assuming OPERATION is a character variable which can have both letters and numbers, first we convert to factor and then to numeric. After that, we are ready to plot within y axis. Edit: I forgot to add the filter for the requested dates!

```<code>df <- df %>% filter(between(PERIOD,"01/01/2019","01/01/2021"))

df %>% ggplot(aes(x= PERIOD, y= as.numeric(as.factor(OPERATION)))) +
geom_line() +
  theme_minimal() +
  labs(y="Operations Number")
</code>```

Body:
My data frame (s1) has this structure:

PERIOD         : Date, format: "2019-02-01" "2019-02-01" "2019-02-01" "2019-02-01" ...

OPERATION      : chr  "SALE" "SALE" "SALE" "SALE"…

My goal: creating a plot showing the number of units sold each month between 2019-2021

When I try to produce it by means of ggplot :

ggplot(s1, aes(x= PERIOD, y= factor(OPERATION)))+
  geom_bar(stat="count", width=0.7, fill="steelblue")+
  theme_minimal()

I get this message:

Error: stat_count() can only have an x or y aesthetic

My questions:

How can I manage to transfor “SALE” which is a categoric variable to become numeric  in order to plot the time serie?

Can you envisage a better solution to produce the desired plot?

Thanks in advance for your advices

Body:
How can I manage to transfor “SALE” which is a categoric variable to
become numeric in order to plot the time serie?

In this case, you can change the parameter to stat or just use only one aes. If you use the x axis, then there is no need to do the counting on y axis, it's automatically computed by the graph using absolute frequency of x.

df %>% ggplot(aes(x= PERIOD)) +
  geom_bar(stat="count", width=0.7, fill="steelblue") +
  theme_minimal()

Can you envisage a better solution to produce the desired plot?

I believe here you aim a time series plot. I gave you a code for it. Assuming OPERATION is a character variable which can have both letters and numbers, first we convert to factor and then to numeric. After that, we are ready to plot within y axis. Edit: I forgot to add the filter for the requested dates!

df <- df %>% filter(between(PERIOD,"01/01/2019","01/01/2021"))

df %>% ggplot(aes(x= PERIOD, y= as.numeric(as.factor(OPERATION)))) +
geom_line() +
  theme_minimal() +
  labs(y="Operations Number")


问:
Using regex in python for a dynamic string
I have a pandas columns with strings which dont have the same pattern, something like this:

```<code>{'iso_2': 'FR', 'iso_3': 'FRA', 'name': 'France'}
{'iso': 'FR', 'iso_2': 'USA', 'name': 'United States of America'}
{'iso_3': 'FR', 'iso_4': 'FRA', 'name': 'France'}
</code>```

How do I only keep the name of the country for every row? I would only like to keep "France", "United States of America", "France".

I tried building the regex pattern: something like this

```<code>r"^\W+[a-z]+_[0-9]\W+"
</code>```

But this turns out to be very specific, and if there is a slight change in the string the pattern wont work. How do we resolve this?

答:
As you have dictionaries in the column, you can get the values of the name  keys:

import pandas as pd
df = pd.DataFrame({'col':[{'iso_2': 'FR', 'iso_3': 'FRA', 'name': 'France'},
    {'iso': 'FR', 'iso_2': 'USA', 'name': 'United States of America'},
    {'iso_3': 'FR', 'iso_4': 'FRA', 'name': 'France'}]})
df['col'] = df['col'].apply(lambda x: x["name"])
```
<p>Output of <code>df['col']</code>:</p>
```0                      France
1    United States of America
2                      France
Name: col, dtype: object
```
<p>If the column contains stringified dictionaries, you can use <code>ast.literal_eval</code> before accessing the <code>name</code> key value:</p>
<pre class="lang-py prettyprint-override"><code>import pandas as pd
import ast

df = pd.DataFrame({'col':["{'iso_2': 'FR', 'iso_3': 'FRA', 'name': 'France'}",
    "{'iso': 'FR', 'iso_2': 'USA', 'name': 'United States of America'}",
    "{'iso_3': 'FR', 'iso_4': 'FRA', 'name': 'France'}"]})

df['col'] = df['col'].apply(lambda x: ast.literal_eval(x)["name"])
</code>```

And in case your column is totally messed up, yes, you can resort to regex:

df['col'] = df['col'].str.extract(r"""['"]name['"]\s*:\s*['"]([^"']+)""")
# or to support escaped " and ':
df['col'] = df['col'].str.extract(r"""['"]name['"]\s*:\s*['"]([^"'\\]+(?:\\.[^'"\\]*)*)""")>>> df['col']
                          0
0                    France
1  United States of America
2                    France
```

See the regex demo.

Body:
I have a pandas columns with strings which dont have the same pattern, something like this:

{'iso_2': 'FR', 'iso_3': 'FRA', 'name': 'France'}
{'iso': 'FR', 'iso_2': 'USA', 'name': 'United States of America'}
{'iso_3': 'FR', 'iso_4': 'FRA', 'name': 'France'}

How do I only keep the name of the country for every row? I would only like to keep "France", "United States of America", "France".

I tried building the regex pattern: something like this

r"^\W+[a-z]+_[0-9]\W+"

But this turns out to be very specific, and if there is a slight change in the string the pattern wont work. How do we resolve this?

Body:
As you have dictionaries in the column, you can get the values of the name  keys:

import pandas as pd
df = pd.DataFrame({'col':[{'iso_2': 'FR', 'iso_3': 'FRA', 'name': 'France'},
    {'iso': 'FR', 'iso_2': 'USA', 'name': 'United States of America'},
    {'iso_3': 'FR', 'iso_4': 'FRA', 'name': 'France'}]})
df['col'] = df['col'].apply(lambda x: x["name"])

Output of df['col']:

0                      France
1    United States of America
2                      France
Name: col, dtype: object

If the column contains stringified dictionaries, you can use ast.literal_eval before accessing the name key value:

import pandas as pd
import ast

df = pd.DataFrame({'col':["{'iso_2': 'FR', 'iso_3': 'FRA', 'name': 'France'}",
    "{'iso': 'FR', 'iso_2': 'USA', 'name': 'United States of America'}",
    "{'iso_3': 'FR', 'iso_4': 'FRA', 'name': 'France'}"]})

df['col'] = df['col'].apply(lambda x: ast.literal_eval(x)["name"])

And in case your column is totally messed up, yes, you can resort to regex:

df['col'] = df['col'].str.extract(r"""['"]name['"]\s*:\s*['"]([^"']+)""")
# or to support escaped " and ':
df['col'] = df['col'].str.extract(r"""['"]name['"]\s*:\s*['"]([^"'\\]+(?:\\.[^'"\\]*)*)""")>>> df['col']
                          0
0                    France
1  United States of America
2                    France

See the regex demo.


问:
In python, appending to a list stored in a dictionary key using += []
Here's what I tried... I get a syntax error on the line "context['x'] += [  ... ]"... basically I want to pass my variables into the function as a dictionary reference... modify a few hash tags and hopefully have a persistent state between function calls.

import numpy as np

def draw_init():
    context={'x': [], 'y' : [], 'n' : [], 'count' : 0}
    return context

def draw_circle(context, radius):
    N0 = 16

    for i in range(0, N0-1):
        t                  = float(i) * 2.0 / float(N0) * np.pi;
        context['x']      += [float(radius) * np.sin(t)]
        context['y']      += [float(radius) * np.cos(t)]
        context['n']      += [context['count']]
        context['count']  += 1

context = draw_init

draw_circle(context, 500)
draw_circle(context, 200)
draw_circle(context, 100)

```

答:
import numpy as np

def draw_init():
    context={'x': [], 'y' : [], 'n' : [], 'count' : 0}
    return context

def draw_circle(context, radius):
    N0 = 16

    for i in range(0, N0-1):
        t                  = float(i) * 2.0 / float(N0) * np.pi;
        context['x']      += [float(radius) * np.sin(t)]
        context['y']      += [float(radius) * np.cos(t)]
        context['n']      += [context['count']]
        context['count']  += 1

context = draw_init()

draw_circle(context, 500)
draw_circle(context, 200)
draw_circle(context, 100)
```
<p>Note the</p>
<pre class="lang-py prettyprint-override"><code>context['n']      += [context['count']]
</code>```

and the

context = draw_init()
```

Body:
import numpy as np

def draw_init():
    context={'x': [], 'y' : [], 'n' : [], 'count' : 0}
    return context

def draw_circle(context, radius):
    N0 = 16

    for i in range(0, N0-1):
        t                  = float(i) * 2.0 / float(N0) * np.pi;
        context['x']      += [float(radius) * np.sin(t)]
        context['y']      += [float(radius) * np.cos(t)]
        context['n']      += [context['count']]
        context['count']  += 1

context = draw_init()

draw_circle(context, 500)
draw_circle(context, 200)
draw_circle(context, 100)

Note the

context['n']      += [context['count']]

and the

context = draw_init()


问:
How can I get a reference of a prefab asset from the code in it self?
So let's say I have this prefab asset, "P" and have this code attached to it.

```<code>public GameObject selfReference;

[ContextMenu("GetReference")]
public void GetReference()
{
    selfReference = gameObject;
}
</code>```

Note that:
the 'reference finding' process is trigger by ContextMenu so it is done in Edit Mode, not Play Mode;
and all of this is happening in the prefab asset "P" itself, not some random instance of it placed in the scene.

So I tried

```<code>selfReference = PrefabUtility.GetNearestPrefabInstanceRoot(gameObject);
</code>```

but it didn't work so tried to load it via path:

```<code>string _path = AssetDatabase.GetAssetPath(gameObject);
</code>```

but it returns only blank string.

Any help plz?

答:
I can only assume -  since it works for me ;)

that your question is rather related to this change not being saved correctly and not being handled for undo/redo.

You should probably do e.g.

```<code>public GameObject selfReference;

[ContextMenu(nameof(GetReference))]
public void GetReference()
{
#if UNITY_EDITOR
    if(!Application.isPlaying)
    {
        UnityEditor.Undo.RecordObject(this, "fetched self-reference");

        if (UnityEditor.PrefabUtility.IsPartOfAnyPrefab(this))
        {
            UnityEditor.PrefabUtility.RecordPrefabInstancePropertyModifications(this);
        }
    }
#endif

    selfReference = gameObject;
}
</code>```

besides that of course it appears a bit redundant to me to have a field for something that is exposed via the property anyway ;)

Body:
So let's say I have this prefab asset, "P" and have this code attached to it.

public GameObject selfReference;

[ContextMenu("GetReference")]
public void GetReference()
{
    selfReference = gameObject;
}

Note that:
the 'reference finding' process is trigger by ContextMenu so it is done in Edit Mode, not Play Mode;
and all of this is happening in the prefab asset "P" itself, not some random instance of it placed in the scene.

So I tried

selfReference = PrefabUtility.GetNearestPrefabInstanceRoot(gameObject);

but it didn't work so tried to load it via path:

string _path = AssetDatabase.GetAssetPath(gameObject);

but it returns only blank string.

Any help plz?

Body:
I can only assume -  since it works for me ;)

that your question is rather related to this change not being saved correctly and not being handled for undo/redo.

You should probably do e.g.

public GameObject selfReference;

[ContextMenu(nameof(GetReference))]
public void GetReference()
{
#if UNITY_EDITOR
    if(!Application.isPlaying)
    {
        UnityEditor.Undo.RecordObject(this, "fetched self-reference");

        if (UnityEditor.PrefabUtility.IsPartOfAnyPrefab(this))
        {
            UnityEditor.PrefabUtility.RecordPrefabInstancePropertyModifications(this);
        }
    }
#endif

    selfReference = gameObject;
}

besides that of course it appears a bit redundant to me to have a field for something that is exposed via the property anyway ;)


问:
TypeError: 'str' object doesn't support item deletion
```<code>x = True
pizza = []
pizza_topping = []

while x:
    pizza_topping = input("Please enter your pizza toppings(type 'quit' to when you're done): ")

    if pizza_topping == 'quit':
        print(f'Thank you for your order. Your pizza toppings are: {pizza}')
        x = False
    else:
        pizza.append(pizza_topping)
        del pizza_topping[0]
</code>```

Dear people,

I get this error:

del pizza_topping[0] TypeError: 'str' object doesn't support item
deletion

What am I doing wrong?

Edit: I changed 'break' to x = False(this was my original code). It got mixed up.

答:
There are several things going on here:

In Python strings are immutable, so you can't change their characters in place.

To remove the first character of the string you can do the following:

```<code> pizza_topping =  pizza_topping[1:]
</code>```

That being said, why do you want to remove the first character here?

Change pizza_topping = [] to pizza_topping = "". This change doesn't affect the run of the program, but as it will hold strings there is no sense in first setting it as a list.

Remove x = True and just check if pizza_topping  equals 'quit' in the while's condition.

It is preferred to give variable meaningful names (x is not meaningful).

Use ','.join to print the contents of pizza.

Putting it all together:

```<code>pizza = []

pizza_topping = ""
while pizza_topping != "quit":
    pizza_topping = input("Please enter your pizza toppings(type 'quit' to when you're done): ")
    pizza.append(pizza_topping)
print("Thank you for your order. Your pizza toppings are: " + ','.join(pizza))

</code>```

Body:
x = True
pizza = []
pizza_topping = []

while x:
    pizza_topping = input("Please enter your pizza toppings(type 'quit' to when you're done): ")

    if pizza_topping == 'quit':
        print(f'Thank you for your order. Your pizza toppings are: {pizza}')
        x = False
    else:
        pizza.append(pizza_topping)
        del pizza_topping[0]

Dear people,

I get this error:

del pizza_topping[0] TypeError: 'str' object doesn't support item
deletion

What am I doing wrong?

Edit: I changed 'break' to x = False(this was my original code). It got mixed up.

Body:
There are several things going on here:

In Python strings are immutable, so you can't change their characters in place.

To remove the first character of the string you can do the following:

 pizza_topping =  pizza_topping[1:]

That being said, why do you want to remove the first character here?

Change pizza_topping = [] to pizza_topping = "". This change doesn't affect the run of the program, but as it will hold strings there is no sense in first setting it as a list.

Remove x = True and just check if pizza_topping  equals 'quit' in the while's condition.

It is preferred to give variable meaningful names (x is not meaningful).

Use ','.join to print the contents of pizza.

Putting it all together:

pizza = []

pizza_topping = ""
while pizza_topping != "quit":
    pizza_topping = input("Please enter your pizza toppings(type 'quit' to when you're done): ")
    pizza.append(pizza_topping)
print("Thank you for your order. Your pizza toppings are: " + ','.join(pizza))


问:
Uses and when to use int16_t,int32_t,int64_t and respectively short int,int,long int,long
Uses and when to use int16_t, int32_t, int64_t and respectively short, int, long.

There are too many damn types in C++. For integers when is it correct to use one over the other?

答:
In MISRA-C 2004 and MISRA-C++ 2008 guidelines, the advisory is to prefer specific-length typedefs:

typedefs that indicate size and signedness should be used in place of
the basic numerical types. [...]

This rule helps to clarify the size
of the storage, but does not guarantee portability because of the
asymmetric behaviour of integral promotion. [...]

Exception is for char type :

The plain char type shall be used only for the storage and use of character values.

However, keep in mind that the MISRA guidelines are for critical systems.

Personally, I follow these guidelines for embedded systems, not for computer applications where I simply use an int when I want an integer, letting the compiler optimize as it wants.


问:
Best way to transfer struct from one process to another?
I have one server process and some clients, the clients need to send to the server as many structures as possible, they will not wait for any answer, just send and send. In which way i can transfer next struct in that case:

```<code>typedef struct tagTDATA {
         BYTE c1;
         DWORD dw1;
         DWORD dw2;
         char ch[255];
} TDATA, *PTDATA;
</code>```

Cant use Boost. The best way its Win API. And i don't need code solution just point me in right direction. Thanks.

答:
In my understanding you can solve this issue by implementing a shared queue. The pattern of your problem is more towards Producer Consumer Problem

You can also take these examples

http://msdn.microsoft.com/en-us/library/dd728068.aspx

How should i write producer /consumer code in C#?

Body:
I have one server process and some clients, the clients need to send to the server as many structures as possible, they will not wait for any answer, just send and send. In which way i can transfer next struct in that case:

typedef struct tagTDATA {
         BYTE c1;
         DWORD dw1;
         DWORD dw2;
         char ch[255];
} TDATA, *PTDATA;

Cant use Boost. The best way its Win API. And i don't need code solution just point me in right direction. Thanks.


问:
c# PDF downloader does not start the download
I have some code that I have researched to hopefully create a PDF document of the data held in an asp repeater table. The repeater and eveything else on the page looks good and works as intended, downloading information from SQL and displaying it to the user.

My problem is that the software owner has requested a function to create a PDF document to use for auditing evidence. I have a button that when clicked runs an onClick event handler to perform the following

```<code>        using (StringWriter sw = new StringWriter())
        {
            using  (HtmlTextWriter hw = new HtmlTextWriter(sw))
            {
                pnlhistory.RenderControl(hw);
                StringReader sr = new StringReader(sw.ToString());
                Document pdfDoc = new Document(PageSize.A4, 10f, 10f, 100f, 0f);
                PdfWriter writer = PdfWriter.GetInstance(pdfDoc, Response.OutputStream);
                pdfDoc.Open();
                XMLWorkerHelper.GetInstance().ParseXHtml(writer, pdfDoc, sr);
                pdfDoc.Close();
                Response.ContentType = "application/pdf";
                Response.AddHeader("content-disposition", "attachment;filename=Panel.pdf");
                Response.Cache.SetCacheability(HttpCacheability.NoCache);
                Response.Write(pdfDoc);
                Response.End();
            }
        }
</code>```

in the researched examples and demos I've seen this starts a downlod of the PDF which can then be viewed. However when I step through the code it all seems okay (no exceptions), then after responce.end(); it returns to the webbrowser where nothing happens.

Have I overlooked an important line of code to tell the browser to start downloading?

the repeater is inside a div which is hidden until a show history checkbox in another table is clicked by the user. The control referenced in the c# is the asp panel seen below.

```<code>   <asp:Panel ID="pnlhistory" runat="server">
                <div class="datagrid" id="divHis" runat="server" visible="false">
                <div style="font-family: Verdana; background:#F8F8FF; padding:0px;" >
                    <asp:Repeater ID="rptHis" runat="server">
                        <HeaderTemplate>
                            <table id="tbl" style="width: 100%; text-align: center;">
                                <tr class="headerColour">
                                    <th>Serial Number</th>
                                    <th>Task ID</th>
                                    <th>Service Request ID</th>
                                    <th>Task Type</th>
                                    <th>Date Raised </th>
                                    <th>Raised By</th>
                                    <th>Current Request Status</th>
                                </tr>
                        </HeaderTemplate>
                        <ItemTemplate>
                            <tr class="<%# Container.ItemIndex % 2 == 0 ? "oddRow" : "evenRow" %>">
                                <td align="center" style="padding: 10px; font-size: 13px;"><asp:Label ID="lbl1" runat="server" Text='<%# Eval("data") %>' /></td>
                                <td align="center" style="padding: 10px; font-size: 13px;"><asp:Label ID="lbl2" runat="server" Text='<%# Eval("data") %>' /></td>
                                <td align="center" style="padding: 10px; font-size: 13px;"><asp:HyperLink ID="hyp1" runat="server" Text='<%# Eval("data") %>' ForeColor="Blue" Target="_blank" NavigateUrl='<%# "website.aspx?id=" + Eval("data") %>'></asp:HyperLink></td>
                                <td align="center" style="padding: 10px; font-size: 13px;"><asp:Label ID="lbl23" runat="server" Text='<%# Eval("data") %>' /></td>
                                <td align="center" style="padding: 10px; font-size: 13px;"><asp:Label ID="Label3" runat="server" Text='<%# method2(Eval("data")) %>' /></td>
                                <td align="center" style="padding: 10px; font-size: 13px;"><asp:Label ID="Label1" runat="server" Text='<%# Eval("data") %>' /></td>
                                <td align="center" style="padding: 10px; font-size: 13px;"><asp:Label ID="Label2" runat="server" Text='<%# Eval("data") %>' /></td>
                            </tr>
                        </ItemTemplate>
                        <FooterTemplate>
                            </table>
                        </FooterTemplate>
                    </asp:Repeater>
                </div>
            </div>
            </asp:Panel>
</code>```

I'm not a complete c# novice but I am largly self taught and would welcome any advice that helps me get a working PDF!

thank you all in advance

答: try to change response contentype in  Response.ContentType = "application/octet-stream"; and Response.Write in Response.WriteFile(file.FullName);

Body:
I have some code that I have researched to hopefully create a PDF document of the data held in an asp repeater table. The repeater and eveything else on the page looks good and works as intended, downloading information from SQL and displaying it to the user.

My problem is that the software owner has requested a function to create a PDF document to use for auditing evidence. I have a button that when clicked runs an onClick event handler to perform the following

        using (StringWriter sw = new StringWriter())
        {
            using  (HtmlTextWriter hw = new HtmlTextWriter(sw))
            {
                pnlhistory.RenderControl(hw);
                StringReader sr = new StringReader(sw.ToString());
                Document pdfDoc = new Document(PageSize.A4, 10f, 10f, 100f, 0f);
                PdfWriter writer = PdfWriter.GetInstance(pdfDoc, Response.OutputStream);
                pdfDoc.Open();
                XMLWorkerHelper.GetInstance().ParseXHtml(writer, pdfDoc, sr);
                pdfDoc.Close();
                Response.ContentType = "application/pdf";
                Response.AddHeader("content-disposition", "attachment;filename=Panel.pdf");
                Response.Cache.SetCacheability(HttpCacheability.NoCache);
                Response.Write(pdfDoc);
                Response.End();
            }
        }

in the researched examples and demos I've seen this starts a downlod of the PDF which can then be viewed. However when I step through the code it all seems okay (no exceptions), then after responce.end(); it returns to the webbrowser where nothing happens.

Have I overlooked an important line of code to tell the browser to start downloading?

the repeater is inside a div which is hidden until a show history checkbox in another table is clicked by the user. The control referenced in the c# is the asp panel seen below.

   <asp:Panel ID="pnlhistory" runat="server">

                    <asp:Repeater ID="rptHis" runat="server">
                        <HeaderTemplate>

                                    Serial Number
                                    Task ID
                                    Service Request ID
                                    Task Type
                                    Date Raised
                                    Raised By
                                    Current Request Status

                        </HeaderTemplate>
                        <ItemTemplate>
                            <tr class="<%# Container.ItemIndex % 2 == 0 ? "oddRow" : "evenRow" %>">
                                <asp:Label ID="lbl1" runat="server" Text='<%# Eval("data") %>' />
                                <asp:Label ID="lbl2" runat="server" Text='<%# Eval("data") %>' />
                                <asp:HyperLink ID="hyp1" runat="server" Text='<%# Eval("data") %>' ForeColor="Blue" Target="_blank" NavigateUrl='<%# "website.aspx?id=" + Eval("data") %>'></asp:HyperLink>
                                <asp:Label ID="lbl23" runat="server" Text='<%# Eval("data") %>' />
                                <asp:Label ID="Label3" runat="server" Text='<%# method2(Eval("data")) %>' />
                                <asp:Label ID="Label1" runat="server" Text='<%# Eval("data") %>' />
                                <asp:Label ID="Label2" runat="server" Text='<%# Eval("data") %>' />

                        </ItemTemplate>
                        <FooterTemplate>

                        </FooterTemplate>
                    </asp:Repeater>

            </asp:Panel>

I'm not a complete c# novice but I am largly self taught and would welcome any advice that helps me get a working PDF!

thank you all in advance


问:
Pyinstaller to package app hidden-import warning/exceptions
So, I've had some trouble setting up my application through Inno setup Compiler, at first I assumed it was a problem within Inno itself but on further inspection I think it is my actual exe. I am able to create a working exe file that runs my program properly but only on my own pc. I am able to create the setup file that also works through Inno setup but it only works on my own pc. I have sent both the actual exe file and the Inno setup file to another computer and downloaded it there and ran it, both meet the same "Fatal Error: failed to run script tk_app.py". Therefore, the problem must be that I have not been able to pavkage the app properly with pyinstaller.

My prgoram has 5 files in total (all in the same folder): main.py, file1.py, file2.py file3.py, tk_app.py

All of them importing each other and using python libraries. I know that pyinstaller supports librarires such as pandas, tkinter and many more without needing the --hidden-impoort= command and that it will pick up all files within the program if there are files that are importing each other.

So I use the file tk_app.py (which contains my tkinter UI and imports main.py which then goes onto import file1.py which import another file so on)

The pyinstaller command line I use to make the exe is as follows:

PS C:\Users\ripta\Desktop\CODING\CSV_Upload> pyinstaller -w --hidden-import=bs4 --hidden-import=fake_useragent --hidden-import=urllib.prase --hidden-import=urllib.request --hidden-import=os --hidden-import=pandas.io.parsers --icon=trademark_icon.ico --onefile  tk_app.py

My question is, will pyinstaller tell me when it needs a given --hidden-import='name' when running becuase it doesn not throw up any errors and does produce a spec file, a build folder and a dist folder containing the exe file.

I have noticed that it throws up WARNINGs or Exceptions (also not sure why it mentions django as I do not import or use it at all inthe application) :

 59182 INFO: Loading module hook 'hook-django.db.backends.py' from 'c:\\users\\ripta\\appdata\\local\\programs\\python\\python36-32\\lib\\site-packages\\PyInstaller\\hooks'... 61711 WARNING: Hidden import "django.db.backends.__pycache__.base" not found! 61712 INFO: Loading module hook 'hook-django.py' from 'c:\\users\\ripta\\appdata\\local\\programs\\python\\python36-32\\lib\\site-packages\\PyInstaller\\hooks'... Traceback (most recent call last): File "<string>", line 21, in walk_packages File "c:\users\ripta\appdata\local\programs\python\python36-32\lib\site-packages\django\contrib\gis\admin\__init__.py", line 5, in <module>

Or show Hidden-import not found (Of imports I have no idea about):

149329 WARNING: Hidden import "pkg_resources.py2_warn" not found! 149330 WARNING: Hidden import "pkg_resources.markers" not found!

The fact that the script fails to run on any other computer besided my own leads me to think that it must be lacking a dependency that is only found on my computer, therefore I am not using pyinstaller correctly but am not too sure where exactly I am making the mistake.

答:
First I've uninstalled Setuptools and reinstalled it with specific version. Then, I've imported pkg_resources.py2_warn as hidden import :

```<code>pip uninstalled setuptools
pip install --upgrade 'setuptools<45.0.0'
pyinstaller --hidden-import='pkg_resources.py2_warn' tk_app.py
</code>```

It worked for me.

Body:
First I've uninstalled Setuptools and reinstalled it with specific version. Then, I've imported pkg_resources.py2_warn as hidden import :

pip uninstalled setuptools
pip install --upgrade 'setuptools<45.0.0'
pyinstaller --hidden-import='pkg_resources.py2_warn' tk_app.py

It worked for me.


问:
Why is Vue undefined in my webpack-bundled app after trying to import it as a client-side library
I have been having an issue for 2 days and finally isolated it. Importing Vue and VueRouter gives me undefined in my frontend code. This is because the output from webpack, using the externals property to load client-side libraries, checks for module.__esModule when you import an externally loaded library, and if true then it returns module.default. In this case Vue.__esModule and VueRouter.__esModule are true while Vue.default and VueRouter.default are undefined. I'm not even sure who to file a bug with, or whether there is something I can add to webpack.config.js to make this work as before. It could be Vue for including __esModule on their global builds. Or it could be something in the internals of webpack.

Here is a section of my package.json

"devDependencies": {
  "@babel/core": "^7.17.0",
  "@babel/preset-env": "^7.16.11",
  "@vue/compiler-sfc": "^3.2.30",
  "babel-loader": "^8.2.3",
  "css-loader": "^6.6.0",
  "mini-css-extract-plugin": "^2.5.3",
  "node-sass": "^7.0.1",
  "sass-loader": "^12.4.0",
  "vue-loader": "^17.0.0",
  "webpack": "^5.68.0",
  "webpack-cli": "^4.9.2"
},
"dependencies": {
    "vue": "^3.2.29"
}
```
<p>And webpack.config.js</p>
<pre class="lang-js prettyprint-override"><code>const VueLoader = require('vue-loader');
const MiniCssExtractPlugin = require("mini-css-extract-plugin");

module.exports = {
    mode: 'production',
    stats: 'errors-warnings',
    entry: [
        './src/app.js',
    ],
    output: {
        filename: 'compiled.js',
        path: __dirname + '/js',
    },
    optimization: {
        minimize: true,
    },
    performance: {
        hints: 'warning',
        maxEntrypointSize: 250000, // JS output 250 kB
        maxAssetSize: 250000, // CSS output 250 kB
    },
    externals: {
        'vue': 'Vue',
        'vuex': 'Vuex',
        'vue-router': 'VueRouter',
    },
    module: {
        rules: [
            {
                test: /\.vue$/,
                loader: 'vue-loader',
            },
            {
                test: /\.m?js$/,
                resolve: {
                    fullySpecified: false,
                },
                use: {
                    loader: 'babel-loader',
                    options: {
                        presets: [
                            ['@babel/preset-env', {targets: '>1%'}],
                        ],
                    },
                },
            },
            {
                test: /\.s?css$/,
                use: [
                    MiniCssExtractPlugin.loader, // add support for `import 'file.scss';` in JS
                    {
                        loader: 'css-loader',
                        options: {
                            url: false, // whether to resolve urls; leave urls in the code as written
                        },
                    },
                    {
                        loader: 'sass-loader',
                        options: {
                            sassOptions: {
                                includePaths: [
                                    //__dirname + '/bower_components/bootstrap-sass/assets/stylesheets',
                                ],
                            },
                        },
                    },
                ],
            },
        ],
    },
    plugins: [
        new VueLoader.VueLoaderPlugin(),
        new MiniCssExtractPlugin({
            // Output destination for compiled CSS
            filename: '../css/compiled.css',
        }),
    ],
};

</code>```

And index.html loads Vue, VueRouter, Vuex, and then my bundled webapp:

```
<p>Then in my frontend code, bundled with webpack:</p>
<pre class="lang-js prettyprint-override"><code>import Vue from "vue";
import Vuex from "vuex";
import VueRouter from "vue-router";

console.log('client', Vue, Vuex, VueRouter);
</code>```

Logs healthcheck {...} {...} {...} in the HTML and then the compiled app logs client undefined, {...}, undefined (Vuex is defined because Vuex.__esModule is undefined)

Any ideas what to do?

答:
Answered here https://github.com/vuejs/core/issues/5380

Vue 3 only supports named imports, like import {createApp} from 'vue';

Body:
I have been having an issue for 2 days and finally isolated it. Importing Vue and VueRouter gives me undefined in my frontend code. This is because the output from webpack, using the externals property to load client-side libraries, checks for module.__esModule when you import an externally loaded library, and if true then it returns module.default. In this case Vue.__esModule and VueRouter.__esModule are true while Vue.default and VueRouter.default are undefined. I'm not even sure who to file a bug with, or whether there is something I can add to webpack.config.js to make this work as before. It could be Vue for including __esModule on their global builds. Or it could be something in the internals of webpack.

Here is a section of my package.json

"devDependencies": {
  "@babel/core": "^7.17.0",
  "@babel/preset-env": "^7.16.11",
  "@vue/compiler-sfc": "^3.2.30",
  "babel-loader": "^8.2.3",
  "css-loader": "^6.6.0",
  "mini-css-extract-plugin": "^2.5.3",
  "node-sass": "^7.0.1",
  "sass-loader": "^12.4.0",
  "vue-loader": "^17.0.0",
  "webpack": "^5.68.0",
  "webpack-cli": "^4.9.2"
},
"dependencies": {
    "vue": "^3.2.29"
}

And webpack.config.js

const VueLoader = require('vue-loader');
const MiniCssExtractPlugin = require("mini-css-extract-plugin");

module.exports = {
    mode: 'production',
    stats: 'errors-warnings',
    entry: [
        './src/app.js',
    ],
    output: {
        filename: 'compiled.js',
        path: __dirname + '/js',
    },
    optimization: {
        minimize: true,
    },
    performance: {
        hints: 'warning',
        maxEntrypointSize: 250000, // JS output 250 kB
        maxAssetSize: 250000, // CSS output 250 kB
    },
    externals: {
        'vue': 'Vue',
        'vuex': 'Vuex',
        'vue-router': 'VueRouter',
    },
    module: {
        rules: [
            {
                test: /\.vue$/,
                loader: 'vue-loader',
            },
            {
                test: /\.m?js$/,
                resolve: {
                    fullySpecified: false,
                },
                use: {
                    loader: 'babel-loader',
                    options: {
                        presets: [
                            ['@babel/preset-env', {targets: '>1%'}],
                        ],
                    },
                },
            },
            {
                test: /\.s?css$/,
                use: [
                    MiniCssExtractPlugin.loader, // add support for `import 'file.scss';` in JS
                    {
                        loader: 'css-loader',
                        options: {
                            url: false, // whether to resolve urls; leave urls in the code as written
                        },
                    },
                    {
                        loader: 'sass-loader',
                        options: {
                            sassOptions: {
                                includePaths: [
                                    //__dirname + '/bower_components/bootstrap-sass/assets/stylesheets',
                                ],
                            },
                        },
                    },
                ],
            },
        ],
    },
    plugins: [
        new VueLoader.VueLoaderPlugin(),
        new MiniCssExtractPlugin({
            // Output destination for compiled CSS
            filename: '../css/compiled.css',
        }),
    ],
};

And index.html loads Vue, VueRouter, Vuex, and then my bundled webapp:

Then in my frontend code, bundled with webpack:

import Vue from "vue";
import Vuex from "vuex";
import VueRouter from "vue-router";

console.log('client', Vue, Vuex, VueRouter);

Logs healthcheck {...} {...} {...} in the HTML and then the compiled app logs client undefined, {...}, undefined (Vuex is defined because Vuex.__esModule is undefined)

Any ideas what to do?


问:
Kafkajs - multiple consumers reading from same topic and partition
I'm planning to use Kafkajs https://kafka.js.org/ and implement it in a NodeJs server.
I would like to know what is the expected behavior in case I have 2 (or more) instances of the server running, each of them having a consumer which is configured with the same group id and topic?

Does this mean that they might read the same messages?
Should I specify a unique consumer group per each server instance ?

I read this - Multiple consumers consuming from same topic but not sure it applies for Kafkajs

答: It's not possible for a single consumer group to have multiple consumers reading from overlapping partitions in any Kafka library. If your topic only has one partition, only one instance of your application will be consuming it, and if that instance dies, the group will rebalance and the other instance will take over (potentially reading some of the same data, due to the nature of at-least-once delivery, but it's not at the same time as the other instance)


问:
PowerShell slow Invoke-Command
I'm writing a PowerShell-Script that reads all shares from all AD-Servers and outputs them into a csv-file. At the same time the script is saving all occuring errors and outputs them into an error-log. Can you guys see any possibility that could fasten up the whole process, because right now it takes quite a long time.

My code:

```<code>    function output1 {

        Get-Content C:\PowerShell\Shares\serverlist.txt | Where-Object { $_.name -like "*" } | ForEach-Object {

            $ErrorActionPreference = 'silentlycontinue'
            $name = $_ + ".domain.com"
            $pathnotnull = $_.Path -ne ""

            invoke-command -ComputerName $name -ArgumentList $name, $pathnotnull -ScriptBlock {

                param($name, $pathnotnull)
                Get-SmbShare | where-object { $_.Path -ne '' } | Select-Object -Property "Name", "Path" | get-acl | Select-Object -Property "PSChildName", "Path", "Group", "AccessToString"

            }

            if (!$?) {
                if ($error[0].exception.message) { Write-Host "Access to $_ failed!" -ForegroundColor Red }
                $error | Set-Content $error_logfile -Encoding Unicode
            }
            else {
                Write-Host "Access to $_ successful!" -ForegroundColor Green
                return $i
            }
        }

    }

    $i = output1
    $i | Out-Null
    $ErrorActionPreference = 'silentlycontinue'

    if ($i -ne $null) {
        $i | export-csv -path C:\PowerShell\Shares\shares.csv -NoTypeInformation -delimiter ";"
    }
    else {
        ""
        write-host "No server could be contacted!" -ForegroundColor Red
        ""
        openerrorlog1
    }
</code>```

答:
Your code could be reduced to this I believe, as mklement0 suggests, Invoke-Command can take an array of computers and invoke the same script block in parallel which will make your script run exponentially faster. I also don't see a need for a function, in this case. As for, capturing the errors from Invoke-Command you could redirect them (2>&1) to the success stream and then you can filter by -is [ErrorRecord]:

```<code>$ErrorActionPreference = 'Continue'

$computers = (Get-Content C:\PowerShell\Shares\serverlist.txt).ForEach({
    if(-not [string]::IsNullOrWhiteSpace($_))
    {
        "$_.domain.com"
    }
})

$remoteCode = {
    Get-SmbShare | Where-Object Path | Get-Acl |
    Select-Object -Property "PSChildName", "Path", "Group", "AccessToString"
}

$results = Invoke-Command -ComputerName $computers -ScriptBlock $remoteCode 2>&1
$errors, $good = $results.Where({$_ -is [System.Management.Automation.ErrorRecord]}, 'Split')

$good | Export-Csv .....

# Here are the Errros
$errors.Exception.Message
</code>```

Title: PowerShell slow Invoke-Command

Body:
I'm writing a PowerShell-Script that reads all shares from all AD-Servers and outputs them into a csv-file. At the same time the script is saving all occuring errors and outputs them into an error-log. Can you guys see any possibility that could fasten up the whole process, because right now it takes quite a long time.

My code:

    function output1 {

        Get-Content C:\PowerShell\Shares\serverlist.txt | Where-Object { $_.name -like "*" } | ForEach-Object {

            $ErrorActionPreference = 'silentlycontinue'
            $name = $_ + ".domain.com"
            $pathnotnull = $_.Path -ne ""

            invoke-command -ComputerName $name -ArgumentList $name, $pathnotnull -ScriptBlock {

                param($name, $pathnotnull)
                Get-SmbShare | where-object { $_.Path -ne '' } | Select-Object -Property "Name", "Path" | get-acl | Select-Object -Property "PSChildName", "Path", "Group", "AccessToString"

            }

            if (!$?) {
                if ($error[0].exception.message) { Write-Host "Access to $_ failed!" -ForegroundColor Red }
                $error | Set-Content $error_logfile -Encoding Unicode
            }
            else {
                Write-Host "Access to $_ successful!" -ForegroundColor Green
                return $i
            }
        }

    }

    $i = output1
    $i | Out-Null
    $ErrorActionPreference = 'silentlycontinue'

    if ($i -ne $null) {
        $i | export-csv -path C:\PowerShell\Shares\shares.csv -NoTypeInformation -delimiter ";"
    }
    else {
        ""
        write-host "No server could be contacted!" -ForegroundColor Red
        ""
        openerrorlog1
    }

Body:
Your code could be reduced to this I believe, as mklement0 suggests, Invoke-Command can take an array of computers and invoke the same script block in parallel which will make your script run exponentially faster. I also don't see a need for a function, in this case. As for, capturing the errors from Invoke-Command you could redirect them (2>&1) to the success stream and then you can filter by -is [ErrorRecord]:

$ErrorActionPreference = 'Continue'

$computers = (Get-Content C:\PowerShell\Shares\serverlist.txt).ForEach({
    if(-not [string]::IsNullOrWhiteSpace($_))
    {
        "$_.domain.com"
    }
})

$remoteCode = {
    Get-SmbShare | Where-Object Path | Get-Acl |
    Select-Object -Property "PSChildName", "Path", "Group", "AccessToString"
}

$results = Invoke-Command -ComputerName $computers -ScriptBlock $remoteCode 2>&1
$errors, $good = $results.Where({$_ -is [System.Management.Automation.ErrorRecord]}, 'Split')

$good | Export-Csv .....

# Here are the Errros
$errors.Exception.Message