问:
I am very new to MySQL and don't know why there are dupliacte items being added to each row?
This is currently all that I have simply because I wanted to test adding items to a table, and making sure that said item wasn't a duplicate. I saw that if you add INSERT IGNORE that it supposedly stops the creation of duplicated items but in my case, that didn't work.</p>
```<code>CREATE TABLE IF NOT EXISTS testing (test0 TEXT, test10 TEXT);

INSERT IGNORE INTO testing VALUES ("TEST1","TEST2");
INSERT IGNORE INTO testing VALUES ("TEST1","TEST2");
INSERT IGNORE INTO testing VALUES ("TEST1","TEST2");
INSERT IGNORE INTO testing VALUES ("TEST1","TEST2");
INSERT IGNORE INTO testing VALUES ("TEST1","TEST2");
</code>```

答:
The purpose of INSERT IGNORE, is to not interrupt the execution of insert statements if one fails for any reason.
For example if you try to insert string value in a Integer column, it would just ignore the error and insert nothing then.

I think in your case you need to create a UNIQUE constraint for desired fields :)

Body:
This is currently all that I have simply because I wanted to test adding items to a table, and making sure that said item wasn't a duplicate. I saw that if you add INSERT IGNORE that it supposedly stops the creation of duplicated items but in my case, that didn't work.

CREATE TABLE IF NOT EXISTS testing (test0 TEXT, test10 TEXT);

INSERT IGNORE INTO testing VALUES ("TEST1","TEST2");
INSERT IGNORE INTO testing VALUES ("TEST1","TEST2");
INSERT IGNORE INTO testing VALUES ("TEST1","TEST2");
INSERT IGNORE INTO testing VALUES ("TEST1","TEST2");
INSERT IGNORE INTO testing VALUES ("TEST1","TEST2");


来源: stackexchange

问:
I have a .txt file and I should scroll and print each element on the screen (list of strings)
I have a .txt file and I should scroll and print each element on the screen (list of strings).
I am new and inexperienced, can you please help me write the file ndex.cshtml (view)?

(Class.cs)

```<code>public class Reader
{
    public static List<string> Read(string path)
     {
            var result = new List<string>();
            String line;

        try
            {
                StreamReader sr = new StreamReader(path);
                line = sr.ReadLine();
            while (line != null)
                {
                    result.Add(line);
                    line = sr.ReadLine();

            }
            }
        catch (Exception e)
        {
            result = new List<string>();
        }

        return result;
    }
}
</code>```

}

(HomeController.cs)
{

```<code>public class HomeController : Controller
{
    public ActionResult Index()
    {
        var result = Reader.Reader.Read("......txt");

        return View(result);
    }
</code>```

答:
Depending on how you want to render the view...

View.cshtml

You can specify a List as your type of model if you wish instead of introducing your own abstractions.  Here is a partial view:

```<code>@model List<string>

@{
    Layout = null;
}

<ul>
@{
    foreach (string line in Model)
    {
        <li>@line</li>
    }
}
</ul>
</code>```

In the razor view you can iterate through the strings in the foreach.  Here I'm putting the output in an unordered list.

You can add c# code to execute in the view using the @ blocks.  @{ } will put in a block of code to execute.

Reading text file

You can also read all lines from the text file using:

```<code>string[] lines = System.IO.File.ReadAllLines(@"...filename.txt");
</code>```

Putting the full path to the filename.  Then you could change your type of model to Enumerable<string> as we are only enumerating through the collection.

ContentLicense: CC BY-SA 4.0

Body:
I have a .txt file and I should scroll and print each element on the screen (list of strings).
I am new and inexperienced, can you please help me write the file ndex.cshtml (view)?

(Class.cs)

public class Reader
{
    public static List<string> Read(string path)
     {
            var result = new List<string>();
            String line;

        try
            {
                StreamReader sr = new StreamReader(path);
                line = sr.ReadLine();
            while (line != null)
                {
                    result.Add(line);
                    line = sr.ReadLine();

            }
            }
        catch (Exception e)
        {
            result = new List<string>();
        }

        return result;
    }
}

}

(HomeController.cs)
{

public class HomeController : Controller
{
    public ActionResult Index()
    {
        var result = Reader.Reader.Read("......txt");

        return View(result);
    }

Body:
Depending on how you want to render the view...

View.cshtml

You can specify a List as your type of model if you wish instead of introducing your own abstractions.  Here is a partial view:

@model List<string>

@{
    Layout = null;
}

<ul>
@{
    foreach (string line in Model)
    {
        <li>@line</li>
    }
}
</ul>

In the razor view you can iterate through the strings in the foreach.  Here I'm putting the output in an unordered list.

You can add c# code to execute in the view using the @ blocks.  @{ } will put in a block of code to execute.

Reading text file

You can also read all lines from the text file using:

string[] lines = System.IO.File.ReadAllLines(@"...filename.txt");

Putting the full path to the filename.  Then you could change your type of model to Enumerable<string> as we are only enumerating through the collection.


问:
How to speed up HTML video on mobile
I have a website and it has a slider that contains a video tag. It worked on desktop well but it's so slowly on mobile.

In mobile, it's playing smaller(MB) video but it's so slowly. The problem website is https://www.drapebutik.com/

What can I do to speed it up?

答: Seems like you are loading all assets on page load making the total size almost 40mb. I would implement some sort of lazy loading of images and videos outside of the viewport. Also if you use Vimeo for serving your videos you would get the added benefit of adaptive bitrate.</p>

Title: How to speed up HTML video on mobile


问:
WebGL Publishing Error on Ubuntu 20.04 and Unity
I am currently working on the Unity Learn Junior Programmer course, and I am starting to run into issues publishing my Unit final. The final project is to post a modded version of a cube rotation file on Unity Play via WebGL, but every time that I try to build the project, it fails. This has been going on for the past week or so. Every time, I get the following error:

```<code>    System.ComponentModel.Win32Exception (2): No such file or directory
at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
at System.Diagnostics.Process.Start()
at Unity.IL2CPP.Shell.SetupAndStart(ExecuteContext context, Boolean asyncMode) in /Users/bokken/build/output/unity/il2cpp/Unity.IL2CPP.Shell/Shell.cs:line 423
at Unity.IL2CPP.Shell.ExecuteAsync(ExecuteArgs executeArgs, IExecuteController controller, Boolean asyncMode) in /Users/bokken/build/output/unity/il2cpp/Unity.IL2CPP.Shell/Shell.cs:line 299
at Unity.IL2CPP.Shell.Execute(ExecuteArgs executeArgs, IExecuteController controller) in /Users/bokken/build/output/unity/il2cpp/Unity.IL2CPP.Shell/Shell.cs:line 355
at Unity.IL2CPP.Building.CppProgramBuilder.ProvideObjectFile(IntermediateObjectFileCompilationData data) in /Users/bokken/build/output/unity/il2cpp/Unity.IL2CPP.Building/CppProgramBuilder.cs:line 334
at Unity.IL2CPP.Building.ParallelFor.<>c__DisplayClass1_0`2.<RunWithResult>b__0(Object o) in /Users/bokken/build/output/unity/il2cpp/Unity.IL2CPP.Building/ParallelFor.cs:line 78
</code>```

I already have the WebGL publisher installed and the build setup switched to WebGL, so do you have any ideas on how to fix this?

答:
Apparently that is a known issue which will not be fixed according to Unitys bug-tracker:

See Issue here

It suggests though that upgrading your Unity version to 2021.2.0b5 or higher might fix the issue as it is marked as "not reproducible" with said versions.

Body:
I am currently working on the Unity Learn Junior Programmer course, and I am starting to run into issues publishing my Unit final. The final project is to post a modded version of a cube rotation file on Unity Play via WebGL, but every time that I try to build the project, it fails. This has been going on for the past week or so. Every time, I get the following error:

    System.ComponentModel.Win32Exception (2): No such file or directory
at System.Diagnostics.Process.ForkAndExecProcess(String filename, String[] argv, String[] envp, String cwd, Boolean redirectStdin, Boolean redirectStdout, Boolean redirectStderr, Boolean setCredentials, UInt32 userId, UInt32 groupId, UInt32[] groups, Int32& stdinFd, Int32& stdoutFd, Int32& stderrFd, Boolean usesTerminal, Boolean throwOnNoExec)
at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
at System.Diagnostics.Process.Start()
at Unity.IL2CPP.Shell.SetupAndStart(ExecuteContext context, Boolean asyncMode) in /Users/bokken/build/output/unity/il2cpp/Unity.IL2CPP.Shell/Shell.cs:line 423
at Unity.IL2CPP.Shell.ExecuteAsync(ExecuteArgs executeArgs, IExecuteController controller, Boolean asyncMode) in /Users/bokken/build/output/unity/il2cpp/Unity.IL2CPP.Shell/Shell.cs:line 299
at Unity.IL2CPP.Shell.Execute(ExecuteArgs executeArgs, IExecuteController controller) in /Users/bokken/build/output/unity/il2cpp/Unity.IL2CPP.Shell/Shell.cs:line 355
at Unity.IL2CPP.Building.CppProgramBuilder.ProvideObjectFile(IntermediateObjectFileCompilationData data) in /Users/bokken/build/output/unity/il2cpp/Unity.IL2CPP.Building/CppProgramBuilder.cs:line 334
at Unity.IL2CPP.Building.ParallelFor.<>c__DisplayClass1_0`2.<RunWithResult>b__0(Object o) in /Users/bokken/build/output/unity/il2cpp/Unity.IL2CPP.Building/ParallelFor.cs:line 78

I already have the WebGL publisher installed and the build setup switched to WebGL, so do you have any ideas on how to fix this?


问:
Why do my list item bullets overlap floating elements
I have an (XHTML Strict) page where I float an image alongside regular paragraphs of text. All goes well, except when a list is used instead of paragraphs. The bullets of the list overlap the floated image.

Changing the margin of the list or the list items does not help. The margin is calculated from the left of the page, but the float pushes the list items to the right inside the li itself. So the margin only helps if I make it wider than the image.

Floating the list next to the image also works, but I don't know when the list is next to a float. I don't want to float every list in my content just to fix this. Also, floating left messes up the layout when an image is floated to the right instead of left of the list.

Setting li { list-style-position: inside } does move the bullets along with the content, but it also causes lines that wrap to start aligned with the bullet, instead of aligned with the line above.

The problem is obviously caused by the bullet being rendered outside the box, the float pushing the contents of the box to the right (not the box itself). This is how IE and FF handle the situation, and as far as I know, not wrong according to the spec. The question is, how can I prevent it?

答: Try list-style-position: inside to change the layout of the bullets.</p>

OwnerDisplayName: Kamiel Wanrooij

ContentLicense: CC BY-SA 3.0

LastEditorDisplayName: Kamiel Wanrooij

OwnerDisplayName: annakata

ContentLicense: CC BY-SA 2.5


问:
Celery tasks get blocked by SSE client
I am running a SSE consumer in order to update a few tables like this

```<code>@app.task(ignore_result=True)
@worker_ready.connect
def xx_sse_nodes_uptime_info_consumer(sender, **kwargs):
   update_tables...
</code>```

By some reason, other tasks are not triggered when we have a .delay calling other tasks. If I remove the task above, other tasks will work normally.

Also, when CELERY_TASK_ALWAYS_EAGER=True and CELERY_TASK_EAGER_PROPAGATES=True everything runs fine.

Since the SSE consumer uses stream=True the xx_sse_nodes_uptime_info_consumer never ends. I am not sure why this blocks others tasks tho as there are enough threads.

答:
Solved creating a function for that</p>
```<code>@worker_ready.connect
def at_worker_ready(sender, **k):
    with sender.app.connection() as conn:
        sender.app.send_task('nodeinfo.tasks.xx_sse_nodes_uptime_info_consumer', connection=conn)
</code>```

Title: Celery tasks get blocked by SSE client

Body:
I am running a SSE consumer in order to update a few tables like this

@app.task(ignore_result=True)
@worker_ready.connect
def xx_sse_nodes_uptime_info_consumer(sender, **kwargs):
   update_tables...

By some reason, other tasks are not triggered when we have a .delay calling other tasks. If I remove the task above, other tasks will work normally.

Also, when CELERY_TASK_ALWAYS_EAGER=True and CELERY_TASK_EAGER_PROPAGATES=True everything runs fine.

Since the SSE consumer uses stream=True the xx_sse_nodes_uptime_info_consumer never ends. I am not sure why this blocks others tasks tho as there are enough threads.

Body:
Solved creating a function for that

@worker_ready.connect
def at_worker_ready(sender, **k):
    with sender.app.connection() as conn:
        sender.app.send_task('nodeinfo.tasks.xx_sse_nodes_uptime_info_consumer', connection=conn)


问:
How to cancel a scheduled job in sidekiq. How to fetch jid of a scheduled job from my code?
job = HardWorker.perform_in(2.minutes, arg1, arg2)

j_id = Sidekiq::ScheduledSet.new.find { |job_id| job.job_id }.item["jid"]

I get the following error :
NoMethodError: undefined method `job_id' for "6512497a4359600501f63038":String

What is the correct way to fetch job id and delete the scheduled job?

答:
From Sidekiq wiki FAQ

How do I cancel a Sidekiq job?

Sidekiq does not provide this functionality; it's safer and better for the application to do it. You should implement something like this:

```<code>class MyWorker
  include Sidekiq::Worker

  def perform(args)
    return if cancelled?
    # do stuff
  end

  def cancelled?
    Sidekiq.redis {|c| c.exists("cancelled-#{jid}") } # Use c.exists? on Redis >= 4.2.0
  end

  def self.cancel!(jid)
    Sidekiq.redis {|c| c.setex("cancelled-#{jid}", 86400, 1) }
  end
end
</code>```

Body:
From Sidekiq wiki FAQ

How do I cancel a Sidekiq job?

Sidekiq does not provide this functionality; it's safer and better for the application to do it. You should implement something like this:

class MyWorker
  include Sidekiq::Worker

  def perform(args)
    return if cancelled?
    # do stuff
  end

  def cancelled?
    Sidekiq.redis {|c| c.exists("cancelled-#{jid}") } # Use c.exists? on Redis >= 4.2.0
  end

  def self.cancel!(jid)
    Sidekiq.redis {|c| c.setex("cancelled-#{jid}", 86400, 1) }
  end
end


问:
swift protoc conversion fails with error "google.type.LatLng" is not defined
I am converting .proto files to swift files using command

```<code>protoc --swift_out=. my_proto_file.proto
</code>```

protoc version being used is libprotoc 3.13.0.

my_proto_file has several import statements like

```<code>import "google/protobuf/duration.proto";
import "google/type/latlng.proto";
</code>```

When I run protoc on this proto file I get error like

```<code>trackProto.proto:8:1: Import "google/type/latlng.proto" was not found or had errors.
trackProto.proto:153:3: "google.type.LatLng" is not defined.
trackProto.proto:169:3: "google.type.LatLng" is not defined.
trackProto.proto:174:3: "google.type.LatLng" is not defined.
</code>```

I believe the dependency is on google/type repo. These are not the well known types like google.protobuf.Any, google.protobuf.Api etc so protoc fails to find the dependency and hence fails to generate .swift files

How can I resolve this dependency ?

答:
I downloaded google/type/latlong.proto from https://github.com/googleapis/googleapis/blob/master/google/type/latlng.proto and put that in a local folder e.g. C:\grpc_includes. I repeated for other protos I needed to import e.g. google/protobuf/timestamp.proto. I matched the folder structure indicated by the path e.g. C:\grpc_includes\google\type\latlong.proto.

Then I added --path_proto=<path to my .proto> --path_proto=C:\grpc_includes to my protoc command line. -I=<path to my .proto> -I=C:\grpc_includes also works.

This allowed protoc to find the imports. To use the resultant types of the imports I needed to include the namespace e.g. google.type.LatLng and google.protobuf.Timestamp when using them in my .proto file.

There is some useful information that lead me to the solution here: https://developers.google.com/protocol-buffers/docs/proto?csw=1#generating and here: https://developers.google.com/protocol-buffers/docs/proto3#importing_definitions

Body:
I am converting .proto files to swift files using command

protoc --swift_out=. my_proto_file.proto

protoc version being used is libprotoc 3.13.0.

my_proto_file has several import statements like

import "google/protobuf/duration.proto";
import "google/type/latlng.proto";

When I run protoc on this proto file I get error like

trackProto.proto:8:1: Import "google/type/latlng.proto" was not found or had errors.
trackProto.proto:153:3: "google.type.LatLng" is not defined.
trackProto.proto:169:3: "google.type.LatLng" is not defined.
trackProto.proto:174:3: "google.type.LatLng" is not defined.

I believe the dependency is on google/type repo. These are not the well known types like google.protobuf.Any, google.protobuf.Api etc so protoc fails to find the dependency and hence fails to generate .swift files

How can I resolve this dependency ?


问:
How to implement shouldChangeValue(to: Int) for UISlider?
I simply have UISlider with minValue 0 and maxValue 100. My currentValue is 40.

I have 6 sliders with custom minValue and maxValue for each of them.

I also have some kind of validation (total sum should not be greater than 200).

Is there a way to check if currentValue from @IBAction

```<code>@objc private func didValueChanged(slider: UISlider) {

    // check if it is not over 200,
    // if not then set previous value.
    // How do I get previous value of slider?
}
</code>```

is not over my limit and do nothing then otherwise set a previous value to slider?

答:
It's not clear exactly what you want to do when the sum of the sliders exceeds the 200 limit, but the general idea...

You'll want to track each slider's last valid value. When the user stops dragging:

get the value of the "active" slider

get sum of the other 5 sliders

add the "active" value to the sum

if the sum is out-of-range (sum > 200)

reset to the active slider's saved value

else

update the active slider's saved value

So, give your sliders this action:

```<code>slider.addTarget(self, action: #selector(didStopSliding(slider:)), for: [.touchUpInside, .touchUpOutside, .touchCancel])
</code>```

which will be called when the users stops sliding (lifts touch or slides off-screen or action is cancelled).

You can keep your current:

```<code>slider.addTarget(self, action: #selector(didValueChanged(slider:)), for: .valueChanged)
</code>```

if you also want to do something while dragging (such as updating value label(s)).

Here's a quick example:

```<code>// struct to associate values and display labels with the UISlider
struct MySliderStruct {
    var minValue: Float = 0
    var maxValue: Float = 0
    var curValue: Float = 0
    var minLabel: UILabel = UILabel()
    var curLabel: UILabel = UILabel()
    var maxLabel: UILabel = UILabel()
    var theSlider: UISlider = UISlider()
}

class SliderCheckVC: UIViewController {

    var theSliders: [MySliderStruct] = []

    let runningTotalLabel: UILabel = {
        let v = UILabel()
        v.textAlignment = .center
        return v
    }()

    override func viewDidLoad() {
        super.viewDidLoad()

        let minMax: [[Float]] = [
            [10, 40],
            [20, 50],
            [0, 10],
            [0, 50],
            [10, 60],
            [0, 100],
        ]
        let colors: [UIColor] = [
            .init(red: 1.0, green: 0.8, blue: 0.8, alpha: 1.0),
            .init(red: 0.5, green: 1.0, blue: 0.5, alpha: 1.0),
            .init(red: 0.8, green: 0.8, blue: 1.0, alpha: 1.0),
            .init(red: 1.0, green: 1.0, blue: 0.0, alpha: 1.0),
            .init(red: 0.0, green: 1.0, blue: 0.0, alpha: 1.0),
            .init(red: 0.0, green: 1.0, blue: 1.0, alpha: 1.0),
        ]

        let outerStack: UIStackView = {
            let v = UIStackView()
            v.axis = .vertical
            v.spacing = 16
            v.translatesAutoresizingMaskIntoConstraints = false
            return v
        }()

        for (pair, clr) in zip(minMax, colors) {

            // vertical stack view for labels + slider
            let vStack: UIStackView = {
                let v = UIStackView()
                v.axis = .vertical
                v.spacing = 4
                return v
            }()
            // horizontal stack view for the labels
            let hStack: UIStackView = {
                let v = UIStackView()
                v.distribution = .fillEqually
                return v
            }()
            let vMin: UILabel = {
                let v = UILabel()
                v.textAlignment = .left
                v.font = .systemFont(ofSize: 13.0)
                v.text = formatVal(pair[0])
                return v
            }()
            let vCur: UILabel = {
                let v = UILabel()
                v.textAlignment = .center
                v.font = .systemFont(ofSize: 13.0)
                v.text = formatVal(pair[0])
                return v
            }()
            let vMax: UILabel = {
                let v = UILabel()
                v.textAlignment = .right
                v.font = .systemFont(ofSize: 13.0)
                v.text = formatVal(pair[1])
                return v
            }()
            hStack.addArrangedSubview(vMin)
            hStack.addArrangedSubview(vCur)
            hStack.addArrangedSubview(vMax)

            let slider = UISlider()
            slider.minimumValue = pair[0]
            slider.maximumValue = pair[1]
            slider.setValue(pair[0], animated: false)
            slider.addTarget(self, action: #selector(didValueChanged(slider:)), for: .valueChanged)
            slider.addTarget(self, action: #selector(didStopSliding(slider:)), for: [.touchUpInside, .touchUpOutside, .touchCancel])

            vStack.backgroundColor = clr

            vStack.addArrangedSubview(hStack)
            vStack.addArrangedSubview(slider)
            outerStack.addArrangedSubview(vStack)

            let thisSlider: MySliderStruct = MySliderStruct(minValue: pair[0],
                                                            maxValue: pair[1],
                                                            curValue: pair[0],
                                                            minLabel: vMin,
                                                            curLabel: vCur,
                                                            maxLabel: vMax,
                                                            theSlider: slider
            )
            theSliders.append(thisSlider)
        }

        // running total line
        let hStack: UIStackView = {
            let v = UIStackView()
            v.distribution = .fillEqually
            return v
        }()
        let v1 = UILabel()
        v1.text = "Current Total"
        v1.textAlignment = .center
        runningTotalLabel.textAlignment = .center
        hStack.addArrangedSubview(v1)
        hStack.addArrangedSubview(runningTotalLabel)
        outerStack.addArrangedSubview(hStack)

        view.addSubview(outerStack)

        let g = view.safeAreaLayoutGuide
        NSLayoutConstraint.activate([
            outerStack.topAnchor.constraint(equalTo: g.topAnchor, constant: 12.0),
            outerStack.leadingAnchor.constraint(equalTo: g.leadingAnchor, constant: 20.0),
            outerStack.trailingAnchor.constraint(equalTo: g.trailingAnchor, constant: -20.0),
        ])

        updateTotalLabel()
    }

    @objc private func didStopSliding(slider: UISlider) {

        guard let thisSliderIDX = theSliders.firstIndex(where: { $0.theSlider == slider})
        else {
            print("Invalide slider setup")
            return
        }

        // get the value of "active" slider
        let thisValue: Float = slider.value.rounded()

        // get sum of all other sliders, plus
        //  this slider value
        var total: Float = 0
        for i in 0..<theSliders.count {
            if i == thisSliderIDX {
                total += thisValue
            } else {
                total += theSliders[i].curValue
            }
        }

        // if sum is less-than max allowed
        if total < 200.0 {
            // update array with new value
            theSliders[thisSliderIDX].curValue = thisValue
            // move slider to rounded Int value
            slider.setValue(thisValue, animated: true)
        } else {
            // reset to last value
            slider.setValue(theSliders[thisSliderIDX].curValue, animated: true)
            // update this slider's current value label
            theSliders[thisSliderIDX].curLabel.text = formatVal(theSliders[thisSliderIDX].curValue)
        }

        updateTotalLabel()

    }
    @objc private func didValueChanged(slider: UISlider) {
        // do something as the slider is dragged
        guard let thisSliderIDX = theSliders.firstIndex(where: { $0.theSlider == slider})
        else {
            print("Invalide slider setup")
            return
        }

        // get the value of "active" slider
        let thisValue: Float = slider.value.rounded()

        // update this slider's current value label
        theSliders[thisSliderIDX].curLabel.text = formatVal(thisValue)

        // get sum of all other sliders, plus
        //  this slider value
        var total: Float = 0
        for i in 0..<theSliders.count {
            if i == thisSliderIDX {
                total += thisValue
            } else {
                total += theSliders[i].curValue
            }
        }

        // update the running total label here, because
        //  we might throw out this value
        runningTotalLabel.text = formatVal(total)
    }

    func updateTotalLabel() {
        let sumOfValues = theSliders.reduce(0) { $0 + ($1.curValue ) }
        runningTotalLabel.text = formatVal(sumOfValues)
    }
    func formatVal(_ val: Float) -> String {
        return "\(Int(val))"
    }
}
</code>```

Looks like this:

On "done sliding" if that slider is dragged to a value that will cause the sum to exceed 200, that slider will be reset to it's last "valid" value.

Body:
I simply have UISlider with minValue 0 and maxValue 100. My currentValue is 40.

I have 6 sliders with custom minValue and maxValue for each of them.

I also have some kind of validation (total sum should not be greater than 200).

Is there a way to check if currentValue from @IBAction

@objc private func didValueChanged(slider: UISlider) {

    // check if it is not over 200,
    // if not then set previous value.
    // How do I get previous value of slider?
}

is not over my limit and do nothing then otherwise set a previous value to slider?

Body:
It's not clear exactly what you want to do when the sum of the sliders exceeds the 200 limit, but the general idea...

You'll want to track each slider's last valid value. When the user stops dragging:

get the value of the "active" slider

get sum of the other 5 sliders

add the "active" value to the sum

if the sum is out-of-range (sum > 200)

reset to the active slider's saved value

else

update the active slider's saved value

So, give your sliders this action:

slider.addTarget(self, action: #selector(didStopSliding(slider:)), for: [.touchUpInside, .touchUpOutside, .touchCancel])

which will be called when the users stops sliding (lifts touch or slides off-screen or action is cancelled).

You can keep your current:

slider.addTarget(self, action: #selector(didValueChanged(slider:)), for: .valueChanged)

if you also want to do something while dragging (such as updating value label(s)).

Here's a quick example:

// struct to associate values and display labels with the UISlider
struct MySliderStruct {
    var minValue: Float = 0
    var maxValue: Float = 0
    var curValue: Float = 0
    var minLabel: UILabel = UILabel()
    var curLabel: UILabel = UILabel()
    var maxLabel: UILabel = UILabel()
    var theSlider: UISlider = UISlider()
}

class SliderCheckVC: UIViewController {

    var theSliders: [MySliderStruct] = []

    let runningTotalLabel: UILabel = {
        let v = UILabel()
        v.textAlignment = .center
        return v
    }()

    override func viewDidLoad() {
        super.viewDidLoad()

        let minMax: [[Float]] = [
            [10, 40],
            [20, 50],
            [0, 10],
            [0, 50],
            [10, 60],
            [0, 100],
        ]
        let colors: [UIColor] = [
            .init(red: 1.0, green: 0.8, blue: 0.8, alpha: 1.0),
            .init(red: 0.5, green: 1.0, blue: 0.5, alpha: 1.0),
            .init(red: 0.8, green: 0.8, blue: 1.0, alpha: 1.0),
            .init(red: 1.0, green: 1.0, blue: 0.0, alpha: 1.0),
            .init(red: 0.0, green: 1.0, blue: 0.0, alpha: 1.0),
            .init(red: 0.0, green: 1.0, blue: 1.0, alpha: 1.0),
        ]

        let outerStack: UIStackView = {
            let v = UIStackView()
            v.axis = .vertical
            v.spacing = 16
            v.translatesAutoresizingMaskIntoConstraints = false
            return v
        }()

        for (pair, clr) in zip(minMax, colors) {

            // vertical stack view for labels + slider
            let vStack: UIStackView = {
                let v = UIStackView()
                v.axis = .vertical
                v.spacing = 4
                return v
            }()
            // horizontal stack view for the labels
            let hStack: UIStackView = {
                let v = UIStackView()
                v.distribution = .fillEqually
                return v
            }()
            let vMin: UILabel = {
                let v = UILabel()
                v.textAlignment = .left
                v.font = .systemFont(ofSize: 13.0)
                v.text = formatVal(pair[0])
                return v
            }()
            let vCur: UILabel = {
                let v = UILabel()
                v.textAlignment = .center
                v.font = .systemFont(ofSize: 13.0)
                v.text = formatVal(pair[0])
                return v
            }()
            let vMax: UILabel = {
                let v = UILabel()
                v.textAlignment = .right
                v.font = .systemFont(ofSize: 13.0)
                v.text = formatVal(pair[1])
                return v
            }()
            hStack.addArrangedSubview(vMin)
            hStack.addArrangedSubview(vCur)
            hStack.addArrangedSubview(vMax)

            let slider = UISlider()
            slider.minimumValue = pair[0]
            slider.maximumValue = pair[1]
            slider.setValue(pair[0], animated: false)
            slider.addTarget(self, action: #selector(didValueChanged(slider:)), for: .valueChanged)
            slider.addTarget(self, action: #selector(didStopSliding(slider:)), for: [.touchUpInside, .touchUpOutside, .touchCancel])

            vStack.backgroundColor = clr

            vStack.addArrangedSubview(hStack)
            vStack.addArrangedSubview(slider)
            outerStack.addArrangedSubview(vStack)

            let thisSlider: MySliderStruct = MySliderStruct(minValue: pair[0],
                                                            maxValue: pair[1],
                                                            curValue: pair[0],
                                                            minLabel: vMin,
                                                            curLabel: vCur,
                                                            maxLabel: vMax,
                                                            theSlider: slider
            )
            theSliders.append(thisSlider)
        }

        // running total line
        let hStack: UIStackView = {
            let v = UIStackView()
            v.distribution = .fillEqually
            return v
        }()
        let v1 = UILabel()
        v1.text = "Current Total"
        v1.textAlignment = .center
        runningTotalLabel.textAlignment = .center
        hStack.addArrangedSubview(v1)
        hStack.addArrangedSubview(runningTotalLabel)
        outerStack.addArrangedSubview(hStack)

        view.addSubview(outerStack)

        let g = view.safeAreaLayoutGuide
        NSLayoutConstraint.activate([
            outerStack.topAnchor.constraint(equalTo: g.topAnchor, constant: 12.0),
            outerStack.leadingAnchor.constraint(equalTo: g.leadingAnchor, constant: 20.0),
            outerStack.trailingAnchor.constraint(equalTo: g.trailingAnchor, constant: -20.0),
        ])

        updateTotalLabel()
    }

    @objc private func didStopSliding(slider: UISlider) {

        guard let thisSliderIDX = theSliders.firstIndex(where: { $0.theSlider == slider})
        else {
            print("Invalide slider setup")
            return
        }

        // get the value of "active" slider
        let thisValue: Float = slider.value.rounded()

        // get sum of all other sliders, plus
        //  this slider value
        var total: Float = 0
        for i in 0..<theSliders.count {
            if i == thisSliderIDX {
                total += thisValue
            } else {
                total += theSliders[i].curValue
            }
        }

        // if sum is less-than max allowed
        if total < 200.0 {
            // update array with new value
            theSliders[thisSliderIDX].curValue = thisValue
            // move slider to rounded Int value
            slider.setValue(thisValue, animated: true)
        } else {
            // reset to last value
            slider.setValue(theSliders[thisSliderIDX].curValue, animated: true)
            // update this slider's current value label
            theSliders[thisSliderIDX].curLabel.text = formatVal(theSliders[thisSliderIDX].curValue)
        }

        updateTotalLabel()

    }
    @objc private func didValueChanged(slider: UISlider) {
        // do something as the slider is dragged
        guard let thisSliderIDX = theSliders.firstIndex(where: { $0.theSlider == slider})
        else {
            print("Invalide slider setup")
            return
        }

        // get the value of "active" slider
        let thisValue: Float = slider.value.rounded()

        // update this slider's current value label
        theSliders[thisSliderIDX].curLabel.text = formatVal(thisValue)

        // get sum of all other sliders, plus
        //  this slider value
        var total: Float = 0
        for i in 0..<theSliders.count {
            if i == thisSliderIDX {
                total += thisValue
            } else {
                total += theSliders[i].curValue
            }
        }

        // update the running total label here, because
        //  we might throw out this value
        runningTotalLabel.text = formatVal(total)
    }

    func updateTotalLabel() {
        let sumOfValues = theSliders.reduce(0) { $0 + ($1.curValue ) }
        runningTotalLabel.text = formatVal(sumOfValues)
    }
    func formatVal(_ val: Float) -> String {
        return "\(Int(val))"
    }
}

Looks like this:

On "done sliding" if that slider is dragged to a value that will cause the sum to exceed 200, that slider will be reset to it's last "valid" value.


问:
Using pyspark to run a job on premises spark cluster
I have a tiny on premises Spark 3.2.0 cluster, with one machine being master, and another 2 being slaves. The cluster is deployed on "bare metal" and everything works fine when I run pyspark from the master machine.

The problem happens when I try to run anything from another machine. Here is my code:

```<code>import pandas as pd
from datetime import datetime
from pyspark.sql import SparkSession, functions

spark = SparkSession.builder.appName("extrair_comex").config("spark.executor.memory", "1g").master("spark://srvsparkm-dev:7077").getOrCreate()

link = 'https://www.stats.govt.nz/assets/Uploads/International-trade/International-trade-September-2021-quarter/Download-data/overseas-trade-indexes-September-2021-quarter-provisional-csv.csv'

arquivo = pd.read_csv(link)

df_spark = spark.createDataFrame(arquivo.astype(str))

df_spark.write.mode('overwrite').parquet(f'hdfs://srvsparkm-dev:9000/lnd/arquivo_extraido_comex.parquet')
</code>```

Where "srvsparkm-dev" is an alias for the spark master IP.

Checking the logs for the "extrair_comex" job, I see this:

The Spark Executor Command:

Spark Executor Command: "/usr/lib/jvm/java-8-openjdk-amd64/bin/java" "-cp" "/home/spark/spark/conf/:/home/spark/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=38571" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@srvairflowcelery-dev:38571" "--executor-id" "157" "--hostname" "srvsparksl1-dev" "--cores" "2" "--app-id" "app-20220204183041-0031" "--worker-url" "spark://Worker@srvsparksl1-dev:37383"

The error:

Where "srvairflowcelery-dev" is the machine where the pyspark script is running.

```<code>Caused by: java.io.IOException: Failed to connect to srvairflowcelery-dev/xx.xxx.xxx.xx:38571
</code>```

Where xx.xxx.xxx.xx is the srvairflowcelery-dev's IP.

It seems to me that the master is assigning to the client to run the task, and that's why it fails.
What can I do about this? Can't I submit jobs from another machine?

答:
I solved the problem. The problem was that the srvairflowcelery is on docker, so only some ports are open. Other than that, the spark master tries to communicate on a random port of the driver (srvairflowcelery), so having some ports closed is a problem.

What I did was:

Opened a range of ports of my airflow workers with:

```<code>airflow-worker:
    <<: *airflow-common
    command: celery worker
    hostname: ${HOSTNAME}
    ports:
      - 8793:8793
      - "51800-51900:51800-51900"

</code>```

Setting on my pyspark code fixed ports:

```<code>spark = SparkSession.builder.appName("extrair_comex_sb") \
        .config("spark.executor.memory", "1g") \
        .config("spark.driver.port", "51810") \
        .config("spark.fileserver.port", "51811") \
        .config("spark.broadcast.port", "51812") \
        .config("spark.replClassServer.port", "51813") \
        .config("spark.blockManager.port", "51814") \
        .config("spark.executor.port", "51815") \
        .master("spark://srvsparkm-dev:7077") \
        .getOrCreate()
</code>```

That fixed the problem.

Body:
I have a tiny on premises Spark 3.2.0 cluster, with one machine being master, and another 2 being slaves. The cluster is deployed on "bare metal" and everything works fine when I run pyspark from the master machine.

The problem happens when I try to run anything from another machine. Here is my code:

import pandas as pd
from datetime import datetime
from pyspark.sql import SparkSession, functions

spark = SparkSession.builder.appName("extrair_comex").config("spark.executor.memory", "1g").master("spark://srvsparkm-dev:7077").getOrCreate()

link = 'https://www.stats.govt.nz/assets/Uploads/International-trade/International-trade-September-2021-quarter/Download-data/overseas-trade-indexes-September-2021-quarter-provisional-csv.csv'

arquivo = pd.read_csv(link)

df_spark = spark.createDataFrame(arquivo.astype(str))

df_spark.write.mode('overwrite').parquet(f'hdfs://srvsparkm-dev:9000/lnd/arquivo_extraido_comex.parquet')

Where "srvsparkm-dev" is an alias for the spark master IP.

Checking the logs for the "extrair_comex" job, I see this:

The Spark Executor Command:

Spark Executor Command: "/usr/lib/jvm/java-8-openjdk-amd64/bin/java" "-cp" "/home/spark/spark/conf/:/home/spark/spark/jars/*" "-Xmx1024M" "-Dspark.driver.port=38571" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@srvairflowcelery-dev:38571" "--executor-id" "157" "--hostname" "srvsparksl1-dev" "--cores" "2" "--app-id" "app-20220204183041-0031" "--worker-url" "spark://Worker@srvsparksl1-dev:37383"

The error:

Where "srvairflowcelery-dev" is the machine where the pyspark script is running.

Caused by: java.io.IOException: Failed to connect to srvairflowcelery-dev/xx.xxx.xxx.xx:38571

Where xx.xxx.xxx.xx is the srvairflowcelery-dev's IP.

It seems to me that the master is assigning to the client to run the task, and that's why it fails.
What can I do about this? Can't I submit jobs from another machine?

Body:
I solved the problem. The problem was that the srvairflowcelery is on docker, so only some ports are open. Other than that, the spark master tries to communicate on a random port of the driver (srvairflowcelery), so having some ports closed is a problem.

What I did was:

Opened a range of ports of my airflow workers with:

airflow-worker:
    <<: *airflow-common
    command: celery worker
    hostname: ${HOSTNAME}
    ports:
      - 8793:8793
      - "51800-51900:51800-51900"

Setting on my pyspark code fixed ports:

spark = SparkSession.builder.appName("extrair_comex_sb") \
        .config("spark.executor.memory", "1g") \
        .config("spark.driver.port", "51810") \
        .config("spark.fileserver.port", "51811") \
        .config("spark.broadcast.port", "51812") \
        .config("spark.replClassServer.port", "51813") \
        .config("spark.blockManager.port", "51814") \
        .config("spark.executor.port", "51815") \
        .master("spark://srvsparkm-dev:7077") \
        .getOrCreate()

That fixed the problem.


问:
Msmdpump.dll OLAP data pump is throwing a 500 error
How do I resolve this issue with requests to msmdpump.dll for connections to SQL Server Analysis Services? I am receiving a 500 Error from the IsapiModule.

On a Windows Server 2012 R2 machine, with IIS 8.5, I have setup the OLAP data pump (msmdpump.dll), using the following instructions: https://msdn.microsoft.com/en-us/library/gg492140.aspx#bkmk_copy

The application pool is configured for .NET CLR v4.0, with Classic Managed pipeline mode. The identity is set to a local service account. (I have also tried a domain account, and I have tried making the local user an Administrator).

I've created an application under the Default Web Site, called OLAP, with an IsapiModule, as per the MSDN article.

As far as I can tell (and I've double and triple checked), everything is configured as laid out in the MSDN article. Also, compared to another server where I have this setup (on a different network), it is essentially the same.

When I request http://localhost/OLAP/msmdpump.dll in a browser on that machine, I receive a 500 Internal Server Error. The error indicates that it is trying to use the OLAP handler that I created. This is not the same error that I would normally expect when doing a GET request to msmdpump.dll. The normal error for a straight GET, when everything is working correctly, is sent back in a SOAP envelope. In my case, the request does not appear to ever be processed by msmdpump.dll.

500 Internal Server Error via browser:

(see below for full screenshot)

```<code>Module   IsapiModule
Notification   ExecuteRequestHandler
Handler   OLAP
Error Code   0x8007007e

Requested URL   http://localhost:80/OLAP/msmdpump.dll
Physical Path   C:\inetpub\wwwroot\OLAP\msmdpump.dll
Logon Method   Anonymous
Logon User   Anonymous
</code>```

500 Internal Server Error via SSMS connection:

I also receive an error when trying to connect to the data pump via SQL Server Management Studio:

Screenshot of the 500 error in the browser:

One appreciable difference between the machine I'm setting up, and the server where the data pump already works, is that there are a few more roles setup on the new server.

The problem server includes:

.NET Extensibility 4.5

ASP.NET 4.5

While the other machine (where the data pump works), does not include those roles. Would the presence of ASP.NET 4.5 or .NET Extensibility 4.5 cause an issue with IIS serving requests for this IsapiModule?

答: This is the perfect article, as expected to see on stackoverflow! I even thought that this case is also mine, but thorough checks shown my problem was crushing msmdpump.dll, and that crash was an exception, caught internally by msmdpump.dll. The only visible clue was a message in Windows Application log stating "an internal error happened" (or smth. alike). Googling alot didn't bring any valuable results, but suddenly this article gave me an idea to check for the LATEST MSQL cumulative update from here, and, after installing it and re-coping msmdpump.dll the crash was gone and cubes finally shown up as expected in SSMS interface. Needless to say that all issues with IIS Identity Pool, double-hop and other security-related stuff was rechecked many times with no success... Realizing that exception is inside the dll itself take some time to come...</p>

Body:
How do I resolve this issue with requests to msmdpump.dll for connections to SQL Server Analysis Services? I am receiving a 500 Error from the IsapiModule.

On a Windows Server 2012 R2 machine, with IIS 8.5, I have setup the OLAP data pump (msmdpump.dll), using the following instructions: https://msdn.microsoft.com/en-us/library/gg492140.aspx#bkmk_copy

The application pool is configured for .NET CLR v4.0, with Classic Managed pipeline mode. The identity is set to a local service account. (I have also tried a domain account, and I have tried making the local user an Administrator).

I've created an application under the Default Web Site, called OLAP, with an IsapiModule, as per the MSDN article.

As far as I can tell (and I've double and triple checked), everything is configured as laid out in the MSDN article. Also, compared to another server where I have this setup (on a different network), it is essentially the same.

When I request http://localhost/OLAP/msmdpump.dll in a browser on that machine, I receive a 500 Internal Server Error. The error indicates that it is trying to use the OLAP handler that I created. This is not the same error that I would normally expect when doing a GET request to msmdpump.dll. The normal error for a straight GET, when everything is working correctly, is sent back in a SOAP envelope. In my case, the request does not appear to ever be processed by msmdpump.dll.

500 Internal Server Error via browser:

(see below for full screenshot)

Module   IsapiModule
Notification   ExecuteRequestHandler
Handler   OLAP
Error Code   0x8007007e

Requested URL   http://localhost:80/OLAP/msmdpump.dll
Physical Path   C:\inetpub\wwwroot\OLAP\msmdpump.dll
Logon Method   Anonymous
Logon User   Anonymous

500 Internal Server Error via SSMS connection:

I also receive an error when trying to connect to the data pump via SQL Server Management Studio:

Screenshot of the 500 error in the browser:

One appreciable difference between the machine I'm setting up, and the server where the data pump already works, is that there are a few more roles setup on the new server.

The problem server includes:

.NET Extensibility 4.5

ASP.NET 4.5

While the other machine (where the data pump works), does not include those roles. Would the presence of ASP.NET 4.5 or .NET Extensibility 4.5 cause an issue with IIS serving requests for this IsapiModule?


问:
C# linq query, select parent list based on the condition of nest list
I have a list of projects with a list of project members for each project as follows

```<code>public List<Project> Projects { get; set; }
    public class Project
    {
        public Guid Id { get; set; }
        public string  Title { get; set; }
        public string Description { get; set; }
        public List<Member> Members { get; set; }
    }

    public class Member{
        public Guid Id { get; set; }
        public string Name { get; set; }
        public string Address { get; set; }
    }
</code>```

I want to select projects for a list of member Ids using LINQ, thank you!

答:
If you have a list of Guids, you can check against it by using .Contains() inside .Where():

```<code>var projects = new List<Project>();
//Add projects

var memberIds = new List<Guid>();
//Add member IDs

var memberProjects = projects
    .Where(p => p.Members.Any(m => memberIds.Contains(m.Id)));
</code>```

Body:
I have a list of projects with a list of project members for each project as follows

public List<Project> Projects { get; set; }
    public class Project
    {
        public Guid Id { get; set; }
        public string  Title { get; set; }
        public string Description { get; set; }
        public List<Member> Members { get; set; }
    }

    public class Member{
        public Guid Id { get; set; }
        public string Name { get; set; }
        public string Address { get; set; }
    }

I want to select projects for a list of member Ids using LINQ, thank you!

Body:
If you have a list of Guids, you can check against it by using .Contains() inside .Where():

var projects = new List<Project>();
//Add projects

var memberIds = new List<Guid>();
//Add member IDs

var memberProjects = projects
    .Where(p => p.Members.Any(m => memberIds.Contains(m.Id)));


问:
PySpark UDF issues when referencing outside of function
I facing the issue that I get the error

TypeError: cannot pickle '_thread.RLock' object

when I try to apply the following code:

```<code>from pyspark.sql.types import *
from pyspark.sql.functions import *

data_1 = [('James','Smith','M',30),('Anna','Rose','F',41),
  ('Robert','Williams','M',62),
]
data_2 = [('Junior','Smith','M',15),('Helga','Rose','F',33),
  ('Mike','Williams','M',77),
]
columns = ["firstname","lastname","gender","age"]
df_1 = spark.createDataFrame(data=data_1, schema = columns)
df_2 = spark.createDataFrame(data=data_2, schema = columns)

def find_n_people_with_higher_age(x):
  return df_2.filter(df_2['age']>=x).count()

find_n_people_with_higher_age_udf = udf(find_n_people_with_higher_age, IntegerType())
df_1.select(find_n_people_with_higher_age_udf(col('category_id')))
</code>```

答:
add additional data:

```<code>from pyspark.sql import Window
from pyspark.sql.types import *
from pyspark.sql.functions import *
    data_1 = [('James','Smith','M',30),('Anna','Rose','F',41),
  ('Robert','Williams','M',62),
]
    # add more data to make it more interesting.
    data_2 = [('Junior','Smith','M',15),('Helga','Rose','F',33),('Gia','Rose','F',34),
      ('Mike','Williams','M',77), ('John','Williams','M',77), ('Bill','Williams','F',79),
    ]
columns = ["firstname","lastname","gender","age"]
df_1 = spark.createDataFrame(data=data_1, schema = columns)
df_2 = spark.createDataFrame(data=data_2, schema = columns)

# dataframe to help fill in missing ages
ref = spark.range( 1, 110, 1).toDF("numbers").withColumn("count", lit(0)).withColumn("rolling_Count", lit(0))

countAges = df_2.groupby("age").count()
#this actually give you the short list of ages
rollingCounts = countAges.withColumn("rolling_Count", sum(col("count")).over(Window.partitionBy().orderBy(col("age").desc())))
#fill in missing ages and remove duplicates
filled = rollingCounts.union(ref).groupBy("age").agg(sum("count").alias("count"))
#add a rolling count across all ages
allAgeCounts = filled.withColumn("rolling_Count", sum(col("count")).over(Window.partitionBy().orderBy(col("age").desc())))
#do inner join because we've filled in all ages.
df_1.join(allAgeCounts, df_1.age == allAgeCounts.age, "inner").show()
+---------+--------+------+---+---+-----+-------------+
|firstname|lastname|gender|age|age|count|rolling_Count|
+---------+--------+------+---+---+-----+-------------+
|     Anna|    Rose|     F| 41| 41|    0|            3|
|   Robert|Williams|     M| 62| 62|    0|            3|
|    James|   Smith|     M| 30| 30|    0|            5|
+---------+--------+------+---+---+-----+-------------+
</code>```

I wouldn't normally want to use a window over an entire table, but here the data it's iterating over <= 110 so this is reasonable.

Body:
I facing the issue that I get the error

TypeError: cannot pickle '_thread.RLock' object

when I try to apply the following code:

from pyspark.sql.types import *
from pyspark.sql.functions import *

data_1 = [('James','Smith','M',30),('Anna','Rose','F',41),
  ('Robert','Williams','M',62),
]
data_2 = [('Junior','Smith','M',15),('Helga','Rose','F',33),
  ('Mike','Williams','M',77),
]
columns = ["firstname","lastname","gender","age"]
df_1 = spark.createDataFrame(data=data_1, schema = columns)
df_2 = spark.createDataFrame(data=data_2, schema = columns)

def find_n_people_with_higher_age(x):
  return df_2.filter(df_2['age']>=x).count()

find_n_people_with_higher_age_udf = udf(find_n_people_with_higher_age, IntegerType())
df_1.select(find_n_people_with_higher_age_udf(col('category_id')))

Body:
add additional data:

from pyspark.sql import Window
from pyspark.sql.types import *
from pyspark.sql.functions import *
    data_1 = [('James','Smith','M',30),('Anna','Rose','F',41),
  ('Robert','Williams','M',62),
]
    # add more data to make it more interesting.
    data_2 = [('Junior','Smith','M',15),('Helga','Rose','F',33),('Gia','Rose','F',34),
      ('Mike','Williams','M',77), ('John','Williams','M',77), ('Bill','Williams','F',79),
    ]
columns = ["firstname","lastname","gender","age"]
df_1 = spark.createDataFrame(data=data_1, schema = columns)
df_2 = spark.createDataFrame(data=data_2, schema = columns)

# dataframe to help fill in missing ages
ref = spark.range( 1, 110, 1).toDF("numbers").withColumn("count", lit(0)).withColumn("rolling_Count", lit(0))

countAges = df_2.groupby("age").count()
#this actually give you the short list of ages
rollingCounts = countAges.withColumn("rolling_Count", sum(col("count")).over(Window.partitionBy().orderBy(col("age").desc())))
#fill in missing ages and remove duplicates
filled = rollingCounts.union(ref).groupBy("age").agg(sum("count").alias("count"))
#add a rolling count across all ages
allAgeCounts = filled.withColumn("rolling_Count", sum(col("count")).over(Window.partitionBy().orderBy(col("age").desc())))
#do inner join because we've filled in all ages.
df_1.join(allAgeCounts, df_1.age == allAgeCounts.age, "inner").show()
+---------+--------+------+---+---+-----+-------------+
|firstname|lastname|gender|age|age|count|rolling_Count|
+---------+--------+------+---+---+-----+-------------+
|     Anna|    Rose|     F| 41| 41|    0|            3|
|   Robert|Williams|     M| 62| 62|    0|            3|
|    James|   Smith|     M| 30| 30|    0|            5|
+---------+--------+------+---+---+-----+-------------+

I wouldn't normally want to use a window over an entire table, but here the data it's iterating over <= 110 so this is reasonable.


问:
Iterate over a spark dataframe and create a nested json string
I'm having a dataframe with the below details and I need to iterate over each row to create the below formatted nested json string.

Employee name
Salary
sick_leave_day
paid_leave_day

Karthi
20000
10-10-2021
Y

Karthi
20000
11-12-2021
Y

Karthi
20000
13-12-2021
N

Rajesh
25000
15-12-2021
Y

Rajesh
25000
17-11-2021
N

Rajesh
25000
10-10-2021
Y

The output json should look like below, Here, Name and salary is having unique data and sick_leave_day,paid_leave_day changes for each row.

{
    "Name": "Karthi",
    "Salary": "20000",
    leave_details: [{
            "sick_leave_day": "10-10-2021",
            "paid_leave_day ": "Y"
        },
        {
            "sick_leave_day": "11-12-2021",
            "paid_leave_day ": "Y"
        },
        {
            "sick_leave_day": "13-12-2021",
            "paid_leave_day ": "N"
        }
    ]
}
```

I have tried to iterate over dataframe using foreach method, but I'm unable to get the logic. If I group the name and salary, I can't get the value of name and salary only once. So, please help me to write the logic by iterating over each row to create this logic. Thanks in advance.

答:
Group by Employee name and collect list of structs for leave_details array and first (or max) Salary then save a json:

val result = df.groupBy(col("Employee name").as("name")).agg(
    first($"Salary").as("Salary"),
    collect_list(struct($"sick_leave_day", $"paid_leave_day")).as("leave_details")
)

result.toJSON.collect.foreach(println(_))

//{"name":"Rajesh","Salary":"25000","leave_details":[{"sick_leave_day":"15-12-2021","paid_leave_day":"Y"},{"sick_leave_day":"17-11-2021","paid_leave_day":"N"},{"sick_leave_day":"10-10-2021","paid_leave_day":"Y"}]}
//{"name":"Karthi","Salary":"20000","leave_details":[{"sick_leave_day":"10-10-2021","paid_leave_day":"Y"},{"sick_leave_day":"11-12-2021","paid_leave_day":"Y"},{"sick_leave_day":"13-12-2021","paid_leave_day":"N"}]}
```

Body:
I'm having a dataframe with the below details and I need to iterate over each row to create the below formatted nested json string.

Employee name
Salary
sick_leave_day
paid_leave_day

Karthi
20000
10-10-2021
Y

Karthi
20000
11-12-2021
Y

Karthi
20000
13-12-2021
N

Rajesh
25000
15-12-2021
Y

Rajesh
25000
17-11-2021
N

Rajesh
25000
10-10-2021
Y

The output json should look like below, Here, Name and salary is having unique data and sick_leave_day,paid_leave_day changes for each row.

{
    "Name": "Karthi",
    "Salary": "20000",
    leave_details: [{
            "sick_leave_day": "10-10-2021",
            "paid_leave_day ": "Y"
        },
        {
            "sick_leave_day": "11-12-2021",
            "paid_leave_day ": "Y"
        },
        {
            "sick_leave_day": "13-12-2021",
            "paid_leave_day ": "N"
        }
    ]
}

I have tried to iterate over dataframe using foreach method, but I'm unable to get the logic. If I group the name and salary, I can't get the value of name and salary only once. So, please help me to write the logic by iterating over each row to create this logic. Thanks in advance.


问:
data type conversion to BIG_INT in odbc driver
How do I convert a datatype of SQL_INTEGER to SQL_C_BIGINT?

I have an odbc driver that is used by power BI to interact with a datasource.
During the call SQLColAttribute, the driver passes the column type as SIGNED SQL_INTEGER.
However, when the SQLFetchData is called by the client, it is expecting the datatype to be
SQL_C_BIGINT.
So basically, I want to convert the data of type SQL_INTEGER to SQL_C_BIGINT before passing it back to the client(power bi).

The challenge I am having is that I am not able to get the format in which a  SQL_C_BIGINT is expected in the driver. If I do a simple cast, the data is appearing incorrectly in the client.

```<code>SQLRETURN SQLGetData(SQLHSTMT StatementHandle,
                             SQLUSMALLINT ColumnNumber, SQLSMALLINT TargetType,
                             SQLPOINTER TargetValue, SQLLEN BufferLength,
                             SQLLEN  *StrLen_or_Ind)
{
     //TargetType is SQL_C_BIGINT
     // Assuming that I have my data in p, how do I copy it to TargetValue so that it is read as SQL_C_BIGINT by the client???
}
</code>```

答: SQL_C_BIGINT is basically an 8 byte integer. All I had to do is assign the value to _int64 variable and pass it back.</p>

Body:
How do I convert a datatype of SQL_INTEGER to SQL_C_BIGINT?

I have an odbc driver that is used by power BI to interact with a datasource.
During the call SQLColAttribute, the driver passes the column type as SIGNED SQL_INTEGER.
However, when the SQLFetchData is called by the client, it is expecting the datatype to be
SQL_C_BIGINT.
So basically, I want to convert the data of type SQL_INTEGER to SQL_C_BIGINT before passing it back to the client(power bi).

The challenge I am having is that I am not able to get the format in which a  SQL_C_BIGINT is expected in the driver. If I do a simple cast, the data is appearing incorrectly in the client.

SQLRETURN SQLGetData(SQLHSTMT StatementHandle,
                             SQLUSMALLINT ColumnNumber, SQLSMALLINT TargetType,
                             SQLPOINTER TargetValue, SQLLEN BufferLength,
                             SQLLEN  *StrLen_or_Ind)
{
     //TargetType is SQL_C_BIGINT
     // Assuming that I have my data in p, how do I copy it to TargetValue so that it is read as SQL_C_BIGINT by the client???
}


问:
Need to converge two matrix value together in form of tuple
Need to converge two matrix value together in form of tuple .

Matrix

0 1 2 3

1 1 1 1

2 2 1 3

3 2 3 1

Matrix

0 1 2 3

1 4 6 7

2 3 5 6

3 1 3 5

Output required is

0    1        2       3

1  (1,4)   (1,6)   (1,7)

2  (2,3)   (1,5)   (3,6)

3  (2.1)   (3,3)   (1,5)

where first row and column is index of the matrix.

答:
Assuming your matrix is list of lists, you can use zip() -

```<code>from random import randint
a = [[randint(1, 10) for _ in range(3)] for _ in range(3)]
b = [[randint(1, 10) for _ in range(3)] for _ in range(3)]
c = [list(zip(x, y)) for x, y in zip(a, b)]
</code>```
```<code>>>> a
[[8, 4, 1], [4, 3, 4], [2, 5, 5]]
>>> b
[[2, 2, 1], [6, 1, 7], [10, 3, 5]]
>>> c
[[(8, 2), (4, 2), (1, 1)], [(4, 6), (3, 1), (4, 7)], [(2, 10), (5, 3), (5, 5)]]
</code>```

If you don't want to use zip, using list comprehension -

```<code>c = [list((a[i][j], b[i][j]) for j in range(len(a[0]))) for i in range(len(a))]
</code>```

Both ways work even when a and b is not square matrix as long as the dimensions of both matrices are identical.

Body:
Assuming your matrix is list of lists, you can use zip() -

from random import randint
a = [[randint(1, 10) for _ in range(3)] for _ in range(3)]
b = [[randint(1, 10) for _ in range(3)] for _ in range(3)]
c = [list(zip(x, y)) for x, y in zip(a, b)]

>>> a
[[8, 4, 1], [4, 3, 4], [2, 5, 5]]
>>> b
[[2, 2, 1], [6, 1, 7], [10, 3, 5]]
>>> c
[[(8, 2), (4, 2), (1, 1)], [(4, 6), (3, 1), (4, 7)], [(2, 10), (5, 3), (5, 5)]]

If you don't want to use zip, using list comprehension -

c = [list((a[i][j], b[i][j]) for j in range(len(a[0]))) for i in range(len(a))]

Both ways work even when a and b is not square matrix as long as the dimensions of both matrices are identical.


问:
Reference an array of Google Map markers
I am trying to reference an array of markers that I created but when I try reference an array element the element receives the message GLOBAL_MARK.0 is null or not an object. I think it might be a scope issue but I'm not sure how to handle it. Here's the code.</p>

```<code>var GLOBAL_MARK = "[]"
var vcount = 0;

 function load() {

   if (GBrowserIsCompatible()) {
   var map = new GMap2(document.getElementById("map"));
   map.addControl(new GSmallMapControl());
   map.addControl(new GMapTypeControl());
   MyLat = "47.614495"
   MyLng =  "-122.341861"
                                                                            map.setCenter(new GLatLng(MyLat, MyLng), 13);

    GDownloadUrl("allmarkers.php", function(data) {
    var xml = GXml.parse(data);
    var markers = xml.documentElement.getElementsByTagName("marker");
    for (var i = 0; i < markers.length; i++) {
        var name = markers[i].getAttribute("name");
        var id = markers[i].getAttribute("id");
        var address = markers[i].getAttribute("address");
        var type = markers[i].getAttribute("type");
        var point = new GLatLng(parseFloat(markers[i].getAttribute("lat")), parseFloat(markers [i].getAttribute("lng")));

        var marker = createMarker(point, name, address, type, id);
        MyAddMarker(marker)
        map.addOverlay(marker);
     }
});
OutTest()
}
}

function MyAddMarker(marker) {
    GLOBAL_MARK[vcount] = marker
    vcount = vcount + 1
}

function OutTest() {
    alert(GLOBAL_MARK[0].getLatLng)
}

function createMarker(point, name, address, type, id, date1, date2) {
var marker = new GMarker(point, customIcons[type]);
var html = "<b>" + name + "</b> <br/>" + address + "<br><br><input type=\"button\"    value=\"  View Posts  \" onclick=\"view_posts('" + id + "','" + type + "','" + date1 + "','" + date2 + "','no')\">"
 GEvent.addListener(marker, 'click', function() {
 marker.openInfoWindowHtml(html);
  });
   return marker;
    }
</code>```

答:
Lots of errors in this code.  GLOBAL_MARK was definitely declared wrong to be used as an object like you use it in MyAddMarker().  I'll try to fix as many as I can see by eye without actually being able to run the code:

```<code>GLOBAL_MARK = [];
GDownloadUrl("allmarkers.php", function(data) {
    var xml = GXml.parse(data);
    var markers = xml.documentElement.getElementsByTagName("marker");
    for (var i = 0; i < markers.length; i++) {
        var name = markers[i].getAttribute("name");
        var id = markers[i].getAttribute("id");
        var address = markers[i].getAttribute("address");
        var type = markers[i].getAttribute("type");
        var point = new GLatLng(
            parseFloat(markers[i].getAttribute("lat")),
            parseFloat(markers[i].getAttribute("lng"))
        );
        var marker = createMarker(point, name, address, type, id);
        MyAddMarker(marker);
        map.addOverlay(marker);
     }
// when the code gets here, the data from allmarkers.php is done
});

OutTest();

function MyAddMarker(marker) {
    GLOBAL_MARK[vcount] = marker;
    vcount = vcount + 1;
}

function OutTest() {
    alert(GLOBAL_MARK[0].getLatLng);
}
</code>```

In addition, I don't see a declaration for vcount anywhere.

And, I suspect you're calling OutTest() before the anonymous function in GDownloadUrl has completed.  That very well may be an asynchronous function that hasn't completed yet when you call OutTest().

Body:
I am trying to reference an array of markers that I created but when I try reference an array element the element receives the message GLOBAL_MARK.0 is null or not an object. I think it might be a scope issue but I'm not sure how to handle it. Here's the code.

var GLOBAL_MARK = "[]"
var vcount = 0;

 function load() {

   if (GBrowserIsCompatible()) {
   var map = new GMap2(document.getElementById("map"));
   map.addControl(new GSmallMapControl());
   map.addControl(new GMapTypeControl());
   MyLat = "47.614495"
   MyLng =  "-122.341861"
                                                                            map.setCenter(new GLatLng(MyLat, MyLng), 13);

    GDownloadUrl("allmarkers.php", function(data) {
    var xml = GXml.parse(data);
    var markers = xml.documentElement.getElementsByTagName("marker");
    for (var i = 0; i < markers.length; i++) {
        var name = markers[i].getAttribute("name");
        var id = markers[i].getAttribute("id");
        var address = markers[i].getAttribute("address");
        var type = markers[i].getAttribute("type");
        var point = new GLatLng(parseFloat(markers[i].getAttribute("lat")), parseFloat(markers [i].getAttribute("lng")));

        var marker = createMarker(point, name, address, type, id);
        MyAddMarker(marker)
        map.addOverlay(marker);
     }
});
OutTest()
}
}

function MyAddMarker(marker) {
    GLOBAL_MARK[vcount] = marker
    vcount = vcount + 1
}

function OutTest() {
    alert(GLOBAL_MARK[0].getLatLng)
}

function createMarker(point, name, address, type, id, date1, date2) {
var marker = new GMarker(point, customIcons[type]);
var html = "" + name + "
" + address + "

<input type=\"button\"    value=\"  View Posts  \" onclick=\"view_posts('" + id + "','" + type + "','" + date1 + "','" + date2 + "','no')\">"
 GEvent.addListener(marker, 'click', function() {
 marker.openInfoWindowHtml(html);
  });
   return marker;
    }

Body:
Lots of errors in this code.  GLOBAL_MARK was definitely declared wrong to be used as an object like you use it in MyAddMarker().  I'll try to fix as many as I can see by eye without actually being able to run the code:

GLOBAL_MARK = [];
GDownloadUrl("allmarkers.php", function(data) {
    var xml = GXml.parse(data);
    var markers = xml.documentElement.getElementsByTagName("marker");
    for (var i = 0; i < markers.length; i++) {
        var name = markers[i].getAttribute("name");
        var id = markers[i].getAttribute("id");
        var address = markers[i].getAttribute("address");
        var type = markers[i].getAttribute("type");
        var point = new GLatLng(
            parseFloat(markers[i].getAttribute("lat")),
            parseFloat(markers[i].getAttribute("lng"))
        );
        var marker = createMarker(point, name, address, type, id);
        MyAddMarker(marker);
        map.addOverlay(marker);
     }
// when the code gets here, the data from allmarkers.php is done
});

OutTest();

function MyAddMarker(marker) {
    GLOBAL_MARK[vcount] = marker;
    vcount = vcount + 1;
}

function OutTest() {
    alert(GLOBAL_MARK[0].getLatLng);
}

In addition, I don't see a declaration for vcount anywhere.

And, I suspect you're calling OutTest() before the anonymous function in GDownloadUrl has completed.  That very well may be an asynchronous function that hasn't completed yet when you call OutTest().


问:
.httacces custom redirectioning
I want to redirect all invalid traffic to my index page.
Invalid traffic in this context means:

-Case 1. Nonexistent directories

-Case 2. Nonexistent files

-Case 3. Existing route but only directory: Like in example "validroute/images" or "validroute/images/"

Actually my working code is

```<code>RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ index.php [R,L,QSA]

Options -Indexes
</code>```

For cases 1 and 2 the conditions and the RewriteRule are redirecting correctly to my index.php
For the third case Im using the "Options -indexes" which avoids user to browse my directory and displays the forbidden error. I want to redirect this last case also to the index.php instead.

The nearest approach i got based on redirect 403 error using .htaccess  was this

```<code>ErrorDocument 403 /index.php
</code>```

Instead of redirecting ALL forbidden (403) when trying to display a valid "folder" (and ONLY in this case) i seek to redirect to index.

By setting the same redirects i expect the user not to be able to see my directories even if he tries to manually input different routes to see if the directory exists or not. Actually he has the hint because some are "forbidden" and others redirects to index.

This is NOT a duplicate of these because given answers didn't fit me or the issues are not what I intend:

.htaccess redirect - Options -Indexes

.htaccess option -indexes redirect?

.htaccess to redirect everything to index.php, but keep a copy of the website in a subdirectory

Maybe is not possible at all or maybe there is another command or maybe a proper regex combined to RewriteRule to fix the selection.

答:
There is a layer above for the server at which i am not able to access or to see in any way, but as i was handling other kind of permissions (folder accesses by password) i noticed that this had its own way of working without allowing developer to modify this.
So i guess there is no way to override such from a higher lvl .htaccess without direct access to server.
I was using Plesk Obsidian.

So if you dont have full control on the server your configurations might generate some conflicts.

Title: .httacces custom redirectioning

Body:
I want to redirect all invalid traffic to my index page.
Invalid traffic in this context means:

-Case 1. Nonexistent directories

-Case 2. Nonexistent files

-Case 3. Existing route but only directory: Like in example "validroute/images" or "validroute/images/"

Actually my working code is

RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^(.*)$ index.php [R,L,QSA]

Options -Indexes

For cases 1 and 2 the conditions and the RewriteRule are redirecting correctly to my index.php
For the third case Im using the "Options -indexes" which avoids user to browse my directory and displays the forbidden error. I want to redirect this last case also to the index.php instead.

The nearest approach i got based on redirect 403 error using .htaccess  was this

ErrorDocument 403 /index.php

Instead of redirecting ALL forbidden (403) when trying to display a valid "folder" (and ONLY in this case) i seek to redirect to index.

By setting the same redirects i expect the user not to be able to see my directories even if he tries to manually input different routes to see if the directory exists or not. Actually he has the hint because some are "forbidden" and others redirects to index.

This is NOT a duplicate of these because given answers didn't fit me or the issues are not what I intend:

.htaccess redirect - Options -Indexes

.htaccess option -indexes redirect?

.htaccess to redirect everything to index.php, but keep a copy of the website in a subdirectory

Maybe is not possible at all or maybe there is another command or maybe a proper regex combined to RewriteRule to fix the selection.


问:
Failing to install IdP configuration with helm in Openshift 4
I'm trying to apply htpasswd IdP configuration with oc apply commands which is working, but when I'm using configuration with helm it is failing to install with following error,

Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: OAuth "cluster" in namespace "" exists and cannot be imported into the current release:

Can someone help ?

Regards
Mallikharjuna Rao Polisetty

答:
This is a helm issue. The helm chart you're using is trying to place an resource of type oauth named cluster onto your OpenShift cluster, which already exists (supposedly because you created it by hand).
Clean up the existing oauth and try again.


问:
spark - filter within map
I am trying to filter inside map function. Basically the way I'll do that in classic map-reduce is mapper wont write anything to context when filter criteria meet. How can I achieve similar with spark? I can't seem to return null from map function as it fails in shuffle step. I can either use filter function but it seems unnecessary iteration of data set while I can perform same task during map. I can also try to output null with dummy key but thats a bad workaround.</p>

答:
Maybe try map_filter(col, lambda-function)
Here are the docs: https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.functions.map_filter.html#pyspark.sql.functions.map_filter

Title: spark - filter within map


问:
web-scraping college timetable keeps giving me 'charmap' error
I'm trying to webscrape for a project and get find free times on a timetable. When i run this code i get the following error "'charmap' codec can't decode byte 0x8d in position 32078: character maps to "
Here is my code

```<code>url = "https://opentimetable.dcu.ie/"
response = requests.get(url)

with open('webpage.html', 'r') as html_file:

content = html_file.read()
</code>```

Any help appreciated

答:
The reason you are not getting anything is because this data is dynamically rendered. You need to select different parameters to query what you are asking for, it will not come in a simple, static request.

With that, there is an api that gives you the option to search by different categories. The category with the least amount of unique values was "Location" so I went with that.

This will go and grab all the location Ids, then feed that into the filter to find what is booked for what time, at each location.

You have in the table the start and end times (and day) of when there is something booked. I will leave it to you to parse through that info to find when there is a day or time open. What I would simply do is have python create a list that has all the unique day and time starts and when time end, sort it, and have it then find where there are gaps/no overlaps.

```<code>import requests
import pandas as pd

s = requests.Session()

url = "https://opentimetable.dcu.ie/broker/api/categoryTypeOptions"
s.get(url)
cookies = s.cookies.get_dict()

cookieStr = ''
for k, v in cookies.items():
    cookieStr += f'{k}={v};'

headers = {
'Accept': 'application/json, text/plain, */*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9,en-GB;q=0.8',
'Authorization': 'basic T64Mdy7m[',
'Connection': 'keep-alive',
'Content-Type': 'application/json',
'Cookie': cookieStr,
'Host': 'opentimetable.dcu.ie',
'Referer': 'https://opentimetable.dcu.ie/',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

url = 'https://opentimetable.dcu.ie/broker/api/CategoryTypes/1e042cb1-547d-41d4-ae93-a1f2c3d34538/Categories/Filter?pageNumber=1'
payload = '[{"Identity": "6359fd0c-1bbe-496a-8998-4fefc5cd18de","Values": ["null"]}]'
jsonData = s.post(url, headers=headers, data=payload).json()

totalPages = jsonData['TotalPages']

print('Page: 1 of %s' %totalPages)
locationList = jsonData['Results']

for page in range(2, totalPages+1):
    print('Page: %s of %s' %(page,totalPages))
    url = 'https://opentimetable.dcu.ie/broker/api/CategoryTypes/1e042cb1-547d-41d4-ae93-a1f2c3d34538/Categories/Filter?pageNumber=%s' %(page)
    jsonData = s.post(url, headers=headers, data=payload).json()
    locationList += jsonData['Results']

locationList = [x['Identity'] for x in locationList]

def update_payload(listOfLocations):
    true = "true"
    false = "false"

    payload = {
      "ViewOptions": {
        "Days": [
          {
            "Name": "Monday",
            "DayOfWeek": 1,
            "IsDefault": true
          },
          {
            "Name": "Tuesday",
            "DayOfWeek": 2,
            "IsDefault": true
          },
          {
            "Name": "Wednesday",
            "DayOfWeek": 3,
            "IsDefault": true
          },
          {
            "Name": "Thursday",
            "DayOfWeek": 4,
            "IsDefault": true
          },
          {
            "Name": "Friday",
            "DayOfWeek": 5,
            "IsDefault": true
          }
        ],
        "Weeks": [
          {
            "WeekNumber": 21,
            "WeekLabel": "21",
            "FirstDayInWeek": "2022-02-07T00:00:00.000Z"
          }
        ],
        "TimePeriods": [
          {
            "Description": "All Day",
            "StartTime": "08:00",
            "EndTime": "22:00",
            "IsDefault": true
          }
        ],
        "DatePeriods": [
          {
            "Description": "This Week",
            "StartDateTime": "2021-09-20T00:00:00.000Z",
            "EndDateTime": "2022-09-20T00:00:00.000Z",
            "IsDefault": true,
            "IsThisWeek": true,
            "IsNextWeek": false,
            "Type": "ThisWeek"
          }
        ],
        "LegendItems": [],
        "InstitutionConfig": {},
        "DateConfig": {
          "FirstDayInWeek": 1,
          "StartDate": "2021-09-20T00:00:00+00:00",
          "EndDate": "2022-09-20T00:00:00+00:00"
        },
        "AllDays": [
          {
            "Name": "Monday",
            "DayOfWeek": 1,
            "IsDefault": true
          },
          {
            "Name": "Tuesday",
            "DayOfWeek": 2,
            "IsDefault": true
          },
          {
            "Name": "Wednesday",
            "DayOfWeek": 3,
            "IsDefault": true
          },
          {
            "Name": "Thursday",
            "DayOfWeek": 4,
            "IsDefault": true
          },
          {
            "Name": "Friday",
            "DayOfWeek": 5,
            "IsDefault": true
          },
          {
            "Name": "Saturday",
            "DayOfWeek": 6,
            "IsDefault": false
          },
          {
            "Name": "Sunday",
            "DayOfWeek": 0,
            "IsDefault": false
          }
        ]
      },
      "CategoryIdentities": listOfLocations

    }

    return payload

x = 20
final_list = lambda test_list, x: [test_list[i:i+x] for i in range(0, len(test_list), x)]
locationChunks = final_list(locationList, x)

locationBooked = []
for count, listOfLocations in enumerate(locationChunks, start=1):
    print('%s of %s' %(count, len(locationChunks)))
    payload = update_payload(listOfLocations)

    url = 'https://opentimetable.dcu.ie/broker/api/categoryTypes/1e042cb1-547d-41d4-ae93-a1f2c3d34538/categories/events/filter'

    response = s.post(url, headers=headers, json=payload).json()

    for each in response:
        if len(each['CategoryEvents']) > 0:
            locationBooked += each['CategoryEvents']

df = pd.DataFrame(locationBooked)
</code>```

Output: First 5 rows of 2936

```<code>print(df.head().to_string())
                          EventIdentity                HostKey                                                Description                EndDateTime  EventType  IsPublished   Location                                 Owner              StartDateTime  IsDeleted                       LastModified                                                                                ExtraProperties  UserManuallyAddedEvent                        StatusIdentity Status StatusBackgroundColor                                                 Name                              Identity
0  026bce78-1cce-9354-43b2-720b96ba9e03       2122#SPLUSCD8040                                                       None  2022-02-10T19:00:00+00:00    Booking         True  AHC.ODG01  b8cf1f5a-9687-4440-86b8-13da2c69fa62  2022-02-10T17:00:00+00:00      False  2022-01-17T17:27:51.9494682+00:00                                                                                [{'Name': 'Activity.TeachingWeekPattern_PatternAsArray', 'DisplayName': 'Weeks', 'Value': '18-26', 'Rank': 3}]                   False  b48c85d4-19aa-4b19-87a6-63a5c6d2e630   None                  None                  Booking - AFU 21-22 (Grainne Reddy)  43c03d98-1c80-4ab5-a47a-19db340ab179
1  5dfe3e92-439f-d7b2-1aaf-384328680d90       2122#SPLUS42C975                                 Contemporary Irish Society  2022-02-10T13:00:00+00:00    Booking         True  AHC.ODG01  b8cf1f5a-9687-4440-86b8-13da2c69fa62  2022-02-10T10:30:00+00:00      False  2022-01-12T14:31:15.0526265+00:00                                                                                [{'Name': 'Activity.TeachingWeekPattern_PatternAsArray', 'DisplayName': 'Weeks', 'Value': '21', 'Rank': 3}]                   False  b48c85d4-19aa-4b19-87a6-63a5c6d2e630   None                  None  Booking - Boston University 21-22 (Sean Harrington)  d259d6bd-6849-42a4-b28c-d6849b2623c1
2  bd60b7f0-633e-b90a-79e6-29fceb4c2ea5  2122ED1009[2]OC/L5/01                                                    RE Cert  2022-02-10T16:00:00+00:00  On Campus         True  AHC.ODG01  b8cf1f5a-9687-4440-86b8-13da2c69fa62  2022-02-10T15:00:00+00:00      False  2021-11-03T10:03:42.4098645+00:00                 [{'Name': 'Module Name', 'DisplayName': 'Module Name', 'Value': 'ED1009[0] Religions, Ethics & Moral Values (CIC)', 'Rank': 1}, {'Name': 'Staff Member', 'DisplayName': 'Staff Member', 'Value': 'Wilkinson J', 'Rank': 2}, {'Name': 'Activity.TeachingWeekPattern_PatternAsArray', 'DisplayName': 'Weeks', 'Value': '17-22, 24-26', 'Rank': 3}]                   False  b48c85d4-19aa-4b19-87a6-63a5c6d2e630   None                  None                                    ED1009[2]OC/L5/01  9589b2ef-17d7-4372-8870-5749c3ae6c37
3  bc161577-4c45-0ce9-a9c0-25fc942b12c0       2122#SPLUS57140A  Teacher as a Reflective Practitioner (School Placement)**  2022-02-09T10:00:00+00:00  On Campus         True  AHC.ODG01  b8cf1f5a-9687-4440-86b8-13da2c69fa62  2022-02-09T09:00:00+00:00      False  2021-10-28T10:35:59.4486955+00:00            [{'Name': 'Module Name', 'DisplayName': 'Module Name', 'Value': 'ED1024[0] Teacher as a Reflective Practitioner (SP)', 'Rank': 1}, {'Name': 'Staff Member', 'DisplayName': 'Staff Member', 'Value': 'Lodge A', 'Rank': 2}, {'Name': 'Activity.TeachingWeekPattern_PatternAsArray', 'DisplayName': 'Weeks', 'Value': '2-11, 17-22, 24-26', 'Rank': 3}]                   False  b48c85d4-19aa-4b19-87a6-63a5c6d2e630   None                  None                                    ED1024[0]OC/T1/07  d213893f-3e0a-4a01-a0d9-7dac1d873627
4  a804c1a7-ef87-d8f0-16be-502d396699d6     2122ED2009[2]L1/01      Religions, Ethics, Morals and Values Education (REMV)  2022-02-11T15:00:00+00:00  On Campus         True  AHC.ODG01  b8cf1f5a-9687-4440-86b8-13da2c69fa62  2022-02-11T14:00:00+00:00      False  2021-09-21T16:50:21.8417239+00:00  [{'Name': 'Module Name', 'DisplayName': 'Module Name', 'Value': 'ED2009[0] Religious, Ethics, Morals & Values Education', 'Rank': 1}, {'Name': 'Staff Member', 'DisplayName': 'Staff Member', 'Value': 'Lodge A, Wilkinson J', 'Rank': 2}, {'Name': 'Activity.TeachingWeekPattern_PatternAsArray', 'DisplayName': 'Weeks', 'Value': '17-22, 24-25', 'Rank': 3}]                   False  b48c85d4-19aa-4b19-87a6-63a5c6d2e630   None                  None                                    ED2009[2]OC/L1/01  6facf5d3-6cb3-45f3-a9c1-3311a74691e7
</code>```

Body:
I'm trying to webscrape for a project and get find free times on a timetable. When i run this code i get the following error "'charmap' codec can't decode byte 0x8d in position 32078: character maps to "
Here is my code

url = "https://opentimetable.dcu.ie/"
response = requests.get(url)

with open('webpage.html', 'r') as html_file:

content = html_file.read()

Any help appreciated

Body:
The reason you are not getting anything is because this data is dynamically rendered. You need to select different parameters to query what you are asking for, it will not come in a simple, static request.

With that, there is an api that gives you the option to search by different categories. The category with the least amount of unique values was "Location" so I went with that.

This will go and grab all the location Ids, then feed that into the filter to find what is booked for what time, at each location.

You have in the table the start and end times (and day) of when there is something booked. I will leave it to you to parse through that info to find when there is a day or time open. What I would simply do is have python create a list that has all the unique day and time starts and when time end, sort it, and have it then find where there are gaps/no overlaps.

import requests
import pandas as pd

s = requests.Session()

url = "https://opentimetable.dcu.ie/broker/api/categoryTypeOptions"
s.get(url)
cookies = s.cookies.get_dict()

cookieStr = ''
for k, v in cookies.items():
    cookieStr += f'{k}={v};'

headers = {
'Accept': 'application/json, text/plain, */*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'en-US,en;q=0.9,en-GB;q=0.8',
'Authorization': 'basic T64Mdy7m[',
'Connection': 'keep-alive',
'Content-Type': 'application/json',
'Cookie': cookieStr,
'Host': 'opentimetable.dcu.ie',
'Referer': 'https://opentimetable.dcu.ie/',
'sec-ch-ua': '" Not A;Brand";v="99", "Chromium";v="96", "Google Chrome";v="96"',
'sec-ch-ua-mobile': '?0',
'sec-ch-ua-platform': '"Windows"',
'Sec-Fetch-Dest': 'empty',
'Sec-Fetch-Mode': 'cors',
'Sec-Fetch-Site': 'same-origin',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

url = 'https://opentimetable.dcu.ie/broker/api/CategoryTypes/1e042cb1-547d-41d4-ae93-a1f2c3d34538/Categories/Filter?pageNumber=1'
payload = '[{"Identity": "6359fd0c-1bbe-496a-8998-4fefc5cd18de","Values": ["null"]}]'
jsonData = s.post(url, headers=headers, data=payload).json()

totalPages = jsonData['TotalPages']

print('Page: 1 of %s' %totalPages)
locationList = jsonData['Results']

for page in range(2, totalPages+1):
    print('Page: %s of %s' %(page,totalPages))
    url = 'https://opentimetable.dcu.ie/broker/api/CategoryTypes/1e042cb1-547d-41d4-ae93-a1f2c3d34538/Categories/Filter?pageNumber=%s' %(page)
    jsonData = s.post(url, headers=headers, data=payload).json()
    locationList += jsonData['Results']

locationList = [x['Identity'] for x in locationList]

def update_payload(listOfLocations):
    true = "true"
    false = "false"

    payload = {
      "ViewOptions": {
        "Days": [
          {
            "Name": "Monday",
            "DayOfWeek": 1,
            "IsDefault": true
          },
          {
            "Name": "Tuesday",
            "DayOfWeek": 2,
            "IsDefault": true
          },
          {
            "Name": "Wednesday",
            "DayOfWeek": 3,
            "IsDefault": true
          },
          {
            "Name": "Thursday",
            "DayOfWeek": 4,
            "IsDefault": true
          },
          {
            "Name": "Friday",
            "DayOfWeek": 5,
            "IsDefault": true
          }
        ],
        "Weeks": [
          {
            "WeekNumber": 21,
            "WeekLabel": "21",
            "FirstDayInWeek": "2022-02-07T00:00:00.000Z"
          }
        ],
        "TimePeriods": [
          {
            "Description": "All Day",
            "StartTime": "08:00",
            "EndTime": "22:00",
            "IsDefault": true
          }
        ],
        "DatePeriods": [
          {
            "Description": "This Week",
            "StartDateTime": "2021-09-20T00:00:00.000Z",
            "EndDateTime": "2022-09-20T00:00:00.000Z",
            "IsDefault": true,
            "IsThisWeek": true,
            "IsNextWeek": false,
            "Type": "ThisWeek"
          }
        ],
        "LegendItems": [],
        "InstitutionConfig": {},
        "DateConfig": {
          "FirstDayInWeek": 1,
          "StartDate": "2021-09-20T00:00:00+00:00",
          "EndDate": "2022-09-20T00:00:00+00:00"
        },
        "AllDays": [
          {
            "Name": "Monday",
            "DayOfWeek": 1,
            "IsDefault": true
          },
          {
            "Name": "Tuesday",
            "DayOfWeek": 2,
            "IsDefault": true
          },
          {
            "Name": "Wednesday",
            "DayOfWeek": 3,
            "IsDefault": true
          },
          {
            "Name": "Thursday",
            "DayOfWeek": 4,
            "IsDefault": true
          },
          {
            "Name": "Friday",
            "DayOfWeek": 5,
            "IsDefault": true
          },
          {
            "Name": "Saturday",
            "DayOfWeek": 6,
            "IsDefault": false
          },
          {
            "Name": "Sunday",
            "DayOfWeek": 0,
            "IsDefault": false
          }
        ]
      },
      "CategoryIdentities": listOfLocations

    }

    return payload

x = 20
final_list = lambda test_list, x: [test_list[i:i+x] for i in range(0, len(test_list), x)]
locationChunks = final_list(locationList, x)

locationBooked = []
for count, listOfLocations in enumerate(locationChunks, start=1):
    print('%s of %s' %(count, len(locationChunks)))
    payload = update_payload(listOfLocations)

    url = 'https://opentimetable.dcu.ie/broker/api/categoryTypes/1e042cb1-547d-41d4-ae93-a1f2c3d34538/categories/events/filter'

    response = s.post(url, headers=headers, json=payload).json()

    for each in response:
        if len(each['CategoryEvents']) > 0:
            locationBooked += each['CategoryEvents']

df = pd.DataFrame(locationBooked)

Output: First 5 rows of 2936

print(df.head().to_string())
                          EventIdentity                HostKey                                                Description                EndDateTime  EventType  IsPublished   Location                                 Owner              StartDateTime  IsDeleted                       LastModified                                                                                ExtraProperties  UserManuallyAddedEvent                        StatusIdentity Status StatusBackgroundColor                                                 Name                              Identity
0  026bce78-1cce-9354-43b2-720b96ba9e03       2122#SPLUSCD8040                                                       None  2022-02-10T19:00:00+00:00    Booking         True  AHC.ODG01  b8cf1f5a-9687-4440-86b8-13da2c69fa62  2022-02-10T17:00:00+00:00      False  2022-01-17T17:27:51.9494682+00:00                                                                                [{'Name': 'Activity.TeachingWeekPattern_PatternAsArray', 'DisplayName': 'Weeks', 'Value': '18-26', 'Rank': 3}]                   False  b48c85d4-19aa-4b19-87a6-63a5c6d2e630   None                  None                  Booking - AFU 21-22 (Grainne Reddy)  43c03d98-1c80-4ab5-a47a-19db340ab179
1  5dfe3e92-439f-d7b2-1aaf-384328680d90       2122#SPLUS42C975                                 Contemporary Irish Society  2022-02-10T13:00:00+00:00    Booking         True  AHC.ODG01  b8cf1f5a-9687-4440-86b8-13da2c69fa62  2022-02-10T10:30:00+00:00      False  2022-01-12T14:31:15.0526265+00:00                                                                                [{'Name': 'Activity.TeachingWeekPattern_PatternAsArray', 'DisplayName': 'Weeks', 'Value': '21', 'Rank': 3}]                   False  b48c85d4-19aa-4b19-87a6-63a5c6d2e630   None                  None  Booking - Boston University 21-22 (Sean Harrington)  d259d6bd-6849-42a4-b28c-d6849b2623c1
2  bd60b7f0-633e-b90a-79e6-29fceb4c2ea5  2122ED1009[2]OC/L5/01                                                    RE Cert  2022-02-10T16:00:00+00:00  On Campus         True  AHC.ODG01  b8cf1f5a-9687-4440-86b8-13da2c69fa62  2022-02-10T15:00:00+00:00      False  2021-11-03T10:03:42.4098645+00:00                 [{'Name': 'Module Name', 'DisplayName': 'Module Name', 'Value': 'ED1009[0] Religions, Ethics & Moral Values (CIC)', 'Rank': 1}, {'Name': 'Staff Member', 'DisplayName': 'Staff Member', 'Value': 'Wilkinson J', 'Rank': 2}, {'Name': 'Activity.TeachingWeekPattern_PatternAsArray', 'DisplayName': 'Weeks', 'Value': '17-22, 24-26', 'Rank': 3}]                   False  b48c85d4-19aa-4b19-87a6-63a5c6d2e630   None                  None                                    ED1009[2]OC/L5/01  9589b2ef-17d7-4372-8870-5749c3ae6c37
3  bc161577-4c45-0ce9-a9c0-25fc942b12c0       2122#SPLUS57140A  Teacher as a Reflective Practitioner (School Placement)**  2022-02-09T10:00:00+00:00  On Campus         True  AHC.ODG01  b8cf1f5a-9687-4440-86b8-13da2c69fa62  2022-02-09T09:00:00+00:00      False  2021-10-28T10:35:59.4486955+00:00            [{'Name': 'Module Name', 'DisplayName': 'Module Name', 'Value': 'ED1024[0] Teacher as a Reflective Practitioner (SP)', 'Rank': 1}, {'Name': 'Staff Member', 'DisplayName': 'Staff Member', 'Value': 'Lodge A', 'Rank': 2}, {'Name': 'Activity.TeachingWeekPattern_PatternAsArray', 'DisplayName': 'Weeks', 'Value': '2-11, 17-22, 24-26', 'Rank': 3}]                   False  b48c85d4-19aa-4b19-87a6-63a5c6d2e630   None                  None                                    ED1024[0]OC/T1/07  d213893f-3e0a-4a01-a0d9-7dac1d873627
4  a804c1a7-ef87-d8f0-16be-502d396699d6     2122ED2009[2]L1/01      Religions, Ethics, Morals and Values Education (REMV)  2022-02-11T15:00:00+00:00  On Campus         True  AHC.ODG01  b8cf1f5a-9687-4440-86b8-13da2c69fa62  2022-02-11T14:00:00+00:00      False  2021-09-21T16:50:21.8417239+00:00  [{'Name': 'Module Name', 'DisplayName': 'Module Name', 'Value': 'ED2009[0] Religious, Ethics, Morals & Values Education', 'Rank': 1}, {'Name': 'Staff Member', 'DisplayName': 'Staff Member', 'Value': 'Lodge A, Wilkinson J', 'Rank': 2}, {'Name': 'Activity.TeachingWeekPattern_PatternAsArray', 'DisplayName': 'Weeks', 'Value': '17-22, 24-25', 'Rank': 3}]                   False  b48c85d4-19aa-4b19-87a6-63a5c6d2e630   None                  None                                    ED2009[2]OC/L1/01  6facf5d3-6cb3-45f3-a9c1-3311a74691e7


问:
How to expose lamba endpoint in serverless.yml only for dev stage?
I'd like to have a debug endpoint available only for stage. One way to do it is to the authorization in server code, but it would be even better not to create the API Gateway endpoint all together. Is it possible to achieve using Serverless framework?</p>

答:
In Serverless Framework, you can use a switch based on your environment to toggle configuration options, within the context of your YAML file.

For example, if you wanted to setup a normal HTTP endpoint via API Gateway you'd add something like the following to your functions section:

```<code>functions:
  ping:
    handler: "src/functions/ping/handler.main"
    events:
      http:
        method: "any"
        path: "/ping"
</code>```

If you want this http event to be available for only a certain stage, you can define a block elsewhere and reference it based on the current stage name. It's commonplace to use the custom block in your serverless.yml file for this. To isolate between the stages prod and dev, with dev being your staging area you'd do this:

```<code>custom:
  pingEvents: "${opt:stage, self:provider.stage}"
    prod:
      http:
        method: "any"
        path: "/ping"
    dev:
      http:

functions:
  ping:
    handler: "src/functions/ping/handler.main"
    events: "${self:custom.pingEvents.${opt:stage, self:provider.stage}}"
</code>```

The above would only expose the endpoint via HTTP when you run sls deploy --stage dev to release your application; any other stage (e.g. – prod) will have it disabled. Do note, that in this example, if you want to support stages outside of dev and prod you'll need to add a new block under custom.pingEvents.

Body:
In Serverless Framework, you can use a switch based on your environment to toggle configuration options, within the context of your YAML file.

For example, if you wanted to setup a normal HTTP endpoint via API Gateway you'd add something like the following to your functions section:

functions:
  ping:
    handler: "src/functions/ping/handler.main"
    events:
      http:
        method: "any"
        path: "/ping"

If you want this http event to be available for only a certain stage, you can define a block elsewhere and reference it based on the current stage name. It's commonplace to use the custom block in your serverless.yml file for this. To isolate between the stages prod and dev, with dev being your staging area you'd do this:

custom:
  pingEvents: "${opt:stage, self:provider.stage}"
    prod:
      http:
        method: "any"
        path: "/ping"
    dev:
      http:

functions:
  ping:
    handler: "src/functions/ping/handler.main"
    events: "${self:custom.pingEvents.${opt:stage, self:provider.stage}}"

The above would only expose the endpoint via HTTP when you run sls deploy --stage dev to release your application; any other stage (e.g. – prod) will have it disabled. Do note, that in this example, if you want to support stages outside of dev and prod you'll need to add a new block under custom.pingEvents.