问:
Copy data in array of a nested array of an object
So guys, I don't how to give a topic to this one. I'm having trouble to copy data from array of Object to another newly created array.

For example, I want to copy and create a new array that contain all category of animals of every person in my database.

```<code> people = [
     {
         name: "Person 1",
         animals: [
             { category: "cat" },
             { category: "dog" },
             { category: "fish" }
         ]
     },
     {
         name: "Person 2",
         animals: [
             { category: "dog" },
             { category: "iguana" }
         ]
     },
     {
         name: "Person 3",
         animals: [
             { category: "cat" }
         ]
     }
 ]
</code>```

So, I created a new array named animalCategory to hold every category available.

```<code> // declare new array to hold category of animals
 let animalCategory = []
</code>```

This is the logic I came up with:-

```<code>// loop all person available
people.forEach(person => {
    // go and loop inside animals array
    person.animals.forEach(animal => {
        // save new category of animals if animalCategory array is EMPTY
        if(animalCategory.length === 0) {
            animalCategory.push(animal.category)
        }

        // if NOT EMPTY, then
        else {
            // loop and check existing animal categories in animalCategory array
            animalCategory.forEach(category => {
                // check if MATCH?
                if(category === animal.category) {
                    break // or just continue or will NOT BE SAVE
                }

                // if NOT MATCH, then
                else {
                    // SAVE new category
                    animalCategory.push(animal.category)
                }
            })
        }
    })
})

// see result
console.log(animalCategory.length)
</code>```

But unfortunately I got like a very big array of animalCategory as a result. And a lot of repetative animals category. (like shown below)

UPDATED: the output I want to seek is:-

```<code>animalCategory: [ 'cat', 'dog', 'iguana', 'fish']
</code>```

How should I change my logic then? And is there any other way I can do this?

答:
We can collect all category property values in array and from it we would remove duplicates values.</p>

```<code>var people = [{
    name: "Person 1",
    animals: [{
        category: "cat"
      },
      {
        category: "dog"
      },
      {
        category: "fish"
      }
    ]
  },
  {
    name: "Person 2",
    animals: [{
        category: "dog"
      },
      {
        category: "iguana"
      }
    ]
  },
  {
    name: "Person 3",
    animals: [{
      category: "cat"
    }]
  }
];
var getArr = [];
var animalsArr = people.map(x => x.animals.map(y => {
  getArr.push(y.category);
}));

var filteredArr = getArr.filter((value, index, self) => self.indexOf(value) === index);
console.log(filteredArr);
</code>```

Body:
So guys, I don't how to give a topic to this one. I'm having trouble to copy data from array of Object to another newly created array.

For example, I want to copy and create a new array that contain all category of animals of every person in my database.

 people = [
     {
         name: "Person 1",
         animals: [
             { category: "cat" },
             { category: "dog" },
             { category: "fish" }
         ]
     },
     {
         name: "Person 2",
         animals: [
             { category: "dog" },
             { category: "iguana" }
         ]
     },
     {
         name: "Person 3",
         animals: [
             { category: "cat" }
         ]
     }
 ]

So, I created a new array named animalCategory to hold every category available.

 // declare new array to hold category of animals
 let animalCategory = []

This is the logic I came up with:-

// loop all person available
people.forEach(person => {
    // go and loop inside animals array
    person.animals.forEach(animal => {
        // save new category of animals if animalCategory array is EMPTY
        if(animalCategory.length === 0) {
            animalCategory.push(animal.category)
        }

        // if NOT EMPTY, then
        else {
            // loop and check existing animal categories in animalCategory array
            animalCategory.forEach(category => {
                // check if MATCH?
                if(category === animal.category) {
                    break // or just continue or will NOT BE SAVE
                }

                // if NOT MATCH, then
                else {
                    // SAVE new category
                    animalCategory.push(animal.category)
                }
            })
        }
    })
})

// see result
console.log(animalCategory.length)

But unfortunately I got like a very big array of animalCategory as a result. And a lot of repetative animals category. (like shown below)

UPDATED: the output I want to seek is:-

animalCategory: [ 'cat', 'dog', 'iguana', 'fish']

How should I change my logic then? And is there any other way I can do this?

Body:
We can collect all category property values in array and from it we would remove duplicates values.

var people = [{
    name: "Person 1",
    animals: [{
        category: "cat"
      },
      {
        category: "dog"
      },
      {
        category: "fish"
      }
    ]
  },
  {
    name: "Person 2",
    animals: [{
        category: "dog"
      },
      {
        category: "iguana"
      }
    ]
  },
  {
    name: "Person 3",
    animals: [{
      category: "cat"
    }]
  }
];
var getArr = [];
var animalsArr = people.map(x => x.animals.map(y => {
  getArr.push(y.category);
}));

var filteredArr = getArr.filter((value, index, self) => self.indexOf(value) === index);
console.log(filteredArr);


来源: stackexchange

问:
How to use Instagram Direct message Api
I am looking for Instagram Direct message API, I tried some private API libraries but I couldn't find any direct message API
How I can use Instagram Direct message API correctly?

I had a look at other posts, but they did not solved my problem

答:
There is no Instagram API for direct messages, I guess there won't be any. You may be able to use the Instagram node.js Private API library

I have to warn you though, that I did try to use the PHP library and Instagram didn't
If you are able to use node.js, it seems that Thread.configureText can be used for sending direct messages on the node.js library.

https://github.com/dilame/instagram-private-api

ContentLicense: CC BY-SA 4.0

Title: How to use Instagram Direct message Api


问:
Angular 8 Typescript How to set value of export const from another component?
I would like to change the value of the variable named bdg_m_1 from an another component. I am able to change the value of bdg_m_1 from the same typescript file.. Here is my code..

Filename : nav.ts

```<code>var bdg_m_1:string = "10" ; // HOW TO CHANGE THIS VALUE FROM ANOTHER COMPONENT

export const navItems: INavData[] = [
  {
    name: "Menu 1",
    url: "/menu_1",
    icon: "icon-speedometer",
     badge: {
       variant: 'info',
       text: bdg_m_1,
     }
  },
];
</code>```

答:
You could send it through an eventEmitter if they are parent-child related. Here is an example on stackblitz, and the code in case it does not works :

child-component.ts:

```<code>import { Component, EventEmitter, Output } from '@angular/core';

@Component({
  selector: 'app-child',
  templateUrl: './child.component.html',
  styleUrls: ['./child.component.css']
})
export class ChildComponent {
  nameToSend = '';
  @Output() newNameEmitter = new EventEmitter<string>();

  emitNewName () {
    this.newNameEmitter.emit(this.nameToSend);
  }
}
</code>```

chil-component.html:

```<code><input type="text" [(ngModel)]="nameToSend"/>
<button (click)="emitNewName()">Send new name</button>
</code>```

app.ts:

```<code>import { Component } from '@angular/core';

@Component({
  selector: 'my-app',
  templateUrl: './app.component.html',
  styleUrls: [ './app.component.css' ]
})
export class AppComponent  {
  name = 'Angular';

  changeName(newName: string){
    this.name = newName;
  }
}
</code>```

app.html:

```<code><app-child (newNameEmitter)="changeName($event)"></app-child>
<br>
name = {{name}}
</code>```

The principle is to emit a value whenever you want. you catch it in the parent with (newNameEmitter)="changeName($event)" (app.html)

Body:
I would like to change the value of the variable named bdg_m_1 from an another component. I am able to change the value of bdg_m_1 from the same typescript file.. Here is my code..

Filename : nav.ts

var bdg_m_1:string = "10" ; // HOW TO CHANGE THIS VALUE FROM ANOTHER COMPONENT

export const navItems: INavData[] = [
  {
    name: "Menu 1",
    url: "/menu_1",
    icon: "icon-speedometer",
     badge: {
       variant: 'info',
       text: bdg_m_1,
     }
  },
];

Body:
You could send it through an eventEmitter if they are parent-child related. Here is an example on stackblitz, and the code in case it does not works :

child-component.ts:

import { Component, EventEmitter, Output } from '@angular/core';

@Component({
  selector: 'app-child',
  templateUrl: './child.component.html',
  styleUrls: ['./child.component.css']
})
export class ChildComponent {
  nameToSend = '';
  @Output() newNameEmitter = new EventEmitter<string>();

  emitNewName () {
    this.newNameEmitter.emit(this.nameToSend);
  }
}

chil-component.html:

<input type="text" [(ngModel)]="nameToSend"/>
<button (click)="emitNewName()">Send new name</button>

app.ts:

import { Component } from '@angular/core';

@Component({
  selector: 'my-app',
  templateUrl: './app.component.html',
  styleUrls: [ './app.component.css' ]
})
export class AppComponent  {
  name = 'Angular';

  changeName(newName: string){
    this.name = newName;
  }
}

app.html:

<app-child (newNameEmitter)="changeName($event)"></app-child>
<br>
name = {{name}}

The principle is to emit a value whenever you want. you catch it in the parent with (newNameEmitter)="changeName($event)" (app.html)


问:
Assign NaN to a column based on another column
The title may be off a little. I will explain this properly. I will receive a DataFrame df with unexpected format of one of the column called marker.

Sometimes, I will marker, mixed of single marker or range marker :

```<code>marker             place1       place2
45                  PQR           STU
145.0-100           ABC           DEF
267.0-175.8         GHI           KLM
</code>```

During transformation, I need to split marker that contains - and becomes like this :

```<code>marker        firstkm    lastkm    place1       place2
45             45         NaN       PQR           STU
145.0-100      145.0      100       ABC           DEF
267.0-175.8    267.0      175.8     GHI           KLM
</code>```

I can also get the following DataFrame where all marker are not a range.

```<code>marker    place1       place2
145.0      ABC           DEF
267.0      GHI           KLM
</code>```

With this piece of code:

```<code>    #Split marker to temporary dataframe , split_m
    split_m = df.marker.str.split('-', expand=True)
    split_m.columns=['firstkm', 'lastkm'] #hitting error here
    split_m = split_km[['firstkm', 'lastkm']].replace([None], np.nan)
</code>```

I can achieve the result I wanted if the DataFrame is like the first example I showed above. However, I am not able to achieve the result if the DataFrame is like the second example I showed. I am getting error :

ValueError: Length mismatch: Expected axis has 1 elements, new values
  have 2 elements

I understand the error is caused by, the DataFrame unable to put any value in lastkm. But I don't know how to handle this..

If I print split_m for the second DataFrame, I will get this :

```<code>marker     firstkm
145.0       145.0
267.0       267.0
</code>```

How can I assign np.nan immediately to lastkm to produce the following result:

```<code>marker     firstkm   lastkm
145.0       145.0     NaN
267.0       267.0     NaN
</code>```

EDIT

Another pattern I encounter :

```<code>marker        firstkm      lastkm    place1       place2
45             45           NaN       PQR           STU
145.0-100      145.0        100       ABC           DEF
267.0-175.8    267.0        175.8     GHI           KLM
18.1J          18.1J        Nan       GHI           KLM
P7.991-54.3    P7.991       54.3      GHI           KLM
UPM Ex 0.5     UPM Ex 0.5   NaN       PPP           SSS
</code>```

UPMEx0.5 is stil acceptable. Case insensitive.

Body:
The title may be off a little. I will explain this properly. I will receive a DataFrame df with unexpected format of one of the column called marker.

Sometimes, I will marker, mixed of single marker or range marker :

marker             place1       place2
45                  PQR           STU
145.0-100           ABC           DEF
267.0-175.8         GHI           KLM

During transformation, I need to split marker that contains - and becomes like this :

marker        firstkm    lastkm    place1       place2
45             45         NaN       PQR           STU
145.0-100      145.0      100       ABC           DEF
267.0-175.8    267.0      175.8     GHI           KLM

I can also get the following DataFrame where all marker are not a range.

marker    place1       place2
145.0      ABC           DEF
267.0      GHI           KLM

With this piece of code:

    #Split marker to temporary dataframe , split_m
    split_m = df.marker.str.split('-', expand=True)
    split_m.columns=['firstkm', 'lastkm'] #hitting error here
    split_m = split_km[['firstkm', 'lastkm']].replace([None], np.nan)

I can achieve the result I wanted if the DataFrame is like the first example I showed above. However, I am not able to achieve the result if the DataFrame is like the second example I showed. I am getting error :

ValueError: Length mismatch: Expected axis has 1 elements, new values
  have 2 elements

I understand the error is caused by, the DataFrame unable to put any value in lastkm. But I don't know how to handle this..

If I print split_m for the second DataFrame, I will get this :

marker     firstkm
145.0       145.0
267.0       267.0

How can I assign np.nan immediately to lastkm to produce the following result:

marker     firstkm   lastkm
145.0       145.0     NaN
267.0       267.0     NaN

EDIT

Another pattern I encounter :

marker        firstkm      lastkm    place1       place2
45             45           NaN       PQR           STU
145.0-100      145.0        100       ABC           DEF
267.0-175.8    267.0        175.8     GHI           KLM
18.1J          18.1J        Nan       GHI           KLM
P7.991-54.3    P7.991       54.3      GHI           KLM
UPM Ex 0.5     UPM Ex 0.5   NaN       PPP           SSS

UPMEx0.5 is stil acceptable. Case insensitive.

Body:
Use str.extract:

print (df["marker"].str.extract("(?P<Start>\d+\.?\d+?)-?(?P<End>\d+\.?\d+?)?"))

   Start    End
0     45    NaN
1  145.0    100
2  267.0  175.8


问:
How would I draw 4 squares in a vertical stack move back and forth?
How would I draw 4 squares in a vertical stack move back and forth using a loop to avoid repeating the same code 4 times when drawing 4 squares. The squares should move as one and when any of them hit the edge of the window, all of them change direction.</p>

```<code>float location;
float sizeWidth;
float sizeHeight;
float direction;
boolean moving;

void setup() {
  size (1280, 720);
  location = 0;
  direction = height/720;
  sizeWidth = 90;
  sizeHeight = 90;
  moving = true;
}

void draw() {

  background(255);

  stroke(0);
  strokeWeight(1);

  for (int i = 0; i < width; i = i + 90) {
    line(0, i, 1280, i);
    line(i, 0, i, 720);
  }

  fill (255, 147, 79);
  int steppedPos = (int)(location/sizeHeight+0.5);
  rect(0, steppedPos*sizeHeight, sizeWidth, sizeHeight);

  if (moving) {
    location = location + direction;
    if (location + sizeHeight > height || location < 0){
      direction = direction * -1;
      location = location + direction;
    }
  }
}

void keyPressed() {
  if (key == ' ') {
    moving = !moving;
  }
}
</code>```

答: Check out Monday’s lecture!  We covered exactly this. </p>

Body:
How would I draw 4 squares in a vertical stack move back and forth using a loop to avoid repeating the same code 4 times when drawing 4 squares. The squares should move as one and when any of them hit the edge of the window, all of them change direction.

float location;
float sizeWidth;
float sizeHeight;
float direction;
boolean moving;

void setup() {
  size (1280, 720);
  location = 0;
  direction = height/720;
  sizeWidth = 90;
  sizeHeight = 90;
  moving = true;
}

void draw() {

  background(255);

  stroke(0);
  strokeWeight(1);

  for (int i = 0; i < width; i = i + 90) {
    line(0, i, 1280, i);
    line(i, 0, i, 720);
  }

  fill (255, 147, 79);
  int steppedPos = (int)(location/sizeHeight+0.5);
  rect(0, steppedPos*sizeHeight, sizeWidth, sizeHeight);

  if (moving) {
    location = location + direction;
    if (location + sizeHeight > height || location < 0){
      direction = direction * -1;
      location = location + direction;
    }
  }
}

void keyPressed() {
  if (key == ' ') {
    moving = !moving;
  }
}


问:
Enterprise Library Logging WriteLog Param
How can i add some custom params to WriteLog procedure (like userid or uri)?</p>

答: The approach taken in Enterprise Library 3.1 Logging Formatter Template - Include URL Request should allow you to do what you want.</p>

ContentLicense: CC BY-SA 3.0


问:
How can I compile a C# visual studio project without the default DLL file?
I'll just cut straight to the point.

I am making a compiler in C#, and creating it using visual studio. When I build my compiler to an executable, it also builds a "runtime" dll, how can I force visual studio to only build the executable without the dll?
(this is a .net core project)

hypec.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
  </PropertyGroup>

</Project>
```

<p>solution</p>

```
Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 16
VisualStudioVersion = 16.0.29509.3
MinimumVisualStudioVersion = 10.0.40219.1
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "hype", "hype.csproj", "{4653DC57-ACFF-419D-A57C-CEF0B5ECA174}"
EndProject
Global
    GlobalSection(SolutionConfigurationPlatforms) = preSolution
        Debug|Any CPU = Debug|Any CPU
        Release|Any CPU = Release|Any CPU
    EndGlobalSection
    GlobalSection(ProjectConfigurationPlatforms) = postSolution
        {4653DC57-ACFF-419D-A57C-CEF0B5ECA174}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
        {4653DC57-ACFF-419D-A57C-CEF0B5ECA174}.Debug|Any CPU.Build.0 = Debug|Any CPU
        {4653DC57-ACFF-419D-A57C-CEF0B5ECA174}.Release|Any CPU.ActiveCfg = Release|Any CPU
        {4653DC57-ACFF-419D-A57C-CEF0B5ECA174}.Release|Any CPU.Build.0 = Release|Any CPU
    EndGlobalSection
    GlobalSection(SolutionProperties) = preSolution
        HideSolutionNode = FALSE
    EndGlobalSection
    GlobalSection(ExtensibilityGlobals) = postSolution
        SolutionGuid = {8D8BF52E-5BB5-4F40-8365-9A6C3EBD042E}
    EndGlobalSection
EndGlobal
```

```
```

答:
From the reference here:
https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotnet-core-3-0

To publish a single-file executable, set the PublishSingleFile in your project or on the command line with the dotnet publish command:

```<code><PropertyGroup>
  <RuntimeIdentifier>win10-x64</RuntimeIdentifier>
  <PublishSingleFile>true</PublishSingleFile>
</PropertyGroup>
</code>```

OR

```<code>dotnet publish -r win10-x64 -p:PublishSingleFile=true
</code>```

Body:
I'll just cut straight to the point.

I am making a compiler in C#, and creating it using visual studio. When I build my compiler to an executable, it also builds a "runtime" dll, how can I force visual studio to only build the executable without the dll?
(this is a .net core project)

hypec.csproj

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp3.0</TargetFramework>
  </PropertyGroup>

</Project>

solution

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 16
VisualStudioVersion = 16.0.29509.3
MinimumVisualStudioVersion = 10.0.40219.1
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "hype", "hype.csproj", "{4653DC57-ACFF-419D-A57C-CEF0B5ECA174}"
EndProject
Global
    GlobalSection(SolutionConfigurationPlatforms) = preSolution
        Debug|Any CPU = Debug|Any CPU
        Release|Any CPU = Release|Any CPU
    EndGlobalSection
    GlobalSection(ProjectConfigurationPlatforms) = postSolution
        {4653DC57-ACFF-419D-A57C-CEF0B5ECA174}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
        {4653DC57-ACFF-419D-A57C-CEF0B5ECA174}.Debug|Any CPU.Build.0 = Debug|Any CPU
        {4653DC57-ACFF-419D-A57C-CEF0B5ECA174}.Release|Any CPU.ActiveCfg = Release|Any CPU
        {4653DC57-ACFF-419D-A57C-CEF0B5ECA174}.Release|Any CPU.Build.0 = Release|Any CPU
    EndGlobalSection
    GlobalSection(SolutionProperties) = preSolution
        HideSolutionNode = FALSE
    EndGlobalSection
    GlobalSection(ExtensibilityGlobals) = postSolution
        SolutionGuid = {8D8BF52E-5BB5-4F40-8365-9A6C3EBD042E}
    EndGlobalSection
EndGlobal

Body:
From the reference here:
https://learn.microsoft.com/en-us/dotnet/core/whats-new/dotnet-core-3-0

To publish a single-file executable, set the PublishSingleFile in your project or on the command line with the dotnet publish command:

<PropertyGroup>
  <RuntimeIdentifier>win10-x64</RuntimeIdentifier>
  <PublishSingleFile>true</PublishSingleFile>
</PropertyGroup>

OR

dotnet publish -r win10-x64 -p:PublishSingleFile=true


问:
How do I generate random integers within a specific range in Java?
How do I generate a random int value in a specific range?

The following methods have bugs related to integer overflow:

randomNum = minimum + (int)(Math.random() * maximum);
// Bug: `randomNum` can be bigger than `maximum`.
```
<pre class="lang-java prettyprint-override"><code>Random rn = new Random();
int n = maximum - minimum + 1;
int i = rn.nextInt() % n;
randomNum =  minimum + i;
// Bug: `randomNum` can be smaller than `minimum`.
</code>```

答:
Use Apache Lang3 Commons

```<code>Integer.parseInt(RandomStringUtils.randomNumeric(6, 6));
</code>```

Min Value 100000 to Max Value 999999

Body:
How do I generate a random int value in a specific range?

The following methods have bugs related to integer overflow:

randomNum = minimum + (int)(Math.random() * maximum);
// Bug: `randomNum` can be bigger than `maximum`.

Random rn = new Random();
int n = maximum - minimum + 1;
int i = rn.nextInt() % n;
randomNum =  minimum + i;
// Bug: `randomNum` can be smaller than `minimum`.

LastEditorDisplayName: mmyers

Body:
Use Apache Lang3 Commons

Integer.parseInt(RandomStringUtils.randomNumeric(6, 6));

Min Value 100000 to Max Value 999999


问:
Get text from ".txt" file and save it in String variable dart
I have a ".txt" file called "bio.txt" inside the Document folder of my application.
I want to read its content and store it inside a String variable.
My problem is that the "reading action" is a "Future" function and I have some troubles to save its value in a "String".

I read the file with :

Future<String> get _localPath async {
  final directory = await getApplicationDocumentsDirectory();
  return directory.path;
}

Future<File> _localFile(String name) async {
  final path = await _localPath;
  return File('$path/$name');
}

Future<String> _read(String filename) async {
  try {
    final file = await _localFile(filename);

    // Read the file.
    return await file.readAsString();
  } catch (e) {
    // If encountering an error, return 0.
    return "Can't read";
  }
}
```

<p>And I would like to save it with :</p>

<pre class="lang-dart prettyprint-override"><code>String _bio = _read("bio.txt").then((value) => String);
</code>```

Do you know why this solution isn't working?
Any suggestion?

答:
You can either use a then block or use the async/await

```<code>_read ('file.txt').then((value){
bio = value;
 });
</code>```

Or:

bio = await _read ('file.txt');

Body:
I have a ".txt" file called "bio.txt" inside the Document folder of my application.
I want to read its content and store it inside a String variable.
My problem is that the "reading action" is a "Future" function and I have some troubles to save its value in a "String".

I read the file with :

Future<String> get _localPath async {
  final directory = await getApplicationDocumentsDirectory();
  return directory.path;
}

Future<File> _localFile(String name) async {
  final path = await _localPath;
  return File('$path/$name');
}

Future<String> _read(String filename) async {
  try {
    final file = await _localFile(filename);

    // Read the file.
    return await file.readAsString();
  } catch (e) {
    // If encountering an error, return 0.
    return "Can't read";
  }
}

And I would like to save it with :

String _bio = _read("bio.txt").then((value) => String);

Do you know why this solution isn't working?
Any suggestion?

Body:
You can either use a then block or use the async/await

_read ('file.txt').then((value){
bio = value;
 });

Or:

bio = await _read ('file.txt');


问:
Android View.Animation size jumped when used with other set of animation. Why?
I follow the view.animation sample here https://developer.android.com/guide/topics/graphics/view-animation and encounter some strange problem.

To illustrate the problem I prolong the duration from 700ms to 2000ms with the same code as below.

```<code><set android:shareInterpolator="false"
    xmlns:android="http://schemas.android.com/apk/res/android">
    <scale
        android:interpolator="@android:anim/accelerate_decelerate_interpolator"
        android:fromXScale="1.0"
        android:toXScale="1.4"
        android:fromYScale="1.0"
        android:toYScale="1.4"
        android:pivotX="50%"
        android:pivotY="50%"
        android:fillAfter="false"
        android:duration="2000" />
    <set android:interpolator="@android:anim/decelerate_interpolator">
        <scale
            android:fromXScale="1.4"
            android:toXScale="0.0"
            android:fromYScale="1.4"
            android:toYScale="0.0"
            android:pivotX="50%"
            android:pivotY="50%"
            android:startOffset="2000"
            android:duration="400"
            android:fillBefore="false" />
        <rotate
            android:fromDegrees="0"
            android:toDegrees="-45"
            android:toYScale="0.0"
            android:pivotX="50%"
            android:pivotY="50%"
            android:startOffset="2000"
            android:duration="400" />
    </set>
</set>
</code>```

You notice the initial animation, the Hello World! size jumped at the start, as shown in the gif below. This is strange because the fromXScale and fromYScale start from 1.0 so I'm expecting a gradual size change instead of a jumped.

If I remove the second set of animation and leave the code as below

```<code><set android:shareInterpolator="false"
    xmlns:android="http://schemas.android.com/apk/res/android">
    <scale
        android:interpolator="@android:anim/accelerate_decelerate_interpolator"
        android:fromXScale="1.0"
        android:toXScale="1.4"
        android:fromYScale="1.0"
        android:toYScale="1.4"
        android:pivotX="50%"
        android:pivotY="50%"
        android:fillAfter="false"
        android:duration="2000" />
</set>
</code>```

then the animation looks smooth, increase it's size from 1.0 to 1.4 accordingly as shown below.

Why is the jump in size?

答:
Apparently the code we need to set the second set fromXScale and fromYScale to 1.0 first.

```<code><set android:shareInterpolator="false"
    xmlns:android="http://schemas.android.com/apk/res/android">
    <scale
        android:interpolator="@android:anim/accelerate_decelerate_interpolator"
        android:fromXScale="1.0"
        android:toXScale="1.4"
        android:fromYScale="1.0"
        android:toYScale="1.4"
        android:pivotX="50%"
        android:pivotY="50%"
        android:fillAfter="false"
        android:duration="2000" />
    <set android:interpolator="@android:anim/decelerate_interpolator">
        <scale
            android:fromXScale="1.0"
            android:toXScale="0.0"
            android:fromYScale="1.0"
            android:toYScale="0.0"
            android:pivotX="50%"
            android:pivotY="50%"
            android:startOffset="2000"
            android:duration="400"
            android:fillBefore="false" />
        <rotate
            android:fromDegrees="0"
            android:toDegrees="-45"
            android:toYScale="0.0"
            android:pivotX="50%"
            android:pivotY="50%"
            android:startOffset="2000"
            android:duration="400" />
    </set>
</set>
</code>```

Then the animation will be smooth as below.

The code in https://developer.android.com/guide/topics/graphics/view-animation does confuses a little on how that behaves.

Body:
I follow the view.animation sample here https://developer.android.com/guide/topics/graphics/view-animation and encounter some strange problem.

To illustrate the problem I prolong the duration from 700ms to 2000ms with the same code as below.

<set android:shareInterpolator="false"
    xmlns:android="http://schemas.android.com/apk/res/android">
    <scale
        android:interpolator="@android:anim/accelerate_decelerate_interpolator"
        android:fromXScale="1.0"
        android:toXScale="1.4"
        android:fromYScale="1.0"
        android:toYScale="1.4"
        android:pivotX="50%"
        android:pivotY="50%"
        android:fillAfter="false"
        android:duration="2000" />
    <set android:interpolator="@android:anim/decelerate_interpolator">
        <scale
            android:fromXScale="1.4"
            android:toXScale="0.0"
            android:fromYScale="1.4"
            android:toYScale="0.0"
            android:pivotX="50%"
            android:pivotY="50%"
            android:startOffset="2000"
            android:duration="400"
            android:fillBefore="false" />
        <rotate
            android:fromDegrees="0"
            android:toDegrees="-45"
            android:toYScale="0.0"
            android:pivotX="50%"
            android:pivotY="50%"
            android:startOffset="2000"
            android:duration="400" />
    </set>
</set>

You notice the initial animation, the Hello World! size jumped at the start, as shown in the gif below. This is strange because the fromXScale and fromYScale start from 1.0 so I'm expecting a gradual size change instead of a jumped.

If I remove the second set of animation and leave the code as below

<set android:shareInterpolator="false"
    xmlns:android="http://schemas.android.com/apk/res/android">
    <scale
        android:interpolator="@android:anim/accelerate_decelerate_interpolator"
        android:fromXScale="1.0"
        android:toXScale="1.4"
        android:fromYScale="1.0"
        android:toYScale="1.4"
        android:pivotX="50%"
        android:pivotY="50%"
        android:fillAfter="false"
        android:duration="2000" />
</set>

then the animation looks smooth, increase it's size from 1.0 to 1.4 accordingly as shown below.

Why is the jump in size?

Body:
Apparently the code we need to set the second set fromXScale and fromYScale to 1.0 first.

<set android:shareInterpolator="false"
    xmlns:android="http://schemas.android.com/apk/res/android">
    <scale
        android:interpolator="@android:anim/accelerate_decelerate_interpolator"
        android:fromXScale="1.0"
        android:toXScale="1.4"
        android:fromYScale="1.0"
        android:toYScale="1.4"
        android:pivotX="50%"
        android:pivotY="50%"
        android:fillAfter="false"
        android:duration="2000" />
    <set android:interpolator="@android:anim/decelerate_interpolator">
        <scale
            android:fromXScale="1.0"
            android:toXScale="0.0"
            android:fromYScale="1.0"
            android:toYScale="0.0"
            android:pivotX="50%"
            android:pivotY="50%"
            android:startOffset="2000"
            android:duration="400"
            android:fillBefore="false" />
        <rotate
            android:fromDegrees="0"
            android:toDegrees="-45"
            android:toYScale="0.0"
            android:pivotX="50%"
            android:pivotY="50%"
            android:startOffset="2000"
            android:duration="400" />
    </set>
</set>

Then the animation will be smooth as below.

The code in https://developer.android.com/guide/topics/graphics/view-animation does confuses a little on how that behaves.


问:
call a php function from jquery
I'm trying to figure out the best way to do this...I am using JQuery to submit data to a PHP function, which sends back data from the DB as JSON, which is working. The thing is, on success, I want the JQuery to execute a PHP function...and I'd rather not have to make yet another AJAX call on top of the first AJAX success - especially since the php function is something I've already used elsewhere on my page. This is my code:

JQUERY:

```<code>$.ajax({
    type: "POST",
    url: post_url,
    success: function(group) //we're calling the response json array 'tree'
    {
        //WANT TO CALL THE PHP FUNCTION HERE
    } //end success
 }); //end AJAX
</code>```

PHP:

```<code><?php
foreach($groups as $group){
    echo '<option value="' . $group->id . '">' . $group->group_name . '</option>';
}
?>
</code>```

答:
Alternatives:

Change the PHP that backs the original AJAX call so that, instead of sending back JSON, it sends back HTML exactly as you want it.

Chain a second AJAX call "inside" the success of the first, so that a second round-trip to the server is done for the JSON->HTML conversion.

Accept that sometimes you need to "duplicate" presentation-rules when you are working with multiple languages, and use something like EJS to do the same <option> stuff that PHP does elsewhere.

Title: call a php function from jquery

Body:
I'm trying to figure out the best way to do this...I am using JQuery to submit data to a PHP function, which sends back data from the DB as JSON, which is working. The thing is, on success, I want the JQuery to execute a PHP function...and I'd rather not have to make yet another AJAX call on top of the first AJAX success - especially since the php function is something I've already used elsewhere on my page. This is my code:

JQUERY:

$.ajax({
    type: "POST",
    url: post_url,
    success: function(group) //we're calling the response json array 'tree'
    {
        //WANT TO CALL THE PHP FUNCTION HERE
    } //end success
 }); //end AJAX

PHP:

<?php
foreach($groups as $group){
    echo '<option value="' . $group->id . '">' . $group->group_name . '</option>';
}
?>


问:
Yup validate password contains at least one special character
I am using the following yup:

```<code>export const validationSchema = Yup.object().shape({
  password: Yup.string()
    .required('Password is required')
    .matches(
      /^[!@#$%^&*()_+\-=\[\]{};':"\\|,.<>\/?]*$/,
      'Need one special character',
    ),
  confirmPassword: Yup.string()
    .oneOf([Yup.ref('password'), null], 'Passwords must match')
    .required('Password confirm is required'),
});
</code>```

to validate if there is one special character in password string.

```<code>@ => valid
@a => invalid
a@ => invalid
aa => invalid, of course
</code>```

I also test the regex pattern online to make sure this pattern will catch any special character in string
I wonder it could be yup bug or I do something wrong

答:
You can try the below regex:

```<code>^[0-9A-Za-z]*[!@#$%^&*()_+\-=\[\]{};':"\\|,.<>\/?][0-9a-zA-Z]*$
</code>```

Explanation of the above regex:

^, $ - Matches start and end of the given string resp.

[0-9A-Za-z]* - Matches any character in the given range zero or more times.

[!@#$%^&*()_+\-=\[\]{};':"\\|,.<>\/?] - Matches exactly one special character among the given set.

You can find the regex demo here.

Body:
I am using the following yup:

export const validationSchema = Yup.object().shape({
  password: Yup.string()
    .required('Password is required')
    .matches(
      /^[!@#$%^&*()_+\-=\[\]{};':"\\|,.<>\/?]*$/,
      'Need one special character',
    ),
  confirmPassword: Yup.string()
    .oneOf([Yup.ref('password'), null], 'Passwords must match')
    .required('Password confirm is required'),
});

to validate if there is one special character in password string.

@ => valid
@a => invalid
a@ => invalid
aa => invalid, of course

I also test the regex pattern online to make sure this pattern will catch any special character in string
I wonder it could be yup bug or I do something wrong

Body:
You can try the below regex:

^[0-9A-Za-z]*[!@#$%^&*()_+\-=\[\]{};':"\\|,.<>\/?][0-9a-zA-Z]*$

Explanation of the above regex:

^, $ - Matches start and end of the given string resp.

[0-9A-Za-z]* - Matches any character in the given range zero or more times.

[!@#$%^&*()_+\-=\[\]{};':"\\|,.<>\/?] - Matches exactly one special character among the given set.

You can find the regex demo here.


问:
Flask restplus exception is not caught by api.errorhandler
I want to define custom error handling for a Flask-restful API

The flask custom error(FlaskGenericException) is raised but is not handled by restplus errorhandler. it just throws a 500 Internal Server error.

Not sure why the dict defined inside exceptions.py is not thrown as api response.

Expected Result

```<code>{'user': 'Lionel Messi', 'company': 'Soccer Brazil','message': 'A generic error'}
</code>```

Here are my files that I am using to test this:

app.py

```<code>import json,os
import logging
import sys
import requests
import config
from os import environ
from flask import Flask, request, jsonify, Response
from flask_jwt_extended import (JWTManager, jwt_required)
from flask_restplus import Api, Resource, reqparse, fields, inputs
from cal_Test import calTest
from exceptions import FlaskGenericException

# Setup Flask Server
app = Flask(__name__,root_path=os.path.join(os.getcwd(), './'))

app.config.from_object(config.Config)
api = Api(app=app,
          version='0.1',
          title="Test Cal API",
          doc="/swagger/",
          description='TestCal = Cal Test')
ns = api.namespace('/', description='Cal Test Api')

@api.errorhandler(FlaskGenericException)
def handle_error(error):
    response = jsonify(error.to_dict())
    response.status_code = error.status_code
    return response

resource_parser = reqparse.RequestParser()
resource_parser.add_argument('test', type=str, required=True)

resource_flds = api.model('Resource', {
    'test': fields.String(required = True),
})

@ns.route('/test', methods=['POST'])
class TestCal(Resource):

    # @jwt_required
    @ns.doc(body=resource_flds)
    @ns.expect(resource_flds, validate=True)
    def post(self):
        request_data = resource_parser.parse_args()
        caltst = calTest()
        result = caltst.cal_test(request_data)
        response = app.response_class(
        response=json.dumps(result),
        status=200,
        mimetype='application/json'
    )
        return response

if __name__ == "__main__":
    app.run(debug=True)
</code>```

cal_Test.py

```<code>import pandas as pd
import pyodbc
import scipy.stats as st
import numpy as np
import pandas as pd
from scipy.stats import norm
from flask import Flask, request, jsonify
import os
from exceptions import FlaskGenericException

class calTest:

def __init__(self):
    self.test_name = 'Hi! I am test'

def cal_test(self,reqdata):
    if self.test_name == reqdata['test']:
        return "Successfull"
    else:
        raise FlaskGenericException('A generic error', status_code=404, payload={'user': 'Lionel Messi', 'company': 'Soccer Brazil'})
</code>```

exceptions.py

```<code>class FlaskGenericException(Exception):
    status_code = 500   # default unless overridden
    def __init__(self, message, status_code=None, payload=None):
        Exception.__init__(self)
        self.message = message
        if status_code is not None:
            self.status_code = status_code
        self.payload = payload

    def to_dict(self):
        rv = dict(self.payload or ())
        rv['message'] = self.message
        return rv
</code>```

答:
Ran into the same issue, I ended up using the abort method from Flask. I don't know if it fits your use case, but give it a try.

from flask import abort

def cal_test(self,reqdata):
    if self.test_name == reqdata['test']:
        return "Successfull"
    else:
        abort(404, "A generic error")
```

Body:
I want to define custom error handling for a Flask-restful API

The flask custom error(FlaskGenericException) is raised but is not handled by restplus errorhandler. it just throws a 500 Internal Server error.

Not sure why the dict defined inside exceptions.py is not thrown as api response.

Expected Result

{'user': 'Lionel Messi', 'company': 'Soccer Brazil','message': 'A generic error'}

Here are my files that I am using to test this:

app.py

import json,os
import logging
import sys
import requests
import config
from os import environ
from flask import Flask, request, jsonify, Response
from flask_jwt_extended import (JWTManager, jwt_required)
from flask_restplus import Api, Resource, reqparse, fields, inputs
from cal_Test import calTest
from exceptions import FlaskGenericException

# Setup Flask Server
app = Flask(__name__,root_path=os.path.join(os.getcwd(), './'))

app.config.from_object(config.Config)
api = Api(app=app,
          version='0.1',
          title="Test Cal API",
          doc="/swagger/",
          description='TestCal = Cal Test')
ns = api.namespace('/', description='Cal Test Api')

@api.errorhandler(FlaskGenericException)
def handle_error(error):
    response = jsonify(error.to_dict())
    response.status_code = error.status_code
    return response

resource_parser = reqparse.RequestParser()
resource_parser.add_argument('test', type=str, required=True)

resource_flds = api.model('Resource', {
    'test': fields.String(required = True),
})

@ns.route('/test', methods=['POST'])
class TestCal(Resource):

    # @jwt_required
    @ns.doc(body=resource_flds)
    @ns.expect(resource_flds, validate=True)
    def post(self):
        request_data = resource_parser.parse_args()
        caltst = calTest()
        result = caltst.cal_test(request_data)
        response = app.response_class(
        response=json.dumps(result),
        status=200,
        mimetype='application/json'
    )
        return response

if __name__ == "__main__":
    app.run(debug=True)

cal_Test.py

import pandas as pd
import pyodbc
import scipy.stats as st
import numpy as np
import pandas as pd
from scipy.stats import norm
from flask import Flask, request, jsonify
import os
from exceptions import FlaskGenericException

class calTest:

def __init__(self):
    self.test_name = 'Hi! I am test'

def cal_test(self,reqdata):
    if self.test_name == reqdata['test']:
        return "Successfull"
    else:
        raise FlaskGenericException('A generic error', status_code=404, payload={'user': 'Lionel Messi', 'company': 'Soccer Brazil'})

exceptions.py

class FlaskGenericException(Exception):
    status_code = 500   # default unless overridden
    def __init__(self, message, status_code=None, payload=None):
        Exception.__init__(self)
        self.message = message
        if status_code is not None:
            self.status_code = status_code
        self.payload = payload

    def to_dict(self):
        rv = dict(self.payload or ())
        rv['message'] = self.message
        return rv


问:
Scraping E-mails from Websites
I have tried several iterations from other posts and nothing seems to be helping or working for my needs.

I have a list of URLs that I want to loop through and pull all associated URLs that contain email addresses.  I then want to store the URLs and Email Addresses into a csv file.

For example, if I went to 10torr.com, the program should find each of the sites within the main URL (ie: 10torr.com/about) and pull any emails.

Below is a list of 5 example websites that are currently in a data frame format when run through my code.  They are saved under the variable small_site.

A helpful answer will include the use of the user defined function listed below called get_info().  Hard coding the the websites is into the Spider itself is not a feasible option as this will be used by many other people with different website list lengths.

```<code>    Website
    http://10torr.com/
    https://www.10000drops.com/
    https://www.11wells.com/
    https://117westspirits.com/
    https://www.onpointdistillery.com/
</code>```

Below is the code that I am running.  The spider seems to run, but there is no output in my csv file.

```<code>
import os
import pandas as pd
import re
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.linkextractors.lxmlhtml import LxmlLinkExtractor

small_site = site.head()

#%% Start Spider
class MailSpider(scrapy.Spider):

    name = 'email'

    def parse(self, response):

        links = LxmlLinkExtractor(allow=()).extract_links(response)
        links = [str(link.url) for link in links]
        links.append(str(response.url))

        for link in links:
            yield scrapy.Request(url=link, callback=self.parse_link)

    def parse_link(self, response):

        for word in self.reject:
            if word in str(response.url):
                return

        html_text = str(response.text)
        mail_list = re.findall('\w+@\w+\.{1}\w+', html_text)

        dic = {'email': mail_list, 'link': str(response.url)}
        df = pd.DataFrame(dic)

        df.to_csv(self.path, mode='a', header=False)
        df.to_csv(self.path, mode='a', header=False)

#%% Preps a CSV File
def ask_user(question):
    response = input(question + ' y/n' + '\n')
    if response == 'y':
        return True
    else:
        return False
def create_file(path):
    response = False
    if os.path.exists(path):
        response = ask_user('File already exists, replace?')
        if response == False: return

    with open(path, 'wb') as file:
        file.close()

#%% Defines function that will extract emails and enter it into CSV
def get_info(url_list, path, reject=[]):

    create_file(path)
    df = pd.DataFrame(columns=['email', 'link'], index=[0])
    df.to_csv(path, mode='w', header=True)

    print('Collecting Google urls...')
    google_urls = url_list

    print('Searching for emails...')
    process = CrawlerProcess({'USER_AGENT': 'Mozilla/5.0'})
    process.start()

    for i in small_site.Website.iteritems():
        print('Searching for emails...')
        process.crawl(MailSpider, start_urls=google_urls, path=path, reject=reject)
        ##process.start()

        print('Cleaning emails...')
        df = pd.read_csv(path, index_col=0)
        df.columns = ['email', 'link']
        df = df.drop_duplicates(subset='email')
        df = df.reset_index(drop=True)
        df.to_csv(path, mode='w', header=True)

    return df

url_list = small_site
path = 'email.csv'

df = get_info(url_list, path)
</code>```

I am not certain where I am going wrong as I am not getting any error messages.  If you need additional information please just ask.  I have been trying to get this for almost a month now and I feel like I am just banging my head against the wall at this point.

The majority of this code was found on the article Web scraping to extract contact information— Part 1: Mailing Lists after a few weeks.  However, I have not been successful in expanding it to my needs.  It worked no problem with one offs while incorporating their google search function to get the base URLs.

Thank you in advance for any assistance you are able to provide.

答:
I modified some scripts are ran the following script via Shell and it works. May be it will provide you as an starting point.

I advise you to use the shell as it always throws errors and other messages during the scraping process

class MailSpider(scrapy.Spider):

    name = 'email'
    start_urls = [
        'http://10torr.com/',
        'https://www.10000drops.com/',
        'https://www.11wells.com/',
        'https://117westspirits.com/',
        'https://www.onpointdistillery.com/',
    ]

    def parse(self, response):
        self.log('A response from %s just arrived!' % response.url)
        links = LxmlLinkExtractor(allow=()).extract_links(response)
        links = [str(link.url) for link in links]
        links.append(str(response.url))

        for link in links:
            yield scrapy.Request(url=link, callback=self.parse_link)

    def parse_link(self, response):

        html_text = str(response.text)
        mail_list = re.findall('\w+@\w+\.{1}\w+', html_text)

        dic = {'email': mail_list, 'link': str(response.url)}

        for key in dic.keys():
            yield {
                'email' : dic['email'],
                'link': dic['link'],
            }
```

<p>This yields the following <strong>output</strong> when crawled via <strong>Anaconda shell</strong> <code>scrapy crawl email -o test.jl</code></p>

```{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/"}
{"email": ["8b4e078a51d04e0e9efdf470027f0ec1@sentry.wixpress", "bundle@3.2", "fetch@3.0", "bolt@2.3", "5oclock@11wells.com", "5oclock@11wells.com", "5oclock@11wells.com"], "link": "https://www.11wells.com"}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/shop?olsPage=search&keywords="}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/shop?olsPage=search&keywords="}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/shop"}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/shop?olsPage=cart"}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/home"}
{"email": ["8b4e078a51d04e0e9efdf470027f0ec1@sentry.wixpress", "bundle@3.2", "fetch@3.0", "bolt@2.3", "5oclock@11wells.com", "5oclock@11wells.com", "5oclock@11wells.com"], "link": "https://www.11wells.com"}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/home"}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/117%C2%B0-west-spirits-1"}
...
...
...
```

Refer Scrapy docs for more information

Title: Scraping E-mails from Websites

Body:
I have tried several iterations from other posts and nothing seems to be helping or working for my needs.

I have a list of URLs that I want to loop through and pull all associated URLs that contain email addresses.  I then want to store the URLs and Email Addresses into a csv file.

For example, if I went to 10torr.com, the program should find each of the sites within the main URL (ie: 10torr.com/about) and pull any emails.

Below is a list of 5 example websites that are currently in a data frame format when run through my code.  They are saved under the variable small_site.

A helpful answer will include the use of the user defined function listed below called get_info().  Hard coding the the websites is into the Spider itself is not a feasible option as this will be used by many other people with different website list lengths.

    Website
    http://10torr.com/
    https://www.10000drops.com/
    https://www.11wells.com/
    https://117westspirits.com/
    https://www.onpointdistillery.com/

Below is the code that I am running.  The spider seems to run, but there is no output in my csv file.

import os
import pandas as pd
import re
import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.linkextractors.lxmlhtml import LxmlLinkExtractor

small_site = site.head()

#%% Start Spider
class MailSpider(scrapy.Spider):

    name = 'email'

    def parse(self, response):

        links = LxmlLinkExtractor(allow=()).extract_links(response)
        links = [str(link.url) for link in links]
        links.append(str(response.url))

        for link in links:
            yield scrapy.Request(url=link, callback=self.parse_link)

    def parse_link(self, response):

        for word in self.reject:
            if word in str(response.url):
                return

        html_text = str(response.text)
        mail_list = re.findall('\w+@\w+\.{1}\w+', html_text)

        dic = {'email': mail_list, 'link': str(response.url)}
        df = pd.DataFrame(dic)

        df.to_csv(self.path, mode='a', header=False)
        df.to_csv(self.path, mode='a', header=False)

#%% Preps a CSV File
def ask_user(question):
    response = input(question + ' y/n' + '\n')
    if response == 'y':
        return True
    else:
        return False
def create_file(path):
    response = False
    if os.path.exists(path):
        response = ask_user('File already exists, replace?')
        if response == False: return

    with open(path, 'wb') as file:
        file.close()

#%% Defines function that will extract emails and enter it into CSV
def get_info(url_list, path, reject=[]):

    create_file(path)
    df = pd.DataFrame(columns=['email', 'link'], index=[0])
    df.to_csv(path, mode='w', header=True)

    print('Collecting Google urls...')
    google_urls = url_list

    print('Searching for emails...')
    process = CrawlerProcess({'USER_AGENT': 'Mozilla/5.0'})
    process.start()

    for i in small_site.Website.iteritems():
        print('Searching for emails...')
        process.crawl(MailSpider, start_urls=google_urls, path=path, reject=reject)
        ##process.start()

        print('Cleaning emails...')
        df = pd.read_csv(path, index_col=0)
        df.columns = ['email', 'link']
        df = df.drop_duplicates(subset='email')
        df = df.reset_index(drop=True)
        df.to_csv(path, mode='w', header=True)

    return df

url_list = small_site
path = 'email.csv'

df = get_info(url_list, path)

I am not certain where I am going wrong as I am not getting any error messages.  If you need additional information please just ask.  I have been trying to get this for almost a month now and I feel like I am just banging my head against the wall at this point.

The majority of this code was found on the article Web scraping to extract contact information— Part 1: Mailing Lists after a few weeks.  However, I have not been successful in expanding it to my needs.  It worked no problem with one offs while incorporating their google search function to get the base URLs.

Thank you in advance for any assistance you are able to provide.

Body:
I modified some scripts are ran the following script via Shell and it works. May be it will provide you as an starting point.

I advise you to use the shell as it always throws errors and other messages during the scraping process

class MailSpider(scrapy.Spider):

    name = 'email'
    start_urls = [
        'http://10torr.com/',
        'https://www.10000drops.com/',
        'https://www.11wells.com/',
        'https://117westspirits.com/',
        'https://www.onpointdistillery.com/',
    ]

    def parse(self, response):
        self.log('A response from %s just arrived!' % response.url)
        links = LxmlLinkExtractor(allow=()).extract_links(response)
        links = [str(link.url) for link in links]
        links.append(str(response.url))

        for link in links:
            yield scrapy.Request(url=link, callback=self.parse_link)

    def parse_link(self, response):

        html_text = str(response.text)
        mail_list = re.findall('\w+@\w+\.{1}\w+', html_text)

        dic = {'email': mail_list, 'link': str(response.url)}

        for key in dic.keys():
            yield {
                'email' : dic['email'],
                'link': dic['link'],
            }

This yields the following output when crawled via Anaconda shell scrapy crawl email -o test.jl

{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/"}
{"email": ["8b4e078a51d04e0e9efdf470027f0ec1@sentry.wixpress", "bundle@3.2", "fetch@3.0", "bolt@2.3", "5oclock@11wells.com", "5oclock@11wells.com", "5oclock@11wells.com"], "link": "https://www.11wells.com"}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/shop?olsPage=search&keywords="}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/shop?olsPage=search&keywords="}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/shop"}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/shop?olsPage=cart"}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/home"}
{"email": ["8b4e078a51d04e0e9efdf470027f0ec1@sentry.wixpress", "bundle@3.2", "fetch@3.0", "bolt@2.3", "5oclock@11wells.com", "5oclock@11wells.com", "5oclock@11wells.com"], "link": "https://www.11wells.com"}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/home"}
{"email": ["info@ndiscovered.com"], "link": "https://117westspirits.com/117%C2%B0-west-spirits-1"}
...
...
...

Refer Scrapy docs for more information


问:
How to get a list of the arguments a method is called with
How do I get a list of the arguments passed to a method, preferably one that I can iterate through?

For example something like

```<code>def foo(a,b,c)
  puts args.inspect
end

foo(1,2,3)
=> [1,2,3]
</code>```

?
Thanks!

答:
If you define your method as you specified, you'll always have 3 args, or the method call is invalid. So "all the args" is already defined for you. So you would just change your method to:

```<code>def foo(a,b,c)
  [a, b, c]
end
</code>```

To define a method that can be called with any args (and to then access those args) you can do something like this:

```<code>def foo(*args)
  args
end
</code>```

What the * does is put all args after that point into an array.

Body:
How do I get a list of the arguments passed to a method, preferably one that I can iterate through?

For example something like

def foo(a,b,c)
  puts args.inspect
end

foo(1,2,3)
=> [1,2,3]

?
Thanks!

Body:
If you define your method as you specified, you'll always have 3 args, or the method call is invalid. So "all the args" is already defined for you. So you would just change your method to:

def foo(a,b,c)
  [a, b, c]
end

To define a method that can be called with any args (and to then access those args) you can do something like this:

def foo(*args)
  args
end

What the * does is put all args after that point into an array.


问:
How to catch the longest sequence of a group
The task is to find the longest sequence of a group

for instance, given DNA sequence: "AGATCAGATCTTTTTTCTAATGTCTAGGATATATCAGATCAGATCAGATCAGATCAGATC"
and it has 7 occurrences of AGATC. (AGATC) matches all occurrences.
Is it possible to write a regular expression that catches only the longest sequence, i.e. AGATCAGATCAGATCAGATCAGATC in the given text?
If this is not possible only with regex, how can I iterate through each sequence (i.e. 1st sequence is AGATCAGATC, 2nd - AGATCAGATCAGATCAGATCAGATC et cetera) in python?

答:
Use re.finditer() to iterate over all matches.  Then use max() with a key function to find the longest.  Make it a function so you can use different groups.

```<code>import re

def find_longest(sequence, group):
    # build pattern
    pattern = fr"(?:{group})+"

    # iterate over all matches
    matches = (match[0] for match in re.finditer(pattern, sequence))

    # find the longest
    return max(matches, key=len)

seq = "AGATCAGATCTTTTTTCTAATGTCTAGGATATATCAGATCAGATCAGATCAGATCAGATC"

find_longest(seq, "AGATC")
</code>```

Body:
Use re.finditer() to iterate over all matches.  Then use max() with a key function to find the longest.  Make it a function so you can use different groups.

import re

def find_longest(sequence, group):
    # build pattern
    pattern = fr"(?:{group})+"

    # iterate over all matches
    matches = (match[0] for match in re.finditer(pattern, sequence))

    # find the longest
    return max(matches, key=len)

seq = "AGATCAGATCTTTTTTCTAATGTCTAGGATATATCAGATCAGATCAGATCAGATCAGATC"

find_longest(seq, "AGATC")


问:
How do I pass extra arguments to a Python decorator?
I have a decorator like below.

```<code>def myDecorator(test_func):
    return callSomeWrapper(test_func)
def callSomeWrapper(test_func):
    return test_func
@myDecorator
def someFunc():
    print 'hello'
</code>```

I want to enhance this decorator to accept another argument like below

```<code>def myDecorator(test_func,logIt):
    if logIt:
        print "Calling Function: " + test_func.__name__
    return callSomeWrapper(test_func)
@myDecorator(False)
def someFunc():
    print 'Hello'
</code>```

But this code gives the error,

TypeError: myDecorator() takes exactly 2 arguments (1 given)

Why is the function not automatically passed? How do I explicitly pass the function to the decorator function?

答:
Now if you want to call a function function1 with a decorator decorator_with_arg and in this case both the function and the decorator take arguments,

```<code>def function1(a, b):
    print (a, b)

decorator_with_arg(10)(function1)(1, 2)
</code>```

Body:
I have a decorator like below.

def myDecorator(test_func):
    return callSomeWrapper(test_func)
def callSomeWrapper(test_func):
    return test_func
@myDecorator
def someFunc():
    print 'hello'

I want to enhance this decorator to accept another argument like below

def myDecorator(test_func,logIt):
    if logIt:
        print "Calling Function: " + test_func.__name__
    return callSomeWrapper(test_func)
@myDecorator(False)
def someFunc():
    print 'Hello'

But this code gives the error,

TypeError: myDecorator() takes exactly 2 arguments (1 given)

Why is the function not automatically passed? How do I explicitly pass the function to the decorator function?

Body:
Now if you want to call a function function1 with a decorator decorator_with_arg and in this case both the function and the decorator take arguments,

def function1(a, b):
    print (a, b)

decorator_with_arg(10)(function1)(1, 2)


问:
How to get elastic search mapping properties name from react frontend
I want to know how to get only the properties inside the mapping from using react frontend. this mappings is in my backend. I want to get only the properties  not the value using elastic search query .backend is nodejs one .
Output should be the account, class, team_members.id, team_members.name, team_members.priority .</p>

```<code>  "mappings": {
            "properties": {
                "account": {"type": "integer"},
                  "class" : { "type" : "keyword",
                            "normalizer" : "my_normalizer"
                          },
                "team_members": {
                    "properties": {
                        "id": {"type": "integer"},
                        "name": {"type": "text"},
                        "priority": {"type": "integer"}
                    }
               }
          }
    }
</code>```

答:
You need to call _mapping endpoint on the ElasticSearch Cluster with the index you want the mappings for. Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#view-mapping

At Frontend, you may have to write a parser that will take this mapping and return the fields. You will need custom logic to handle this.

The mappings you specified, I think belong to ElasticSearch v7.

Note: You may have to write your own parser logic in this case as there can be other solutions and each solution will depend on requirements.

Body:
I want to know how to get only the properties inside the mapping from using react frontend. this mappings is in my backend. I want to get only the properties  not the value using elastic search query .backend is nodejs one .
Output should be the account, class, team_members.id, team_members.name, team_members.priority .

  "mappings": {
            "properties": {
                "account": {"type": "integer"},
                  "class" : { "type" : "keyword",
                            "normalizer" : "my_normalizer"
                          },
                "team_members": {
                    "properties": {
                        "id": {"type": "integer"},
                        "name": {"type": "text"},
                        "priority": {"type": "integer"}
                    }
               }
          }
    }


问:
How to create a PHP page that loads dynamic/different content depending on which link is clicked?
I have an HTML page with a list of  links. Each link will link to the same PHP page. I want the PHP page to dynamically load a different  link depending on which link id is clicked on the previous HTML page.

HTML Page ↓

```<code><a id="link1" href="test.php">Download</a>
<a id="link2" href="test.php">Download</a>
<a id="link3" href="test.php">Download</a>
...
</code>```

PHP page will print out a specific link depending on which a id was clicked on the previous HTML page ↓

```<code><a id="link1" href="thisisaspecificlink.com">Link here</a>
</code>```

I'm a total noob at PHP and would greatly appreciate assistance of any kind.

答:
There are many ways to achieve these but at the most basic level you can pass a GET variable and use that as your link:

e.g.

```<code><a id="link1" href="test.php?link=download.com">Download</a>
<a id="link2" href="test.php?link=download2.com">Download</a>
<a id="link3" href="test.php?link=download3.com">Download</a>

<a href="http://<?=$_GET['link'];?>">Link here</a>
</code>```

In this example you are passing link as a $_GET variable and then using it in the  tag on the 2nd page

Another way is to, again pass a $_GET variable but then use an IF statement on the 2nd page e.g.

```<code><a id="link1" href="test.php?link=1">Download</a>
<a id="link2" href="test.php?link=2">Download</a>
<a id="link3" href="test.php?link=3">Download</a>

$link = $_GET['link'];
if($link == 1) { $linkURL = "http://www.download.com";}
if($link == 2) { $linkURL = "http://www.download2.com";}
if($link == 3) { $linkURL = "http://www.download3.com";}

<a href="<?=$linkURL;?>">Link here</a>
</code>```

The above example is a little bit longer but wanted to include so you can see different ways of achieving what you're looking for. And give you a better idea of how to pass and handle variables from page to page.

Body:
I have an HTML page with a list of  links. Each link will link to the same PHP page. I want the PHP page to dynamically load a different  link depending on which link id is clicked on the previous HTML page.

HTML Page ↓

Download
Download
Download
...

PHP page will print out a specific link depending on which a id was clicked on the previous HTML page ↓

Link here

I'm a total noob at PHP and would greatly appreciate assistance of any kind.

Body:
There are many ways to achieve these but at the most basic level you can pass a GET variable and use that as your link:

e.g.

<a id="link1" href="test.php?link=download.com">Download</a>
<a id="link2" href="test.php?link=download2.com">Download</a>
<a id="link3" href="test.php?link=download3.com">Download</a>

<a href="http://<?=$_GET['link'];?>">Link here</a>

In this example you are passing link as a $_GET variable and then using it in the <a> tag on the 2nd page

Another way is to, again pass a $_GET variable but then use an IF statement on the 2nd page e.g.

<a id="link1" href="test.php?link=1">Download</a>
<a id="link2" href="test.php?link=2">Download</a>
<a id="link3" href="test.php?link=3">Download</a>

$link = $_GET['link'];
if($link == 1) { $linkURL = "http://www.download.com";}
if($link == 2) { $linkURL = "http://www.download2.com";}
if($link == 3) { $linkURL = "http://www.download3.com";}

<a href="<?=$linkURL;?>">Link here</a>

The above example is a little bit longer but wanted to include so you can see different ways of achieving what you're looking for. And give you a better idea of how to pass and handle variables from page to page.


问:
Open CSV: open csv how to show quote character only one field
I'm using OpenCsv library to create a .csv file in my Spring Boot application.
I need to put double quotaion symbol at the beginning and the end of just one field avoiding in the rest.

Here is that field named description that I want not to avoid ",

```<code>"\"" + description.replaceAll(",","").replaceAll("\"","").replaceAll("\n","").replaceAll("\r","") + "\""
</code>```

Here is my CSVWriter implementation,

```<code>StatefulBeanToCsv<DtoClass> beanToCsv = new StatefulBeanToCsvBuilder<DtoClass>(writer)
                .withQuotechar(CSVWriter.NO_QUOTE_CHARACTER)
                .withMappingStrategy(mapStrategy)
                .withSeparator(',')
                .build();

beanToCsv.write(dtoList);
</code>```

But when I see in the result .csv file the library adds an additional " in the beginning and the end,

For ex:
The string I'm passing:

```<code>"Export Logs and save it for future reference."
</code>```

What it writes in the resulted .csv file,

```<code>""Export Logs and save it for future reference.""
</code>```

How can I avoid having this extra double quote?

My Environment: Spring Boot 2.2.4 | Java 8 | com.opencsv:opencsv:5.1

答:
Not sure if it is possible to change the library but it is easier in jackson-dataformat-csv</p>

```<code><dependency>
    <groupId>com.fasterxml.jackson.dataformat</groupId>
    <artifactId>jackson-dataformat-csv</artifactId>
    <version>2.6.3</version></dependency>

CsvMapper mapper = new CsvMapper();
CsvSchema schema = mapper.schemaFor(objects).withHeader();
schema = schema..withoutQuoteChar();
</code>```

Body:
I'm using OpenCsv library to create a .csv file in my Spring Boot application.
I need to put double quotaion symbol at the beginning and the end of just one field avoiding in the rest.

Here is that field named description that I want not to avoid ",

"\"" + description.replaceAll(",","").replaceAll("\"","").replaceAll("\n","").replaceAll("\r","") + "\""

Here is my CSVWriter implementation,

StatefulBeanToCsv<DtoClass> beanToCsv = new StatefulBeanToCsvBuilder<DtoClass>(writer)
                .withQuotechar(CSVWriter.NO_QUOTE_CHARACTER)
                .withMappingStrategy(mapStrategy)
                .withSeparator(',')
                .build();

beanToCsv.write(dtoList);

But when I see in the result .csv file the library adds an additional " in the beginning and the end,

For ex:
The string I'm passing:

"Export Logs and save it for future reference."

What it writes in the resulted .csv file,

""Export Logs and save it for future reference.""

How can I avoid having this extra double quote?

My Environment: Spring Boot 2.2.4 | Java 8 | com.opencsv:opencsv:5.1

Body:
Not sure if it is possible to change the library but it is easier in jackson-dataformat-csv

<dependency>
    <groupId>com.fasterxml.jackson.dataformat</groupId>
    <artifactId>jackson-dataformat-csv</artifactId>
    <version>2.6.3</version></dependency>

CsvMapper mapper = new CsvMapper();
CsvSchema schema = mapper.schemaFor(objects).withHeader();
schema = schema..withoutQuoteChar();


Title: How do I calculate the overlap days in a dataframe in PySpark?

Body:
I need to calculate the overlap days, row wise, in a data frame.
Data looks like this:

+-------+-------------------+-------------------+------------------+
|id|             begin|                end|              days|
+-------+-------------------+-------------------+------------------+
|1|2019-01-01 00:00:00|2019-01-08 02:10:00| 7.090277777777778|
|1|2019-02-04 05:28:00|2019-03-05 19:29:00|29.584027777777777|
|1|2019-06-05 22:18:00|2020-01-01 00:00:00|209.07083333333333|
|1|2019-05-17 16:25:00|2019-06-05 22:18:00| 19.24513888888889|
|1|2019-05-03 05:05:00|2019-05-17 16:25:00|14.472222222222221|
|1|2019-01-08 02:10:00|2019-02-04 05:28:00|           27.1375|
|1|2019-01-01 00:00:00|2020-01-01 00:00:00|             365.0|
|1|2019-04-22 18:45:00|2019-05-03 05:05:00|10.430555555555555|
|1|2019-03-05 19:29:00|2019-04-22 18:45:00| 47.96944444444444|
+-------+-------------------+-------------------+------------------+

Here, the first entry spans across the year 2019 (365 days).
All other entries are overlapping with the first entry.
I want a function to calculate total days, which is 365 days in the dataset after removing the overlapping days.

I actually solved this problem in R but I'm unable to run for loops in PySpark.

I'm looking for an output like this.

+-------+-------------------+-------------------+------------------+------------------+
|     id|              begin|                end|              days|           overlap|
+-------+-------------------+-------------------+------------------+------------------+
|1      |2019-01-01 00:00:00|2020-01-01 00:00:00|             365.0|            0|
|1      |2019-01-01 00:00:00|2019-01-08 02:10:00| 7.090277777777778| 7.090277777777778|
|1      |2019-01-08 02:10:00|2019-02-04 05:28:00|           27.1375|           27.1375|
|1      |2019-02-04 05:28:00|2019-03-05 19:29:00|29.584027777777777|29.584027777777777|
|1      |2019-03-05 19:29:00|2019-04-22 18:45:00| 47.96944444444444| 47.96944444444444|
|1      |2019-04-22 18:45:00|2019-05-03 05:05:00|10.430555555555555|10.430555555555555|
|1      |2019-05-03 05:05:00|2019-05-17 16:25:00|14.472222222222221|14.472222222222221|
|1      |2019-05-17 16:25:00|2019-06-05 22:18:00| 19.24513888888889| 19.24513888888889|
|1      |2019-06-05 22:18:00|2020-01-01 00:00:00|209.07083333333333|209.07083333333333|
+-------+-------------------+-------------------+------------------+------------------+

The dates are never in order and there are scenarios where there is no overlap.

Scenario 2: No overlap

+-------+-------------------+-------------------+-----+-----+
|  id   |              begin|                end| days| over|
+-------+-------------------+-------------------+-----+-----+
|2      |2019-01-01 00:00:00|2019-12-25 00:00:00|358.0|    0|
|2      |2019-12-25 00:00:00|2020-01-01 00:00:00|  7.0|    0|
+-------+-------------------+-------------------+-----+-----+

Scenario 3: Partial overlap

+-------+-------------------+-------------------+-----+-----+
|     id|              begin|                end| days| over|
+-------+-------------------+-------------------+-----+-----+
|3      |2019-01-01 00:00:00|2019-12-25 00:00:00|358.0|    0|
|3      |2019-12-20 00:00:00|2020-01-01 00:00:00| 12.0|    5|
+-------+-------------------+-------------------+-----+-----+

Scenario 4: More complicated
Here the first entry spans across first 358 days of 2019. Second entry is completely overlapping with the first and therefore all of the days are under over. The third entry doesn't overlap with second entry but overlaps 5 days with the first and therefore 5 days under column "over".

+-------+-------------------+-------------------+-----+-----+
|     id|              begin|                end| days| over|
+-------+-------------------+-------------------+-----+-----+
|4      |2019-01-01 00:00:00|2019-12-25 00:00:00|358.0|    0|
|4      |2019-01-01 00:00:00|2019-11-25 00:00:00|328.0|328.0|
|4      |2019-12-20 00:00:00|2020-01-01 00:00:00| 12.0|    5|
+-------+-------------------+-------------------+-----+-----+

Basically, I want to know for how long a particular id was in force. I can't just take the max and min date and subtract them because there could be breaks between periods.

In R, I created another column called "overlap" and used the Overlap function in for loop to check all values against others.

R function that produces the desired output:

abc<-data.frame()
for (i in id) {
  xyz<- dataset %>% filter(id==i) %>% arrange(begin)

  for(j in 1:(nrow(xyz)-1)){
    k=j
    while(k<nrow(xyz)){
      xyz$overlap[j]<- xyz$overlap[j] + Overlap(c(xyz$begin[j], xyz$end[j]), c(xyz$begin[k+1], xyz$end[k+1]))
      k=k+1
    }

  }
  abc<- bind_rows(abc,xyz)
}

I'm still learning pyspark and need help with this.

Response to the code snippet by @murtihash

Hi,
It looks closer to the answer but still not the result I'm looking for.
Output from your code

+-------+-------------------+-------------------+-----------------+-------+
|     id|              begin|                end|             days|overlap|
+-------+-------------------+-------------------+-----------------+-------+
|7777777|2019-01-05 01:00:00|2019-04-04 00:00:00|88.95833333333333|      0|
|7777777|2019-04-04 00:00:00|2019-07-11 00:00:00|             98.0|      0|
|7777777|2019-07-11 00:00:00|2019-09-17 00:00:00|             68.0|      1|
|7777777|2019-09-17 00:00:00|2019-09-19 22:01:00|2.917361111111111|      0|
|7777777|2019-09-19 22:01:00|2020-01-01 00:00:00|103.0826388888889|     -1|
|7777777|2019-09-19 22:01:00|2020-01-01 00:00:00|103.0826388888889|     -1|
+-------+-------------------+-------------------+-----------------+-------+

Desired output should be:

+-------+-------------------+-------------------+-----------------+-------+
|     id|              begin|                end|             days|overlap|
+-------+-------------------+-------------------+-----------------+-------+
|7777777|2019-01-05 01:00:00|2019-04-04 00:00:00|88.95833333333333|      0|
|7777777|2019-04-04 00:00:00|2019-07-11 00:00:00|             98.0|      0|
|7777777|2019-07-11 00:00:00|2019-09-17 00:00:00|             68.0|      0|
|7777777|2019-09-17 00:00:00|2019-09-19 22:01:00|2.917361111111111|      0|
|7777777|2019-09-19 22:01:00|2020-01-01 00:00:00|103.0826388888889|103.082|
|7777777|2019-09-19 22:01:00|2020-01-01 00:00:00|103.0826388888889|      0|
+-------+-------------------+-------------------+-----------------+-------+

Explanation: First four rows have no overlaps. The fifth and sixth row are exactly the same period(and not overlapping with the other rows), so for one of 5th or 6th row, the overlap should be 103.08 days

Update:
Not able to work on this particular scenario.
Output from code snippet @murtihash

+-------+-------------------+-------------------+------------------+-------+
|  imono|              begin|                end|              days|overlap|
+-------+-------------------+-------------------+------------------+-------+
|9347774|2019-01-01 00:00:00|2019-01-08 02:10:00| 7.090277777777778|    0.0|
|9347774|2019-01-08 02:10:00|2019-02-04 05:28:00|           27.1375|    0.0|
|9347774|2019-02-04 05:28:00|2019-03-05 19:29:00|29.584027777777777|    0.0|
|9347774|2019-03-05 19:29:00|2019-04-22 18:45:00| 47.96944444444444|    0.0|
|9347774|2019-04-22 18:45:00|2019-05-03 05:05:00|10.430555555555555|    0.0|
|9347774|2019-05-03 05:05:00|2019-05-17 16:25:00|14.472222222222221|    0.0|
|9347774|2019-05-17 16:25:00|2019-06-05 22:18:00| 19.24513888888889|    0.0|
|9347774|2019-01-01 00:00:00|2020-01-01 00:00:00|             365.0|    7.0|
|9347774|2019-06-05 22:18:00|2020-01-01 00:00:00|209.07083333333333|    0.0|
+-------+-------------------+-------------------+------------------+-------+

Desired output:
This

 +-------+-------------------+-------------------+------------------+-------+
    |  imono|              begin|                end|              days|overlap|
    +-------+-------------------+-------------------+------------------+-------+
    |9347774|2019-01-01 00:00:00|2019-01-08 02:10:00| 7.090277777777778|    0.0|
    |9347774|2019-01-08 02:10:00|2019-02-04 05:28:00|           27.1375|    0.0|
    |9347774|2019-02-04 05:28:00|2019-03-05 19:29:00|29.584027777777777|    0.0|
    |9347774|2019-03-05 19:29:00|2019-04-22 18:45:00| 47.96944444444444|    0.0|
    |9347774|2019-04-22 18:45:00|2019-05-03 05:05:00|10.430555555555555|    0.0|
    |9347774|2019-05-03 05:05:00|2019-05-17 16:25:00|14.472222222222221|    0.0|
    |9347774|2019-05-17 16:25:00|2019-06-05 22:18:00| 19.24513888888889|    0.0|
    |9347774|2019-01-01 00:00:00|2020-01-01 00:00:00|             365.0|    365|
    |9347774|2019-06-05 22:18:00|2020-01-01 00:00:00|209.07083333333333|    0.0|
    +-------+-------------------+-------------------+------------------+-------+

Or

 +-------+-------------------+-------------------+------------------+-------+
|  imono|              begin|                end|              days|overlap|

+-------+-------------------+-------------------+------------------+-------+
|9347774|2019-01-01 00:00:00|2019-01-08 02:10:00| 7.090277777777778|    7.1|
|9347774|2019-01-08 02:10:00|2019-02-04 05:28:00|           27.1375|   27.1|
|9347774|2019-02-04 05:28:00|2019-03-05 19:29:00|29.584027777777777|   29.5|
|9347774|2019-03-05 19:29:00|2019-04-22 18:45:00| 47.96944444444444|   48.0|
|9347774|2019-04-22 18:45:00|2019-05-03 05:05:00|10.430555555555555|   10.4|
|9347774|2019-05-03 05:05:00|2019-05-17 16:25:00|14.472222222222221|   14.5|
|9347774|2019-05-17 16:25:00|2019-06-05 22:18:00| 19.24513888888889|   19.2|
|9347774|2019-01-01 00:00:00|2020-01-01 00:00:00|             365.0|    0.0|
|9347774|2019-06-05 22:18:00|2020-01-01 00:00:00|209.07083333333333|  209.1|
+-------+-------------------+-------------------+------------------+-------+

Explanation: second last entry spans the full year and all other entries are overlapping with that. So either the output is that the second last entry overlap= 365 or all the other entries have their days as overlap and second last entry have 0 days as overlap.

Update2:
Not able to work on this particular scenario.
Output from code snippet @murtihash (Update2)

+-------+-------------------+-------------------+------------------+-------+
|  imono|              begin|                end|              days|overlap|
+-------+-------------------+-------------------+------------------+-------+
|9395123|2019-01-19 05:01:00|2019-02-06 00:00:00|17.790972222222223|   17.0|
|9395123|2019-02-06 00:00:00|2019-06-17 00:00:00|             131.0|    0.0|
|9395123|2019-01-19 05:01:00|2020-01-01 00:00:00| 346.7909722222222|    0.0|
|9395123|2019-06-17 00:00:00|2020-01-01 00:00:00|             198.0|    0.0|
+-------+-------------------+-------------------+------------------+-------+

Desired output:

+-------+-------------------+-------------------+------------------+-------+
|  id   |              begin|                end|              days|overlap|
+-------+-------------------+-------------------+------------------+-------+
|8888888|2019-01-19 05:01:00|2019-02-06 00:00:00|17.790972222222223|   17.8|
|8888888|2019-02-06 00:00:00|2019-06-17 00:00:00|             131.0|    0.0|
|8888888|2019-01-19 05:01:00|2020-01-01 00:00:00| 346.7909722222222|    329|
|8888888|2019-06-17 00:00:00|2020-01-01 00:00:00|             198.0|    0.0|
+-------+-------------------+-------------------+------------------+-------+

I really don't understand what your code snippet does and therefore I'm unable to tweak it for my purpose. Thanks for your help!

Body:
For Spark2.4+, you can use sequence(generate date range) , collect_list, and use a combination of array functions and higher order functions to get your desired overlaps.

df.show() #sample dataframe
#+---+-------------------+-------------------+-----+
#| id|              begin|                end| days|
#+---+-------------------+-------------------+-----+
#|  2|2019-01-01 00:00:00|2019-12-25 00:00:00|358.0|
#|  2|2019-12-25 00:00:00|2020-01-01 00:00:00|  7.0|
#|  3|2019-01-01 00:00:00|2019-12-25 00:00:00|358.0|
#|  3|2019-12-20 00:00:00|2020-01-01 00:00:00| 12.0|
#|  4|2019-01-01 00:00:00|2019-12-25 00:00:00|358.0|
#|  4|2019-01-01 00:00:00|2019-11-25 00:00:00|328.0|
#|  4|2019-12-20 00:00:00|2020-01-01 00:00:00| 12.0|
#+---+-------------------+-------------------+-----+

from pyspark.sql import functions as F
from pyspark.sql.window import Window

w1=Window().partitionBy("id").orderBy("begin")
df.withColumn("seq", F.expr("""sequence(to_timestamp(begin), to_timestamp(end),interval 1 day)"""))\
  .withColumn("seq1", F.expr("""flatten(filter(collect_list(seq) over\
                                (partition by id),x-> arrays_overlap(x,seq)==True and seq!=x))"""))\
  .withColumn("overlap", F.when(F.row_number().over(w1)==1, F.lit(0))\
              .otherwise(F.size(F.array_intersect("seq","seq1"))-1)).orderBy("id","end").drop("seq","seq1").show()

#+---+-------------------+-------------------+-----+-------+
#| id|              begin|                end| days|overlap|
#+---+-------------------+-------------------+-----+-------+
#|  2|2019-01-01 00:00:00|2019-12-25 00:00:00|358.0|      0|
#|  2|2019-12-25 00:00:00|2020-01-01 00:00:00|  7.0|      0|
#|  3|2019-01-01 00:00:00|2019-12-25 00:00:00|358.0|      0|
#|  3|2019-12-20 00:00:00|2020-01-01 00:00:00| 12.0|      5|
#|  4|2019-01-01 00:00:00|2019-11-25 00:00:00|328.0|    328|
#|  4|2019-01-01 00:00:00|2019-12-25 00:00:00|358.0|      0|
#|  4|2019-12-20 00:00:00|2020-01-01 00:00:00| 12.0|      5|
#+---+-------------------+-------------------+-----+-------+

UPDATE:

This should cover all cases:

from pyspark.sql import functions as F
from pyspark.sql.window import Window

w1=Window().partitionBy("id").orderBy("begin")
w2=Window().partitionBy("id","begin","end").orderBy("begin")
w3=Window().partitionBy("id","begin","end")
w4=Window().partitionBy("id","begin","end","maxrownum").orderBy("begin")
df.withColumn("seq", F.expr("""sequence(to_timestamp(begin), to_timestamp(end),interval 1 day)"""))\
  .withColumn('maxrownum', F.max(F.row_number().over(w2)).over(w3))\
  .withColumn('rowNum', F.row_number().over(w4))\
  .withColumn("seq1", F.expr("""flatten(filter(collect_list(seq) over\
                                (partition by id order by begin),x-> arrays_overlap(x,seq)==True and seq!=x))"""))\
  .withColumn("overlap", F.when(F.row_number().over(w1)==1, F.lit(0))\
              .when(F.size(F.array_intersect("seq","seq1"))!=0,F.size(F.array_intersect("seq","seq1"))-1)
              .when((F.col("maxrownum")!=1)&(F.col("rowNum")<F.col("maxrownum")),F.col("days"))\
              .otherwise(F.lit(0)))\
         .orderBy("id","end").drop("seq","seq1","maxrownum","rowNum").show()

#+-------+-------------------+-------------------+-----------------+-----------------+
#|     id|              begin|                end|             days|          overlap|
#+-------+-------------------+-------------------+-----------------+-----------------+
#|7777777|2019-01-05 01:00:00|2019-04-04 00:00:00|88.95833333333333|              0.0|
#|7777777|2019-04-04 00:00:00|2019-07-11 00:00:00|             98.0|              0.0|
#|7777777|2019-07-11 00:00:00|2019-09-17 00:00:00|             68.0|              0.0|
#|7777777|2019-09-17 00:00:00|2019-09-19 22:01:00|2.917361111111111|              0.0|
#|7777777|2019-09-19 22:01:00|2020-01-01 00:00:00|103.0826388888889|103.0826388888889|
#|7777777|2019-09-19 22:01:00|2020-01-01 00:00:00|103.0826388888889|              0.0|
#+-------+-------------------+-------------------+-----------------+-----------------+

UPDATE2:

from pyspark.sql import functions as F
from pyspark.sql.window import Window

w1=Window().partitionBy("id").orderBy("begin")
w2=Window().partitionBy("id","begin","end").orderBy("begin")
w3=Window().partitionBy("id","begin","end")
w4=Window().partitionBy("id","begin","end","maxrownum").orderBy("begin")
df.withColumn("seq", F.expr("""sequence(to_timestamp(begin), to_timestamp(end),interval 1 day)"""))\
  .withColumn('maxrownum', F.max(F.row_number().over(w2)).over(w3))\
  .withColumn('rowNum', F.row_number().over(w4))\
  .withColumn("seq1", F.expr("""flatten(filter(collect_list(seq) over\
                                (partition by id),x-> arrays_overlap(x,seq)==True and seq!=x))"""))\
  .withColumn("overlap", F.when(F.row_number().over(w1)==1, F.lit(0))\
              .when(F.size(F.array_intersect("seq","seq1"))!=0,F.size(F.array_intersect("seq","seq1"))-1)
              .when((F.col("maxrownum")!=1)&(F.col("rowNum")<F.col("maxrownum")),F.col("days"))\
              .when(F.col("maxrownum")==1,F.col("days"))\
              .otherwise(F.lit(0)))\
              .replace(1,0)\
         .orderBy("id","end").drop("seq","seq1","rowNum","maxrownum").show()

#+-------+-------------------+-------------------+------------------+------------------+
#|     id|              begin|                end|              days|           overlap|
#+-------+-------------------+-------------------+------------------+------------------+
#|9347774|2019-01-01 00:00:00|2019-01-08 02:10:00| 7.090277777777778|               7.0|
#|9347774|2019-01-08 02:10:00|2019-02-04 05:28:00|           27.1375|           27.1375|
#|9347774|2019-02-04 05:28:00|2019-03-05 19:29:00|29.584027777777777|29.584027777777777|
#|9347774|2019-03-05 19:29:00|2019-04-22 18:45:00| 47.96944444444444| 47.96944444444444|
#|9347774|2019-04-22 18:45:00|2019-05-03 05:05:00|10.430555555555555|10.430555555555555|
#|9347774|2019-05-03 05:05:00|2019-05-17 16:25:00|14.472222222222221|14.472222222222221|
#|9347774|2019-05-17 16:25:00|2019-06-05 22:18:00| 19.24513888888889| 19.24513888888889|
#|9347774|2019-01-01 00:00:00|2020-01-01 00:00:00|             365.0|               0.0|
#|9347774|2019-06-05 22:18:00|2020-01-01 00:00:00|209.07083333333333|209.07083333333333|
#+-------+-------------------+-------------------+------------------+------------------+


问:
Reading two Ys on a line which have been split
I have split two the lines at the commas(taking them away)
All I am left with is the name of the Lennings,George,Y,Y but since the it has been split, I cant read the Ys as part of same line. I know this is very simple but I am incredibly confused(also its reading from a file).

```<code>empFile = "Employee.txt"
confFile = "confpacks.txt"

confFile = open("confpacks.txt", "r")
packsread = confFile.readlines()

readFile = open("Employee.txt", "r")
records = readFile.readlines()

    for line in records:
      line = line[:-1]
      fields = line.split(",")

      for field in fields:
        print(field, end=" , ")

        if field == "Y":
          print("Only attended one day therefore recieves " + packsread[0])
</code>```

expected output

if there is the the name and two Y Y so

```<code>John, Steward, Y , Y, attended both days.
</code>```

if there is the the name and one Y so

```<code>John, Steward,Y attended one day
</code>```

if there are no Ys at all

```<code>John, Steward, did not attend any days
</code>```

答:
You can count Ys in the end of the line and add the message based on the counter:

```<code>for line in file:
    line = line.strip()
    fields = line.split(",")

    count = 0
    for v in reversed(fields):
        if v == 'Y':
            count += 1
        else:
            break

    print(', '.join(fields), end=', ')

    if not count:
        print('did not attend any days')
    elif count == 1:
        print('attended one day')
    else:
        print('attended both days')
</code>```

Body:
I have split two the lines at the commas(taking them away)
All I am left with is the name of the Lennings,George,Y,Y but since the it has been split, I cant read the Ys as part of same line. I know this is very simple but I am incredibly confused(also its reading from a file).

empFile = "Employee.txt"
confFile = "confpacks.txt"

confFile = open("confpacks.txt", "r")
packsread = confFile.readlines()

readFile = open("Employee.txt", "r")
records = readFile.readlines()

    for line in records:
      line = line[:-1]
      fields = line.split(",")

      for field in fields:
        print(field, end=" , ")

        if field == "Y":
          print("Only attended one day therefore recieves " + packsread[0])

expected output

if there is the the name and two Y Y so

John, Steward, Y , Y, attended both days.

if there is the the name and one Y so

John, Steward,Y attended one day

if there are no Ys at all

John, Steward, did not attend any days

Body:
You can count Ys in the end of the line and add the message based on the counter:

for line in file:
    line = line.strip()
    fields = line.split(",")

    count = 0
    for v in reversed(fields):
        if v == 'Y':
            count += 1
        else:
            break

    print(', '.join(fields), end=', ')

    if not count:
        print('did not attend any days')
    elif count == 1:
        print('attended one day')
    else:
        print('attended both days')


问:
Android Studio Invoke Red Lightbulb
I am using Android Studio and I love using the little red light bulb for things like define a method. When I click on a red error it appears.

Mostly it works fine but sometimes it disappears too quickly to click on and sometimes it does not appear at all.

Is there a shortcut key to make it appear ? Or what is the name of the lightbulb so that I can Google for it.

答:
On Mac, you can do</p>

```<code>option + return
</code>```

Title: Android Studio Invoke Red Lightbulb

Body:
On Mac, you can do

option + return


问:
Multiple server builds with the same image - one will not run a specific crontab line
We have a watchdog script that runs on a number of servers that will restart some services if they fail.  We have them running successfully on a number of sites, however we have one site which will not run the crontab entry that triggers the watchdog.  If we run the entry from the command line - it works fine

When the watchdog is installed it puts the following line into crontab.  You just remove the '#' to enable it

```<code>#*/5 * * * * root /usr/local/fusion/scripts/watch_fusion_services 60
</code>```

Other entries in Crontab do run - it's just this one line

I have done the following to try and resolve the issue

Removed crontab entries for the watchdog and reinstalled watchdog

Checked syslog getting this error:

```<code> Error: bad hour; while reading /etc/crontab
</code>```

Changed the crontab line to be 6 minutes instead of 5 (as there was another cron job running every 5 minutes as well at this site)

Syslog error no longer occurring, however watchdog still does not work via crontab.  No error messages in syslog

tested running the crontab line from the command prompt - this works okay

Attempted same process on a test VM - worked okay

Attempted same process in live environment - tested okay

checked versions of the ICA - both the same - GNU/Linux 3.13.0-117-generic x86_64

ran ntqp -p on server that is having issues - time is 'LOCAL'.

typed entry by hand - same issues occurring

I could try rebooting the server - but it seems a bit extreme for one crontab entry not working

Does anyone have any ideas about this one?

答:
There was a line of junk in crontab that we all missed.  One of the other technicians I work with pointed it as he walked past our department.

Sometimes you just can't see the wood from the trees when you are looking at a problem too long

Body:
We have a watchdog script that runs on a number of servers that will restart some services if they fail.  We have them running successfully on a number of sites, however we have one site which will not run the crontab entry that triggers the watchdog.  If we run the entry from the command line - it works fine

When the watchdog is installed it puts the following line into crontab.  You just remove the '#' to enable it

#*/5 * * * * root /usr/local/fusion/scripts/watch_fusion_services 60

Other entries in Crontab do run - it's just this one line

I have done the following to try and resolve the issue

Removed crontab entries for the watchdog and reinstalled watchdog

Checked syslog getting this error:

 Error: bad hour; while reading /etc/crontab

Changed the crontab line to be 6 minutes instead of 5 (as there was another cron job running every 5 minutes as well at this site)

Syslog error no longer occurring, however watchdog still does not work via crontab.  No error messages in syslog

tested running the crontab line from the command prompt - this works okay

Attempted same process on a test VM - worked okay

Attempted same process in live environment - tested okay

checked versions of the ICA - both the same - GNU/Linux 3.13.0-117-generic x86_64

ran ntqp -p on server that is having issues - time is 'LOCAL'.

typed entry by hand - same issues occurring

I could try rebooting the server - but it seems a bit extreme for one crontab entry not working

Does anyone have any ideas about this one?