问:
Data Lake: fix corrupted files on Ingestion vs ETL

Objective

I'm building datalake, the general flow looks like Nifi -> Storage -> ETL -> Storage -> Data Warehouse.

The general rule for Data Lake sounds like no pre-processing on ingestion stage. All ongoing processing should happen at ETL, so you have provenance over raw & processed data.

Issue

Source system sends corrupted CSV files. Means besides header and data, the first too lines are always of free format metadata we'll never use. Only single table is corrupted, the corrupted CSV is used by single Spark job at the moment (lets call it X).

Question

Is it a good approach to remove those two lines at Nifi layer? See option 3 at "Workarounds".

Workarounds

Handle the corrupted records inside Spark job X. IMHO, this is bad approach, because we gonna use that file at different tools in future (data governance schema crawlers, maybe some Athena/ADLA-like engines over ADLS/S3). Means corrupted records handling logic should be implemented at multiple places.

Fix corrupted files on ETL layer and store them at "fixed" layer. All ongoing activities (ETL, data governance, MPP engines) will work only with "fixed" layer, instead of "raw" layer. This sounds for me as an overhead, to create a new layer for single CSV.

Fix (remove the first two strings from the CSV) at Nifi layer. Means "raw" storage layer will always contain readable data. IMHO, this is good because it's simple and the handling logic is implemented at one place.

答:
I like the philosophy offered in the accepted answer but I'd like to provide a more tactical answer...

Use the handle 'bad records' option on the spark read, e.g.:

```<code>spark.read
  .option("badRecordsPath", "/tmp/badRecordsPath")
  .format("csv")
  .load("/input/csvFile.csv")
</code>```

Reference "Handling bad records and files"

Reference "CSV files"

You can use this with a schema option .schema(customSchema) code to get a level of schema verification too (and better performance) on the read side of your jobs.

To perform schema checks on write, take a
look at Delta Lake open source project which has schema on write enforcement and ACID transactions for more reliability.

Managed Delta Lake will let you bin pack your small files with the OPTIMIZE command Databricks Delta Lake Optimize command

Because of ACID transactions and bin packing, Spark Structured Streaming and Delta Lake work really well together to continue the streaming data acquisition Nifi is performing.

Body:
I like the philosophy offered in the accepted answer but I'd like to provide a more tactical answer...

Use the handle 'bad records' option on the spark read, e.g.:

spark.read
  .option("badRecordsPath", "/tmp/badRecordsPath")
  .format("csv")
  .load("/input/csvFile.csv")

Reference "Handling bad records and files"

Reference "CSV files"

You can use this with a schema option .schema(customSchema) code to get a level of schema verification too (and better performance) on the read side of your jobs.

To perform schema checks on write, take a
look at Delta Lake open source project which has schema on write enforcement and ACID transactions for more reliability.

Managed Delta Lake will let you bin pack your small files with the OPTIMIZE command Databricks Delta Lake Optimize command

Because of ACID transactions and bin packing, Spark Structured Streaming and Delta Lake work really well together to continue the streaming data acquisition Nifi is performing.


来源: stackexchange

问:
Syntax for three-piece segmented regression using NLS in R when concave
My goal is to fit a three-piece (i.e., two break-point) regression model to make predictions using propagate's predictNLS function, making sure to define knots as parameters, but my model formula seems off.

I've used the segmented package to estimate the breakpoint locations (used as starting values in NLS), but would like to keep my models in the NLS format, specifically, nlsLM {minipack.lm} because I am fitting other types of curves to my data using NLS, want to allow NLS to optimize the knot values, am sometimes using variable weights, and need to be able to easily calculate the Monte Carlo confidence intervals from propagate. Though I'm very close to having the right syntax for the formula, I'm not getting the expected/required behaviour near the breakpoint(s). The segments SHOULD meet directly at the breakpoints (without any jumps), but at least on this data, I'm getting a weird local minimum at the breakpoint (see plots below).

Below is an example of my data and general process. I believe my issue to be in the NLS formula.

```<code>library(minpack.lm)
library(segmented)

y <- c(-3.99448113, -3.82447011, -3.65447803, -3.48447030, -3.31447855, -3.14448753, -2.97447972, -2.80448401, -2.63448380, -2.46448069, -2.29448796, -2.12448912, -1.95448783, -1.78448797, -1.61448563, -1.44448719, -1.27448469, -1.10448651, -0.93448525, -0.76448637, -0.59448626, -0.42448586, -0.25448588, -0.08448548,  0.08551417,  0.25551393,  0.42551411,  0.59551395,  0.76551389,  0.93551398)

x <- c(61586.1711, 60330.5550, 54219.9925, 50927.5381, 48402.8700, 45661.9175, 37375.6023, 33249.1248, 30808.6131, 28378.6508, 22533.3782, 13901.0882, 11716.5669, 11004.7305, 10340.3429,  9587.7994,  8736.3200,  8372.1482,  8074.3709,  7788.1847,  7499.6721,  7204.3168,  6870.8192,  6413.0828,  5523.8097,  3961.6114,  3460.0913,  2907.8614, 2016.1158,   452.8841)

df<- data.frame(x,y)

#Use Segmented to get estimates for parameters with 2 breakpoints
my.seg2 <- segmented(lm(y ~ x, data = df), seg.Z = ~ x, npsi = 2)

#extract knot, intercept, and coefficient values to use as NLS start points
my.knot1 <- my.seg2$psi[1,2]
my.knot2 <- my.seg2$psi[2,2]
my.m_2 <- slope(my.seg2)$x[1,1]
my.b1 <- my.seg2$coefficients[[1]]
my.b2 <- my.seg2$coefficients[[2]]
my.b3 <- my.seg2$coefficients[[3]]

#Fit a NLS model to ~replicate segmented model. Presumably my model formula is where the problem lies
my.model <- nlsLM(y~m*x+b+(b2*(ifelse(x>=knot1&x<=knot2,1,0)*(x-knot1))+(b3*ifelse(x>knot2,1,0)*(x-knot2-knot1))),data=df, start = c(m = my.m_2, b = my.b1, b2 = my.b2, b3 = my.b3, knot1 = my.knot1, knot2 = my.knot2))
</code>```

How it should look

```<code>plot(my.seg2)
</code>```

How it does look

```<code>plot(x, y)
lines(x=x, y=predict(my.model), col='black', lty = 1, lwd = 1)
</code>```

I was pretty sure I had it "right", but when the 95% confidence intervals are plotted with the line and prediction resolution (e.g., the density of x points) is increased, things seem dramatically incorrect.

Thank you all for your help.

答:
Define g to be a grouping vector having the same length as x which takes on values 1, 2, 3 for the 3 sections of the X axis and create an nls model from these.  The resulting plot looks ok.

```<code>my.knots <- c(my.knot1, my.knot2)
g <- cut(x, c(-Inf, my.knots, Inf), label = FALSE)
fm <- nls(y ~ a[g] + b[g] * x, df, start = list(a = c(1, 1, 1), b = c(1, 1, 1)))

plot(y ~ x, df)
lines(fitted(fm) ~ x, df, col = "red")
</code>```

(continued after graph)

Constraints

Although the above looks ok and may be sufficient it does not guarantee that the segments intersect at the knots. To do that we must impose the constraints that both sides are equal at the knots:

```<code>a[2] + b[2] * my.knots[1] = a[1] + b[1] * my.knots[1]
a[3] + b[3] * my.knots[2] = a[2] + b[2] * my.knots[2]
</code>```

so

```<code>a[2] = a[1] + (b[1] - b[2]) * my.knots[1]
a[3] = a[2] + (b[2] - b[3]) * my.knots[2]
     = a[1] + (b[1] - b[2]) * my.knots[1] + (b[2] - b[3]) * my.knots[2]
</code>```

giving:

```<code># returns a vector of the three a values
avals <- function(a1, b) unname(cumsum(c(a1, -diff(b) * my.knots)))

fm2 <- nls(y ~ avals(a1, b)[g] + b[g] * x, df, start = list(a1 = 1, b = c(1, 1, 1)))
</code>```

To get the three a values we can use:

```<code>co <- coef(fm2)
avals(co[1], co[-1])
</code>```

To get the residual sum of squares:

```<code>deviance(fm2)
## [1] 0.193077
</code>```

Polynomial

Although it involves a large number of parameters, a polynomial fit could be used in place of the segmented linear regression.  A 12th degree polynomial involves 13 parameters but has a lower residual sum of squares than the segmented linear regression.  A lower degree could be used with corresponding increase in residual sum of squares.  A 7th degree polynomial involves 8 parameters and visually looks not too bad although it has a higher residual sum of squares.

```<code>fm12 <- nls(y ~ cbind(1, poly(x, 12)) %*% b, df, start = list(b = rep(1, 13)))

deviance(fm12)
## [1] 0.1899218
</code>```

ContentLicense: CC BY-SA 4.0

Body:
My goal is to fit a three-piece (i.e., two break-point) regression model to make predictions using propagate's predictNLS function, making sure to define knots as parameters, but my model formula seems off.

I've used the segmented package to estimate the breakpoint locations (used as starting values in NLS), but would like to keep my models in the NLS format, specifically, nlsLM {minipack.lm} because I am fitting other types of curves to my data using NLS, want to allow NLS to optimize the knot values, am sometimes using variable weights, and need to be able to easily calculate the Monte Carlo confidence intervals from propagate. Though I'm very close to having the right syntax for the formula, I'm not getting the expected/required behaviour near the breakpoint(s). The segments SHOULD meet directly at the breakpoints (without any jumps), but at least on this data, I'm getting a weird local minimum at the breakpoint (see plots below).

Below is an example of my data and general process. I believe my issue to be in the NLS formula.

library(minpack.lm)
library(segmented)

y <- c(-3.99448113, -3.82447011, -3.65447803, -3.48447030, -3.31447855, -3.14448753, -2.97447972, -2.80448401, -2.63448380, -2.46448069, -2.29448796, -2.12448912, -1.95448783, -1.78448797, -1.61448563, -1.44448719, -1.27448469, -1.10448651, -0.93448525, -0.76448637, -0.59448626, -0.42448586, -0.25448588, -0.08448548,  0.08551417,  0.25551393,  0.42551411,  0.59551395,  0.76551389,  0.93551398)

x <- c(61586.1711, 60330.5550, 54219.9925, 50927.5381, 48402.8700, 45661.9175, 37375.6023, 33249.1248, 30808.6131, 28378.6508, 22533.3782, 13901.0882, 11716.5669, 11004.7305, 10340.3429,  9587.7994,  8736.3200,  8372.1482,  8074.3709,  7788.1847,  7499.6721,  7204.3168,  6870.8192,  6413.0828,  5523.8097,  3961.6114,  3460.0913,  2907.8614, 2016.1158,   452.8841)

df<- data.frame(x,y)

#Use Segmented to get estimates for parameters with 2 breakpoints
my.seg2 <- segmented(lm(y ~ x, data = df), seg.Z = ~ x, npsi = 2)

#extract knot, intercept, and coefficient values to use as NLS start points
my.knot1 <- my.seg2$psi[1,2]
my.knot2 <- my.seg2$psi[2,2]
my.m_2 <- slope(my.seg2)$x[1,1]
my.b1 <- my.seg2$coefficients[[1]]
my.b2 <- my.seg2$coefficients[[2]]
my.b3 <- my.seg2$coefficients[[3]]

#Fit a NLS model to ~replicate segmented model. Presumably my model formula is where the problem lies
my.model <- nlsLM(y~m*x+b+(b2*(ifelse(x>=knot1&x<=knot2,1,0)*(x-knot1))+(b3*ifelse(x>knot2,1,0)*(x-knot2-knot1))),data=df, start = c(m = my.m_2, b = my.b1, b2 = my.b2, b3 = my.b3, knot1 = my.knot1, knot2 = my.knot2))

How it should look

plot(my.seg2)

How it does look

plot(x, y)
lines(x=x, y=predict(my.model), col='black', lty = 1, lwd = 1)

I was pretty sure I had it "right", but when the 95% confidence intervals are plotted with the line and prediction resolution (e.g., the density of x points) is increased, things seem dramatically incorrect.

Thank you all for your help.

Body:
Define g to be a grouping vector having the same length as x which takes on values 1, 2, 3 for the 3 sections of the X axis and create an nls model from these.  The resulting plot looks ok.

my.knots <- c(my.knot1, my.knot2)
g <- cut(x, c(-Inf, my.knots, Inf), label = FALSE)
fm <- nls(y ~ a[g] + b[g] * x, df, start = list(a = c(1, 1, 1), b = c(1, 1, 1)))

plot(y ~ x, df)
lines(fitted(fm) ~ x, df, col = "red")

(continued after graph)

Constraints

Although the above looks ok and may be sufficient it does not guarantee that the segments intersect at the knots. To do that we must impose the constraints that both sides are equal at the knots:

a[2] + b[2] * my.knots[1] = a[1] + b[1] * my.knots[1]
a[3] + b[3] * my.knots[2] = a[2] + b[2] * my.knots[2]

so

a[2] = a[1] + (b[1] - b[2]) * my.knots[1]
a[3] = a[2] + (b[2] - b[3]) * my.knots[2]
     = a[1] + (b[1] - b[2]) * my.knots[1] + (b[2] - b[3]) * my.knots[2]

giving:

# returns a vector of the three a values
avals <- function(a1, b) unname(cumsum(c(a1, -diff(b) * my.knots)))

fm2 <- nls(y ~ avals(a1, b)[g] + b[g] * x, df, start = list(a1 = 1, b = c(1, 1, 1)))

To get the three a values we can use:

co <- coef(fm2)
avals(co[1], co[-1])

To get the residual sum of squares:

deviance(fm2)
## [1] 0.193077

Polynomial

Although it involves a large number of parameters, a polynomial fit could be used in place of the segmented linear regression.  A 12th degree polynomial involves 13 parameters but has a lower residual sum of squares than the segmented linear regression.  A lower degree could be used with corresponding increase in residual sum of squares.  A 7th degree polynomial involves 8 parameters and visually looks not too bad although it has a higher residual sum of squares.

fm12 <- nls(y ~ cbind(1, poly(x, 12)) %*% b, df, start = list(b = rep(1, 13)))

deviance(fm12)
## [1] 0.1899218


问:
swiper.js SVG's stacking on top of each other
I'm new to Swiper.js, I have a full screen swiper for a hero section that acts as a background slider. In the middle of this hero section I have another swiper that needs to display some SVG's with text. The hero swiper isn't the problem. The problem is that this SVG swiper is stacking the SVG's on top of each other, making a complete mess.

This is how the SVG swiper starts:

And this is how it looks when the final slide is showing up:

And then it cleans up, and starts the process of stacking the SVG's on top of each other again.

Here is my HTML:

```<code> <div class="swiper-container textslider">
            <div class="swiper-wrapper">
                <div class="swiper-slide text--1"></div>
                <div class="swiper-slide text--2"></div>
                <div class="swiper-slide text--3"></div>
                <div class="swiper-slide text--4"></div>
            </div>
        </div>

        <div class="player-bg">
            <a href="#" class="player"><span id="player__text">Reproducir <i class="fas fa-play"></i></span></a>
        </div>

        <div class="swiper-container background">
            <div class="swiper-wrapper">
                <div class="swiper-slide slide--1"></div>
                <div class="swiper-slide slide--2"></div>
                <div class="swiper-slide slide--3"></div>
                <div class="swiper-slide slide--4"></div>
            </div>
        </div>
    </div>

    <script>
        var swiper = new Swiper('.swiper-container', {
            spaceBetween: 30,
            effect: 'fade',
            speed: 2000,
            autoplay: {
                delay: 4000,
                disableOnInteraction: false,
            },
        });

        var textswiper = new Swiper('.textslider', {
            spaceBetween: 30,
            effect: 'fade',
            speed: 2000,
            autoplay: {
                delay: 4000,
                disableOnInteraction: false,
            },
        });
    </script>
</code>```

And this is my CSS:

```<code>.background {
    width: 100%;
    height: 100vh;
    position: absolute;
    top: 0;
    z-index: -99;
}

.textslider {
    width: 500px;
    height: 500px;
    position: relative;
    z-index: 7;
}

.slide--1 {
    background-image: url(../images/hero/slider-0.jpg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.slide--2 {
    background-image: url(../images/hero/slider-1.jpg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.slide--3 {
    background-image: url(../images/hero/slider-2.jpg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.slide--4 {
    background-image: url(../images/hero/slider-3.jpg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.text--1 {
    background-image: url(../images/bible-verses/verse2.svg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.text--2 {
    background-image: url(../images/bible-verses/verse1.svg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.text--3 {
    background-image: url(../images/bible-verses/verse3.svg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.text--4 {
    background-image: url(../images/bible-verses/verse4.svg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}
</code>```

What am I doing wrong?

答:
All the SVGs are positioned exactly the same (on top of each other), and since they have transparent backgrounds you're able to see "through" them.

Swiper controls visibility of the slides by adjusting the opacity CSS attribute. In your case, the problem is that the default Swiper effect sets opacity to 1 (i.e. visible) on multiple slides at the same time.

However, Swiper has an option that enables cross fades, which is the behavior you're looking for. Add the fadeEffect option on your "text" slider:

var textswiper = new Swiper('.textslider', {
  spaceBetween: 30,
  effect: 'fade',
  fadeEffect: { // Add this
    crossFade: true,
  }
  speed: 2000,
  autoplay: {
    delay: 4000,
    disableOnInteraction: false,
  },
});
```

Body:
I'm new to Swiper.js, I have a full screen swiper for a hero section that acts as a background slider. In the middle of this hero section I have another swiper that needs to display some SVG's with text. The hero swiper isn't the problem. The problem is that this SVG swiper is stacking the SVG's on top of each other, making a complete mess.

This is how the SVG swiper starts:

And this is how it looks when the final slide is showing up:

And then it cleans up, and starts the process of stacking the SVG's on top of each other again.

Here is my HTML:

            Reproducir

And this is my CSS:

.background {
    width: 100%;
    height: 100vh;
    position: absolute;
    top: 0;
    z-index: -99;
}

.textslider {
    width: 500px;
    height: 500px;
    position: relative;
    z-index: 7;
}

.slide--1 {
    background-image: url(../images/hero/slider-0.jpg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.slide--2 {
    background-image: url(../images/hero/slider-1.jpg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.slide--3 {
    background-image: url(../images/hero/slider-2.jpg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.slide--4 {
    background-image: url(../images/hero/slider-3.jpg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.text--1 {
    background-image: url(../images/bible-verses/verse2.svg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.text--2 {
    background-image: url(../images/bible-verses/verse1.svg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.text--3 {
    background-image: url(../images/bible-verses/verse3.svg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

.text--4 {
    background-image: url(../images/bible-verses/verse4.svg);
    background-position: center center;
    background-repeat: no-repeat;
    background-size: cover;
}

What am I doing wrong?


问:
How can one copy a matplotlib ax object
Instead of the following:

fig, ax = plt.subplots()
x = [1, 2]
y = [4, 5]
ax.scatter(x, y, s=100)
```

<p>Which looks as
<img src="https://i.stack.imgur.com/SEzID.png" alt="enter image description here"></p>

<p>I would like to have:</p>

<pre class="lang-py prettyprint-override"><code>
def plot(ax, x, y):
    ax_c = copy.deepcopy(ax)
    ax_c.scatter(x, y, s=100)
    return ax_c

fig, ax = plt.subplots()
x = [1, 2]
y = [4, 5]
ax = plot(ax, x, y)
</code>```

However, this doesn't work, with the following error:

```<code>NotImplementedError: TransformNode instances can not be copied. Consider using frozen() instead.
</code>```

So, I'm wondering how I go about writing a function which takes a matplotlib ax as input, creates a copy of it, alters the copy, and returns it. The reason that I'd like to do so is so that I can have functions which are self-contained.

edit

the following is not a suitable solution:

def plot(ax_c, x, y):
    ax_c.scatter(x, y, s=100)
    return ax_c

```

because here the return statement could be replaced with None and it would still function fine.

答:
You can simply use the following way in your function </p>

```<code>def plot(ax_c, x, y):
    ax_c.scatter(x, y, s=100)
    return ax_c
</code>```

Title: How can one copy a matplotlib ax object

Body:
Instead of the following:

fig, ax = plt.subplots()
x = [1, 2]
y = [4, 5]
ax.scatter(x, y, s=100)

Which looks as

I would like to have:

def plot(ax, x, y):
    ax_c = copy.deepcopy(ax)
    ax_c.scatter(x, y, s=100)
    return ax_c

fig, ax = plt.subplots()
x = [1, 2]
y = [4, 5]
ax = plot(ax, x, y)

However, this doesn't work, with the following error:

NotImplementedError: TransformNode instances can not be copied. Consider using frozen() instead.

So, I'm wondering how I go about writing a function which takes a matplotlib ax as input, creates a copy of it, alters the copy, and returns it. The reason that I'd like to do so is so that I can have functions which are self-contained.

edit

the following is not a suitable solution:

def plot(ax_c, x, y):
    ax_c.scatter(x, y, s=100)
    return ax_c

because here the return statement could be replaced with None and it would still function fine.

Body:
You can simply use the following way in your function

def plot(ax_c, x, y):
    ax_c.scatter(x, y, s=100)
    return ax_c


问:
Phusion Passenger: Error starting web application - Deploying Probot app (NodeJS) with Plesk nginx
I'm trying to deploy a GitHub Probot App (NodeJS application) to my webserver running Plesk 18.0.27 U1 with the NodeJS Extension 1.3.6-117. When running the probot app on my local machine, the app starts just fine and is accessible via localhost.

Plesk is apparently using the Phusion Passenger application server to serve NodeJS apps.

When accessing the deployed website, I get the following errror:

Screenshot: Passenger problem location

And in /var/log/nginx/error.log:

```<code>[ E 2020-05-30 10:06:31.7393 21506/Th age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/vhosts/example.org/node_root: A timeout occurred while spawning an application process.
  Error ID: 5f02dec5
  Error details saved to: /tmp/passenger-error-y6AeCv.html

[ E 2020-05-30 10:06:31.7466 21506/T5 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-2] Cannot checkout session because a spawning error occurred. The identifier of the error is 5f02dec5. Please see earlier logs for details about the error.
</code>```

Things I've tested so far

I increased the timeout for starting the app in nginx passenger_start_timeout 300;

I disabled php and proxy_mode in Plesk for this website (Nginx proxies requests to Apache by default, but passenger is running on nginx only)

I wrote a custom startup script to disable passenger autoinstall and set the webserver port in the node dotenv file to PORT="passenger"

```<code>#!/usr/bin/env node

const { Probot } = require('Probot')

// @ts-ignore
if (typeof(PhusionPassenger) !== 'undefined') {
    //@ts-ignore
    PhusionPassenger.configure({ autoInstall: false });
}

Probot.run(process.argv)

</code>```

Starting questions

Is my application failing or is passenger unable to bind the port of the app?

Are there more detailed logs or an option to enable verbose output?

Thanks in advance!

答:
After long research I figured it out myself. Posting my solution in case anyone is dealing with the same problem.

Setting the passenger log level. You can put passenger_log_level 7 in your nginx configuration /etc/nginx/conf.d/phusion-passenger.conf (for Plesk)

I started the app as a standalone passenger server and got the actual application stdout output. Example: cd in your node app root-folder and run passenger start --startup-file lib/startup.js --nodejs /opt/plesk/node/12/bin/node --log-level 3 --app-type node.

At this point I saw the module Probot couldn't be found in my above mentioned startup script. So I dug a little deeper how a probot app is actually started and stumbled upon the probot run command. By default a probot app isn't run with node ./lib/index.js but probot run ./lib/index.js.

Using a custom passenger app start command. I added the nginx option passenger_app_start_command "/opt/plesk/node/12/bin/npm start"; to set the custom start command to use the probot run command defined in the package.json start script instead of the default node ./lib/startup.js from passenger

I learned a lot about these tools and hope this will save someones weekend :D

Body:
I'm trying to deploy a GitHub Probot App (NodeJS application) to my webserver running Plesk 18.0.27 U1 with the NodeJS Extension 1.3.6-117. When running the probot app on my local machine, the app starts just fine and is accessible via localhost.

Plesk is apparently using the Phusion Passenger application server to serve NodeJS apps.

When accessing the deployed website, I get the following errror:

Screenshot: Passenger problem location

And in /var/log/nginx/error.log:

[ E 2020-05-30 10:06:31.7393 21506/Th age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/vhosts/example.org/node_root: A timeout occurred while spawning an application process.
  Error ID: 5f02dec5
  Error details saved to: /tmp/passenger-error-y6AeCv.html

[ E 2020-05-30 10:06:31.7466 21506/T5 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-2] Cannot checkout session because a spawning error occurred. The identifier of the error is 5f02dec5. Please see earlier logs for details about the error.

Things I've tested so far

I increased the timeout for starting the app in nginx passenger_start_timeout 300;

I disabled php and proxy_mode in Plesk for this website (Nginx proxies requests to Apache by default, but passenger is running on nginx only)

I wrote a custom startup script to disable passenger autoinstall and set the webserver port in the node dotenv file to PORT="passenger"

#!/usr/bin/env node

const { Probot } = require('Probot')

// @ts-ignore
if (typeof(PhusionPassenger) !== 'undefined') {
    //@ts-ignore
    PhusionPassenger.configure({ autoInstall: false });
}

Probot.run(process.argv)

Starting questions

Is my application failing or is passenger unable to bind the port of the app?

Are there more detailed logs or an option to enable verbose output?

Thanks in advance!


问:
How can I edit a matrix recursively given certain conditions?
```<code>grid = [[5 3 0 0 7 0 0 0 0];[6 0 0 1 9 5 0 0 0];[0 9 8 0 0 0 0 6 0];[8 0 0 0 6 0 0 0 3];[4 0 0 8 0 3 0 0 1];[7 0 0 0 2 0 0 0 6];[0 6 0 0 0 0 2 8 0];[0 0 0 4 1 9 0 0 5];[0 0 0 0 8 0 0 7 9]];

function ov = possibility(grid,x,y,n)
ov = true;
for i = 1:9
    if (grid(x,i) | grid(i,y)) == n
        ov = false;
        return
    end
end

xO = floor((x-1)/3)*3;
yO = floor((y-1)/3)*3;

for m = 1:3
    for k = 1:3
        if grid(xO + m, yO + k) == n
            ov = false;
            return
        end
    end
end
end

function solve(grid)
for x = 1:9
    for y = 1:9
        if grid(x,y) == 0
            for n = 1:9
                if possibility(grid,x,y,n)
                    grid(x,y) = n;
                    solve(grid)
                else
                    grid(x,y) = 0;
                end
            end
        end
    end
end
end
</code>```

Just a disclaimer, this code is not my original work, I'm only trying to figure out recursion because it's always been an odd concept to me. This is from the youtube video https://youtu.be/G_UYXzGuqvM where the program was written in python.

I don't quite understand how to systematically go back through the solve function and allow the code to decide which 'squares' it needs to change, because the way I see it: If grid(x,y) were to be 0, and then set to the first possible option, but that option was wrong, the code would never be able to find a solution for the puzzle. I don't get how it would determine to go back and change the incorrect choice to the next available choice which also may or may not be correct.

答:
I figured it out and this is my solution, thanks for all the feedback guys :)</p>

```<code>function Sudoku_Solver()
clear; close; clc

% manually input sudoku puzzle
gridGrid = [5 3 0 0 7 0 0 0 0;6 0 0 1 9 5 0 0 0;0 9 8 0 0 0 0 6 0;8 0 0 0 6 0 0 0 3;4 0 0 8 0 3 0 0 1;7 0 0 0 2 0 0 0 6;0 6 0 0 0 0 2 8 0;0 0 0 4 1 9 0 0 5;0 0 0 0 8 0 0 7 9];

%{
establishes possibility vector for each index, starts with all possible
solutions 1-9 and removes the non-working ones
%}
gridCell = cellfun(@double, mat2cell(repmat(1:9,9,9),ones(1,9),9*ones(1,9)),'UniformOutput',false);

% used to check common 3x3 grid
squareCheck = repelem(reshape(1:9,3,3)',3,3);

solve()

    function solve()
        % intialize a for loop for indices by row and column
        for x = 1:9
            for y = 1:9
                % Check that the current index is established or not
                if gridGrid(x,y) == 0
                    % if the index is not established check possibilities
                    for n = 1:9
                        possible(x,y,n)
                    end
                    % Check that the possibility is not a singular value
                    if size(gridCell{x,y},2) == 1
                        % if a singular value then establish the index
                        gridGrid(x,y) = gridCell{x,y};
                        % call function with newly established index
                        solve()
                    end
                end
            end
        end
    end

    % establish function to check the integrity of options 1-9 per index
    function possible(deltaX,deltaY,n)
        % initialize a for loop to check row/column
        for deltak = 1:9
            % Check entire row and column of index that we are checking
            if gridGrid(deltaX,deltak) == n || gridGrid(deltak,deltaY) == n
                %{
                if any value in the row/column is n that means it is not
                an option and we remove it from the list
                %}
                gridCell{deltaX,deltaY} = gridCell{deltaX,deltaY}(gridCell{deltaX,deltaY} ~= n);
            end

            %{
            intialize a for loop to check the 3x3 grid that the index
            belongs to
            %}
            for deltai = 1:9
                for deltaj = 1:9
                    % check indices that share a common 3x3 grid
                    if squareCheck(deltai,deltaj) == squareCheck(deltaX,deltaY)
                        if gridGrid(deltai,deltaj) == n
                            % remove any values that are not an option
                            gridCell{deltaX,deltaY} = gridCell{deltaX,deltaY}(gridCell{deltaX,deltaY} ~= n);
                        end
                    end
                end
            end
        end
    end

% print solution to command window
gridGrid

end
</code>```

Body:
grid = [[5 3 0 0 7 0 0 0 0];[6 0 0 1 9 5 0 0 0];[0 9 8 0 0 0 0 6 0];[8 0 0 0 6 0 0 0 3];[4 0 0 8 0 3 0 0 1];[7 0 0 0 2 0 0 0 6];[0 6 0 0 0 0 2 8 0];[0 0 0 4 1 9 0 0 5];[0 0 0 0 8 0 0 7 9]];

function ov = possibility(grid,x,y,n)
ov = true;
for i = 1:9
    if (grid(x,i) | grid(i,y)) == n
        ov = false;
        return
    end
end

xO = floor((x-1)/3)*3;
yO = floor((y-1)/3)*3;

for m = 1:3
    for k = 1:3
        if grid(xO + m, yO + k) == n
            ov = false;
            return
        end
    end
end
end

function solve(grid)
for x = 1:9
    for y = 1:9
        if grid(x,y) == 0
            for n = 1:9
                if possibility(grid,x,y,n)
                    grid(x,y) = n;
                    solve(grid)
                else
                    grid(x,y) = 0;
                end
            end
        end
    end
end
end

Just a disclaimer, this code is not my original work, I'm only trying to figure out recursion because it's always been an odd concept to me. This is from the youtube video https://youtu.be/G_UYXzGuqvM where the program was written in python.

I don't quite understand how to systematically go back through the solve function and allow the code to decide which 'squares' it needs to change, because the way I see it: If grid(x,y) were to be 0, and then set to the first possible option, but that option was wrong, the code would never be able to find a solution for the puzzle. I don't get how it would determine to go back and change the incorrect choice to the next available choice which also may or may not be correct.

Body:
I figured it out and this is my solution, thanks for all the feedback guys :)

function Sudoku_Solver()
clear; close; clc

% manually input sudoku puzzle
gridGrid = [5 3 0 0 7 0 0 0 0;6 0 0 1 9 5 0 0 0;0 9 8 0 0 0 0 6 0;8 0 0 0 6 0 0 0 3;4 0 0 8 0 3 0 0 1;7 0 0 0 2 0 0 0 6;0 6 0 0 0 0 2 8 0;0 0 0 4 1 9 0 0 5;0 0 0 0 8 0 0 7 9];

%{
establishes possibility vector for each index, starts with all possible
solutions 1-9 and removes the non-working ones
%}
gridCell = cellfun(@double, mat2cell(repmat(1:9,9,9),ones(1,9),9*ones(1,9)),'UniformOutput',false);

% used to check common 3x3 grid
squareCheck = repelem(reshape(1:9,3,3)',3,3);

solve()

    function solve()
        % intialize a for loop for indices by row and column
        for x = 1:9
            for y = 1:9
                % Check that the current index is established or not
                if gridGrid(x,y) == 0
                    % if the index is not established check possibilities
                    for n = 1:9
                        possible(x,y,n)
                    end
                    % Check that the possibility is not a singular value
                    if size(gridCell{x,y},2) == 1
                        % if a singular value then establish the index
                        gridGrid(x,y) = gridCell{x,y};
                        % call function with newly established index
                        solve()
                    end
                end
            end
        end
    end

    % establish function to check the integrity of options 1-9 per index
    function possible(deltaX,deltaY,n)
        % initialize a for loop to check row/column
        for deltak = 1:9
            % Check entire row and column of index that we are checking
            if gridGrid(deltaX,deltak) == n || gridGrid(deltak,deltaY) == n
                %{
                if any value in the row/column is n that means it is not
                an option and we remove it from the list
                %}
                gridCell{deltaX,deltaY} = gridCell{deltaX,deltaY}(gridCell{deltaX,deltaY} ~= n);
            end

            %{
            intialize a for loop to check the 3x3 grid that the index
            belongs to
            %}
            for deltai = 1:9
                for deltaj = 1:9
                    % check indices that share a common 3x3 grid
                    if squareCheck(deltai,deltaj) == squareCheck(deltaX,deltaY)
                        if gridGrid(deltai,deltaj) == n
                            % remove any values that are not an option
                            gridCell{deltaX,deltaY} = gridCell{deltaX,deltaY}(gridCell{deltaX,deltaY} ~= n);
                        end
                    end
                end
            end
        end
    end

% print solution to command window
gridGrid

end


问:
How to make the console print something by pressing a key like F2 or F3
UPDATE! I figured out how to find the keys and defined them in my code. The problem is I don't know how to make it so the console prints the selection and it gives u time to press a key. Then it gets plugged into to the switch which prints the current gun depending on the key pressed. I am also using windows 10. I also have encountered another problem I want it to print out Guns first then ask for an input (this part works fine) but then instead of just printing the attachments, it prints out both attachments and the humanized level.</p>

```<code>#include <iostream>
#include <istream>
#include <fstream>
#include <string.h>
#include <conio.h>
#include <string>

using namespace std;

#define KEY_F1 17
#define KEY_F2 18
#define KEY_F3 19
#define KEY_F4 20
#define KEY_F5 21
#define KEY_F6 22
#define KEY_F7 23
#define KEY_F8 24
#define KEY_F9 25
#define KEY_F10 26
#define KEY_F11 27
#define KEY_F12 28

#define KEY_N7 103
#define KEY_N8 104
#define KEY_N9 105

int main() {

    char gun;
    char att;
    char hum;

    char key = _getch();
    int value = key;

    cout << "" << endl;

    cout << "Developed by ! CaptaiN#9999" << endl;

    cout << "" << endl;

    cout << "Guns" << endl;

    cout << "" << endl;

    cout << "[F1]   AK" << endl;
    cout << "[F2]   MP5" << endl;
    cout << "[F3]   M2" << endl;
    cout << "[F4]   SAR" << endl;
    cout << "[F5]   Tommy" << endl;
    cout << "[F6]   Custom" << endl;
    cin >> gun;

    cout << "" << endl;

    cout << "Attachnments" << endl;

    cout << "" << endl;

    cout << "[F7]   Holosight" << endl;
    cout << "[F8]   Simplesight" << endl;
    cout << "[F9]   Silencer" << endl;
    cout << "[F10]  8x Zoom Scope" << endl;
    cout << "[F11]  16x Zoom Scope" << endl;
    cout << "[F12]  No Attachment" << endl;
    cin >> att;

    cout << "" << endl;

    cout << "Humanize Levels" << endl;

    cout << "" << endl;

    cout << "[Numpad 7]  3 (HIGH)" << endl;
    cout << "[Numpad 8]  2 (MODERATE)" << endl;
    cout << "[Numpad 9]  1 (LOW)" << endl;
    cin >> hum;

    //system("CLS");

    while (value != KEY_X) {

        switch (_getch()) {

        case KEY_F1:
            cout << "Current Gun: AK" << endl;
            break;
        case KEY_F2:
            cout << "Current Gun: MP5" << endl;
            break;
        case KEY_F3:
            cout << "Current Gun: M2" << endl;
            break;
        case KEY_F4:
            cout << "Current Gun: SAR" << endl;
            break;
        case KEY_F5:
            cout << "Current Gun: Tommy" << endl;
            break;
        case KEY_F6:
            cout << "Current Gun: Custom" << endl;
            break;
        }

        switch (att) {

        case KEY_F7:
            cout << "Current Attachment: Holosight" << endl;
            break;
        case KEY_F8:
            cout << "Current Attachment: Simplesight" << endl;
            break;
        case KEY_F9:
            cout << "Current Attachment: Silencer" << endl;
            break;
        case KEY_F10:
            cout << "Current Attachment: 8x Zoom Scope" << endl;
            break;
        case KEY_F11:
            cout << "Current Attachment: 16x Zoom Scope" << endl;
            break;
        case KEY_F12:
            cout << "Current Attachment: None" << endl;
            break;
        }

        switch (hum) {

        case KEY_N7:
            cout << "Humanized Level: 3" << endl;
            break;
        case KEY_N8:
            cout << "Humanized Level: 2" << endl;
            break;
        case KEY_N9:
            cout << "Humanized Level: 1" << endl;
            break;
        }
    }
}
</code>```

答:
I think pm100's answer is a perfectly acceptable way to do this, but the resulting code from such a technique would be fragile and dependent on the platform (because there's no standard representation as he mentions).

If that's fine with you, use his answer. A more portable way to do this would be to use a terminal control library, the most common one being curses
(it's technically a c library FYI). There are two main implementations: pdcurses mainly for Windows and ncurses for Mac and Linux. I won't go over set up here since that will mostly depend on the library, but if you want to make it cross-platform you can use cmake but that isn't super easy (because you have to handle pdcurses and ncurses separately).

Regardless of curses implementation, a minimal example of capturing the F1-F12 keys is below:

#ifdef _WIN32
#include <pdcurses.h>
#else
#include <ncurses.h>
#endif

int main()
{
    initscr();
    cbreak();
    noecho();
    keypad(stdscr, TRUE);

    while (true) {
        int ch = getch();
        if (ch >= KEY_F(1) && ch <= KEY_F(12)) {
            printw("F* key pressed\n");
        }
    }

    endwin();
    return 0;
}
```

for additional details on the library I recommend this programming guide.

as a side note, you might want to consider something like std::map to map your keys to std::string as it would be a little less verbose than using switch statements.

Body:
UPDATE! I figured out how to find the keys and defined them in my code. The problem is I don't know how to make it so the console prints the selection and it gives u time to press a key. Then it gets plugged into to the switch which prints the current gun depending on the key pressed. I am also using windows 10. I also have encountered another problem I want it to print out Guns first then ask for an input (this part works fine) but then instead of just printing the attachments, it prints out both attachments and the humanized level.

#include <iostream>
#include <istream>
#include <fstream>
#include <string.h>
#include <conio.h>
#include <string>

using namespace std;

#define KEY_F1 17
#define KEY_F2 18
#define KEY_F3 19
#define KEY_F4 20
#define KEY_F5 21
#define KEY_F6 22
#define KEY_F7 23
#define KEY_F8 24
#define KEY_F9 25
#define KEY_F10 26
#define KEY_F11 27
#define KEY_F12 28

#define KEY_N7 103
#define KEY_N8 104
#define KEY_N9 105

int main() {

    char gun;
    char att;
    char hum;

    char key = _getch();
    int value = key;

    cout << "" << endl;

    cout << "Developed by ! CaptaiN#9999" << endl;

    cout << "" << endl;

    cout << "Guns" << endl;

    cout << "" << endl;

    cout << "[F1]   AK" << endl;
    cout << "[F2]   MP5" << endl;
    cout << "[F3]   M2" << endl;
    cout << "[F4]   SAR" << endl;
    cout << "[F5]   Tommy" << endl;
    cout << "[F6]   Custom" << endl;
    cin >> gun;

    cout << "" << endl;

    cout << "Attachnments" << endl;

    cout << "" << endl;

    cout << "[F7]   Holosight" << endl;
    cout << "[F8]   Simplesight" << endl;
    cout << "[F9]   Silencer" << endl;
    cout << "[F10]  8x Zoom Scope" << endl;
    cout << "[F11]  16x Zoom Scope" << endl;
    cout << "[F12]  No Attachment" << endl;
    cin >> att;

    cout << "" << endl;

    cout << "Humanize Levels" << endl;

    cout << "" << endl;

    cout << "[Numpad 7]  3 (HIGH)" << endl;
    cout << "[Numpad 8]  2 (MODERATE)" << endl;
    cout << "[Numpad 9]  1 (LOW)" << endl;
    cin >> hum;

    //system("CLS");

    while (value != KEY_X) {

        switch (_getch()) {

        case KEY_F1:
            cout << "Current Gun: AK" << endl;
            break;
        case KEY_F2:
            cout << "Current Gun: MP5" << endl;
            break;
        case KEY_F3:
            cout << "Current Gun: M2" << endl;
            break;
        case KEY_F4:
            cout << "Current Gun: SAR" << endl;
            break;
        case KEY_F5:
            cout << "Current Gun: Tommy" << endl;
            break;
        case KEY_F6:
            cout << "Current Gun: Custom" << endl;
            break;
        }

        switch (att) {

        case KEY_F7:
            cout << "Current Attachment: Holosight" << endl;
            break;
        case KEY_F8:
            cout << "Current Attachment: Simplesight" << endl;
            break;
        case KEY_F9:
            cout << "Current Attachment: Silencer" << endl;
            break;
        case KEY_F10:
            cout << "Current Attachment: 8x Zoom Scope" << endl;
            break;
        case KEY_F11:
            cout << "Current Attachment: 16x Zoom Scope" << endl;
            break;
        case KEY_F12:
            cout << "Current Attachment: None" << endl;
            break;
        }

        switch (hum) {

        case KEY_N7:
            cout << "Humanized Level: 3" << endl;
            break;
        case KEY_N8:
            cout << "Humanized Level: 2" << endl;
            break;
        case KEY_N9:
            cout << "Humanized Level: 1" << endl;
            break;
        }
    }
}

Body:
I think pm100's answer is a perfectly acceptable way to do this, but the resulting code from such a technique would be fragile and dependent on the platform (because there's no standard representation as he mentions).

If that's fine with you, use his answer. A more portable way to do this would be to use a terminal control library, the most common one being curses
(it's technically a c library FYI). There are two main implementations: pdcurses mainly for Windows and ncurses for Mac and Linux. I won't go over set up here since that will mostly depend on the library, but if you want to make it cross-platform you can use cmake but that isn't super easy (because you have to handle pdcurses and ncurses separately).

Regardless of curses implementation, a minimal example of capturing the F1-F12 keys is below:

#ifdef _WIN32
#include <pdcurses.h>
#else
#include <ncurses.h>
#endif

int main()
{
    initscr();
    cbreak();
    noecho();
    keypad(stdscr, TRUE);

    while (true) {
        int ch = getch();
        if (ch >= KEY_F(1) && ch <= KEY_F(12)) {
            printw("F* key pressed\n");
        }
    }

    endwin();
    return 0;
}

for additional details on the library I recommend this programming guide.

as a side note, you might want to consider something like std::map to map your keys to std::string as it would be a little less verbose than using switch statements.


问:
Splunk: Unable to get the correct min and max values
I'm a newbie as far as Splunk is concerned with modest regex skills.

We have events with the following patterns:

```<code>fallbackAPIStatus={api1=133:..., api2=472:...,api3=498:...}
fallbackAPIStatus={api1=3535:...}
fallbackAPIStatus={api2=252:...,api3=655:...}
</code>```

The numeric value indicates the response times and the ellipsis inidcates fields that I'm not interested in.

The number of apis within the braces is dynamic (between 1 and 4)

I want to be able to create a table as follows:

```<code>apiName   TotalRequests  Max-Response-Time  Min-Response-Time
api1           2            3535           133
api2           2             472           252
api2           2             655           498
</code>```

Here's my search:

```<code>index=my_logs  sourcetype=my_sourcetype | rex field=_raw "fallbackAPIStatus=\{(?P<fallBackApis>[^\}]+)\}" | eval temp=split(fallBackApis,",") | rex field=temp "(?P<apiName>[a-zA-Z-]+)=(?P<responseTime>[0-9]+):"|stats count as TotalRequests  max(responseTime) as Max-Response-Time min(responseTime) as Min-Response-Time by apiName
</code>```

I'm able to get the TotalRequests right but I'm not able to get the correct max and min response times

Can someone advise what I'm doing wrong here?

答:
I think there is an issue with your field extraction, the following works fine </p>

```<code>| eval temp=split(fallBackApis,",") | rex field=temp "(?<apiName>\S+)=(?<responseTime>\d+):"
</code>```

Body:
I'm a newbie as far as Splunk is concerned with modest regex skills.

We have events with the following patterns:

fallbackAPIStatus={api1=133:..., api2=472:...,api3=498:...}
fallbackAPIStatus={api1=3535:...}
fallbackAPIStatus={api2=252:...,api3=655:...}

The numeric value indicates the response times and the ellipsis inidcates fields that I'm not interested in.

The number of apis within the braces is dynamic (between 1 and 4)

I want to be able to create a table as follows:

apiName   TotalRequests  Max-Response-Time  Min-Response-Time
api1           2            3535           133
api2           2             472           252
api2           2             655           498

Here's my search:

index=my_logs  sourcetype=my_sourcetype | rex field=_raw "fallbackAPIStatus=\{(?P<fallBackApis>[^\}]+)\}" | eval temp=split(fallBackApis,",") | rex field=temp "(?P<apiName>[a-zA-Z-]+)=(?P<responseTime>[0-9]+):"|stats count as TotalRequests  max(responseTime) as Max-Response-Time min(responseTime) as Min-Response-Time by apiName

I'm able to get the TotalRequests right but I'm not able to get the correct max and min response times

Can someone advise what I'm doing wrong here?

Body:
I think there is an issue with your field extraction, the following works fine

| eval temp=split(fallBackApis,",") | rex field=temp "(?<apiName>\S+)=(?<responseTime>\d+):"


问:
Bash doesn't print string with question mark in for loop
I have a text file with the following contents,

```<code>My test
strings
that dont have
a question
mark except this line?
but not
these two
</code>```

and when i try to read the file in bash using, for example,

```<code>ph_lines="/path/to/file.txt"
for l in $(cat "$ph_lines")
do
    echo "$l"
done
</code>```

everything prints on the output except for the string with the question mark in it.

I have tried using while read line; echo line; done < $filename and it still has the same problem

The only thing that would work to capture all of the lines is when i used sed to remove question marks.

```<code>for l in $(cat ${ph_lines} | sed $'s/\?//' )
</code>```

Thank you!

答:
To correctly read from a file, use the example from Read a file line by line assigning the value to a variable:

```<code>while IFS= read -r line; do
    echo "$line"
done < my_filename.txt
</code>```

The difference is not that it's a while loop (because you've tried that), but because this loop is correctly quoted. The problem you're seeing happens because you secretly enabled nullglob first, and then neglected to quote:

```<code>$ shopt -s nullglob
$ var='question?'

$ echo "$var"
question?

$ echo $var
(blank line)
</code>```

Unquoted expansion causes pathname expansion, and since you enabled nullglob and have no matching files, the previous example shows nothing. If you had some matching files, you'd see those instead:

```<code>$ touch questions question2

$ echo $var
question2 questions
</code>```

You can set up shellcheck in your editor to get automatic warnings about these issues.

Body:
I have a text file with the following contents,

My test
strings
that dont have
a question
mark except this line?
but not
these two

and when i try to read the file in bash using, for example,

ph_lines="/path/to/file.txt"
for l in $(cat "$ph_lines")
do
    echo "$l"
done

everything prints on the output except for the string with the question mark in it.

I have tried using while read line; echo line; done < $filename and it still has the same problem

The only thing that would work to capture all of the lines is when i used sed to remove question marks.

for l in $(cat ${ph_lines} | sed $'s/\?//' )

Thank you!

Body:
To correctly read from a file, use the example from Read a file line by line assigning the value to a variable:

while IFS= read -r line; do
    echo "$line"
done < my_filename.txt

The difference is not that it's a while loop (because you've tried that), but because this loop is correctly quoted. The problem you're seeing happens because you secretly enabled nullglob first, and then neglected to quote:

$ shopt -s nullglob
$ var='question?'

$ echo "$var"
question?

$ echo $var
(blank line)

Unquoted expansion causes pathname expansion, and since you enabled nullglob and have no matching files, the previous example shows nothing. If you had some matching files, you'd see those instead:

$ touch questions question2

$ echo $var
question2 questions

You can set up shellcheck in your editor to get automatic warnings about these issues.


问:
Getting Json formatted objects
Ok, so I have a big JSON file that has arrays and objects setup for controls for a google map project I am working on...looks like this:

```<code>{
"settings":
{

    "DEFAULT_MAP_SETTING":  "DEFAULT_MAP_SETTINGS",

    "DEFAULT_MAP_RES":      "county",
    "DEFAULT_MAP_CAT":      "popden"
},

"map_settings":
{
    "DEFAULT_MAP_SETTINGS":
    {
        "map_options":
        {
            "center":               [39.828175, -94.5795],
            "mapTypeId":            "TERRAIN",
            "streetViewControl":    false,
            "scrollwheel":          false,
            "overviewMapControl":   false,
            "mapTypeControl":       false,
            "zoom":                 4
        },

        "map_bounds":
        {
            "upper-left":           [98.70, -127.50],
            "lower-right":          [48.85, -55.90]
        },

    }

}
}
</code>```

My question is how do I go about getting this data in json format as it has to be in json format to load up options and what not in google maps. For instance I have

```<code>var myMapOptions = {
            "zoom": 4,
            "minZoom" : 4,
            "scrollWheel" : false,
            "center": initialLoc,
            "mapTypeId": google.maps.MapTypeId.ROADMAP,
    };

var theMap = new google.maps.Map(document.getElementById("map_canvas"), myMapOptions);
</code>```

Which loads up the google map. But I cant figure out how to set myMapOptions with my big Json file. Every method that I know of returns the json data in strings and gets rid of the formatting. Like $.ajax in jQuery will go get the json for me but it strips it of its formatting.

Can anyone help me out?

答:
jQuery $.ajax() will automatically transform you JSON "file" into a javascript object if your request come back with a JSON header.

However you can override this by using dataType of $.ajax:

```<code>$.ajax({
  url: "test.html",
  dataType: "text",
  success: function(data){
    alert(data);
  }
});
</code>```

The previous script will alert you JSON file like you want.

[Update]
You can try to use eval():

```<code>$.ajax({
  url: "test.html",
  dataType: "text",
  success: function(data){
    var options = eval(data);
  }
});
</code>```

In this case your data need to be executable by javascript.

[Update 2]
JsonP style, your json file need to looks like that:

```<code>var options = {
   "zoom": 4, "minZoom" : 4,
   "scrollWheel" : false,
   "center": new google.maps.LatLng(39.828175, -94.5795),
   "mapTypeId": google.maps.MapTypeId.ROADMAP
};
</code>```

Notice the "var options ="

Your ajax request need to look like that:

```<code>$.ajax({
  url: "test.html",
  dataType: "text",
  success: function(data){
    alert(data);// will show your data as text
    eval(data); this will execute your "JSON file"

    //after eval you can now use options as a variable
    alert(options);
  }
});
</code>```

ContentLicense: CC BY-SA 3.0

Title: Getting Json formatted objects

Body:
Ok, so I have a big JSON file that has arrays and objects setup for controls for a google map project I am working on...looks like this:

{
"settings":
{

    "DEFAULT_MAP_SETTING":  "DEFAULT_MAP_SETTINGS",

    "DEFAULT_MAP_RES":      "county",
    "DEFAULT_MAP_CAT":      "popden"
},

"map_settings":
{
    "DEFAULT_MAP_SETTINGS":
    {
        "map_options":
        {
            "center":               [39.828175, -94.5795],
            "mapTypeId":            "TERRAIN",
            "streetViewControl":    false,
            "scrollwheel":          false,
            "overviewMapControl":   false,
            "mapTypeControl":       false,
            "zoom":                 4
        },

        "map_bounds":
        {
            "upper-left":           [98.70, -127.50],
            "lower-right":          [48.85, -55.90]
        },

    }

}
}

My question is how do I go about getting this data in json format as it has to be in json format to load up options and what not in google maps. For instance I have

var myMapOptions = {
            "zoom": 4,
            "minZoom" : 4,
            "scrollWheel" : false,
            "center": initialLoc,
            "mapTypeId": google.maps.MapTypeId.ROADMAP,
    };

var theMap = new google.maps.Map(document.getElementById("map_canvas"), myMapOptions);

Which loads up the google map. But I cant figure out how to set myMapOptions with my big Json file. Every method that I know of returns the json data in strings and gets rid of the formatting. Like $.ajax in jQuery will go get the json for me but it strips it of its formatting.

Can anyone help me out?

Body:
jQuery $.ajax() will automatically transform you JSON "file" into a javascript object if your request come back with a JSON header.

However you can override this by using dataType of $.ajax:

$.ajax({
  url: "test.html",
  dataType: "text",
  success: function(data){
    alert(data);
  }
});

The previous script will alert you JSON file like you want.

[Update]
You can try to use eval():

$.ajax({
  url: "test.html",
  dataType: "text",
  success: function(data){
    var options = eval(data);
  }
});

In this case your data need to be executable by javascript.

[Update 2]
JsonP style, your json file need to looks like that:

var options = {
   "zoom": 4, "minZoom" : 4,
   "scrollWheel" : false,
   "center": new google.maps.LatLng(39.828175, -94.5795),
   "mapTypeId": google.maps.MapTypeId.ROADMAP
};

Notice the "var options ="

Your ajax request need to look like that:

$.ajax({
  url: "test.html",
  dataType: "text",
  success: function(data){
    alert(data);// will show your data as text
    eval(data); this will execute your "JSON file"

    //after eval you can now use options as a variable
    alert(options);
  }
});


问:
How do I add a reaction emoji to the message the bot sends after doing the user's command?
I get this error: discord.ext.commands.errors.CommandInvokeError: Command raised an exception: AttributeError: 'Bot' object has no attribute 'message'--when trying to do await self.client.message.add_reaction(emoji).

I tried changing it to await ctx.message.add_reaction(emoji), and I realized that it reacted to the command the user sent rather than the bot's new message.

```<code>import discord
from discord.ext import commands

class MovieNight(commands.Cog):
    """Polls for Movie Night."""

    def __init__(self, client):
        self.client = client

    @commands.command(aliases=['m'])
    async def movie(self, ctx, year, *movie):
        movie_title = ' '.join(movie[:-1])
        await ctx.send(f"`{year}` - `{movie_title}` | **{movie[-1]}**")
        emoji = ''
        await self.client.message.add_reaction(emoji)

def setup(client):
    client.add_cog(MovieNight(client))
</code>```

答:
self.client doesn't know about the message, the is stored as part of the invocation context:

```<code>await ctx.message.add_reaction(emoji)
</code>```

Body:
I get this error: discord.ext.commands.errors.CommandInvokeError: Command raised an exception: AttributeError: 'Bot' object has no attribute 'message'--when trying to do await self.client.message.add_reaction(emoji).

I tried changing it to await ctx.message.add_reaction(emoji), and I realized that it reacted to the command the user sent rather than the bot's new message.

import discord
from discord.ext import commands

class MovieNight(commands.Cog):
    """Polls for Movie Night."""

    def __init__(self, client):
        self.client = client

    @commands.command(aliases=['m'])
    async def movie(self, ctx, year, *movie):
        movie_title = ' '.join(movie[:-1])
        await ctx.send(f"`{year}` - `{movie_title}` | **{movie[-1]}**")
        emoji = ''
        await self.client.message.add_reaction(emoji)

def setup(client):
    client.add_cog(MovieNight(client))

Body:
self.client doesn't know about the message, the is stored as part of the invocation context:

await ctx.message.add_reaction(emoji)


问:
How to safely send data from a service to running activity
Lets say I have a String test = "value" available in a service and want to send it to my running mainactivity.
How does one send that data safely?
Im looking for the most simple example possible, where the data can not be seen by other apps.
When I look at intents it says:

A broadcast is a message that any app can receive. The system delivers
  various broadcasts for system events, such as when the system boots up
  or the device starts charging. You can deliver a broadcast to >>>>other
  apps<<< by passing an Intent to sendBroadcast() or
  sendOrderedBroadcast().

However I want to send it to my app only as it contains private data.
In my service I tried:

```<code>Intent intent = new Intent(this, MainActivity.class);
intent.setFlags(Intent.FLAG_ACTIVITY_SINGLE_TOP);
intent.putExtra(getString(R.string.notification_intent), "6");
startActivity(intent);
</code>```

Where MainActivity.class is the class that should receive the intent, but in this way I get:

```<code>Calling startActivity() from outside of an Activity  context requires the FLAG_ACTIVITY_NEW_TASK flag. Is this really what you want?
</code>```

How can I safely send data from a service to an activity?

答:
Q: How do I send data from an Android service to my activity?

A: You have several alternatives:

1. Use intents:

How to get data from service to activity

Send msg from service:

```<code>private static void sendMessageToActivity(Location l, String msg) {
    Intent intent = new Intent("GPSLocationUpdates");
    // You can also include some extra data.
    intent.putExtra("Status", msg);
    Bundle b = new Bundle();
    b.putParcelable("Location", l);
    intent.putExtra("Location", b);
    LocalBroadcastManager.getInstance(context).sendBroadcast(intent);
</code>```

Register to receive message in activity:

```<code>LocalBroadcastManager.getInstance(getActivity()).registerReceiver(
            mMessageReceiver, new IntentFilter("GPSLocationUpdates"));
</code>```

Custom message receiver in activity:

```<code>private BroadcastReceiver mMessageReceiver = new BroadcastReceiver() {
    @Override
    public void onReceive(Context context, Intent intent) {
        // Get extra data included in the Intent
        String message = intent.getStringExtra("Status");
        Bundle b = intent.getBundleExtra("Location");
        lastKnownLoc = (Location) b.getParcelable("Location");
        ...
</code>```

I would NOT characterize this as "unsafe" - it can be a perfectly reasonable approach.

2. Have the activity bind to the service

https://developer.android.com/guide/components/bound-services

Service:

```<code>public class LocalService extends Service {
    // Binder given to clients
    private final IBinder binder = new LocalBinder();
    ...
public class LocalBinder extends Binder {
        LocalService getService() {
            // Return this instance of LocalService so clients can call public methods
            return LocalService.this;
        }
    }

    @Override
    public IBinder onBind(Intent intent) {
        return binder;
    }
</code>```

Activity:

```<code>@Override
    protected void onStart() {
        super.onStart();
        // Bind to LocalService
        Intent intent = new Intent(this, LocalService.class);
        bindService(intent, connection, Context.BIND_AUTO_CREATE);
        ...
     @Override
        protected void onStop() {
            super.onStop();
            unbindService(connection);
            mBound = false;
        ...
    /** Defines callbacks for service binding, passed to bindService() */
        private ServiceConnection connection = new ServiceConnection() {

        @Override
        public void onServiceConnected(ComponentName className,
                IBinder service) {
            // We've bound to LocalService, cast the IBinder and get LocalService instance
            LocalBinder binder = (LocalBinder) service;
            mService = binder.getService();
            mBound = true;
        }
    ...
    // To use the service, your client would call mService.someMethod()...
</code>```

I'm not sure exactly what you're looking for, but example 1 is probably your best bet.

Here's a tutorial that might help give you more details/more ideas:

Basics Of Services In Android:

Part 1: Basics

Part 2: Binding services to an activity

Part 3: Using Messenger

https://developer.android.com/guide/components/broadcasts

Android provides three ways for apps to send broadcast:

The sendOrderedBroadcast(Intent, String) method sends broadcasts to one receiver at a time.

sendBroadcast(Intent) method sends broadcasts to all receivers in an undefined order. This is called a Normal Broadcast. This is more
  efficient, but means that receivers cannot read results from other
  receivers, propagate data received from the broadcast, or abort the
  broadcast.

LocalBroadcastManager.sendBroadcast method sends broadcasts to receivers that are in the same app as the sender. If you don't need to
  send broadcasts across apps, use local broadcasts. The implementation
  is much more efficient (no interprocess communication needed) and you
  don't need to worry about any security issues related to other apps
  being able to receive or send your broadcasts.

Additionally:

https://developer.android.com/guide/components/broadcasts#restrict-broadcasts-permissions

Restricting broadcasts with permissions

Permissions allow you to restrict broadcasts to the set of apps that hold certain permissions. You can enforce restrictions on either
  the sender or receiver of a broadcast.

Sending with permissions

When you call sendBroadcast(Intent, String) or sendOrderedBroadcast(Intent, String, BroadcastReceiver, Handler, int,
  String, Bundle), you can specify a permission parameter. Only
  receivers who have requested that permission with the tag in their
  manifest (and subsequently been granted the permission if it is
  dangerous) can receive the broadcast. For example, the following code
  sends a broadcast:

Personally, I don't see any "security" issue at all with simply using an intent.

But if you want or need to, you can use the above techniques to further lock down communications.

'Hope that helps!

Body:
Lets say I have a String test = "value" available in a service and want to send it to my running mainactivity.
How does one send that data safely?
Im looking for the most simple example possible, where the data can not be seen by other apps.
When I look at intents it says:

A broadcast is a message that any app can receive. The system delivers
  various broadcasts for system events, such as when the system boots up
  or the device starts charging. You can deliver a broadcast to >>>>other
  apps<<< by passing an Intent to sendBroadcast() or
  sendOrderedBroadcast().

However I want to send it to my app only as it contains private data.
In my service I tried:

Intent intent = new Intent(this, MainActivity.class);
intent.setFlags(Intent.FLAG_ACTIVITY_SINGLE_TOP);
intent.putExtra(getString(R.string.notification_intent), "6");
startActivity(intent);

Where MainActivity.class is the class that should receive the intent, but in this way I get:

Calling startActivity() from outside of an Activity  context requires the FLAG_ACTIVITY_NEW_TASK flag. Is this really what you want?

How can I safely send data from a service to an activity?

Body:
Q: How do I send data from an Android service to my activity?

A: You have several alternatives:

1. Use intents:

How to get data from service to activity

Send msg from service:

private static void sendMessageToActivity(Location l, String msg) {
    Intent intent = new Intent("GPSLocationUpdates");
    // You can also include some extra data.
    intent.putExtra("Status", msg);
    Bundle b = new Bundle();
    b.putParcelable("Location", l);
    intent.putExtra("Location", b);
    LocalBroadcastManager.getInstance(context).sendBroadcast(intent);

Register to receive message in activity:

LocalBroadcastManager.getInstance(getActivity()).registerReceiver(
            mMessageReceiver, new IntentFilter("GPSLocationUpdates"));

Custom message receiver in activity:

private BroadcastReceiver mMessageReceiver = new BroadcastReceiver() {
    @Override
    public void onReceive(Context context, Intent intent) {
        // Get extra data included in the Intent
        String message = intent.getStringExtra("Status");
        Bundle b = intent.getBundleExtra("Location");
        lastKnownLoc = (Location) b.getParcelable("Location");
        ...

I would NOT characterize this as "unsafe" - it can be a perfectly reasonable approach.

2. Have the activity bind to the service

https://developer.android.com/guide/components/bound-services

Service:

public class LocalService extends Service {
    // Binder given to clients
    private final IBinder binder = new LocalBinder();
    ...
public class LocalBinder extends Binder {
        LocalService getService() {
            // Return this instance of LocalService so clients can call public methods
            return LocalService.this;
        }
    }

    @Override
    public IBinder onBind(Intent intent) {
        return binder;
    }

Activity:

@Override
    protected void onStart() {
        super.onStart();
        // Bind to LocalService
        Intent intent = new Intent(this, LocalService.class);
        bindService(intent, connection, Context.BIND_AUTO_CREATE);
        ...
     @Override
        protected void onStop() {
            super.onStop();
            unbindService(connection);
            mBound = false;
        ...
    /** Defines callbacks for service binding, passed to bindService() */
        private ServiceConnection connection = new ServiceConnection() {

        @Override
        public void onServiceConnected(ComponentName className,
                IBinder service) {
            // We've bound to LocalService, cast the IBinder and get LocalService instance
            LocalBinder binder = (LocalBinder) service;
            mService = binder.getService();
            mBound = true;
        }
    ...
    // To use the service, your client would call mService.someMethod()...

I'm not sure exactly what you're looking for, but example 1 is probably your best bet.

Here's a tutorial that might help give you more details/more ideas:

Basics Of Services In Android:

Part 1: Basics

Part 2: Binding services to an activity

Part 3: Using Messenger

https://developer.android.com/guide/components/broadcasts

Android provides three ways for apps to send broadcast:

The sendOrderedBroadcast(Intent, String) method sends broadcasts to one receiver at a time.

sendBroadcast(Intent) method sends broadcasts to all receivers in an undefined order. This is called a Normal Broadcast. This is more
  efficient, but means that receivers cannot read results from other
  receivers, propagate data received from the broadcast, or abort the
  broadcast.

LocalBroadcastManager.sendBroadcast method sends broadcasts to receivers that are in the same app as the sender. If you don't need to
  send broadcasts across apps, use local broadcasts. The implementation
  is much more efficient (no interprocess communication needed) and you
  don't need to worry about any security issues related to other apps
  being able to receive or send your broadcasts.

Additionally:

https://developer.android.com/guide/components/broadcasts#restrict-broadcasts-permissions

Restricting broadcasts with permissions

Permissions allow you to restrict broadcasts to the set of apps that hold certain permissions. You can enforce restrictions on either
  the sender or receiver of a broadcast.

Sending with permissions

When you call sendBroadcast(Intent, String) or sendOrderedBroadcast(Intent, String, BroadcastReceiver, Handler, int,
  String, Bundle), you can specify a permission parameter. Only
  receivers who have requested that permission with the tag in their
  manifest (and subsequently been granted the permission if it is
  dangerous) can receive the broadcast. For example, the following code
  sends a broadcast:

Personally, I don't see any "security" issue at all with simply using an intent.

But if you want or need to, you can use the above techniques to further lock down communications.

'Hope that helps!


问:
Run parallel inside steps of a stage in declarative jenkins
So, I want to run my parallel stages inside a stage but I also want to write some shared code by each parallel stage which I have written in steps of parallel parent stage
The problem I faced is that that the parallel stages are not being run</p>

```<code>stages {
   stage('partent stage 1'){
      something here
   }
   stage('parent stage 2') {
      steps {
         // common code for parallel stages

         parallel {
            stage ('1'){
               // some shell command
            }
            stage('2') {
               // some shell command
            }
         }

      }
   }
}
</code>```

答:
For executing shared code you can define variables and functions outside of the declarative pipeline:

```<code>def foo = true

def checkFoo {
  return foo
}

pipeline {
  stage('parallel stage') {
    parallel {
      stage('stage 1') {
        steps {
          script {
            def baz = checkFoo()
          }
          sh “echo ${baz}”
        }
      }
      stage('stage 2') {
        steps {
          script {
            def baz = checkFoo()
          }
          sh “echo ${baz}”
        }
      }
    }
  }
}
</code>```

You can also write a shared library, which you can use in all or certain jobs.

I’ve deleted my first answer, since it was pure BS.

Body:
So, I want to run my parallel stages inside a stage but I also want to write some shared code by each parallel stage which I have written in steps of parallel parent stage
The problem I faced is that that the parallel stages are not being run

stages {
   stage('partent stage 1'){
      something here
   }
   stage('parent stage 2') {
      steps {
         // common code for parallel stages

         parallel {
            stage ('1'){
               // some shell command
            }
            stage('2') {
               // some shell command
            }
         }

      }
   }
}

Body:
For executing shared code you can define variables and functions outside of the declarative pipeline:

def foo = true

def checkFoo {
  return foo
}

pipeline {
  stage('parallel stage') {
    parallel {
      stage('stage 1') {
        steps {
          script {
            def baz = checkFoo()
          }
          sh “echo ${baz}”
        }
      }
      stage('stage 2') {
        steps {
          script {
            def baz = checkFoo()
          }
          sh “echo ${baz}”
        }
      }
    }
  }
}

You can also write a shared library, which you can use in all or certain jobs.

I’ve deleted my first answer, since it was pure BS.


问:
SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 81
I am currently new to robot framework.I am currently using latest window version of chrome and chromedriver which is 80 but when i try to run the test it gives the message "SessionNotCreatedException: Message: session not created: This version of ChromeDriver only supports Chrome version 81" in pycharm but currently beta version of 81 is only available. I have tried uninstalling everthing and reinstalling it again but nothing works can anyone help me with this.Thank you!

Screenshots below:

答:
The solution is:-

Check out the chrome version of your chrome.From chrome://settings/help

Check out which version of ChromeDriver is compatible with your current
chrome version from here

download the compatible one and replace the existing ChromeDriver with
a new ChromeDriver.

Now run the code


问:
read_pdf FileNotFoundError: [Errno 2] No such file or directory: in Python
I am trying to scrape tables from pdf with read_pdf in python. I am using read_pdf but it doesn't do the job. Also, to mention, I do this in MAC with Jupiter notebook.
This is what I do:

```<code>from tabula import read_pdf
file = read_pdf(r'C:\Users\myname\Rprojects\Reports_scraping\data_scraped\icnarc_29052020\icnarc_200529.pdf')
</code>```

I get this error:

```<code>FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\myname\\Rprojects\\Reports_scraping\\data_scraped\\icnarc_29052020\\icnarc_200529.pdf'
</code>```

How I can solve this issue?

答:
just to check that the file exist, do you get True when running this:

```<code>import os

file_path = r'C:\Users\myname\Rprojects\Reports_scraping\data_scraped\icnarc_29052020\icnarc_200529.pdf'
print( os.path.isfile(file_path))
</code>```

Edit file_path with wherever is the file(using Python 3).
And did you change "myname" in the path with your actual username... (just in case)

It is preferable to build your paths using os.path.join to make things compatible, on windows it will need to create a root "config.py" file, see

how to get the root folder on windows

#

having discussed with GaB, it seemed that he is using Jupyter notebook on Mac, which explains issues, I saw this link, but can't help more.

Jupyter - import pdf

os.path.join doc

Body:
I am trying to scrape tables from pdf with read_pdf in python. I am using read_pdf but it doesn't do the job. Also, to mention, I do this in MAC with Jupiter notebook.
This is what I do:

from tabula import read_pdf
file = read_pdf(r'C:\Users\myname\Rprojects\Reports_scraping\data_scraped\icnarc_29052020\icnarc_200529.pdf')

I get this error:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\myname\\Rprojects\\Reports_scraping\\data_scraped\\icnarc_29052020\\icnarc_200529.pdf'

How I can solve this issue?

Body:
just to check that the file exist, do you get True when running this:

import os

file_path = r'C:\Users\myname\Rprojects\Reports_scraping\data_scraped\icnarc_29052020\icnarc_200529.pdf'
print( os.path.isfile(file_path))

Edit file_path with wherever is the file(using Python 3).
And did you change "myname" in the path with your actual username... (just in case)

It is preferable to build your paths using os.path.join to make things compatible, on windows it will need to create a root "config.py" file, see

how to get the root folder on windows

#

having discussed with GaB, it seemed that he is using Jupyter notebook on Mac, which explains issues, I saw this link, but can't help more.

Jupyter - import pdf

os.path.join doc


问:
Script to send email based on value in a column - problem with script
New to GAS.  I have a Google Form feeding into a spreadsheet. After watching tutorial videos and reading other posts, I attempted to create a script that will send an email to an address in column 2, and send a different email based on either a yes or no in another column (column 25).  It also includes another column (26) that I want to have the date populated into when the email is sent, ensuring that every time I run this script, there are no duplicates sent.  I have debugged and nothing comes up as an error, but it's not working - I have not received an email.  Help would be greatly appreciated!  Here is an example spreadsheet: https://docs.google.com/spreadsheets/d/1AKaSOk1ZbnfKeadgB_mugnqplKzJJ3bDQ_KsRY0sMvQ/edit?usp=sharing</p>

```<code>function sendEmail() {
  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var responses = ss.getSheetByName("Field Trip Requests");

  var data = responses.getRange(2,1,responses.getLastRow()-1,25).getValues();

  data.forEach(function(row,i) {
    // variables
    var recipient = row[1];
    var destination = row[6];
    var approval = row[24];
    var emailSent = row[25];

    if(emailSent == ' ') {

      if(approval == "Y") {
        var body = "Your field trip request for " +
          destination +
            " has been approved! If you requested transportation, Najma will make arrangements and contact you if she requires more information." +
              "<br><br>" +
                "Cheers," +
                  "<br><br>" +
                    "Boaty McBoatface";

      }
      else if(approval == "N") {
        var body = "Your field trip request for " +
          destination +
            " has not been approved.  Please come and see me and we can chat!" +
              "<br><br>" +
                "Cheers," +
                  "<br><br>" +
                    "Boaty McBoatface";

      }

      var subject = "Your Field Trip Request";

      MailApp.sendEmail(recipient, subject, body)

      var d = new Date();

      responses.getRange(i + 1, 25).setValue(d);

    }

  })

}

</code>```

答:
One problem of the script is that you it's using 1 based indexes when it should be using 0 based in

```<code>var recipient = row[1];
var destination = row[6];
var approval = row[24];
var emailSent = row[25];
</code>```

The above is because JavaScript uses 0 based indexes for Array elements (as well as for other things), so intestead of the above use

```<code>var recipient = row[0];
var destination = row[5];
var approval = row[23];
var emailSent = row[24];
</code>```

Another problem is the following condition:

```<code>emailSent == ' ' /* is emailSent equal to a blank space? */
</code>```

it should be

```<code>emailSent == '' /* is emailSent equal to an empty string? */
</code>```

the above because getValues() returns an empty string for empty cells

Body:
New to GAS.  I have a Google Form feeding into a spreadsheet. After watching tutorial videos and reading other posts, I attempted to create a script that will send an email to an address in column 2, and send a different email based on either a yes or no in another column (column 25).  It also includes another column (26) that I want to have the date populated into when the email is sent, ensuring that every time I run this script, there are no duplicates sent.  I have debugged and nothing comes up as an error, but it's not working - I have not received an email.  Help would be greatly appreciated!  Here is an example spreadsheet: https://docs.google.com/spreadsheets/d/1AKaSOk1ZbnfKeadgB_mugnqplKzJJ3bDQ_KsRY0sMvQ/edit?usp=sharing

function sendEmail() {
  var ss = SpreadsheetApp.getActiveSpreadsheet();
  var responses = ss.getSheetByName("Field Trip Requests");

  var data = responses.getRange(2,1,responses.getLastRow()-1,25).getValues();

  data.forEach(function(row,i) {
    // variables
    var recipient = row[1];
    var destination = row[6];
    var approval = row[24];
    var emailSent = row[25];

    if(emailSent == ' ') {

      if(approval == "Y") {
        var body = "Your field trip request for " +
          destination +
            " has been approved! If you requested transportation, Najma will make arrangements and contact you if she requires more information." +
              "

" +
                "Cheers," +
                  "

" +
                    "Boaty McBoatface";

      }
      else if(approval == "N") {
        var body = "Your field trip request for " +
          destination +
            " has not been approved.  Please come and see me and we can chat!" +
              "

" +
                "Cheers," +
                  "

" +
                    "Boaty McBoatface";

      }

      var subject = "Your Field Trip Request";

      MailApp.sendEmail(recipient, subject, body)

      var d = new Date();

      responses.getRange(i + 1, 25).setValue(d);

    }

  })

}

Body:
One problem of the script is that you it's using 1 based indexes when it should be using 0 based in

var recipient = row[1];
var destination = row[6];
var approval = row[24];
var emailSent = row[25];

The above is because JavaScript uses 0 based indexes for Array elements (as well as for other things), so intestead of the above use

var recipient = row[0];
var destination = row[5];
var approval = row[23];
var emailSent = row[24];

Another problem is the following condition:

emailSent == ' ' /* is emailSent equal to a blank space? */

it should be

emailSent == '' /* is emailSent equal to an empty string? */

the above because getValues() returns an empty string for empty cells


问:
How to filter Spark sql by nested array field (array within array)?
My spark Dataframe schema

```<code>|-- goods: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- brand_id: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- product_id: string (nullable = true)
</code>```

I'm able to filter the Dataframe by product_id

```<code>select * from goodsInfo where array_contains(goods.product_id, 'f31ee3f8-9ba2-49cb-86e2-ceb44e34efd9')
</code>```

But I'm unable to filter by brand_id which is an array within array..

I get the error when I try

```<code>select * from goodsInfo where array_contains(goods.brand_id, '45c060b9-3645-49ad-86eb-65f3cd4e9081')
</code>```

Error:

```<code>function array_contains should have been array followed by a value with same element type, but it's [array<array<string>>, string].; line 1 pos 45;
</code>```

Can someone please help ?

Thanks in advance

答:
I have created a sample DataFrame and exploded the array column and then filter the records. PFB sample code for the same.

```<code>import org.apache.spark.sql.functions._

val jsonData = """{
 "id": "0001",
 "batters":
  {
   "batter":
    [
      1001,
      1002,
      1003
    ]
  }
}
"""

// Create DataFrame
val df = spark.read.json(Seq(jsonData).toDS).toDF

// Explode the array column and then filter.
val newdf = df.select($"id", explode($"batters.batter").as("batter")).filter("batter = 1001")
newdf.show()
</code>```

// Output

```<code>+----+------+
|  id|batter|
+----+------+
|0001|  1001|
+----+------+
</code>```

You can also refer to my blog for "Working with JSON in Apache Spark"

I hope this helps.

Body:
My spark Dataframe schema

|-- goods: array (nullable = true)
 |    |-- element: struct (containsNull = true)
 |    |    |-- brand_id: array (nullable = true)
 |    |    |    |-- element: string (containsNull = true)
 |    |    |-- product_id: string (nullable = true)

I'm able to filter the Dataframe by product_id

select * from goodsInfo where array_contains(goods.product_id, 'f31ee3f8-9ba2-49cb-86e2-ceb44e34efd9')

But I'm unable to filter by brand_id which is an array within array..

I get the error when I try

select * from goodsInfo where array_contains(goods.brand_id, '45c060b9-3645-49ad-86eb-65f3cd4e9081')

Error:

function array_contains should have been array followed by a value with same element type, but it's [array<array<string>>, string].; line 1 pos 45;

Can someone please help ?

Thanks in advance

Body:
I have created a sample DataFrame and exploded the array column and then filter the records. PFB sample code for the same.

import org.apache.spark.sql.functions._

val jsonData = """{
 "id": "0001",
 "batters":
  {
   "batter":
    [
      1001,
      1002,
      1003
    ]
  }
}
"""

// Create DataFrame
val df = spark.read.json(Seq(jsonData).toDS).toDF

// Explode the array column and then filter.
val newdf = df.select($"id", explode($"batters.batter").as("batter")).filter("batter = 1001")
newdf.show()

// Output

+----+------+
|  id|batter|
+----+------+
|0001|  1001|
+----+------+

You can also refer to my blog for "Working with JSON in Apache Spark"

I hope this helps.


问:
What modules does mod_rewrite depend on?
proxy? proxy_http? what else?

When I include all of the available modules, rewrite works, but I'm having a hard time figuring out which ones are actually making the difference.

ANSWER: vhost_alias was one. seems to work now.

答:
To have mod_rewrite working,
first, you have to turn it on.
in httpd.conf
uncomment this: LoadModule rewrite_module libexec/mod_rewrite.so
and restart Apache.

If you use .htaccess to manage the mod_rewrite, then you need to activate that as well.
and set AllowOverride.

reference: http://www.usenet-forums.com/apache-web-server/10742-activate-mod_rewrite.html

and http://www.apachefriends.org/f/viewtopic.php?f=16&t=33314
and http://httpd.apache.org/docs/2.0/en/mod/core.html#allowoverride

OwnerDisplayName: ThomasGHenry

OwnerDisplayName: Murvinlai

ContentLicense: CC BY-SA 2.5