问:
Laravel command exec output
I am trying to make artisan command which would execute a non-ending script (more precisely it is a gulp watch command). Is it possible to output something to console in real time and not waiting for script to end?

```<code>exec($basedir . "test.sh", $output);

foreach ($output as $item) {
    $this->info($item);
}
</code>```

test.sh looks like this (shortened for brevity):

```<code>echo "something"
echo "something else"

gulp watch
</code>```

Since command doesn't actually end, I can't retrieve two echos running inside it. Is there a way to do it?

答:
I have resolved it by doing the following:</p>

```<code>while (@ ob_end_flush()) ; // end all output buffers if any

// this section streams output to terminal
$proc = popen($basedir . "test.sh", 'r');
while (!feof($proc)) {
    $this->info(fread($proc, 4096));
    @ flush();
}
</code>```

Title: Laravel command exec output

Body:
I am trying to make artisan command which would execute a non-ending script (more precisely it is a gulp watch command). Is it possible to output something to console in real time and not waiting for script to end?

exec($basedir . "test.sh", $output);

foreach ($output as $item) {
    $this->info($item);
}

test.sh looks like this (shortened for brevity):

echo "something"
echo "something else"

gulp watch

Since command doesn't actually end, I can't retrieve two echos running inside it. Is there a way to do it?

Body:
I have resolved it by doing the following:

while (@ ob_end_flush()) ; // end all output buffers if any

// this section streams output to terminal
$proc = popen($basedir . "test.sh", 'r');
while (!feof($proc)) {
    $this->info(fread($proc, 4096));
    @ flush();
}


来源: stackexchange

问:
How to get debian packages from older releases
This might be one basic question.
but I wanted to download packages of specific debian version from debian mirror.

e.g I wanted to download debian 9.11 packages but when I checked on debian mirror(http://ftp.debian.com/debian/dists/)

I could see only debian 9.12 entry and no entry of 9.11
screenshot of debian mirror

I know debian must archive these packages and mirror somewhere else but I am currently not able to find it

Once a developer told that he used jigdo images to get the archived mirror. But I am not able to extract debian mirror url from jigdo images

Thanks

答:
I after searching over the internet, I got the solution.

I found mirror url from jidgo image uploaded by debian.

I downloaded jidgo image of debian 9.11 from here
https://cdimage.debian.org/cdimage/archive/9.11.0/amd64/iso-cd/

and used zcat command on .jidgo file and found mirror at the end of the file
the mirror was something like this
http://snapshot.debian.org/archive/debian/20190909T181138Z/

ContentLicense: CC BY-SA 4.0


问:
DB design : Config Data, Actual Data, Log Data
I want to know if there is any typical approach to differenciate this kind of data

I have to listing devices (for example) in a db, ane everyone will have

Configuration data

Actual data

Log data

I commonly mix Config/Actual Data in the same table and another table for Log data,

This seems to be an usual issue, so I wonder if there is any standar way to do it.

EDIT::

Here an Example

A semaphore at the street :

Config data = position (street intersection), type (for passengers,
cars), etc..

Actual data = colorstate=red, green, walk, stop.. functionality=ok, being repaired, etc..

Log data = date/time + colorstate + (any other of the Actual data needed to be logged)

Thanks

答:
it looks like you would benefit from properly normalizing this information into several tables.

it does not look like you should consider jamming it all into one table.

ContentLicense: CC BY-SA 3.0


问:
Library "TableOne" multiple comparisons. Calculate line by line p-values
I received a comment from a reviewer who wanted to have all the p-values for each line of specific variables levels in a demographic characteristic table (Table 1). Even though the request appears quite strange (and inexact) to me, I would like to agree with his suggestion.

```<code>    library(tableone)
## Load data
library(survival); data(pbc)

# drop ID from variable list
vars <- names(pbc)[-1]

## Create Table 1 stratified by trt (can add more stratifying variables)
tableOne <- CreateTableOne(vars = vars, strata = c("trt"), data = pbc, factorVars = c("status","edema","stage"))

print(tableOne, nonnormal = c("bili","chol","copper","alk.phos","trig"), exact = c("status","stage"),  smd = TRUE)
</code>```

the output:

I need to have the p-values for each level of the variables status, edema and stage, with Bonferroni correction. I went through the documentation without success.
In addition, is it correct to use chi-squared to compare sample sizes across rows?

UPDATE:

I'm not sure if my approach is correct, however I would like to share it with you.  I generated for the variable status a dummy variable for each strata, than I calculated the chisq .

```<code>    library(tableone)
    ## Load data
    library(survival); data(pbc)

    d <- pbc[,c("status", "trt")]

    # Convert dummy variables
    d$status.0 <- ifelse(d$status==0, 1,0)
    d$status.1 <- ifelse(d$status==1, 1,0)
    d$status.2 <- ifelse(d$status==2, 1,0)

t <- rbind(
    chisq.test(d$status.0, d$trt),
    # p-value = 0.7202

    chisq.test(d$status.1, d$trt),
    # p-value = 1

    chisq.test(d$status.2, d$trt)
    #p-value = 0.7818
)
t
</code>```

BONFERRONI ADJ FOR MULTIPLE COMPARISONS:

```<code>p <- t[,"p.value"]

p.adjust(p, method = "bonferroni")
</code>```

答:
This question was posted some time ago, so I supose you already answered the reviewer.

I don't really understand why computing adjusted p values for just three varibles. In fact, adjusting p values depends on the number of comparisons made. If you use p.adjust() with a vector of 3 p values, results will not really be "adjusted" by the amount of comparison made (you really did more than a dozen and a half!)

I show how to extract all p-values so you can compute the adjusted ones.
To extract pValues from package tableOne there is a way calling object attributes (explained first), and two quick and dirty ways (at the bottom part).

To extract them, first I copy your code to create your tableOne:

```<code>library(tableone)
## Load data
library(survival); data(pbc)

# drop ID from variable list
vars <- names(pbc)[-1]

## Create Table 1 stratified by trt (can add more stratifying variables)
tableOne <- CreateTableOne(vars = vars, strata = c("trt"), data = pbc, factorVars = c("status","edema","stage"))
</code>```

You can see what your "tableOne" object has via attributes()

```<code>attributes(tableOne)
</code>```

You can see a tableOne usually has a table for continuous and categorical variables. You can use attributes() in them too

```<code>attributes(tableOne$CatTable)
# you can notice $pValues
</code>```

Now you know "where" the pValues are, you can extract them with attr()

```<code>attr(tableOne$CatTable, "pValues")
</code>```

Something similar with numerical variables:

```<code>attributes(tableOne$ContTable)
# $pValues are there
attr(tableOne$ContTable, "pValues")
</code>```

You have pValues for Normal and NonNormal variables.
As you set them before, you can extract both

```<code>mypCont <- attr(tableOne$ContTable, "pValues") # put them in an object
nonnormal = c("bili","chol","copper","alk.phos","trig") # copied from your code

mypCont[rownames(mypCont) %in% c(nonnormal), "pNonNormal"] # extract NonNormal

"%!in%" <- Negate("%in%")
mypCont[rownames(mypCont) %!in% c(nonnormal), "pNonNormal"] # extract Normal
</code>```

All that said, and your pValues extracted, I think there are two much more convenient quick and dirty ways to accomplish the same:

Quick and dirty way A: using dput() with your printed tableOne. Then search in the console where the pValues are and copy-paste them to the script, to store them in an object

Quick and dirty way B: If you look in tableOne vignette there is an "Exporting" section, you can use print(tableOne, quote = TRUE) and then just copy and paste to a spreadsheet (like LibreOffice, Excel...).
Then I would select the column with pValue, transpose it, and get it back to R, to compute adjusted p values with p.adjust() and copy them back to the spreadsheet for journal submission

Body:
I received a comment from a reviewer who wanted to have all the p-values for each line of specific variables levels in a demographic characteristic table (Table 1). Even though the request appears quite strange (and inexact) to me, I would like to agree with his suggestion.

    library(tableone)
## Load data
library(survival); data(pbc)

# drop ID from variable list
vars <- names(pbc)[-1]

## Create Table 1 stratified by trt (can add more stratifying variables)
tableOne <- CreateTableOne(vars = vars, strata = c("trt"), data = pbc, factorVars = c("status","edema","stage"))

print(tableOne, nonnormal = c("bili","chol","copper","alk.phos","trig"), exact = c("status","stage"),  smd = TRUE)

the output:

I need to have the p-values for each level of the variables status, edema and stage, with Bonferroni correction. I went through the documentation without success.
In addition, is it correct to use chi-squared to compare sample sizes across rows?

UPDATE:

I'm not sure if my approach is correct, however I would like to share it with you.  I generated for the variable status a dummy variable for each strata, than I calculated the chisq .

    library(tableone)
    ## Load data
    library(survival); data(pbc)

    d <- pbc[,c("status", "trt")]

    # Convert dummy variables
    d$status.0 <- ifelse(d$status==0, 1,0)
    d$status.1 <- ifelse(d$status==1, 1,0)
    d$status.2 <- ifelse(d$status==2, 1,0)

t <- rbind(
    chisq.test(d$status.0, d$trt),
    # p-value = 0.7202

    chisq.test(d$status.1, d$trt),
    # p-value = 1

    chisq.test(d$status.2, d$trt)
    #p-value = 0.7818
)
t

BONFERRONI ADJ FOR MULTIPLE COMPARISONS:

p <- t[,"p.value"]

p.adjust(p, method = "bonferroni")

Body:
This question was posted some time ago, so I supose you already answered the reviewer.

I don't really understand why computing adjusted p values for just three varibles. In fact, adjusting p values depends on the number of comparisons made. If you use p.adjust() with a vector of 3 p values, results will not really be "adjusted" by the amount of comparison made (you really did more than a dozen and a half!)

I show how to extract all p-values so you can compute the adjusted ones.
To extract pValues from package tableOne there is a way calling object attributes (explained first), and two quick and dirty ways (at the bottom part).

To extract them, first I copy your code to create your tableOne:

library(tableone)
## Load data
library(survival); data(pbc)

# drop ID from variable list
vars <- names(pbc)[-1]

## Create Table 1 stratified by trt (can add more stratifying variables)
tableOne <- CreateTableOne(vars = vars, strata = c("trt"), data = pbc, factorVars = c("status","edema","stage"))

You can see what your "tableOne" object has via attributes()

attributes(tableOne)

You can see a tableOne usually has a table for continuous and categorical variables. You can use attributes() in them too

attributes(tableOne$CatTable)
# you can notice $pValues

Now you know "where" the pValues are, you can extract them with attr()

attr(tableOne$CatTable, "pValues")

Something similar with numerical variables:

attributes(tableOne$ContTable)
# $pValues are there
attr(tableOne$ContTable, "pValues")

You have pValues for Normal and NonNormal variables.
As you set them before, you can extract both

mypCont <- attr(tableOne$ContTable, "pValues") # put them in an object
nonnormal = c("bili","chol","copper","alk.phos","trig") # copied from your code

mypCont[rownames(mypCont) %in% c(nonnormal), "pNonNormal"] # extract NonNormal

"%!in%" <- Negate("%in%")
mypCont[rownames(mypCont) %!in% c(nonnormal), "pNonNormal"] # extract Normal

All that said, and your pValues extracted, I think there are two much more convenient quick and dirty ways to accomplish the same:

Quick and dirty way A: using dput() with your printed tableOne. Then search in the console where the pValues are and copy-paste them to the script, to store them in an object

Quick and dirty way B: If you look in tableOne vignette there is an "Exporting" section, you can use print(tableOne, quote = TRUE) and then just copy and paste to a spreadsheet (like LibreOffice, Excel...).
Then I would select the column with pValue, transpose it, and get it back to R, to compute adjusted p values with p.adjust() and copy them back to the spreadsheet for journal submission


问:
Add and remove to Local Storage
i'm creating a list with add to favorite link.
I want to store the 'foo' value on local storage using pure JS. Then once I click again it is remove.
Maybe I can not bind the identifier to the style name.
I wanted to repeat the way it is here https://jsfiddle.net/farhadB/ocgcejg2/ , but I can’t.

document.addEventListener('click', function(e) {
  if (e.target && e.target.classList.contains('photo')) {
    e.target.classList = "foo";
  } else if (e.target && e.target.classList.contains('foo')) {
    e.target.classList = "photo";
  }
});
```

答:
Hope this helps

document.addEventListener('click', function(e) {
  if (e.target && e.target.classList.contains('photo')) {
    e.target.classList.add("foo");
    e.target.classList.remove("photo");
    localStoarge.favlink = <link name here>;
  } else if (e.target && e.target.classList.contains('foo')) {
    e.target.classList.add("photo");
    e.target.classList.remove("foo");
    localStoarge.favlink = null; // Delete fav link
  }
});
```

Title: Add and remove to Local Storage


问:
Perform NLTK in pyspark
I am very new in pyspark and I have developed a program to perform NLTK on HDFS file, The following are the steps for that.I'm using spark 2.3.1

1. Get file from HDFS

2. perform Lemmatization

3. Remove punctuation mark.

4. Convert RDD to DataFrame

5. Perform Tokenizer

6. Remove Stop words

7. Explode columns data to create a unique row for each record

8. I want to keep all files data into a single file so I am merging the output with old fil

9. Now write this entire merged output into HDFS

10. Then deleting old file and renaming spark created file to different name

11. I am doing this for all bigram and trigram files.

Here is my pyspark code.

```<code>%pyspark

import os
import pyspark
import csv
import nltk
import json
import string
import re

from pyspark.ml.feature import Tokenizer, StopWordsRemover
from pyspark.ml.feature import NGram
from pyspark import SparkContext, SparkConf as sc
from pyspark.sql.types import StringType

from nltk.corpus import stopwords
nltk.download('stopwords')

from pyspark.sql import SQLContext
from pyspark.sql.functions import explode,regexp_replace

import pandas
import hdfs

nltk.download('punkt')
from nltk.stem import WordNetLemmatizer
nltk.download('wordnet')

from pyspark import SparkContext, SparkConf
# conf = SparkConf().setAppName("PySpark App")
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)

hdfs_dst_dir = "/user/zeppelin/achyuttest.csv/"
counter=0

#Lemmatizen

def lemma(x):

    lemmatizer = WordNetLemmatizer()
    return lemmatizer.lemmatize(x)

for i in range(1,50001):
    data = sc.textFile('hdfs:///user/spark/Patentdata/ElectronicsPatents/Link\ {}/abstract.txt'.format(i), use_unicode=False)

    print(type(data))

    if data.isEmpty():
        continue

    else:
        lem_words = data.map(lemma)

        list_punct=list(string.punctuation)

        len_list = lem_words.collect()

        test_str = len_list[0]
        test_df = test_str.split(' ')

        data_df = data.map(lambda x: (x, )).toDF(['lem_words'])

# Perform Tokenizer

        tokenizer =  Tokenizer(inputCol="lem_words", outputCol="tokenized_data")
        outputdata = tokenizer.transform(data_df)
        outputdata = outputdata.select('tokenized_data')

    # Remove stop words

        remover = StopWordsRemover(inputCol='tokenized_data', outputCol='words_clean')
        outputdata = remover.transform(outputdata).select('words_clean')

#Explode one Row into multiple Row with value

        result_df = outputdata.withColumn("exploded", explode("words_clean")).select("exploded")

        result_df=result_df.select(regexp_replace('exploded',"[^a-zA-Z\\s]",""))

        print("Link  ========>",i)
#Merge with old output

        if counter>0:
            old_data = sc.textFile('hdfs:///user/zeppelin/achyuttest.csv/unigram.csv', use_unicode=False)
            old_data_df = old_data.map(lambda x: (x, )).toDF(['words_clean'])

            result_df = old_data_df.union(result_df)

        else:
            pass

#Write DataFrame to HDFS

        result_df.coalesce(1).write.mode('append').csv(hdfs_dst_dir)

        fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration())

    # Rename file

    #list files in the directory

        list_status = fs.listStatus(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir))

    #filter name of the file starts with part-

        print("Get FileName")
        file_name = [file.getPath().getName() for file in list_status if file.getPath().getName().startswith('part-')][0]

        print(file_name)
    #rename the file

        new_filename = "unigram.csv"

    # Remove Old file

        fs.delete(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))
        fs.rename(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+file_name),spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))

## Bigrams

        bigram = NGram(n=2, inputCol="words_clean", outputCol="bigrams")

        bigramDataFrame = bigram.transform(outputdata)

    #Explode one Row into multiple Row with value

        result_df = bigramDataFrame.withColumn("exploded", explode("bigrams")).select("exploded")
        result_df=result_df.select(regexp_replace('exploded',"[^a-zA-Z\\s]",""))

    #Merge with old output

        if counter>0:
            old_data = sc.textFile('hdfs:///user/zeppelin/achyuttest.csv/bigram.csv', use_unicode=False)
            old_data_df = old_data.map(lambda x: (x, )).toDF(["exploded"])

            result_df = old_data_df.union(result_df)

        else:
            pass

    # Write Output in file

        result_df.coalesce(1).write.mode('append').csv('hdfs:///user/zeppelin/achyuttest.csv')

    # Rename file

    #list files in the directory

        list_status = fs.listStatus(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir))

    #filter name of the file starts with part-

        file_name = [file.getPath().getName() for file in list_status if file.getPath().getName().startswith('part-')][0]

    #rename the file

        new_filename = "bigram.csv"

        fs.delete(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))
        fs.rename(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+file_name),spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))

## TriGram

        trigram = NGram(n=3, inputCol="words_clean", outputCol="trigrams")

        trigramDataFrame = trigram.transform(outputdata)

    #Explode one Row into multiple Row with value

        result_df = trigramDataFrame.withColumn("exploded", explode("trigrams")).select("exploded")
        result_df=result_df.select(regexp_replace('exploded',"[^a-zA-Z\\s]",""))

    #Merge with old output

        if counter>0:
            old_data = sc.textFile('hdfs:///user/zeppelin/achyuttest.csv/trigram.csv', use_unicode=False)
            old_data_df = old_data.map(lambda x: (x, )).toDF(["exploded"])

            result_df = old_data_df.union(result_df)

        else:
            pass

#Save DataFrame in HDFS
        result_df.coalesce(1).write.mode('append').csv('hdfs:///user/zeppelin/achyuttest.csv')

    # Rename file

    #list files in the directory

        list_status = fs.listStatus(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir))

    #filter name of the file starts with part-

        file_name = [file.getPath().getName() for file in list_status if file.getPath().getName().startswith('part-')][0]

    #rename the file

        new_filename = "trigram.csv"

        fs.delete(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))
        fs.rename(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+file_name),spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))

        counter = counter+1
</code>```

I am performing this code on 50K files, and my spark is taking too much time to perform this program. (Passed 2 days and still going ...)

I'm running HDP in Virtual machine(running one node HDP Sandbox)Here is my system specification...

====> Guest OS::

Memory: 12930 MB

CPU: 6CPUs

===> YARN Specifications::

1.Memory: 4608 MB

Maximum Container memory: 4608 MB

Maximum Container size(Vcores): 4

Number of virtual core: 4

===> Zeppelin Pyspark Interpreter Specification::
1. spark.executor.memory: Blank (it's mean 1g as per specified in the documentation)

So I have two questions.

Is my code proper or not?

Which value I have to specify in YARN and Zeppelin Interpreter so it will work fast and efficiently.

Thank you.

答:
I'm answering my first question.

According to the old code, I was making an RDD for each file located in folder, So It was taking too much time (To process 3K files it was taking 19 hr.)

But Now What I have done is to Read all input files in Single RDD operation, and perform all operations on it. (Now New code is taking ~15 min to process 3K files.)

Final Code

Comments are used for extra understanding

Patentdetect-local.py

```<code>"""
To Run this code
Set Pyspark_python

$ export PYSPARK_PYTHON=/usr/bin/python3
$ pip install nltk

RUN ON Spark::

$ ./bin/spark-submit file_path/Patentdetect-local.py
"""

import pyspark
import nltk
import string
import os
import re

from pyspark import SparkContext
from nltk.stem import WordNetLemmatizer

from pyspark.ml.feature import NGram
from pyspark.sql.types import ArrayType,StructType,StructField,StringType
from pyspark.sql.functions import explode,array,split,collect_list
from pyspark.sql.window import Window
from pyspark.sql import SparkSession

sc = SparkContext.getOrCreate()

spark = SparkSession.builder.appName('Spark Example').getOrCreate()

Source_path="<path>/*/abstract.txt"

Destination_path="<path>/spark-outputs/parquet/Electronics-50/"

data=sc.textFile(Source_path)

data.persist()
lower_casetext = data.map(lambda x:x.lower())

# splitting_rdd = lower_casetext.map(lambda x:x.split(" "))
# print(splitting_rdd.collect())

# Function to perform Sentence tokeniaztion
def sent_TokenizeFunct(x):
    return nltk.sent_tokenize(x)

sentencetokenization_rdd = lower_casetext.map(sent_TokenizeFunct)

# Function to perform Word tokenization

def word_TokenizeFunct(x):
    splitted = [word for line in x for word in line.split()]
    return splitted

wordtokenization_rdd = sentencetokenization_rdd.map(word_TokenizeFunct)

# Remove Stop Words

def removeStopWordsFunct(x):
    from nltk.corpus import stopwords
    stop_words=set(stopwords.words('english'))
    filteredSentence = [w for w in x if not w in stop_words]
    return filteredSentence
stopwordRDD = wordtokenization_rdd.map(removeStopWordsFunct)

# Remove Punctuation marks

def removePunctuationsFunct(x):
    list_punct=list(string.punctuation)
    filtered = [''.join(c for c in s if c not in list_punct) for s in x]
    filtered_space = [s for s in filtered if s] #remove empty space
    return filtered
rmvPunctRDD = stopwordRDD.map(removePunctuationsFunct)

# Perform Lemmatization

def lemma(x):

    lemmatizer = WordNetLemmatizer()

    final_rdd = [lemmatizer.lemmatize(s) for s in x]
    return final_rdd

lem_wordsRDD = rmvPunctRDD.map(lemma)

# Join tokens

# def joinTokensFunct(x):
#     joinedTokens_list = []
#     x = " ".join(x)
#     return x

# joinedTokensRDD = lem_wordsRDD.map(joinTokensFunct)

##Create DataFrame from RDD

df = lem_wordsRDD.map(lambda x: (x, )).toDF(["features"])

tokenized_df = df.withColumn("values", explode("features")).select("values")

## Write DataFrame Output

# tokenized_df.write.mode('append').csv(Destination_path)

## Change File-name

# for old_file_name in os.listdir(Destination_path):
#   src = Destination_path+old_file_name
#   dst = Destination_path+"unigram.csv"

#   if old_file_name.startswith("part-"):
#       os.rename(src, dst)
        # break

## For Bigrams following commented line is enough
# # tokenized_df.select(F.concat_ws(" ",F.col("values"),F.lead("values").over(Window.orderBy(F.lit(None))))).show()

## Create Final DataFrme

final_df = tokenized_df.select(collect_list("values").alias("listed_data"))

# final_df.show(truncate=False)

final_df.persist()

## Unigram

unigram = NGram(n=1, inputCol="listed_data", outputCol="unigrams")

unigramDataFrame = unigram.transform(final_df)

unigram_FinalDataFrame = unigramDataFrame.withColumn("unigram_final",explode("unigrams")).select("unigram_final")

## Write DataFrame Outputs

unigram_FinalDataFrame.write.mode('append').parquet(Destination_path)

# Change filename
for old_file_name in os.listdir(Destination_path):
    src = Destination_path+old_file_name
    dst = Destination_path+"unigram.parquet"

    if old_file_name.startswith("part-"):
        os.rename(src, dst)

## Bigram

bigram = NGram(n=2, inputCol="listed_data", outputCol="bigrams")

bigramDataFrame = bigram.transform(final_df)

bigram_FinalDataFrame = bigramDataFrame.withColumn("bigram_final",explode("bigrams")).select("bigram_final")

## Write DataFrame Outputs

bigram_FinalDataFrame.write.mode('append').parquet(Destination_path)

## Change filename
for old_file_name in os.listdir(Destination_path):
    src = Destination_path+old_file_name
    dst = Destination_path+"bigram.parquet"

    if old_file_name.startswith("part-"):
        os.rename(src, dst)
        # break

## Trigram

trigram = NGram(n=3, inputCol="listed_data", outputCol="trigram")

trigramDataFrame = trigram.transform(final_df)

trigram_FinalDataFrame = trigramDataFrame.withColumn("trigram_final",explode("trigram")).select("trigram_final")

## Write DataFrame Outputs
trigram_FinalDataFrame.write.mode('append').parquet(Destination_path)

# Change Filename
for old_file_name in os.listdir(Destination_path):
    src = Destination_path+old_file_name
    dst = Destination_path+"trigram.parquet"

    if old_file_name.startswith("part-"):
        os.rename(src, dst)
        # break

final_df.unpersist()
data.unpersist()
</code>```

Title: Perform NLTK in pyspark

Body:
I am very new in pyspark and I have developed a program to perform NLTK on HDFS file, The following are the steps for that.I'm using spark 2.3.1

1. Get file from HDFS

2. perform Lemmatization

3. Remove punctuation mark.

4. Convert RDD to DataFrame

5. Perform Tokenizer

6. Remove Stop words

7. Explode columns data to create a unique row for each record

8. I want to keep all files data into a single file so I am merging the output with old fil

9. Now write this entire merged output into HDFS

10. Then deleting old file and renaming spark created file to different name

11. I am doing this for all bigram and trigram files.

Here is my pyspark code.

%pyspark

import os
import pyspark
import csv
import nltk
import json
import string
import re

from pyspark.ml.feature import Tokenizer, StopWordsRemover
from pyspark.ml.feature import NGram
from pyspark import SparkContext, SparkConf as sc
from pyspark.sql.types import StringType

from nltk.corpus import stopwords
nltk.download('stopwords')

from pyspark.sql import SQLContext
from pyspark.sql.functions import explode,regexp_replace

import pandas
import hdfs

nltk.download('punkt')
from nltk.stem import WordNetLemmatizer
nltk.download('wordnet')

from pyspark import SparkContext, SparkConf
# conf = SparkConf().setAppName("PySpark App")
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)

hdfs_dst_dir = "/user/zeppelin/achyuttest.csv/"
counter=0

#Lemmatizen

def lemma(x):

    lemmatizer = WordNetLemmatizer()
    return lemmatizer.lemmatize(x)

for i in range(1,50001):
    data = sc.textFile('hdfs:///user/spark/Patentdata/ElectronicsPatents/Link\ {}/abstract.txt'.format(i), use_unicode=False)

    print(type(data))

    if data.isEmpty():
        continue

    else:
        lem_words = data.map(lemma)

        list_punct=list(string.punctuation)

        len_list = lem_words.collect()

        test_str = len_list[0]
        test_df = test_str.split(' ')

        data_df = data.map(lambda x: (x, )).toDF(['lem_words'])

# Perform Tokenizer

        tokenizer =  Tokenizer(inputCol="lem_words", outputCol="tokenized_data")
        outputdata = tokenizer.transform(data_df)
        outputdata = outputdata.select('tokenized_data')

    # Remove stop words

        remover = StopWordsRemover(inputCol='tokenized_data', outputCol='words_clean')
        outputdata = remover.transform(outputdata).select('words_clean')

#Explode one Row into multiple Row with value

        result_df = outputdata.withColumn("exploded", explode("words_clean")).select("exploded")

        result_df=result_df.select(regexp_replace('exploded',"[^a-zA-Z\\s]",""))

        print("Link  ========>",i)
#Merge with old output

        if counter>0:
            old_data = sc.textFile('hdfs:///user/zeppelin/achyuttest.csv/unigram.csv', use_unicode=False)
            old_data_df = old_data.map(lambda x: (x, )).toDF(['words_clean'])

            result_df = old_data_df.union(result_df)

        else:
            pass

#Write DataFrame to HDFS

        result_df.coalesce(1).write.mode('append').csv(hdfs_dst_dir)

        fs = spark._jvm.org.apache.hadoop.fs.FileSystem.get(spark._jsc.hadoopConfiguration())

    # Rename file

    #list files in the directory

        list_status = fs.listStatus(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir))

    #filter name of the file starts with part-

        print("Get FileName")
        file_name = [file.getPath().getName() for file in list_status if file.getPath().getName().startswith('part-')][0]

        print(file_name)
    #rename the file

        new_filename = "unigram.csv"

    # Remove Old file

        fs.delete(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))
        fs.rename(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+file_name),spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))

## Bigrams

        bigram = NGram(n=2, inputCol="words_clean", outputCol="bigrams")

        bigramDataFrame = bigram.transform(outputdata)

    #Explode one Row into multiple Row with value

        result_df = bigramDataFrame.withColumn("exploded", explode("bigrams")).select("exploded")
        result_df=result_df.select(regexp_replace('exploded',"[^a-zA-Z\\s]",""))

    #Merge with old output

        if counter>0:
            old_data = sc.textFile('hdfs:///user/zeppelin/achyuttest.csv/bigram.csv', use_unicode=False)
            old_data_df = old_data.map(lambda x: (x, )).toDF(["exploded"])

            result_df = old_data_df.union(result_df)

        else:
            pass

    # Write Output in file

        result_df.coalesce(1).write.mode('append').csv('hdfs:///user/zeppelin/achyuttest.csv')

    # Rename file

    #list files in the directory

        list_status = fs.listStatus(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir))

    #filter name of the file starts with part-

        file_name = [file.getPath().getName() for file in list_status if file.getPath().getName().startswith('part-')][0]

    #rename the file

        new_filename = "bigram.csv"

        fs.delete(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))
        fs.rename(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+file_name),spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))

## TriGram

        trigram = NGram(n=3, inputCol="words_clean", outputCol="trigrams")

        trigramDataFrame = trigram.transform(outputdata)

    #Explode one Row into multiple Row with value

        result_df = trigramDataFrame.withColumn("exploded", explode("trigrams")).select("exploded")
        result_df=result_df.select(regexp_replace('exploded',"[^a-zA-Z\\s]",""))

    #Merge with old output

        if counter>0:
            old_data = sc.textFile('hdfs:///user/zeppelin/achyuttest.csv/trigram.csv', use_unicode=False)
            old_data_df = old_data.map(lambda x: (x, )).toDF(["exploded"])

            result_df = old_data_df.union(result_df)

        else:
            pass

#Save DataFrame in HDFS
        result_df.coalesce(1).write.mode('append').csv('hdfs:///user/zeppelin/achyuttest.csv')

    # Rename file

    #list files in the directory

        list_status = fs.listStatus(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir))

    #filter name of the file starts with part-

        file_name = [file.getPath().getName() for file in list_status if file.getPath().getName().startswith('part-')][0]

    #rename the file

        new_filename = "trigram.csv"

        fs.delete(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))
        fs.rename(spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+file_name),spark._jvm.org.apache.hadoop.fs.Path(hdfs_dst_dir+''+new_filename))

        counter = counter+1

I am performing this code on 50K files, and my spark is taking too much time to perform this program. (Passed 2 days and still going ...)

I'm running HDP in Virtual machine(running one node HDP Sandbox)Here is my system specification...

====> Guest OS::

Memory: 12930 MB

CPU: 6CPUs

===> YARN Specifications::

1.Memory: 4608 MB

Maximum Container memory: 4608 MB

Maximum Container size(Vcores): 4

Number of virtual core: 4

===> Zeppelin Pyspark Interpreter Specification::
1. spark.executor.memory: Blank (it's mean 1g as per specified in the documentation)

So I have two questions.

Is my code proper or not?

Which value I have to specify in YARN and Zeppelin Interpreter so it will work fast and efficiently.

Thank you.

Body:
I'm answering my first question.

According to the old code, I was making an RDD for each file located in folder, So It was taking too much time (To process 3K files it was taking 19 hr.)

But Now What I have done is to Read all input files in Single RDD operation, and perform all operations on it. (Now New code is taking ~15 min to process 3K files.)

Final Code

Comments are used for extra understanding

Patentdetect-local.py

"""
To Run this code
Set Pyspark_python

$ export PYSPARK_PYTHON=/usr/bin/python3
$ pip install nltk

RUN ON Spark::

$ ./bin/spark-submit file_path/Patentdetect-local.py
"""

import pyspark
import nltk
import string
import os
import re

from pyspark import SparkContext
from nltk.stem import WordNetLemmatizer

from pyspark.ml.feature import NGram
from pyspark.sql.types import ArrayType,StructType,StructField,StringType
from pyspark.sql.functions import explode,array,split,collect_list
from pyspark.sql.window import Window
from pyspark.sql import SparkSession

sc = SparkContext.getOrCreate()

spark = SparkSession.builder.appName('Spark Example').getOrCreate()

Source_path="<path>/*/abstract.txt"

Destination_path="<path>/spark-outputs/parquet/Electronics-50/"

data=sc.textFile(Source_path)

data.persist()
lower_casetext = data.map(lambda x:x.lower())

# splitting_rdd = lower_casetext.map(lambda x:x.split(" "))
# print(splitting_rdd.collect())

# Function to perform Sentence tokeniaztion
def sent_TokenizeFunct(x):
    return nltk.sent_tokenize(x)

sentencetokenization_rdd = lower_casetext.map(sent_TokenizeFunct)

# Function to perform Word tokenization

def word_TokenizeFunct(x):
    splitted = [word for line in x for word in line.split()]
    return splitted

wordtokenization_rdd = sentencetokenization_rdd.map(word_TokenizeFunct)

# Remove Stop Words

def removeStopWordsFunct(x):
    from nltk.corpus import stopwords
    stop_words=set(stopwords.words('english'))
    filteredSentence = [w for w in x if not w in stop_words]
    return filteredSentence
stopwordRDD = wordtokenization_rdd.map(removeStopWordsFunct)

# Remove Punctuation marks

def removePunctuationsFunct(x):
    list_punct=list(string.punctuation)
    filtered = [''.join(c for c in s if c not in list_punct) for s in x]
    filtered_space = [s for s in filtered if s] #remove empty space
    return filtered
rmvPunctRDD = stopwordRDD.map(removePunctuationsFunct)

# Perform Lemmatization

def lemma(x):

    lemmatizer = WordNetLemmatizer()

    final_rdd = [lemmatizer.lemmatize(s) for s in x]
    return final_rdd

lem_wordsRDD = rmvPunctRDD.map(lemma)

# Join tokens

# def joinTokensFunct(x):
#     joinedTokens_list = []
#     x = " ".join(x)
#     return x

# joinedTokensRDD = lem_wordsRDD.map(joinTokensFunct)

##Create DataFrame from RDD

df = lem_wordsRDD.map(lambda x: (x, )).toDF(["features"])

tokenized_df = df.withColumn("values", explode("features")).select("values")

## Write DataFrame Output

# tokenized_df.write.mode('append').csv(Destination_path)

## Change File-name

# for old_file_name in os.listdir(Destination_path):
#   src = Destination_path+old_file_name
#   dst = Destination_path+"unigram.csv"

#   if old_file_name.startswith("part-"):
#       os.rename(src, dst)
        # break

## For Bigrams following commented line is enough
# # tokenized_df.select(F.concat_ws(" ",F.col("values"),F.lead("values").over(Window.orderBy(F.lit(None))))).show()

## Create Final DataFrme

final_df = tokenized_df.select(collect_list("values").alias("listed_data"))

# final_df.show(truncate=False)

final_df.persist()

## Unigram

unigram = NGram(n=1, inputCol="listed_data", outputCol="unigrams")

unigramDataFrame = unigram.transform(final_df)

unigram_FinalDataFrame = unigramDataFrame.withColumn("unigram_final",explode("unigrams")).select("unigram_final")

## Write DataFrame Outputs

unigram_FinalDataFrame.write.mode('append').parquet(Destination_path)

# Change filename
for old_file_name in os.listdir(Destination_path):
    src = Destination_path+old_file_name
    dst = Destination_path+"unigram.parquet"

    if old_file_name.startswith("part-"):
        os.rename(src, dst)

## Bigram

bigram = NGram(n=2, inputCol="listed_data", outputCol="bigrams")

bigramDataFrame = bigram.transform(final_df)

bigram_FinalDataFrame = bigramDataFrame.withColumn("bigram_final",explode("bigrams")).select("bigram_final")

## Write DataFrame Outputs

bigram_FinalDataFrame.write.mode('append').parquet(Destination_path)

## Change filename
for old_file_name in os.listdir(Destination_path):
    src = Destination_path+old_file_name
    dst = Destination_path+"bigram.parquet"

    if old_file_name.startswith("part-"):
        os.rename(src, dst)
        # break

## Trigram

trigram = NGram(n=3, inputCol="listed_data", outputCol="trigram")

trigramDataFrame = trigram.transform(final_df)

trigram_FinalDataFrame = trigramDataFrame.withColumn("trigram_final",explode("trigram")).select("trigram_final")

## Write DataFrame Outputs
trigram_FinalDataFrame.write.mode('append').parquet(Destination_path)

# Change Filename
for old_file_name in os.listdir(Destination_path):
    src = Destination_path+old_file_name
    dst = Destination_path+"trigram.parquet"

    if old_file_name.startswith("part-"):
        os.rename(src, dst)
        # break

final_df.unpersist()
data.unpersist()


问:
OpenCV (cv2 in Python) VideoCapture not releasing camera after deletion
I am relatively new to Python, just having learnt it over the past month or so and have hacked this together based off examples and others' code I found online.

I have gotten a Tkinter GUI to display the feed from a webcam as a loop of continuously updated images on a canvas. Quitting the GUI and re-running the script every other time results in this error:

```<code>Exception in Tkinter callback
Traceback (most recent call last):
    File "C:\Python27\lib\lib-tk\Tkinter.py", line 1410, in __call__
        return self.func(*args)
    File "C:\Python27\lib\lib-tk\Tkinter.py", line 495, in callit
        func(*args)
   File "C:\...\cv2_cam_v8.py", line 20, in update_video
        (self.readsuccessful,self.f) = self.cam.read()
SystemError: NULL object passed to Py_BuildValue
</code>```

When the error happens no images get read and the videofeed recieves no images to update the canvas. The script runs normally with no errors the first time and every second time. From previous tests with the VideoCapture function in the cv2 module, I found that I had to delete the camera object to release it so that subsequent runs are able to capture the camera stream with no issue. Checks on the namespace by typing who in the console do not show cam so I know it is being deleted properly after the GUI is closed. I do not understand why cv2's read function is giving an error. I think it is only happening every second time because when the error occurs, some garbage collection or error handling deletes or frees up something to do with the camera but I do not know what this is...

Here is my code:

```<code>import cv2
import Tkinter as tk
from PIL import Image, ImageTk

class vid():
    def __init__(self,cam,root,canvas):
        self.cam = cam
        self.root = root
        self.canvas = canvas

    def update_video(self):
        (self.readsuccessful,self.f) = self.cam.read()
        self.gray_im = cv2.cvtColor(self.f, cv2.COLOR_RGB2GRAY)
        self.a = Image.fromarray(self.gray_im)
        self.b = ImageTk.PhotoImage(image=self.a)
        self.canvas.create_image(0,0,image=self.b,anchor=tk.NW)
        self.root.update()
        self.root.after(33,self.update_video)

if __name__ == '__main__':
    root = tk.Tk()
    videoframe = tk.LabelFrame(root,text='Captured video')
    videoframe.grid(column=0,row=0,columnspan=1,rowspan=1,padx=5, pady=5, ipadx=5, ipady=5)
    canvas = tk.Canvas(videoframe, width=640,height=480)
    canvas.grid(column=0,row=0)
    cam = cv2.VideoCapture(2)
    x = vid(cam,root,canvas)
    root.after(0,x.update_video)
    button = tk.Button(text='Quit',master=videoframe,command=root.destroy)
    button.grid(column=0,row=1)
    root.mainloop()
    del cam
</code>```

Refactoring the code like this:

```<code>def update_video(cam,root,canvas):
    (readsuccessful,f) = cam.read()
    gray_im = cv2.cvtColor(f, cv2.COLOR_RGB2GRAY)
    a = Image.fromarray(gray_im)
    b = ImageTk.PhotoImage(image=a)
    canvas.create_image(0,0,image=b,anchor=tk.NW)
    root.update()
    root.after(33,update_video(cam,root,canvas))

if __name__ == '__main__':
    root = tk.Tk()
    videoframe = tk.LabelFrame(root,text='Captured video')
    videoframe.grid(column=0,row=0,columnspan=1,rowspan=1,padx=5, pady=5, ipadx=5, ipady=5)
    canvas = tk.Canvas(videoframe, width=640,height=480)
    canvas.grid(column=0,row=0)
    cam = cv2.VideoCapture(2)
    root.after(0,update_video(cam,root,canvas))
    button = tk.Button(text='Quit',master=videoframe,command=root.destroy)
    button.grid(column=0,row=1)
    root.mainloop()
    del cam
</code>```

does not display the button in the GUI and gives this error after closing the window:

```<code>RuntimeError: Too early to create image
</code>```

I have 3 questions

1 - How can I prevent either exception?
UPDATE: changing "root.after(0,update_video(cam,root,canvas))" to "root.after(0,lambda: update_video(cam,root,canvas))" and "update_video(cam,root,canvas)" to "update_video(cam,root,canvas,event=None)" OR passing the arguments to the callback using this format: "root.after(time_to_wait, callback, arguments, master)" fixes the second error (and others I did not post). Also as kobejohn pointed out, adding a try: except block also fixes the second error. Please see his answer for more details.

2 - Is there a faster, more efficient function than .read() in cv2? Edit: Is there a way to refactor my code to get higher framerates? The read function is the only one listed in the docs and I just read somewhere that if it is not in the docs, then it is not available. This method only gives me about 5fps, where 10-20fps would be much more acceptable.
UPDATE: From the discrepancies between kobejohn's tests and mine with different cameras, the low framerate is a result of poor quality webcams. Better quality webcams yield higher framerates.

3 - I have been reading that update() should be avoided as much as possible but how do I get the canvas to redraw the image otherwise (or implement update_idletasks() with this code)?. Do I have to implement some sort of threading or can I avoid that?
UPDATE: I have gotten the code to work without using the update() method but have to look at implementing threading anyway because when I start recording the videofeed from a button the main GUI, it freezes/ becomes unresponsive.

The finished program will be used in Ubuntu and windows (possibly on macs as well). I am running Windows 7, IDE is Spyder 2.1.11 (Python 2.7.3).

Thank you in advance, any advice and/or solutions will be much appreciated!

Regards,

S. Chia

答:
Set the environment variable before you initialize the camera object in opencv.

```<code>os.environ['OPENCV_VIDEOIO_PRIORITY_MSMF'] = '0'
</code>```

This released the camera even after closing the camera object in my code.

Body:
I am relatively new to Python, just having learnt it over the past month or so and have hacked this together based off examples and others' code I found online.

I have gotten a Tkinter GUI to display the feed from a webcam as a loop of continuously updated images on a canvas. Quitting the GUI and re-running the script every other time results in this error:

Exception in Tkinter callback
Traceback (most recent call last):
    File "C:\Python27\lib\lib-tk\Tkinter.py", line 1410, in __call__
        return self.func(*args)
    File "C:\Python27\lib\lib-tk\Tkinter.py", line 495, in callit
        func(*args)
   File "C:\...\cv2_cam_v8.py", line 20, in update_video
        (self.readsuccessful,self.f) = self.cam.read()
SystemError: NULL object passed to Py_BuildValue

When the error happens no images get read and the videofeed recieves no images to update the canvas. The script runs normally with no errors the first time and every second time. From previous tests with the VideoCapture function in the cv2 module, I found that I had to delete the camera object to release it so that subsequent runs are able to capture the camera stream with no issue. Checks on the namespace by typing who in the console do not show cam so I know it is being deleted properly after the GUI is closed. I do not understand why cv2's read function is giving an error. I think it is only happening every second time because when the error occurs, some garbage collection or error handling deletes or frees up something to do with the camera but I do not know what this is...

Here is my code:

import cv2
import Tkinter as tk
from PIL import Image, ImageTk

class vid():
    def __init__(self,cam,root,canvas):
        self.cam = cam
        self.root = root
        self.canvas = canvas

    def update_video(self):
        (self.readsuccessful,self.f) = self.cam.read()
        self.gray_im = cv2.cvtColor(self.f, cv2.COLOR_RGB2GRAY)
        self.a = Image.fromarray(self.gray_im)
        self.b = ImageTk.PhotoImage(image=self.a)
        self.canvas.create_image(0,0,image=self.b,anchor=tk.NW)
        self.root.update()
        self.root.after(33,self.update_video)

if __name__ == '__main__':
    root = tk.Tk()
    videoframe = tk.LabelFrame(root,text='Captured video')
    videoframe.grid(column=0,row=0,columnspan=1,rowspan=1,padx=5, pady=5, ipadx=5, ipady=5)
    canvas = tk.Canvas(videoframe, width=640,height=480)
    canvas.grid(column=0,row=0)
    cam = cv2.VideoCapture(2)
    x = vid(cam,root,canvas)
    root.after(0,x.update_video)
    button = tk.Button(text='Quit',master=videoframe,command=root.destroy)
    button.grid(column=0,row=1)
    root.mainloop()
    del cam

Refactoring the code like this:

def update_video(cam,root,canvas):
    (readsuccessful,f) = cam.read()
    gray_im = cv2.cvtColor(f, cv2.COLOR_RGB2GRAY)
    a = Image.fromarray(gray_im)
    b = ImageTk.PhotoImage(image=a)
    canvas.create_image(0,0,image=b,anchor=tk.NW)
    root.update()
    root.after(33,update_video(cam,root,canvas))

if __name__ == '__main__':
    root = tk.Tk()
    videoframe = tk.LabelFrame(root,text='Captured video')
    videoframe.grid(column=0,row=0,columnspan=1,rowspan=1,padx=5, pady=5, ipadx=5, ipady=5)
    canvas = tk.Canvas(videoframe, width=640,height=480)
    canvas.grid(column=0,row=0)
    cam = cv2.VideoCapture(2)
    root.after(0,update_video(cam,root,canvas))
    button = tk.Button(text='Quit',master=videoframe,command=root.destroy)
    button.grid(column=0,row=1)
    root.mainloop()
    del cam

does not display the button in the GUI and gives this error after closing the window:

RuntimeError: Too early to create image

I have 3 questions

1 - How can I prevent either exception?
UPDATE: changing "root.after(0,update_video(cam,root,canvas))" to "root.after(0,lambda: update_video(cam,root,canvas))" and "update_video(cam,root,canvas)" to "update_video(cam,root,canvas,event=None)" OR passing the arguments to the callback using this format: "root.after(time_to_wait, callback, arguments, master)" fixes the second error (and others I did not post). Also as kobejohn pointed out, adding a try: except block also fixes the second error. Please see his answer for more details.

2 - Is there a faster, more efficient function than .read() in cv2? Edit: Is there a way to refactor my code to get higher framerates? The read function is the only one listed in the docs and I just read somewhere that if it is not in the docs, then it is not available. This method only gives me about 5fps, where 10-20fps would be much more acceptable.
UPDATE: From the discrepancies between kobejohn's tests and mine with different cameras, the low framerate is a result of poor quality webcams. Better quality webcams yield higher framerates.

3 - I have been reading that update() should be avoided as much as possible but how do I get the canvas to redraw the image otherwise (or implement update_idletasks() with this code)?. Do I have to implement some sort of threading or can I avoid that?
UPDATE: I have gotten the code to work without using the update() method but have to look at implementing threading anyway because when I start recording the videofeed from a button the main GUI, it freezes/ becomes unresponsive.

The finished program will be used in Ubuntu and windows (possibly on macs as well). I am running Windows 7, IDE is Spyder 2.1.11 (Python 2.7.3).

Thank you in advance, any advice and/or solutions will be much appreciated!

Regards,

S. Chia

Body:
Set the environment variable before you initialize the camera object in opencv.

os.environ['OPENCV_VIDEOIO_PRIORITY_MSMF'] = '0'

This released the camera even after closing the camera object in my code.


问:
What is the Deno equivalent of process.argv in Node.js?
When working with NodeJS, I can pass the arguments to a Node script like this:

$ node node-server.js arg1 arg2=arg2-val arg3

And can get the arguments like so:

```<code>// print process.argv
process.argv.forEach(function (val, index, array) {
  console.log(index + ': ' + val);
});
//Output
0: node
1: /Users/umar/work/node/node-server.js
2: arg1
3: arg2=arg2-val
4: arg3
</code>```

How to get the command-line arguments in Deno?

Some experts suggested me to solve the problem by answers to the question

答:
Deno executable path ~ process.argv[0]:

```<code>Deno.execPath()
</code>```

File URL of executed script ~ process.argv[1]:

```<code>Deno.mainModule
</code>```

You can use path.fromFileUrl for conversions of URL to path string:

```<code>import { fromFileUrl } from "https://deno.land/std@0.55.0/path/mod.ts";
const modPath = fromFileUrl(import.meta.url)
</code>```

Command-line arguments ~ process.argv.slice(2):

```<code>Deno.args
</code>```

Example

```<code>deno run --allow-read test.ts -foo -bar=baz 42
</code>```

Sample output (Windows):

```<code>Deno.execPath(): <scoop path>\apps\deno\current\deno.exe
import.meta.url: file:///C:/path/to/project/test.ts
  as path: C:\path\to\project\test.ts
Deno.args: [ "-foo", "-bar=baz", "42" ]
</code>```

Body:
When working with NodeJS, I can pass the arguments to a Node script like this:

$ node node-server.js arg1 arg2=arg2-val arg3

And can get the arguments like so:

// print process.argv
process.argv.forEach(function (val, index, array) {
  console.log(index + ': ' + val);
});
//Output
0: node
1: /Users/umar/work/node/node-server.js
2: arg1
3: arg2=arg2-val
4: arg3

How to get the command-line arguments in Deno?

Some experts suggested me to solve the problem by answers to the question

Body:
Deno executable path ~ process.argv[0]:

Deno.execPath()

File URL of executed script ~ process.argv[1]:

Deno.mainModule

You can use path.fromFileUrl for conversions of URL to path string:

import { fromFileUrl } from "https://deno.land/std@0.55.0/path/mod.ts";
const modPath = fromFileUrl(import.meta.url)

Command-line arguments ~ process.argv.slice(2):

Deno.args

Example

deno run --allow-read test.ts -foo -bar=baz 42

Sample output (Windows):

Deno.execPath(): <scoop path>\apps\deno\current\deno.exe
import.meta.url: file:///C:/path/to/project/test.ts
  as path: C:\path\to\project\test.ts
Deno.args: [ "-foo", "-bar=baz", "42" ]


问:
Session variable was empty in php
login.php

```<code>    <?php
    ob_start();
    session_start();
    include '../config.php';
    if( (isset($_SESSION['can-id']) )) {
    header('location: ../home/profile.php');
    }
    if(isset($_POST['can-login']))
    {
        $email=$_POST['email'];
        $password=$_POST['password'];
    $sql="SELECT * FROM `user_credentials` WHERE `email`=:email AND `password`=:password";
    $pdoResult=$conn->prepare($sql);
    $pdoExec=$pdoResult->execute(array(":email"=>$email,":password"=>$password));
    $pdoResult->setFetchMode(PDO::FETCH_ASSOC);
    $count=0;
    $uid='';
    while ($r=$pdoResult->fetch()) {
      # code...
    $count+=1;
    $uid=$r['email'];
    }

    if ($count==1) {
      # code...
      $_SESSION['can-id']=$uid;
      header('location: ../home/profile.php');

    }
    else
    {
      $_SESSION['error']="login failed";
    }
    }
    ?>

<html>
....
</html>
</code>```

profile.php

```<code><?php
    ob_start();
    session_start();
    if (!(isset($_SESSION['can-id']))) {
        # code...
        header('location: ../login/');

    }
    else
    {
        $cid=$_SESSION['can-id'];
    }
?>
<h1 ><?php echo $cid;?></h1>
</code>```

This is my code after log in the page was redirected to profile.php page but in profile page session variable doesn't printed I don't know why but this problem was not occuring every time I log in It occurs sometimes so I can't find what is the problem. Anyone knows  please help me to solve the problem.

答:
Remove  ob_start() from your login.php
Don't put session_start() in all of your file

```<code> e.g login.php, profile.php, etc
</code>```

but instead, add this to your config.php for example:

```<code><?php
 session_start();
 //.. config variables here
</code>```

Then, include config.php also in your profile.php.

Title: Session variable was empty in php

Body:
login.php

    <?php
    ob_start();
    session_start();
    include '../config.php';
    if( (isset($_SESSION['can-id']) )) {
    header('location: ../home/profile.php');
    }
    if(isset($_POST['can-login']))
    {
        $email=$_POST['email'];
        $password=$_POST['password'];
    $sql="SELECT * FROM `user_credentials` WHERE `email`=:email AND `password`=:password";
    $pdoResult=$conn->prepare($sql);
    $pdoExec=$pdoResult->execute(array(":email"=>$email,":password"=>$password));
    $pdoResult->setFetchMode(PDO::FETCH_ASSOC);
    $count=0;
    $uid='';
    while ($r=$pdoResult->fetch()) {
      # code...
    $count+=1;
    $uid=$r['email'];
    }

    if ($count==1) {
      # code...
      $_SESSION['can-id']=$uid;
      header('location: ../home/profile.php');

    }
    else
    {
      $_SESSION['error']="login failed";
    }
    }
    ?>

....

profile.php

<?php
    ob_start();
    session_start();
    if (!(isset($_SESSION['can-id']))) {
        # code...
        header('location: ../login/');

    }
    else
    {
        $cid=$_SESSION['can-id'];
    }
?>

<?php echo $cid;?>

This is my code after log in the page was redirected to profile.php page but in profile page session variable doesn't printed I don't know why but this problem was not occuring every time I log in It occurs sometimes so I can't find what is the problem. Anyone knows  please help me to solve the problem.

Body:
Remove  ob_start() from your login.php
Don't put session_start() in all of your file

 e.g login.php, profile.php, etc

but instead, add this to your config.php for example:

<?php
 session_start();
 //.. config variables here

Then, include config.php also in your profile.php.


问:
xtrabackup does not write gtid information in xtrabackup_binlog_info
Software versions:

```<code>xtrabackup 8.0.12
percona-xtradb-cluster-server 8.0.18-9
</code>```

I am running xtrabackup with this options:

```<code>--defaults-file=/etc/mysql/my.cnf --backup --user=backup
--password=**** --parallel=4 --no-timestamp --target-dir=/my-backup-dir
</code>```

Some of server options:

```<code>binlog_format | ROW
gtid_mode     | ON_PERMISSIVE
enforce_gtid_consistency | ON
</code>```

File xtrabackup_binlog_info has only binlog file name and position:

```<code>mysql-bin.000159        251
</code>```

No GTID, so I can not create GTID-based replication restoring a slave from this backup.

What should I do to make xtrabackup include this information?

UPDATE:

Check if GTIDs are enabled:

```<code>show global variables like '%gtid%';

| Variable_name                    | Value                                                                                |
| binlog_gtid_simple_recovery      | ON                                                                                |
| enforce_gtid_consistency         | ON                                                                                |
| gtid_executed                    | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3399594,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |
| gtid_executed_compression_period | 1000                                                                                |
| gtid_mode                        | ON_PERMISSIVE                                                                                |
| gtid_owned                       |                                                                                |
| gtid_purged                      | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-2661056,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |
| session_track_gtids              | OFF                                                                                |
</code>```

```<code>show master status;

| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                                                                                |
| mysql-bin.000166 | 15285372 |              |                  | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3358798,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |

</code>```

```<code>(pavel.selivanov@localhost) [kassa_prod]> show binlog events in 'mysql-bin.000166' limit 7;

| Log_name         | Pos  | Event_type     | Server_id | End_log_pos | Info
| mysql-bin.000166 |    4 | Format_desc    |      4315 |         124 | Server ver: 8.0.18-9, Binlog ver: 4                                                                           |
| mysql-bin.000166 |  124 | Previous_gtids |      4315 |         251 | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3347159,
de211648-2642-ee18-628d-dc48283b005c:3697598:3877279-10141440 |
| mysql-bin.000166 |  251 | Gtid           |      4315 |         330 | SET @@SESSION.GTID_NEXT= 'c0e3de06-a2a6-11ea-913c-c7b046cf5782:3347160'                                       |
| mysql-bin.000166 |  330 | Query          |      4315 |         411 | BEGIN                                                                                |
| mysql-bin.000166 |  411 | Table_map      |      4315 |         499 | table_id: 150 (db.table)                                                                      |
| mysql-bin.000166 |  499 | Update_rows    |      4315 |        3475 | table_id: 150 flags: STMT_END_F                                                                               |
| mysql-bin.000166 | 3475 | Xid            |      4315 |        3506 | COMMIT /* xid=9611331 */
</code>```

答:
From the manual:

ON_PERMISSIVE: New transactions are GTID transactions. Replicated transactions can be either anonymous or GTID transactions.

Check with show global variables like 'gt%'; or in your binary logs, if you actually have GTID transactions.

You actually don't have anything special to do, to have xtrabackup include GTIDs in the xtrabackup_binlog_info file: How to create a new (or repair a broken) GTID based slave

Body:
Software versions:

xtrabackup 8.0.12
percona-xtradb-cluster-server 8.0.18-9

I am running xtrabackup with this options:

--defaults-file=/etc/mysql/my.cnf --backup --user=backup
--password=**** --parallel=4 --no-timestamp --target-dir=/my-backup-dir

Some of server options:

binlog_format | ROW
gtid_mode     | ON_PERMISSIVE
enforce_gtid_consistency | ON

File xtrabackup_binlog_info has only binlog file name and position:

mysql-bin.000159        251

No GTID, so I can not create GTID-based replication restoring a slave from this backup.

What should I do to make xtrabackup include this information?

UPDATE:

Check if GTIDs are enabled:

show global variables like '%gtid%';

| Variable_name                    | Value                                                                                |
| binlog_gtid_simple_recovery      | ON                                                                                |
| enforce_gtid_consistency         | ON                                                                                |
| gtid_executed                    | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3399594,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |
| gtid_executed_compression_period | 1000                                                                                |
| gtid_mode                        | ON_PERMISSIVE                                                                                |
| gtid_owned                       |                                                                                |
| gtid_purged                      | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-2661056,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |
| session_track_gtids              | OFF                                                                                |

show master status;

| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                                                                                |
| mysql-bin.000166 | 15285372 |              |                  | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3358798,
de211648-2642-ee18-628d-dc48283b005c:1-3697598:3877279-10141440 |

(pavel.selivanov@localhost) [kassa_prod]> show binlog events in 'mysql-bin.000166' limit 7;

| Log_name         | Pos  | Event_type     | Server_id | End_log_pos | Info
| mysql-bin.000166 |    4 | Format_desc    |      4315 |         124 | Server ver: 8.0.18-9, Binlog ver: 4                                                                           |
| mysql-bin.000166 |  124 | Previous_gtids |      4315 |         251 | c0e3de06-a2a6-11ea-913c-c7b046cf5782:1-3347159,
de211648-2642-ee18-628d-dc48283b005c:3697598:3877279-10141440 |
| mysql-bin.000166 |  251 | Gtid           |      4315 |         330 | SET @@SESSION.GTID_NEXT= 'c0e3de06-a2a6-11ea-913c-c7b046cf5782:3347160'                                       |
| mysql-bin.000166 |  330 | Query          |      4315 |         411 | BEGIN                                                                                |
| mysql-bin.000166 |  411 | Table_map      |      4315 |         499 | table_id: 150 (db.table)                                                                      |
| mysql-bin.000166 |  499 | Update_rows    |      4315 |        3475 | table_id: 150 flags: STMT_END_F                                                                               |
| mysql-bin.000166 | 3475 | Xid            |      4315 |        3506 | COMMIT /* xid=9611331 */


问:
Java recursively list the files from directory of specific pattern
I've below directory/file structure

```<code>ABC
  -- Apps
  -- Tests
     -- file1.xml
     -- file2.xml
  -- AggTests
  -- UnitTests
PQR
  -- Apps
  -- Tests
     -- file3.xml
     -- file4.xml
  -- AggTests
  -- UnitTests

</code>```

Here I just want to a List of Files from Tests directory. How can I achieve it in java, I found this is something helpful https://stackoverflow.com/a/24006711/1665592

Something below is listing down all XML files but I need it from specific directory called Tests?

```<code>try (Stream<Path> walk = Files.walk(Paths.get("C:\\projects"))) {

    List<String> fileList = walk.map(x -> x.toString())
            .filter(f -> f.endsWith(".xml")).collect(Collectors.toList());

    fileList.forEach(System.out::println);

} catch (IOException e) {
    e.printStackTrace();
}

</code>```

Ultimately, I need fileList = [file1.xml, file2.xml, file3.xml, file4.xml]

答:
```<code>List<String> fileList = walk.filter(x -> x.getParent().endsWith("Tests")).map(x -> x.toString())
                    .filter(f -> f.endsWith(".xml")).collect(Collectors.toList());
</code>```

if you just need the filenames, without the whole path, you could do:

```<code>List<String> fileList = walk.filter(x -> x.getParent().endsWith("Tests")).map(x -> x.getFileName().toString())
                    .filter(f -> f.endsWith(".xml")).collect(Collectors.toList());
</code>```

Body:
I've below directory/file structure

ABC
  -- Apps
  -- Tests
     -- file1.xml
     -- file2.xml
  -- AggTests
  -- UnitTests
PQR
  -- Apps
  -- Tests
     -- file3.xml
     -- file4.xml
  -- AggTests
  -- UnitTests

Here I just want to a List of Files from Tests directory. How can I achieve it in java, I found this is something helpful https://stackoverflow.com/a/24006711/1665592

Something below is listing down all XML files but I need it from specific directory called Tests?

try (Stream<Path> walk = Files.walk(Paths.get("C:\\projects"))) {

    List<String> fileList = walk.map(x -> x.toString())
            .filter(f -> f.endsWith(".xml")).collect(Collectors.toList());

    fileList.forEach(System.out::println);

} catch (IOException e) {
    e.printStackTrace();
}

Ultimately, I need fileList = [file1.xml, file2.xml, file3.xml, file4.xml]

Body:
List<String> fileList = walk.filter(x -> x.getParent().endsWith("Tests")).map(x -> x.toString())
                    .filter(f -> f.endsWith(".xml")).collect(Collectors.toList());

if you just need the filenames, without the whole path, you could do:

List<String> fileList = walk.filter(x -> x.getParent().endsWith("Tests")).map(x -> x.getFileName().toString())
                    .filter(f -> f.endsWith(".xml")).collect(Collectors.toList());


问:
PARTITION BY with TUMBLE causes exception in FlinkSQL
I would like to select last element of processing window using FlinkSQL. Tried to achieve that with ROW_NUMBER in Blink planner. Tried following query:

```<code>SELECT * FROM (
  SELECT key, value, ROW_NUMBER() OVER w AS rn
    FROM InputTable
  WINDOW w AS (PARTITION BY key, TUMBLE(rt, INTERVAL '15' MINUTE) ORDER BY -ts)
) WHERE rn = 1

// rt = ts.rowtime, ts is Long
</code>```

Unfortunately, this causes exception

```<code>org.apache.flink.table.planner.codegen.CodeGenException: Unsupported call: TUMBLE(TIMESTAMP(3) *ROWTIME*, INTERVAL SECOND(3) NOT NULL)
If you think this function should be supported, you can create an issue and start a discussion for it.
</code>```

Any idea what am I doing wrong? I was thinking about TUMBLE function as something "equivalent"  to calculating rowtime % interval.

答:
TUMBLE (as well as HOP and SESSION) are special built-in functions in Flink SQL (version 2.11) that can only be used in a GROUP BY clause. In principle you are right and it should be OK to use TUMBLE in this context, but it is simply not supported at this point.

You could implement a user-defined function to re-implement the grouping logic of TUMBLE however, I would not recommend that because the query would not perform well. Flink SQL would not be aware that a partition (PARTITION BY key, TUMBLE(rt, INTERVAL '15' MINUTE) would only be "active" for 35 minutes and keep its state forever. Hence, the query would accumulate more and more state over time which slows down checkpointing and recovery. IMO, such time-based OVER partitions should be supported in the future, but AFAIK they are not yet.

Body:
I would like to select last element of processing window using FlinkSQL. Tried to achieve that with ROW_NUMBER in Blink planner. Tried following query:

SELECT * FROM (
  SELECT key, value, ROW_NUMBER() OVER w AS rn
    FROM InputTable
  WINDOW w AS (PARTITION BY key, TUMBLE(rt, INTERVAL '15' MINUTE) ORDER BY -ts)
) WHERE rn = 1

// rt = ts.rowtime, ts is Long

Unfortunately, this causes exception

org.apache.flink.table.planner.codegen.CodeGenException: Unsupported call: TUMBLE(TIMESTAMP(3) *ROWTIME*, INTERVAL SECOND(3) NOT NULL)
If you think this function should be supported, you can create an issue and start a discussion for it.

Any idea what am I doing wrong? I was thinking about TUMBLE function as something "equivalent"  to calculating rowtime % interval.


问:
Process same file in two threads using ifstream
I have an input file in my application that contains a vast amount of information. Reading over it sequentially, and at only a single file offset at a time is not sufficient for my application's usage. Ideally, I'd like to have two threads, that have separate and distinct ifstreams reading from two unique file offsets of the same file. I can't just start one ifstream up, and then make a copy of it using its copy constructor (since its uncopyable). So, how do I handle this?

Immediately I can think of two ways,

Construct a new ifstream for the second thread, open it on the same file.

Share a single instance of an open ifstream across both threads (using for instance boost::shared_ptr<>). Seek to the appropriate file offset that current thread is currently interested in, when the thread gets a time slice.

Is one of these two methods preferred?

Is there a third (or fourth) option that I have not yet thought of?

Obviously I am ultimately limited by the hard drive having to spin back and forth, but what I am interested in taking advantage of (if possible), is some OS level disk caching at both file offsets simultaneously.

Thanks.

答:
Other option:

Memory-map the file, create as many memory istream objects as you want.  (istrstream is good for this, istringstream is not).


问:
How to get the data csv or raw numbers from cloud watch
I am monitoring Network In Average and Network Out Average by cloud watch.

I can check the graph on the web.

However I want to get the csv or raw numbers data.

I have checkws around but not found the some download way or showing raw numbers.

I am familiar with php/python or crawling, but not familiar with aws.

Where should I start ???

答:
Unfortunately, there is no build in functionality for that in console. Thus, the get the raw data, you have to use get_metric_data or get_metric_statistics methods in Python using boto3 (AWS SDK for Python).

Which one and how to use it is explained in a recent AWS blog post;

Should I use GetMetricData or GetMetricStatistics for CloudWatch metrics?

It's a best practice to use the GetMetricData API instead of GetMetricStatistics, because you can retrieve data faster at scale with GetMetricData.

Instead of writing your own program for that, you can have a look at already developed tools for that. For example:

cloudwatch_exporter


问:
Components with uppercase function names are rendered multiple times in React Hooks
I've been developing with Hooks for a long time, But I met a situation where I was confused.

Let me briefly describe this strange problem with a particular case.

I created a very simple functional component called Test:

// test.js
export default function Test () {

  useEffect(() => {
    console.log('Test')
  }, [])

  return (

      Test Component

  )
}
```

<p>It will print something at the first render</p>

<p>And the component will be referenced in <code>App</code></p>

<pre class="lang-js prettyprint-override"><code>// App.js
import React, { useEffect, useState } from "react";
import Test from './test';

export default function App() {
  const [number, updateNumber] = useState(0);

  const renderComponent = () => {
    return (
      <Test />
    )
  }

  const RenderComponent = () => {
    return (
      <Test />
    )
  }

  useEffect(() => {
    updateNumber(1)
    updateNumber(2)
  }, [])

  console.log('every-render')

  return (
    <div className="App">

      {renderComponent()}

      {/* <RenderComponent /> */}

      <h1>Rerender times: {number}</h1>
    </div>
  );
}
</code>```

So something strange happened

When I use this way, the Test component will be printed once

// Case 1
const renderComponent = () => {
    return (
      <Test />
    )
}

// ...

renderComponent()
```

<p>Something that it prints in Console</p>

<p>But when I use this way, the <code>Test</code> component will be printed twice</p>

<pre class="lang-js prettyprint-override"><code>// Case 2
const RenderComponent = () => {
    return (
      <Test />
    )
}

// ...

<RenderComponent />
</code>```

Something that it prints in Console

Of course, I wrote a demo for you to debug:
https://codesandbox.io/s/demo-for-hooks-8gkml?file=/src/App.js

I am confused why in case 2 the Test component will be remounted each time the Hooks are rerendered.

答:
Every render cycle the RenderComponent component is recreated, so it is mounted every render, and thus, mounts its children. Consider the following code where you render <Test /> directly, it's output is identical to {renderComponent()}.

```<code>export default function App() {
  const [number, updateNumber] = useState(0);

  const renderComponent = () => {
    return <Test />;
  };

  const RenderComponent = () => {
    return <Test />;
  };

  useEffect(() => {
    updateNumber(1);
    updateNumber(2);
  }, []);

  useEffect(() => {
    console.log("every-render");
  });

  return (
    <div className="App">
      {/* {renderComponent()} */}
      {/* <RenderComponent /> */}
      <Test /> // <-- "Mounted Test", "every-render", "every-render"

      <h1>Rerender times: {number}</h1>
    </div>
  );
}
</code>```

The function renderComponent is executed and the return value is <Test />, thus equivalent to just rendering <Test /> directly, whereas RenderComponent is a new component each cycle.

Body:
I've been developing with Hooks for a long time, But I met a situation where I was confused.

Let me briefly describe this strange problem with a particular case.

I created a very simple functional component called Test:

// test.js
export default function Test () {

  useEffect(() => {
    console.log('Test')
  }, [])

  return (

      Test Component

  )
}

It will print something at the first render

And the component will be referenced in App

// App.js
import React, { useEffect, useState } from "react";
import Test from './test';

export default function App() {
  const [number, updateNumber] = useState(0);

  const renderComponent = () => {
    return (
      <Test />
    )
  }

  const RenderComponent = () => {
    return (
      <Test />
    )
  }

  useEffect(() => {
    updateNumber(1)
    updateNumber(2)
  }, [])

  console.log('every-render')

  return (

      {renderComponent()}

      {/* <RenderComponent /> */}

Rerender times: {number}

  );
}

So something strange happened

When I use this way, the Test component will be printed once

// Case 1
const renderComponent = () => {
    return (
      <Test />
    )
}

// ...

renderComponent()

Something that it prints in Console

But when I use this way, the Test component will be printed twice

// Case 2
const RenderComponent = () => {
    return (
      <Test />
    )
}

// ...

<RenderComponent />

Something that it prints in Console

Of course, I wrote a demo for you to debug:
https://codesandbox.io/s/demo-for-hooks-8gkml?file=/src/App.js

I am confused why in case 2 the Test component will be remounted each time the Hooks are rerendered.

Body:
Every render cycle the RenderComponent component is recreated, so it is mounted every render, and thus, mounts its children. Consider the following code where you render <Test /> directly, it's output is identical to {renderComponent()}.

export default function App() {
  const [number, updateNumber] = useState(0);

  const renderComponent = () => {
    return <Test />;
  };

  const RenderComponent = () => {
    return <Test />;
  };

  useEffect(() => {
    updateNumber(1);
    updateNumber(2);
  }, []);

  useEffect(() => {
    console.log("every-render");
  });

  return (
    <div className="App">
      {/* {renderComponent()} */}
      {/* <RenderComponent /> */}
      <Test /> // <-- "Mounted Test", "every-render", "every-render"

      <h1>Rerender times: {number}</h1>
    </div>
  );
}

The function renderComponent is executed and the return value is <Test />, thus equivalent to just rendering <Test /> directly, whereas RenderComponent is a new component each cycle.


问:
how can it do other work except "async_wait" in one single thread using boost.asio.deadline_timer
I rewrite a boost.asio example "Timer.3 - Binding arguments to a handler". I think "worker" can work forever and "print" can be called each second. But "print" only is called once. It is very diffcult for me to think why.

If this can't work,how can I do other things except "async_wait" in one single thread.

```<code>#include <iostream>
#include <boost/asio.hpp>
#include <boost/bind.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>

int num = 0;
void worker(){
    while(1){
        ++num;
    }
}
void print(const boost::system::error_code& /*e*/,
    boost::asio::deadline_timer* t, int* count)
{
    std::cout << *count << "\n";
    ++(*count);

    t->expires_at(t->expires_at() + boost::posix_time::seconds(1));
    t->async_wait(boost::bind(
        print, boost::asio::placeholders::error, t, count
    ));
    if(*count == 1){
        worker();
    }
}

int main()
{
    boost::asio::io_service io;

    int count = 0;
    boost::asio::deadline_timer t(io, boost::posix_time::seconds(1));
    t.async_wait(boost::bind(print,
        boost::asio::placeholders::error, &t, &count
    ));
    io.run();

    std::cout << "Final count is " << count << "\n";

    return 0;
}
</code>```

答:
You have an infinite loop

```<code>void worker() {
    while(1) {
        ++num;
    }
}
</code>```

so control never returns the first time print() is invoked.

Body:
I rewrite a boost.asio example "Timer.3 - Binding arguments to a handler". I think "worker" can work forever and "print" can be called each second. But "print" only is called once. It is very diffcult for me to think why.

If this can't work,how can I do other things except "async_wait" in one single thread.

#include <iostream>
#include <boost/asio.hpp>
#include <boost/bind.hpp>
#include <boost/date_time/posix_time/posix_time.hpp>

int num = 0;
void worker(){
    while(1){
        ++num;
    }
}
void print(const boost::system::error_code& /*e*/,
    boost::asio::deadline_timer* t, int* count)
{
    std::cout << *count << "\n";
    ++(*count);

    t->expires_at(t->expires_at() + boost::posix_time::seconds(1));
    t->async_wait(boost::bind(
        print, boost::asio::placeholders::error, t, count
    ));
    if(*count == 1){
        worker();
    }
}

int main()
{
    boost::asio::io_service io;

    int count = 0;
    boost::asio::deadline_timer t(io, boost::posix_time::seconds(1));
    t.async_wait(boost::bind(print,
        boost::asio::placeholders::error, &t, &count
    ));
    io.run();

    std::cout << "Final count is " << count << "\n";

    return 0;
}

Body:
You have an infinite loop

void worker() {
    while(1) {
        ++num;
    }
}

so control never returns the first time print() is invoked.


问:
Update/set state dynamically in reactjs
I am new to reactjs and used to load/update data through javascript/jquery.

In my example I've got 3 buttons. Whenever I click one one button, I would like to update the text above. Right now, I am using same state for the counter. Can't you somehow get the current divs ID - and update the state inside it?

I could just change the state to 3 different states.
But the reason why I am not doing this is if I ever want to load something dynamically they probably have the same state?

jsfiddle:
https://jsfiddle.net/t9am76fs/1/

```<code>constructor(props) {
    super(props);

    this.state = {
        count: 0,
    };
}
incrementCounter = (value) => {
    this.setState({ count: this.state.count + value })
};

render() {
    return (
        <div>
            <div id='div1'>
                  <p>You clicked {this.state.count} times</p>
                  <button onClick={() => this.incrementCounter(1)}>+1</button>
            </div>

            <div id='div2'>
                  <p>You clicked {this.state.count} times</p>
                  <button onClick={() => this.incrementCounter(1)}>+1</button>
            </div>

            <div id='div3'>
                  <p>You clicked {this.state.count} times</p>
                  <button onClick={() => this.incrementCounter(1)}>+1</button>
            </div>
        </div>
    );
}
</code>```

Thanks

答:
Try it:

class Counter extends React.Component {
    constructor(props) {
        super(props);

        this.state = {
            count1: 0,
            count2: 0,
            count3: 0,
        };
    }
    incrementCounter = (type) => {
        this.setState({ [type]: this.state[type] + 1 })
    };

    render() {
        return (

You clicked {this.state.count1} times

                      <button onClick={() => this.incrementCounter('count1')}>+1</button>

You clicked {this.state.count2} times

                      <button onClick={() => this.incrementCounter('count2')}>+1</button>

You clicked {this.state.count3} times

                      <button onClick={() => this.incrementCounter('count3')}>+1</button>

        );
    }
}

ReactDOM.render(<Counter />, document.querySelector("#app")); ```

Title: Update/set state dynamically in reactjs

Body:
I am new to reactjs and used to load/update data through javascript/jquery.

In my example I've got 3 buttons. Whenever I click one one button, I would like to update the text above. Right now, I am using same state for the counter. Can't you somehow get the current divs ID - and update the state inside it?

I could just change the state to 3 different states.
But the reason why I am not doing this is if I ever want to load something dynamically they probably have the same state?

jsfiddle:
https://jsfiddle.net/t9am76fs/1/

constructor(props) {
    super(props);

    this.state = {
        count: 0,
    };
}
incrementCounter = (value) => {
    this.setState({ count: this.state.count + value })
};

render() {
    return (

You clicked {this.state.count} times

                  <button onClick={() => this.incrementCounter(1)}>+1</button>

You clicked {this.state.count} times

                  <button onClick={() => this.incrementCounter(1)}>+1</button>

You clicked {this.state.count} times

                  <button onClick={() => this.incrementCounter(1)}>+1</button>

    );
}

Thanks

Body:
Try it:

class Counter extends React.Component {
    constructor(props) {
        super(props);

        this.state = {
            count1: 0,
            count2: 0,
            count3: 0,
        };
    }
    incrementCounter = (type) => {
        this.setState({ [type]: this.state[type] + 1 })
    };

    render() {
        return (
            <div>
                <div id='div1'>
                      <p>You clicked {this.state.count1} times</p>
                      <button onClick={() => this.incrementCounter('count1')}>+1</button>
                </div>

                <div id='div2'>
                      <p>You clicked {this.state.count2} times</p>
                      <button onClick={() => this.incrementCounter('count2')}>+1</button>
                </div>

                <div id='div3'>
                      <p>You clicked {this.state.count3} times</p>
                      <button onClick={() => this.incrementCounter('count3')}>+1</button>
                </div>
            </div>
        );
    }
}

ReactDOM.render(<Counter />, document.querySelector("#app"));


问:
Find whole word that starts at character position in R
```<code>Str <- "I love chocolate pudding"
pos <- 8
</code>```

I need to return the word that starts with the letter c at pos 8, which is chocolate. How can I do that?

答:
Using stringr and ignoring the 'starting with 'c' condition:

Str %>%
  str_sub(pos) %>%
  word(1)
```

Body:
Str <- "I love chocolate pudding"
pos <- 8

I need to return the word that starts with the letter c at pos 8, which is chocolate. How can I do that?