问:
Validate single record on continuous form
So I have a form and a table called Variables.  The table is simply the fields VarID, VarDescription, and VarValue.  It's only three items that are all network locations of things.  The VarValue is the only thing that can be changed through the form, and thus it is the only thing validated.  I validate those records on the form with the Before Update Event in the control by using logic such as:

```<code>    If Me.VarID = 1 Then
        If Me.Tex like "*:\* Then....
        End If
        If GetAttr(Me.Tex) = vbDirectory Then
        End
    End If

    If Me.VarID = 2 Then
        If Me.Tex like "*:\* Then....
        End If
        If GetAttr(Me.Tex) = vbNormal Or GetAttr(Me.Tex) = vbArchive Then
        End If
    End If
</code>```

This all works great.  However my issue comes in when multiple locations become invalid at once.  I get stuck in one cell because the other VarValue's are invalid as well.  How can I validate only the cell I have changed?  I tried playing around with various Dirty and Focus events/methods, but those seem to be form specific, not cell specific.

答: This seems to just be a reference issue as I am looking on the database on two completely separate networks.  The original uses all Office 16.0 libraries whereas the instance I was having the issue with was using Office 15.0 libraries.  I played around with the original and it works fine, even if all the locations become bad at the same time.</p>

Body:
So I have a form and a table called Variables.  The table is simply the fields VarID, VarDescription, and VarValue.  It's only three items that are all network locations of things.  The VarValue is the only thing that can be changed through the form, and thus it is the only thing validated.  I validate those records on the form with the Before Update Event in the control by using logic such as:

    If Me.VarID = 1 Then
        If Me.Tex like "*:\* Then....
        End If
        If GetAttr(Me.Tex) = vbDirectory Then
        End
    End If

    If Me.VarID = 2 Then
        If Me.Tex like "*:\* Then....
        End If
        If GetAttr(Me.Tex) = vbNormal Or GetAttr(Me.Tex) = vbArchive Then
        End If
    End If

This all works great.  However my issue comes in when multiple locations become invalid at once.  I get stuck in one cell because the other VarValue's are invalid as well.  How can I validate only the cell I have changed?  I tried playing around with various Dirty and Focus events/methods, but those seem to be form specific, not cell specific.


来源: stackexchange

问:
How can I divide single values of a dataframe by monthly averages?
I have the following 15 minute data as a dataframe for 3 years. With the first two columns being the index.

```<code>2014-01-01 00:15:00  1269.6
2014-01-01 00:30:00  1161.6
2014-01-01 00:45:00  1466.4
2014-01-01 01:00:00  1365.6
2014-01-01 01:15:00  1362.6
2014-01-01 01:30:00  1064.0
2014-01-01 01:45:00  1171.2
2014-01-01 02:00:00  1171.0
2014-01-01 02:15:00  1330.4
2014-01-01 02:30:00  1309.6
2014-01-01 02:45:00  1308.4
2014-01-01 03:00:00  1494.0
</code>```

I have used resample to get a second series with monthly averages.

```<code>data_Monthly = data.resample('1M', how='mean')
</code>```

How can I divide the values in the last column by their monthly average with the result being still a time series on 15 minute granularity?

答:
I think it is generally recommended to use Grouper instead of TimeGrouper. Have a look at this.
For example, if your column is called Date, use </p>

```<code>grouper = pd.Grouper(key='Date', freq='M')
</code>```

<p>instead of using TimeGrouper and then continue as @Zelazny7 suggested.
If your column is not a datetime index then use </p>

```<code>df['Date'] = pd.to_datetime(df['Date'])
</code>```

ContentLicense: CC BY-SA 3.0

Body:
I have the following 15 minute data as a dataframe for 3 years. With the first two columns being the index.

2014-01-01 00:15:00  1269.6
2014-01-01 00:30:00  1161.6
2014-01-01 00:45:00  1466.4
2014-01-01 01:00:00  1365.6
2014-01-01 01:15:00  1362.6
2014-01-01 01:30:00  1064.0
2014-01-01 01:45:00  1171.2
2014-01-01 02:00:00  1171.0
2014-01-01 02:15:00  1330.4
2014-01-01 02:30:00  1309.6
2014-01-01 02:45:00  1308.4
2014-01-01 03:00:00  1494.0

I have used resample to get a second series with monthly averages.

data_Monthly = data.resample('1M', how='mean')

How can I divide the values in the last column by their monthly average with the result being still a time series on 15 minute granularity?

Body:
I think it is generally recommended to use Grouper instead of TimeGrouper. Have a look at this.
For example, if your column is called Date, use

grouper = pd.Grouper(key='Date', freq='M')

instead of using TimeGrouper and then continue as @Zelazny7 suggested.
If your column is not a datetime index then use

df['Date'] = pd.to_datetime(df['Date'])


问:
Reading multiple txt files and saving them into one numpy array: how to concatenate numpy arrays
a) Here's my try. It works but I'm wondering if I can do it without if else conditions and indexing - more pythonic way.   </p>

```<code>sensor_files = ['file1.csv', 'file2.csv']

for idx,sensor_file in enumerate(sensor_files):
    with open(file, 'rb') as f:
        clean_lines = (line.replace(b';',b',')
        sensors = np.genfromtxt(clean_lines, dtype=int, delimiter=',')
        if idx == 0:
            acc_measurements = sensors
        else:
            acc_measurements = np.concatenate((acc_measurements, sensors))
</code>```

答:
A list append version:

```<code>alist = []
for sensor_file in sensor_files:
    with open(sensor_file, 'rb') as f:
        clean_lines = (line.replace(b';',b',')
        sensors = np.genfromtxt(clean_lines, dtype=int, delimiter=',')
        alist.append(sensors)
acc_measurements = np.concatenate(alist)
</code>```

genfromtxt does something similar.  It iterates on file lines, parses each and appends it to a list.  Then at the end it converts that list of lists into an array with the correct dtype.

Each concatenate creates a new array, copying data from the arguments.  List append operates 'in-place' adding just a pointer to the new source.  Lists are designed for fast appends.

Body:
a) Here's my try. It works but I'm wondering if I can do it without if else conditions and indexing - more pythonic way.

sensor_files = ['file1.csv', 'file2.csv']

for idx,sensor_file in enumerate(sensor_files):
    with open(file, 'rb') as f:
        clean_lines = (line.replace(b';',b',')
        sensors = np.genfromtxt(clean_lines, dtype=int, delimiter=',')
        if idx == 0:
            acc_measurements = sensors
        else:
            acc_measurements = np.concatenate((acc_measurements, sensors))

Body:
A list append version:

alist = []
for sensor_file in sensor_files:
    with open(sensor_file, 'rb') as f:
        clean_lines = (line.replace(b';',b',')
        sensors = np.genfromtxt(clean_lines, dtype=int, delimiter=',')
        alist.append(sensors)
acc_measurements = np.concatenate(alist)

genfromtxt does something similar.  It iterates on file lines, parses each and appends it to a list.  Then at the end it converts that list of lists into an array with the correct dtype.

Each concatenate creates a new array, copying data from the arguments.  List append operates 'in-place' adding just a pointer to the new source.  Lists are designed for fast appends.


问:
How to ignore SSL policy to perform a HTTPClient request?
I'm trying to ignore ssl policy to do a request with HttpClient but ServerCertificateValidationCallback is never called. What is the correct place to invoke ServerCertificateValidationCallback in ASP.NET Core App? I put it before HttpClient using.

Note: This class is in net462 Class Library.

Code:

```<code> System.Net.ServicePointManager.ServerCertificateValidationCallback +=
 delegate (object sender,
 System.Security.Cryptography.X509Certificates.X509Certificate certificate,
 System.Security.Cryptography.X509Certificates.X509Chain chain,
 System.Net.Security.SslPolicyErrors sslPolicyErrors)
 {
     return true;
 };

 using (var client = new HttpClient())
 {
    client.BaseAddress = new Uri("https://myapi");
    ...

    var result = client.PostAsync(method, httpContent).Result;
    ...
 }
</code>```

答:
Consider trying it like this</p>

```<code>//Handle TLS protocols
System.Net.ServicePointManager.SecurityProtocol =
    System.Net.SecurityProtocolType.Tls
    | System.Net.SecurityProtocolType.Tls11
    | System.Net.SecurityProtocolType.Tls12;

var client = new HttpClient();
client.BaseAddress = new Uri("https://myapi");
//...

var result = await client.PostAsync(method, httpContent);
</code>```

Body:
I'm trying to ignore ssl policy to do a request with HttpClient but ServerCertificateValidationCallback is never called. What is the correct place to invoke ServerCertificateValidationCallback in ASP.NET Core App? I put it before HttpClient using.

Note: This class is in net462 Class Library.

Code:

 System.Net.ServicePointManager.ServerCertificateValidationCallback +=
 delegate (object sender,
 System.Security.Cryptography.X509Certificates.X509Certificate certificate,
 System.Security.Cryptography.X509Certificates.X509Chain chain,
 System.Net.Security.SslPolicyErrors sslPolicyErrors)
 {
     return true;
 };

 using (var client = new HttpClient())
 {
    client.BaseAddress = new Uri("https://myapi");
    ...

    var result = client.PostAsync(method, httpContent).Result;
    ...
 }

Body:
Consider trying it like this

//Handle TLS protocols
System.Net.ServicePointManager.SecurityProtocol =
    System.Net.SecurityProtocolType.Tls
    | System.Net.SecurityProtocolType.Tls11
    | System.Net.SecurityProtocolType.Tls12;

var client = new HttpClient();
client.BaseAddress = new Uri("https://myapi");
//...

var result = await client.PostAsync(method, httpContent);


问:
Echo mysql data using group_concat in PHP
This is my SQL Fiddle: SQL Fiddle

I am trying to echo this result using PHP. This is my expected result:

```<code>// Some Stuff Here

The Dark Knight Rises -  7.5
Batman Begins - 7.5
Iron Man - 7.3
The Lord of the Rings: The Return of the King - 8.1
etc...

// Some more Stuff here
</code>```

My PHP Code:

```<code>$stmt = $conn->prepare("SELECT tmdb_movies.movie_title
,GROUP_CONCAT(recommendations.recommendations_title  ORDER BY recommendations.recommendations_title) as recommendations_title
,GROUP_CONCAT(recommendations.recommendations_vote_average ORDER BY recommendations.recommendations_title) as recommendations_vote_average

FROM tmdb_movies

LEFT JOIN cast ON cast.cast_tmdb_id=tmdb_movies.tmdb_id
LEFT JOIN recommendations ON recommendations.recommendations_tmdb_id=tmdb_movies.tmdb_id

Where tmdb_movies.tmdb_id= 155

GROUP BY tmdb_movies.movie_title
 ");

     // Then fire it up
     $stmt->execute();
     // Pick up the result as an array
     $result = $stmt->fetchAll();

    // Now you run through this array in many ways, for example
     for($x=0, $n=count($result); $x < $n; $x++){
$recommendations_name_list = explode(',',$result[$x]["recommendations_title"]);
$recommendations_vote_average = explode(',',$result[$x]["recommendations_vote_average"]);
foreach( $recommendations_name_list as $index => $recommendations_title ) {
           echo'
    <p>'.$recommendations_title.'- '.$recommendations_vote_average[$index].'</p>';
    }
}
</code>```

Output of this code:

```<code>Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Lord of th- 8
</code>```

But If i remove this line LEFT JOIN cast ON cast.cast_tmdb_id=tmdb_movies.tmdb_id
 then, my php code works fine.

Expected Result:

```<code>Batman Begins- 7.5

Captain America: The First Avenger- 6.6

Inception- 8

Iron Man- 7.3

Iron Man 2- 6.6

The Dark Knight Rises- 7.5

The Lord of the Rings: The Fellowship of the Ring- 8

The Lord of the Rings: The Return of the King- 8.1

The Lord of the Rings: The Two Towers- 7.9

The Matrix- 7.9
</code>```

答:
The use of the   LEFT JOIN cast

is unuseful for the select values (the results are select ayway because are based on maun table movie_title
adn don't return useful info because there are no columns from di table in select clause ) and  produce  a resulting set with more  rows

so if you want insist on thi left join you can add DISTINCT to your originale query

```<code> SELECT DISTICNT
  tmdb_movies.movie_title
  ,GROUP_CONCAT(recommendations.recommendations_title
        ORDER BY recommendations.recommendations_title) as recommendations_title
  ,GROUP_CONCAT(recommendations.recommendations_vote_average
      ORDER BY recommendations.recommendations_title) as recommendations_vote_average

FROM tmdb_movies

LEFT JOIN cast ON cast.cast_tmdb_id=tmdb_movies.tmdb_id
LEFT JOIN recommendations
    ON recommendations.recommendations_tmdb_id=tmdb_movies.tmdb_id
Where tmdb_movies.tmdb_id= 155
GROUP BY tmdb_movies.movie_title
</code>```

or better don use left join pn cast

```<code>   SELECT
    tmdb_movies.movie_title
    ,GROUP_CONCAT(recommendations.recommendations_title
          ORDER BY recommendations.recommendations_title) as recommendations_title
    ,GROUP_CONCAT(recommendations.recommendations_vote_average
        ORDER BY recommendations.recommendations_title) as recommendations_vote_average

  FROM tmdb_movies

  LEFT JOIN recommendations
      ON recommendations.recommendations_tmdb_id=tmdb_movies.tmdb_id
  Where tmdb_movies.tmdb_id= 155
  GROUP BY tmdb_movies.movie_title
</code>```

Body:
This is my SQL Fiddle: SQL Fiddle

I am trying to echo this result using PHP. This is my expected result:

// Some Stuff Here

The Dark Knight Rises -  7.5
Batman Begins - 7.5
Iron Man - 7.3
The Lord of the Rings: The Return of the King - 8.1
etc...

// Some more Stuff here

My PHP Code:

$stmt = $conn->prepare("SELECT tmdb_movies.movie_title
,GROUP_CONCAT(recommendations.recommendations_title  ORDER BY recommendations.recommendations_title) as recommendations_title
,GROUP_CONCAT(recommendations.recommendations_vote_average ORDER BY recommendations.recommendations_title) as recommendations_vote_average

FROM tmdb_movies

LEFT JOIN cast ON cast.cast_tmdb_id=tmdb_movies.tmdb_id
LEFT JOIN recommendations ON recommendations.recommendations_tmdb_id=tmdb_movies.tmdb_id

Where tmdb_movies.tmdb_id= 155

GROUP BY tmdb_movies.movie_title
 ");

     // Then fire it up
     $stmt->execute();
     // Pick up the result as an array
     $result = $stmt->fetchAll();

    // Now you run through this array in many ways, for example
     for($x=0, $n=count($result); $x < $n; $x++){
$recommendations_name_list = explode(',',$result[$x]["recommendations_title"]);
$recommendations_vote_average = explode(',',$result[$x]["recommendations_vote_average"]);
foreach( $recommendations_name_list as $index => $recommendations_title ) {
           echo'

'.$recommendations_title.'- '.$recommendations_vote_average[$index].'
';
    }
}

Output of this code:

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Batman Begins- 7.5

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Captain America: The First Avenger- 6.6

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Inception- 8

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man- 7.3

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

Iron Man 2- 6.6

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Dark Knight Rises- 7.5

The Lord of th- 8

But If i remove this line LEFT JOIN cast ON cast.cast_tmdb_id=tmdb_movies.tmdb_id
 then, my php code works fine.

Expected Result:

Batman Begins- 7.5

Captain America: The First Avenger- 6.6

Inception- 8

Iron Man- 7.3

Iron Man 2- 6.6

The Dark Knight Rises- 7.5

The Lord of the Rings: The Fellowship of the Ring- 8

The Lord of the Rings: The Return of the King- 8.1

The Lord of the Rings: The Two Towers- 7.9

The Matrix- 7.9

Body:
The use of the   LEFT JOIN cast

is unuseful for the select values (the results are select ayway because are based on maun table movie_title
adn don't return useful info because there are no columns from di table in select clause ) and  produce  a resulting set with more  rows

so if you want insist on thi left join you can add DISTINCT to your originale query

 SELECT DISTICNT
  tmdb_movies.movie_title
  ,GROUP_CONCAT(recommendations.recommendations_title
        ORDER BY recommendations.recommendations_title) as recommendations_title
  ,GROUP_CONCAT(recommendations.recommendations_vote_average
      ORDER BY recommendations.recommendations_title) as recommendations_vote_average

FROM tmdb_movies

LEFT JOIN cast ON cast.cast_tmdb_id=tmdb_movies.tmdb_id
LEFT JOIN recommendations
    ON recommendations.recommendations_tmdb_id=tmdb_movies.tmdb_id
Where tmdb_movies.tmdb_id= 155
GROUP BY tmdb_movies.movie_title

or better don use left join pn cast

   SELECT
    tmdb_movies.movie_title
    ,GROUP_CONCAT(recommendations.recommendations_title
          ORDER BY recommendations.recommendations_title) as recommendations_title
    ,GROUP_CONCAT(recommendations.recommendations_vote_average
        ORDER BY recommendations.recommendations_title) as recommendations_vote_average

  FROM tmdb_movies

  LEFT JOIN recommendations
      ON recommendations.recommendations_tmdb_id=tmdb_movies.tmdb_id
  Where tmdb_movies.tmdb_id= 155
  GROUP BY tmdb_movies.movie_title


问:
Broadcast Hash Join (BHJ) in Spark for full outer join (outer, full, fullouter)
How do I force full outer join for Dataframes in spark to use Boradcast Hash Join? Here is the code snippet:

```<code>sparkConfiguration.set("spark.sql.autoBroadcastJoinThreshold", "1000000000")
val Result = BigTable.join(
  org.apache.spark.sql.functions.broadcast(SmallTable),
  Seq("X", "Y", "Z", "W", "V"),
  "outer"
)
</code>```

My SmallTable's size is way smaller than the autoBroadcastJoinThreshold as specified above. Also, if I use inner, left_outer, or right_outer join, I see from the DAG visualization that the join is using BroadcastHashJoin as expected.

However, when I use "outer" as the type of join, spark decides to use SortMergeJoin for some unknown reason. Does anyone know how to solve this problem? Based on the performance I see with left outer join, BroadcastHashJoin would help speed up my application multiple-folds.

答:
spark decides to use SortMergeJoin for some unknown reason. Does
  anyone know how to solve this problem?

Reason : FullOuter (mean any keyword outer, full, fullouter) wont support Broadcast hash join(aka map side join)

How to prove this ?

lets have one example :

```
package com.examples

import org.apache.log4j.{Level, Logger}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

/**
  * Join Example and some basics demonstration using sample data.
  *
  * @author : Ram Ghadiyaram
  */
object JoinExamples extends Logging {
  // switch off  un necessary logs
  Logger.getLogger("org").setLevel(Level.OFF)
   val spark: SparkSession = SparkSession.builder.config("spark.master", "local").getOrCreate;
  case class Person(name: String, age: Int, personid: Int)

  case class Profile(name: String, personId: Int, profileDescription: String)

  /**
    * main
    *
    * @param args Array[String]
    */
  def main(args: Array[String]): Unit = {
    spark.conf.set("spark.sql.join.preferSortMergeJoin", "false")
    import spark.implicits._

    spark.sparkContext.getConf.getAllWithPrefix("spark.sql").foreach(x => logInfo(x.toString()))
    /**
      * create 2 dataframes here using case classes one is Person df1 and another one is profile df2
      */
    val df1 = spark.sqlContext.createDataFrame(
      spark.sparkContext.parallelize(
        Person("Sarath", 33, 2)
          :: Person("KangarooWest", 30, 2)
          :: Person("Ravikumar Ramasamy", 34, 5)
          :: Person("Ram Ghadiyaram", 42, 9)
          :: Person("Ravi chandra Kancharla", 43, 9)
          :: Nil))

    val df2 = spark.sqlContext.createDataFrame(
      Profile("Spark", 2, "SparkSQLMaster")
        :: Profile("Spark", 5, "SparkGuru")
        :: Profile("Spark", 9, "DevHunter")
        :: Nil
    )

    // you can do alias to refer column name with aliases to  increase readablity

    val df_asPerson = df1.as("dfperson")
    val df_asProfile = df2.as("dfprofile")
    /** *
      * Example displays how to join them in the dataframe level
      * next example demonstrates using sql with createOrReplaceTempView
      */
    val joined_df = df_asPerson.join(
      broadcast(df_asProfile)
      , col("dfperson.personid") === col("dfprofile.personid")
      , "outer")
    val joined = joined_df.select(
      col("dfperson.name")
      , col("dfperson.age")
      , col("dfprofile.name")
      , col("dfprofile.profileDescription"))
    joined.explain(false) // it will show which join was used
    joined.show

  }

}
```

I tried to use broadcast hint for fullouter join but framework is ignoring and its taking SortMergeJoin below is the explain plan for this.
Result :

```
== Physical Plan ==
*Project [name#4, age#5, name#11, profileDescription#13]
+- SortMergeJoin [personid#6], [personid#12], FullOuter
   :- *Sort [personid#6 ASC NULLS FIRST], false, 0
   :  +- Exchange hashpartitioning(personid#6, 200)
   :     +- *SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, assertnotnull(input[0, com.examples.JoinExamples$Person, true]).name, true) AS name#4, assertnotnull(input[0, com.examples.JoinExamples$Person, true]).age AS age#5, assertnotnull(input[0, com.examples.JoinExamples$Person, true]).personid AS personid#6]
   :        +- Scan ExternalRDDScan[obj#3]
   +- *Sort [personid#12 ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(personid#12, 200)
         +- LocalTableScan [name#11, personId#12, profileDescription#13]
+--------------------+---+-----+------------------+
|                name|age| name|profileDescription|
+--------------------+---+-----+------------------+
|  Ravikumar Ramasamy| 34|Spark|         SparkGuru|
|      Ram Ghadiyaram| 42|Spark|         DevHunter|
|Ravi chandra Kanc...| 43|Spark|         DevHunter|
|              Sarath| 33|Spark|    SparkSQLMaster|
|        KangarooWest| 30|Spark|    SparkSQLMaster|
+--------------------+---+-----+------------------+
```

From spark 2.3 Merge-Sort join is the default join algorithm in spark.
  However, this can be turned down by using the internal parameter
  ‘spark.sql.join.preferSortMergeJoin’ which by default is true.

Other case except fullouter join ... If you dont want spark to use sortmergejoin in any case you can set the below property.

```<code>sparkSession.conf.set("spark.sql.join.preferSortMergeJoin", "false")
</code>```

this is instructions to code SparkStrategies.scala (which is responsible & Converts a logical plan into zero or more SparkPlans) that you don't want to use sortmergejoin.

Imp. Note :

This property spark.sql.join.preferSortMergeJoin When true, prefer sort merge join over shuffle hash join by this PREFER_SORTMERGEJOIN property.

Setting false means spark cant only select broadcasthashjoin it can be anything else also(for example shuffle hash join).

The below doc is in SparkStrategies.scala i.e. on top of object JoinSelection extends Strategy with PredicateHelper ...

Broadcast: if one side of the join has an estimated physical size that is smaller than the
    user-configurable [[SQLConf.AUTO_BROADCASTJOIN_THRESHOLD]] threshold
    or if that side has an explicit broadcast hint (e.g. the user applied the
    [[org.apache.spark.sql.functions.broadcast()]] function to a DataFrame), then that side
    of the join will be broadcasted and the other side will be streamed, with no shuffling
   performed. If both sides of the join are eligible to be broadcasted then the

Shuffle hash join: if the average size of a single partition is small enough to build a hash table.

Sort merge: if the matching join keys are sortable.

ContentLicense: CC BY-SA 4.0

Body:
How do I force full outer join for Dataframes in spark to use Boradcast Hash Join? Here is the code snippet:

sparkConfiguration.set("spark.sql.autoBroadcastJoinThreshold", "1000000000")
val Result = BigTable.join(
  org.apache.spark.sql.functions.broadcast(SmallTable),
  Seq("X", "Y", "Z", "W", "V"),
  "outer"
)

My SmallTable's size is way smaller than the autoBroadcastJoinThreshold as specified above. Also, if I use inner, left_outer, or right_outer join, I see from the DAG visualization that the join is using BroadcastHashJoin as expected.

However, when I use "outer" as the type of join, spark decides to use SortMergeJoin for some unknown reason. Does anyone know how to solve this problem? Based on the performance I see with left outer join, BroadcastHashJoin would help speed up my application multiple-folds.

Body:
spark decides to use SortMergeJoin for some unknown reason. Does
  anyone know how to solve this problem?

Reason : FullOuter (mean any keyword outer, full, fullouter) wont support Broadcast hash join(aka map side join)

How to prove this ?

lets have one example :

package com.examples

import org.apache.log4j.{Level, Logger}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._

/**
  * Join Example and some basics demonstration using sample data.
  *
  * @author : Ram Ghadiyaram
  */
object JoinExamples extends Logging {
  // switch off  un necessary logs
  Logger.getLogger("org").setLevel(Level.OFF)
   val spark: SparkSession = SparkSession.builder.config("spark.master", "local").getOrCreate;
  case class Person(name: String, age: Int, personid: Int)

  case class Profile(name: String, personId: Int, profileDescription: String)

  /**
    * main
    *
    * @param args Array[String]
    */
  def main(args: Array[String]): Unit = {
    spark.conf.set("spark.sql.join.preferSortMergeJoin", "false")
    import spark.implicits._

    spark.sparkContext.getConf.getAllWithPrefix("spark.sql").foreach(x => logInfo(x.toString()))
    /**
      * create 2 dataframes here using case classes one is Person df1 and another one is profile df2
      */
    val df1 = spark.sqlContext.createDataFrame(
      spark.sparkContext.parallelize(
        Person("Sarath", 33, 2)
          :: Person("KangarooWest", 30, 2)
          :: Person("Ravikumar Ramasamy", 34, 5)
          :: Person("Ram Ghadiyaram", 42, 9)
          :: Person("Ravi chandra Kancharla", 43, 9)
          :: Nil))

    val df2 = spark.sqlContext.createDataFrame(
      Profile("Spark", 2, "SparkSQLMaster")
        :: Profile("Spark", 5, "SparkGuru")
        :: Profile("Spark", 9, "DevHunter")
        :: Nil
    )

    // you can do alias to refer column name with aliases to  increase readablity

    val df_asPerson = df1.as("dfperson")
    val df_asProfile = df2.as("dfprofile")
    /** *
      * Example displays how to join them in the dataframe level
      * next example demonstrates using sql with createOrReplaceTempView
      */
    val joined_df = df_asPerson.join(
      broadcast(df_asProfile)
      , col("dfperson.personid") === col("dfprofile.personid")
      , "outer")
    val joined = joined_df.select(
      col("dfperson.name")
      , col("dfperson.age")
      , col("dfprofile.name")
      , col("dfprofile.profileDescription"))
    joined.explain(false) // it will show which join was used
    joined.show

  }

}

I tried to use broadcast hint for fullouter join but framework is ignoring and its taking SortMergeJoin below is the explain plan for this.
Result :

== Physical Plan ==
*Project [name#4, age#5, name#11, profileDescription#13]
+- SortMergeJoin [personid#6], [personid#12], FullOuter
   :- *Sort [personid#6 ASC NULLS FIRST], false, 0
   :  +- Exchange hashpartitioning(personid#6, 200)
   :     +- *SerializeFromObject [staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, assertnotnull(input[0, com.examples.JoinExamples$Person, true]).name, true) AS name#4, assertnotnull(input[0, com.examples.JoinExamples$Person, true]).age AS age#5, assertnotnull(input[0, com.examples.JoinExamples$Person, true]).personid AS personid#6]
   :        +- Scan ExternalRDDScan[obj#3]
   +- *Sort [personid#12 ASC NULLS FIRST], false, 0
      +- Exchange hashpartitioning(personid#12, 200)
         +- LocalTableScan [name#11, personId#12, profileDescription#13]
+--------------------+---+-----+------------------+
|                name|age| name|profileDescription|
+--------------------+---+-----+------------------+
|  Ravikumar Ramasamy| 34|Spark|         SparkGuru|
|      Ram Ghadiyaram| 42|Spark|         DevHunter|
|Ravi chandra Kanc...| 43|Spark|         DevHunter|
|              Sarath| 33|Spark|    SparkSQLMaster|
|        KangarooWest| 30|Spark|    SparkSQLMaster|
+--------------------+---+-----+------------------+

From spark 2.3 Merge-Sort join is the default join algorithm in spark.
  However, this can be turned down by using the internal parameter
  ‘spark.sql.join.preferSortMergeJoin’ which by default is true.

Other case except fullouter join ... If you dont want spark to use sortmergejoin in any case you can set the below property.

sparkSession.conf.set("spark.sql.join.preferSortMergeJoin", "false")

this is instructions to code SparkStrategies.scala (which is responsible & Converts a logical plan into zero or more SparkPlans) that you don't want to use sortmergejoin.

Imp. Note :

This property spark.sql.join.preferSortMergeJoin When true, prefer sort merge join over shuffle hash join by this PREFER_SORTMERGEJOIN property.

Setting false means spark cant only select broadcasthashjoin it can be anything else also(for example shuffle hash join).

The below doc is in SparkStrategies.scala i.e. on top of object JoinSelection extends Strategy with PredicateHelper ...

Broadcast: if one side of the join has an estimated physical size that is smaller than the
    user-configurable [[SQLConf.AUTO_BROADCASTJOIN_THRESHOLD]] threshold
    or if that side has an explicit broadcast hint (e.g. the user applied the
    [[org.apache.spark.sql.functions.broadcast()]] function to a DataFrame), then that side
    of the join will be broadcasted and the other side will be streamed, with no shuffling
   performed. If both sides of the join are eligible to be broadcasted then the

Shuffle hash join: if the average size of a single partition is small enough to build a hash table.

Sort merge: if the matching join keys are sortable.


问:
Geokit Rails distance and find(:all)
I have 2 questions regarding the gem 'geokit-rails'

This is my sample code

```<code>class Outlet < ActiveRecord::Base
  acts_as_mappable :lat_column_name => :address_latitude,
                   :lng_column_name => :address_longitude,
                   :default_units => :kms
end
</code>```

I cant use the find(:all)

sample Outlet.find(:all, :origin =>[32.951613,-96.958444], :within=>10)
it returns
ActiveRecord::RecordNotFound: Couldn't find all Outlets with 'id': (all, {:origin=>[32.951613, -96.958444], :within=>10})

I have also tested on this query Outlet.find(:all)
it returns ActiveRecord::RecordNotFound: Couldn't find Outlet with 'id'=all

Distance. I dont really understand how does this work. Should we add distance column into the the outlet model ? If yes, whats the column type? And is that possible to return the record with the distance value based on the given lat lng ? Outlet.by_distance(:origin =>[32.951613, -96.958444])
it will return the records with the distance to that point. (2kms, 3kms, 4km, ...)

答:
find is an ActiveRecord method. It has nothing to do with Geokit. It takes model id such as 1 for the first Outlet you created or 2 for the second Outlet. You used Outlet.find(:all) so, ActiveRecord complained that it could not find any Outlet with id all.

I think what you actually wanted to do is

```<code>Outlet.within(10, :origin => [32.951613,-96.958444])
</code>```

Note that you can also pass an object for origin, for example,

```<code>@the_latest_outlet = Outlet.last
@outlets_nearby_the_latest_outlet = Outlet.within(10, origin: @the_latest_outlet)
</code>```

No, you don't have to add a column. The field distance is added to ActiveRecord instances automatically. I still haven't found any use of the parameter :distance_field_name. My best guess is that if your model has a field named distance, you should use distance_field_name parameter to avoid the conflict.

2.1. If you want to measure distance between 2 points, you need distance_to as this example:

```<code>a = Outlet.find(1)
b = Outlet.find(2)
distance_from_a_to_b = a.distance_to(b)
</code>```

2.2. by_distance is to get all your instances sorted by distance.

```<code>a_place_to_compare = [lat, lng]
all_outlets_sorted_by_distance_from_the_place = Outlet.by_distance(a_place_to_compare)
</code>```

Title: Geokit Rails distance and find(:all)

Body:
I have 2 questions regarding the gem 'geokit-rails'

This is my sample code

class Outlet < ActiveRecord::Base
  acts_as_mappable :lat_column_name => :address_latitude,
                   :lng_column_name => :address_longitude,
                   :default_units => :kms
end

I cant use the find(:all)

sample Outlet.find(:all, :origin =>[32.951613,-96.958444], :within=>10)
it returns
ActiveRecord::RecordNotFound: Couldn't find all Outlets with 'id': (all, {:origin=>[32.951613, -96.958444], :within=>10})

I have also tested on this query Outlet.find(:all)
it returns ActiveRecord::RecordNotFound: Couldn't find Outlet with 'id'=all

Distance. I dont really understand how does this work. Should we add distance column into the the outlet model ? If yes, whats the column type? And is that possible to return the record with the distance value based on the given lat lng ? Outlet.by_distance(:origin =>[32.951613, -96.958444])
it will return the records with the distance to that point. (2kms, 3kms, 4km, ...)

Body:
find is an ActiveRecord method. It has nothing to do with Geokit. It takes model id such as 1 for the first Outlet you created or 2 for the second Outlet. You used Outlet.find(:all) so, ActiveRecord complained that it could not find any Outlet with id all.

I think what you actually wanted to do is

Outlet.within(10, :origin => [32.951613,-96.958444])

Note that you can also pass an object for origin, for example,

@the_latest_outlet = Outlet.last
@outlets_nearby_the_latest_outlet = Outlet.within(10, origin: @the_latest_outlet)

No, you don't have to add a column. The field distance is added to ActiveRecord instances automatically. I still haven't found any use of the parameter :distance_field_name. My best guess is that if your model has a field named distance, you should use distance_field_name parameter to avoid the conflict.

2.1. If you want to measure distance between 2 points, you need distance_to as this example:

a = Outlet.find(1)
b = Outlet.find(2)
distance_from_a_to_b = a.distance_to(b)

2.2. by_distance is to get all your instances sorted by distance.

a_place_to_compare = [lat, lng]
all_outlets_sorted_by_distance_from_the_place = Outlet.by_distance(a_place_to_compare)


问:
Hiding a private overloaded virtual function?
I have a class hierarchy that works approximately like this:

```<code>class A
{
protected:
    virtual void f(int) = 0;
};

class B
{
protected:
    virtual void f(char*) = 0;
};

class DA : A
{
private:
    virtual void f(int) override {}
};

class DB : public DA, B
{
private:
    virtual void f(char*) override {}
};
</code>```

When I try to compile with clang (or gcc, for that matter), it gives me the warning

```<code><source>:22:18: warning: 'DB::f' hides overloaded virtual function [-Woverloaded-virtual]
    virtual void f(char*) override {}
                 ^
<source>:16:18: note: hidden overloaded virtual function 'DA::f' declared here: type mismatch at 1st parameter ('int' vs 'char *')
    virtual void f(int) override {}
                 ^
</code>```

Which I understand, but should it really give that warning? After all, DB cannot even see the hidden function (which might even be named the same by coincidence).

If it wasn't private, I could use

```<code>using DA::f;
</code>```

to clarify, of course, but the function is private, DB doesn't even know about it, and certainly shouldn't expose it.

Is there a way to fix this without deactivating that warning, i.e. telling the compiler that everything is designed as is intended?

答:
A possible way to fix the warning is to duplicate code of DA::f inside DB:

```<code>class DB : public DA, B
{
    virtual void f(int i) override
    {
         // copy implementation of DA::f(int) as you cannot do
         // return DA::f(i);
    }
private:
    virtual void f(char*) override {}
};
</code>```

Demo

but a local pragma to remove warning seems more appropriate.

Body:
I have a class hierarchy that works approximately like this:

class A
{
protected:
    virtual void f(int) = 0;
};

class B
{
protected:
    virtual void f(char*) = 0;
};

class DA : A
{
private:
    virtual void f(int) override {}
};

class DB : public DA, B
{
private:
    virtual void f(char*) override {}
};

When I try to compile with clang (or gcc, for that matter), it gives me the warning

<source>:22:18: warning: 'DB::f' hides overloaded virtual function [-Woverloaded-virtual]
    virtual void f(char*) override {}
                 ^
<source>:16:18: note: hidden overloaded virtual function 'DA::f' declared here: type mismatch at 1st parameter ('int' vs 'char *')
    virtual void f(int) override {}
                 ^

Which I understand, but should it really give that warning? After all, DB cannot even see the hidden function (which might even be named the same by coincidence).

If it wasn't private, I could use

using DA::f;

to clarify, of course, but the function is private, DB doesn't even know about it, and certainly shouldn't expose it.

Is there a way to fix this without deactivating that warning, i.e. telling the compiler that everything is designed as is intended?

Body:
A possible way to fix the warning is to duplicate code of DA::f inside DB:

class DB : public DA, B
{
    virtual void f(int i) override
    {
         // copy implementation of DA::f(int) as you cannot do
         // return DA::f(i);
    }
private:
    virtual void f(char*) override {}
};

Demo

but a local pragma to remove warning seems more appropriate.


问:
Numerical differentiation using Cauchy (CIF)
I am trying to create a module with a mathematical class for Taylor series, to have it easily accessible for other projects. Hence I wish to optimize it as far as I can.

For those who are not too familiar with Taylor series, it will be a necessity to be able to differentiate a function in a point many times. Given that the normal definition of the mathematical derivative of a function will require immense precision for higher order derivatives, I've decided to use Cauchy's integral formula instead. With a little bit of work, I've managed to rearrange the formula a little bit, as you can see on this picture: Rearranged formula. This provided me with much more accurate results on higher order derivatives than the traditional definition of the derivative. Here is the function i am currently using to differentiate a function in a point:

```<code>def myDerivative(f, x, dTheta, degree):
  riemannSum = 0
  theta = 0

  while theta < 2*np.pi:
      functionArgument = np.complex128(x + np.exp(1j*theta))
      secondFactor = np.complex128(np.exp(-1j * degree * theta))
      riemannSum += f(functionArgument) * secondFactor * dTheta
      theta += dTheta
  return factorial(degree)/(2*np.pi) * riemannSum.real
</code>```

I've tested this function in my main function with a carefully thought out mathematical function which I know the derivatives of, namely f(x) = sin(x).

```<code>def main():
  print(myDerivative(f, 0, 2*np.pi/(4*4096), 16))
  pass
</code>```

These derivatives seems to freak out at around the derivative of degree 16. I've also tried to play around with dTheta, but with no luck. I would like to have higher orders as well, but I fear I've run into some kind of machine precission.

My question is in it's simplest form: What can I do to improve this function in order to get higher order of my derivatives?

答: Well, since you are working with a discrete approximation of the derivative (via dTheta), sooner or later you must run into trouble. I'm surprised you were able to get at least 15 accurate derivatives -- good work! But to get derivatives of all orders, either you have to put a limit on what you're willing to accept and say it's good enough, or else compute the derivatives symbolically. Take a look at Sympy for that. Sympy probably has some functions for computing Taylor series too.</p>

Body:
I am trying to create a module with a mathematical class for Taylor series, to have it easily accessible for other projects. Hence I wish to optimize it as far as I can.

For those who are not too familiar with Taylor series, it will be a necessity to be able to differentiate a function in a point many times. Given that the normal definition of the mathematical derivative of a function will require immense precision for higher order derivatives, I've decided to use Cauchy's integral formula instead. With a little bit of work, I've managed to rearrange the formula a little bit, as you can see on this picture: Rearranged formula. This provided me with much more accurate results on higher order derivatives than the traditional definition of the derivative. Here is the function i am currently using to differentiate a function in a point:

def myDerivative(f, x, dTheta, degree):
  riemannSum = 0
  theta = 0

  while theta < 2*np.pi:
      functionArgument = np.complex128(x + np.exp(1j*theta))
      secondFactor = np.complex128(np.exp(-1j * degree * theta))
      riemannSum += f(functionArgument) * secondFactor * dTheta
      theta += dTheta
  return factorial(degree)/(2*np.pi) * riemannSum.real

I've tested this function in my main function with a carefully thought out mathematical function which I know the derivatives of, namely f(x) = sin(x).

def main():
  print(myDerivative(f, 0, 2*np.pi/(4*4096), 16))
  pass

These derivatives seems to freak out at around the derivative of degree 16. I've also tried to play around with dTheta, but with no luck. I would like to have higher orders as well, but I fear I've run into some kind of machine precission.

My question is in it's simplest form: What can I do to improve this function in order to get higher order of my derivatives?


问:
Trained TensorFlow model always outputs zero
I am training an autonomous driving convolutional neural network in TensorFlow. It is a simple regression network that takes an image and outputs a single value (a steering angle).

This is the function in which the network is defined:

```<code>def cnn_model_fn(features, labels, mode):
    conv1 = tf.layers.conv2d(
        inputs=features,
        filters=32,
        kernel_size=5,
        padding="same",
        activation=tf.nn.relu
    )

    pool1 = tf.layers.max_pooling2d(
        inputs=conv1,
        pool_size=2,
        strides=2
    )

    pool1_flat = tf.reshape(pool1, [-1, 2764800])

    dense1 = tf.layers.dense(
        inputs=pool1_flat,
        units=128,
        activation=tf.nn.relu
    )

    dropout = tf.layers.dropout(
        inputs=dense1,
        rate=0.4,
        training=mode == learn.ModeKeys.TRAIN
    )

    dense2 = tf.layers.dense(
        inputs=dropout,
        units=1,
        activation=tf.nn.relu
    )

    predictions = tf.reshape(dense2, [-1])

    loss = None
    train_op = None

    if mode != learn.ModeKeys.INFER:
        loss = tf.losses.mean_squared_error(
            labels=labels,
            predictions=predictions
        )

    if mode == learn.ModeKeys.TRAIN:
        train_op = tf.contrib.layers.optimize_loss(
            loss=loss,
            global_step=tf.contrib.framework.get_global_step(),
            learning_rate=0.001,
            optimizer="SGD"
        )

    return model_fn_lib.ModelFnOps(
        mode=mode,
        predictions=predictions,
        loss=loss,
        train_op=train_op
    )
</code>```

Elsewhere in the program, I initiate the classifier's training like so:

```<code>def main(_):
    # Gather data
    images, labels = get_data("./data/labels.csv")

    # Create the estimator
    classifier = learn.Estimator(
        model_fn=cnn_model_fn,
        model_dir="/tmp/network2"
    )

    # Train the model
    classifier.fit(
        x=images,
        y=labels,
        batch_size=10,
        steps=20
    )

    for v in tf.trainable_variables():
        print(v)
</code>```

labels is a simple one-dimensional NumPy containing all of the steering angles for the training examples. They are being read from a CSV file. The values in the file are quite close to 0, and have an average around zero.

When they are read directly from the file, or multiplied by a scalar, the network converges reasonably well, and achieves a low loss function. When I add a constant, it fails to converge or diverges. I have a suspicion that all of the weights of the network are converging on zero.

Does anybody see any problems with my methodology?

答:
That dropout regularization might be the culprit:

```<code>dropout = tf.layers.dropout(
        inputs=dense1,
        rate=0.4,
        training=mode == learn.ModeKeys.TRAIN
    )
</code>```

What you are describing, the weights failing to adequately converge or falling close to zero, is highly descriptive of a high-bias problem. Removing or reducing the degree of regularization, adding more parameters to your network, or otherwise increasing the variance are common ways to fix this type of problem.

Body:
I am training an autonomous driving convolutional neural network in TensorFlow. It is a simple regression network that takes an image and outputs a single value (a steering angle).

This is the function in which the network is defined:

def cnn_model_fn(features, labels, mode):
    conv1 = tf.layers.conv2d(
        inputs=features,
        filters=32,
        kernel_size=5,
        padding="same",
        activation=tf.nn.relu
    )

    pool1 = tf.layers.max_pooling2d(
        inputs=conv1,
        pool_size=2,
        strides=2
    )

    pool1_flat = tf.reshape(pool1, [-1, 2764800])

    dense1 = tf.layers.dense(
        inputs=pool1_flat,
        units=128,
        activation=tf.nn.relu
    )

    dropout = tf.layers.dropout(
        inputs=dense1,
        rate=0.4,
        training=mode == learn.ModeKeys.TRAIN
    )

    dense2 = tf.layers.dense(
        inputs=dropout,
        units=1,
        activation=tf.nn.relu
    )

    predictions = tf.reshape(dense2, [-1])

    loss = None
    train_op = None

    if mode != learn.ModeKeys.INFER:
        loss = tf.losses.mean_squared_error(
            labels=labels,
            predictions=predictions
        )

    if mode == learn.ModeKeys.TRAIN:
        train_op = tf.contrib.layers.optimize_loss(
            loss=loss,
            global_step=tf.contrib.framework.get_global_step(),
            learning_rate=0.001,
            optimizer="SGD"
        )

    return model_fn_lib.ModelFnOps(
        mode=mode,
        predictions=predictions,
        loss=loss,
        train_op=train_op
    )

Elsewhere in the program, I initiate the classifier's training like so:

def main(_):
    # Gather data
    images, labels = get_data("./data/labels.csv")

    # Create the estimator
    classifier = learn.Estimator(
        model_fn=cnn_model_fn,
        model_dir="/tmp/network2"
    )

    # Train the model
    classifier.fit(
        x=images,
        y=labels,
        batch_size=10,
        steps=20
    )

    for v in tf.trainable_variables():
        print(v)

labels is a simple one-dimensional NumPy containing all of the steering angles for the training examples. They are being read from a CSV file. The values in the file are quite close to 0, and have an average around zero.

When they are read directly from the file, or multiplied by a scalar, the network converges reasonably well, and achieves a low loss function. When I add a constant, it fails to converge or diverges. I have a suspicion that all of the weights of the network are converging on zero.

Does anybody see any problems with my methodology?

Body:
That dropout regularization might be the culprit:

dropout = tf.layers.dropout(
        inputs=dense1,
        rate=0.4,
        training=mode == learn.ModeKeys.TRAIN
    )

What you are describing, the weights failing to adequately converge or falling close to zero, is highly descriptive of a high-bias problem. Removing or reducing the degree of regularization, adding more parameters to your network, or otherwise increasing the variance are common ways to fix this type of problem.


答:
For interpolation in Python you could use scipy.interpolate.InterpolatedUnivariateSpline

```<code>import numpy as np
from scipy.interpolate import InterpolatedUnivariateSpline
x = [57136,57688,58046,58480,58730,59210,59775,60275,60900,61365,62030]
y = [3.87, 3.55, 3.75, 2.04, 2.66, 3.1, 3.38, 4.13, 3.7, 4, 5.78]

interpolation_function = InterpolatedUnivariateSpline(x,y)
new_x = np.arange(57000,62001,250)
new_y = interpolation_function(new_x)
</code>```

The output will be numpy arrays, which can then be put into a pandas dataframe.

This will most certainly not get you the values you indicate in your answer, as the original y-values are all in the range [2, 6], so one would expect the output to also be in this range (for interpolated values) as pointed out by @Prune.

InterpolatedUnivariateSpline by default allows for extrapolation(see ext parameter). If you want linear interpolation instead of cubic interpolation(k=3, the default) you can specify k=1 as an argument.

Pandas also has its own interpolation method interpolate which you could use if your starting point is a Dataframe

Title: interpolate/extrapolate between two x-values for a given number of steps python

Body:
I want to create a new df that given a starting value x0 and an end value x1 the output interpolates/extrapolates for a given n number of points.

For example, given the df below, I want to create a new df between x0=57000 and x1=62000 in steps of 250, or n=21 points:

    x = [57136,57688,58046,58480,58730,59210,59775,60275,60900,61365,62030]
    y = [3.87, 3.55, 3.75, 2.04, 2.66, 3.1, 3.38, 4.13, 3.7, 4, 5.78]

    df = pd.DataFrame(data=[x,y]).transpose()
    df.columns=['x','y']

Given df1, I want to create a new df2 such that the output will be:

    >>>print(df2)
            x         y
    0       57000     2.78745
    1       57250     2.74425
    2       57500     2.70106
    3       57750     2.72185
    4       58000     2.93666
    5       58250     2.34479
    6       58500     1.67233
    7       58750     2.13959
    8       59000     2.31422
    9       59250     2.47805
    10      59500     2.58523
    11      59750     2.69242
    12      60000     2.97746
    13      60250     3.28227
    14      60500     3.18627
    15      60750     3.04574
    16      61000     3.04658
    17      61250     3.25947
    18      61500     3.62019
    19      61750     4.10685
    20      62000     4.59351

Body:
For interpolation in Python you could use scipy.interpolate.InterpolatedUnivariateSpline

import numpy as np
from scipy.interpolate import InterpolatedUnivariateSpline
x = [57136,57688,58046,58480,58730,59210,59775,60275,60900,61365,62030]
y = [3.87, 3.55, 3.75, 2.04, 2.66, 3.1, 3.38, 4.13, 3.7, 4, 5.78]

interpolation_function = InterpolatedUnivariateSpline(x,y)
new_x = np.arange(57000,62001,250)
new_y = interpolation_function(new_x)

The output will be numpy arrays, which can then be put into a pandas dataframe.

This will most certainly not get you the values you indicate in your answer, as the original y-values are all in the range [2, 6], so one would expect the output to also be in this range (for interpolated values) as pointed out by @Prune.

InterpolatedUnivariateSpline by default allows for extrapolation(see ext parameter). If you want linear interpolation instead of cubic interpolation(k=3, the default) you can specify k=1 as an argument.

Pandas also has its own interpolation method interpolate which you could use if your starting point is a Dataframe


问:
Multiple Request to Single Field in Scrapy
I am trying to scrape a website using Scrapy. Example Link is: Here.
I am able to get some data using css selectors. I also need to fetch all image urls of each item. Now an item can have multiple colours. When we click on another colour, it actually fetch images from another url in the browser. So, I need to generate manual requests (due to multiple colours) and attach "meta" to store image urls from others urls into a SINGLE ITEM FIELD.

Here is my Scrapy code:

```<code>def get_image_urls(self, response):
    item = response.meta['item']
    if 'image_urls' in item:
        urls = item['image_urls']
    else:
        urls = []
    urls.extend(response.css('.product-image-link::attr(href)').extract())
    item['image_urls'] = urls
    next_url = response.css('.va-color .emptyswatch a::attr(href)').extract()
    #print(item['image_urls'])
    yield Request(next_url[0], callback=self.get_image_urls, meta={'item': item})

def parse(self, response):
    output = JulesProduct()
    output['name'] = self.get_name(response)

    # Now get the recursive img urls
    response.meta['item'] = output
    self.get_image_urls(response)
    return output
</code>```

Ideally, I should return output object to have all of the required data. My question is why I am not getting output['image_urls']? Because when I uncomment print statement in get_image_urls function, I see 3 crawled urls and 3 print statements with url appended after each other. I need them in the parse function. I'm not sure if I'm able to dictate my issue. Can anybody help?

答:
Your parse method is returning the output before the get_image_urls requests are done.

You should only yield or return your final item and at the end of your recursive logic. Something like this should work:

```<code>def parse(self, response):
    output = JulesProduct()
    output['name'] = self.get_name(response)
    yield Request(response.url, callback=self.get_image_urls, meta={'item': item}, dont_filter=True)

def get_image_urls(self, response):
    item = response.meta['item']
    if 'image_urls' in item:
        urls = item['image_urls']
    else:
        urls = []
    urls.extend(response.css('.product-image-link::attr(href)').extract())
    item['image_urls'] = urls
    next_url = response.css('.va-color .emptyswatch a::attr(href)').extract()

    if len(next_url) > 0:
        yield Request(next_url[0], callback=self.get_image_urls, meta={'item': item})
    else:
        yield item
</code>```

Body:
I am trying to scrape a website using Scrapy. Example Link is: Here.
I am able to get some data using css selectors. I also need to fetch all image urls of each item. Now an item can have multiple colours. When we click on another colour, it actually fetch images from another url in the browser. So, I need to generate manual requests (due to multiple colours) and attach "meta" to store image urls from others urls into a SINGLE ITEM FIELD.

Here is my Scrapy code:

def get_image_urls(self, response):
    item = response.meta['item']
    if 'image_urls' in item:
        urls = item['image_urls']
    else:
        urls = []
    urls.extend(response.css('.product-image-link::attr(href)').extract())
    item['image_urls'] = urls
    next_url = response.css('.va-color .emptyswatch a::attr(href)').extract()
    #print(item['image_urls'])
    yield Request(next_url[0], callback=self.get_image_urls, meta={'item': item})

def parse(self, response):
    output = JulesProduct()
    output['name'] = self.get_name(response)

    # Now get the recursive img urls
    response.meta['item'] = output
    self.get_image_urls(response)
    return output

Ideally, I should return output object to have all of the required data. My question is why I am not getting output['image_urls']? Because when I uncomment print statement in get_image_urls function, I see 3 crawled urls and 3 print statements with url appended after each other. I need them in the parse function. I'm not sure if I'm able to dictate my issue. Can anybody help?

Body:
Your parse method is returning the output before the get_image_urls requests are done.

You should only yield or return your final item and at the end of your recursive logic. Something like this should work:

def parse(self, response):
    output = JulesProduct()
    output['name'] = self.get_name(response)
    yield Request(response.url, callback=self.get_image_urls, meta={'item': item}, dont_filter=True)

def get_image_urls(self, response):
    item = response.meta['item']
    if 'image_urls' in item:
        urls = item['image_urls']
    else:
        urls = []
    urls.extend(response.css('.product-image-link::attr(href)').extract())
    item['image_urls'] = urls
    next_url = response.css('.va-color .emptyswatch a::attr(href)').extract()

    if len(next_url) > 0:
        yield Request(next_url[0], callback=self.get_image_urls, meta={'item': item})
    else:
        yield item


问:
Remove a specific string only if it's in the very end of a code
I'm parsing a html file to extract part of its text to create an epub. The problem is that in the text extracted sometimes the last paragraph is an empty one that I would like to remove. So... how can I remove this blank paragraph (and any additional blank spaces after that) with the condition that there's no additional text later on in any other line of code?

For example:

```<code><p> </p>
<p>“Lorem ipsum dolor sit amet, consectetur adipiscing elit.”</p>
<p>Sed ut perspiciatis unde omnis iste natus error...</p>
<p> </p>
<p>“Lorem ipsum dolor sit amet.”</p>
<p>Omnis iste natus error sit voluptatem.</p>
<p> </p>
</code>```

So, what can I do to remove only this last instance of

 (and any additional blank spaces/newlines after that) from the code above?

I have tried using this negative lookahead

\s*?(?!
) on perl-regex to exclude later paragraphs from the search, but it still finds the previous instances of

 and I only need to remove this paragraph when it's the last one in the file.

Thank you in advance!

EDIT-1:

To be clear, I want this:

```<code><p> </p>
<p>“Lorem ipsum dolor sit amet, consectetur adipiscing elit.”</p>
<p>Sed ut perspiciatis unde omnis iste natus error...</p>
<p> </p>
<p>“Lorem ipsum dolor sit amet.”</p>
<p>Omnis iste natus error sit voluptatem.</p>
<p> </p>
</code>```

to become this:

```<code><p> </p>
<p>“Lorem ipsum dolor sit amet, consectetur adipiscing elit.”</p>
<p>Sed ut perspiciatis unde omnis iste natus error...</p>
<p> </p>
<p>“Lorem ipsum dolor sit amet.”</p>
<p>Omnis iste natus error sit voluptatem.</p>
</code>```

i.e. I want to remove the

 only in the last line (where there's no other text after it), so I want to know which perl-regex search I should use to find only this particular instance of this string so I can replace it with nothing and with that remove it from the code.

EDIT-2.:

Based on a suggestion by ikegami, I used \s*(?>

\s*)(?!
)\s* as the search string to find only the last blank paragraph (

)of the html code to be removed. What really made the difference in his answer seems to have been the use of an atomic grouping (?>...). Without it, I was picking other instances of the same string from other lines of the code that I didn't want. Not sure why (really not an expert on regex), but that's what I got from my tests.

I'm just using some basic find/replace operations of perl-regex in an ebook editor to clean the code, so I'm not sure about how it will behave under other circumstances. Anyway, I appreciate the other attempts of helping me, some of which were even too technical for me to grasp and I hope this answer will help someone in the future with a similar problem. Thank you again!

答:
```<code>EDIT-2.:

Based on a suggestion by ikegami, I used
\s*(?><p> </p>\s*)(?!<p>)\s* as the search string to find only
the last blank paragraph (<p> </p>)of the html code to be
removed. What really made the difference in his answer seems to have
been the use of an atomic grouping (?>...). Without it, I was picking
other instances of the same string from other lines of the code that I
didn't want. Not sure why (really not an expert on regex), but that's
what I got from my tests.
</code>```

You should not use something you don't understand.

In reality, you should learn regex instead of asking somebody to do it for you.

Just a short education here, the same is achieved with this regex

\s*

(?>\s*)(?!
)\s*

in that the juxtaposition of these two (?>\s*) (?!
) prevents

the engine from giving up a single whitespace to satisfy the assertion.

If you had no whitespace between paragraph markers, an atomic group is

not necessary.

To further explore this, the assertion (?!
) presumes there exists

a paragraph marker directly after

\s*.

This is a bad choice for a design. If there is no paragraph marker there

it will match it, even if it exists further down somewhere.

Other design considerations:

You state this is html, which means there will be other tags following

the last

. This means you shouldn't need the EOS anchor $ or \zin the
regex, unless you use a regex that @ikegami suggests (?>

\s*)(?=(?:(?!
).)*\z)

which need's a (?s) to go with it.

However, this regex is a terrible idea !!

Each time it finds a

 it has to stop and search all the way to

the end of the string.

If you have 500

 in the document, it is mostly equivalent to searching

the same document text 500 times.

The Safe and Sane approach

This way -> (?s).*\K

\s*

It just goes directly to the last one via (?s).*

then skips that part of the match using \K,

leaving only the

\s* to be replaced with nothing.

Benchmark test:

The target sample contains 10 duplicates of the OP's sample @ 30

```<code>@ikegami Regex:   (?s)(?><p> </p>\s*)(?=(?:(?!<p>).)*\z)
Options:  < none >
Completed iterations:   50  /  50     ( x 1000 )
Matches found per iteration:   1
Elapsed Time:    3.24 s,   3236.42 ms,   3236424 µs

Safe and Sane Regex:   (?s).*\K<p> </p>\s*
Options:  < none >
Completed iterations:   50  /  50     ( x 1000 )
Matches found per iteration:   1
Elapsed Time:    0.10 s,   102.04 ms,   102044 µs
</code>```

Body:
I'm parsing a html file to extract part of its text to create an epub. The problem is that in the text extracted sometimes the last paragraph is an empty one that I would like to remove. So... how can I remove this blank paragraph (and any additional blank spaces after that) with the condition that there's no additional text later on in any other line of code?

For example:

“Lorem ipsum dolor sit amet, consectetur adipiscing elit.”

Sed ut perspiciatis unde omnis iste natus error...

“Lorem ipsum dolor sit amet.”

Omnis iste natus error sit voluptatem.

So, what can I do to remove only this last instance of

 (and any additional blank spaces/newlines after that) from the code above?

I have tried using this negative lookahead

\s*?(?!
) on perl-regex to exclude later paragraphs from the search, but it still finds the previous instances of

 and I only need to remove this paragraph when it's the last one in the file.

Thank you in advance!

EDIT-1:

To be clear, I want this:

“Lorem ipsum dolor sit amet, consectetur adipiscing elit.”

Sed ut perspiciatis unde omnis iste natus error...

“Lorem ipsum dolor sit amet.”

Omnis iste natus error sit voluptatem.

to become this:

“Lorem ipsum dolor sit amet, consectetur adipiscing elit.”

Sed ut perspiciatis unde omnis iste natus error...

“Lorem ipsum dolor sit amet.”

Omnis iste natus error sit voluptatem.

i.e. I want to remove the

 only in the last line (where there's no other text after it), so I want to know which perl-regex search I should use to find only this particular instance of this string so I can replace it with nothing and with that remove it from the code.

EDIT-2.:

Based on a suggestion by ikegami, I used \s*(?>

\s*)(?!
)\s* as the search string to find only the last blank paragraph (

)of the html code to be removed. What really made the difference in his answer seems to have been the use of an atomic grouping (?>...). Without it, I was picking other instances of the same string from other lines of the code that I didn't want. Not sure why (really not an expert on regex), but that's what I got from my tests.

I'm just using some basic find/replace operations of perl-regex in an ebook editor to clean the code, so I'm not sure about how it will behave under other circumstances. Anyway, I appreciate the other attempts of helping me, some of which were even too technical for me to grasp and I hope this answer will help someone in the future with a similar problem. Thank you again!

Body:
EDIT-2.:

Based on a suggestion by ikegami, I used
\s*(?><p>&nbsp;</p>\s*)(?!<p>)\s* as the search string to find only
the last blank paragraph (<p>&nbsp;</p>)of the html code to be
removed. What really made the difference in his answer seems to have
been the use of an atomic grouping (?>...). Without it, I was picking
other instances of the same string from other lines of the code that I
didn't want. Not sure why (really not an expert on regex), but that's
what I got from my tests.

You should not use something you don't understand.

In reality, you should learn regex instead of asking somebody to do it for you.

Just a short education here, the same is achieved with this regex

\s*<p>&nbsp;</p>(?>\s*)(?!<p>)\s*

in that the juxtaposition of these two (?>\s*) (?!<p>) prevents

the engine from giving up a single whitespace to satisfy the assertion.

If you had no whitespace between paragraph markers, an atomic group is

not necessary.

To further explore this, the assertion (?!<p>) presumes there exists

a paragraph marker directly after <p>&nbsp;</p>\s*.

This is a bad choice for a design. If there is no paragraph marker there

it will match it, even if it exists further down somewhere.

Other design considerations:

You state this is html, which means there will be other tags following

the last <p>&nbsp;</p>. This means you shouldn't need the EOS anchor $ or \zin the
regex, unless you use a regex that @ikegami suggests (?><p>&nbsp;</p>\s*)(?=(?:(?!<p>).)*\z)

which need's a (?s) to go with it.

However, this regex is a terrible idea !!

Each time it finds a <p>&nbsp;</p> it has to stop and search all the way to

the end of the string.

If you have 500  <p>&nbsp;</p> in the document, it is mostly equivalent to searching

the same document text 500 times.

The Safe and Sane approach

This way -> (?s).*\K<p>&nbsp;</p>\s*

It just goes directly to the last one via (?s).*

then skips that part of the match using \K,

leaving only the <p>&nbsp;</p>\s* to be replaced with nothing.

Benchmark test:

The target sample contains 10 duplicates of the OP's sample @ 30 <p>&nbsp;</p>

@ikegami Regex:   (?s)(?><p>&nbsp;</p>\s*)(?=(?:(?!<p>).)*\z)
Options:  < none >
Completed iterations:   50  /  50     ( x 1000 )
Matches found per iteration:   1
Elapsed Time:    3.24 s,   3236.42 ms,   3236424 µs

Safe and Sane Regex:   (?s).*\K<p>&nbsp;</p>\s*
Options:  < none >
Completed iterations:   50  /  50     ( x 1000 )
Matches found per iteration:   1
Elapsed Time:    0.10 s,   102.04 ms,   102044 µs


问:
Overriding ReadOnly Property in a subclass to make it Read/Write (VB.NET or C#)
This doesn't seem possible in VB.NET with properties since the property statement itself must describe whether it is ReadOnly or not.

In my example below, it doesn't let me make the ReadWriteChild compile.  I guess I could make the parent Read/Write, and then have the ReadOnlyChild's setter not do anything, but that seems sort of hacky.  The best alternative seems to be abandoning properties in favor of getter/setter methods in this case.

Public MustInherit Class Parent

    Public MustOverride ReadOnly Property Foo() As String

End Class

Public Class ReadOnlyChild
    Inherits Parent

    Public Overrides ReadOnly Property Foo() As String
        Get
            ' Get the Property
        End Get
    End Property

End Class

Public Class ReadWriteChild
    Inherits Parent

    Public Overrides Property Foo() As String
        Get
            ' Get the property.
        End Get
        Set(ByVal value As String)
           ' Set the property.
        End Set
    End Property

End Class
```

答:
You need to add another level of hierarchy; unfortunately, the only way to really scope things properly is to use nested classes:

Public Class IntermediateClassForPrivateInheritanceOnly
  Inherits Parent
  Public Overrides ReadOnly Property Foo() As String
  ' etc.
  Private Sub New(whatever)
    MyBase.New(whatever)
  End Sub

  ' Most other stuff for the class goes here.

  Public Class ReadWriteChild
    Inherits IntermediateClassForPrivateInheritanceOnly
    Shadows Property Foo()
    ' etc.
    Public Sub New(whatever)
      MyBase.New(whatever)
    End Sub
  End Class
End Class

Public Class ReadWriteChild  ' Would use an alias, if such things existed
  Inherits IntermediateClassForPrivateInheritanceOnly
  Public Sub New(whatever)
    MyBase.New(whatever)
  End Sub
End Class
```


问:
Delete unused ViewController from memory
I have 3 ViewController.
The first ViewController is checking if the user is logged in.
If yes performSegue to the mainVC and if no performSegue to loginVC.

When I am in loginVC, I log in and performSegue to mainVC.

What I want now is, I want to have all ViewControllers which are unused being "deleted", to save memory.
How is that going to work?

I found here in StackOverflow this piece of code:

```<code>class ManualSegue: UIStoryboardSegue {

  override func perform() {
    sourceViewController.presentViewController(destinationViewController, animated: true) {
      self.sourceViewController.navigationController?.popToRootViewControllerAnimated(false)
      UIApplication.sharedApplication().delegate?.window??.rootViewController = self.destinationViewController
    }
  }
}
</code>```

Is that going to do what I want? It seems like yes because this method is popping the ViewController.

I am using "Show Detail" - segues only, except when using this method I created a custom Segue Segue.

答:
Memory question

Yes, that is how it works. You do not need to take care of freeing view controllers.

The system will keep track of references to view controller objects. When you do not have references to these anymore then the memory is deallocated. You can read about this more in swift language documentation:

https://developer.apple.com/library/content/documentation/Swift/Conceptual/Swift_Programming_Language/AutomaticReferenceCounting.html

What the code is doing

presentViewController method is showing a view controller modally. The completion closure is performed after presenting the new view controller finishes. Inside closer 2 things happen

popToRootViewControllerRemoves all view controllers inside the sourceViewController object.

rootViewController of the window is set to new value.

This practically changes the root view controller to another one. This seems like a valid action after successful login.

I do not know if step 1 is necessary. That navigation view controller is going to go away anyway so why to pop view controllers inside it?

More about view controllers

You might be also interested in view controller life cycle. UIKit developer documentation contains in-depth details about view controllers:

https://developer.apple.com/documentation/uikit/uiviewcontroller

Body:
I have 3 ViewController.
The first ViewController is checking if the user is logged in.
If yes performSegue to the mainVC and if no performSegue to loginVC.

When I am in loginVC, I log in and performSegue to mainVC.

What I want now is, I want to have all ViewControllers which are unused being "deleted", to save memory.
How is that going to work?

I found here in StackOverflow this piece of code:

class ManualSegue: UIStoryboardSegue {

  override func perform() {
    sourceViewController.presentViewController(destinationViewController, animated: true) {
      self.sourceViewController.navigationController?.popToRootViewControllerAnimated(false)
      UIApplication.sharedApplication().delegate?.window??.rootViewController = self.destinationViewController
    }
  }
}

Is that going to do what I want? It seems like yes because this method is popping the ViewController.

I am using "Show Detail" - segues only, except when using this method I created a custom Segue Segue.


问:
string modifiers and properties do not work
I am trying to use Methods and Properties of the String Class to modify a string, but I keep getting an Invalid qualifier compile error. I even copied the following code* directly from the MSDN website and it throws the same error.

```<code>Public Sub Main()
    Dim original As String
    original = "aaabbb"
    Dim modified As String
    modified = original.Insert(3, " ")
End Sub

'This is the original code, but I had to change it slightly because the word-vba
'programming environment didn't like the syntax and highlighted everything red.

'Public Sub Main()
   'Dim original As String = "aaabbb"
   'Console.WriteLine("The original string: '{0}'", original)
   'Dim modified As String = original.Insert(3, " ")
   'Console.WriteLine("The modified string: '{0}'", modified)
'End Sub
</code>```

Does word-vba not support string class modifiers and properties, am I not initializing the string correctly, or is there some other problem?

答:
```<code>modified = original.Insert(3, " ")
</code>```

You're thinking in VB.NET, but writing VBA. Strings (or any primitive or UDT type) don't have members in VBA. Not your fault, finding official VBA documentation is getting harder every day, with every "VBA" search yielding results for VB.NET.

That original code is clearly VB.NET.

If you mean to concatenate 3 spaces in front of original, then what you want to do is this:

```<code>modified = String(3, " ") & original
</code>```

If you mean to get a new string in which a specified string is inserted at a specified index position in this instance (MSDN), then you want to do this (thanks @A.S.H!):

```<code>modified = Left$(original, 3) & " " & Right$(original, Len(original) - 3)
</code>```

Body:
I am trying to use Methods and Properties of the String Class to modify a string, but I keep getting an Invalid qualifier compile error. I even copied the following code* directly from the MSDN website and it throws the same error.

Public Sub Main()
    Dim original As String
    original = "aaabbb"
    Dim modified As String
    modified = original.Insert(3, " ")
End Sub

'This is the original code, but I had to change it slightly because the word-vba
'programming environment didn't like the syntax and highlighted everything red.

'Public Sub Main()
   'Dim original As String = "aaabbb"
   'Console.WriteLine("The original string: '{0}'", original)
   'Dim modified As String = original.Insert(3, " ")
   'Console.WriteLine("The modified string: '{0}'", modified)
'End Sub

Does word-vba not support string class modifiers and properties, am I not initializing the string correctly, or is there some other problem?

Body:
modified = original.Insert(3, " ")

You're thinking in VB.NET, but writing VBA. Strings (or any primitive or UDT type) don't have members in VBA. Not your fault, finding official VBA documentation is getting harder every day, with every "VBA" search yielding results for VB.NET.

That original code is clearly VB.NET.

If you mean to concatenate 3 spaces in front of original, then what you want to do is this:

modified = String(3, " ") & original

If you mean to get a new string in which a specified string is inserted at a specified index position in this instance (MSDN), then you want to do this (thanks @A.S.H!):

modified = Left$(original, 3) & " " & Right$(original, Len(original) - 3)


问:
Find number of non-zero elements in a tensor along an aixs
I want to find the number of non-zero elements in a tensor along a particular axis. Is there any PyTorch function which can do this?

I tried to use the nonzero() method in PyTorch.

```<code>torch.nonzero(losses).size(0)
</code>```

Here, lossess is a tensor of shape 64 x 1. When I run the above statement, it gives me the following error.

```<code>TypeError: Type Variable doesn't implement stateless method nonzero
</code>```

But if I run, torch.nonzero(losses.data).size(0), then it works fine. Any clue, why this is happening or what the error means?

答: Meaning of the error message - TypeError: Type Variable doesn't implement stateless method nonzero is, we cannot use torch.nonzero() on autograd.Variable but only on simple tensors. Also it should be noted that, tensors are stateless while the Variables are stateful.

Body:
I want to find the number of non-zero elements in a tensor along a particular axis. Is there any PyTorch function which can do this?

I tried to use the nonzero() method in PyTorch.

torch.nonzero(losses).size(0)

Here, lossess is a tensor of shape 64 x 1. When I run the above statement, it gives me the following error.

TypeError: Type Variable doesn't implement stateless method nonzero

But if I run, torch.nonzero(losses.data).size(0), then it works fine. Any clue, why this is happening or what the error means?


问:
vim/nvim line spacing when printing
I'm trying to use nvim for everything, including writing, and all is great but the text is too crammed when I print it out.

Is it possible to adjust the line spacing when printing with :hardcopy?

My idea of a kludge fix would be to insert a second newline character for every carriage return, including those automatically inserted by line wrapping. Is this possible?

答:
One of the least frustrating ways to do it would be to use a tool like pandoc to convert the text (for example Markdown) to whatever format you need — preferably PDF. However, pandoc uses LaTeX to create the resulting PDF, so regarding the styling you would have to tinker with supplying it a template or other options.

You can do it even easier with (for example) a Node.js tool called mdpdf. After installation, just run

```<code>mdpdf file.md --style styles.css
</code>```

to supply it with a CSS stylesheet in which you can modify the resulting text output with every feature CSS permits. Using larger line spacing would be something like this:

```<code>body { line-spacing: 150%; }
</code>```

This results in a 1.5x line spacing for everything in the document.

Of course, you can also set up a custom Vim command to automate this for you, putting something like the following into your .vimrc:

```<code>command MdToPDF !mdpdf %:t --style /full/path/to/styles.css
</code>```

Calling :MdToPDF in Vim will then run that command for you.

Finally, if you're happy with the output, just print the PDF.

Title: vim/nvim line spacing when printing

Body:
One of the least frustrating ways to do it would be to use a tool like pandoc to convert the text (for example Markdown) to whatever format you need — preferably PDF. However, pandoc uses LaTeX to create the resulting PDF, so regarding the styling you would have to tinker with supplying it a template or other options.

You can do it even easier with (for example) a Node.js tool called mdpdf. After installation, just run

mdpdf file.md --style styles.css

to supply it with a CSS stylesheet in which you can modify the resulting text output with every feature CSS permits. Using larger line spacing would be something like this:

body { line-spacing: 150%; }

This results in a 1.5x line spacing for everything in the document.

Of course, you can also set up a custom Vim command to automate this for you, putting something like the following into your .vimrc:

command MdToPDF !mdpdf %:t --style /full/path/to/styles.css

Calling :MdToPDF in Vim will then run that command for you.

Finally, if you're happy with the output, just print the PDF.


问:
Multi-series bar chart in DC-js
I'm using DC.js ( lib on top of D3 ) and have a great example of a single series bar chart:

```<code> var xf = crossfilter(data);
 var dim = xf.dimension(function (d) { return d["EmployeeName"]; });
 var group = dim.group().reduceSum(function (d) { return d["AverageSale"]; });

 var chart = dc.barChart(elm);
 chart.barPadding(0.1)
 chart.outerPadding(0.05)
 chart.brushOn(false)
 chart.x(d3.scale.ordinal());
 chart.xUnits(dc.units.ordinal);
 chart.elasticY(true);

 chart.dimension(dim).group(group);
 chart.render();
</code>```

but I was wondering if it was possible to create a multi dimensional bar chart using this library.  For example: Group by Store Name then Group By Employee then y-axis display average sale value ( already calculated by my backend ).

The data looks like:

```<code> [{
    "EmployeeName": "Heather",
    "StoreName" : "Plaza",
    "AverageSaleValue": 200
 }{
    "EmployeeName": "Mellisa",
    "StoreName" : "Plaza",
    "AverageSaleValue": 240
 }, {
    "EmployeeName": "Sarah",
    "StoreName" : "Oak Park",
    "AverageSaleValue": 300
 } ... ]
</code>```

答:
I know I’m late for this but it might help someone else.

To create grouped-bar-chart in dc.js without overwrite the original dc code you can take advantage of ‘pretransition’ event and split the bars to create a group.

I've created an example (jsfiddle)

The magic happens here:

```<code>let scaleSubChartBarWidth = chart => {
  let subs = chart.selectAll(".sub");

  if (typeof barPadding === 'undefined') { // first draw only
    // to percentage
    barPadding = BAR_PADDING / subs.size() * 100;
    // each bar gets half the padding
    barPadding = barPadding / 2;
  }

  let startAt, endAt,
      subScale = d3.scale.linear().domain([0, subs.size()]).range([0, 100]);

  subs.each(function (d, i) {

    startAt = subScale(i + 1) - subScale(1);
    endAt = subScale(i + 1);

    startAt += barPadding;
    endAt -= barPadding;

    // polygon(
    //  top-left-vertical top-left-horizontal,
    //  top-right-vertical top-right-horizontal,
    //  bottom-right-vertical bottom-right-horizontal,
    //  bottom-left-vertical bottom-left-horizontal,
    // )

    d3.select(this)
      .selectAll('rect')
      .attr("clip-path", `polygon(${startAt}% 0, ${endAt}% 0, ${endAt}% 100%, ${startAt}% 100%)`);

  });
};
</code>```

...

```<code>.on("pretransition", chart => {
    scaleSubChartBarWidth(chart);
})
</code>```

Complete code:

markup

```<code><div id="chart-container"></div>
</code>```

Js

```<code>//'use strict';
let compositeChart = dc.compositeChart("#chart-container");

const BAR_PADDING = .1; // percentage the padding will take from the bar
const RANGE_BAND_PADDING = .5; // padding between 'groups'
const OUTER_RANGE_BAND_PADDING = 0.5; // padding from each side of the chart

let sizing = chart => {
  chart
    .width(window.innerWidth)
    .height(window.innerHeight)
    .redraw();
};

let resizing = chart => window.onresize = () => sizing(chart);

let barPadding;
let scaleSubChartBarWidth = chart => {
  let subs = chart.selectAll(".sub");

  if (typeof barPadding === 'undefined') { // first draw only
    // to percentage
    barPadding = BAR_PADDING / subs.size() * 100;
    // each bar gets half the padding
    barPadding = barPadding / 2;
  }

  let startAt, endAt,
      subScale = d3.scale.linear().domain([0, subs.size()]).range([0, 100]);

  subs.each(function (d, i) {

    startAt = subScale(i + 1) - subScale(1);
    endAt = subScale(i + 1);

    startAt += barPadding;
    endAt -= barPadding;

    // polygon(
    //  top-left-vertical top-left-horizontal,
    //  top-right-vertical top-right-horizontal,
    //  bottom-right-vertical bottom-right-horizontal,
    //  bottom-left-vertical bottom-left-horizontal,
    // )

    d3.select(this)
      .selectAll('rect')
      .attr("clip-path", `polygon(${startAt}% 0, ${endAt}% 0, ${endAt}% 100%, ${startAt}% 100%)`);

  });
};

let data = [
  {
    key: "First",
    value: [
      {key: 1, value: 0.18},
      {key: 2, value: 0.28},
      {key: 3, value: 0.68}
    ]
  },
  {
    key: "Second",
    value: [
      {key: 1, value: 0.72},
      {key: 2, value: 0.32},
      {key: 3, value: 0.82}
    ]
  },
  {
    key: "Third",
    value: [
      {key: 1, value: 0.3},
      {key: 2, value: 0.22},
      {key: 3, value: 0.7}
    ]
  },
  {
    key: "Fourth",
    value: [
      {key: 1, value: 0.18},
      {key: 2, value: 0.58},
      {key: 3, value: 0.48}
    ]
  }
];

let ndx = crossfilter(data),
    dimension = ndx.dimension(d => d.key),
    group = {all: () => data}; // for simplicity sake (take a look at crossfilter group().reduce())

let barChart1 = dc.barChart(compositeChart)
    .barPadding(0)
    .valueAccessor(d => d.value[0].value)
    .title(d => d.key + `[${d.value[0].key}]: ` + d.value[0].value)
    .colors(['#000']);

let barChart2 = dc.barChart(compositeChart)
    .barPadding(0)
    .valueAccessor(d => d.value[1].value)
    .title(d => d.key + `[${d.value[1].key}]: ` + d.value[1].value)
    .colors(['#57B4F0']);

let barChart3 = dc.barChart(compositeChart)
    .barPadding(0)
    .valueAccessor(d => d.value[2].value)
    .title(d => d.key + `[${d.value[2].key}]: ` + d.value[2].value)
    .colors(['#47a64a']);

compositeChart
  .shareTitle(false)
  .dimension(dimension)
  .group(group)
  ._rangeBandPadding(RANGE_BAND_PADDING)
  ._outerRangeBandPadding(OUTER_RANGE_BAND_PADDING)
  .x(d3.scale.ordinal())
  .y(d3.scale.linear().domain([0, 1]))
  .xUnits(dc.units.ordinal)
  .compose([barChart1, barChart2, barChart3])
  .on("pretransition", chart => {
  scaleSubChartBarWidth(chart)
})
  .on("filtered", (chart, filter) => {
  console.log(chart, filter);
})
  .on("preRedraw", chart => {
  chart.rescale();
})
  .on("preRender", chart => {
  chart.rescale();
})
  .render();

sizing(compositeChart);
resizing(compositeChart);
</code>```

It's not perfect but it could give a starting point.

Title: Multi-series bar chart in DC-js

Body:
I'm using DC.js ( lib on top of D3 ) and have a great example of a single series bar chart:

 var xf = crossfilter(data);
 var dim = xf.dimension(function (d) { return d["EmployeeName"]; });
 var group = dim.group().reduceSum(function (d) { return d["AverageSale"]; });

 var chart = dc.barChart(elm);
 chart.barPadding(0.1)
 chart.outerPadding(0.05)
 chart.brushOn(false)
 chart.x(d3.scale.ordinal());
 chart.xUnits(dc.units.ordinal);
 chart.elasticY(true);

 chart.dimension(dim).group(group);
 chart.render();

but I was wondering if it was possible to create a multi dimensional bar chart using this library.  For example: Group by Store Name then Group By Employee then y-axis display average sale value ( already calculated by my backend ).

The data looks like:

 [{
    "EmployeeName": "Heather",
    "StoreName" : "Plaza",
    "AverageSaleValue": 200
 }{
    "EmployeeName": "Mellisa",
    "StoreName" : "Plaza",
    "AverageSaleValue": 240
 }, {
    "EmployeeName": "Sarah",
    "StoreName" : "Oak Park",
    "AverageSaleValue": 300
 } ... ]

Body:
I know I’m late for this but it might help someone else.

To create grouped-bar-chart in dc.js without overwrite the original dc code you can take advantage of ‘pretransition’ event and split the bars to create a group.

I've created an example (jsfiddle)

The magic happens here:

let scaleSubChartBarWidth = chart => {
  let subs = chart.selectAll(".sub");

  if (typeof barPadding === 'undefined') { // first draw only
    // to percentage
    barPadding = BAR_PADDING / subs.size() * 100;
    // each bar gets half the padding
    barPadding = barPadding / 2;
  }

  let startAt, endAt,
      subScale = d3.scale.linear().domain([0, subs.size()]).range([0, 100]);

  subs.each(function (d, i) {

    startAt = subScale(i + 1) - subScale(1);
    endAt = subScale(i + 1);

    startAt += barPadding;
    endAt -= barPadding;

    // polygon(
    //  top-left-vertical top-left-horizontal,
    //  top-right-vertical top-right-horizontal,
    //  bottom-right-vertical bottom-right-horizontal,
    //  bottom-left-vertical bottom-left-horizontal,
    // )

    d3.select(this)
      .selectAll('rect')
      .attr("clip-path", `polygon(${startAt}% 0, ${endAt}% 0, ${endAt}% 100%, ${startAt}% 100%)`);

  });
};

...

.on("pretransition", chart => {
    scaleSubChartBarWidth(chart);
})

Complete code:

markup

<div id="chart-container"></div>

Js

//'use strict';
let compositeChart = dc.compositeChart("#chart-container");

const BAR_PADDING = .1; // percentage the padding will take from the bar
const RANGE_BAND_PADDING = .5; // padding between 'groups'
const OUTER_RANGE_BAND_PADDING = 0.5; // padding from each side of the chart

let sizing = chart => {
  chart
    .width(window.innerWidth)
    .height(window.innerHeight)
    .redraw();
};

let resizing = chart => window.onresize = () => sizing(chart);

let barPadding;
let scaleSubChartBarWidth = chart => {
  let subs = chart.selectAll(".sub");

  if (typeof barPadding === 'undefined') { // first draw only
    // to percentage
    barPadding = BAR_PADDING / subs.size() * 100;
    // each bar gets half the padding
    barPadding = barPadding / 2;
  }

  let startAt, endAt,
      subScale = d3.scale.linear().domain([0, subs.size()]).range([0, 100]);

  subs.each(function (d, i) {

    startAt = subScale(i + 1) - subScale(1);
    endAt = subScale(i + 1);

    startAt += barPadding;
    endAt -= barPadding;

    // polygon(
    //  top-left-vertical top-left-horizontal,
    //  top-right-vertical top-right-horizontal,
    //  bottom-right-vertical bottom-right-horizontal,
    //  bottom-left-vertical bottom-left-horizontal,
    // )

    d3.select(this)
      .selectAll('rect')
      .attr("clip-path", `polygon(${startAt}% 0, ${endAt}% 0, ${endAt}% 100%, ${startAt}% 100%)`);

  });
};

let data = [
  {
    key: "First",
    value: [
      {key: 1, value: 0.18},
      {key: 2, value: 0.28},
      {key: 3, value: 0.68}
    ]
  },
  {
    key: "Second",
    value: [
      {key: 1, value: 0.72},
      {key: 2, value: 0.32},
      {key: 3, value: 0.82}
    ]
  },
  {
    key: "Third",
    value: [
      {key: 1, value: 0.3},
      {key: 2, value: 0.22},
      {key: 3, value: 0.7}
    ]
  },
  {
    key: "Fourth",
    value: [
      {key: 1, value: 0.18},
      {key: 2, value: 0.58},
      {key: 3, value: 0.48}
    ]
  }
];

let ndx = crossfilter(data),
    dimension = ndx.dimension(d => d.key),
    group = {all: () => data}; // for simplicity sake (take a look at crossfilter group().reduce())

let barChart1 = dc.barChart(compositeChart)
    .barPadding(0)
    .valueAccessor(d => d.value[0].value)
    .title(d => d.key + `[${d.value[0].key}]: ` + d.value[0].value)
    .colors(['#000']);

let barChart2 = dc.barChart(compositeChart)
    .barPadding(0)
    .valueAccessor(d => d.value[1].value)
    .title(d => d.key + `[${d.value[1].key}]: ` + d.value[1].value)
    .colors(['#57B4F0']);

let barChart3 = dc.barChart(compositeChart)
    .barPadding(0)
    .valueAccessor(d => d.value[2].value)
    .title(d => d.key + `[${d.value[2].key}]: ` + d.value[2].value)
    .colors(['#47a64a']);

compositeChart
  .shareTitle(false)
  .dimension(dimension)
  .group(group)
  ._rangeBandPadding(RANGE_BAND_PADDING)
  ._outerRangeBandPadding(OUTER_RANGE_BAND_PADDING)
  .x(d3.scale.ordinal())
  .y(d3.scale.linear().domain([0, 1]))
  .xUnits(dc.units.ordinal)
  .compose([barChart1, barChart2, barChart3])
  .on("pretransition", chart => {
  scaleSubChartBarWidth(chart)
})
  .on("filtered", (chart, filter) => {
  console.log(chart, filter);
})
  .on("preRedraw", chart => {
  chart.rescale();
})
  .on("preRender", chart => {
  chart.rescale();
})
  .render();

sizing(compositeChart);
resizing(compositeChart);

It's not perfect but it could give a starting point.


问:
Printing index of duplicate numbers in array javascript
I need to find the first duplicate number in array and then print index of this number in console. In my code below I reach the moment when code finds the duplicate but prints its index. In simple words, 2 is duplictaing in array but it's stands in first place so I need to print index "0".

var sameNum = [2, 4, 5, 2, 3, 5, 1, 2, 4];
var firstIndex = [];

for (var i = 0; i < sameNum.length; i++) {
  for (var j = i; j < sameNum.length; j++) {
    if (i != j && sameNum[i] == sameNum[j]) {
      firstIndex = [i];
    }
  }
}
console.log(firstIndex);```

答:
If I'm understanding the question correctly (you're looking for the first index of a number that occurs more than once), then the value of i is what you're looking for as that represents the index.</p>

```<code>if(i != j && sameNum[i] == sameNum[j]) {
  firstIndex.push(i);
  // or print it //
  console.log(i)
}
</code>```

Body:
If I'm understanding the question correctly (you're looking for the first index of a number that occurs more than once), then the value of i is what you're looking for as that represents the index.

if(i != j && sameNum[i] == sameNum[j]) {
  firstIndex.push(i);
  // or print it //
  console.log(i)
}


问:
How to build regex for following, i have done half?
http://con.google.com/video/daTUTOEl-OMBccnY9.mp4

I need to check for URL which has all this attribute

```<code>/https?:\/\/con\.google\.com\/video\//
</code>```

With above Regex I can get to video/ but after that I need to check if its seperated by a - or not daTUTOEl => 9characters both number and string
OMBccnY9 => 9characters both number and string.

答:
Those are 8 characters not 9,

Try following regex

```<code>https?:\/\/con\.google\.com\/video\/(?:\w+){8}-(?:\w+){8}.mp4
</code>```

See it working here

https://regex101.com/r/IvNKc1/1

Body:
http://con.google.com/video/daTUTOEl-OMBccnY9.mp4

I need to check for URL which has all this attribute

/https?:\/\/con\.google\.com\/video\//

With above Regex I can get to video/ but after that I need to check if its seperated by a - or not daTUTOEl => 9characters both number and string
OMBccnY9 => 9characters both number and string.

Body:
Those are 8 characters not 9,

Try following regex

https?:\/\/con\.google\.com\/video\/(?:\w+){8}-(?:\w+){8}.mp4

See it working here

https://regex101.com/r/IvNKc1/1


问:
Retain KDB Long Epoch Time When Importing Into Pandas
I have epoch times from KDB e.g. 530782044475144833. But any data type I have used in Pandas e.g. int, float64 severely round it up to 5.31e+17, rendering it useless. How do I retain the entire length of the KDB epoch time when reading into Pandas? Thanks in advance.

Body:
In the recent (> 4.0) versions of PyQ, kdb+ timestamp vectors can be easily converted to numpy arrays. For example:

>>> p = q('2#.z.P')
>>> p
k('2017.07.05D13:37:41.058130000 2017.07.05D13:37:41.058130000')
>>> p.long
k('552577061058130000 552577061058130000')
>>> a = np.array(p)
>>> a
array(['2017-07-05T13:37:41.058130000', '2017-07-05T13:37:41.058130000'], dtype='datetime64[ns]')

This does an automatic epoch conversion and does not loose any precision

>>> p == a
True

The resulting array can be cast into a panda's DatetimeIndex:

>>> pd.DatetimeIndex(a)
DatetimeIndex(['2017-07-05 13:37:41.058130', '2017-07-05 13:37:41.058130'], dtype='datetime64[ns]', freq=None)

or placed in a DataFrame:

>>> pd.DataFrame({'timestamp':a})
                   timestamp
0 2017-07-05 13:37:41.058130
1 2017-07-05 13:37:41.058130