问:
npm ERR! code Z_BUF_ERROR when install
In my server(CentOS 7.2) I install the dependencies:

```<code>npm install
</code>```

But I get bellow error:

```<code>npm ERR! code Z_BUF_ERROR
npm ERR! errno -5
npm ERR! unexpected end of file

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-02-11T21_03_20_261Z-debug.log
</code>```

in the /root/.npm/_logs/2018-02-11T21_03_20_261Z-debug.log, the info is bellow:

```<code>10234 verbose bundle EBUNDLEOVERRIDE: Replacing npm@1.4.29's bundled version of readable-stream with readable-stream@1.0.34.
10235 verbose unlock done using /root/.npm/_locks/staging-ace74a3b0cf47932.lock for /home/ubuntu/source_code_web/vue_admin_site/node_modules/.staging
10236 warn The package iview is included as both a dev and production dependency.
10237 warn npm@1.4.29 had bundled packages that do not match the required version(s). They have been replaced with non-bundled versions.
10238 verbose type OperationalError
10239 verbose stack Error: unexpected end of file
10239 verbose stack     at Gunzip.zlibOnError (zlib.js:152:15)
10240 verbose cwd /home/ubuntu/source_code_web/vue_admin_site
10241 verbose Linux 3.10.0-327.el7.x86_64
10242 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
10243 verbose node v8.4.0
10244 verbose npm  v5.3.0
10245 error code Z_BUF_ERROR
10246 error errno -5
10247 error unexpected end of file
10248 verbose exit [ -5, true ]
</code>```

I tried use：

```<code>npm cache clean
</code>```

to clean the npm, but fails:

```<code>npm ERR! As of npm@5, the npm cache self-heals from corruption issues and data extracted from the cache is guaranteed to be valid. If you want to make sure everything is consistent, use 'npm cache verify' instead.
npm ERR!
npm ERR! If you're sure you want to delete the entire cache, rerun this command with --force.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-02-11T21_13_51_943Z-debug.log
</code>```

答:
npm install --cache /tmp/empty-cache

This is what worked for me

Title: npm ERR! code Z_BUF_ERROR when install

Body:
In my server(CentOS 7.2) I install the dependencies:

npm install

But I get bellow error:

npm ERR! code Z_BUF_ERROR
npm ERR! errno -5
npm ERR! unexpected end of file

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-02-11T21_03_20_261Z-debug.log

in the /root/.npm/_logs/2018-02-11T21_03_20_261Z-debug.log, the info is bellow:

10234 verbose bundle EBUNDLEOVERRIDE: Replacing npm@1.4.29's bundled version of readable-stream with readable-stream@1.0.34.
10235 verbose unlock done using /root/.npm/_locks/staging-ace74a3b0cf47932.lock for /home/ubuntu/source_code_web/vue_admin_site/node_modules/.staging
10236 warn The package iview is included as both a dev and production dependency.
10237 warn npm@1.4.29 had bundled packages that do not match the required version(s). They have been replaced with non-bundled versions.
10238 verbose type OperationalError
10239 verbose stack Error: unexpected end of file
10239 verbose stack     at Gunzip.zlibOnError (zlib.js:152:15)
10240 verbose cwd /home/ubuntu/source_code_web/vue_admin_site
10241 verbose Linux 3.10.0-327.el7.x86_64
10242 verbose argv "/usr/local/bin/node" "/usr/local/bin/npm" "install"
10243 verbose node v8.4.0
10244 verbose npm  v5.3.0
10245 error code Z_BUF_ERROR
10246 error errno -5
10247 error unexpected end of file
10248 verbose exit [ -5, true ]

I tried use：

npm cache clean

to clean the npm, but fails:

npm ERR! As of npm@5, the npm cache self-heals from corruption issues and data extracted from the cache is guaranteed to be valid. If you want to make sure everything is consistent, use 'npm cache verify' instead.
npm ERR!
npm ERR! If you're sure you want to delete the entire cache, rerun this command with --force.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-02-11T21_13_51_943Z-debug.log


来源: stackexchange

问:
what's wrong in where condition statement
```<code>SqlConnection con =
   new SqlConnection
     (@"Data Source=SAMA-PC\SQLEXPRESS;Initial Catalog=advCenter;
          Integrated Security=True");
SqlCommand com1 = new SqlCommand(
     "select visited_link
      from links
      where [user_email]=@ue and [visited_link]=@vl",con);
com1.Parameters.AddWithValue("@ue",Convert.ToString(Session["mail"]));
com1.Parameters.AddWithValue("@vl", ImageButton1.ID);
con.Open();
SqlDataReader dr;
dr = com1.ExecuteReader();
if (dr.HasRows)
{
    Label2.Text = "wrong";
}
</code>```

答:
Use the following:

```<code>"... and  CAST([visited_link] AS NVARCHAR(MAX))=@vl "
</code>```

Refer to the following

CAST and CONVERT (Transact-SQL)

HOW TO: Compare Values in NTEXT Field

ContentLicense: CC BY-SA 3.0

Body:
Use the following:

"... and  CAST([visited_link] AS NVARCHAR(MAX))=@vl "

Refer to the following

CAST and CONVERT (Transact-SQL)

HOW TO: Compare Values in NTEXT Field


问:
Are operators a subset of statements?
Basically in all high-level languages (as I know) we have two main categories of language mechanisms to create a program: statements and expressions.

Usually statements are represented by some subset of language's keywords: if/else/switch, for/foreach/while, {} (or BEGIN/END), etc.

Expressions are represented by literals (which represent some data) and operators: literals: 1, 2, -100, testTest, etc; operators: +, -, /, *, ==, ===, etc.

If we think deeper, we can notice that statements usually answer on question "what?" and expressions -- on question "how?". Statements represent actions, expressions represent the context of actions.

Then we may look in expressions' parts again: literals and operators. Operators are actions too.

And here is my question again: are operators a subset of statement?

P.S. Generally, I understand that statements and expression are used together to aim some programming target. Separation of this categories is mostly theoretical.

答:
In general, "operator" describes a kind of syntactic form, which can be used to produce an expression, a statement, or some other class of language entity.  So, technically, the answer to your question is, "no".

For example, Haskell uses a | operator to generate an algebraic type spec, which is neither an expression nor a statement:

```<code>data Maybe a = Just a | Nothing
</code>```

ContentLicense: CC BY-SA 4.0

Title: Are operators a subset of statements?

Body:
In general, "operator" describes a kind of syntactic form, which can be used to produce an expression, a statement, or some other class of language entity.  So, technically, the answer to your question is, "no".

For example, Haskell uses a | operator to generate an algebraic type spec, which is neither an expression nor a statement:

data Maybe a = Just a | Nothing


问:
How to use Flutter to build a bottom sheet widget which is able to drag up to full screen?
I want to build a bottom sheet widget which is able to drag up to full screen.

Can anyone tell me how to do this?

Google map has a widget is able to do this. Here is the screenshot (check attached image):

Before drag up:

After drag up:

答:
Use the DraggableBottomSheet widget with the Stack widget:

Here's the gist for the entire page in this^ GIF, or try the Codepen.

Here's the structure of the entire page:

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Stack(
        children: <Widget>[
          CustomGoogleMap(),
          CustomHeader(),
          DraggableScrollableSheet(
            initialChildSize: 0.30,
            minChildSize: 0.15,
            builder: (BuildContext context, ScrollController scrollController) {
              return SingleChildScrollView(
                controller: scrollController,
                child: CustomScrollViewContent(),
              );
            },
          ),
        ],
      ),
    );
  }
```

<p><br>In the <code>Stack</code>:
<br> - The Google map is the lower most layer.
<br> - The Header (search field + horizontally scrolling chips) is above the map.
<br> - The DraggableBottomSheet is above the Header.</p>

<p>Some useful parameters as defined in <code>draggable_scrollable_sheet.dart</code>:</p>

<pre class="lang-dart prettyprint-override"><code>  /// The initial fractional value of the parent container's height to use when
  /// displaying the widget.
  ///
  /// The default value is `0.5`.
  final double initialChildSize;

  /// The minimum fractional value of the parent container's height to use when
  /// displaying the widget.
  ///
  /// The default value is `0.25`.
  final double minChildSize;

  /// The maximum fractional value of the parent container's height to use when
  /// displaying the widget.
  ///
  /// The default value is `1.0`.
  final double maxChildSize;
</code>```

Body:
Use the DraggableBottomSheet widget with the Stack widget:

Here's the gist for the entire page in this^ GIF, or try the Codepen.

Here's the structure of the entire page:

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      body: Stack(
        children: <Widget>[
          CustomGoogleMap(),
          CustomHeader(),
          DraggableScrollableSheet(
            initialChildSize: 0.30,
            minChildSize: 0.15,
            builder: (BuildContext context, ScrollController scrollController) {
              return SingleChildScrollView(
                controller: scrollController,
                child: CustomScrollViewContent(),
              );
            },
          ),
        ],
      ),
    );
  }

In the Stack:

 - The Google map is the lower most layer.

 - The Header (search field + horizontally scrolling chips) is above the map.

 - The DraggableBottomSheet is above the Header.

Some useful parameters as defined in draggable_scrollable_sheet.dart:

  /// The initial fractional value of the parent container's height to use when
  /// displaying the widget.
  ///
  /// The default value is `0.5`.
  final double initialChildSize;

  /// The minimum fractional value of the parent container's height to use when
  /// displaying the widget.
  ///
  /// The default value is `0.25`.
  final double minChildSize;

  /// The maximum fractional value of the parent container's height to use when
  /// displaying the widget.
  ///
  /// The default value is `1.0`.
  final double maxChildSize;


问:
Is there a way in Python to take a string and find the list it is associated with?
```<code>l1 = ["a","b","c"]
l2 = ["d","e","f"]

var1 = input("What list do you want to use? (l1 or l2)"

print (var1)
output: l1
</code>```

how do I make it so if the user types in l1 the output will be: ["a","b","c"] ?

The closest example I have found is:

```<code>x='buffalo'
exec("%s = %d" % (x,2))
</code>```

答:
```<code>if var1 == 'I1':
  print(I1)
elif var1 == 'I2':
  print(I2)
</code>```

That's just a basic if/else.

Body:
l1 = ["a","b","c"]
l2 = ["d","e","f"]

var1 = input("What list do you want to use? (l1 or l2)"

print (var1)
output: l1

how do I make it so if the user types in l1 the output will be: ["a","b","c"] ?

The closest example I have found is:

x='buffalo'
exec("%s = %d" % (x,2))

Body:
if var1 == 'I1':
  print(I1)
elif var1 == 'I2':
  print(I2)

That's just a basic if/else.


问:
Check if a products date is overdue
I want to create a kind of check, thats check if a products date is overdue. In my product table I have a column named veilingGesloten (auctionClosed) that can contain the value 'yes' or 'no'.

So I made this:

```<code>create TRIGGER [dbo].[trg_validategesloten]
ON [dbo].[voorwerp]
after update, insert
AS
begin
    set nocount on;

    update [dbo].[voorwerp]
        set veilinGesloten =
        case when ( inserted.looptijdeindeDag <= GETDATE() ) then 'wel'
        else 'niet'
        end
    from [dbo].[voorwerp]
    inner join inserted on [dbo].[voorwerp].voorwerpnummer = inserted.voorwerpnummer;
end
</code>```

Now im wondering how to actually make this an auto process without any insert or update events

答:
You don't trigger it - you use a computed column:

```<code>alter table voorwerp add veilinGesloten
    as (case when looptijdeindeDag <= getdate() then 'wel' else 'niet' end);
</code>```

Personally I would use a bit field and convert to meaningful words at the front-end.

```<code>alter table voorwerp add veilinGesloten
    as (convert(bit, case when looptijdeindeDag <= getdate() then 1 else 0 end));
</code>```

Title: Check if a products date is overdue

Body:
I want to create a kind of check, thats check if a products date is overdue. In my product table I have a column named veilingGesloten (auctionClosed) that can contain the value 'yes' or 'no'.

So I made this:

create TRIGGER [dbo].[trg_validategesloten]
ON [dbo].[voorwerp]
after update, insert
AS
begin
    set nocount on;

    update [dbo].[voorwerp]
        set veilinGesloten =
        case when ( inserted.looptijdeindeDag <= GETDATE() ) then 'wel'
        else 'niet'
        end
    from [dbo].[voorwerp]
    inner join inserted on [dbo].[voorwerp].voorwerpnummer = inserted.voorwerpnummer;
end

Now im wondering how to actually make this an auto process without any insert or update events

Body:
You don't trigger it - you use a computed column:

alter table voorwerp add veilinGesloten
    as (case when looptijdeindeDag <= getdate() then 'wel' else 'niet' end);

Personally I would use a bit field and convert to meaningful words at the front-end.

alter table voorwerp add veilinGesloten
    as (convert(bit, case when looptijdeindeDag <= getdate() then 1 else 0 end));


问:
EC2 Name tag and its position
I have this code, it works but I soon realized standards is not something we do. In my code I am getting the Tags value for 'Name'. However, the way i do it, I am assuming it is the first tag at position 0. I am wrong to assume. How can I make it more robust to just look for value of the key 'Name' irrelevant of position?</p>

```<code> response = client.describe_instances(Filters=[{'Name':'tag-key','Values':['Name']}])

for item in response['Reservations']:
    #pprint(item['Instances'])
        print("AWS Account ID: {}".format(item['OwnerId']))
        for instance_id in item['Instances']:
            #print(instance_id)
            Tags = instance_id['Tags'][0]['Value']
            State = instance_id['State']['Name']
            print("EC2 Name: {}".format(Tags))
            print("Instance Id is: {}\nInstance Type is: {}".format(instance_id['InstanceId'],instance_id['InstanceType']))

</code>```

答:
One way would be to iterate over Tags and search for the tag with Key of Name:

```<code>for item in response['Reservations']:

        print("AWS Account ID: {}".format(item['OwnerId']))

        for instance_id in item['Instances']:

            Tags = instance_id['Tags']

            tag_name_value = ""

            for tag in Tags:
                if tag['Key'] == "Name":
                    tag_name_value = tag["Value"]
                    break

            State = instance_id['State']['Name']

            print("EC2 Name: {}".format(tag_name_value))
            print("Instance Id is: {}\nInstance Type is: {}".format(
                    instance_id['InstanceId'],instance_id['InstanceType']))
</code>```

Title: EC2 Name tag and its position

Body:
I have this code, it works but I soon realized standards is not something we do. In my code I am getting the Tags value for 'Name'. However, the way i do it, I am assuming it is the first tag at position 0. I am wrong to assume. How can I make it more robust to just look for value of the key 'Name' irrelevant of position?

 response = client.describe_instances(Filters=[{'Name':'tag-key','Values':['Name']}])

for item in response['Reservations']:
    #pprint(item['Instances'])
        print("AWS Account ID: {}".format(item['OwnerId']))
        for instance_id in item['Instances']:
            #print(instance_id)
            Tags = instance_id['Tags'][0]['Value']
            State = instance_id['State']['Name']
            print("EC2 Name: {}".format(Tags))
            print("Instance Id is: {}\nInstance Type is: {}".format(instance_id['InstanceId'],instance_id['InstanceType']))

Body:
One way would be to iterate over Tags and search for the tag with Key of Name:

for item in response['Reservations']:

        print("AWS Account ID: {}".format(item['OwnerId']))

        for instance_id in item['Instances']:

            Tags = instance_id['Tags']

            tag_name_value = ""

            for tag in Tags:
                if tag['Key'] == "Name":
                    tag_name_value = tag["Value"]
                    break

            State = instance_id['State']['Name']

            print("EC2 Name: {}".format(tag_name_value))
            print("Instance Id is: {}\nInstance Type is: {}".format(
                    instance_id['InstanceId'],instance_id['InstanceType']))


问:
ggplot2 - a custom histogram with a rug plot
I am trying to create a custom histogram with a rug plot showing the original values on the X axis.

I am going to use the mtcars dataset to illustrate.  Its not be best dataset for this question...but hopefully the reader will understand what I am trying to achieve...

Below shows the basic histogram, without any rug plot attempt.

I want to create the histogram using geom_bar as this allows for more flexibility with custom bins.

I also want a small gap between the histgram bars (i.e width = 0.95) .... which adds to this
problem's complexity.

```<code>library(dplyr)
library(ggplot2)

# create custom bins
vct_seq <- c(seq(from = 10, to = 25, by = 5), 34)
mtcars$bin <- cut(mtcars$mpg, breaks = vct_seq)

# create data.frame for the ggplot graph..using bins above
df_mtcars_count <- mtcars %>% group_by(bin) %>% summarise(count = n())

# indicative labels
vct_labels <- c("bin 1", "bin 2", "bin 3", "bin 4")

# attempt 1 - basic plot -- no rug plot
p <- ggplot(data = df_mtcars_count, aes(x = bin, y = count))
p <- p + geom_bar(stat = "identity", width = 0.95)
p <- p + geom_text(aes(label = count), vjust = -0.5)
p <- p + scale_x_discrete("x title to go here", labels = df_mtcars_count$bin, breaks = df_mtcars_count$bin)
p

</code>```

Next, try and add a basic rug plot on the X axis.  This obviously doesn't work as the geom_bar and geom_rug have completely different scales.

```<code># attempt 2 with no scaling.... doesn't work as x scale for ordinal (bins) and
# x scale for continuous (mpg) do not match
p <- ggplot(data = df_mtcars_count, aes(x = bin, y = count))
p <- p + geom_bar(stat = "identity", width = 0.95)
p <- p + geom_text(aes(label = count), vjust = -0.5)
p <- p + scale_x_discrete("x title to go here", labels = df_mtcars_count$bin, breaks = df_mtcars_count$bin)
p <- p + geom_rug(data = mtcars, aes(x = mpg), inherit.aes = F, alpha = 0.3)
p
</code>```

Now, try and rescale the mpg column to match with the ordinal scale....

First define a linear mapping function...

```<code>fn_linear_map <- function(vct_existing_val, vct_new_range) {
  # example....converts 1:20 into the range 1 to 10 like this:
  # fn_linear_map(1:20, c(1, 10))
  fn_r_diff <- function(x) x %>% range() %>% diff()
  flt_ratio <- fn_r_diff(vct_new_range) / fn_r_diff(vct_existing_val)
  vct_old_min_offset <- vct_existing_val  - min(vct_existing_val)
  vct_new_range_val <- (vct_old_min_offset * flt_ratio) + min(vct_new_range)
  return(vct_new_range_val)
}
</code>```

Now apply the function...we try and map mpg to the range 1 to 4 (which is an attempt to match
the ordinal scale)

```<code>mtcars$mpg_remap <- fn_linear_map(mtcars$mpg, c(1, 4))
</code>```

Try the plot again.... getting closer ... but not really accurate...

```<code># attempt 3:  getting closer but doesn't really match the ordinal scale
p <- ggplot(data = df_mtcars_count, aes(x = bin, y = count))
p <- p + geom_bar(stat = "identity", width = 0.95)
p <- p + geom_text(aes(label = count), vjust = -0.5)
p <- p + scale_x_discrete("x title to go here", labels = df_mtcars_count$bin, breaks = df_mtcars_count$bin)
p <- p + geom_rug(data = mtcars, aes(x = mpg_remap), inherit.aes = F, alpha = 0.3)
p
</code>```

The graph above is getting close to what I want....but rug plot does not line up
with the actual data  ... example the max observation (33.9) should be displayed
almost aligning with the right hand side of the bar.. see below:

```<code>mtcars %>% filter(bin == "(25,34]") %>% arrange(mpg) %>% dplyr::select(mpg, mpg_remap)

</code>```

答:
Your scale makes no sense to me, as you are showing a bin that is twice as wide using the same bar width. Doing that in combination with a rug strikes me as confusing as best and misleading at worst. I suggest you plot the bars with their correct widths, after which the rug is trivial.

I think the best solution is to just use geom_histogram:

```<code>ggplot(mtcars, aes(mpg)) +
  geom_histogram(breaks = vct_seq, col = 'grey80') +
  geom_rug(aes(mpg, y = NULL))
</code>```

If you really want the gaps between the bars you'll have to do more work:

```<code>library(tidyr)
d <- mtcars %>%
  count(bin) %>%
  separate(bin, c('min', 'max'), sep = ',', remove = FALSE) %>%
  mutate_at(vars('min', 'max'), readr::parse_number) %>%
  mutate(
    middle = min + (max - min) / 2,
    width = 0.9 * (max - min)
  )

ggplot(d, aes(middle, n)) +
  geom_col(width = d$width) +
  geom_rug(aes(mpg, y = NULL), mtcars)
</code>```

Body:
I am trying to create a custom histogram with a rug plot showing the original values on the X axis.

I am going to use the mtcars dataset to illustrate.  Its not be best dataset for this question...but hopefully the reader will understand what I am trying to achieve...

Below shows the basic histogram, without any rug plot attempt.

I want to create the histogram using geom_bar as this allows for more flexibility with custom bins.

I also want a small gap between the histgram bars (i.e width = 0.95) .... which adds to this
problem's complexity.

library(dplyr)
library(ggplot2)

# create custom bins
vct_seq <- c(seq(from = 10, to = 25, by = 5), 34)
mtcars$bin <- cut(mtcars$mpg, breaks = vct_seq)

# create data.frame for the ggplot graph..using bins above
df_mtcars_count <- mtcars %>% group_by(bin) %>% summarise(count = n())

# indicative labels
vct_labels <- c("bin 1", "bin 2", "bin 3", "bin 4")

# attempt 1 - basic plot -- no rug plot
p <- ggplot(data = df_mtcars_count, aes(x = bin, y = count))
p <- p + geom_bar(stat = "identity", width = 0.95)
p <- p + geom_text(aes(label = count), vjust = -0.5)
p <- p + scale_x_discrete("x title to go here", labels = df_mtcars_count$bin, breaks = df_mtcars_count$bin)
p

Next, try and add a basic rug plot on the X axis.  This obviously doesn't work as the geom_bar and geom_rug have completely different scales.

# attempt 2 with no scaling.... doesn't work as x scale for ordinal (bins) and
# x scale for continuous (mpg) do not match
p <- ggplot(data = df_mtcars_count, aes(x = bin, y = count))
p <- p + geom_bar(stat = "identity", width = 0.95)
p <- p + geom_text(aes(label = count), vjust = -0.5)
p <- p + scale_x_discrete("x title to go here", labels = df_mtcars_count$bin, breaks = df_mtcars_count$bin)
p <- p + geom_rug(data = mtcars, aes(x = mpg), inherit.aes = F, alpha = 0.3)
p

Now, try and rescale the mpg column to match with the ordinal scale....

First define a linear mapping function...

fn_linear_map <- function(vct_existing_val, vct_new_range) {
  # example....converts 1:20 into the range 1 to 10 like this:
  # fn_linear_map(1:20, c(1, 10))
  fn_r_diff <- function(x) x %>% range() %>% diff()
  flt_ratio <- fn_r_diff(vct_new_range) / fn_r_diff(vct_existing_val)
  vct_old_min_offset <- vct_existing_val  - min(vct_existing_val)
  vct_new_range_val <- (vct_old_min_offset * flt_ratio) + min(vct_new_range)
  return(vct_new_range_val)
}

Now apply the function...we try and map mpg to the range 1 to 4 (which is an attempt to match
the ordinal scale)

mtcars$mpg_remap <- fn_linear_map(mtcars$mpg, c(1, 4))

Try the plot again.... getting closer ... but not really accurate...

# attempt 3:  getting closer but doesn't really match the ordinal scale
p <- ggplot(data = df_mtcars_count, aes(x = bin, y = count))
p <- p + geom_bar(stat = "identity", width = 0.95)
p <- p + geom_text(aes(label = count), vjust = -0.5)
p <- p + scale_x_discrete("x title to go here", labels = df_mtcars_count$bin, breaks = df_mtcars_count$bin)
p <- p + geom_rug(data = mtcars, aes(x = mpg_remap), inherit.aes = F, alpha = 0.3)
p

The graph above is getting close to what I want....but rug plot does not line up
with the actual data  ... example the max observation (33.9) should be displayed
almost aligning with the right hand side of the bar.. see below:

mtcars %>% filter(bin == "(25,34]") %>% arrange(mpg) %>% dplyr::select(mpg, mpg_remap)

Body:
Your scale makes no sense to me, as you are showing a bin that is twice as wide using the same bar width. Doing that in combination with a rug strikes me as confusing as best and misleading at worst. I suggest you plot the bars with their correct widths, after which the rug is trivial.

I think the best solution is to just use geom_histogram:

ggplot(mtcars, aes(mpg)) +
  geom_histogram(breaks = vct_seq, col = 'grey80') +
  geom_rug(aes(mpg, y = NULL))

If you really want the gaps between the bars you'll have to do more work:

library(tidyr)
d <- mtcars %>%
  count(bin) %>%
  separate(bin, c('min', 'max'), sep = ',', remove = FALSE) %>%
  mutate_at(vars('min', 'max'), readr::parse_number) %>%
  mutate(
    middle = min + (max - min) / 2,
    width = 0.9 * (max - min)
  )

ggplot(d, aes(middle, n)) +
  geom_col(width = d$width) +
  geom_rug(aes(mpg, y = NULL), mtcars)


问:
Azure Web app connect to VM service on private network
I've created windows server virtual machine with active directory controller and my web app is communicating with it on public ip successfully (by ldap), but when i try connect to ldap on private ip (virtual network) it does not respond

to my Web app i have set VNet Configuration

my virtual machine network tab:

But it does not work when i try to connect to ldap://10.0.1.4:389

What should i do?

I can't even test my connection between webapp (by tcpping) and vm neither on public and private ip

i couldn't add to my webapp vnet integration on 10.0.1.0/24 because there was written that this subnet is already in use (or sth like that)

what should i type when i do vnet integration?

答:
VNet Integration with Azure web app is required a dedicated unused subnet for the network connection. The subnet should be a subnet where no other resources like Azure VM located.

Your configuration is correct. You could follow these troubleshooting steps to verify if the VNet integration is working well. You can test via port 3389 on Azure VM. In my case, there is not a DC deployed on Azure VM.

However, please note that

There are some things that VNet Integration doesn't support, like:

Mounting a drive.

Active Directory integration.

NetBIOS.

Thus, I suppose Idap does not work with VNet Integration.


问:
How do I sort an R double by the second column, while preserving parings?
I have a double where the first column is a course name and the second column is a lambda value (these are coefficients from a penalized regression model). My double is a few thousand rows long, with most of the lambda values being zero. However, there are a few non-zero values. How can I filter this double so that only the non-zero coefficients remain? Here is what I have:

To see the coefficients with the minimum cross-validation error:

```<code>course_coef1 <- as.matrix(coef(lasso_reg, lasso_reg$lambda.min))

head(course_coef1)
                    1
(Intercept) 0.4170463
PHYS 1116   0.0000000
VISST 2511  0.0000000
MATH 1920   0.0000000
PHIL 1110   0.0000000
FREN 1220   0.0000000
</code>```

when I do this, I remove the left column of the double, which I don't want to do. I want to be able to see the course to which coefficient refers as well

```<code>non_zero <-  course_coef1[course_coef1[,1] != 0]

non_zero
 [1]  4.170463e-01  1.186766e-02  1.022153e-02 -1.728692e-02 -1.267802e-02  2.953045e-02 -7.366728e-04 -6.825617e-02  2.581637e-02  1.030888e-01
[11] -6.815507e-02 -6.177919e-04  3.138149e-02  1.297283e-05  7.753567e-02 -1.562090e-01 -2.301548e-01 -2.635691e-02 -1.382577e-02  1.487066e-02
[21] -3.922772e-04 -2.267470e-02 -2.668698e-02  3.372374e-02  2.309662e-02  4.383800e-02  8.291964e-03  2.643610e-04 -2.237277e-03 -3.068006e-04
</code>```

答:
Two problems.

You are grabbing columns, since you have omitted the comma, try [... != 0,] (add the comma);

It might be dropping from a matrix to a vector, fix with ,drop=FALSE.

z <- as.matrix(coef(lm(mpg~disp+factor(cyl), data=mtcars)))
z
#                     [,1]
# (Intercept)  29.53476781
# disp         -0.02730864
# factor(cyl)6 -4.78584624
# factor(cyl)8 -4.79208587

z[z[,1] < 0]
# [1] -0.02730864 -4.78584624 -4.79208587
z[z[,1] < 0,]
#         disp factor(cyl)6 factor(cyl)8
#  -0.02730864  -4.78584624  -4.79208587
z[z[,1] < 0,, drop = FALSE]
#                     [,1]
# disp         -0.02730864
# factor(cyl)6 -4.78584624
# factor(cyl)8 -4.79208587
```

Body:
I have a double where the first column is a course name and the second column is a lambda value (these are coefficients from a penalized regression model). My double is a few thousand rows long, with most of the lambda values being zero. However, there are a few non-zero values. How can I filter this double so that only the non-zero coefficients remain? Here is what I have:

To see the coefficients with the minimum cross-validation error:

course_coef1 <- as.matrix(coef(lasso_reg, lasso_reg$lambda.min))

head(course_coef1)
                    1
(Intercept) 0.4170463
PHYS 1116   0.0000000
VISST 2511  0.0000000
MATH 1920   0.0000000
PHIL 1110   0.0000000
FREN 1220   0.0000000

when I do this, I remove the left column of the double, which I don't want to do. I want to be able to see the course to which coefficient refers as well

non_zero <-  course_coef1[course_coef1[,1] != 0]

non_zero
 [1]  4.170463e-01  1.186766e-02  1.022153e-02 -1.728692e-02 -1.267802e-02  2.953045e-02 -7.366728e-04 -6.825617e-02  2.581637e-02  1.030888e-01
[11] -6.815507e-02 -6.177919e-04  3.138149e-02  1.297283e-05  7.753567e-02 -1.562090e-01 -2.301548e-01 -2.635691e-02 -1.382577e-02  1.487066e-02
[21] -3.922772e-04 -2.267470e-02 -2.668698e-02  3.372374e-02  2.309662e-02  4.383800e-02  8.291964e-03  2.643610e-04 -2.237277e-03 -3.068006e-04


问:
Is the working directory of the dagster main process different of the scheduler processes
I'm having an issue with the loading of a file from dagster code (setup, not pipelines). Say I have the following project structure:

```<code>pipelines
-app/
--environments
----schedules.yaml
--repository.py
--repository.yaml
</code>```

When I run dagit while inside the project folder($cd project && dagit -y app/repository.yaml), this folder becomes the working dir and inside the repository.py I could load a file knowing the root is project

# repository.py

with open('app/evironments/schedules.yaml', 'r'):
   # do something with the file
```

<p>However, if I set up a schedule the pipelines in the project do not run. Checking the cron logs it seems the <code>open</code> line throws a file not found exception. I was wondering if this happens because the working directory is different when executing the cron.</p>

<p>For context, I'm loading a config file with parameters of cron_schedules for each pipeline. Also, here's the tail of the stacktrace in my case:</p>

```  File "/home/user/.local/share/virtualenvs/pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/core/definitions/handle.py", line 190, in from_yaml
    return LoaderEntrypoint.from_file_target(
  File "/home/user/.local/share/virtualenvs/pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/core/definitions/handle.py", line 161, in from_file_target
    module = import_module_from_path(module_name, os.path.abspath(python_file))
  File "/home/user/.local/share/virtualenvs/pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/seven/__init__.py", line 75, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/user/pipelines/app/repository.py", line 28, in <module>
    schedule_builder = ScheduleBuilder(settings.CRON_PRESET, settings.ENV_DICT)
  File "/home/user/pipelines/app/schedules.py", line 12, in __init__
    self.cron_schedules = self._load_schedules_yaml()
  File "/home/user/pipelines/app/schedules.py", line 16, in _load_schedules_yaml
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'app/environments/schedules.yaml'
```

答:
You could open the file using the absolute path of the file so that it opens correctly.

```<code>from dagster.utils import file_relative_path

with open(file_relative_path(__file__, './environments/schedules.yaml'), 'r'):
   # do something with the file
</code>```

All file_relative_path is simply doing the following, so you can call the os.path methods directly if you prefer:

```<code>def file_relative_path(dunderfile, relative_path):
    os.path.join(os.path.dirname(dunderfile), relative_path)
</code>```

Body:
I'm having an issue with the loading of a file from dagster code (setup, not pipelines). Say I have the following project structure:

pipelines
-app/
--environments
----schedules.yaml
--repository.py
--repository.yaml

When I run dagit while inside the project folder($cd project && dagit -y app/repository.yaml), this folder becomes the working dir and inside the repository.py I could load a file knowing the root is project

# repository.py

with open('app/evironments/schedules.yaml', 'r'):
   # do something with the file

However, if I set up a schedule the pipelines in the project do not run. Checking the cron logs it seems the open line throws a file not found exception. I was wondering if this happens because the working directory is different when executing the cron.

For context, I'm loading a config file with parameters of cron_schedules for each pipeline. Also, here's the tail of the stacktrace in my case:

  File "/home/user/.local/share/virtualenvs/pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/core/definitions/handle.py", line 190, in from_yaml
    return LoaderEntrypoint.from_file_target(
  File "/home/user/.local/share/virtualenvs/pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/core/definitions/handle.py", line 161, in from_file_target
    module = import_module_from_path(module_name, os.path.abspath(python_file))
  File "/home/user/.local/share/virtualenvs/pipelines-mfP13m0c/lib/python3.8/site-packages/dagster/seven/__init__.py", line 75, in import_module_from_path
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/user/pipelines/app/repository.py", line 28, in <module>
    schedule_builder = ScheduleBuilder(settings.CRON_PRESET, settings.ENV_DICT)
  File "/home/user/pipelines/app/schedules.py", line 12, in __init__
    self.cron_schedules = self._load_schedules_yaml()
  File "/home/user/pipelines/app/schedules.py", line 16, in _load_schedules_yaml
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: 'app/environments/schedules.yaml'

Body:
You could open the file using the absolute path of the file so that it opens correctly.

from dagster.utils import file_relative_path

with open(file_relative_path(__file__, './environments/schedules.yaml'), 'r'):
   # do something with the file

All file_relative_path is simply doing the following, so you can call the os.path methods directly if you prefer:

def file_relative_path(dunderfile, relative_path):
    os.path.join(os.path.dirname(dunderfile), relative_path)


答:
A dictionary would be a great solution here.

```<code>mapping = {
    "l1": ["a","b","c"],
    "l2": ["d","e","f"]
}

var1 = input("What list do you want to use? (l1 or l2)")

mapping[var1]  # this is the list.
</code>```

If var1 is not a key in the dictionary, though, the above code will raise a KeyError. You can check for this and print a useful error message with a try-except block.

```<code>try:
    my_list = mapping[var1]
except KeyError:
    print("unknown input: {}".format(var1))
</code>```

Title: Is there a way in Python to take a string and find the list it is associated with?

Body:
A dictionary would be a great solution here.

mapping = {
    "l1": ["a","b","c"],
    "l2": ["d","e","f"]
}

var1 = input("What list do you want to use? (l1 or l2)")

mapping[var1]  # this is the list.

If var1 is not a key in the dictionary, though, the above code will raise a KeyError. You can check for this and print a useful error message with a try-except block.

try:
    my_list = mapping[var1]
except KeyError:
    print("unknown input: {}".format(var1))


问:
WebClient.UploadString appears to be passing a GET when I'm explicitly doing a POST
This is a weird one can I'm not sure what is happening.  I have a testing project that calls my api.  Recently, it appears that when I'm calling a none https website (ie http), it's changing my verb from a "POST" to a "GET".  My code is

```<code>response = client.UploadString(URL, "POST", myItemsObject);
</code>```

On my API, the first thing I do is log the request, including the verb.

答:
Doh!  I figured it out... On my web servers in the web.config I had

```<code><sessionState cookieless="AutoDetect" mode="InProc" />
</code>```

This was causing AspxAutoDetectCookieSupport=1 to be passed in the URL as part of the query string.  So it was hitting the web server as GET instead of a PUT.  I changed cookieless="false" and it works.

Body:
This is a weird one can I'm not sure what is happening.  I have a testing project that calls my api.  Recently, it appears that when I'm calling a none https website (ie http), it's changing my verb from a "POST" to a "GET".  My code is

response = client.UploadString(URL, "POST", myItemsObject);

On my API, the first thing I do is log the request, including the verb.

Body:
Doh!  I figured it out... On my web servers in the web.config I had

<sessionState cookieless="AutoDetect" mode="InProc" />

This was causing AspxAutoDetectCookieSupport=1 to be passed in the URL as part of the query string.  So it was hitting the web server as GET instead of a PUT.  I changed cookieless="false" and it works.


问:
How to use Google fonts without internet connection
Is there any way to download Google fonts and link them offline on localhost web page?</p>

答:
Try downloading the font folder and add this to your projectFolder and use font faces.

Check this https://answers.themler.io/articles/9064/how-to-use-google-fonts-locally


问:
How to Calculate number of hours left for the current date in Swift
I would like to count how many hours left till midnight. I have found tons of tutorials about making countdown timers, but I can't find any for calculating the left time.

Any ideas how should I do it?

Thanks!

答:
Gargoyle's code no longer builds. Here is Swift 5 production code (without the force unwrapping) that I've tested.  </p>

```<code>extension Date {
    var minutesToMidnight: Int {
        let calendar = NSCalendar.current
        guard let noonishTomorow = calendar.date(bySetting: Calendar.Component.hour, value: 12, of: Date()) else {
            return 0
        }
        let midnight = calendar.startOfDay(for: noonishTomorow)
        let components = calendar.dateComponents([.hour, .minute], from: Date(), to: midnight)
        var hoursUntilMidnight = components.hour ?? 0
        var minutesLeftInHour = components.minute ?? 0
            if minutesLeftInHour < 0 {
            minutesLeftInHour += 60
            hoursUntilMidnight -= 1
        }
        if hoursUntilMidnight < 0 {
            hoursUntilMidnight += 24
        }
        print("\(hoursUntilMidnight) hours and \(minutesLeftInHour) minutes till midnight")
        return hoursUntilMidnight * 60 + minutesLeftInHour
    }
}
</code>```

Body:
Gargoyle's code no longer builds. Here is Swift 5 production code (without the force unwrapping) that I've tested.

extension Date {
    var minutesToMidnight: Int {
        let calendar = NSCalendar.current
        guard let noonishTomorow = calendar.date(bySetting: Calendar.Component.hour, value: 12, of: Date()) else {
            return 0
        }
        let midnight = calendar.startOfDay(for: noonishTomorow)
        let components = calendar.dateComponents([.hour, .minute], from: Date(), to: midnight)
        var hoursUntilMidnight = components.hour ?? 0
        var minutesLeftInHour = components.minute ?? 0
            if minutesLeftInHour < 0 {
            minutesLeftInHour += 60
            hoursUntilMidnight -= 1
        }
        if hoursUntilMidnight < 0 {
            hoursUntilMidnight += 24
        }
        print("\(hoursUntilMidnight) hours and \(minutesLeftInHour) minutes till midnight")
        return hoursUntilMidnight * 60 + minutesLeftInHour
    }
}


问:
Trying to display 2 tables side by side but using Display: inline-table or float:left doesn't seem to work. Suggestions?
Overview:

I'm trying to get 2 tables to be aligned side by side. In another thread I found out that you can do that using display:inline-table or float: left. However when I added these elements to my CSS, nothing happens.

Here is how my code looks at the moment (also, apologies if my code is messy):

```<code><html>
<head>
<meta charset="utf-8">
<style>
body {
background-color:#FFFFFF;
}

table {
float: left;
font-family: tahoma, sans-serif;
border-collapse: collapse;
width: 100%;
}

td, th {
border: 0px solid #dddddd;
text-align: left;
padding: 8px;
}

tr:nth-child(even) {
background-color: #dddddd;
}
tr:hover {background-color:#FFDB80;}
</style>
</head>
<body>
<table cellpadding="10" cellspacing="0"  style="height: 100px; font-family:tahoma;">
<tbody>
<tr>
<td colspan="3" width="700px" valign="center" bgcolor="#192A34" style="color:white;"><font size="4">LIST OF ISBT 128 STANDARDS</font></td>
<td width="100px" valign="center" bgcolor="#192A34" style="color:white;"><font size="2"> 05 Nov 2018</font></td>
</tr>
<tr>
<td width="100px" valign="center" bgcolor="#941A1D" style="color:white;"><font size="2">Identification</font></td>
<td width="600" valign="center" bgcolor="#941A1D" style="color:white;"><font size="2">Document Title</font></td>
<td width="100" valign="center" bgcolor="#941A1D" style="color:white;"><font size="2">Active Version</font></td>
<td valign="center" bgcolor="#941A1D" style="color:white;"><font size="2">Date Issued</font></td>
</tr>
<tr>
<td> </td>
</tr>
<tr>
<td><font size="2">ST-001</font></td>
<td><font size="2">ISBT 128 Standard Technical Specification</font></td>
<td><font size="2">5.10.0</font></td>
<td><font size="2"> Jan 2019</font></td>
</tr>
</tbody>
</table>
<table cellpadding="10" cellspacing="0"  style="height: 100px; font-family:tahoma;">
<tbody>
<tr>
<td colspan="3" width="700px" valign="center" bgcolor="#192A34" style="color:white;"><font size="4">LIST OF ISBT 128 TECHNICAL BULLETINS</font></td>
<td width="100px" valign="center" bgcolor="#192A34" style="color:white;"><font size="2"> 05 Nov 2018</font></td>
</tr>
<tr>
<td width="100px" valign="center" bgcolor="#941A1D" style="color:white;"><font size="2">Identification</font></td>
<td width="600" valign="center" bgcolor="#941A1D" style="color:white;"><font size="2">Document Title</font></td>
<td width="100" valign="center" bgcolor="#941A1D" style="color:white;"><font size="2">Active Version</font></td>
<td valign="center" bgcolor="#941A1D" style="color:white;"><font size="2">Date Issued</font></td>
</tr>
<tr>
<td> </td>
</tr>
<tr>
<td><font size="2">TB-001</font></td>
<td><font size="2">Implementing ISBT 128: Guidance on Handling non-ISBT 128 Labeled Collections</font></td>
<td><font size="2">5.10.0</font></td>
<td><font size="2"> Jan 2019</font></td>
</tr>
</tbody>
</table>
</body>
</html>
</code>```

I suspect it has something to do with a lot of the td width elements and other html elements I injected into the table. I'm still a beginner so any suggestions are welcome to clean up the code.

答:
You have your width set on the tables as 100% so each of them is taking up the full width of the browser and they can't fit side by side. Reduce that to 50% and you'll see that works.

Note: , valign and bgcolor have been obsolete for many years. Do not use them.

Body:
Overview:

I'm trying to get 2 tables to be aligned side by side. In another thread I found out that you can do that using display:inline-table or float: left. However when I added these elements to my CSS, nothing happens.

Here is how my code looks at the moment (also, apologies if my code is messy):

<meta charset="utf-8">

LIST OF ISBT 128 STANDARDS
 05 Nov 2018

Identification
Document Title
Active Version
Date Issued

ST-001
ISBT 128 Standard Technical Specification
5.10.0
 Jan 2019

LIST OF ISBT 128 TECHNICAL BULLETINS
 05 Nov 2018

Identification
Document Title
Active Version
Date Issued

TB-001
Implementing ISBT 128: Guidance on Handling non-ISBT 128 Labeled Collections
5.10.0
 Jan 2019

I suspect it has something to do with a lot of the td width elements and other html elements I injected into the table. I'm still a beginner so any suggestions are welcome to clean up the code.

Body:
You have your width set on the tables as 100% so each of them is taking up the full width of the browser and they can't fit side by side. Reduce that to 50% and you'll see that works.

Note: <font>, valign and bgcolor have been obsolete for many years. Do not use them.


问:
AWS amplify TruncatingBigNumber can not be converted to an String
so, im trying to reset the password using the forgotPasswordSubmit from amplify, i pass as argument the security code previously asked that is sent to an email and the new password two times, when i send the values i get an error in the console:

```<code>ERROR Error: Uncaught (in promise): Object: {"code":"SerializationException","name":"SerializationException","message":"class com.amazon.coral.value.json.numbers.TruncatingBigNumber can not be converted to an String"}
    at resolvePromise (zone-evergreen.js:798)
    at resolvePromise (zone-evergreen.js:750)
    at zone-evergreen.js:860
    at ZoneDelegate.invokeTask (zone-evergreen.js:399)
    at Object.onInvokeTask (core.js:41264)
    at ZoneDelegate.invokeTask (zone-evergreen.js:398)
    at Zone.runTask (zone-evergreen.js:167)
    at drainMicroTaskQueue (zone-evergreen.js:569)
</code>```

This is my html code:

```<code><div *ngIf="newPassword">
 <form  [formGroup]="passwordForm"(ngSubmit)="verifyPassword(passwordForm.value)">
  <class="form-group">
   <label for="">Código</label>
   <input type="number" class="form-control" id="exampleInputverifycode1"  #verifycode formControlName = "code" placeholder="Ingresa el código">
   </div>
   <div>
    <label for="">Contraseña</label>
    <input type="password" name="password" id="exampleInputverifycode1"  #verifycode formControlName = "password" placeholder="Ingresa tu contraseña">
   </div>
   <div>
    <label for="">Contraseña</label>
    <input type="password" name="password" id="exampleInputverifycode1"  #verifycode formControlName = "passwordRepeat" placeholder="vuelve a escribir tu contraseña">
   </div>
   <div class="modal-footer">
    <button role="button" class="btn btn-forms-submit w-100" >Aceptar</button>
   </div>
</form>
</code>```

And this is the code that verifies the values given through the form:

```<code>    passwordForm = new FormGroup({
     code: new FormControl('', [Validators.required]),
   password: new FormControl('',[Validators.required]),
     passwordRepeat: new FormControl('',[Validators.required])
   })
    verifyPassword(value){
     console.log(value);
     Auth.forgotPasswordSubmit(value.code,value.password, value.passwordRepeat)
      .then(data=> console.log(data)
      )
    }
</code>```

答: Try changing your number field to text for your security code (exampleInputverifycode1 in your example). I saw the same issue when using the verifyTotpToken method from the aws-amplify with the Formik library and this issue pointed to a solution. If you try converting the numbers to a string further down the line from the form like I first tried, it's possible that e.g. 000123 will be interpreted as 123 causing a different type of error.  In summary, make sure your form is sending a string and not integers.

Body:
so, im trying to reset the password using the forgotPasswordSubmit from amplify, i pass as argument the security code previously asked that is sent to an email and the new password two times, when i send the values i get an error in the console:

ERROR Error: Uncaught (in promise): Object: {"code":"SerializationException","name":"SerializationException","message":"class com.amazon.coral.value.json.numbers.TruncatingBigNumber can not be converted to an String"}
    at resolvePromise (zone-evergreen.js:798)
    at resolvePromise (zone-evergreen.js:750)
    at zone-evergreen.js:860
    at ZoneDelegate.invokeTask (zone-evergreen.js:399)
    at Object.onInvokeTask (core.js:41264)
    at ZoneDelegate.invokeTask (zone-evergreen.js:398)
    at Zone.runTask (zone-evergreen.js:167)
    at drainMicroTaskQueue (zone-evergreen.js:569)

This is my html code:

 <form  [formGroup]="passwordForm"(ngSubmit)="verifyPassword(passwordForm.value)">
  <class="form-group">
   Código
   <input type="number" class="form-control" id="exampleInputverifycode1"  #verifycode formControlName = "code" placeholder="Ingresa el código">

    Contraseña
    <input type="password" name="password" id="exampleInputverifycode1"  #verifycode formControlName = "password" placeholder="Ingresa tu contraseña">

    Contraseña
    <input type="password" name="password" id="exampleInputverifycode1"  #verifycode formControlName = "passwordRepeat" placeholder="vuelve a escribir tu contraseña">

    <button role="button" class="btn btn-forms-submit w-100" >Aceptar</button>

</form>

And this is the code that verifies the values given through the form:

    passwordForm = new FormGroup({
     code: new FormControl('', [Validators.required]),
   password: new FormControl('',[Validators.required]),
     passwordRepeat: new FormControl('',[Validators.required])
   })
    verifyPassword(value){
     console.log(value);
     Auth.forgotPasswordSubmit(value.code,value.password, value.passwordRepeat)
      .then(data=> console.log(data)
      )
    }


问:
Choice of loss function
Here is a word2vec implementation:

```<code>%reset -f

import torch
from torch.autograd import Variable
import numpy as np
import torch.functional as F
import torch.nn.functional as F

corpus = [
    'this test',
    'this separate test'
]

def get_input_layer(word_idx):
    x = torch.zeros(vocabulary_size).float()
    x[word_idx] = 1.0
    return x

def tokenize_corpus(corpus):
    tokens = [x.split() for x in corpus]
    return tokens

tokenized_corpus = tokenize_corpus(corpus)

vocabulary = []
for sentence in tokenized_corpus:
    for token in sentence:
        if token not in vocabulary:
            vocabulary.append(token)

word2idx = {w: idx for (idx, w) in enumerate(vocabulary)}
idx2word = {idx: w for (idx, w) in enumerate(vocabulary)}

window_size = 2
idx_pairs = []
# for each sentence
for sentence in tokenized_corpus:
    indices = [word2idx[word] for word in sentence]
    # for each word, threated as center word
    for center_word_pos in range(len(indices)):
        # for each window position
        for w in range(-window_size, window_size + 1):
            context_word_pos = center_word_pos + w
            # make soure not jump out sentence
            if context_word_pos < 0 or context_word_pos >= len(indices) or center_word_pos == context_word_pos:
                continue
            context_word_idx = indices[context_word_pos]
            idx_pairs.append((indices[center_word_pos], context_word_idx))

idx_pairs = np.array(idx_pairs) # it will be useful to have this as numpy array

vocabulary_size = len(vocabulary)

embedding_dims = 4
W1 = Variable(torch.randn(embedding_dims, vocabulary_size).float(), requires_grad=True)
W2 = Variable(torch.randn(vocabulary_size, embedding_dims).float(), requires_grad=True)
num_epochs = 1
learning_rate = 0.001

for epo in range(num_epochs):
    loss_val = 0
    for data, target in idx_pairs:
        x = Variable(get_input_layer(data)).float()
        y_true = Variable(torch.from_numpy(np.array([target])).long())

        z1 = torch.matmul(W1, x)
        z2 = torch.matmul(W2, z1)

        log_softmax = F.log_softmax(z2, dim=0)

        loss = F.nll_loss(log_softmax.view(1,-1), y_true)
        print(float(loss))

        loss_val += loss.data.item()
        loss.backward()
        W1.data -= learning_rate * W1.grad.data
        W2.data -= learning_rate * W2.grad.data

        W1.grad.data.zero_()
        W2.grad.data.zero_()

        print(W1.shape)
        print(W2.shape)

        print(f'Loss at epo {epo}: {loss_val/len(idx_pairs)}')
</code>```

This prints:

```<code>0.33185482025146484
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.041481852531433105
3.302438735961914
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.45428669452667236
2.3144636154174805
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.7435946464538574
0.33418864011764526
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.7853682264685631
1.0644199848175049
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.9184207245707512
0.4970806837081909
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.980555810034275
3.2861199378967285
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 1.3913208022713661
6.170125961303711
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 2.16258654743433
</code>```

Modifying the code to use mse_loss change y_true to float :

```<code>y_true = Variable(torch.from_numpy(np.array([target])).float())
</code>```

Use mse_loss :

```<code>loss = F.mse_loss(log_softmax.view(1,-1), y_true)
</code>```

Incorporated updates:

```<code>for epo in range(num_epochs):
    loss_val = 0
    for data, target in idx_pairs:
        x = Variable(get_input_layer(data)).float()
        y_true = Variable(torch.from_numpy(np.array([target])).float())

        z1 = torch.matmul(W1, x)
        z2 = torch.matmul(W2, z1)

        log_softmax = F.log_softmax(z2, dim=0)

        loss = F.mse_loss(log_softmax.view(1,-1), y_true)
        print(float(loss))

        loss_val += loss.data.item()
        loss.backward()
        W1.data -= learning_rate * W1.grad.data
        W2.data -= learning_rate * W2.grad.data

        W1.grad.data.zero_()
        W2.grad.data.zero_()

        print(W1.shape)
        print(W2.shape)

        print(f'Loss at epo {epo}: {loss_val/len(idx_pairs)}')
</code>```

Output is now:

```<code>41.75048828125
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 5.21881103515625
16.929386138916016
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 7.334984302520752
50.63690948486328
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 13.664597988128662
36.21110534667969
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 18.190986156463623
5.304859638214111
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 18.854093611240387
9.802173614501953
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 20.07936531305313
15.515325546264648
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 22.018781006336212
30.408292770385742
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 25.81981760263443

-c:12: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 3])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
</code>```

Why does mse loss not work as well as nll loss ? Is it related to the PyTorch warning:

```<code>Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 3])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
</code>```

?

答:
The input and target must have the same size for the nn.MSELoss, because it is calculated by squaring the difference of the i-th element of the input and the i-th element of the target, i.e. mse_i = (input_i - target_i) ** 2.

Furthermore, your targets are non-negative integers in the range [0, vocabulary_size], but you use log-softmax, which has values in range [-∞, 0]. With MSE the idea is to get the prediction value to the same value as the target, but the only overlap of the two intervals is 0. That means that every class besides 0 is unreachable.

MSE is a regression loss function and is simply not appropriate in this situation, since you are dealing with categorical data.

Title: Choice of loss function

Body:
Here is a word2vec implementation:

%reset -f

import torch
from torch.autograd import Variable
import numpy as np
import torch.functional as F
import torch.nn.functional as F

corpus = [
    'this test',
    'this separate test'
]

def get_input_layer(word_idx):
    x = torch.zeros(vocabulary_size).float()
    x[word_idx] = 1.0
    return x

def tokenize_corpus(corpus):
    tokens = [x.split() for x in corpus]
    return tokens

tokenized_corpus = tokenize_corpus(corpus)

vocabulary = []
for sentence in tokenized_corpus:
    for token in sentence:
        if token not in vocabulary:
            vocabulary.append(token)

word2idx = {w: idx for (idx, w) in enumerate(vocabulary)}
idx2word = {idx: w for (idx, w) in enumerate(vocabulary)}

window_size = 2
idx_pairs = []
# for each sentence
for sentence in tokenized_corpus:
    indices = [word2idx[word] for word in sentence]
    # for each word, threated as center word
    for center_word_pos in range(len(indices)):
        # for each window position
        for w in range(-window_size, window_size + 1):
            context_word_pos = center_word_pos + w
            # make soure not jump out sentence
            if context_word_pos < 0 or context_word_pos >= len(indices) or center_word_pos == context_word_pos:
                continue
            context_word_idx = indices[context_word_pos]
            idx_pairs.append((indices[center_word_pos], context_word_idx))

idx_pairs = np.array(idx_pairs) # it will be useful to have this as numpy array

vocabulary_size = len(vocabulary)

embedding_dims = 4
W1 = Variable(torch.randn(embedding_dims, vocabulary_size).float(), requires_grad=True)
W2 = Variable(torch.randn(vocabulary_size, embedding_dims).float(), requires_grad=True)
num_epochs = 1
learning_rate = 0.001

for epo in range(num_epochs):
    loss_val = 0
    for data, target in idx_pairs:
        x = Variable(get_input_layer(data)).float()
        y_true = Variable(torch.from_numpy(np.array([target])).long())

        z1 = torch.matmul(W1, x)
        z2 = torch.matmul(W2, z1)

        log_softmax = F.log_softmax(z2, dim=0)

        loss = F.nll_loss(log_softmax.view(1,-1), y_true)
        print(float(loss))

        loss_val += loss.data.item()
        loss.backward()
        W1.data -= learning_rate * W1.grad.data
        W2.data -= learning_rate * W2.grad.data

        W1.grad.data.zero_()
        W2.grad.data.zero_()

        print(W1.shape)
        print(W2.shape)

        print(f'Loss at epo {epo}: {loss_val/len(idx_pairs)}')

This prints:

0.33185482025146484
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.041481852531433105
3.302438735961914
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.45428669452667236
2.3144636154174805
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.7435946464538574
0.33418864011764526
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.7853682264685631
1.0644199848175049
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.9184207245707512
0.4970806837081909
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 0.980555810034275
3.2861199378967285
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 1.3913208022713661
6.170125961303711
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 2.16258654743433

Modifying the code to use mse_loss change y_true to float :

y_true = Variable(torch.from_numpy(np.array([target])).float())

Use mse_loss :

loss = F.mse_loss(log_softmax.view(1,-1), y_true)

Incorporated updates:

for epo in range(num_epochs):
    loss_val = 0
    for data, target in idx_pairs:
        x = Variable(get_input_layer(data)).float()
        y_true = Variable(torch.from_numpy(np.array([target])).float())

        z1 = torch.matmul(W1, x)
        z2 = torch.matmul(W2, z1)

        log_softmax = F.log_softmax(z2, dim=0)

        loss = F.mse_loss(log_softmax.view(1,-1), y_true)
        print(float(loss))

        loss_val += loss.data.item()
        loss.backward()
        W1.data -= learning_rate * W1.grad.data
        W2.data -= learning_rate * W2.grad.data

        W1.grad.data.zero_()
        W2.grad.data.zero_()

        print(W1.shape)
        print(W2.shape)

        print(f'Loss at epo {epo}: {loss_val/len(idx_pairs)}')

Output is now:

41.75048828125
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 5.21881103515625
16.929386138916016
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 7.334984302520752
50.63690948486328
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 13.664597988128662
36.21110534667969
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 18.190986156463623
5.304859638214111
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 18.854093611240387
9.802173614501953
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 20.07936531305313
15.515325546264648
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 22.018781006336212
30.408292770385742
torch.Size([4, 3])
torch.Size([3, 4])
Loss at epo 0: 25.81981760263443

-c:12: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 3])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

Why does mse loss not work as well as nll loss ? Is it related to the PyTorch warning:

Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 3])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.

?


问:
Somewhat Infinite Output in Python
I have written Python code where I tried to contain all the possible combinations of words with or without meaning but it seems to be somewhat infinite [although it has limits]. I mean, using the letters abcd...wxyz to construct those possible words leading to 10 billion+ results. Is it possible to run this kind of code where you'll be having the expected output after some time? I have manage to make my own code but it makes my PC slower temporarily so I decided to terminate it before it totally crashes my OS.</p>

答:
Using itertools.product will get you what you want, and you can cycle through the many, MANY combinations at whatever speed you want. Here's an example of how to generate all the combinations efficiently:

```<code>import itertools
alphabet = "abcdefghijklmnopqrstuvwxyz"
length = 5
alphabets = [alphabet] * length
combos_generator = itertools.product(*alphabets)
print(len(list(combos_generator)))
</code>```

Note that the number of combinations will be 26 ^ num_letters, or in this case, over 11million combinations for just 5-letter words (aaaaa through zzzzz). If you actually want to access them efficiently, use the interator-style access:

```<code>for combo in combos_generator:
    <do something, output to file, etc>
    ...
</code>```

Happy Coding!

Title: Somewhat Infinite Output in Python

Body:
Using itertools.product will get you what you want, and you can cycle through the many, MANY combinations at whatever speed you want. Here's an example of how to generate all the combinations efficiently:

import itertools
alphabet = "abcdefghijklmnopqrstuvwxyz"
length = 5
alphabets = [alphabet] * length
combos_generator = itertools.product(*alphabets)
print(len(list(combos_generator)))

Note that the number of combinations will be 26 ^ num_letters, or in this case, over 11million combinations for just 5-letter words (aaaaa through zzzzz). If you actually want to access them efficiently, use the interator-style access:

for combo in combos_generator:
    <do something, output to file, etc>
    ...

Happy Coding!


问:
pandas select_dtypes not returning all columns of dtype
I'm encountering strange behavior with select_dtypes in Pandas.

I am using a JupyterNotebook v6.0.1 with Python 3.7.4 on Ubuntu Linux.

I'm working with a dataset of bulldozer auction results.

I started by importing the dataset into pandas dataframe from a csv file and dropped a couple of features that I knew that I would not want to use and limited the dataset by year because I wanted to use recent data.

```<code>drop_features = ['SalesID','ModelID','MachineID','fiProductClassDesc']

bulldozers_df = pd.read_csv("bluebook-for-bulldozers/Train.csv").drop(drop_features,axis=1)
bulldozers_df = bulldozers_df[pd.to_datetime(bulldozers_df['saledate']).dt.year > 2004]
</code>```

Then I listed out the types of the columns.

```<code>bulldozers_df.dtypes.value_counts()

object     44
int64       3
float64     2
dtype: int64

</code>```

Then I created a new synthetic column with a feature that I thought I would want to use.

```<code>bulldozers_df = bulldozers_df.assign(YearsMadeAgo = bulldozers_df['saledate'].astype('datetime64').dt.year - bulldozers_df['YearMade'].map(lambda x: bulldozers_df['YearMade'].mean() if pd.isna(x) else x))
</code>```

And then, I checked the columns again.

```<code>SalePrice                     int64
datasource                    int64
auctioneerID                float64
YearMade                      int64
MachineHoursCurrentMeter    float64
UsageBand                    object
saledate                     object
fiModelDesc                  object
fiBaseModel                  object

(... additional features...)

Travel_Controls              object
Differential_Type            object
Steering_Controls            object
YearsMadeAgo                  int64
</code>```

Then I decided to select the 'int64' columns. I expected to get 4 columns back: SalePrice, datasource,YearMade and YearsMadeAgo (the column that I made).

However, when I run the command, first I only get back SalePrice, datasource and YearMade.

Then, once I try again, pandas appears to break and just returns the index to me.

```<code>bulldozers_df.select_dtypes('int64')

SalePrice   datasource  YearMade
0   66000   121 2004
3   38500   121 2001
4   11000   121 2007
5   26500   121 2004
7   27000   121 2001
... ... ... ...
401120  10500   149 2005
401121  11000   149 2005
401122  11500   149 2005
401123  9000    149 2005
401124  7750    149 2005
</code>```

```<code>bulldozers_df.select_dtypes('int64')
0
3
4
5
7
...
401120
401121
401122
401123
401124
</code>```

Does anyone know what might be going on? df.select_dtypes appeared to work before I added the YearsMadeAgo column.

Just Hardware specs in case it is relevant, I have 24 GB of RAM with 20 GB free and 1.2% CPU usage with an Intel(R) Core(TM) i5-4460  CPU @ 3.20GHz.

答:
What environment are you using? Because pandas will visually truncate output to fit in a particular cell or terminal output.

Something like

```<code>bulldozers_df.info(verbose=True)
</code>```

should work

also something like this

```<code>with pd.option_context('display.max_rows', None, 'display.max_columns', None):
    print(df.dtypes)
</code>```

As shown here

Pandas printing ALL dtypes

Body:
I'm encountering strange behavior with select_dtypes in Pandas.

I am using a JupyterNotebook v6.0.1 with Python 3.7.4 on Ubuntu Linux.

I'm working with a dataset of bulldozer auction results.

I started by importing the dataset into pandas dataframe from a csv file and dropped a couple of features that I knew that I would not want to use and limited the dataset by year because I wanted to use recent data.

drop_features = ['SalesID','ModelID','MachineID','fiProductClassDesc']

bulldozers_df = pd.read_csv("bluebook-for-bulldozers/Train.csv").drop(drop_features,axis=1)
bulldozers_df = bulldozers_df[pd.to_datetime(bulldozers_df['saledate']).dt.year > 2004]

Then I listed out the types of the columns.

bulldozers_df.dtypes.value_counts()

object     44
int64       3
float64     2
dtype: int64

Then I created a new synthetic column with a feature that I thought I would want to use.

bulldozers_df = bulldozers_df.assign(YearsMadeAgo = bulldozers_df['saledate'].astype('datetime64').dt.year - bulldozers_df['YearMade'].map(lambda x: bulldozers_df['YearMade'].mean() if pd.isna(x) else x))

And then, I checked the columns again.

SalePrice                     int64
datasource                    int64
auctioneerID                float64
YearMade                      int64
MachineHoursCurrentMeter    float64
UsageBand                    object
saledate                     object
fiModelDesc                  object
fiBaseModel                  object

(... additional features...)

Travel_Controls              object
Differential_Type            object
Steering_Controls            object
YearsMadeAgo                  int64

Then I decided to select the 'int64' columns. I expected to get 4 columns back: SalePrice, datasource,YearMade and YearsMadeAgo (the column that I made).

However, when I run the command, first I only get back SalePrice, datasource and YearMade.

Then, once I try again, pandas appears to break and just returns the index to me.

bulldozers_df.select_dtypes('int64')

SalePrice   datasource  YearMade
0   66000   121 2004
3   38500   121 2001
4   11000   121 2007
5   26500   121 2004
7   27000   121 2001
... ... ... ...
401120  10500   149 2005
401121  11000   149 2005
401122  11500   149 2005
401123  9000    149 2005
401124  7750    149 2005

bulldozers_df.select_dtypes('int64')
0
3
4
5
7
...
401120
401121
401122
401123
401124

Does anyone know what might be going on? df.select_dtypes appeared to work before I added the YearsMadeAgo column.

Just Hardware specs in case it is relevant, I have 24 GB of RAM with 20 GB free and 1.2% CPU usage with an Intel(R) Core(TM) i5-4460  CPU @ 3.20GHz.

Body:
What environment are you using? Because pandas will visually truncate output to fit in a particular cell or terminal output.

Something like

bulldozers_df.info(verbose=True)

should work

also something like this

with pd.option_context('display.max_rows', None, 'display.max_columns', None):
    print(df.dtypes)

As shown here

Pandas printing ALL dtypes


问:
Memory Leak in C#
Is it ever possible in a managed system to leak memory when you make sure that all handles, things that implement IDispose are disposed?

Would there be cases where some variables are left out?

答: Reflection emit is another potential source of leaks, with e.g. built-in object deserializers and fancy SOAP/XML clients. At least in earlier versions of the framework, generated code in dependent AppDomains was never unloaded...</p>

OwnerDisplayName: Joan Venge

ContentLicense: CC BY-SA 2.5

Title: Memory Leak in C#

LastEditorDisplayName: strager

OwnerDisplayName: Pontus Gagge


问:
"Re-announcing" service periodically when using WCF ServiceDiscoveryBehavior announcement endpoint?
I have a Managed Discovery Service hosted with a known URI. I have a discoverable service that when it starts, it announces itself using an AnnouncementEndpoint added to the ServiceDiscoveryBehavior of the service.

The specific use case I would like to solve is the following:

Managed Discovery service starts.

A discoverable service starts and announces itself to the Managed Discovery service.

The Managed Discovery service is restarted (for any various possible reasons).

How then does the discoverable service refresh itself (re-announce) to the Managed Discovery service?

I know the Managed Discovery service can persist endpoints and restore them upon start but I want everything to be dynamic and self repairing so that there's no chance of stale endpoint information.

An alternative use case would be:

An existing discoverable service is running.

A new Managed Discovery service is brought online.

How do we force or invoke the same Announcement service contract call to the new Managed Discovery service?

I hope this is enough information about what I want to accomplish.

答:
I found the answer myself. In the scenario where you need to control announcements outside of the ServiceDiscoveryBehavior , you would use the AnnouncementClient class.</p>

```<code>AnnouncementClient client = new AnnouncementClient(announcementEndpoint);
var endpointDiscoveryMetadata = EndpointDiscoveryMetadata.FromServiceEndpoint(netTcpEndpoint);

client.AnnounceOnline(endpointDiscoveryMetadata);
</code>```

Body:
I found the answer myself. In the scenario where you need to control announcements outside of the ServiceDiscoveryBehavior , you would use the AnnouncementClient class.

AnnouncementClient client = new AnnouncementClient(announcementEndpoint);
var endpointDiscoveryMetadata = EndpointDiscoveryMetadata.FromServiceEndpoint(netTcpEndpoint);

client.AnnounceOnline(endpointDiscoveryMetadata);


问:
Why am I not able to run sparkPi example on a Kubernetes (K8s) cluster?
I have a K8s cluster up and running, on VMs inside VMWare Workstation, as of now. I'm trying to deploy a Spark application natively using the official documentation from here. However, I also landed on this article which made it clearer, I felt.

Now, earlier my setup was running inside nested VMs, basically my machine is on Win10 and I had an Ubuntu VM inside which I had 3 more VMs running for the cluster (not the best idea, I know).

When I tried to run my setup by following the article mentioned, I first created a service account inside the cluster called spark, then created a clusterrolebinding called spark-role, gave edit as the clusterrole and assigned it to the spark service account so that Spark driver pod has sufficient permissions.

I then try to run the example SparkPi job using this command line:

```<code>bin/spark-submit \
  --master k8s://https://<k8-cluster-ip>:<k8-cluster-port> \
  --deploy-mode cluster \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=2 \
  --conf spark.kubernetes.container.image=kmaster:5000/spark:latest \
  --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 100
</code>```

And it fails within a few seconds after it has created the driver-pod, it goes into Running state and after like 3 seconds goes into Error state.

On giving the command kubectl logs spark-pi-driver this is the log I get.

The second Caused by: is always either as mentioned above i.e:

Caused by: java.net.SocketException: Broken pipe (Write failed) or,

Caused by: okhttp3.internal.http2.ConnectionShutdownException

Log #2 for reference.

After running into dead-ends with this, I tried giving --deploy-mode client to see if it makes a difference and get more verbose logs. You can read the difference between client and cluster mode from here.

On deploying the job as client mode it still fails, however, now I see that each time the driver pod (now running not as a pod but as a process on the local machine) tries to create an executor pod, it goes into a loop infinitely trying to create an executor pod with a count-number appended to the pod name, as the last one goes into a terminated state. Also, now I can see the Spark UI on the 4040 port but the job doesn't move forward as it's stuck on trying to create even a single executor pod.

I get this log.

To me, this makes it pretty apparent that it's a resource crunch maybe?

So to be sure, I delete the nested VMs and setup 2 new VMs on my main machine and connect them using a NAT network and setup the same K8s cluster.

But now when I try to do the exact same thing it fails with the same error (Broken Pipe/ShutdownException), except now it tells me that it fails even at creating a driver-pod.

This is the log for reference.

Now I can't even fetch logs as to why it fails, because it's never even created.

I've broken my head over this and can't figure out why it's failing. Now, I tried out a lot of things to rule them out but so far nothing has worked except one (which is a completely different solution).

I tried the spark-on-k8-operator from GCP from here and it worked for me. I wasn't able to see the Spark UI as it runs briefly but it prints the Pi value in the shell window, so I know it works.
I'm guessing, that even this spark-on-k8s-operator 'internally' does the same thing but I really need to be able to deploy it natively, or at least know why it fails.

Any help here will be appreciated (I know it's a long post). Thank you.

答:
Spark on K8s operator example uses a Spark image (from gcr.io) that works. You can find the image tag in spark-on-k8s-operator/examples/spark-pi.yaml

```<code>spec:
  ...
  image: "gcr.io/spark-operator/spark:v2.4.5"
...
</code>```

I tried to replace the image config in the bin/spark-submit and it worked for me.

```<code>bin/spark-submit \
--master k8s://https://192.168.99.100:8443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.driver.cores=1 \
--conf spark.driver.memory=512m \
--conf spark.executor.instances=2 \
--conf spark.executor.memory=512m \
--conf spark.executor.cores=1 \
--conf spark.kubernetes.container.image=gcr.io/spark-operator/spark:v2.4.5 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar
</code>```

Body:
I have a K8s cluster up and running, on VMs inside VMWare Workstation, as of now. I'm trying to deploy a Spark application natively using the official documentation from here. However, I also landed on this article which made it clearer, I felt.

Now, earlier my setup was running inside nested VMs, basically my machine is on Win10 and I had an Ubuntu VM inside which I had 3 more VMs running for the cluster (not the best idea, I know).

When I tried to run my setup by following the article mentioned, I first created a service account inside the cluster called spark, then created a clusterrolebinding called spark-role, gave edit as the clusterrole and assigned it to the spark service account so that Spark driver pod has sufficient permissions.

I then try to run the example SparkPi job using this command line:

bin/spark-submit \
  --master k8s://https://<k8-cluster-ip>:<k8-cluster-port> \
  --deploy-mode cluster \
  --name spark-pi \
  --class org.apache.spark.examples.SparkPi \
  --conf spark.executor.instances=2 \
  --conf spark.kubernetes.container.image=kmaster:5000/spark:latest \
  --conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar 100

And it fails within a few seconds after it has created the driver-pod, it goes into Running state and after like 3 seconds goes into Error state.

On giving the command kubectl logs spark-pi-driver this is the log I get.

The second Caused by: is always either as mentioned above i.e:

Caused by: java.net.SocketException: Broken pipe (Write failed) or,

Caused by: okhttp3.internal.http2.ConnectionShutdownException

Log #2 for reference.

After running into dead-ends with this, I tried giving --deploy-mode client to see if it makes a difference and get more verbose logs. You can read the difference between client and cluster mode from here.

On deploying the job as client mode it still fails, however, now I see that each time the driver pod (now running not as a pod but as a process on the local machine) tries to create an executor pod, it goes into a loop infinitely trying to create an executor pod with a count-number appended to the pod name, as the last one goes into a terminated state. Also, now I can see the Spark UI on the 4040 port but the job doesn't move forward as it's stuck on trying to create even a single executor pod.

I get this log.

To me, this makes it pretty apparent that it's a resource crunch maybe?

So to be sure, I delete the nested VMs and setup 2 new VMs on my main machine and connect them using a NAT network and setup the same K8s cluster.

But now when I try to do the exact same thing it fails with the same error (Broken Pipe/ShutdownException), except now it tells me that it fails even at creating a driver-pod.

This is the log for reference.

Now I can't even fetch logs as to why it fails, because it's never even created.

I've broken my head over this and can't figure out why it's failing. Now, I tried out a lot of things to rule them out but so far nothing has worked except one (which is a completely different solution).

I tried the spark-on-k8-operator from GCP from here and it worked for me. I wasn't able to see the Spark UI as it runs briefly but it prints the Pi value in the shell window, so I know it works.
I'm guessing, that even this spark-on-k8s-operator 'internally' does the same thing but I really need to be able to deploy it natively, or at least know why it fails.

Any help here will be appreciated (I know it's a long post). Thank you.

Body:
Spark on K8s operator example uses a Spark image (from gcr.io) that works. You can find the image tag in spark-on-k8s-operator/examples/spark-pi.yaml

spec:
  ...
  image: "gcr.io/spark-operator/spark:v2.4.5"
...

I tried to replace the image config in the bin/spark-submit and it worked for me.

bin/spark-submit \
--master k8s://https://192.168.99.100:8443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.driver.cores=1 \
--conf spark.driver.memory=512m \
--conf spark.executor.instances=2 \
--conf spark.executor.memory=512m \
--conf spark.executor.cores=1 \
--conf spark.kubernetes.container.image=gcr.io/spark-operator/spark:v2.4.5 \
--conf spark.kubernetes.container.image.pullPolicy=IfNotPresent \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
local:///opt/spark/examples/jars/spark-examples_2.11-2.4.5.jar


问:
Faster way to find the first empty row in a Google Sheet column
I've made a script that every few hours adds a new row to a Google Apps spreadsheet.

This is the function I've made to find the first empty row:

function getFirstEmptyRow() {
  var spr = SpreadsheetApp.getActiveSpreadsheet();
  var cell = spr.getRange('a1');
  var ct = 0;
  while ( cell.offset(ct, 0).getValue() != "" ) {
    ct++;
  }
  return (ct);
}
```

It works fine, but when reaching about 100 rows, it gets really slow, even ten seconds.
I'm worried that when reaching thousands of rows, it will be too slow, maybe going in timeout or worse.
Is there a better way?

答:
Here is a list of what the code should do:

Give a correct answer if there are no empty cells

Be fast

Return the correct row number - not the index number of the array

Get the correct row number of the empty cell even when other columns in the sheet tab have more rows with data

Have good variable names

Answer the original question

Avoid unnecessary data processing

Provide comment explanations for what the code does

Be generic enough to adapt to the readers conditions

This solution uses the array method some which will stop iterating the loop when the condition is true.  This avoids wasting time spent looping through every element of the array, and yet uses an array method rather than a for or while loop.

The some method only returns true or false, but there is a way to capture the index number because the some method halts looping when the condition is true.

The index number is assigned to a variable in the scope outside of the array function.  This does not slow down the processing.

Code:

```<code>function getFirstEmptyCellIn_A_Column(po) {
  var foundEmptyCell,rng,sh,ss,values,x;

  /*
    po.sheetTabName - The name of the sheet tab to get
    po.ssID - the file ID of the spreadsheet
    po.getActive - boolean - true - get the active spreadsheet -
  */

  /*  Ive tested the code for speed using many different ways to do this and using array.some
    is the fastest way - when array.some finds the first true statement it stops iterating -
  */

  if (po.getActive || ! po.ssID) {
    ss =  SpreadsheetApp.getActiveSpreadsheet();
  } else {
    ss = SpreadsheetApp.openById(po.ssID);
  }

  sh = ss.getSheetByName(po.sheetTabName);
  rng = sh.getRange('A:A');//This is the fastest - Its faster than getting the last row and getting a
  //specific range that goes only to the last row

  values = rng.getValues(); // get all the data in the column - This is a 2D array

  x = 0;//Set counter to zero - this is outside of the scope of the array function but still accessible to it

  foundEmptyCell = values.some(function(e,i){
    //Logger.log(i)
    //Logger.log(e[0])
    //Logger.log(e[0] == "")

    x = i;//Set the value every time - its faster than first testing for a reason to set the value
    return e[0] == "";//The first time that this is true it stops looping
  });

  //Logger.log('x + 1: ' + (x + 1))//x is the index of the value in the array - which is one less than the row number
  //Logger.log('foundEmptyCell: ' + foundEmptyCell)

  return foundEmptyCell ? x + 1 : false;
}

function testMycode() {

  getFirstEmptyCellIn_A_Column({"sheetTabName":"Put Sheet tab name here","ssID":"Put your ss file ID here"})

}
</code>```

Body:
I've made a script that every few hours adds a new row to a Google Apps spreadsheet.

This is the function I've made to find the first empty row:

function getFirstEmptyRow() {
  var spr = SpreadsheetApp.getActiveSpreadsheet();
  var cell = spr.getRange('a1');
  var ct = 0;
  while ( cell.offset(ct, 0).getValue() != "" ) {
    ct++;
  }
  return (ct);
}

It works fine, but when reaching about 100 rows, it gets really slow, even ten seconds.
I'm worried that when reaching thousands of rows, it will be too slow, maybe going in timeout or worse.
Is there a better way?

Body:
Here is a list of what the code should do:

Give a correct answer if there are no empty cells

Be fast

Return the correct row number - not the index number of the array

Get the correct row number of the empty cell even when other columns in the sheet tab have more rows with data

Have good variable names

Answer the original question

Avoid unnecessary data processing

Provide comment explanations for what the code does

Be generic enough to adapt to the readers conditions

This solution uses the array method some which will stop iterating the loop when the condition is true.  This avoids wasting time spent looping through every element of the array, and yet uses an array method rather than a for or while loop.

The some method only returns true or false, but there is a way to capture the index number because the some method halts looping when the condition is true.

The index number is assigned to a variable in the scope outside of the array function.  This does not slow down the processing.

Code:

function getFirstEmptyCellIn_A_Column(po) {
  var foundEmptyCell,rng,sh,ss,values,x;

  /*
    po.sheetTabName - The name of the sheet tab to get
    po.ssID - the file ID of the spreadsheet
    po.getActive - boolean - true - get the active spreadsheet -
  */

  /*  Ive tested the code for speed using many different ways to do this and using array.some
    is the fastest way - when array.some finds the first true statement it stops iterating -
  */

  if (po.getActive || ! po.ssID) {
    ss =  SpreadsheetApp.getActiveSpreadsheet();
  } else {
    ss = SpreadsheetApp.openById(po.ssID);
  }

  sh = ss.getSheetByName(po.sheetTabName);
  rng = sh.getRange('A:A');//This is the fastest - Its faster than getting the last row and getting a
  //specific range that goes only to the last row

  values = rng.getValues(); // get all the data in the column - This is a 2D array

  x = 0;//Set counter to zero - this is outside of the scope of the array function but still accessible to it

  foundEmptyCell = values.some(function(e,i){
    //Logger.log(i)
    //Logger.log(e[0])
    //Logger.log(e[0] == "")

    x = i;//Set the value every time - its faster than first testing for a reason to set the value
    return e[0] == "";//The first time that this is true it stops looping
  });

  //Logger.log('x + 1: ' + (x + 1))//x is the index of the value in the array - which is one less than the row number
  //Logger.log('foundEmptyCell: ' + foundEmptyCell)

  return foundEmptyCell ? x + 1 : false;
}

function testMycode() {

  getFirstEmptyCellIn_A_Column({"sheetTabName":"Put Sheet tab name here","ssID":"Put your ss file ID here"})

}


问:
Not able to reduce number of task in Pyspark
I am not able to reduce number of task . Spark always show 200 tasks in UI. I am not sure but is it because of default minimum partitions while reading data ? I tried all parameters like numPartitions , partitions, minPartitions in my command mentioned below but none of them worked and spark throwing an error like TypeError: csv() got an unexpected keyword argument 'partitions' .  I am using spark2. Please assist. Apologies I am not able to format code mention below.</p>

```<code>df_ord = spark.read.csv(props.get(env, 'input.dir') + "/orders", header=False,inferSchema=True,**numPartitions=10**) \
              .toDF("ord_id", "ord_dt", "cust_id", "ord_status")
</code>```

答:
Spark have a default property spark.sql.shuffle.partitions which is responsible for partitions. The default value of this is 200 that's why you are seeing 200 partitions on UI.

To change this you can pass conf while submitting the job or using below command

```<code>spark.conf.set("spark.sql.shuffle.partitions", 100)
</code>```

Alternatively you can use repartition or coalesce to reduce the number of partitions as per your requirement.

References:
https://spark.apache.org/docs/latest/sql-performance-tuning.html

Body:
I am not able to reduce number of task . Spark always show 200 tasks in UI. I am not sure but is it because of default minimum partitions while reading data ? I tried all parameters like numPartitions , partitions, minPartitions in my command mentioned below but none of them worked and spark throwing an error like TypeError: csv() got an unexpected keyword argument 'partitions' .  I am using spark2. Please assist. Apologies I am not able to format code mention below.

df_ord = spark.read.csv(props.get(env, 'input.dir') + "/orders", header=False,inferSchema=True,**numPartitions=10**) \
              .toDF("ord_id", "ord_dt", "cust_id", "ord_status")

Body:
Spark have a default property spark.sql.shuffle.partitions which is responsible for partitions. The default value of this is 200 that's why you are seeing 200 partitions on UI.

To change this you can pass conf while submitting the job or using below command

spark.conf.set("spark.sql.shuffle.partitions", 100)

Alternatively you can use repartition or coalesce to reduce the number of partitions as per your requirement.

References:
https://spark.apache.org/docs/latest/sql-performance-tuning.html


问:
Is there a way in Python to take a string and find the list it is associated with?
```<code>l1 = ["a","b","c"]
l2 = ["d","e","f"]

var1 = input("What list do you want to use? (l1 or l2)"

print (var1)
output: l1
</code>```

how do I make it so if the user types in l1 the output will be: ["a","b","c"] ?

The closest example I have found is:

```<code>x='buffalo'
exec("%s = %d" % (x,2))
</code>```

答:
You might want to look at eval

https://docs.python.org/3/library/functions.html#eval

Sample code:

```<code>l1 = ["a","b","c"]
l2 = ["d","e","f"]

var1 = input("What list do you want to use? (l1 or l2)")

print (var1)

eval("print(" + var1 + ")")
</code>```

Output:

```<code>What list do you want to use? (l1 or l2)l1
l1
['a', 'b', 'c']
</code>```

Body:
l1 = ["a","b","c"]
l2 = ["d","e","f"]

var1 = input("What list do you want to use? (l1 or l2)"

print (var1)
output: l1

how do I make it so if the user types in l1 the output will be: ["a","b","c"] ?

The closest example I have found is:

x='buffalo'
exec("%s = %d" % (x,2))

Body:
You might want to look at eval

https://docs.python.org/3/library/functions.html#eval

Sample code:

l1 = ["a","b","c"]
l2 = ["d","e","f"]

var1 = input("What list do you want to use? (l1 or l2)")

print (var1)

eval("print(" + var1 + ")")

Output:

What list do you want to use? (l1 or l2)l1
l1
['a', 'b', 'c']


问:
How to find all stored procedures that delete rows from a particular table
If there's a way of doing this without regular expressions, that's great. If there isn't, here's what I've got so far:

I've written a simple CLR user-defined function (which as you can see I've called CLR_RegExMatch) that performs regex matches on a supplied string. I use it to search for patterns inside stored procedures, triggers, functions etc.

Here's an example of its use - searching for inserts into a table called ExampleTable:

```<code>SELECT O.name, O.type_desc
FROM
    SYS.OBJECTS O
    INNER JOIN SYS.SQL_MODULES M ON
        M.object_id = O.object_id
        AND dbo.CLR_RegExMatch('INSERT\s+(INTO\s+)?ExampleTable\b', M.definition) = 1
</code>```

The issue I've got is that I can't come up with a regex pattern to find all routines that delete rows from a given table. Obviously I could substitute the following for the last line in the previous example:

```<code>        AND dbo.CLR_RegExMatch('DELETE\s+(FROM\s+)?ExampleTable\b', M.definition) = 1
</code>```

and that gets me part of the way there. However it wouldn't pick up the following:

```<code>DELETE T1
FROM
    ExampleTable T1
    INNER JOIN AnotherTable T2 ON T2.ParentId = T1.Id
</code>```

So what I'm looking for is either a regex pattern that will match deletes as above, or alternatively a different way of going about this.

N.B. The reason that I'm querying the definition column of SYS.SQL_MODULES instead of the ROUTINE_DEFINITION column of INFORMATION_SCHEMA.ROUTINES is that the latter only contains the first 4000 characters a routine definition, whereas the former contains the full text.

答:
If can help, i use this stored procedure that can find a string in all modules in the database/s

usage:

```<code>exec find_text 'text to search', 'db_name'
-- if no db_name specified search in all DB
</code>```

Code below:

```<code>CREATE PROCEDURE [dbo].[find_text]
@text varchar(250),
@dbname varchar(64) = null
AS BEGIN
SET NOCOUNT ON;

if @dbname is null
 begin
  -- enumerate all databases.
  DECLARE #db CURSOR FOR Select Name from master..sysdatabases
  declare @c_dbname varchar(64)

  OPEN #db FETCH #db INTO @c_dbname
  while @@FETCH_STATUS <> -1
  begin
    execute find_text @text, @c_dbname
    FETCH #db INTO @c_dbname
  end
  CLOSE #db DEALLOCATE #db
 end
else
 begin
    declare @sql varchar(250)
    --create the find like command
    select @sql = 'select ''' + @dbname + ''' as db, o.name,m.definition '
    select @sql = @sql + ' from '+@dbname+'.sys.sql_modules m '
    select @sql = @sql + ' inner join '+@dbname+'..sysobjects o on m.object_id=o.id'
    select @sql = @sql + ' where [definition] like ''%'+@text+'%'''
    select @sql = @sql + ' order by o.name'

    execute (@sql)
 end
END
</code>```

Body:
If there's a way of doing this without regular expressions, that's great. If there isn't, here's what I've got so far:

I've written a simple CLR user-defined function (which as you can see I've called CLR_RegExMatch) that performs regex matches on a supplied string. I use it to search for patterns inside stored procedures, triggers, functions etc.

Here's an example of its use - searching for inserts into a table called ExampleTable:

SELECT O.name, O.type_desc
FROM
    SYS.OBJECTS O
    INNER JOIN SYS.SQL_MODULES M ON
        M.object_id = O.object_id
        AND dbo.CLR_RegExMatch('INSERT\s+(INTO\s+)?ExampleTable\b', M.definition) = 1

The issue I've got is that I can't come up with a regex pattern to find all routines that delete rows from a given table. Obviously I could substitute the following for the last line in the previous example:

        AND dbo.CLR_RegExMatch('DELETE\s+(FROM\s+)?ExampleTable\b', M.definition) = 1

and that gets me part of the way there. However it wouldn't pick up the following:

DELETE T1
FROM
    ExampleTable T1
    INNER JOIN AnotherTable T2 ON T2.ParentId = T1.Id

So what I'm looking for is either a regex pattern that will match deletes as above, or alternatively a different way of going about this.

N.B. The reason that I'm querying the definition column of SYS.SQL_MODULES instead of the ROUTINE_DEFINITION column of INFORMATION_SCHEMA.ROUTINES is that the latter only contains the first 4000 characters a routine definition, whereas the former contains the full text.

Body:
If can help, i use this stored procedure that can find a string in all modules in the database/s

usage:

exec find_text 'text to search', 'db_name'
-- if no db_name specified search in all DB

Code below:

CREATE PROCEDURE [dbo].[find_text]
@text varchar(250),
@dbname varchar(64) = null
AS BEGIN
SET NOCOUNT ON;

if @dbname is null
 begin
  -- enumerate all databases.
  DECLARE #db CURSOR FOR Select Name from master..sysdatabases
  declare @c_dbname varchar(64)

  OPEN #db FETCH #db INTO @c_dbname
  while @@FETCH_STATUS <> -1
  begin
    execute find_text @text, @c_dbname
    FETCH #db INTO @c_dbname
  end
  CLOSE #db DEALLOCATE #db
 end
else
 begin
    declare @sql varchar(250)
    --create the find like command
    select @sql = 'select ''' + @dbname + ''' as db, o.name,m.definition '
    select @sql = @sql + ' from '+@dbname+'.sys.sql_modules m '
    select @sql = @sql + ' inner join '+@dbname+'..sysobjects o on m.object_id=o.id'
    select @sql = @sql + ' where [definition] like ''%'+@text+'%'''
    select @sql = @sql + ' order by o.name'

    execute (@sql)
 end
END