问:
Varnish 4: Always miss
I'm currently setting up Varnish 4 for my domain.
My Varnish caches all files like charm, except of the DOC-request.
But I can't figure out, why the DOC-request is always a "MISS".

I hope, that you guys can help me.

All details below. If you need more, tell me.

Thank you so much for your help!

Regards,
Hermsi

Varnish-Version:

```<code>/ # varnishd -V
varnishd (varnish-4.1.3 revision 5e3b6d2)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS
</code>```

curl request:

```<code>$ curl -s -D - "https://www.example.com" -o /dev/null
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 0
Cache-Control: max-age=0
Content-Type: text/html; charset=UTF-8
Date: Wed, 15 Nov 2017 20:25:14 GMT
Expires: Wed, 15 Nov 2017 20:25:14 GMT
Vary: Accept-Encoding
X-Cache: MISS
X-Cache-Age:
X-Cache-Grace:
X-Cache-Ttl-Remaining:
X-Hits: 0
X-Host: example.com
X-Varnish-Cookie-Debug: Cleaned request cookies:
Transfer-Encoding: chunked
</code>```

Varnishlog:

```<code>*   << BeReq    >> 786983
-   Begin          bereq 786982 pass
-   Timestamp      Start: 1510849297.605099 0.000000 0.000000
-   BereqMethod    GET
-   BereqURL       /
-   BereqProtocol  HTTP/1.1
-   BereqHeader    Host: www.example.com
-   BereqHeader    User-Agent: curl/7.26.0
-   BereqHeader    Accept: */*
-   BereqHeader    X-Forwarded-Host: www.example.com
-   BereqHeader    X-Forwarded-Port: 443
-   BereqHeader    X-Forwarded-Proto: https
-   BereqHeader    X-Forwarded-Server: lb-proxy-1
-   BereqHeader    X-Real-Ip: XXX.XX.XXX.XXX
-   BereqHeader    X-Forwarded-For: XXX.XX.XXX.XXX
-   BereqHeader    Accept-Encoding: gzip
-   BereqHeader    X-Device: desktop
-   BereqHeader    X-Varnish: 786983
-   VCL_call       BACKEND_FETCH
-   VCL_return     fetch
-   BackendOpen    23 boot.default XXX.XX.XXX.XXX 80 10.42.70.133 39122
-   BackendStart   XXX.XX.XXX.XXX 80
-   Timestamp      Bereq: 1510849297.619798 0.014699 0.014699
-   Timestamp      Beresp: 1510849297.676958 0.071859 0.057160
-   BerespProtocol HTTP/1.1
-   BerespStatus   200
-   BerespReason   OK
-   BerespHeader   Date: Thu, 16 Nov 2017 16:21:37 GMT
-   BerespHeader   Server: Apache
-   BerespHeader   Cache-Control: max-age=0
-   BerespHeader   Expires: Thu, 16 Nov 2017 16:21:37 GMT
-   BerespHeader   Vary: Accept-Encoding
-   BerespHeader   Content-Encoding: gzip
-   BerespHeader   Content-Length: 3252
-   BerespHeader   Content-Type: text/html; charset=UTF-8
-   TTL            RFC 0 10 -1 1510849298 1510849298 1510849297 1510849297 0
-   VCL_call       BACKEND_RESPONSE
-   BerespHeader   X-Varnish-Cookie-Debug: Cleaned request cookies:
-   BerespHeader   x-url: /
-   BerespHeader   x-host: www.example.com
-   BerespHeader   X-Cache: ZIP
-   BereqHeader    stale-while-revalidate: max-age=0
-   TTL            VCL -1 7200 0 1510849298
-   TTL            VCL 120 7200 0 1510849298
-   VCL_return     deliver
-   Storage        malloc Transient
-   ObjProtocol    HTTP/1.1
-   ObjStatus      200
-   ObjReason      OK
-   ObjHeader      Date: Thu, 16 Nov 2017 16:21:37 GMT
-   ObjHeader      Server: Apache
-   ObjHeader      Cache-Control: max-age=0
-   ObjHeader      Expires: Thu, 16 Nov 2017 16:21:37 GMT
-   ObjHeader      Vary: Accept-Encoding
-   ObjHeader      Content-Encoding: gzip
-   ObjHeader      Content-Length: 3252
-   ObjHeader      Content-Type: text/html; charset=UTF-8
-   ObjHeader      X-Varnish-Cookie-Debug: Cleaned request cookies:
-   ObjHeader      x-url: /
-   ObjHeader      x-host: www.example.com
-   ObjHeader      X-Cache: ZIP
-   Fetch_Body     3 length stream
-   Gzip           u F - 3252 10723 80 80 25946
-   BackendReuse   23 boot.default
-   Timestamp      BerespBody: 1510849297.677148 0.072049 0.000190
-   Length         3252
-   BereqAcct      328 0 328 247 3252 3499
-   End

*   << Request  >> 786982
-   Begin          req 786981 rxreq
-   Timestamp      Start: 1510849297.605008 0.000000 0.000000
-   Timestamp      Req: 1510849297.605008 0.000000 0.000000
-   ReqStart       10.42.115.200 35666
-   ReqMethod      GET
-   ReqURL         /
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: www.example.com
-   ReqHeader      User-Agent: curl/7.26.0
-   ReqHeader      Accept: */*
-   ReqHeader      X-Forwarded-For: XXX.XX.XXX.XXX
-   ReqHeader      X-Forwarded-Host: www.example.com
-   ReqHeader      X-Forwarded-Port: 443
-   ReqHeader      X-Forwarded-Proto: https
-   ReqHeader      X-Forwarded-Server: lb-proxy-1
-   ReqHeader      X-Real-Ip: XXX.XX.XXX.XXX
-   ReqHeader      Accept-Encoding: gzip
-   ReqUnset       X-Forwarded-For: XXX.XX.XXX.XXX
-   ReqHeader      X-Forwarded-For: XXX.XX.XXX.XXX, 10.42.115.200
-   VCL_call       RECV
-   ReqUnset       X-Forwarded-For: XXX.XX.XXX.XXX, 10.42.115.200
-   ReqHeader      X-Forwarded-For: XXX.XX.XXX.XXX
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqUnset       Accept-Encoding: gzip
-   ReqHeader      Accept-Encoding: gzip
-   ReqHeader      X-Device: desktop
-   VCL_return     hash
-   VCL_call       HASH
-   VCL_return     lookup
-   Debug          "XXXX HIT-FOR-PASS"
-   HitPass        786980
-   VCL_call       PASS
-   VCL_return     fetch
-   Link           bereq 786983 pass
-   Timestamp      Fetch: 1510849297.677182 0.072175 0.072175
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
-   RespHeader     Date: Thu, 16 Nov 2017 16:21:37 GMT
-   RespHeader     Server: Apache
-   RespHeader     Cache-Control: max-age=0
-   RespHeader     Expires: Thu, 16 Nov 2017 16:21:37 GMT
-   RespHeader     Vary: Accept-Encoding
-   RespHeader     Content-Encoding: gzip
-   RespHeader     Content-Length: 3252
-   RespHeader     Content-Type: text/html; charset=UTF-8
-   RespHeader     X-Varnish-Cookie-Debug: Cleaned request cookies:
-   RespHeader     x-url: /
-   RespHeader     x-host: www.example.com
-   RespHeader     X-Cache: ZIP
-   RespHeader     X-Varnish: 786982
-   RespHeader     Age: 0
-   RespHeader     Via: 1.1 varnish-v4
-   VCL_call       DELIVER
-   RespUnset      x-url: /
-   RespUnset      x-host: www.example.com
-   RespUnset      X-Cache: ZIP
-   RespHeader     X-Cache: MISS
-   RespHeader     X-Hits: 0
-   RespHeader     X-Host: example.com
-   RespUnset      Server: Apache
-   RespUnset      X-Varnish: 786982
-   RespUnset      Via: 1.1 varnish-v4
-   RespHeader     X-Cache-TTL-Remaining:
-   RespHeader     X-Cache-Age:
-   RespHeader     X-Cache-Grace:
-   VCL_return     deliver
-   Timestamp      Process: 1510849297.677222 0.072215 0.000040
-   RespHeader     Accept-Ranges: bytes
-   Debug          "RES_MODE 2"
-   RespHeader     Connection: keep-alive
-   Timestamp      Resp: 1510849297.677275 0.072268 0.000053
-   ReqAcct        290 0 290 446 3252 3698
-   End
</code>```

This is my VCL:

```<code>vcl 4.0;

import std;

include "cache_invalidation.vcl";
include "custom_subroutines.vcl";

backend default {
    .host = "VARNISH_BACKEND_ADDRESS";  # IP or Hostname of backend
    .port = "VARNISH_BACKEND_PORT";     # Port Apache or whatever is listening
    .connect_timeout = 5s;              # Wait a maximum of 5s for backend connection (Apache, Nginx, etc...)
    .first_byte_timeout = 10s;           # Wait a maximum of 10s for the first byte to come from your backend
    .between_bytes_timeout = 2s;        # Wait a maximum of 2s between each bytes sent
    .max_connections = 300;             # Max parallel connections to our backend
}

sub vcl_recv {

    # IP forwarding.
    if (req.restarts == 0) {
        if (req.http.X-Forwarded-For && !req.http.X-Real-IP) {
            set req.http.X-Real-IP = regsub(req.http.X-Forwarded-For, ".*\b(?!10|127|172)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*", "\1");
        }else{
            set req.http.X-Forwarded-For = req.http.X-Real-IP;
        }
    }

    # Enforce redirect to HTTPS
    if (req.http.X-Forwarded-Proto !~ "(?i)https") {
        return(synth(720, "https://" + req.http.host + req.url));
    }

    # Remove the Google Analytics added parameters, useless for our backend
      if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=") {
        set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
        set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
        set req.url = regsub(req.url, "\?&", "?");
        set req.url = regsub(req.url, "\?$", "");
      }

    # Remove anchor or hash references
      if(req.url ~ "\#"){
        set req.url = regsub(req.url, "\#.*$", "");
      }

    # Remove incomplete query string
      if(req.url ~ "\?$"){
        set req.url = regsub(req.url, "\?$", "");
      }

    # Remove Google Analytics Cookies, our server does not need it
    set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", "");

    # Remove DoubleClick offensive cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");

    # Remove the Quant Capital cookies (added by some plugin, all __qca)
    set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");

    # Remove the AddThis cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");

    # Remove a ";" prefix in the cookie if present
    set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");

    # Are there cookies left with only spaces or that are empty?
    if (req.http.cookie ~ "^\s*$") {
        unset req.http.cookie;
    }

    if (req.url ~ "^[^?]*\.(bmp|bz2|css|doc|eot|flv|gif|gz|ico|jpeg|jpg|js|less|pdf|png|rtf|swf|txt|woff|xml)(\?.*)?$") {
        unset req.http.Cookie;
        return (hash);
    }

    if (req.url ~ "^[^?]*\.(mp[34]|rar|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$") {
        unset req.http.Cookie;
        return (hash);
    }

    if(req.http.Accept-Encoding)
    {
       if(req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp[34]|ogg)$")
       {
        # No point compressing these extensions
        unset req.http.Accept-Encoding;
       } elsif(req.http.Accept-Encoding ~ "gzip")
       {
        set req.http.Accept-Encoding = "gzip";
       } elsif(req.http.Accept-Encoding ~ "deflate")
       {
        set req.http.Accept-Encoding = "deflate";
       } elsif(req.http.Accept-Encoding ~ "sdch")
       {
        set req.http.Accept-Encoding = "sdch";
       } else
       {
        # Unknown Algorithm
        unset req.http.Accept-Encoding;
       }
    }

}

# At this point, it is our last chance to make some decision (set|unset|mod) about some header that the backend should see.

sub vcl_backend_fetch {

    unset bereq.http.X-Varnish-SESS-ID;
}

sub vcl_hit {

    # Custom Headers for debug purposes
    set req.http.X-Cache-TTL-Remaining = obj.ttl;
    set req.http.X-Cache-Age = obj.keep - obj.ttl;
    set req.http.X-Cache-Grace = obj.grace;
}

sub vcl_backend_response {

    if (bereq.url ~ "^[^?]*\.(mp[34]|rar|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$") {
        unset beresp.http.set-cookie;
        set beresp.do_stream = true;  # Check memory usage it'll grow in fetch_chunksize blocks (128k by default) if the backend doesn't send a Content-Length header, so only enable it for big objects
        set beresp.do_gzip = false;   # Don't try to compress it for storage
    }

    # Do not Gzip image files in Varnish
    if (beresp.http.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|mp4|ogg|swf)$") {
        set beresp.do_gzip = false;
    }
    else {
        set beresp.do_gzip = true;
        set beresp.http.X-Cache = "ZIP";
    }

    # GZip the cached content if possible
    if (beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    # If we are performing a redirection 301 / 302 from the backend and our web server and varnish instances are in the same node, apache mod_rewrite could append it's port
    # typically varnish :80 apache/nginx/lighthttp :8080, so the redirect can then often redirect the end-user to a URL on :8080, where it should be :80.

    if(beresp.status == 301 || beresp.status == 302) {
        set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+", "");
    }

    # The example below is explained in detail in load_balancing.vcl as part of one of the strategies that we can take to handle Varnish Grace Mode

    set bereq.http.stale-while-revalidate = regsub(beresp.http.Cache-Control, ".*stale-while-revalidate\=([0-9]+).*", "\1");

    if(std.real(bereq.http.stale-while-revalidate, 0.0) > 0)
    {
        set beresp.grace = std.duration(bereq.http.stale-while-revalidate + "s", 10s);
    } else {
        set beresp.grace = 2h;
    }

}

sub vcl_deliver {

    # At this point this is our last change to set or unset the response headers before deliver it

    if (obj.hits > 0)
    {
        set resp.http.X-Cache = "HIT";
    } else
    {
        set resp.http.X-Cache = "MISS";
    }

    set resp.http.X-Hits = obj.hits;

    # Remove www of our host in the backend response
    set resp.http.X-Host = regsub(req.http.host, "^www\.", "");

    # Remove PHP , Apache and Varnish versions
    unset resp.http.x-powered-by;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;

    # Here we are ensuring to add this headers in the response (defined in the hit subroutine) for debug purposes
    set resp.http.X-Cache-TTL-Remaining = req.http.X-Cache-TTL-Remaining;
    set resp.http.X-Cache-Age = req.http.X-Cache-Age;
    set resp.http.X-Cache-Grace = req.http.X-Cache-Grace;

}
</code>```

答:
The varnish log you are providing does not match the curl request you are doing (wrong user agent for example).

If it is still relevant, in your varnishlog you can see that the backend request contains the headers :

```<code>BereqHeader    Cache-Control: no-cache
BereqHeader    Pragma: no-cache
</code>```

and the backend response contains the following

```<code>BerespHeader   Cache-Control: max-age=0
</code>```

which prevent Varnish from caching the response.

Can you share the varnish log part for the Request (not the BackendRequest) ?

Without it, my guess could be that you always get a miss because the builtin.vcl make your request a pass (you do not do any return (pass) in your conf nonetheless the backend request comes from the pass state

```<code>Begin          bereq 1048612 pass
</code>```

It could be triggered by these two rules in the builtin.vcl :

```<code>if (req.method != "GET" && req.method != "HEAD") {
    /* We only deal with GET and HEAD by default */
    return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
    /* Not cacheable by default */
    return (pass);
}
</code>```

EDIT

That's weird. You should not have any Cookie using Curl.
Try scoping your cookie handling code and use req.http.Cookie (with an uppercase C) like this :

```<code>if (req.http.Cookie) {
  # Remove Google Analytics Cookies, our server does not need it
  set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", "");

  # Remove DoubleClick offensive cookies
  set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");

  # Remove the Quant Capital cookies (added by some plugin, all __qca)
  set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");

  # Remove the AddThis cookies
  set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");

  # Remove a ";" prefix in the cookie if present
  set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");

  # Are there cookies left with only spaces or that are empty?
  if (req.http.Cookie ~ "^\s*$") {
      unset req.http.Cookie;
  }
}
</code>```

Another thing here, in your vcl log you can see that Varnish is doing Hit-For-Pass, that is, Varnish is caching the fact that your request is not cached. If you want to truly test, you'll need to purge the content beforehands.
One last thing : your backend server (apache) responds with a Cache-Control : max-age =0 header which prevent caching, you should change that if you want to cache content.

Title: Varnish 4: Always miss

Body:
I'm currently setting up Varnish 4 for my domain.
My Varnish caches all files like charm, except of the DOC-request.
But I can't figure out, why the DOC-request is always a "MISS".

I hope, that you guys can help me.

All details below. If you need more, tell me.

Thank you so much for your help!

Regards,
Hermsi

Varnish-Version:

/ # varnishd -V
varnishd (varnish-4.1.3 revision 5e3b6d2)
Copyright (c) 2006 Verdens Gang AS
Copyright (c) 2006-2015 Varnish Software AS

curl request:

$ curl -s -D - "https://www.example.com" -o /dev/null
HTTP/1.1 200 OK
Accept-Ranges: bytes
Age: 0
Cache-Control: max-age=0
Content-Type: text/html; charset=UTF-8
Date: Wed, 15 Nov 2017 20:25:14 GMT
Expires: Wed, 15 Nov 2017 20:25:14 GMT
Vary: Accept-Encoding
X-Cache: MISS
X-Cache-Age:
X-Cache-Grace:
X-Cache-Ttl-Remaining:
X-Hits: 0
X-Host: example.com
X-Varnish-Cookie-Debug: Cleaned request cookies:
Transfer-Encoding: chunked

Varnishlog:

*   << BeReq    >> 786983
-   Begin          bereq 786982 pass
-   Timestamp      Start: 1510849297.605099 0.000000 0.000000
-   BereqMethod    GET
-   BereqURL       /
-   BereqProtocol  HTTP/1.1
-   BereqHeader    Host: www.example.com
-   BereqHeader    User-Agent: curl/7.26.0
-   BereqHeader    Accept: */*
-   BereqHeader    X-Forwarded-Host: www.example.com
-   BereqHeader    X-Forwarded-Port: 443
-   BereqHeader    X-Forwarded-Proto: https
-   BereqHeader    X-Forwarded-Server: lb-proxy-1
-   BereqHeader    X-Real-Ip: XXX.XX.XXX.XXX
-   BereqHeader    X-Forwarded-For: XXX.XX.XXX.XXX
-   BereqHeader    Accept-Encoding: gzip
-   BereqHeader    X-Device: desktop
-   BereqHeader    X-Varnish: 786983
-   VCL_call       BACKEND_FETCH
-   VCL_return     fetch
-   BackendOpen    23 boot.default XXX.XX.XXX.XXX 80 10.42.70.133 39122
-   BackendStart   XXX.XX.XXX.XXX 80
-   Timestamp      Bereq: 1510849297.619798 0.014699 0.014699
-   Timestamp      Beresp: 1510849297.676958 0.071859 0.057160
-   BerespProtocol HTTP/1.1
-   BerespStatus   200
-   BerespReason   OK
-   BerespHeader   Date: Thu, 16 Nov 2017 16:21:37 GMT
-   BerespHeader   Server: Apache
-   BerespHeader   Cache-Control: max-age=0
-   BerespHeader   Expires: Thu, 16 Nov 2017 16:21:37 GMT
-   BerespHeader   Vary: Accept-Encoding
-   BerespHeader   Content-Encoding: gzip
-   BerespHeader   Content-Length: 3252
-   BerespHeader   Content-Type: text/html; charset=UTF-8
-   TTL            RFC 0 10 -1 1510849298 1510849298 1510849297 1510849297 0
-   VCL_call       BACKEND_RESPONSE
-   BerespHeader   X-Varnish-Cookie-Debug: Cleaned request cookies:
-   BerespHeader   x-url: /
-   BerespHeader   x-host: www.example.com
-   BerespHeader   X-Cache: ZIP
-   BereqHeader    stale-while-revalidate: max-age=0
-   TTL            VCL -1 7200 0 1510849298
-   TTL            VCL 120 7200 0 1510849298
-   VCL_return     deliver
-   Storage        malloc Transient
-   ObjProtocol    HTTP/1.1
-   ObjStatus      200
-   ObjReason      OK
-   ObjHeader      Date: Thu, 16 Nov 2017 16:21:37 GMT
-   ObjHeader      Server: Apache
-   ObjHeader      Cache-Control: max-age=0
-   ObjHeader      Expires: Thu, 16 Nov 2017 16:21:37 GMT
-   ObjHeader      Vary: Accept-Encoding
-   ObjHeader      Content-Encoding: gzip
-   ObjHeader      Content-Length: 3252
-   ObjHeader      Content-Type: text/html; charset=UTF-8
-   ObjHeader      X-Varnish-Cookie-Debug: Cleaned request cookies:
-   ObjHeader      x-url: /
-   ObjHeader      x-host: www.example.com
-   ObjHeader      X-Cache: ZIP
-   Fetch_Body     3 length stream
-   Gzip           u F - 3252 10723 80 80 25946
-   BackendReuse   23 boot.default
-   Timestamp      BerespBody: 1510849297.677148 0.072049 0.000190
-   Length         3252
-   BereqAcct      328 0 328 247 3252 3499
-   End

*   << Request  >> 786982
-   Begin          req 786981 rxreq
-   Timestamp      Start: 1510849297.605008 0.000000 0.000000
-   Timestamp      Req: 1510849297.605008 0.000000 0.000000
-   ReqStart       10.42.115.200 35666
-   ReqMethod      GET
-   ReqURL         /
-   ReqProtocol    HTTP/1.1
-   ReqHeader      Host: www.example.com
-   ReqHeader      User-Agent: curl/7.26.0
-   ReqHeader      Accept: */*
-   ReqHeader      X-Forwarded-For: XXX.XX.XXX.XXX
-   ReqHeader      X-Forwarded-Host: www.example.com
-   ReqHeader      X-Forwarded-Port: 443
-   ReqHeader      X-Forwarded-Proto: https
-   ReqHeader      X-Forwarded-Server: lb-proxy-1
-   ReqHeader      X-Real-Ip: XXX.XX.XXX.XXX
-   ReqHeader      Accept-Encoding: gzip
-   ReqUnset       X-Forwarded-For: XXX.XX.XXX.XXX
-   ReqHeader      X-Forwarded-For: XXX.XX.XXX.XXX, 10.42.115.200
-   VCL_call       RECV
-   ReqUnset       X-Forwarded-For: XXX.XX.XXX.XXX, 10.42.115.200
-   ReqHeader      X-Forwarded-For: XXX.XX.XXX.XXX
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqHeader      Cookie:
-   ReqUnset       Cookie:
-   ReqUnset       Accept-Encoding: gzip
-   ReqHeader      Accept-Encoding: gzip
-   ReqHeader      X-Device: desktop
-   VCL_return     hash
-   VCL_call       HASH
-   VCL_return     lookup
-   Debug          "XXXX HIT-FOR-PASS"
-   HitPass        786980
-   VCL_call       PASS
-   VCL_return     fetch
-   Link           bereq 786983 pass
-   Timestamp      Fetch: 1510849297.677182 0.072175 0.072175
-   RespProtocol   HTTP/1.1
-   RespStatus     200
-   RespReason     OK
-   RespHeader     Date: Thu, 16 Nov 2017 16:21:37 GMT
-   RespHeader     Server: Apache
-   RespHeader     Cache-Control: max-age=0
-   RespHeader     Expires: Thu, 16 Nov 2017 16:21:37 GMT
-   RespHeader     Vary: Accept-Encoding
-   RespHeader     Content-Encoding: gzip
-   RespHeader     Content-Length: 3252
-   RespHeader     Content-Type: text/html; charset=UTF-8
-   RespHeader     X-Varnish-Cookie-Debug: Cleaned request cookies:
-   RespHeader     x-url: /
-   RespHeader     x-host: www.example.com
-   RespHeader     X-Cache: ZIP
-   RespHeader     X-Varnish: 786982
-   RespHeader     Age: 0
-   RespHeader     Via: 1.1 varnish-v4
-   VCL_call       DELIVER
-   RespUnset      x-url: /
-   RespUnset      x-host: www.example.com
-   RespUnset      X-Cache: ZIP
-   RespHeader     X-Cache: MISS
-   RespHeader     X-Hits: 0
-   RespHeader     X-Host: example.com
-   RespUnset      Server: Apache
-   RespUnset      X-Varnish: 786982
-   RespUnset      Via: 1.1 varnish-v4
-   RespHeader     X-Cache-TTL-Remaining:
-   RespHeader     X-Cache-Age:
-   RespHeader     X-Cache-Grace:
-   VCL_return     deliver
-   Timestamp      Process: 1510849297.677222 0.072215 0.000040
-   RespHeader     Accept-Ranges: bytes
-   Debug          "RES_MODE 2"
-   RespHeader     Connection: keep-alive
-   Timestamp      Resp: 1510849297.677275 0.072268 0.000053
-   ReqAcct        290 0 290 446 3252 3698
-   End

This is my VCL:

vcl 4.0;

import std;

include "cache_invalidation.vcl";
include "custom_subroutines.vcl";

backend default {
    .host = "VARNISH_BACKEND_ADDRESS";  # IP or Hostname of backend
    .port = "VARNISH_BACKEND_PORT";     # Port Apache or whatever is listening
    .connect_timeout = 5s;              # Wait a maximum of 5s for backend connection (Apache, Nginx, etc...)
    .first_byte_timeout = 10s;           # Wait a maximum of 10s for the first byte to come from your backend
    .between_bytes_timeout = 2s;        # Wait a maximum of 2s between each bytes sent
    .max_connections = 300;             # Max parallel connections to our backend
}

sub vcl_recv {

    # IP forwarding.
    if (req.restarts == 0) {
        if (req.http.X-Forwarded-For && !req.http.X-Real-IP) {
            set req.http.X-Real-IP = regsub(req.http.X-Forwarded-For, ".*\b(?!10|127|172)(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}).*", "\1");
        }else{
            set req.http.X-Forwarded-For = req.http.X-Real-IP;
        }
    }

    # Enforce redirect to HTTPS
    if (req.http.X-Forwarded-Proto !~ "(?i)https") {
        return(synth(720, "https://" + req.http.host + req.url));
    }

    # Remove the Google Analytics added parameters, useless for our backend
      if (req.url ~ "(\?|&)(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=") {
        set req.url = regsuball(req.url, "&(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "");
        set req.url = regsuball(req.url, "\?(utm_source|utm_medium|utm_campaign|utm_content|gclid|cx|ie|cof|siteurl)=([A-z0-9_\-\.%25]+)", "?");
        set req.url = regsub(req.url, "\?&", "?");
        set req.url = regsub(req.url, "\?$", "");
      }

    # Remove anchor or hash references
      if(req.url ~ "\#"){
        set req.url = regsub(req.url, "\#.*$", "");
      }

    # Remove incomplete query string
      if(req.url ~ "\?$"){
        set req.url = regsub(req.url, "\?$", "");
      }

    # Remove Google Analytics Cookies, our server does not need it
    set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", "");
    set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", "");

    # Remove DoubleClick offensive cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");

    # Remove the Quant Capital cookies (added by some plugin, all __qca)
    set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");

    # Remove the AddThis cookies
    set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");

    # Remove a ";" prefix in the cookie if present
    set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");

    # Are there cookies left with only spaces or that are empty?
    if (req.http.cookie ~ "^\s*$") {
        unset req.http.cookie;
    }

    if (req.url ~ "^[^?]*\.(bmp|bz2|css|doc|eot|flv|gif|gz|ico|jpeg|jpg|js|less|pdf|png|rtf|swf|txt|woff|xml)(\?.*)?$") {
        unset req.http.Cookie;
        return (hash);
    }

    if (req.url ~ "^[^?]*\.(mp[34]|rar|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$") {
        unset req.http.Cookie;
        return (hash);
    }

    if(req.http.Accept-Encoding)
    {
       if(req.url ~ "\.(jpg|png|gif|gz|tgz|bz2|tbz|mp[34]|ogg)$")
       {
        # No point compressing these extensions
        unset req.http.Accept-Encoding;
       } elsif(req.http.Accept-Encoding ~ "gzip")
       {
        set req.http.Accept-Encoding = "gzip";
       } elsif(req.http.Accept-Encoding ~ "deflate")
       {
        set req.http.Accept-Encoding = "deflate";
       } elsif(req.http.Accept-Encoding ~ "sdch")
       {
        set req.http.Accept-Encoding = "sdch";
       } else
       {
        # Unknown Algorithm
        unset req.http.Accept-Encoding;
       }
    }

}

# At this point, it is our last chance to make some decision (set|unset|mod) about some header that the backend should see.

sub vcl_backend_fetch {

    unset bereq.http.X-Varnish-SESS-ID;
}

sub vcl_hit {

    # Custom Headers for debug purposes
    set req.http.X-Cache-TTL-Remaining = obj.ttl;
    set req.http.X-Cache-Age = obj.keep - obj.ttl;
    set req.http.X-Cache-Grace = obj.grace;
}

sub vcl_backend_response {

    if (bereq.url ~ "^[^?]*\.(mp[34]|rar|tar|tgz|gz|wav|zip|bz2|xz|7z|avi|mov|ogm|mpe?g|mk[av]|webm)(\?.*)?$") {
        unset beresp.http.set-cookie;
        set beresp.do_stream = true;  # Check memory usage it'll grow in fetch_chunksize blocks (128k by default) if the backend doesn't send a Content-Length header, so only enable it for big objects
        set beresp.do_gzip = false;   # Don't try to compress it for storage
    }

    # Do not Gzip image files in Varnish
    if (beresp.http.url ~ "\.(jpg|jpeg|png|gif|gz|tgz|bz2|tbz|mp3|mp4|ogg|swf)$") {
        set beresp.do_gzip = false;
    }
    else {
        set beresp.do_gzip = true;
        set beresp.http.X-Cache = "ZIP";
    }

    # GZip the cached content if possible
    if (beresp.http.content-type ~ "text") {
        set beresp.do_gzip = true;
    }

    # If we are performing a redirection 301 / 302 from the backend and our web server and varnish instances are in the same node, apache mod_rewrite could append it's port
    # typically varnish :80 apache/nginx/lighthttp :8080, so the redirect can then often redirect the end-user to a URL on :8080, where it should be :80.

    if(beresp.status == 301 || beresp.status == 302) {
        set beresp.http.Location = regsub(beresp.http.Location, ":[0-9]+", "");
    }

    # The example below is explained in detail in load_balancing.vcl as part of one of the strategies that we can take to handle Varnish Grace Mode

    set bereq.http.stale-while-revalidate = regsub(beresp.http.Cache-Control, ".*stale-while-revalidate\=([0-9]+).*", "\1");

    if(std.real(bereq.http.stale-while-revalidate, 0.0) > 0)
    {
        set beresp.grace = std.duration(bereq.http.stale-while-revalidate + "s", 10s);
    } else {
        set beresp.grace = 2h;
    }

}

sub vcl_deliver {

    # At this point this is our last change to set or unset the response headers before deliver it

    if (obj.hits > 0)
    {
        set resp.http.X-Cache = "HIT";
    } else
    {
        set resp.http.X-Cache = "MISS";
    }

    set resp.http.X-Hits = obj.hits;

    # Remove www of our host in the backend response
    set resp.http.X-Host = regsub(req.http.host, "^www\.", "");

    # Remove PHP , Apache and Varnish versions
    unset resp.http.x-powered-by;
    unset resp.http.Server;
    unset resp.http.X-Varnish;
    unset resp.http.Via;

    # Here we are ensuring to add this headers in the response (defined in the hit subroutine) for debug purposes
    set resp.http.X-Cache-TTL-Remaining = req.http.X-Cache-TTL-Remaining;
    set resp.http.X-Cache-Age = req.http.X-Cache-Age;
    set resp.http.X-Cache-Grace = req.http.X-Cache-Grace;

}

Body:
The varnish log you are providing does not match the curl request you are doing (wrong user agent for example).

If it is still relevant, in your varnishlog you can see that the backend request contains the headers :

BereqHeader    Cache-Control: no-cache
BereqHeader    Pragma: no-cache

and the backend response contains the following

BerespHeader   Cache-Control: max-age=0

which prevent Varnish from caching the response.

Can you share the varnish log part for the Request (not the BackendRequest) ?

Without it, my guess could be that you always get a miss because the builtin.vcl make your request a pass (you do not do any return (pass) in your conf nonetheless the backend request comes from the pass state

Begin          bereq 1048612 pass

It could be triggered by these two rules in the builtin.vcl :

if (req.method != "GET" && req.method != "HEAD") {
    /* We only deal with GET and HEAD by default */
    return (pass);
}
if (req.http.Authorization || req.http.Cookie) {
    /* Not cacheable by default */
    return (pass);
}

EDIT

That's weird. You should not have any Cookie using Curl.
Try scoping your cookie handling code and use req.http.Cookie (with an uppercase C) like this :

if (req.http.Cookie) {
  # Remove Google Analytics Cookies, our server does not need it
  set req.http.Cookie = regsuball(req.http.Cookie, "__utm.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "_ga=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "_gat=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmctr=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmcmd.=[^;]+(; )?", "");
  set req.http.Cookie = regsuball(req.http.Cookie, "utmccn.=[^;]+(; )?", "");

  # Remove DoubleClick offensive cookies
  set req.http.Cookie = regsuball(req.http.Cookie, "__gads=[^;]+(; )?", "");

  # Remove the Quant Capital cookies (added by some plugin, all __qca)
  set req.http.Cookie = regsuball(req.http.Cookie, "__qc.=[^;]+(; )?", "");

  # Remove the AddThis cookies
  set req.http.Cookie = regsuball(req.http.Cookie, "__atuv.=[^;]+(; )?", "");

  # Remove a ";" prefix in the cookie if present
  set req.http.Cookie = regsuball(req.http.Cookie, "^;\s*", "");

  # Are there cookies left with only spaces or that are empty?
  if (req.http.Cookie ~ "^\s*$") {
      unset req.http.Cookie;
  }
}

Another thing here, in your vcl log you can see that Varnish is doing Hit-For-Pass, that is, Varnish is caching the fact that your request is not cached. If you want to truly test, you'll need to purge the content beforehands.
One last thing : your backend server (apache) responds with a Cache-Control : max-age =0 header which prevent caching, you should change that if you want to cache content.


来源: stackexchange

问:
Why are only final variables accessible in anonymous class?

a can only be final here.  Why?  How can I reassign a in onClick() method without keeping it as private member?

```<code>private void f(Button b, final int a){
    b.addClickHandler(new ClickHandler() {

        @Override
        public void onClick(ClickEvent event) {
            int b = a*5;

        }
    });
}
</code>```

How can I return the 5 * a when it clicked? I mean,

```<code>private void f(Button b, final int a){
    b.addClickHandler(new ClickHandler() {

        @Override
        public void onClick(ClickEvent event) {
             int b = a*5;
             return b; // but return type is void
        }
    });
}
</code>```

答: Methods within an anonomyous inner class may be invoked well after the thread that spawned it has terminated. In your example, the inner class will be invoked on the event dispatch thread and not in the same thread as that which created it. Hence, the scope of the variables will be different. So to protect such variable assignment scope issues you must declare them final.</p>

ContentLicense: CC BY-SA 3.0

Body:
a can only be final here.  Why?  How can I reassign a in onClick() method without keeping it as private member?

private void f(Button b, final int a){
    b.addClickHandler(new ClickHandler() {

        @Override
        public void onClick(ClickEvent event) {
            int b = a*5;

        }
    });
}

How can I return the 5 * a when it clicked? I mean,

private void f(Button b, final int a){
    b.addClickHandler(new ClickHandler() {

        @Override
        public void onClick(ClickEvent event) {
             int b = a*5;
             return b; // but return type is void
        }
    });
}

ContentLicense: CC BY-SA 2.5


问:
Memory efficient way to split large numpy array into train and test
I have a large numpy array and when I run scikit learn's train_test_split to split the array into training and test data, I always run into memory errors.  What would be a more memory efficient method of splitting into train and test, and why does the train_test_split cause this?

The follow code results in a memory error and causes a crash

```<code>import numpy as np
from sklearn.cross_validation import train_test_split

X = np.random.random((10000,70000))
Y = np.random.random((10000,))
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.33, random_state=42)
</code>```

答: I faced the same problem with my code. I was using a dense array like you and ran out of memory. I converted my training data to sparse (I am doing document classification) and solved my issue.</p>

Body:
I have a large numpy array and when I run scikit learn's train_test_split to split the array into training and test data, I always run into memory errors.  What would be a more memory efficient method of splitting into train and test, and why does the train_test_split cause this?

The follow code results in a memory error and causes a crash

import numpy as np
from sklearn.cross_validation import train_test_split

X = np.random.random((10000,70000))
Y = np.random.random((10000,))
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.33, random_state=42)


问:
Excel - Finding a name then inserting a specific value
Alright guys, I do not know exactly how to phrase this question but I have a picture and an example of exactly what I am trying to do.

Here is my excel sheet screenshot:

So I have a list of players, their points and a space for their team name.
I have a team already listed in E1:E5, there will be more teams defined in F1:F5 and so on... There are a lot of players in this sheet and the order of them are scrambled.

Instead of manually finding each player and typing in their team name, I wish to just type in the team name and its players into a block and have column A automatically update with the corresponding team name. I tried to fiddle around with VLOOKUP and =Index(match()) to no avail. Is this possible?

Thank you.

答:
Formula in A2:

```<code>=INDEX($E$1:$G$1,1,SUMPRODUCT(($E$2:$G$5=B2)*(COLUMN($E$1:$G$1)-4)))
</code>```

($E$2:$G$5=B2) is like a full array lookup, returns an array of true and false

(COLUMN($E$1:$G$1)-4) gives an array containing the column numbers. -4 because E is column 5, so that the first team, Optic becomes column number 1 after subtracting 4, TeamB becomes column number 2 (column 6 - 4), etc

SUMPRODUCT(($E$2:$G$5=B2)*(COLUMN($E$1:$G$1)-4)) thus gives the column number in which the match was found.

Note: this formula will not work properly if there are more than 1 match, and will return #VALUE when no match is found.

Body:
Formula in A2:

=INDEX($E$1:$G$1,1,SUMPRODUCT(($E$2:$G$5=B2)*(COLUMN($E$1:$G$1)-4)))

($E$2:$G$5=B2) is like a full array lookup, returns an array of true and false

(COLUMN($E$1:$G$1)-4) gives an array containing the column numbers. -4 because E is column 5, so that the first team, Optic becomes column number 1 after subtracting 4, TeamB becomes column number 2 (column 6 - 4), etc

SUMPRODUCT(($E$2:$G$5=B2)*(COLUMN($E$1:$G$1)-4)) thus gives the column number in which the match was found.

Note: this formula will not work properly if there are more than 1 match, and will return #VALUE when no match is found.


问:
get the wanted objects from the array of objects
This is my first question in stackoverflow, So any kind of help is appreciated.

I have an array like this

```<code>var products = [{status:'available', ...},{status:'none', ...},{status:'available', ...},{status:'', ...},{status:'none', ...}]
</code>```

how can I get an array that contains only  products[i].status = 'available' .

I already did this with for loop but isn't there a better method? My code is practically full of loops now & it's not easy to read.

Thank you for you help.

答:
Using arrow functions (ES6) is a shortest one

```<code>products.filter(product => product.status == 'available')
</code>```

Will return an appropriate array of products where statuses arr "available"

Body:
This is my first question in stackoverflow, So any kind of help is appreciated.

I have an array like this

var products = [{status:'available', ...},{status:'none', ...},{status:'available', ...},{status:'', ...},{status:'none', ...}]

how can I get an array that contains only  products[i].status = 'available' .

I already did this with for loop but isn't there a better method? My code is practically full of loops now & it's not easy to read.

Thank you for you help.

Body:
Using arrow functions (ES6) is a shortest one

products.filter(product => product.status == 'available')

Will return an appropriate array of products where statuses arr "available"


问:
Is in Java method reference a part of the reflection API
In Kotlin, the result of a method-reference is placed in the package kotlin.reflect. Is the method-reference in Java also part of the reflection-api?</p>

答:
The actual type implementing the functional interface for method references is intentionally unspecified. All that matters, is, that it is provided by the current JRE.

Practically, in the case of HotSpot/OpenJDK, it is a runtime generated class that resides within the same package and class loading context as the class containing the method reference. It does not inherit from a special class and does not implement other interfaces besides the interfaces defined by its target type. So while this class has some special properties, i.e. it can not get looked up by name and it may get garbage collected regardless of the reachability of its defining class loader, but it’s not part of the Reflection API.

The properties of the instance created for a method reference are specified by the Java Language Specification as follows:

The value of a method reference expression is a reference to an instance of a class with the following properties:

The class implements the targeted functional interface type and, if the target type is an intersection type, every other interface type mentioned in the intersection.

Where the method reference expression has type U, for each non-static member method m of U:

If the function type of U has a subsignature of the signature of m, then the class declares an invocation method that overrides m. The invocation method's body invokes the referenced method, creates a class instance, or creates an array, as described below. If the invocation method's result is not void, then the body returns the result of the method invocation or object creation, after any necessary assignment conversions (§5.2).

If the erasure of the type of a method being overridden differs in its signature from the erasure of the function type of U, then before the method invocation or object creation, an invocation method's body checks that each argument value is an instance of a subclass or subinterface of the erasure of the corresponding parameter type in the function type of U; if not, a ClassCastException is thrown.

The class overrides no other methods of the functional interface type or other interface types mentioned above, although it may override methods of the Object class.


问:
How do I know if tensorflow using cuda and cudnn or not?
I am using Ubuntu 16.04. Here is tensorflow info:

```<code>>>> pip show tensorflow-gpu
pip show tensorflow-gpu
Name: tensorflow-gpu
Version: 1.2.0
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
License: Apache 2.0
Location: /home/xxxx/anaconda3/envs/tensorflow/lib/python3.5/site-packages
Requires: markdown, backports.weakref, wheel, bleach, html5lib, protobuf, numpy, six, werkzeug
</code>```

The cuda info:

```<code>nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17
</code>```

When I import tensorflow in Python from Ubuntu terminal I get no loading information as below.

```<code>>>> import tensorflow
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
</code>```

If I run the python program in terminal, I get different information.

```<code>2017-06-20 16:08:18.075709: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-20 16:08:18.075733: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-20 16:08:18.075740: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-20 16:08:18.075744: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-20 16:08:18.075750: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-06-20 16:08:18.260629: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-06-20 16:08:18.261462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties:
name: Quadro K620M
major: 5 minor: 0 memoryClockRate (GHz) 1.124
pciBusID 0000:08:00.0
Total memory: 1.96GiB
Free memory: 1.58GiB
2017-06-20 16:08:18.261514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0
2017-06-20 16:08:18.261524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y
2017-06-20 16:08:18.261550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Quadro K620M, pci bus id: 0000:08:00.0)
2
</code>```

How do I know if tensorflow is using cuda and cudnn or not? What other information do I need to provide?

答: You can check with nvidia-smi if the GPU is used by the python/tensorflow process. If there is no process using the GPU, tensorflow doesn't use cuda and cudnn.

Body:
I am using Ubuntu 16.04. Here is tensorflow info:

>>> pip show tensorflow-gpu
pip show tensorflow-gpu
Name: tensorflow-gpu
Version: 1.2.0
Summary: TensorFlow helps the tensors flow
Home-page: http://tensorflow.org/
Author: Google Inc.
Author-email: opensource@google.com
License: Apache 2.0
Location: /home/xxxx/anaconda3/envs/tensorflow/lib/python3.5/site-packages
Requires: markdown, backports.weakref, wheel, bleach, html5lib, protobuf, numpy, six, werkzeug

The cuda info:

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2015 NVIDIA Corporation
Built on Tue_Aug_11_14:27:32_CDT_2015
Cuda compilation tools, release 7.5, V7.5.17

When I import tensorflow in Python from Ubuntu terminal I get no loading information as below.

>>> import tensorflow
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally

If I run the python program in terminal, I get different information.

2017-06-20 16:08:18.075709: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-20 16:08:18.075733: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-20 16:08:18.075740: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-20 16:08:18.075744: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-20 16:08:18.075750: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2017-06-20 16:08:18.260629: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:893] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2017-06-20 16:08:18.261462: I tensorflow/core/common_runtime/gpu/gpu_device.cc:940] Found device 0 with properties:
name: Quadro K620M
major: 5 minor: 0 memoryClockRate (GHz) 1.124
pciBusID 0000:08:00.0
Total memory: 1.96GiB
Free memory: 1.58GiB
2017-06-20 16:08:18.261514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:961] DMA: 0
2017-06-20 16:08:18.261524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   Y
2017-06-20 16:08:18.261550: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Quadro K620M, pci bus id: 0000:08:00.0)
2

How do I know if tensorflow is using cuda and cudnn or not? What other information do I need to provide?


问:
Relaying denied. IP name possibly forged [216.136.38.14]')} Python error
I am trying to send mail using SMTP server but I am getting error but the same code is working using gmail.so can you help me to fingd the solution.

```<code>from smtplib import SMTP
import datetime

debuglevel = 0

smtp = SMTP()
smtp.set_debuglevel(debuglevel)
smtp.connect('e3-smtp-gtm.zxp.com', 25)

from_addr = "from mail id"
to_addr = "to mail id"

subj = "hello"
date = datetime.datetime.now().strftime( "%d/%m/%Y %H:%M" )

message_text = "Hello\nThis is a mail from your server\n\nBye\n"

msg = "From: %s\nTo: %s\nSubject: %s\nDate: %s\n\n%s" % ( from_addr, to_addr, subj, date, message_text )
smtp.sendmail(from_addr, to_addr, msg)
smtp.quit()
</code>```

When I run above code I am getting error as below

{'to mail id': (550, b'5.7.1 ...Relaying denied. IP name possibly forged [216.136.38.14]')}

Can anyone help me what is main reason for getting this error

答:
An SMTP server is a protected resource without public access. Without the restrictions spammers would misuse it a lot.

While everybody may submit messages that will be delivered locally (incoming mail), only authorized users may submit messages to be forwarded to other servers on Internet (outgoing mail). Usually the authorization is based on IP address or user/password login.

The meaning of "Relaying denied" is that the SMTP server does not recognize you as an authorized user who can send email wherever he wants. You should provide login/password given to you by the server's admin. In most cases that requires STARTTLS prior to login.

Also, connecting to port 25 is discouraged. This port is intended for server to server mail transfer. The initial message submition should go to the TCP port 587.

Body:
I am trying to send mail using SMTP server but I am getting error but the same code is working using gmail.so can you help me to fingd the solution.

from smtplib import SMTP
import datetime

debuglevel = 0

smtp = SMTP()
smtp.set_debuglevel(debuglevel)
smtp.connect('e3-smtp-gtm.zxp.com', 25)

from_addr = "from mail id"
to_addr = "to mail id"

subj = "hello"
date = datetime.datetime.now().strftime( "%d/%m/%Y %H:%M" )

message_text = "Hello\nThis is a mail from your server\n\nBye\n"

msg = "From: %s\nTo: %s\nSubject: %s\nDate: %s\n\n%s" % ( from_addr, to_addr, subj, date, message_text )
smtp.sendmail(from_addr, to_addr, msg)
smtp.quit()

When I run above code I am getting error as below

{'to mail id': (550, b'5.7.1 ...Relaying denied. IP name possibly forged [216.136.38.14]')}

Can anyone help me what is main reason for getting this error


问:
How to remove TIMED_WAITING threads generated by C3P0 settings
I am very new in c3p0 integration...

I have these settings with c3p0-0.9.5-pre5.jar, hibernate-c3p0-3.5.6-Final.jar, hibernate-core-3.5.6-Final.jar and mchange-commons-java-0.2.6.3.jar jars like below...

```<code>    <property name="cache.provider_class">org.hibernate.cache.NoCacheProvider</property>
    <property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
    <property name="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>

    <property name="hibernate.c3p0.min_size">1</property>
    <property name="hibernate.c3p0.max_size">5</property>
    <property name="hibernate.c3p0.timeout">40</property>
    <property name="hibernate.c3p0.idle_test_period">30</property>
    <!--<property name="hibernate.c3p0.max_statements">50</property>-->
    <property name="hibernate.c3p0.maxStatementsPerConnection">5</property> <!--Instead of max_statements-->
    <property name="hibernate.c3p0.validate">true</property>

    <property name="hibernate.connection.pool_size">25</property>
    <property name="hibernate.c3p0.acquire_increment">1</property>
    <property name="hibernate.c3p0.automaticTestTable">con_test</property>
    <property name="hibernate.c3p0.privilegeSpawnedThreads">true</property>
    <property name="hibernate.c3p0.contextClassLoaderSource">library</property>
    <property name="hibernate.c3p0.maxAdministrativeTaskTime">30</property>
    <property name="hibernate.c3p0.numHelperThreads">20</property>
</code>```

The problem is, application generates thousands of threads and keep those as Time waited Threads.

I have print some of those threads by a loop ..

```<code>"<br/>" + c++ +". "+ t.getState() + " (" + t.isAlive() + ") : " + t.getName();
</code>```

results is below...

```<code>    147. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|2feccbb3]-HelperThread-#0
    148. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|8d0e89c]-HelperThread-#3
    149. WAITING (true) : Reference Handler
    150. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|8d0e89c]-HelperThread-#2
    151. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|1045f6be]-HelperThread-#8
    152. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|1045f6be]-HelperThread-#19
    153. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|3b0c81d2]-HelperThread-#17
    154. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|2feccbb3]-HelperThread-#3
    155. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|2feccbb3]-HelperThread-#1
    156. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|70cef37d]-HelperThread-#19
</code>```

This is increasing very first when retrieve data from database by application.

App developed by Java, Struts-1, Hibernate, Oracle(BD).

How can I remove/kill these threads

答:
One way or some other, if you are seeing thousand of these Threads, you are leaking DataSources. That is, your application is constructing c3p0 DataSources, each of which has its own complement of Threads, then it is losing or dereferencing or replacing them without first close()ing them.

A pooled DataSource should be constructed once, placed somewhere with shared availability, and reused over and over again. If, unusually, a DataSource needs to be reconstructed for some reason, you need to close() c3p0 DataSources or their Threads will live forever.

Perhaps the most common error that leads to this sort of thing are applications that hot-redeploy. If on app initialization a DataSource is created, in a shutdown hook in the redeploy cycle you must take car that the same DataSource gets destroyed.

Note that in the list of Threads above, you show many Threads from different DataSources (as they have different identity tokens after the common VMID portion before the |). You are definitely creating but not close()ing lots of DataSources.

Body:
I am very new in c3p0 integration...

I have these settings with c3p0-0.9.5-pre5.jar, hibernate-c3p0-3.5.6-Final.jar, hibernate-core-3.5.6-Final.jar and mchange-commons-java-0.2.6.3.jar jars like below...

    <property name="cache.provider_class">org.hibernate.cache.NoCacheProvider</property>
    <property name="hibernate.connection.provider_class">org.hibernate.connection.C3P0ConnectionProvider</property>
    <property name="hibernate.transaction.factory_class">org.hibernate.transaction.JDBCTransactionFactory</property>

    <property name="hibernate.c3p0.min_size">1</property>
    <property name="hibernate.c3p0.max_size">5</property>
    <property name="hibernate.c3p0.timeout">40</property>
    <property name="hibernate.c3p0.idle_test_period">30</property>

    <property name="hibernate.c3p0.maxStatementsPerConnection">5</property>
    <property name="hibernate.c3p0.validate">true</property>

    <property name="hibernate.connection.pool_size">25</property>
    <property name="hibernate.c3p0.acquire_increment">1</property>
    <property name="hibernate.c3p0.automaticTestTable">con_test</property>
    <property name="hibernate.c3p0.privilegeSpawnedThreads">true</property>
    <property name="hibernate.c3p0.contextClassLoaderSource">library</property>
    <property name="hibernate.c3p0.maxAdministrativeTaskTime">30</property>
    <property name="hibernate.c3p0.numHelperThreads">20</property>

The problem is, application generates thousands of threads and keep those as Time waited Threads.

I have print some of those threads by a loop ..

"
" + c++ +". "+ t.getState() + " (" + t.isAlive() + ") : " + t.getName();

results is below...

    147. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|2feccbb3]-HelperThread-#0
    148. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|8d0e89c]-HelperThread-#3
    149. WAITING (true) : Reference Handler
    150. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|8d0e89c]-HelperThread-#2
    151. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|1045f6be]-HelperThread-#8
    152. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|1045f6be]-HelperThread-#19
    153. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|3b0c81d2]-HelperThread-#17
    154. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|2feccbb3]-HelperThread-#3
    155. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|2feccbb3]-HelperThread-#1
    156. TIMED_WAITING (true) : C3P0PooledConnectionPoolManager[identityToken->1hge86f9r1gp0vs6si1few|70cef37d]-HelperThread-#19

This is increasing very first when retrieve data from database by application.

App developed by Java, Struts-1, Hibernate, Oracle(BD).

How can I remove/kill these threads


问:
Java ME object persistence
I know that Serialization (Serializable) is not available in the Micro Edition of Java.
It's kinda straight forward to save primitives like int and java.lang.String objects with the RMS.
But if I want to save (make persistant) an arbitrary object? Is that possible?</p>

答:
You can use the Connection API to write the data to a File/Stream. Please keep in mind that it is easy to persist the data in to a Stream but inorder to restore the data from the Stream you will have to write your own code. Usually reflection is used for this purpose and it becomes a bottle neck on many devices.

So instead of re-inventing the wheel I would advise using Floggy (http://floggy.sourceforge.net/). Floggy is a free object persistence framework for J2ME/MIDP applications. The main goal of this framework is to abstract the data persistence details from the developer, reducing the development and maintenance effort. You can check the examples at http://floggy.sourceforge.net/getting-started.html#framework

OwnerDisplayName: anon

LastEditorDisplayName: Roger

Title: Java ME object persistence

OwnerDisplayName: Faisal Feroz


问:
ffmpeg - overlay multiple fading texts with different colors
I have problem with this ffmpeg command, it works fine if the fading text is in white font color, but if I change the fontcolor to something else (for example black), the fading text will not appear, any ideas?

```<code>ffmpeg -i ./based_video/480/clip3.mp4 -filter_complex "color=black:100x100[c]; [c][0]scale2ref[ct][mv0]; \
[ct]setsar=1,split=3[t1][t2][t3]; \
[t1]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,split[text1][alpha1]; \
[text1][alpha1]alphamerge,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta1]; \
[t2]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,split[text2][alpha2]; \
[text2][alpha2]alphamerge,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta2]; \
[t3]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,split[text3][alpha3]; \
[text3][alpha3]alphamerge,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta3]; \
[mv0][txta1]overlay=x='100':y='200':shortest=1[mv1]; \
[mv1][txta2]overlay=x='300':y='200':shortest=1[mv2]; \
[mv2][txta3]overlay=x='500':y='200':shortest=1" \
-c:v libx264 -c:a copy ./output_video/testnew-clip3-output.mp4
</code>```

full log is here :
https://docs.google.com/document/d/1y9Dnn0Df75J8P_hZ6LjHTX2dk-8z97UnTjlX8dnc0v0/edit?usp=sharing

Thanks in advance

答:
You're feeding the plane with the drawn text as the alpha. If the text is black, then the alpha is black and so the text won't appear. You can skip the alpha creation and merging altogether.</p>

```<code>ffmpeg -i ./based_video/480/clip3.mp4 -filter_complex "color=black@0:100x100,format=yuva444p[c]; [c][0]scale2ref[ct][mv0]; \
[ct]setsar=1,split=3[t1][t2][t3]; \
[t1]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta1]; \
[t2]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta2]; \
[t3]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta3]; \
[mv0][txta1]overlay=x='100':y='200':shortest=1[mv1]; \
[mv1][txta2]overlay=x='300':y='200':shortest=1[mv2]; \
[mv2][txta3]overlay=x='500':y='200':shortest=1" \
-c:v libx264 -c:a copy ./output_video/testnew-clip3-output.mp4
</code>```

Body:
I have problem with this ffmpeg command, it works fine if the fading text is in white font color, but if I change the fontcolor to something else (for example black), the fading text will not appear, any ideas?

ffmpeg -i ./based_video/480/clip3.mp4 -filter_complex "color=black:100x100[c]; [c][0]scale2ref[ct][mv0]; \
[ct]setsar=1,split=3[t1][t2][t3]; \
[t1]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,split[text1][alpha1]; \
[text1][alpha1]alphamerge,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta1]; \
[t2]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,split[text2][alpha2]; \
[text2][alpha2]alphamerge,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta2]; \
[t3]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,split[text3][alpha3]; \
[text3][alpha3]alphamerge,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta3]; \
[mv0][txta1]overlay=x='100':y='200':shortest=1[mv1]; \
[mv1][txta2]overlay=x='300':y='200':shortest=1[mv2]; \
[mv2][txta3]overlay=x='500':y='200':shortest=1" \
-c:v libx264 -c:a copy ./output_video/testnew-clip3-output.mp4

full log is here :
https://docs.google.com/document/d/1y9Dnn0Df75J8P_hZ6LjHTX2dk-8z97UnTjlX8dnc0v0/edit?usp=sharing

Thanks in advance

Body:
You're feeding the plane with the drawn text as the alpha. If the text is black, then the alpha is black and so the text won't appear. You can skip the alpha creation and merging altogether.

ffmpeg -i ./based_video/480/clip3.mp4 -filter_complex "color=black@0:100x100,format=yuva444p[c]; [c][0]scale2ref[ct][mv0]; \
[ct]setsar=1,split=3[t1][t2][t3]; \
[t1]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta1]; \
[t2]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta2]; \
[t3]drawtext=fontfile=/usr/share/fonts/truetype/roboto/Roboto-Bold.ttf:text='\$30,000.0':fontsize=40:fontcolor=white,fade=t=in:st=1:d=1:alpha=1,fade=t=out:st=5:d=1:alpha=1[txta3]; \
[mv0][txta1]overlay=x='100':y='200':shortest=1[mv1]; \
[mv1][txta2]overlay=x='300':y='200':shortest=1[mv2]; \
[mv2][txta3]overlay=x='500':y='200':shortest=1" \
-c:v libx264 -c:a copy ./output_video/testnew-clip3-output.mp4


问:
Postgres - simplify repeat of select subqueries based on dates
Newbie with Postgres (9.6).

I have 2 tables

sales_account (list of customers)

journal (transactions line invoices payments)

I want to create another table called balances which has a summary of outstanding transactions based on periods (like 7 days, 14 days, 30 days, 60 days or older).

I have written the query in long hand as sub-queries (it takes far too long to generate - there are over 15m journal records) but am looking for any hints as to how I can simplify the query.

Thank you.

```<code>INSERT INTO balances (sales_account,
    balance,
    period_1,
    period_2,
    period_3,
    period_4,
    period_5)

SELECT  sales_account.id,

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active'), -- overall balance

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active' AND
journal.reversed_by is null AND
journal.original_id is null AND
journal.transaction_date <= CURRENT_DATE - interval '7 days'), -- current

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active' AND
journal.reversed_by is null AND
journal.original_id is null AND
journal.transaction_date <= CURRENT_DATE  - interval '14 days' AND
journal.transaction_date >= CURRENT_DATE - interval '8 days'), -- 8-14 days

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active' AND
journal.transaction_date <=CURRENT_DATE  - interval '30 days' AND
journal.transaction_date >= CURRENT_DATE - interval '15 days'), -- 22-30 days

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active' AND
journal.transaction_date >= CURRENT_DATE - interval '31 days' AND
journal.transaction_date <=CURRENT_DATE  - interval '60 days'), -- 31-60 days

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active' AND
journal.transaction_date >= CURRENT_DATE - interval '61 days') -- 61+ days

FROM sales_account;
</code>```

答:
This can be simplified to one query using case:</p>

```<code>   SELECT
       coalesce(sum(case
         when journal.transaction_date >= CURRENT_DATE - interval '8 days' AND
              journal.transaction_date <= CURRENT_DATE  - interval '14 days'
         then
          journal.outstanding else 0
         end),0) days814,
       coalesce(sum(case
         when journal.transaction_date >= CURRENT_DATE - interval '22 days' AND
              journal.transaction_date <= CURRENT_DATE  - interval '30 days'
         then
          journal.outstanding else 0
         end),0) days2230,
      coalesce(sum(case
         when journal.transaction_date >= CURRENT_DATE - interval '31 days' AND
              journal.transaction_date <= CURRENT_DATE  - interval '60 days'
         then
          journal.outstanding else 0
         end),0) days3160,
      coalesce(sum(case
         when journal.transaction_date >= CURRENT_DATE - interval '61 days'
         then
          journal.outstanding else 0
         end),0) days61,
    FROM journal, sales_account
    WHERE journal.sales_account = sales_account.id AND
    journal.status='Active'
</code>```

Body:
Newbie with Postgres (9.6).

I have 2 tables

sales_account (list of customers)

journal (transactions line invoices payments)

I want to create another table called balances which has a summary of outstanding transactions based on periods (like 7 days, 14 days, 30 days, 60 days or older).

I have written the query in long hand as sub-queries (it takes far too long to generate - there are over 15m journal records) but am looking for any hints as to how I can simplify the query.

Thank you.

INSERT INTO balances (sales_account,
    balance,
    period_1,
    period_2,
    period_3,
    period_4,
    period_5)

SELECT  sales_account.id,

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active'), -- overall balance

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active' AND
journal.reversed_by is null AND
journal.original_id is null AND
journal.transaction_date <= CURRENT_DATE - interval '7 days'), -- current

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active' AND
journal.reversed_by is null AND
journal.original_id is null AND
journal.transaction_date <= CURRENT_DATE  - interval '14 days' AND
journal.transaction_date >= CURRENT_DATE - interval '8 days'), -- 8-14 days

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active' AND
journal.transaction_date <=CURRENT_DATE  - interval '30 days' AND
journal.transaction_date >= CURRENT_DATE - interval '15 days'), -- 22-30 days

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active' AND
journal.transaction_date >= CURRENT_DATE - interval '31 days' AND
journal.transaction_date <=CURRENT_DATE  - interval '60 days'), -- 31-60 days

(SELECT coalesce(sum(journal.outstanding),0)
FROM journal
WHERE journal.sales_account = sales_account.id AND
journal.status='Active' AND
journal.transaction_date >= CURRENT_DATE - interval '61 days') -- 61+ days

FROM sales_account;

Body:
This can be simplified to one query using case:

   SELECT
       coalesce(sum(case
         when journal.transaction_date >= CURRENT_DATE - interval '8 days' AND
              journal.transaction_date <= CURRENT_DATE  - interval '14 days'
         then
          journal.outstanding else 0
         end),0) days814,
       coalesce(sum(case
         when journal.transaction_date >= CURRENT_DATE - interval '22 days' AND
              journal.transaction_date <= CURRENT_DATE  - interval '30 days'
         then
          journal.outstanding else 0
         end),0) days2230,
      coalesce(sum(case
         when journal.transaction_date >= CURRENT_DATE - interval '31 days' AND
              journal.transaction_date <= CURRENT_DATE  - interval '60 days'
         then
          journal.outstanding else 0
         end),0) days3160,
      coalesce(sum(case
         when journal.transaction_date >= CURRENT_DATE - interval '61 days'
         then
          journal.outstanding else 0
         end),0) days61,
    FROM journal, sales_account
    WHERE journal.sales_account = sales_account.id AND
    journal.status='Active'


问:
Can't load 'mnist-original' dataset using sklearn
This question is similar to what asked here and here. Unfortunately, in my case the suggested solution didn't fix the problem.

I need to work with the MNIST dataset but I can't fetch it, even if I specify the address of the scikit_learn_data/mldata/ folder (see below). How can I fix this?

In case it might help, I'm using Anaconda.

Code:

```<code>from sklearn.datasets.mldata import fetch_mldata

dataset = fetch_mldata('mnist-original', data_home='/Users/michelangelo/scikit_learn_data/mldata/')
mnist = fetch_mldata('MNIST original')
</code>```

Error:

```<code>---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-5-dc4d45bc928e> in <module>()
----> 1 mnist = fetch_mldata('MNIST original')

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/sklearn/datasets/mldata.pyc in fetch_mldata(dataname, target_name, data_name, transpose_data, data_home)
    168     # load dataset matlab file
    169     with open(filename, 'rb') as matlab_file:
--> 170         matlab_dict = io.loadmat(matlab_file, struct_as_record=True)
    171
    172     # -- extract data from matlab_dict

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/scipy/io/matlab/mio.pyc in loadmat(file_name, mdict, appendmat, **kwargs)
    134     variable_names = kwargs.pop('variable_names', None)
    135     MR = mat_reader_factory(file_name, appendmat, **kwargs)
--> 136     matfile_dict = MR.get_variables(variable_names)
    137     if mdict is not None:
    138         mdict.update(matfile_dict)

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/scipy/io/matlab/mio5.pyc in get_variables(self, variable_names)
    290                 continue
    291             try:
--> 292                 res = self.read_var_array(hdr, process)
    293             except MatReadError as err:
    294                 warnings.warn(

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/scipy/io/matlab/mio5.pyc in read_var_array(self, header, process)
    250            `process`.
    251         '''
--> 252         return self._matrix_reader.array_from_header(header, process)
    253
    254     def get_variables(self, variable_names=None):

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.array_from_header()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.array_from_header()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_real_complex()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_numeric()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_element()

streams.pyx in scipy.io.matlab.streams.FileStream.read_string()

IOError: could not read bytes
</code>```

答: For people having the same issue: it was a connection problem. If you get a similar error, check that you have the entire mnist-original.mat file, as suggested by @vivek-kumar. Current file size: 55.4 MB.

Body:
This question is similar to what asked here and here. Unfortunately, in my case the suggested solution didn't fix the problem.

I need to work with the MNIST dataset but I can't fetch it, even if I specify the address of the scikit_learn_data/mldata/ folder (see below). How can I fix this?

In case it might help, I'm using Anaconda.

Code:

from sklearn.datasets.mldata import fetch_mldata

dataset = fetch_mldata('mnist-original', data_home='/Users/michelangelo/scikit_learn_data/mldata/')
mnist = fetch_mldata('MNIST original')

Error:

---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-5-dc4d45bc928e> in <module>()
----> 1 mnist = fetch_mldata('MNIST original')

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/sklearn/datasets/mldata.pyc in fetch_mldata(dataname, target_name, data_name, transpose_data, data_home)
    168     # load dataset matlab file
    169     with open(filename, 'rb') as matlab_file:
--> 170         matlab_dict = io.loadmat(matlab_file, struct_as_record=True)
    171
    172     # -- extract data from matlab_dict

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/scipy/io/matlab/mio.pyc in loadmat(file_name, mdict, appendmat, **kwargs)
    134     variable_names = kwargs.pop('variable_names', None)
    135     MR = mat_reader_factory(file_name, appendmat, **kwargs)
--> 136     matfile_dict = MR.get_variables(variable_names)
    137     if mdict is not None:
    138         mdict.update(matfile_dict)

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/scipy/io/matlab/mio5.pyc in get_variables(self, variable_names)
    290                 continue
    291             try:
--> 292                 res = self.read_var_array(hdr, process)
    293             except MatReadError as err:
    294                 warnings.warn(

/Users/michelangelo/anaconda2/lib/python2.7/site-packages/scipy/io/matlab/mio5.pyc in read_var_array(self, header, process)
    250            `process`.
    251         '''
--> 252         return self._matrix_reader.array_from_header(header, process)
    253
    254     def get_variables(self, variable_names=None):

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.array_from_header()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.array_from_header()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_real_complex()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_numeric()

mio5_utils.pyx in scipy.io.matlab.mio5_utils.VarReader5.read_element()

streams.pyx in scipy.io.matlab.streams.FileStream.read_string()

IOError: could not read bytes


问:
Regex real number
I need real numbers in TextBox. I try my code online here

"^-{0,1}[0-9]{1,3},{0,1}[0-9]{1,2}$"

and its working perfectly, but in my project not working.
Please show me how I must do it

答:
Would TryParse works for you? From MSDN:

Converts the string representation of a number in a specified style and culture-specific format to its double-precision floating-point number equivalent. A return value indicates whether the conversion succeeded or failed.

And for you, it could be something like this:

```<code>private void c_PreviewTextInput(object sender, TextCompositionEventArgs e)
{
    double num;
    e.Handled = double.TryParse(e.Text, out num);
    // if e.Text is a number, e.Handled will be true and num = e.Text
}
</code>```

Title: Regex real number

Body:
Would TryParse works for you? From MSDN:

Converts the string representation of a number in a specified style and culture-specific format to its double-precision floating-point number equivalent. A return value indicates whether the conversion succeeded or failed.

And for you, it could be something like this:

private void c_PreviewTextInput(object sender, TextCompositionEventArgs e)
{
    double num;
    e.Handled = double.TryParse(e.Text, out num);
    // if e.Text is a number, e.Handled will be true and num = e.Text
}


问:
Check the count of column value is equal to one based on condition in sql server
I have two tables

Student

```<code>StudentId    |  StudentName
----------   |  --------------
1            |  John
2            |  Susan
3            |  Andy
4            |  Joe
</code>```

Department

```<code>StudentId    |    DepartmentId
----------   |    ------------
1            |    123
1            |    234
2            |    123
2            |    456
3            |    123
4            |    456
</code>```

Each student can be in multiple departments but I have to find those students which are only in one department like student 3 and 4

Any help?

答:
Simply join them & aggregate them by Group by clause with having.. count() function

```<code>SELECT s.StudentId, s.StudentName
FROM student s
     JOIN Department d ON d.StudentId = s.StudentId
GROUP BY s.StudentId, s.StudentName
HAVING COUNT(d.DepartmentId) = 1;
</code>```

Result :

```<code>studentid   studentname
3           Andy
4           Joe
</code>```

Body:
I have two tables

Student

StudentId    |  StudentName
----------   |  --------------
1            |  John
2            |  Susan
3            |  Andy
4            |  Joe

Department

StudentId    |    DepartmentId
----------   |    ------------
1            |    123
1            |    234
2            |    123
2            |    456
3            |    123
4            |    456

Each student can be in multiple departments but I have to find those students which are only in one department like student 3 and 4

Any help?

Body:
Simply join them & aggregate them by Group by clause with having.. count() function

SELECT s.StudentId, s.StudentName
FROM student s
     JOIN Department d ON d.StudentId = s.StudentId
GROUP BY s.StudentId, s.StudentName
HAVING COUNT(d.DepartmentId) = 1;

Result :

studentid   studentname
3           Andy
4           Joe


问:
Powershell invoke script in new process and within the same runspace
I got within my application (process) an already configured c# Runspace v3.
I want to execute my PowerShell script in a new process, but within the same Runspace.
How could I accomplish this?
I thought, when I start a PowerShell within an existing runspace, then that Runspace will be inherited to the new PowerShell: Unfortunately, this was not the case...

I have seen following interesting methods, but got no solution how these could help:
RunspaceFactory.CreateOutOfProcessRunspace
Invoke-Command

PS: I can not change within my app the access from Runspace to a PowerShellProcessInstance  and also I can not use PSRemoting

答:
What you describe is impossible - a single runspace cannot span multiple processes.

Allow me to demonstrate the relevant hierarchical relationships with this state of the art graphic I whipped up in mspaint:

A runspace is bound to a single .NET AppDomain, and an AppDomain cannot span multiple processes.


答:
You can use IN with department table group by StudentId alongwith having count(Student_Id) = 1 in sub-query  as below.

```<code>SELECT *
FROM student s
WHERE s.StudentId IN (
        SELECT d.StudentId
        FROM Department d
        GROUP BY d.StudentId
        HAVING count(d.DepartmentId) = 1
        );
</code>```

Result:

```<code>+-----------+-------------+
| studentid | studentname |
+-----------+-------------+
|         3 | Andy        |
|         4 | Joe         |
+-----------+-------------+
</code>```

DEMO

Title: Check the count of column value is equal to one based on condition in sql server

Body:
You can use IN with department table group by StudentId alongwith having count(Student_Id) = 1 in sub-query  as below.

SELECT *
FROM student s
WHERE s.StudentId IN (
        SELECT d.StudentId
        FROM Department d
        GROUP BY d.StudentId
        HAVING count(d.DepartmentId) = 1
        );

Result:

+-----------+-------------+
| studentid | studentname |
+-----------+-------------+
|         3 | Andy        |
|         4 | Joe         |
+-----------+-------------+

DEMO


问:
How to access services in a different Kubernetes cluster
For improved performance and availability we'd like to distribute certain services from out stack across different Kubernetes clusters in different parts of the world (GCP regions).

The majority of our stack will continue to run in one cluster / region but some user facing services will be deployed all over the world.

Some of these services need to access other services in our main cluster.

Q: How can we reliably access services in a different Kubernetes cluster?

Using internal load balancers seems to be out of the question as those are per region only.

We'd like to keep the communication between our services inside the private GCP network and avoid going over the public internet. So an public ingress also wouldn't work.

答:
Take a look at VPN Peering: https://cloud.google.com/vpc/docs/vpc-peering

It allows you to connect two vpcs (in different regions) so that they can communicate privately.
You may have to recreate/reconfigure your Kubernetes in order to support this vpc architecture.


问:
Optimization for Matrix Multiply (OpenMP) - C
I am learning a little bit about openMP and trying to use it here to multiply two matrices together.

```<code>void matrix_multiply(matrix *A, matrix *B, matrix *C) {

    #pragma omp parallel
    {
        #pragma omp for
        for(int i = 0; i < A->dim.rows; i++) {
            for(int j = 0; j < B->dim.cols; j++) {
                C->data[i][j] = 0;
                for (int k = 0; k < A->dim.cols; k++) {
                C->data[i][j] += A->data[i][k] * B->data[k][j];
               }
           }
       }
   }
}

typedef struct shape {
    int rows;
    int cols;
} shape;

typedef struct matrix {
    shape dim;
    float** data;
} matrix;
</code>```

Still a little new to this, so are there any simple changes to improve performance or have I already done that? Also am I running into any data races by not using reduction?

答:
Your current implementation can probably not be improved a lot. At this point it comes down to the compiler and cache usage. An interesting point is made here by Intel that GCC requires two loops to be swapped in order to vectorize the multiplications (i.e. use SIMD). For very large matrices, you might consider dividing the matrices not in stripes but in blocks. This introduces complexity and overhead, but can improve cache usage.

The reduction clause is only needed if you are summing a single variable with multiple threads, which is not the case here since you only sum over k.

Finally (but this is completely personal) note that you can replace the two directives by a single one

```<code>#pragma omp parallel for
</code>```

which in my opinion looks somewhat cleaner.

Body:
I am learning a little bit about openMP and trying to use it here to multiply two matrices together.

void matrix_multiply(matrix *A, matrix *B, matrix *C) {

    #pragma omp parallel
    {
        #pragma omp for
        for(int i = 0; i dim.rows; i++) {
            for(int j = 0; j dim.cols; j++) {
                C->data[i][j] = 0;
                for (int k = 0; k dim.cols; k++) {
                C->data[i][j] += A->data[i][k] * B->data[k][j];
               }
           }
       }
   }
}

typedef struct shape {
    int rows;
    int cols;
} shape;

typedef struct matrix {
    shape dim;
    float** data;
} matrix;

Still a little new to this, so are there any simple changes to improve performance or have I already done that? Also am I running into any data races by not using reduction?

Body:
Your current implementation can probably not be improved a lot. At this point it comes down to the compiler and cache usage. An interesting point is made here by Intel that GCC requires two loops to be swapped in order to vectorize the multiplications (i.e. use SIMD). For very large matrices, you might consider dividing the matrices not in stripes but in blocks. This introduces complexity and overhead, but can improve cache usage.

The reduction clause is only needed if you are summing a single variable with multiple threads, which is not the case here since you only sum over k.

Finally (but this is completely personal) note that you can replace the two directives by a single one

#pragma omp parallel for

which in my opinion looks somewhat cleaner.


问:
Nodejs Redis and Waterfall not passing value to next function
I've been trying to follow the async waterfall example shown as below;

```<code>async.waterfall([
    function(callback) {
        callback(null, 'one', 'two');
    },
    function(arg1, arg2, callback) {
        // arg1 now equals 'one' and arg2 now equals 'two'
        callback(null, 'three');
    },
    function(arg1, callback) {
        // arg1 now equals 'three'
        callback(null, 'done');
    }
], function (err, result) {
    // result now equals 'done'
});
</code>```

Here is my version of the code;

```<code>var successfulRecords = 0 //successfulRecords holds the number of successfully processed rows

       //Delete existing records for prepaid model before adding new records
        async.waterfall([
            function(callback){
                redis.del("prepaid",function(err,data){
                    if (err){
                        logger.debug(err);
                    }

                    SCORES_CACHE = new Map(
                        rows.map(function(row){
                            redis.zadd("prepaid",row[1],row[0],function(err,response){

                                if (err){
                                    logger.debug(err);
                                }

                                if (response==1){
                                    successfulRecords=successfulRecords+1;
                                    console.log(successfulRecords);
                                }
                            })
                        })
                        )

                })
                callback(null,successfulRecords);
            },

            function(arg1,callback){
                console.log('hello, waiting for arg1: ',arg1);
            }
            ], function(err,result){

            });
</code>```

Code supposed to work like this

Insert record into redis DB

If successful, update the successful record counter

Pass the final count of the successfulRecord variable to the next function

However, i keep getting this result instead;

```<code>hello, waiting for arg1:  0
</code>```

How can i fix this? Been working on this for hours. Did I not place the callback correctly?

Thanks in advance!

答:
This callback(null,successfulRecords); is called before the redis.del is complete. Move it one line up.

```<code>function(callback){
    redis.del("prepaid",function(err,data){
        if (err){
            logger.debug(err);
        }

        SCORES_CACHE = new Map(
            rows.map(function(row){
                redis.zadd("prepaid",row[1],row[0],function(err,response){

                    if (err){
                        logger.debug(err);
                    }

                    if (response==1){
                        successfulRecords=successfulRecords+1;
                        console.log(successfulRecords);
                    }
                })
            })
        )
        callback(null,successfulRecords); //One line up
    })

}
</code>```

Body:
I've been trying to follow the async waterfall example shown as below;

async.waterfall([
    function(callback) {
        callback(null, 'one', 'two');
    },
    function(arg1, arg2, callback) {
        // arg1 now equals 'one' and arg2 now equals 'two'
        callback(null, 'three');
    },
    function(arg1, callback) {
        // arg1 now equals 'three'
        callback(null, 'done');
    }
], function (err, result) {
    // result now equals 'done'
});

Here is my version of the code;

var successfulRecords = 0 //successfulRecords holds the number of successfully processed rows

       //Delete existing records for prepaid model before adding new records
        async.waterfall([
            function(callback){
                redis.del("prepaid",function(err,data){
                    if (err){
                        logger.debug(err);
                    }

                    SCORES_CACHE = new Map(
                        rows.map(function(row){
                            redis.zadd("prepaid",row[1],row[0],function(err,response){

                                if (err){
                                    logger.debug(err);
                                }

                                if (response==1){
                                    successfulRecords=successfulRecords+1;
                                    console.log(successfulRecords);
                                }
                            })
                        })
                        )

                })
                callback(null,successfulRecords);
            },

            function(arg1,callback){
                console.log('hello, waiting for arg1: ',arg1);
            }
            ], function(err,result){

            });

Code supposed to work like this

Insert record into redis DB

If successful, update the successful record counter

Pass the final count of the successfulRecord variable to the next function

However, i keep getting this result instead;

hello, waiting for arg1:  0

How can i fix this? Been working on this for hours. Did I not place the callback correctly?

Thanks in advance!

Body:
This callback(null,successfulRecords); is called before the redis.del is complete. Move it one line up.

function(callback){
    redis.del("prepaid",function(err,data){
        if (err){
            logger.debug(err);
        }

        SCORES_CACHE = new Map(
            rows.map(function(row){
                redis.zadd("prepaid",row[1],row[0],function(err,response){

                    if (err){
                        logger.debug(err);
                    }

                    if (response==1){
                        successfulRecords=successfulRecords+1;
                        console.log(successfulRecords);
                    }
                })
            })
        )
        callback(null,successfulRecords); //One line up
    })

}


问:
How can I get the choose data (such as foreign data) like the RestFramework browser?
How can I get the choose data (such as foreign data) like the RestFramework browser?

As all we know, if I have a model:

```<code>class Disk(models.Model):
    diskEssenceType = models.ForeignKey(to=DiskEssenceType, related_name='disks')  #  "SAS", "SSD"
    diskOsType = models.ForeignKey(to=DiskOSType, related_name='disks') #
    hostType = models.ForeignKey(to=HostType, related_name='disks', on_delete=models.CASCADE)
    price = models.DecimalField(max_digits=8, decimal_places=2, default=0.00)  # price
</code>```

There are three foreign fields.

In the serializers:

```<code>class DiskCreateSerializer(ModelSerializer):

    class Meta:
        model = Disk
        fields = "__all__"
</code>```

And the views.py:

```<code>class DiskCreateAPIView(CreateAPIView):
    serializer_class = DiskCreateSerializer
    permission_classes = []
    queryset = Disk.objects.all()
</code>```

When I access this create link:

You see, there are serial optional drop-down boxes we can choose to add in my Disk instance creation.

So, how can we do this like in our custom website html so easily?

Or whether rest framework have return the drop-down boxes data, but I don't know.

答:
So, how can we do this like in our custom website html so easily?

This used to be possible by making an OPTION request on the resource. This pull request removed it.

So it's down to:

either write your own metadata class to reintroduce it

either add another endpoint that lists all the associated data

Body:
How can I get the choose data (such as foreign data) like the RestFramework browser?

As all we know, if I have a model:

class Disk(models.Model):
    diskEssenceType = models.ForeignKey(to=DiskEssenceType, related_name='disks')  #  "SAS", "SSD"
    diskOsType = models.ForeignKey(to=DiskOSType, related_name='disks') #
    hostType = models.ForeignKey(to=HostType, related_name='disks', on_delete=models.CASCADE)
    price = models.DecimalField(max_digits=8, decimal_places=2, default=0.00)  # price

There are three foreign fields.

In the serializers:

class DiskCreateSerializer(ModelSerializer):

    class Meta:
        model = Disk
        fields = "__all__"

And the views.py:

class DiskCreateAPIView(CreateAPIView):
    serializer_class = DiskCreateSerializer
    permission_classes = []
    queryset = Disk.objects.all()

When I access this create link:

You see, there are serial optional drop-down boxes we can choose to add in my Disk instance creation.

So, how can we do this like in our custom website html so easily?

Or whether rest framework have return the drop-down boxes data, but I don't know.