问:
Linux, UDP datagrams, and kernel timestamps: Lots of examples and stackoversflow entries later, and still cannot get timestamps at all
I have been trying and failing to get Linux (kernel 4.1.4) to give me timestamps for when UDP datagrams are sent and received.  I have read the original kernel docs (https://www.kernel.org/doc/Documentation/networking/timestamping.txt), along with lots of examples and a number of stackoverflow entries.  I can send datagrams between sender and receiver with no problems.  But I cannot get timestamps for sending or receiving datagrams, and I can't figure out what I'm doing wrong.

One bizarre thing is that when I use the MSG_ERRQUEUE channel for getting timestamp info on a sent datagram, I do get the original outgoing packet, and I do get the first ancillary message (SOL_IP, IP_RECVERR), but I do not get a second message (which should be level SOL_SOCKET, type SCM_TIMESTAMPING).

In another stackoverflow entry on getting timestamps for sent packets (Timestamp outgoing packets), someone mentioned that some drivers might not implement the call to skb_tx_timestamp, but I checked mine (Realtek), and that call is definitely in there.

Here's how I set up the UDP receiver (error handling code not shown):

```<code>inf->fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

timestampOn = SOF_TIMESTAMPING_RX_SOFTWARE | SOF_TIMESTAMPING_RX_HARDWARE;
r = setsockopt(inf->fd, SOL_SOCKET, SO_TIMESTAMPING, &timestampOn, sizeof(timestampOn));

r = setsockopt(inf->fd, SOL_SOCKET, SO_REUSEPORT, &on, sizeof(on));

memset(&(inf->local), 0, sizeof(struct sockaddr_in));
inf->local.sin_family = AF_INET;
inf->local.sin_port = htons(port);
inf->local.sin_addr.s_addr = htonl(INADDR_ANY);

r = bind(inf->fd, (struct sockaddr *)&(inf->local), sizeof(struct sockaddr_in));
</code>```

Using SO_REUSEPORT or not doesn't seem to matter.

For receiving, my understanding is that we don't use MSG_ERRQUEUE.  That's only if we want timestamps for sent messages.  Besides, when I use MSG_ERRQUEUE with recvmsg, I get "resource temporarily unavailable."  Here's how I receive datagrams:

```<code>int recv_len;
struct msghdr   msg;
struct iovec    iov;

memset(&msg, 0, sizeof(msg));
memset(&iov, 0, sizeof(iov));

// Space for control message info plus timestamp
char ctrl[2048];
memset(ctrl, 0, sizeof(ctrl));
//struct cmsghdr *cmsg = (struct cmsghdr *) &ctrl;

// Ancillary data buffer and length
msg.msg_control      = (char *) ctrl;
msg.msg_controllen   = sizeof(ctrl);

// Dest address info
msg.msg_name         = (struct sockaddr *) &(inf->remote);
msg.msg_namelen      = sizeof(struct sockaddr_in);

// Array of data buffers (scatter/gather)
msg.msg_iov          = &iov;
msg.msg_iovlen       = 1;

// Data buffer pointer and length
iov.iov_base         = buf;
iov.iov_len          = len;

recv_len = recvmsg(inf->fd, &msg, 0);
</code>```

And then I pass a pointer to msg to another function (handle_time) that does this:

```<code>struct timespec* ts = NULL;
struct cmsghdr* cmsg;
struct sock_extended_err *ext;

for( cmsg = CMSG_FIRSTHDR(msg); cmsg; cmsg = CMSG_NXTHDR(msg,cmsg) ) {
    printf("level=%d, type=%d, len=%zu\n", cmsg->cmsg_level, cmsg->cmsg_type, cmsg->cmsg_len);
}
</code>```

Zero messages are received.  So that's the first problem.  My setup code above matches like half a dozen other examples I've found on the web, but I'm getting no ancillary data from this.

Next, let's turn to sending datagrams.  Here's the setup:

```<code>inf->port = port;
inf->fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

memset(&(inf->remote), 0, sizeof(struct sockaddr_in));
inf->remote.sin_family = AF_INET;
inf->remote.sin_port = htons(port);

timestampOn = SOF_TIMESTAMPING_TX_SOFTWARE | SOF_TIMESTAMPING_TX_HARDWARE;
r = setsockopt(inf->fd, SOL_SOCKET, SO_TIMESTAMPING, &timestampOn, sizeof(timestampOn));

on = 1;
r = setsockopt(inf->fd, SOL_SOCKET, SO_BROADCAST, &on, sizeof(on));

r = inet_aton(address, &(inf->remote.sin_addr));
</code>```

And this is how I send a datagram:

```<code>int send_len, r, i;
struct msghdr   msg;
struct iovec    iov;

memset(&msg, 0, sizeof(msg));
memset(&iov, 0, sizeof(iov));

// Space for control message info plus timestamp
char ctrl[2048];
memset(ctrl, 0, sizeof(ctrl));
//struct cmsghdr *cmsg = (struct cmsghdr *) &ctrl;

// Ancillary data buffer and length
//msg.msg_control      = (char *) ctrl;
//msg.msg_controllen   = sizeof(ctrl);

// Dest address info
msg.msg_name         = (struct sockaddr *) &(inf->remote);
msg.msg_namelen      = sizeof(struct sockaddr_in);

// Array of data buffers (scatter/gather)
msg.msg_iov          = &iov;
msg.msg_iovlen       = 1;

// Data buffer pointer and length
iov.iov_base         = buf;
iov.iov_len          = len;

send_len = sendmsg(inf->fd, &msg, 0);
</code>```

Examples I've seen reuse the msg and iov data structures, but in my experimentation, I added code to make sure things were cleared, just in case the send left anything behind, although it didn't make any difference.  Here's the code for getting the timestamp:

```<code>memset(&msg, 0, sizeof(msg));
memset(&iov, 0, sizeof(iov));
memset(ctrl, 0, sizeof(ctrl));
msg.msg_control      = (char *) ctrl;
msg.msg_controllen   = sizeof(ctrl);
msg.msg_name         = (struct sockaddr *) &(inf->remote);
msg.msg_namelen      = sizeof(struct sockaddr_in);
msg.msg_iov          = &iov;
msg.msg_iovlen       = 1;
iov.iov_base         = junk_buf;
iov.iov_len          = sizeof(junk_buf);

for (;;) {
    r = recvmsg(inf->fd, &msg, MSG_ERRQUEUE);
    if (r<0) {
        fprintf(stderr, "Didn't get kernel time\n");
        return send_len;
    }

    printf("recvmsg returned %d\n", r);
    handle_time(&msg);
}
</code>```

The data buffer contains the original datagram as expected.  The ancillary data I get back includes a single message, which handle_time prints as:

```<code>level=0, type=11, len=48
</code>```

This is level SOL_IP and type IP_RECVERR, which is expected according to the docs.  Looking into the payload (a struct sock_extended_err), the errno is 42 (ENOMSG, No message of desired type) and origin is 4 (SO_EE_ORIGIN_TXSTATUS).  From the docs, this is supposed to happen and demonstrates that in fact I did manage to inform the kernel that I want TX status messages.  But there is no second ancillary message!

I have tried to see if there is any kernel compile option that might disable this, but I haven't found any.  So I'm just completely baffled here.  Can anyone help me figure out what I'm doing wrong?

Thanks!

UPDATE:  I tried running this same code on another Linux machine, this time CentOS 7 (kernel 3.10.0-693.2.2.el7.x86_64).  I can't figure out what what kind of NIC that machine has, but when I try to send datagrams, I get some other weird behavior.  For the very first datagram, when I start this program, I get back the message and a single ancillary message, just as above.  For every subsequent sendmsg call, errno tells me that I get an "Invalid argument" error.  This error goes away if I don't enable timestamps on the socket.

UPDATE 2:  I discovered that I had not been making an ioctl necessary to enable timestamps in the driver.  Unfortunately, when I do this call, I get ENODEV from errno (no such device).  Here's how I'm trying to do it (which I'm imitating from https://github.com/majek/openonload/blob/master/src/tests/onload/hwtimestamping/tx_timestamping.c):

```<code>struct ifreq ifr;
struct hwtstamp_config hwc;

inf->fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

memset(&ifr, 0, sizeof(ifr));
hwc.flags = 0;
hwc.tx_type = HWTSTAMP_TX_ON;
hwc.rx_filter = HWTSTAMP_FILTER_ALL;
ifr.ifr_data = (char*)&hwc;
r = ioctl(inf->fd, SIOCSHWTSTAMP, &ifr);
</code>```

That being said, I'd be relatively happy with software timestamps, which should not need this call.  So I'm not sure this is helpful anyhow.

UPDATE 3:  A compilable example was requested.  The whole program is pretty minimal, so I put it into pastebin here:  https://pastebin.com/qd0gspRc

Also, here's the output from ethtool:

```<code>Time stamping parameters for eth0:
Capabilities:
        software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
        software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
        software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none
</code>```

Since this obviously doesn't support hardware timestamps, the ioctl is moot.  I tried changing the SO_TIMESTAMPING setting to SOF_TIMESTAMPING_TX_SOFTWARE and SOF_TIMESTAMPING_RX_SOFTWARE for sender and receiver.  That didn't help.

Then I tried adding SOF_TIMESTAMPING_SOFTWARE to both.  I finally started getting something:

```<code>level=1, type=37, len=64
</code>```

Level 1 is SOL_SOCKET, and type 37 is SCM_TIMESTAMPING.  I'll go back to the docs and figure out how to interpret this.  It says something about passing an array of three time structures.  The driver's call to skb_tx_timestamp should have been sufficient so that it wouldn't require that I enable "fake" software timestamps to get something out.

答:
Like I say in comment the use of SOF_TIMESTAMPING_SOFTWARE and SOF_TIMESTAMPING_RAW_HARDWARE is necessary because if I understand correctly the documentation, some bits are to generate the timestamp and some bits are here to report them in control message:

1.3.1 Timestamp Generation

Some bits are requests to the stack to try to generate timestamps. Any
  combination of them is valid. Changes to these bits apply to newly
  created packets, not to packets already in the stack. As a result, it
  is possible to selectively request timestamps for a subset of packets
  (e.g., for sampling) by embedding an send() call within two setsockopt
  calls, one to enable timestamp generation and one to disable it.
  Timestamps may also be generated for reasons other than being
  requested by a particular socket, such as when receive timestamping is
  enabled system wide, as explained earlier.

1.3.2 Timestamp Reporting

The other three bits control which timestamps will be reported in a
  generated control message. Changes to the bits take immediate effect
  at the timestamp reporting locations in the stack. Timestamps are only
  reported for packets that also have the relevant timestamp generation
  request set.

After, to use the data documentation say:

2.1 SCM_TIMESTAMPING records

These timestamps are returned in a control message with cmsg_level
  SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type

struct scm_timestamping {    struct timespec ts[3]; };

...

The structure can return up to three timestamps. This is a legacy
  feature. At least one field is non-zero at any time. Most timestamps
  are passed in ts[0]. Hardware timestamps are passed in ts[2].

To get transmit timestamp this require some configuration, first you need to know that software timestamp are not always available, I only achieve to get hardware transmit timestamp. But I'm not an expert in these domain, I just try to implemented timestamp with information that I found.

Secondly, I needed to activate hardware feature with linuxptp tool, I use hwstamp_cli:

```<code>hwstamp_ctl -i eth0 -r 1 -t 1
</code>```

With this and some modification on your code I achieve to get hardware transmit timestamp but only with ethX interface because lo interface don't have these feature AFAIK so the final code is:

```<code>#include <arpa/inet.h>
#include <errno.h>
#include <inttypes.h>
#include <linux/errqueue.h>
#include <linux/net_tstamp.h>
#include <linux/sockios.h>
#include <net/if.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>

#define UDP_MAX_LENGTH 1500

typedef struct {
  int fd;
  int port;
  int err_no;
  struct sockaddr_in local;
  struct sockaddr_in remote;
  struct timeval time_kernel;
  struct timeval time_user;
  int64_t prev_serialnum;
} socket_info;

static int setup_udp_receiver(socket_info *inf, int port) {
  inf->port = port;
  inf->fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
  if (inf->fd < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_server: socket failed: %s\n",
            strerror(inf->err_no));
    return inf->fd;
  }

  int timestampOn =
      SOF_TIMESTAMPING_RX_SOFTWARE | SOF_TIMESTAMPING_TX_SOFTWARE |
      SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_RX_HARDWARE |
      SOF_TIMESTAMPING_TX_HARDWARE | SOF_TIMESTAMPING_RAW_HARDWARE |
      // SOF_TIMESTAMPING_OPT_TSONLY |
      0;
  int r = setsockopt(inf->fd, SOL_SOCKET, SO_TIMESTAMPING, &timestampOn,
                     sizeof timestampOn);
  if (r < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_server: setsockopt failed: %s\n",
            strerror(inf->err_no));
    return r;
  }

  int on = 1;
  r = setsockopt(inf->fd, SOL_SOCKET, SO_REUSEPORT, &on, sizeof on);
  if (r < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_server: setsockopt2 failed: %s\n",
            strerror(inf->err_no));
    return r;
  }

  inf->local = (struct sockaddr_in){.sin_family = AF_INET,
                                    .sin_port = htons((uint16_t)port),
                                    .sin_addr.s_addr = htonl(INADDR_ANY)};
  r = bind(inf->fd, (struct sockaddr *)&inf->local, sizeof inf->local);
  if (r < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_server: bind failed: %s\n",
            strerror(inf->err_no));
    return r;
  }

  inf->prev_serialnum = -1;

  return 0;
}

static int setup_udp_sender(socket_info *inf, int port, char *address) {
  inf->port = port;
  inf->fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
  if (inf->fd < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_client: socket failed: %s\n",
            strerror(inf->err_no));
    return inf->fd;
  }

  int timestampOn =
      SOF_TIMESTAMPING_RX_SOFTWARE | SOF_TIMESTAMPING_TX_SOFTWARE |
      SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_RX_HARDWARE |
      SOF_TIMESTAMPING_TX_HARDWARE | SOF_TIMESTAMPING_RAW_HARDWARE |
      // SOF_TIMESTAMPING_OPT_TSONLY |
      0;
  int r = setsockopt(inf->fd, SOL_SOCKET, SO_TIMESTAMPING, &timestampOn,
                     sizeof timestampOn);
  if (r < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_server: setsockopt failed: %s\n",
            strerror(inf->err_no));
    return r;
  }

  inf->remote = (struct sockaddr_in){.sin_family = AF_INET,
                                     .sin_port = htons((uint16_t)port)};
  r = inet_aton(address, &inf->remote.sin_addr);
  if (r == 0) {
    fprintf(stderr, "setup_udp_client: inet_aton failed\n");
    inf->err_no = 0;
    return -1;
  }

  inf->local = (struct sockaddr_in){.sin_family = AF_INET,
                                    .sin_port = htons(0),
                                    .sin_addr.s_addr = htonl(INADDR_ANY)};
  inf->prev_serialnum = -1;

  return 0;
}

static void handle_scm_timestamping(struct scm_timestamping *ts) {
  for (size_t i = 0; i < sizeof ts->ts / sizeof *ts->ts; i++) {
    printf("timestamp: %lld.%.9lds\n", (long long)ts->ts[i].tv_sec,
           ts->ts[i].tv_nsec);
  }
}

static void handle_time(struct msghdr *msg) {

  for (struct cmsghdr *cmsg = CMSG_FIRSTHDR(msg); cmsg;
       cmsg = CMSG_NXTHDR(msg, cmsg)) {
    printf("level=%d, type=%d, len=%zu\n", cmsg->cmsg_level, cmsg->cmsg_type,
           cmsg->cmsg_len);

    if (cmsg->cmsg_level == SOL_IP && cmsg->cmsg_type == IP_RECVERR) {
      struct sock_extended_err *ext =
          (struct sock_extended_err *)CMSG_DATA(cmsg);
      printf("errno=%d, origin=%d\n", ext->ee_errno, ext->ee_origin);
      continue;
    }

    if (cmsg->cmsg_level != SOL_SOCKET)
      continue;

    switch (cmsg->cmsg_type) {
    case SO_TIMESTAMPNS: {
      struct scm_timestamping *ts = (struct scm_timestamping *)CMSG_DATA(cmsg);
      handle_scm_timestamping(ts);
    } break;
    case SO_TIMESTAMPING: {
      struct scm_timestamping *ts = (struct scm_timestamping *)CMSG_DATA(cmsg);
      handle_scm_timestamping(ts);
    } break;
    default:
      /* Ignore other cmsg options */
      break;
    }
  }
  printf("End messages\n");
}

static ssize_t udp_receive(socket_info *inf, char *buf, size_t len) {
  char ctrl[2048];
  struct iovec iov = (struct iovec){.iov_base = buf, .iov_len = len};
  struct msghdr msg = (struct msghdr){.msg_control = ctrl,
                                      .msg_controllen = sizeof ctrl,
                                      .msg_name = &inf->remote,
                                      .msg_namelen = sizeof inf->remote,
                                      .msg_iov = &iov,
                                      .msg_iovlen = 1};
  ssize_t recv_len = recvmsg(inf->fd, &msg, 0);
  gettimeofday(&inf->time_user, NULL);

  if (recv_len < 0) {
    inf->err_no = errno;
    fprintf(stderr, "udp_receive: recvfrom failed: %s\n",
            strerror(inf->err_no));
  }

  handle_time(&msg);

  return recv_len;
}

static ssize_t udp_send(socket_info *inf, char *buf, size_t len) {
  struct iovec iov = (struct iovec){.iov_base = buf, .iov_len = len};
  struct msghdr msg = (struct msghdr){.msg_name = &inf->remote,
                                      .msg_namelen = sizeof inf->remote,
                                      .msg_iov = &iov,
                                      .msg_iovlen = 1};
  gettimeofday(&inf->time_user, NULL);
  ssize_t send_len = sendmsg(inf->fd, &msg, 0);
  if (send_len < 0) {
    inf->err_no = errno;
    fprintf(stderr, "udp_send: sendmsg failed: %s\n", strerror(inf->err_no));
  }

  return send_len;
}

static ssize_t meq_receive(socket_info *inf, char *buf, size_t len) {
  struct iovec iov = (struct iovec){.iov_base = buf, .iov_len = len};
  char ctrl[2048];
  struct msghdr msg = (struct msghdr){.msg_control = ctrl,
                                      .msg_controllen = sizeof ctrl,
                                      .msg_name = &inf->remote,
                                      .msg_namelen = sizeof inf->remote,
                                      .msg_iov = &iov,
                                      .msg_iovlen = 1};
  ssize_t recv_len = recvmsg(inf->fd, &msg, MSG_ERRQUEUE);
  if (recv_len < 0) {
    inf->err_no = errno;
    if (errno != EAGAIN) {
      fprintf(stderr, "meq_receive: recvmsg failed: %s\n",
              strerror(inf->err_no));
    }
    return recv_len;
  }
  handle_time(&msg);

  return recv_len;
}

typedef struct {
  int64_t serialnum;

  int64_t user_time_serialnum;
  int64_t user_time;

  int64_t kernel_time_serialnum;
  int64_t kernel_time;

  size_t message_bytes;
} message_header;

static const size_t payload_max = UDP_MAX_LENGTH - sizeof(message_header);

static ssize_t generate_random_message(socket_info *inf, char *buf,
                                       size_t len) {
  if (len < sizeof(message_header)) {
    return -1;
  }
  message_header *header = (message_header *)buf;
  char *payload = (char *)(header + 1);
  size_t payload_len = (size_t)random() % (payload_max + 1);
  if (payload_len > len - sizeof(message_header)) {
    payload_len = len - sizeof(message_header);
  }
  for (size_t i = 0; i < payload_len; i++) {
    payload[i] = (char)random();
  }

  static int64_t serial_num = 0;
  *header = (message_header){
      .user_time_serialnum = inf->prev_serialnum,
      .user_time = inf->time_user.tv_sec * 1000000000L + inf->time_user.tv_usec,
      .kernel_time_serialnum = inf->prev_serialnum,
      .kernel_time =
          inf->time_kernel.tv_sec * 1000000000L + inf->time_kernel.tv_usec,
      .serialnum = serial_num,
      .message_bytes = payload_len};
  size_t total = payload_len + sizeof *header;

  printf("uts%5" PRId64 ": kt=%" PRId64 ", ut=%" PRId64 ", sn=%" PRId64
         ": s=%zu\n",
         header->user_time_serialnum, header->kernel_time, header->user_time,
         header->serialnum, total);

  inf->prev_serialnum = serial_num++;

  return (ssize_t)total;
}

static void sender_loop(char *host) {
  socket_info inf;
  int ret = setup_udp_sender(&inf, 8000, host);
  if (ret < 0) {
    return;
  }

  for (int i = 0; i < 2000; i++) {
    useconds_t t = random() % 2000000;
    usleep(t);
    char packet_buffer[4096];
    ssize_t len =
        generate_random_message(&inf, packet_buffer, sizeof packet_buffer);
    if (len < 0) {
      return;
    }
    udp_send(&inf, packet_buffer, (size_t)len);
    while (meq_receive(&inf, packet_buffer, sizeof packet_buffer) != -1) {
    }
  }
}

static void receiver_loop(void) {
  socket_info inf;
  int ret = setup_udp_receiver(&inf, 8000);
  if (ret < 0) {
    return;
  }

  for (int i = 0; i < 1000; i++) {
    char packet_buffer[4096];
    udp_receive(&inf, packet_buffer, sizeof packet_buffer);
  }
}

#define USAGE "Usage: %s [-r | -s host]\n"

int main(int argc, char *argv[]) {
  if (argc < 2) {
    fprintf(stderr, USAGE, argv[0]);
    return 0;
  }

  if (0 == strcmp(argv[1], "-s")) {
    if (argc < 3) {
      fprintf(stderr, USAGE, argv[0]);
      return 0;
    }
    sender_loop(argv[2]);
  } else if (0 == strcmp(argv[1], "-r")) {
    receiver_loop();
  } else {
    fprintf(stderr, USAGE, argv[0]);
  }
}
</code>```

Exemple output:

```<code>$ ./a.out -r
level=1, type=37, len=64
timestamp: 1511196758.087209387s
timestamp: 0.000000000s
timestamp: 0.000000000s
End messages
level=1, type=37, len=64
timestamp: 1511196759.333507671s
timestamp: 0.000000000s
timestamp: 0.000000000s
End messages
$ ./a.out -s "8.8.8.8"
uts   -1: kt=238059712, ut=140918979990070, sn=0: s=482
uts    0: kt=238059712, ut=1511197522000237457, sn=1: s=132
level=1, type=37, len=64
timestamp: 0.000000000s
timestamp: 0.000000000s
timestamp: 1511197359.637050597s
level=0, type=11, len=48
errno=42, origin=4
End messages
uts    1: kt=238059712, ut=1511197523000483805, sn=2: s=1454
level=1, type=37, len=64
timestamp: 0.000000000s
timestamp: 0.000000000s
timestamp: 1511197360.883295397s
level=0, type=11, len=48
errno=42, origin=4
End messages
</code>```

Live test: sender, receiver

Body:
I have been trying and failing to get Linux (kernel 4.1.4) to give me timestamps for when UDP datagrams are sent and received.  I have read the original kernel docs (https://www.kernel.org/doc/Documentation/networking/timestamping.txt), along with lots of examples and a number of stackoverflow entries.  I can send datagrams between sender and receiver with no problems.  But I cannot get timestamps for sending or receiving datagrams, and I can't figure out what I'm doing wrong.

One bizarre thing is that when I use the MSG_ERRQUEUE channel for getting timestamp info on a sent datagram, I do get the original outgoing packet, and I do get the first ancillary message (SOL_IP, IP_RECVERR), but I do not get a second message (which should be level SOL_SOCKET, type SCM_TIMESTAMPING).

In another stackoverflow entry on getting timestamps for sent packets (Timestamp outgoing packets), someone mentioned that some drivers might not implement the call to skb_tx_timestamp, but I checked mine (Realtek), and that call is definitely in there.

Here's how I set up the UDP receiver (error handling code not shown):

inf->fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

timestampOn = SOF_TIMESTAMPING_RX_SOFTWARE | SOF_TIMESTAMPING_RX_HARDWARE;
r = setsockopt(inf->fd, SOL_SOCKET, SO_TIMESTAMPING, &timestampOn, sizeof(timestampOn));

r = setsockopt(inf->fd, SOL_SOCKET, SO_REUSEPORT, &on, sizeof(on));

memset(&(inf->local), 0, sizeof(struct sockaddr_in));
inf->local.sin_family = AF_INET;
inf->local.sin_port = htons(port);
inf->local.sin_addr.s_addr = htonl(INADDR_ANY);

r = bind(inf->fd, (struct sockaddr *)&(inf->local), sizeof(struct sockaddr_in));

Using SO_REUSEPORT or not doesn't seem to matter.

For receiving, my understanding is that we don't use MSG_ERRQUEUE.  That's only if we want timestamps for sent messages.  Besides, when I use MSG_ERRQUEUE with recvmsg, I get "resource temporarily unavailable."  Here's how I receive datagrams:

int recv_len;
struct msghdr   msg;
struct iovec    iov;

memset(&msg, 0, sizeof(msg));
memset(&iov, 0, sizeof(iov));

// Space for control message info plus timestamp
char ctrl[2048];
memset(ctrl, 0, sizeof(ctrl));
//struct cmsghdr *cmsg = (struct cmsghdr *) &ctrl;

// Ancillary data buffer and length
msg.msg_control      = (char *) ctrl;
msg.msg_controllen   = sizeof(ctrl);

// Dest address info
msg.msg_name         = (struct sockaddr *) &(inf->remote);
msg.msg_namelen      = sizeof(struct sockaddr_in);

// Array of data buffers (scatter/gather)
msg.msg_iov          = &iov;
msg.msg_iovlen       = 1;

// Data buffer pointer and length
iov.iov_base         = buf;
iov.iov_len          = len;

recv_len = recvmsg(inf->fd, &msg, 0);

And then I pass a pointer to msg to another function (handle_time) that does this:

struct timespec* ts = NULL;
struct cmsghdr* cmsg;
struct sock_extended_err *ext;

for( cmsg = CMSG_FIRSTHDR(msg); cmsg; cmsg = CMSG_NXTHDR(msg,cmsg) ) {
    printf("level=%d, type=%d, len=%zu\n", cmsg->cmsg_level, cmsg->cmsg_type, cmsg->cmsg_len);
}

Zero messages are received.  So that's the first problem.  My setup code above matches like half a dozen other examples I've found on the web, but I'm getting no ancillary data from this.

Next, let's turn to sending datagrams.  Here's the setup:

inf->port = port;
inf->fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

memset(&(inf->remote), 0, sizeof(struct sockaddr_in));
inf->remote.sin_family = AF_INET;
inf->remote.sin_port = htons(port);

timestampOn = SOF_TIMESTAMPING_TX_SOFTWARE | SOF_TIMESTAMPING_TX_HARDWARE;
r = setsockopt(inf->fd, SOL_SOCKET, SO_TIMESTAMPING, &timestampOn, sizeof(timestampOn));

on = 1;
r = setsockopt(inf->fd, SOL_SOCKET, SO_BROADCAST, &on, sizeof(on));

r = inet_aton(address, &(inf->remote.sin_addr));

And this is how I send a datagram:

int send_len, r, i;
struct msghdr   msg;
struct iovec    iov;

memset(&msg, 0, sizeof(msg));
memset(&iov, 0, sizeof(iov));

// Space for control message info plus timestamp
char ctrl[2048];
memset(ctrl, 0, sizeof(ctrl));
//struct cmsghdr *cmsg = (struct cmsghdr *) &ctrl;

// Ancillary data buffer and length
//msg.msg_control      = (char *) ctrl;
//msg.msg_controllen   = sizeof(ctrl);

// Dest address info
msg.msg_name         = (struct sockaddr *) &(inf->remote);
msg.msg_namelen      = sizeof(struct sockaddr_in);

// Array of data buffers (scatter/gather)
msg.msg_iov          = &iov;
msg.msg_iovlen       = 1;

// Data buffer pointer and length
iov.iov_base         = buf;
iov.iov_len          = len;

send_len = sendmsg(inf->fd, &msg, 0);

Examples I've seen reuse the msg and iov data structures, but in my experimentation, I added code to make sure things were cleared, just in case the send left anything behind, although it didn't make any difference.  Here's the code for getting the timestamp:

memset(&msg, 0, sizeof(msg));
memset(&iov, 0, sizeof(iov));
memset(ctrl, 0, sizeof(ctrl));
msg.msg_control      = (char *) ctrl;
msg.msg_controllen   = sizeof(ctrl);
msg.msg_name         = (struct sockaddr *) &(inf->remote);
msg.msg_namelen      = sizeof(struct sockaddr_in);
msg.msg_iov          = &iov;
msg.msg_iovlen       = 1;
iov.iov_base         = junk_buf;
iov.iov_len          = sizeof(junk_buf);

for (;;) {
    r = recvmsg(inf->fd, &msg, MSG_ERRQUEUE);
    if (r<0) {
        fprintf(stderr, "Didn't get kernel time\n");
        return send_len;
    }

    printf("recvmsg returned %d\n", r);
    handle_time(&msg);
}

The data buffer contains the original datagram as expected.  The ancillary data I get back includes a single message, which handle_time prints as:

level=0, type=11, len=48

This is level SOL_IP and type IP_RECVERR, which is expected according to the docs.  Looking into the payload (a struct sock_extended_err), the errno is 42 (ENOMSG, No message of desired type) and origin is 4 (SO_EE_ORIGIN_TXSTATUS).  From the docs, this is supposed to happen and demonstrates that in fact I did manage to inform the kernel that I want TX status messages.  But there is no second ancillary message!

I have tried to see if there is any kernel compile option that might disable this, but I haven't found any.  So I'm just completely baffled here.  Can anyone help me figure out what I'm doing wrong?

Thanks!

UPDATE:  I tried running this same code on another Linux machine, this time CentOS 7 (kernel 3.10.0-693.2.2.el7.x86_64).  I can't figure out what what kind of NIC that machine has, but when I try to send datagrams, I get some other weird behavior.  For the very first datagram, when I start this program, I get back the message and a single ancillary message, just as above.  For every subsequent sendmsg call, errno tells me that I get an "Invalid argument" error.  This error goes away if I don't enable timestamps on the socket.

UPDATE 2:  I discovered that I had not been making an ioctl necessary to enable timestamps in the driver.  Unfortunately, when I do this call, I get ENODEV from errno (no such device).  Here's how I'm trying to do it (which I'm imitating from https://github.com/majek/openonload/blob/master/src/tests/onload/hwtimestamping/tx_timestamping.c):

struct ifreq ifr;
struct hwtstamp_config hwc;

inf->fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);

memset(&ifr, 0, sizeof(ifr));
hwc.flags = 0;
hwc.tx_type = HWTSTAMP_TX_ON;
hwc.rx_filter = HWTSTAMP_FILTER_ALL;
ifr.ifr_data = (char*)&hwc;
r = ioctl(inf->fd, SIOCSHWTSTAMP, &ifr);

That being said, I'd be relatively happy with software timestamps, which should not need this call.  So I'm not sure this is helpful anyhow.

UPDATE 3:  A compilable example was requested.  The whole program is pretty minimal, so I put it into pastebin here:  https://pastebin.com/qd0gspRc

Also, here's the output from ethtool:

Time stamping parameters for eth0:
Capabilities:
        software-transmit     (SOF_TIMESTAMPING_TX_SOFTWARE)
        software-receive      (SOF_TIMESTAMPING_RX_SOFTWARE)
        software-system-clock (SOF_TIMESTAMPING_SOFTWARE)
PTP Hardware Clock: none
Hardware Transmit Timestamp Modes: none
Hardware Receive Filter Modes: none

Since this obviously doesn't support hardware timestamps, the ioctl is moot.  I tried changing the SO_TIMESTAMPING setting to SOF_TIMESTAMPING_TX_SOFTWARE and SOF_TIMESTAMPING_RX_SOFTWARE for sender and receiver.  That didn't help.

Then I tried adding SOF_TIMESTAMPING_SOFTWARE to both.  I finally started getting something:

level=1, type=37, len=64

Level 1 is SOL_SOCKET, and type 37 is SCM_TIMESTAMPING.  I'll go back to the docs and figure out how to interpret this.  It says something about passing an array of three time structures.  The driver's call to skb_tx_timestamp should have been sufficient so that it wouldn't require that I enable "fake" software timestamps to get something out.

Body:
Like I say in comment the use of SOF_TIMESTAMPING_SOFTWARE and SOF_TIMESTAMPING_RAW_HARDWARE is necessary because if I understand correctly the documentation, some bits are to generate the timestamp and some bits are here to report them in control message:

1.3.1 Timestamp Generation

Some bits are requests to the stack to try to generate timestamps. Any
  combination of them is valid. Changes to these bits apply to newly
  created packets, not to packets already in the stack. As a result, it
  is possible to selectively request timestamps for a subset of packets
  (e.g., for sampling) by embedding an send() call within two setsockopt
  calls, one to enable timestamp generation and one to disable it.
  Timestamps may also be generated for reasons other than being
  requested by a particular socket, such as when receive timestamping is
  enabled system wide, as explained earlier.

1.3.2 Timestamp Reporting

The other three bits control which timestamps will be reported in a
  generated control message. Changes to the bits take immediate effect
  at the timestamp reporting locations in the stack. Timestamps are only
  reported for packets that also have the relevant timestamp generation
  request set.

After, to use the data documentation say:

2.1 SCM_TIMESTAMPING records

These timestamps are returned in a control message with cmsg_level
  SOL_SOCKET, cmsg_type SCM_TIMESTAMPING, and payload of type

struct scm_timestamping {    struct timespec ts[3]; };

...

The structure can return up to three timestamps. This is a legacy
  feature. At least one field is non-zero at any time. Most timestamps
  are passed in ts[0]. Hardware timestamps are passed in ts[2].

To get transmit timestamp this require some configuration, first you need to know that software timestamp are not always available, I only achieve to get hardware transmit timestamp. But I'm not an expert in these domain, I just try to implemented timestamp with information that I found.

Secondly, I needed to activate hardware feature with linuxptp tool, I use hwstamp_cli:

hwstamp_ctl -i eth0 -r 1 -t 1

With this and some modification on your code I achieve to get hardware transmit timestamp but only with ethX interface because lo interface don't have these feature AFAIK so the final code is:

#include <arpa/inet.h>
#include <errno.h>
#include <inttypes.h>
#include <linux/errqueue.h>
#include <linux/net_tstamp.h>
#include <linux/sockios.h>
#include <net/if.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <sys/time.h>
#include <sys/types.h>
#include <unistd.h>

#define UDP_MAX_LENGTH 1500

typedef struct {
  int fd;
  int port;
  int err_no;
  struct sockaddr_in local;
  struct sockaddr_in remote;
  struct timeval time_kernel;
  struct timeval time_user;
  int64_t prev_serialnum;
} socket_info;

static int setup_udp_receiver(socket_info *inf, int port) {
  inf->port = port;
  inf->fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
  if (inf->fd < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_server: socket failed: %s\n",
            strerror(inf->err_no));
    return inf->fd;
  }

  int timestampOn =
      SOF_TIMESTAMPING_RX_SOFTWARE | SOF_TIMESTAMPING_TX_SOFTWARE |
      SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_RX_HARDWARE |
      SOF_TIMESTAMPING_TX_HARDWARE | SOF_TIMESTAMPING_RAW_HARDWARE |
      // SOF_TIMESTAMPING_OPT_TSONLY |
      0;
  int r = setsockopt(inf->fd, SOL_SOCKET, SO_TIMESTAMPING, &timestampOn,
                     sizeof timestampOn);
  if (r < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_server: setsockopt failed: %s\n",
            strerror(inf->err_no));
    return r;
  }

  int on = 1;
  r = setsockopt(inf->fd, SOL_SOCKET, SO_REUSEPORT, &on, sizeof on);
  if (r < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_server: setsockopt2 failed: %s\n",
            strerror(inf->err_no));
    return r;
  }

  inf->local = (struct sockaddr_in){.sin_family = AF_INET,
                                    .sin_port = htons((uint16_t)port),
                                    .sin_addr.s_addr = htonl(INADDR_ANY)};
  r = bind(inf->fd, (struct sockaddr *)&inf->local, sizeof inf->local);
  if (r < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_server: bind failed: %s\n",
            strerror(inf->err_no));
    return r;
  }

  inf->prev_serialnum = -1;

  return 0;
}

static int setup_udp_sender(socket_info *inf, int port, char *address) {
  inf->port = port;
  inf->fd = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
  if (inf->fd < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_client: socket failed: %s\n",
            strerror(inf->err_no));
    return inf->fd;
  }

  int timestampOn =
      SOF_TIMESTAMPING_RX_SOFTWARE | SOF_TIMESTAMPING_TX_SOFTWARE |
      SOF_TIMESTAMPING_SOFTWARE | SOF_TIMESTAMPING_RX_HARDWARE |
      SOF_TIMESTAMPING_TX_HARDWARE | SOF_TIMESTAMPING_RAW_HARDWARE |
      // SOF_TIMESTAMPING_OPT_TSONLY |
      0;
  int r = setsockopt(inf->fd, SOL_SOCKET, SO_TIMESTAMPING, &timestampOn,
                     sizeof timestampOn);
  if (r < 0) {
    inf->err_no = errno;
    fprintf(stderr, "setup_udp_server: setsockopt failed: %s\n",
            strerror(inf->err_no));
    return r;
  }

  inf->remote = (struct sockaddr_in){.sin_family = AF_INET,
                                     .sin_port = htons((uint16_t)port)};
  r = inet_aton(address, &inf->remote.sin_addr);
  if (r == 0) {
    fprintf(stderr, "setup_udp_client: inet_aton failed\n");
    inf->err_no = 0;
    return -1;
  }

  inf->local = (struct sockaddr_in){.sin_family = AF_INET,
                                    .sin_port = htons(0),
                                    .sin_addr.s_addr = htonl(INADDR_ANY)};
  inf->prev_serialnum = -1;

  return 0;
}

static void handle_scm_timestamping(struct scm_timestamping *ts) {
  for (size_t i = 0; i < sizeof ts->ts / sizeof *ts->ts; i++) {
    printf("timestamp: %lld.%.9lds\n", (long long)ts->ts[i].tv_sec,
           ts->ts[i].tv_nsec);
  }
}

static void handle_time(struct msghdr *msg) {

  for (struct cmsghdr *cmsg = CMSG_FIRSTHDR(msg); cmsg;
       cmsg = CMSG_NXTHDR(msg, cmsg)) {
    printf("level=%d, type=%d, len=%zu\n", cmsg->cmsg_level, cmsg->cmsg_type,
           cmsg->cmsg_len);

    if (cmsg->cmsg_level == SOL_IP && cmsg->cmsg_type == IP_RECVERR) {
      struct sock_extended_err *ext =
          (struct sock_extended_err *)CMSG_DATA(cmsg);
      printf("errno=%d, origin=%d\n", ext->ee_errno, ext->ee_origin);
      continue;
    }

    if (cmsg->cmsg_level != SOL_SOCKET)
      continue;

    switch (cmsg->cmsg_type) {
    case SO_TIMESTAMPNS: {
      struct scm_timestamping *ts = (struct scm_timestamping *)CMSG_DATA(cmsg);
      handle_scm_timestamping(ts);
    } break;
    case SO_TIMESTAMPING: {
      struct scm_timestamping *ts = (struct scm_timestamping *)CMSG_DATA(cmsg);
      handle_scm_timestamping(ts);
    } break;
    default:
      /* Ignore other cmsg options */
      break;
    }
  }
  printf("End messages\n");
}

static ssize_t udp_receive(socket_info *inf, char *buf, size_t len) {
  char ctrl[2048];
  struct iovec iov = (struct iovec){.iov_base = buf, .iov_len = len};
  struct msghdr msg = (struct msghdr){.msg_control = ctrl,
                                      .msg_controllen = sizeof ctrl,
                                      .msg_name = &inf->remote,
                                      .msg_namelen = sizeof inf->remote,
                                      .msg_iov = &iov,
                                      .msg_iovlen = 1};
  ssize_t recv_len = recvmsg(inf->fd, &msg, 0);
  gettimeofday(&inf->time_user, NULL);

  if (recv_len < 0) {
    inf->err_no = errno;
    fprintf(stderr, "udp_receive: recvfrom failed: %s\n",
            strerror(inf->err_no));
  }

  handle_time(&msg);

  return recv_len;
}

static ssize_t udp_send(socket_info *inf, char *buf, size_t len) {
  struct iovec iov = (struct iovec){.iov_base = buf, .iov_len = len};
  struct msghdr msg = (struct msghdr){.msg_name = &inf->remote,
                                      .msg_namelen = sizeof inf->remote,
                                      .msg_iov = &iov,
                                      .msg_iovlen = 1};
  gettimeofday(&inf->time_user, NULL);
  ssize_t send_len = sendmsg(inf->fd, &msg, 0);
  if (send_len < 0) {
    inf->err_no = errno;
    fprintf(stderr, "udp_send: sendmsg failed: %s\n", strerror(inf->err_no));
  }

  return send_len;
}

static ssize_t meq_receive(socket_info *inf, char *buf, size_t len) {
  struct iovec iov = (struct iovec){.iov_base = buf, .iov_len = len};
  char ctrl[2048];
  struct msghdr msg = (struct msghdr){.msg_control = ctrl,
                                      .msg_controllen = sizeof ctrl,
                                      .msg_name = &inf->remote,
                                      .msg_namelen = sizeof inf->remote,
                                      .msg_iov = &iov,
                                      .msg_iovlen = 1};
  ssize_t recv_len = recvmsg(inf->fd, &msg, MSG_ERRQUEUE);
  if (recv_len < 0) {
    inf->err_no = errno;
    if (errno != EAGAIN) {
      fprintf(stderr, "meq_receive: recvmsg failed: %s\n",
              strerror(inf->err_no));
    }
    return recv_len;
  }
  handle_time(&msg);

  return recv_len;
}

typedef struct {
  int64_t serialnum;

  int64_t user_time_serialnum;
  int64_t user_time;

  int64_t kernel_time_serialnum;
  int64_t kernel_time;

  size_t message_bytes;
} message_header;

static const size_t payload_max = UDP_MAX_LENGTH - sizeof(message_header);

static ssize_t generate_random_message(socket_info *inf, char *buf,
                                       size_t len) {
  if (len < sizeof(message_header)) {
    return -1;
  }
  message_header *header = (message_header *)buf;
  char *payload = (char *)(header + 1);
  size_t payload_len = (size_t)random() % (payload_max + 1);
  if (payload_len > len - sizeof(message_header)) {
    payload_len = len - sizeof(message_header);
  }
  for (size_t i = 0; i < payload_len; i++) {
    payload[i] = (char)random();
  }

  static int64_t serial_num = 0;
  *header = (message_header){
      .user_time_serialnum = inf->prev_serialnum,
      .user_time = inf->time_user.tv_sec * 1000000000L + inf->time_user.tv_usec,
      .kernel_time_serialnum = inf->prev_serialnum,
      .kernel_time =
          inf->time_kernel.tv_sec * 1000000000L + inf->time_kernel.tv_usec,
      .serialnum = serial_num,
      .message_bytes = payload_len};
  size_t total = payload_len + sizeof *header;

  printf("uts%5" PRId64 ": kt=%" PRId64 ", ut=%" PRId64 ", sn=%" PRId64
         ": s=%zu\n",
         header->user_time_serialnum, header->kernel_time, header->user_time,
         header->serialnum, total);

  inf->prev_serialnum = serial_num++;

  return (ssize_t)total;
}

static void sender_loop(char *host) {
  socket_info inf;
  int ret = setup_udp_sender(&inf, 8000, host);
  if (ret < 0) {
    return;
  }

  for (int i = 0; i < 2000; i++) {
    useconds_t t = random() % 2000000;
    usleep(t);
    char packet_buffer[4096];
    ssize_t len =
        generate_random_message(&inf, packet_buffer, sizeof packet_buffer);
    if (len < 0) {
      return;
    }
    udp_send(&inf, packet_buffer, (size_t)len);
    while (meq_receive(&inf, packet_buffer, sizeof packet_buffer) != -1) {
    }
  }
}

static void receiver_loop(void) {
  socket_info inf;
  int ret = setup_udp_receiver(&inf, 8000);
  if (ret < 0) {
    return;
  }

  for (int i = 0; i < 1000; i++) {
    char packet_buffer[4096];
    udp_receive(&inf, packet_buffer, sizeof packet_buffer);
  }
}

#define USAGE "Usage: %s [-r | -s host]\n"

int main(int argc, char *argv[]) {
  if (argc < 2) {
    fprintf(stderr, USAGE, argv[0]);
    return 0;
  }

  if (0 == strcmp(argv[1], "-s")) {
    if (argc < 3) {
      fprintf(stderr, USAGE, argv[0]);
      return 0;
    }
    sender_loop(argv[2]);
  } else if (0 == strcmp(argv[1], "-r")) {
    receiver_loop();
  } else {
    fprintf(stderr, USAGE, argv[0]);
  }
}

Exemple output:

$ ./a.out -r
level=1, type=37, len=64
timestamp: 1511196758.087209387s
timestamp: 0.000000000s
timestamp: 0.000000000s
End messages
level=1, type=37, len=64
timestamp: 1511196759.333507671s
timestamp: 0.000000000s
timestamp: 0.000000000s
End messages
$ ./a.out -s "8.8.8.8"
uts   -1: kt=238059712, ut=140918979990070, sn=0: s=482
uts    0: kt=238059712, ut=1511197522000237457, sn=1: s=132
level=1, type=37, len=64
timestamp: 0.000000000s
timestamp: 0.000000000s
timestamp: 1511197359.637050597s
level=0, type=11, len=48
errno=42, origin=4
End messages
uts    1: kt=238059712, ut=1511197523000483805, sn=2: s=1454
level=1, type=37, len=64
timestamp: 0.000000000s
timestamp: 0.000000000s
timestamp: 1511197360.883295397s
level=0, type=11, len=48
errno=42, origin=4
End messages

Live test: sender, receiver


来源: stackexchange

问:
How to check that string contains is english only
i have a string which contains values like:

```<code>string str ="Abhigyan Prakash,Primeshow,NewsPoint,NCP,Inflation,सरकार,राहुल,प्राइम शो,न्यूजप्वाइंट,कमजोर,एनसीपी,अभिज्ञान प्रकाश,Rahul";
</code>```

i have used below code to convert it to array list:

```<code>    ArrayList altags = new ArrayList( str.Split(','));
</code>```

now i wants to remove all those strings from arraylist which belongs from non-english language (in my context those words which are written in "hindi" should be removed)

please suggest me how to check that string is written in english (with numeric and symbols) or in other language..

NOTE: i have no issue with generic list. i cant take it also. but please tell me how to check that string contains only alphabets + numeric + symbols
Thanks

答:
If you need to filter strings with latin symbols, numeric symbols and spaces, you may use regular expressions. </p>

```<code>        var regex = new Regex("[a-zA-Z0-9 ]*");
        var result = str.Split(',')
                        .Where(s => regex.Match(s).Value == s)
                        .ToArray();
</code>```

ContentLicense: CC BY-SA 2.5

Body:
i have a string which contains values like:

string str ="Abhigyan Prakash,Primeshow,NewsPoint,NCP,Inflation,सरकार,राहुल,प्राइम शो,न्यूजप्वाइंट,कमजोर,एनसीपी,अभिज्ञान प्रकाश,Rahul";

i have used below code to convert it to array list:

    ArrayList altags = new ArrayList( str.Split(','));

now i wants to remove all those strings from arraylist which belongs from non-english language (in my context those words which are written in "hindi" should be removed)

please suggest me how to check that string is written in english (with numeric and symbols) or in other language..

NOTE: i have no issue with generic list. i cant take it also. but please tell me how to check that string contains only alphabets + numeric + symbols
Thanks

Body:
If you need to filter strings with latin symbols, numeric symbols and spaces, you may use regular expressions.

        var regex = new Regex("[a-zA-Z0-9 ]*");
        var result = str.Split(',')
                        .Where(s => regex.Match(s).Value == s)
                        .ToArray();


问:
Websockets reverse proxy in IIS 8
I'm attempting to connect to a websockets server (websockify) through a reverse proxy on IIS. The IIS and websockets server reside on the same physical server (Windows Server 2012 R2, IIS 8.5, ARR 3, Websockets enabled). I've seen a few questions about this and it's suggested this should work with IIS 8 and ARR 3, but no actual solutions as yet. I have some experience with http/https reverse proxies in IIS, but this is my first attempt working with websockets.

For example:

The original url:
ws://10.2.1.10/websockify

The reverse proxy needs to translate this to:
ws://10.2.1.10:5901/websockify

Overly general sample rule in web.config:

```<code><rewrite>
     <rules>
        <rule name="WS reverse proxy" stopProcessing="true">
          <match url="(.*)" />
          <conditions> <add input="{CACHE_URL}" pattern="^(.+)://" />
          </conditions>
          <action type="Rewrite" url="{C:1}://10.2.1.10:5901/websockify"/>
        </rule>
     </rules>
</rewrite>
</code>```

Per the Failed Request Trace, the url appears to be translated, but for some reason it doesn't reach the websocket server at 10.2.1.10:5901.

The end goal is to incorporate noVNC/websockify to provide browser based client access to multiple VNC servers on the network. Any help understanding how to reverse proxy the websockets is appreciated.

答:
As stated by @jowo, IIS8/8.5 don't seem to support Sec-Websocket-Extensions. That being said, the workaround I applied is to simply rewrite the server variable in question. First add HTTP_SEC_WEBSOCKET_EXTENSIONS to the allowed server variable list, then add it to your rule:

<serverVariables><set name="HTTP_SEC_WEBSOCKET_EXTENSIONS" value="" /></serverVariables>

That way, the destination won't receive the troublesome permessage-deflate :)

ContentLicense: CC BY-SA 3.0

Title: Websockets reverse proxy in IIS 8

Body:
I'm attempting to connect to a websockets server (websockify) through a reverse proxy on IIS. The IIS and websockets server reside on the same physical server (Windows Server 2012 R2, IIS 8.5, ARR 3, Websockets enabled). I've seen a few questions about this and it's suggested this should work with IIS 8 and ARR 3, but no actual solutions as yet. I have some experience with http/https reverse proxies in IIS, but this is my first attempt working with websockets.

For example:

The original url:
ws://10.2.1.10/websockify

The reverse proxy needs to translate this to:
ws://10.2.1.10:5901/websockify

Overly general sample rule in web.config:

<rewrite>
     <rules>
        <rule name="WS reverse proxy" stopProcessing="true">
          <match url="(.*)" />
          <conditions> <add input="{CACHE_URL}" pattern="^(.+)://" />
          </conditions>
          <action type="Rewrite" url="{C:1}://10.2.1.10:5901/websockify"/>
        </rule>
     </rules>
</rewrite>

Per the Failed Request Trace, the url appears to be translated, but for some reason it doesn't reach the websocket server at 10.2.1.10:5901.

The end goal is to incorporate noVNC/websockify to provide browser based client access to multiple VNC servers on the network. Any help understanding how to reverse proxy the websockets is appreciated.


问:
Login to windows domain on Linux container
My company is exclusively using Windows machines and when I login to the windows domain it gives me access to some shared drives and databases.
Now I would like to run R in a container, for instance following tutorial
https://ropenscilabs.github.io/r-docker-tutorial/02-Launching-Docker.html

My question is the following: is there a way that my R scripts executed in this container inherit from the permissions of the host OS?
It seems especially tricky for access to MSSQL databases that are using Windows authentication.....

答:
RockScience,

I see two options here, they both enable you to inherit permissions, as opposed to passing in user and password credentials. To address your specific question, around Linux take a look at:

Active Directory Authentication with SQL Server on Linux

If you take the approach of running Windows containers, take a look at Active Directory gMSA (Group Managed Service Accounts) accounts and the following MSDN article and video:

Active Directory Service Accounts for Windows Containers

Walk the Path to Containerization - transforming workloads into containers

this would enable you to create a Windows container and R environment.

Active Directory Authentication with SQL Server on Linux

The tutorial explains how to configure SQL Server on Linux to support Active Directory (AD) authentication, also known as integrated authentication. AD Authentication enables domain-joined clients on either Windows or Linux to authenticate to SQL Server using their domain credentials and the Kerberos protocol.

AD Authentication has the following advantages over SQL Server Authentication:

Users authenticate via single sign-on, without being prompted for a password.

By creating logins for AD groups, you can manage access and permissions in SQL Server using AD group memberships.

Each user has a single identity across your organization, so you don’t have to keep track of which SQL Server logins correspond to which people.

AD enables you to enforce a centralised password policy across your organisation.

The tutorial consists of the following tasks:

Join SQL Server host to AD domain

Create AD user for SQL Server and set SPN

Configure SQL Server service keytab

Create AD-based logins in Transact-SQL

Connect to SQL Server using AD Authentication

Active Directory Service Accounts for Windows Containers

Today, group Managed Service Accounts are often used to secure connections between one computer or service to another. The general steps to use one are:

Create a gMSA

Configure the service to run as the gMSA

Give the domain-joined host running the service access to the gMSA secrets in Active Directory

Allow access to gMSA on the other service such as a database or file Shares

When the service is launched, the domain-joined host automatically gets the gMSA secrets from Active Directory, and runs the service using that account. Since that service is running as the gMSA, it can access any resources the gMSA is allowed to.

Windows Containers follow a similar process:

Create a gMSA. By default, a domain administrator or account operator must do this. Otherwise they can delegate the privileges to create & manage gMSAs to admins who manage services which use them. See gMSA Getting started

Give the domain-joined container host access to the gMSA

Allow access to gMSA on the other service such as a database or file Shares

Use the CredentialSpec PowerShell module from windows-server-container-tools to store settings needed to use the gMSA

Start the container with an extra option --security-opt "credentialspec=..."

When the container is launched, the installed services running as Local System or Network Service will appear to run as the gMSA. This is similar to how those accounts work on a domain-joined hosts, except a gMSA is used instead of a computer account.

Example Uses

SQL Connection Strings

When a service is running as Local System or Network Service in a container, it can use Windows Integrated Authentication to connect to a Microsoft SQL Server.

Example:

Copy

```<code>Server=sql.contoso.com;Database=MusicStore;Integrated Security=True;MultipleActiveResultSets=True;Connect Timeout=30
</code>```

On the Microsoft SQL Server, create a login using the domain and gMSA name, followed by a $. Once the login is created, it can be added to a user on a database and given appropriate access permissions.

Example:

SQL

Copy

```<code>CREATE LOGIN "DEMO\WebApplication1$"
    FROM WINDOWS
    WITH DEFAULT_DATABASE = "MusicStore"
GO

USE MusicStore
GO
CREATE USER WebApplication1 FOR LOGIN "DEMO\WebApplication1$"
GO

EXEC sp_addrolemember 'db_datareader', 'WebApplication1'
EXEC sp_addrolemember 'db_datawriter', 'WebApplication1'
</code>```

To see it in action, check out the recorded demo available from Microsoft Ignite 2016 in the session "Walk the Path to Containerization - transforming workloads into containers".

Body:
RockScience,

I see two options here, they both enable you to inherit permissions, as opposed to passing in user and password credentials. To address your specific question, around Linux take a look at:

Active Directory Authentication with SQL Server on Linux

If you take the approach of running Windows containers, take a look at Active Directory gMSA (Group Managed Service Accounts) accounts and the following MSDN article and video:

Active Directory Service Accounts for Windows Containers

Walk the Path to Containerization - transforming workloads into containers

this would enable you to create a Windows container and R environment.

Active Directory Authentication with SQL Server on Linux

The tutorial explains how to configure SQL Server on Linux to support Active Directory (AD) authentication, also known as integrated authentication. AD Authentication enables domain-joined clients on either Windows or Linux to authenticate to SQL Server using their domain credentials and the Kerberos protocol.

AD Authentication has the following advantages over SQL Server Authentication:

Users authenticate via single sign-on, without being prompted for a password.

By creating logins for AD groups, you can manage access and permissions in SQL Server using AD group memberships.

Each user has a single identity across your organization, so you don’t have to keep track of which SQL Server logins correspond to which people.

AD enables you to enforce a centralised password policy across your organisation.

The tutorial consists of the following tasks:

Join SQL Server host to AD domain

Create AD user for SQL Server and set SPN

Configure SQL Server service keytab

Create AD-based logins in Transact-SQL

Connect to SQL Server using AD Authentication

Active Directory Service Accounts for Windows Containers

Today, group Managed Service Accounts are often used to secure connections between one computer or service to another. The general steps to use one are:

Create a gMSA

Configure the service to run as the gMSA

Give the domain-joined host running the service access to the gMSA secrets in Active Directory

Allow access to gMSA on the other service such as a database or file Shares

When the service is launched, the domain-joined host automatically gets the gMSA secrets from Active Directory, and runs the service using that account. Since that service is running as the gMSA, it can access any resources the gMSA is allowed to.

Windows Containers follow a similar process:

Create a gMSA. By default, a domain administrator or account operator must do this. Otherwise they can delegate the privileges to create & manage gMSAs to admins who manage services which use them. See gMSA Getting started

Give the domain-joined container host access to the gMSA

Allow access to gMSA on the other service such as a database or file Shares

Use the CredentialSpec PowerShell module from windows-server-container-tools to store settings needed to use the gMSA

Start the container with an extra option --security-opt "credentialspec=..."

When the container is launched, the installed services running as Local System or Network Service will appear to run as the gMSA. This is similar to how those accounts work on a domain-joined hosts, except a gMSA is used instead of a computer account.

Example Uses

SQL Connection Strings

When a service is running as Local System or Network Service in a container, it can use Windows Integrated Authentication to connect to a Microsoft SQL Server.

Example:

Copy

Server=sql.contoso.com;Database=MusicStore;Integrated Security=True;MultipleActiveResultSets=True;Connect Timeout=30

On the Microsoft SQL Server, create a login using the domain and gMSA name, followed by a $. Once the login is created, it can be added to a user on a database and given appropriate access permissions.

Example:

SQL

Copy

CREATE LOGIN "DEMO\WebApplication1$"
    FROM WINDOWS
    WITH DEFAULT_DATABASE = "MusicStore"
GO

USE MusicStore
GO
CREATE USER WebApplication1 FOR LOGIN "DEMO\WebApplication1$"
GO

EXEC sp_addrolemember 'db_datareader', 'WebApplication1'
EXEC sp_addrolemember 'db_datawriter', 'WebApplication1'

To see it in action, check out the recorded demo available from Microsoft Ignite 2016 in the session "Walk the Path to Containerization - transforming workloads into containers".


问:
Is it possible to use filecheck to match values from another file inside a CMAKE project?
I am working on a LLVM pass that dumps a json file as one of its output when run over a test.ll file (represented by %s in filecheck)

Here is the commands inside my test.ll file

```<code>; RUN: opt-load %bindir/MY-PASS.so -json-filename output.json -run-my-pass %s -S -disable-output
; RUN: cat output.json | FileCheck %testdir/output-target.json
</code>```

My ${PROJ_DIR}/Tests folder has:

${PROJ_DIR}/Tests/test.ll file

${PROJ_DIR}/Tests/output-target.json file

After configuring my project with CMAKE on ${BUILD_DIR} folder and running make check I face the issue where make check cannot find output-target.json

make check is able to generate output.json in  ${BUILD_DIR}/Tests/ folder but it is not able to find the target file %testdir/output-target.json which expands to ${BUILD_DIR}/Tests/Output/test.ll.tmpestdir/output-target.json

Should I be using something different than %testdir?

答:
You could use the predefined substitutions in lit, more specifically the %p along with the test_source_root (see the same URL provided) configuration variable for test discovery.

So, what I usually end up doing is to provide a lit.cfg.in template for cmake's configure_file command, in order to get a lit.cfg in the build directory, something along the lines of:

```<code># aggregate unit test targets under a pseudo-target
add_custom_target(check)

configure_file("${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.in" "lit.cfg")

add_custom_target(lit_tests
COMMAND ${PYTHON_EXECUTABLE} -m lit.main "${CMAKE_CURRENT_BINARY_DIR}" -v)

add_dependencies(lit_tests ${TESTEE_LIB})
add_dependencies(check lit_tests)
</code>```

This also allows me to run lit tests by typing make check in the build directory.

Then in my lit.cfg.in I add this:

```<code>config.test_source_root = '@CMAKE_CURRENT_SOURCE_DIR@'
config.substitutions.append(('%outputdatadir', "%p/data/output"))
</code>```

So in my case, %p ends up being [absolute-path-to-project-test-root-dir]/data/output, where I keep expected text outputs from test cases.

Finally in a specific test case .ll file I can use:

```<code>; RUN: diff -w %t %outputdatadir/output-target.json
</code>```

or something along those lines.

Body:
I am working on a LLVM pass that dumps a json file as one of its output when run over a test.ll file (represented by %s in filecheck)

Here is the commands inside my test.ll file

; RUN: opt-load %bindir/MY-PASS.so -json-filename output.json -run-my-pass %s -S -disable-output
; RUN: cat output.json | FileCheck %testdir/output-target.json

My ${PROJ_DIR}/Tests folder has:

${PROJ_DIR}/Tests/test.ll file

${PROJ_DIR}/Tests/output-target.json file

After configuring my project with CMAKE on ${BUILD_DIR} folder and running make check I face the issue where make check cannot find output-target.json

make check is able to generate output.json in  ${BUILD_DIR}/Tests/ folder but it is not able to find the target file %testdir/output-target.json which expands to ${BUILD_DIR}/Tests/Output/test.ll.tmpestdir/output-target.json

Should I be using something different than %testdir?

Body:
You could use the predefined substitutions in lit, more specifically the %p along with the test_source_root (see the same URL provided) configuration variable for test discovery.

So, what I usually end up doing is to provide a lit.cfg.in template for cmake's configure_file command, in order to get a lit.cfg in the build directory, something along the lines of:

# aggregate unit test targets under a pseudo-target
add_custom_target(check)

configure_file("${CMAKE_CURRENT_SOURCE_DIR}/lit.cfg.in" "lit.cfg")

add_custom_target(lit_tests
COMMAND ${PYTHON_EXECUTABLE} -m lit.main "${CMAKE_CURRENT_BINARY_DIR}" -v)

add_dependencies(lit_tests ${TESTEE_LIB})
add_dependencies(check lit_tests)

This also allows me to run lit tests by typing make check in the build directory.

Then in my lit.cfg.in I add this:

config.test_source_root = '@CMAKE_CURRENT_SOURCE_DIR@'
config.substitutions.append(('%outputdatadir', "%p/data/output"))

So in my case, %p ends up being [absolute-path-to-project-test-root-dir]/data/output, where I keep expected text outputs from test cases.

Finally in a specific test case .ll file I can use:

; RUN: diff -w %t %outputdatadir/output-target.json

or something along those lines.


问:
Getting longitude and latidude from postgis database with sqlalchemy
I am using the sqlalchemy package to state queries to my postgis database which is filled with .osm data of a city. I want to retrieve the longitude and latitude values from lets say the planet_osm_point table.
I state the sql query which looks like this:

```<code>      SELECT st_y(st_asewkt(st_transform(way, 4326))) as lat,
             st_x(st_asewkt(st_transform(way, 4326))) as lon
             "addr:housenumber" AS husenumber,
             "addr:street" AS street,
             "addr:postcode" AS postcode
      FROM planet_osm_point
</code>```

Sqlalchemy throws me this error:

```<code>sqlalchemy.exc.InternalError: (psycopg2.InternalError) FEHLER:  Argument to ST_Y() must be a point
</code>```

The only problem is the ST_Y() and ST_X() function.

答:
ST_X/ST_Y Return floats. You could either use the floats or cast them to text.

Using ST_AsEWKT is problematic here since both ST_Y/ST_X return floats and ST_AsEWKT expects a geometry.

Use the floats you get:

```<code>SELECT st_y(st_transform(way, 4326)) AS lat,
             st_x(st_transform(way, 4326)) AS lon
             "addr:housenumber" AS husenumber,
             "addr:street" AS street,
             "addr:postcode" AS postcode
      FROM planet_osm_point
</code>```

Or cast to text:

```<code>SELECT cast(st_y(st_transform(way, 4326)) as text) AS lat,
             cast(st_x(st_transform(way, 4326)) as text) AS lon
             "addr:housenumber" AS husenumber,
             "addr:street" AS street,
             "addr:postcode" AS postcode
      FROM planet_osm_point
</code>```

Body:
I am using the sqlalchemy package to state queries to my postgis database which is filled with .osm data of a city. I want to retrieve the longitude and latitude values from lets say the planet_osm_point table.
I state the sql query which looks like this:

      SELECT st_y(st_asewkt(st_transform(way, 4326))) as lat,
             st_x(st_asewkt(st_transform(way, 4326))) as lon
             "addr:housenumber" AS husenumber,
             "addr:street" AS street,
             "addr:postcode" AS postcode
      FROM planet_osm_point

Sqlalchemy throws me this error:

sqlalchemy.exc.InternalError: (psycopg2.InternalError) FEHLER:  Argument to ST_Y() must be a point

The only problem is the ST_Y() and ST_X() function.

Body:
ST_X/ST_Y Return floats. You could either use the floats or cast them to text.

Using ST_AsEWKT is problematic here since both ST_Y/ST_X return floats and ST_AsEWKT expects a geometry.

Use the floats you get:

SELECT st_y(st_transform(way, 4326)) AS lat,
             st_x(st_transform(way, 4326)) AS lon
             "addr:housenumber" AS husenumber,
             "addr:street" AS street,
             "addr:postcode" AS postcode
      FROM planet_osm_point

Or cast to text:

SELECT cast(st_y(st_transform(way, 4326)) as text) AS lat,
             cast(st_x(st_transform(way, 4326)) as text) AS lon
             "addr:housenumber" AS husenumber,
             "addr:street" AS street,
             "addr:postcode" AS postcode
      FROM planet_osm_point


问:
Scala .map(toString()) gives List of Char
I tried to make my own map method in Scala and this is the result:

```<code>def map[A, B](f: A => B)(as: List[A]): List[B] =
    for (i <- as) yield f(i)
</code>```

And if I try to convert a List[Int] to a List[String]:

```<code>val list = List(1, 2, 3, 4, 5)   //> list  : List[Int] = List(1, 2, 3, 4, 5)
map(toString()) (list)           //> res0: List[Char] = List(e, b, u, n, g)
</code>```

I get the same result if I try:

```<code>list.map(toString())             //> res1: List[Char] = List(e, b, u, n, g)
</code>```

My Question: Why gives toString() not a List("1","2"...)?

答:
You're calling toString() on the current object (which apparently returns something like "?ebung...") and passing the resulting string as an argument to map.

So you might expect that to produce a type error because map expects a function, but you're passing a string. However strings count as functions too (because RichString implements the Function1[Int, Char] trait) that take an integer and return the character at that index. So that's why you get a list containing those characters.

To do what you meant to do (create a function that calls toString on its argument), you can either use x => x.toString or _.toString for short.

Body:
I tried to make my own map method in Scala and this is the result:

def map[A, B](f: A => B)(as: List[A]): List[B] =
    for (i <- as) yield f(i)

And if I try to convert a List[Int] to a List[String]:

val list = List(1, 2, 3, 4, 5)   //> list  : List[Int] = List(1, 2, 3, 4, 5)
map(toString()) (list)           //> res0: List[Char] = List(e, b, u, n, g)

I get the same result if I try:

list.map(toString())             //> res1: List[Char] = List(e, b, u, n, g)

My Question: Why gives toString() not a List("1","2"...)?


问:
i need a query which returns 1 column 5 rows and 2nd column 10 rows
here is my query

```<code>SELECT
ISNULL(
     (
     select subjectshrt from subject_definition sd  join student_subject ss
     on(sd.sdsrno =ss.sdsrno1) where sd.tpbp LIKE ('p%')
      )
,0)A,
ISNULL(
     (
     select subjectshrt from subject_definition sd  join student_subject ss
     on(sd.sdsrno =ss.sdsrno2) where sd.tpbp LIKE ('p%')
      )
,0)b
</code>```

here first select query return 1 row and 2nd returns multiple rows
and am getting a error like

Subquery returned more than 1 value. This is not permitted when the subquery follows =, !=, <, <= , >, >= or when the subquery is used as an expression.

so what should i do to make this query work

答:
Can't you modify the code to use one query and check ON (sd.sdsrno IN (ss.sdsrno1,ss.sdsrno2))? it seems like that is the only change here.

(Or maybe I'm missing the end result here. if so, please feel free to post a couple of example outputs you'd like to see)

```<code>SELECT  COALESCE(subjectshirt,0) AS subjectshirt
FROM    subject_definition
  JOIN  student_subject
  ON    (sd.sdsrno IN (ss.sdsrno1,ss.sdsrno2))
WHERE sd.tpbp LIKE 'p%'
</code>```

Or "SELECT TOP 2" is preferred.

Body:
here is my query

SELECT
ISNULL(
     (
     select subjectshrt from subject_definition sd  join student_subject ss
     on(sd.sdsrno =ss.sdsrno1) where sd.tpbp LIKE ('p%')
      )
,0)A,
ISNULL(
     (
     select subjectshrt from subject_definition sd  join student_subject ss
     on(sd.sdsrno =ss.sdsrno2) where sd.tpbp LIKE ('p%')
      )
,0)b

here first select query return 1 row and 2nd returns multiple rows
and am getting a error like

Subquery returned more than 1 value. This is not permitted when the subquery follows =, !=, <, <= , >, >= or when the subquery is used as an expression.

so what should i do to make this query work

Body:
Can't you modify the code to use one query and check ON (sd.sdsrno IN (ss.sdsrno1,ss.sdsrno2))? it seems like that is the only change here.

(Or maybe I'm missing the end result here. if so, please feel free to post a couple of example outputs you'd like to see)

SELECT  COALESCE(subjectshirt,0) AS subjectshirt
FROM    subject_definition
  JOIN  student_subject
  ON    (sd.sdsrno IN (ss.sdsrno1,ss.sdsrno2))
WHERE sd.tpbp LIKE 'p%'

Or "SELECT TOP 2" is preferred.


问:
Do we really need a notary that validates?
At the risk of sounding naive, I ask myself "Is a validating notary necessary?", given all the issues with it - transaction and dependencies leak, exposure of state model, to name a few.
The answer I hear has something to do with the potential attack where a dishonest node tries to steal someone else's asset. For example, in a legitimate transaction, partyA sold partyB some asset S subject to a Move contract. Immediately afterwards, partyA creates a self-signed transaction that transfers S back to himself subject to a dummy contract in a bogus flow that does not even run the ledger transaction verify(). But when he calls FinaltyFlow to satisfy the simple notary to commit the transaction on the ledger, it will fail the verifyContracts() because S appoints to the Move contract which should say owner partyB must sign the bogus transaction.
So that does not convince me of the need for a validating notary.
Apparently, I must have missed something. Can someone enlighten me?

Thanks.

\Sean

答:
As you say, the advantage of the validating notary is that it prevents a malicious party from "wedging" a state - that is, creating an invalid transaction that consumes an unconsumed state that they are aware of. Although the transaction is invalid, the non-validating notary would not be aware of this, and would mark the state as spent.

You're correct that this invalid transaction would fail the FinalityFlow's call to verifyContracts(). However, we cannot force a malicious node to use FinalityFlow to send something to the notary. If the malicious node was sufficiently motivated, they could build the invalid transaction hash by hand and send that to the notary directly, for example.

However, note that in the non-validating notary case, there are still multiple layers of protection against wedging:

The malicious party has to be aware of the state(s) they want to wedge. Since information in Corda is only distributed on a need-to-know basis, the node would only be aware of a small subset of unconsumed states

If the state is wedged by accident, the node can show the invalid transaction to the notary. Upon seeing that the transaction is invalid, the notary will mark the re-mark the state as unconsumed

As Corda is a permissioned network, the notary can keep a record of the legal identity of everyone who submits a transaction. If a malicious node is submitting invalid transactions to consume states, their identity will be known, and the dispute can be resolved outside of the platform, based on the agreements governing participation in the network

On the other hand, SGX addresses the data leak issue associated with validating notaries. If the validation of the transaction occurs within an SGX enclave, the validating notary gains no information about the contents of the transaction they are validating.

Ultimately, the choice between validating and non-validating notaries comes down to the threat model associated with a given deployment.  If you are using Corda in a setting where you are sure the participants won't deliberately change their node's code or act maliciously, then a validating notary is not needed.

But if you assume somebody WILL try to cheat and would be willing to write their own code to do so, then a validating notary provides an extra layer of protection.

So Corda provides choices:

Choose to reveal more to a notary cluster if you trust the participants relatively less...

Choose to reveal less to a notary cluster if you consider the risk of revealing too much to the notaries to be the bigger problem

(and use SGX if you're paranoid about everybody!)


问:
Jenkins get user who aborted a build
Is there a way to get the username when a build is aborted by a user?
Preferably using jenkins pipeline code.

When a build is aborted by a user, it logs:

```<code>Aborted by <username>
</code>```

so I hope it is stored as a variable for a brief period.

Use case: username to be later used to inform the user itself or other users via email or other means of messaging.

答:
It seems that a InterruptedBuildAction object is inserted in to the list of build action if a job is aborted. This object can be used to retrieve the user that aborted the build. I use the following function in my Jenkinsfile:</p>

```<code>@NonCPS
def getAbortUser()
{
    def causee = ''
    def actions = currentBuild.getRawBuild().getActions(jenkins.model.InterruptedBuildAction)
    for (action in actions) {
        def causes = action.getCauses()

        // on cancellation, report who cancelled the build
        for (cause in causes) {
            causee = cause.getUser().getDisplayName()
            cause = null
        }
        causes = null
        action = null
    }
    actions = null

    return causee
}
</code>```

Title: Jenkins get user who aborted a build

Body:
Is there a way to get the username when a build is aborted by a user?
Preferably using jenkins pipeline code.

When a build is aborted by a user, it logs:

Aborted by <username>

so I hope it is stored as a variable for a brief period.

Use case: username to be later used to inform the user itself or other users via email or other means of messaging.

Body:
It seems that a InterruptedBuildAction object is inserted in to the list of build action if a job is aborted. This object can be used to retrieve the user that aborted the build. I use the following function in my Jenkinsfile:

@NonCPS
def getAbortUser()
{
    def causee = ''
    def actions = currentBuild.getRawBuild().getActions(jenkins.model.InterruptedBuildAction)
    for (action in actions) {
        def causes = action.getCauses()

        // on cancellation, report who cancelled the build
        for (cause in causes) {
            causee = cause.getUser().getDisplayName()
            cause = null
        }
        causes = null
        action = null
    }
    actions = null

    return causee
}


问:
How long will take Bubble sort with array of 1 000 000 ints?
Bubble Sort have Computational Complexity O(n^2). So if we have for example CPU 3.5 GHz, these calculation are true?
1 000 000 * 1 000 000 =10^12
3.5 GHz make ~6 000 000 per mike (I think so, please correct me if this is not true)

(10^12/6 * 10^6)/60=~2777 hours
This is true?

答:
I think you misunderstood BigO notation.

In theoretical informatics we use BigO to express the upper and lower bound so it does not mean you just plug-in the value and find some time. In-case you don't know with BigO notation O(10N) = O(N).

For example look at the complexity of quicksort, with tilde notation it is ~1.39NlogN and with BigO it's O(NlogN).

BigO or tilde notation has nothing to do with miliseconds. However the only way to know time is by using ratio test.

BigO is used to express number of compares.


问:
Bootstrap 3 Grid: Responsive Position Absolute Bottom Width Issue
I need to position an element at the bottom of a container which is using the Bootstrap grid.

The issue I'm having is that the anchor is spanning outside of the parent width. I need the anchor to span 100% and remain within the width of the parent div content-wrapper.

I need to keep the 15px padding on each side of the button that comes from the parent container.

Here is a screenshot of the issue:

HTML:

```<code><div class="content-wrapper col-xs-12">
  <div class="content">
    <a class="button" href="#">See more</a>
  </div>
</div>
</code>```

CSS:

```<code>.content-wrapper {
  width: 100%;
  height: 450px;
  background-color: gray;
  position: relative;
}

.button {
  display: block;
  background: black;
  width: 100%;
  text-align: center;
  height: 50px;
  padding-top: 13px;
  position: absolute;
  bottom: 0;
}
</code>```

Here is a JSFIDDLE to see the issue in action.

答:
I have added row class and container-fluid class over your HTML structure along with some CSS changes.
The class container-fluid was not working with your import bootstrap. so I have added it additionally in HTML part for your reference.

By this changes, you can achieve the desired result with 15px gap as mentioned above with default bootstrap structure.

Here is the working code:

/* Latest compiled and minified CSS included as External Resource*/

/* Optional theme */
/* @import url('//netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap-theme.min.css'); */

.content-wrapper {
  width: 100%;
  height: 450px;
  background-color: gray;
}
.content {
    display: inline-block;
    width: 100%;
    position: relative;
    height: 100%;
}
.button {
  display: block;
  background: black;
  width: 100%;
  text-align: center;
  height: 50px;
  padding-top: 13px;
  position: absolute;
  bottom: 0;
  left: 0;

}```
<pre class="snippet-code-html lang-html prettyprint-override"><code><!-- Latest compiled and minified CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
<div class="container-fluid">
<div class="row">

<div class="content-wrapper col-xs-12">
  <div class="content">
    See more
  </div>
</div>
</div>
</div></code>```

Body:
I need to position an element at the bottom of a container which is using the Bootstrap grid.

The issue I'm having is that the anchor is spanning outside of the parent width. I need the anchor to span 100% and remain within the width of the parent div content-wrapper.

I need to keep the 15px padding on each side of the button that comes from the parent container.

Here is a screenshot of the issue:

HTML:

    See more

CSS:

.content-wrapper {
  width: 100%;
  height: 450px;
  background-color: gray;
  position: relative;
}

.button {
  display: block;
  background: black;
  width: 100%;
  text-align: center;
  height: 50px;
  padding-top: 13px;
  position: absolute;
  bottom: 0;
}

Here is a JSFIDDLE to see the issue in action.

Body:
I have added row class and container-fluid class over your HTML structure along with some CSS changes.
The class container-fluid was not working with your import bootstrap. so I have added it additionally in HTML part for your reference.

By this changes, you can achieve the desired result with 15px gap as mentioned above with default bootstrap structure.

Here is the working code:

/* Latest compiled and minified CSS included as External Resource*/

/* Optional theme */
/* @import url('//netdna.bootstrapcdn.com/bootstrap/3.0.0/css/bootstrap-theme.min.css'); */

.content-wrapper {
  width: 100%;
  height: 450px;
  background-color: gray;
}
.content {
    display: inline-block;
    width: 100%;
    position: relative;
    height: 100%;
}
.button {
  display: block;
  background: black;
  width: 100%;
  text-align: center;
  height: 50px;
  padding-top: 13px;
  position: absolute;
  bottom: 0;
  left: 0;

}

<!-- Latest compiled and minified CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
<div class="container-fluid">
<div class="row">

<div class="content-wrapper col-xs-12">
  <div class="content">
    <a class="button" href="#">See more</a>
  </div>
</div>
</div>
</div>


问:
Swift 3 better way to write this if-statement
I am comparing values in a GKNoise field and then setting tiles into a tileMap based on the level of noise

This if-statement is comparing the values by rounding up to the nearest tenth, is there a better way to write it?

```<code>if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -1.0 {
    tileMap.setTileGroup(tileGroup4, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.9 {
    tileMap.setTileGroup(tileGroup4, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.8 {
    tileMap.setTileGroup(tileGroup4, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.7 {
    tileMap.setTileGroup(tileGroup4, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.6 {
    tileMap.setTileGroup(tileGroup4, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.5 {
    tileMap.setTileGroup(tileGroup3, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.4 {
    tileMap.setTileGroup(tileGroup3, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.3 {
    tileMap.setTileGroup(tileGroup3, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.2 {
    tileMap.setTileGroup(tileGroup3, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.1 {
    tileMap.setTileGroup(tileGroup3, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.0 {
    tileMap.setTileGroup(tileGroup2, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.1 {
    tileMap.setTileGroup(tileGroup2, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.2 {
    tileMap.setTileGroup(tileGroup2, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.3 {
    tileMap.setTileGroup(tileGroup2, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.4 {
    tileMap.setTileGroup(tileGroup2, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.5 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.6 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.7 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.8 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.9 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 1.0 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
</code>```

It seems a little drawn out and not very efficient.

答:
One relatively simple improvement is to compute round outside the conditional, use a variable to store the decision of which title group to use, and convert the chain to if-then-else. This would let you use <= and => in place of ==:

```<code>let rounded = round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows))))
var tileGroup : SKTileGroup = nil
if rounded <= -0.6 {
    tileGroup = titleGroup4
} else if rounded <= -0.1 {
    tileGroup = titleGroup3
} else if rounded <= 0.4 {
    tileGroup = titleGroup2
} else if rounded <= 1.0 {
    tileGroup = titleGroup1
}
tileMap.setTileGroup(tileGroup, forColumn: columns, row: rows)
</code>```

Body:
I am comparing values in a GKNoise field and then setting tiles into a tileMap based on the level of noise

This if-statement is comparing the values by rounding up to the nearest tenth, is there a better way to write it?

if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -1.0 {
    tileMap.setTileGroup(tileGroup4, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.9 {
    tileMap.setTileGroup(tileGroup4, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.8 {
    tileMap.setTileGroup(tileGroup4, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.7 {
    tileMap.setTileGroup(tileGroup4, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.6 {
    tileMap.setTileGroup(tileGroup4, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.5 {
    tileMap.setTileGroup(tileGroup3, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.4 {
    tileMap.setTileGroup(tileGroup3, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.3 {
    tileMap.setTileGroup(tileGroup3, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.2 {
    tileMap.setTileGroup(tileGroup3, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == -0.1 {
    tileMap.setTileGroup(tileGroup3, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.0 {
    tileMap.setTileGroup(tileGroup2, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.1 {
    tileMap.setTileGroup(tileGroup2, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.2 {
    tileMap.setTileGroup(tileGroup2, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.3 {
    tileMap.setTileGroup(tileGroup2, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.4 {
    tileMap.setTileGroup(tileGroup2, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.5 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.6 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.7 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.8 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 0.9 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}
if round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows)))) == 1.0 {
    tileMap.setTileGroup(tileGroup1, forColumn: columns, row: rows)
}

It seems a little drawn out and not very efficient.

Body:
One relatively simple improvement is to compute round outside the conditional, use a variable to store the decision of which title group to use, and convert the chain to if-then-else. This would let you use <= and => in place of ==:

let rounded = round(MasterNoise.value(atPosition: vector_float2(Float(columns),Float(rows))))
var tileGroup : SKTileGroup = nil
if rounded <= -0.6 {
    tileGroup = titleGroup4
} else if rounded <= -0.1 {
    tileGroup = titleGroup3
} else if rounded <= 0.4 {
    tileGroup = titleGroup2
} else if rounded <= 1.0 {
    tileGroup = titleGroup1
}
tileMap.setTileGroup(tileGroup, forColumn: columns, row: rows)


问:
How to remove all files with specific extension in folder?
I am looking for command to delete all files with a specific extension in a given folder. Both, for windows and mac.

Thanks!

答:
Before calling the del command you must also change the directory, for example a file bat could be like this

```<code>cd c:\MyFolderFullOfTmp
del *.tmp
</code>```

Body:
Before calling the del command you must also change the directory, for example a file bat could be like this

cd c:\MyFolderFullOfTmp
del *.tmp


问:
SQL Selecting max date by year for 3 unique IDs
I've got a list like this where I need to get the max date per year for each combination of unique IDs.

```<code>Calculated-Date|ID_A | ID_B | ID_C
   2017/11/5   |  AB |  A   |  C
   2017/10/5   |  AB |  A   |  C
   2016/10/5   |  AB |  A   |  C
   2017/10/5   |  AB |  C   |  C
   2016/10/5   |  AB |  C   |  C
   2016/10/30  |  AB |  C   |  C
   2015/10/5   |  AB |  C   |  C
</code>```

What I would like to get is this

```<code>    Calculated-Date|ID_A | ID_B | ID_C
       2017/11/5   |  AB |  A   |  C
       2016/10/5   |  AB |  A   |  C
       2017/10/5   |  AB |  C   |  C
       2016/10/30  |  AB |  C   |  C
       2015/10/5   |  AB |  C   |  C
</code>```

I've tried a variation of subqueries but I am fairly new to SQL and is truly stuck at the moment. Any help is much appreciated!

I've tried this:

```<code>select distinct max(Calculated-Date), ID_A, ID_B, ID_C
    from TABLE_1
    group by Calculated-Date, ID_A, ID_B, ID_C
order by ID_A, ID_B, ID_C
</code>```

答:
You can use GROUP BY . . . with some date arithmetic:

```<code>select id_a, id_b, id_c, max(calculated_date)
from t
group by id_a, id_b, id_c, year(calculated_date);
</code>```

Note:  This assumes that the date is, indeed, stored using a date/time date type.

Body:
I've got a list like this where I need to get the max date per year for each combination of unique IDs.

Calculated-Date|ID_A | ID_B | ID_C
   2017/11/5   |  AB |  A   |  C
   2017/10/5   |  AB |  A   |  C
   2016/10/5   |  AB |  A   |  C
   2017/10/5   |  AB |  C   |  C
   2016/10/5   |  AB |  C   |  C
   2016/10/30  |  AB |  C   |  C
   2015/10/5   |  AB |  C   |  C

What I would like to get is this

    Calculated-Date|ID_A | ID_B | ID_C
       2017/11/5   |  AB |  A   |  C
       2016/10/5   |  AB |  A   |  C
       2017/10/5   |  AB |  C   |  C
       2016/10/30  |  AB |  C   |  C
       2015/10/5   |  AB |  C   |  C

I've tried a variation of subqueries but I am fairly new to SQL and is truly stuck at the moment. Any help is much appreciated!

I've tried this:

select distinct max(Calculated-Date), ID_A, ID_B, ID_C
    from TABLE_1
    group by Calculated-Date, ID_A, ID_B, ID_C
order by ID_A, ID_B, ID_C

Body:
You can use GROUP BY . . . with some date arithmetic:

select id_a, id_b, id_c, max(calculated_date)
from t
group by id_a, id_b, id_c, year(calculated_date);

Note:  This assumes that the date is, indeed, stored using a date/time date type.


问:
Python: subprocess.call, stdout to file, stderr to file, display stderr on screen in real time
I have a command line tool (actually, several) that I am writing a wrapper for in Python.

The tool is generally used like this:

```<code> $ path_to_tool -option1 -option2 > file_out
</code>```

The user gets the output written to file_out, and is also able to see various status messages of the tool as it is running.

I want to replicate this behavior, while also logging stderr (the status messages) to a file.

What I have is this:

```<code>from subprocess import call
call(['path_to_tool','-option1','option2'], stdout = file_out, stderr = log_file)
</code>```

This works fine EXCEPT that stderr is not written to the screen.
I can add code to print the contents of the log_file to the screen of course, but then the user will see it after everything is done rather than while it is happening.

To recap, desired behavior is:

use call(), or subprocess()

direct stdout to a file

direct stderr to a file, while also writing stderr to the screen in real time as if the
tool had been called directly from the command line.

I have a feeling I'm either missing something really simple, or this is much more complicated than I thought...thanks for any help!

EDIT: this only needs to work on Linux.

答:
I had to make a few changes to @abarnert's answer for Python 3. This seems to work:</p>

```<code>def tee_pipe(pipe, f1, f2):
    for line in pipe:
        f1.write(line)
        f2.write(line)

proc = subprocess.Popen(["/bin/echo", "hello"],
                        stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE)

# Open the output files for stdout/err in unbuffered mode.
out_file = open("stderr.log", "wb", 0)
err_file = open("stdout.log", "wb", 0)

stdout = sys.stdout
stderr = sys.stderr

# On Python3 these are wrapped with BufferedTextIO objects that we don't
# want.
if sys.version_info[0] >= 3:
    stdout = stdout.buffer
    stderr = stderr.buffer

# Start threads to duplicate the pipes.
out_thread = threading.Thread(target=tee_pipe,
                              args=(proc.stdout, out_file, stdout))
err_thread = threading.Thread(target=tee_pipe,
                              args=(proc.stderr, err_file, stderr))

out_thread.start()
err_thread.start()

# Wait for the command to finish.
proc.wait()

# Join the pipe threads.
out_thread.join()
err_thread.join()
</code>```

Body:
I have a command line tool (actually, several) that I am writing a wrapper for in Python.

The tool is generally used like this:

 $ path_to_tool -option1 -option2 > file_out

The user gets the output written to file_out, and is also able to see various status messages of the tool as it is running.

I want to replicate this behavior, while also logging stderr (the status messages) to a file.

What I have is this:

from subprocess import call
call(['path_to_tool','-option1','option2'], stdout = file_out, stderr = log_file)

This works fine EXCEPT that stderr is not written to the screen.
I can add code to print the contents of the log_file to the screen of course, but then the user will see it after everything is done rather than while it is happening.

To recap, desired behavior is:

use call(), or subprocess()

direct stdout to a file

direct stderr to a file, while also writing stderr to the screen in real time as if the
tool had been called directly from the command line.

I have a feeling I'm either missing something really simple, or this is much more complicated than I thought...thanks for any help!

EDIT: this only needs to work on Linux.

Body:
I had to make a few changes to @abarnert's answer for Python 3. This seems to work:

def tee_pipe(pipe, f1, f2):
    for line in pipe:
        f1.write(line)
        f2.write(line)

proc = subprocess.Popen(["/bin/echo", "hello"],
                        stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE)

# Open the output files for stdout/err in unbuffered mode.
out_file = open("stderr.log", "wb", 0)
err_file = open("stdout.log", "wb", 0)

stdout = sys.stdout
stderr = sys.stderr

# On Python3 these are wrapped with BufferedTextIO objects that we don't
# want.
if sys.version_info[0] >= 3:
    stdout = stdout.buffer
    stderr = stderr.buffer

# Start threads to duplicate the pipes.
out_thread = threading.Thread(target=tee_pipe,
                              args=(proc.stdout, out_file, stdout))
err_thread = threading.Thread(target=tee_pipe,
                              args=(proc.stderr, err_file, stderr))

out_thread.start()
err_thread.start()

# Wait for the command to finish.
proc.wait()

# Join the pipe threads.
out_thread.join()
err_thread.join()