问:
ZeroMQ: very high latency with PUSH/PULL

Summary

I see very high latency at the receiver using zmq.PUSH/PULL sockets.

Details

I'm new to ZeroMQ, and am trying to send a few MBytes from the sender to the receiver (my final aim is to send a camera stream from the sender to the receiver). The sender and receiver are different machines on the same local network (one is a Mac and the other is running Ubuntu 18.04, if that matters). The sender is able to send its packets very fast (I'm guessing they get buffered in some zmq/tcp queue), but the receiver receives them very slowly with increasing latency per packet.

Here's the sender :

```<code>  1 """
  2 Sender
  3 """
  4 import zmq
  5 from time import time
  6 import sys
  7
  8 addr = "tcp://192.168.86.33:5555"
  9
 10 context = zmq.Context()
 11 socket = context.socket(zmq.PUSH)
 12 socket.bind(addr)
 13
 14 num = 0
 15 while True:
 16     frame = [0] * 1000000
 17     ts = time()
 18     num += 1
 19     socket.send_pyobj(dict(num=num, frame=frame, ts=ts))
 20
 21     delay = time() - ts
 22     print("Sender:: pkt_num: {}, delay: {}".format(num, delay))

</code>```

And the receiver :

```<code>  1 """
  2 Receiver
  3 """
  4 import zmq
  5 from time import time
  6
  7 addr = "tcp://192.168.86.33:5555"
  8 context = zmq.Context()
  9
 10 socket = context.socket(zmq.PULL)
 11 socket.connect(addr)
 12
 13 while True:
 14     msg = socket.recv_pyobj()
 15     frame = msg['frame']
 16     num = msg['num']
 17     ts = msg['ts']
 18
 19     delay = time() - ts
 20
 21     if True:
 22         print("Receiver:: pkt_num: {} latency: {}".format(num, delay))
</code>```

When I run this, I see the sender is able to send its packets very quickly :

```<code>Sender:: pkt_num: 1, delay: 0.026965618133544922
Sender:: pkt_num: 2, delay: 0.018309354782104492
Sender:: pkt_num: 3, delay: 0.01821303367614746
Sender:: pkt_num: 4, delay: 0.016669273376464844
Sender:: pkt_num: 5, delay: 0.01674652099609375
Sender:: pkt_num: 6, delay: 0.01668095588684082
Sender:: pkt_num: 7, delay: 0.015082836151123047
Sender:: pkt_num: 8, delay: 0.014363527297973633
Sender:: pkt_num: 9, delay: 0.014063835144042969
Sender:: pkt_num: 10, delay: 0.014398813247680664
</code>```

But the receiver sees very high and growing packet latencies :

```<code>Receiver:: pkt_num: 1 latency: 0.1272585391998291
Receiver:: pkt_num: 2 latency: 0.2539491653442383
Receiver:: pkt_num: 3 latency: 0.40800905227661133
Receiver:: pkt_num: 4 latency: 0.5737316608428955
Receiver:: pkt_num: 5 latency: 0.7272651195526123
Receiver:: pkt_num: 6 latency: 0.9418754577636719
Receiver:: pkt_num: 7 latency: 1.0799565315246582
Receiver:: pkt_num: 8 latency: 1.228663682937622
Receiver:: pkt_num: 9 latency: 1.3731486797332764
Receiver:: pkt_num: 10 latency: 1.5067603588104248
</code>```

I tried swapping the sender and receiver between the Mac and Linux machines, and saw the same behavior. Since my goal is to send a video stream from the sender to the receiver, these high latencies make this unusable for that purpose.

Edit 1

Based on user3666197's suggestion, I edited the sender/receiver test code to remove some overheads. On the sender side keep sending the same dict. I also added more prints.

Sender:

```<code> 14 num = 0
 15 frame = [0] * 1000000
 16 payload = dict(num=0, frame=frame, ts=0.0)
 17 while True:
 18     payload['num'] += 1
 19     payload['ts'] = time()
 20     socket.send_pyobj(payload)
 21
 22     delay = time() - payload['ts']
 23     print("Sender:: pkt_num: {:>6d}, delay: {:6f}" \
 24           .format(payload['num'], delay))
</code>```

Receiver

```<code> 10 socket = context.socket(zmq.PULL)
 11 socket.connect(addr)
 12 clk = zmq.Stopwatch()
 13 clk.start()
 14
 15 while True:
 16     iterT = clk.stop()
 17     clk.start()
 18     msg = socket.recv_pyobj()
 19     rcvT = clk.stop()
 20     delay = time() - msg['ts']
 21
 22     print("Server:: pkt_num: {:>6d} latency: {:>6f} iterT: {} rcvT: {}" \
 23          .format(msg['num'], delay, iterT, rcvT))
 24     clk.start()
</code>```

The sender's per packet delay has reduced further. An interesting datapoint revealed is that the receiver takes almost 0.15s to receive each packet, which seems to be the main problem.

```<code>Sender:: pkt_num:      1, delay: 1.797830
Sender:: pkt_num:      2, delay: 0.025297
Sender:: pkt_num:      3, delay: 0.019500
Sender:: pkt_num:      4, delay: 0.019500
Sender:: pkt_num:      5, delay: 0.018166
Sender:: pkt_num:      6, delay: 0.017320
Sender:: pkt_num:      7, delay: 0.017258
Sender:: pkt_num:      8, delay: 0.017277
Sender:: pkt_num:      9, delay: 0.017426
Sender:: pkt_num:     10, delay: 0.017340
</code>```

In the receiver's prints, rcvT is the per-packet receive time in microseconds.

```<code>Server:: pkt_num:      1 latency: 2.395570 iterT: 1 rcvT: 331601
Server:: pkt_num:      2 latency: 0.735229 iterT: 1 rcvT: 137547
Server:: pkt_num:      3 latency: 0.844345 iterT: 1 rcvT: 134385
Server:: pkt_num:      4 latency: 0.991852 iterT: 1 rcvT: 166980
Server:: pkt_num:      5 latency: 1.089429 iterT: 2 rcvT: 117047
Server:: pkt_num:      6 latency: 1.190770 iterT: 2 rcvT: 119466
Server:: pkt_num:      7 latency: 1.348077 iterT: 2 rcvT: 174566
Server:: pkt_num:      8 latency: 1.460732 iterT: 1 rcvT: 129858
Server:: pkt_num:      9 latency: 1.585445 iterT: 2 rcvT: 141948
Server:: pkt_num:     10 latency: 1.717757 iterT: 1 rcvT: 149666
</code>```

Edit 2

I implemented the solution pointed out in this answer that uses PUB/SUB, and it works perfectly; I get 30fps video on the receiver. I still do not understand why my PUSH/PULL sample sees a delay.

答:
Q : "ZeroMQ: very high latency with PUSH/PULL"

The ZeroMQ PUSH/PULL archetype is a part of the story. Let's decompose the claimed latency :

1 )

Let's measure the actual payload assembly costs [us] :

```<code>aClk = zmq.Stopwatch()
num  = 0
while True:
    aClk.start() #--------------------------------------- .start() the microsecond timer
    frame     = [0] * 1000000
    ts        = time()
    num      += 1
    aPayLOAD  = dict( num   = num,
                      frame = frame,
                      ts    = ts
                      )
    _1 = aClk.stop() #---------------------------------- .stop() the microsecond timer
    aPayLOAD['ts'] = time()

    aClk.start() #-------------------------------------- .start() the microsecond timer
    socket.send_pyobj( aPayLOAD )
    _2 = aClk.stop() #---------------------------------- .stop() the microsecond timer
    delay        = time() - aPayLOAD['ts']

    print( "Sender:: pkt_num: {0:>6d}, assy: {1:>6d} [us] .send_pyobj(): {2:>6d} [us] 'delay': {3:>10.6f} [s] ".format( num, _1, _2, delay ) )
</code>```

2 )

Doing the same for the receiving side, you have a clear picture of how many [us] got consumed on the Python side, before the ZeroMQ has ever touched the first byte of the payload.

3 )

Next comes the performance tuning :

Avoid Python garbage collection

Improve the wasted memory-management ( avoid creating new instances, better inject data into memory-efficient "handlers" ( numpy.ndarray.data-zones instead of assigning re-composing list-s into a python variable )

refactor the code to avoid expensive dict-based key-value mapping and rather use compact binary maps, as available in struct-module ( your proposed mapping is both static & trivial )

test adding a compression-step ( may improve a bit the over-the-network hop latency )

4 )

Only now the ZeroMQ part comes :

may test improved performance envelopes from adding more IO-threads into either of Context( nIO_threads )-instances

may go in for .setsockopt( zmq.CONFLATE, 1 ) to ignore all non-"last" frames, as video-streaming may get no added value from "late" re-creation of already "old" frames

Body:
Summary

I see very high latency at the receiver using zmq.PUSH/PULL sockets.

Details

I'm new to ZeroMQ, and am trying to send a few MBytes from the sender to the receiver (my final aim is to send a camera stream from the sender to the receiver). The sender and receiver are different machines on the same local network (one is a Mac and the other is running Ubuntu 18.04, if that matters). The sender is able to send its packets very fast (I'm guessing they get buffered in some zmq/tcp queue), but the receiver receives them very slowly with increasing latency per packet.

Here's the sender :

  1 """
  2 Sender
  3 """
  4 import zmq
  5 from time import time
  6 import sys
  7
  8 addr = "tcp://192.168.86.33:5555"
  9
 10 context = zmq.Context()
 11 socket = context.socket(zmq.PUSH)
 12 socket.bind(addr)
 13
 14 num = 0
 15 while True:
 16     frame = [0] * 1000000
 17     ts = time()
 18     num += 1
 19     socket.send_pyobj(dict(num=num, frame=frame, ts=ts))
 20
 21     delay = time() - ts
 22     print("Sender:: pkt_num: {}, delay: {}".format(num, delay))

And the receiver :

  1 """
  2 Receiver
  3 """
  4 import zmq
  5 from time import time
  6
  7 addr = "tcp://192.168.86.33:5555"
  8 context = zmq.Context()
  9
 10 socket = context.socket(zmq.PULL)
 11 socket.connect(addr)
 12
 13 while True:
 14     msg = socket.recv_pyobj()
 15     frame = msg['frame']
 16     num = msg['num']
 17     ts = msg['ts']
 18
 19     delay = time() - ts
 20
 21     if True:
 22         print("Receiver:: pkt_num: {} latency: {}".format(num, delay))

When I run this, I see the sender is able to send its packets very quickly :

Sender:: pkt_num: 1, delay: 0.026965618133544922
Sender:: pkt_num: 2, delay: 0.018309354782104492
Sender:: pkt_num: 3, delay: 0.01821303367614746
Sender:: pkt_num: 4, delay: 0.016669273376464844
Sender:: pkt_num: 5, delay: 0.01674652099609375
Sender:: pkt_num: 6, delay: 0.01668095588684082
Sender:: pkt_num: 7, delay: 0.015082836151123047
Sender:: pkt_num: 8, delay: 0.014363527297973633
Sender:: pkt_num: 9, delay: 0.014063835144042969
Sender:: pkt_num: 10, delay: 0.014398813247680664

But the receiver sees very high and growing packet latencies :

Receiver:: pkt_num: 1 latency: 0.1272585391998291
Receiver:: pkt_num: 2 latency: 0.2539491653442383
Receiver:: pkt_num: 3 latency: 0.40800905227661133
Receiver:: pkt_num: 4 latency: 0.5737316608428955
Receiver:: pkt_num: 5 latency: 0.7272651195526123
Receiver:: pkt_num: 6 latency: 0.9418754577636719
Receiver:: pkt_num: 7 latency: 1.0799565315246582
Receiver:: pkt_num: 8 latency: 1.228663682937622
Receiver:: pkt_num: 9 latency: 1.3731486797332764
Receiver:: pkt_num: 10 latency: 1.5067603588104248

I tried swapping the sender and receiver between the Mac and Linux machines, and saw the same behavior. Since my goal is to send a video stream from the sender to the receiver, these high latencies make this unusable for that purpose.

Edit 1

Based on user3666197's suggestion, I edited the sender/receiver test code to remove some overheads. On the sender side keep sending the same dict. I also added more prints.

Sender:

 14 num = 0
 15 frame = [0] * 1000000
 16 payload = dict(num=0, frame=frame, ts=0.0)
 17 while True:
 18     payload['num'] += 1
 19     payload['ts'] = time()
 20     socket.send_pyobj(payload)
 21
 22     delay = time() - payload['ts']
 23     print("Sender:: pkt_num: {:>6d}, delay: {:6f}" \
 24           .format(payload['num'], delay))

Receiver

 10 socket = context.socket(zmq.PULL)
 11 socket.connect(addr)
 12 clk = zmq.Stopwatch()
 13 clk.start()
 14
 15 while True:
 16     iterT = clk.stop()
 17     clk.start()
 18     msg = socket.recv_pyobj()
 19     rcvT = clk.stop()
 20     delay = time() - msg['ts']
 21
 22     print("Server:: pkt_num: {:>6d} latency: {:>6f} iterT: {} rcvT: {}" \
 23          .format(msg['num'], delay, iterT, rcvT))
 24     clk.start()

The sender's per packet delay has reduced further. An interesting datapoint revealed is that the receiver takes almost 0.15s to receive each packet, which seems to be the main problem.

Sender:: pkt_num:      1, delay: 1.797830
Sender:: pkt_num:      2, delay: 0.025297
Sender:: pkt_num:      3, delay: 0.019500
Sender:: pkt_num:      4, delay: 0.019500
Sender:: pkt_num:      5, delay: 0.018166
Sender:: pkt_num:      6, delay: 0.017320
Sender:: pkt_num:      7, delay: 0.017258
Sender:: pkt_num:      8, delay: 0.017277
Sender:: pkt_num:      9, delay: 0.017426
Sender:: pkt_num:     10, delay: 0.017340

In the receiver's prints, rcvT is the per-packet receive time in microseconds.

Server:: pkt_num:      1 latency: 2.395570 iterT: 1 rcvT: 331601
Server:: pkt_num:      2 latency: 0.735229 iterT: 1 rcvT: 137547
Server:: pkt_num:      3 latency: 0.844345 iterT: 1 rcvT: 134385
Server:: pkt_num:      4 latency: 0.991852 iterT: 1 rcvT: 166980
Server:: pkt_num:      5 latency: 1.089429 iterT: 2 rcvT: 117047
Server:: pkt_num:      6 latency: 1.190770 iterT: 2 rcvT: 119466
Server:: pkt_num:      7 latency: 1.348077 iterT: 2 rcvT: 174566
Server:: pkt_num:      8 latency: 1.460732 iterT: 1 rcvT: 129858
Server:: pkt_num:      9 latency: 1.585445 iterT: 2 rcvT: 141948
Server:: pkt_num:     10 latency: 1.717757 iterT: 1 rcvT: 149666

Edit 2

I implemented the solution pointed out in this answer that uses PUB/SUB, and it works perfectly; I get 30fps video on the receiver. I still do not understand why my PUSH/PULL sample sees a delay.

Body:
Q : "ZeroMQ: very high latency with PUSH/PULL"

The ZeroMQ PUSH/PULL archetype is a part of the story. Let's decompose the claimed latency :

1 )

Let's measure the actual payload assembly costs [us] :

aClk = zmq.Stopwatch()
num  = 0
while True:
    aClk.start() #--------------------------------------- .start() the microsecond timer
    frame     = [0] * 1000000
    ts        = time()
    num      += 1
    aPayLOAD  = dict( num   = num,
                      frame = frame,
                      ts    = ts
                      )
    _1 = aClk.stop() #---------------------------------- .stop() the microsecond timer
    aPayLOAD['ts'] = time()

    aClk.start() #-------------------------------------- .start() the microsecond timer
    socket.send_pyobj( aPayLOAD )
    _2 = aClk.stop() #---------------------------------- .stop() the microsecond timer
    delay        = time() - aPayLOAD['ts']

    print( "Sender:: pkt_num: {0:>6d}, assy: {1:>6d} [us] .send_pyobj(): {2:>6d} [us] 'delay': {3:>10.6f} [s] ".format( num, _1, _2, delay ) )

2 )

Doing the same for the receiving side, you have a clear picture of how many [us] got consumed on the Python side, before the ZeroMQ has ever touched the first byte of the payload.

3 )

Next comes the performance tuning :

Avoid Python garbage collection

Improve the wasted memory-management ( avoid creating new instances, better inject data into memory-efficient "handlers" ( numpy.ndarray.data-zones instead of assigning re-composing list-s into a python variable )

refactor the code to avoid expensive dict-based key-value mapping and rather use compact binary maps, as available in struct-module ( your proposed mapping is both static & trivial )

test adding a compression-step ( may improve a bit the over-the-network hop latency )

4 )

Only now the ZeroMQ part comes :

may test improved performance envelopes from adding more IO-threads into either of Context( nIO_threads )-instances

may go in for .setsockopt( zmq.CONFLATE, 1 ) to ignore all non-"last" frames, as video-streaming may get no added value from "late" re-creation of already "old" frames


来源: stackexchange

问:
How do I make python read a value after a string, then comparing and outputting the value?
For example, say I have a set of data.txt:

```<code>Name1 | Points:100
Name2 | Points:150
Name3 | Points:200
</code>```

I want python to read the value after "Points:", compare it to a variable "j" and output the line only if the value is less than the variable "j".

I'm a complete beginner to Python, so apologies if this seems very easy. Thanks for any replies :)

答:
you may use this:

This code reads all the lines from the file and iterates them one by one. we find the location where the number begins and we convert it from string to int to compare.

```<code>path = "some/path/to/file.txt"
with open(path, "r") as datafile:
    lines = datafile.readlines()

j = 150
for row in lines:
    row = row.replace("\n", "")
    num_start = row.find(":") + 1
    # get chars from num_start till the end and convert to int
    point = int(row[num_start:])
    print(point)
    if point > j:
        print(row)
</code>```

ContentLicense: CC BY-SA 4.0

Body:
For example, say I have a set of data.txt:

Name1 | Points:100
Name2 | Points:150
Name3 | Points:200

I want python to read the value after "Points:", compare it to a variable "j" and output the line only if the value is less than the variable "j".

I'm a complete beginner to Python, so apologies if this seems very easy. Thanks for any replies :)

Body:
you may use this:

This code reads all the lines from the file and iterates them one by one. we find the location where the number begins and we convert it from string to int to compare.

path = "some/path/to/file.txt"
with open(path, "r") as datafile:
    lines = datafile.readlines()

j = 150
for row in lines:
    row = row.replace("\n", "")
    num_start = row.find(":") + 1
    # get chars from num_start till the end and convert to int
    point = int(row[num_start:])
    print(point)
    if point > j:
        print(row)


问:
How to implement a custom dataset to pytorch project
I’d like to train a NN with a given dataset (all including some kind of object, for example: a dog), after the training the NN should help me classifying my images (downloaded from instagram) as “image includes a dog (with probability:0.XX)” or “image doesn’t include a dog (with probability: 0.XX)”.

Obviously images from instagram-images do not always have the same size (but they all have the same format (.jpg) due to filtering), and the images from my dataset do not have the same size as well.

While testing, I'm getting this Error:

```<code>    Traceback (most recent call last):
  File "/venv/nn.py", line 129, in <module>
    train(model=globalModel, hardware=hw, train_loader=loader, optimizer=optimizer, epoch=1)
  File "/venv/nn.py", line 74, in train
    for batch_idx, (data, target) in enumerate(train_loader):
  File "\venv\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "\venv\lib\site-packages\torch\utils\data\dataloader.py", line 384, in _next_data
    index = self._next_index()  # may raise StopIteration
  File "\venv\lib\site-packages\torch\utils\data\dataloader.py", line 339, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "\venv\lib\site-packages\torch\utils\data\sampler.py", line 200, in __iter__
    for idx in self.sampler:
  File "\venv\lib\site-packages\torch\utils\data\sampler.py", line 62, in __iter__
    return iter(range(len(self.data_source)))
TypeError: object of type 'type' has no len()
</code>```

with this code: https://pastebin.com/DcvbeMcq

Does anyone know how to implement a custom dataset right?

答:
Looks like the problem is that you're passing not the instance of customDataset, but the class type itself.
Try changing your loader creation code to

```<code>loader = torch.utils.data.DataLoader(customDataset(), batch_size=4)
</code>```

Body:
I’d like to train a NN with a given dataset (all including some kind of object, for example: a dog), after the training the NN should help me classifying my images (downloaded from instagram) as “image includes a dog (with probability:0.XX)” or “image doesn’t include a dog (with probability: 0.XX)”.

Obviously images from instagram-images do not always have the same size (but they all have the same format (.jpg) due to filtering), and the images from my dataset do not have the same size as well.

While testing, I'm getting this Error:

    Traceback (most recent call last):
  File "/venv/nn.py", line 129, in <module>
    train(model=globalModel, hardware=hw, train_loader=loader, optimizer=optimizer, epoch=1)
  File "/venv/nn.py", line 74, in train
    for batch_idx, (data, target) in enumerate(train_loader):
  File "\venv\lib\site-packages\torch\utils\data\dataloader.py", line 345, in __next__
    data = self._next_data()
  File "\venv\lib\site-packages\torch\utils\data\dataloader.py", line 384, in _next_data
    index = self._next_index()  # may raise StopIteration
  File "\venv\lib\site-packages\torch\utils\data\dataloader.py", line 339, in _next_index
    return next(self._sampler_iter)  # may raise StopIteration
  File "\venv\lib\site-packages\torch\utils\data\sampler.py", line 200, in __iter__
    for idx in self.sampler:
  File "\venv\lib\site-packages\torch\utils\data\sampler.py", line 62, in __iter__
    return iter(range(len(self.data_source)))
TypeError: object of type 'type' has no len()

with this code: https://pastebin.com/DcvbeMcq

Does anyone know how to implement a custom dataset right?

Body:
Looks like the problem is that you're passing not the instance of customDataset, but the class type itself.
Try changing your loader creation code to

loader = torch.utils.data.DataLoader(customDataset(), batch_size=4)


问:
Get a vector with the count of consecutive specific values in R
I have a dataframe that looks like this:

```<code>    (TimeDate,Movement,Booked)

    1   2017-01-02 09:00:13 FALSE   TRUE
    2   2017-01-02 09:01:13 FALSE   TRUE
    3   2017-01-02 09:02:15 TRUE    TRUE
    4   2017-01-02 09:03:13 TRUE    TRUE
    5   2017-01-02 09:04:13 FALSE   FALSE
    6   2017-01-02 09:05:13 FALSE   FALSE
    7   2017-01-02 09:06:13 FALSE   TRUE
    8   2017-01-02 09:07:13 FALSE   TRUE
    9   2017-01-02 09:08:14 TRUE    TRUE
    10  2017-01-02 09:09:13 FALSE   TRUE
</code>```

It has a minute by minute log with two booleans corresponding to movement detected and room booked. What I want to obtain is a vector that counts the consecutive occurrences of (FALSE,TRUE) for Movement and Booked comumns, so in this case I would want a resulting vector like this one:

```<code>    c(2,2,1)
</code>```

Where the first element is the count of lines 1 and 2, the second for lines 7 and 8 and the last one for line 10.

Honestly I don't even know how to start with this.
Thanks!

答:
Here is one option with rleid where we create a grouping column based on the FALSE in 'Movement' and Booked TRUE, get the count of the column

```<code>library(dplyr)
library(data.table)
df1 %>%
     mutate(rl = rleid(!Movement, Booked)) %>%
     filter(!(! Movement & Booked)) %>%
     count(rl) %>%
     pull(n)
#[1] 2 2 1
</code>```

Or with rle from base R

```<code>with(df1, with(rle(paste(Movement, Booked)), lengths[values == 'FALSE TRUE']))
#[1] 2 2 1
</code>```

or using a compact way

```<code>with(df1, table(rleid(Movement, Booked)[!Movement & Booked]))

#1 4 6
#2 2 1
</code>```

data

```<code>df1 <- structure(list(TimeDate = c("2017-01-02 09:00:13", "2017-01-02 09:01:13",
"2017-01-02 09:02:15", "2017-01-02 09:03:13", "2017-01-02 09:04:13",
"2017-01-02 09:05:13", "2017-01-02 09:06:13", "2017-01-02 09:07:13",
"2017-01-02 09:08:14", "2017-01-02 09:09:13"), Movement = c(FALSE,
FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE),
    Booked = c(TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, TRUE, TRUE,
    TRUE, TRUE)), class = "data.frame", row.names = c("1", "2",
"3", "4", "5", "6", "7", "8", "9", "10"))
</code>```

Body:
I have a dataframe that looks like this:

    (TimeDate,Movement,Booked)

    1   2017-01-02 09:00:13 FALSE   TRUE
    2   2017-01-02 09:01:13 FALSE   TRUE
    3   2017-01-02 09:02:15 TRUE    TRUE
    4   2017-01-02 09:03:13 TRUE    TRUE
    5   2017-01-02 09:04:13 FALSE   FALSE
    6   2017-01-02 09:05:13 FALSE   FALSE
    7   2017-01-02 09:06:13 FALSE   TRUE
    8   2017-01-02 09:07:13 FALSE   TRUE
    9   2017-01-02 09:08:14 TRUE    TRUE
    10  2017-01-02 09:09:13 FALSE   TRUE

It has a minute by minute log with two booleans corresponding to movement detected and room booked. What I want to obtain is a vector that counts the consecutive occurrences of (FALSE,TRUE) for Movement and Booked comumns, so in this case I would want a resulting vector like this one:

    c(2,2,1)

Where the first element is the count of lines 1 and 2, the second for lines 7 and 8 and the last one for line 10.

Honestly I don't even know how to start with this.
Thanks!

Body:
Here is one option with rleid where we create a grouping column based on the FALSE in 'Movement' and Booked TRUE, get the count of the column

library(dplyr)
library(data.table)
df1 %>%
     mutate(rl = rleid(!Movement, Booked)) %>%
     filter(!(! Movement & Booked)) %>%
     count(rl) %>%
     pull(n)
#[1] 2 2 1

Or with rle from base R

with(df1, with(rle(paste(Movement, Booked)), lengths[values == 'FALSE TRUE']))
#[1] 2 2 1

or using a compact way

with(df1, table(rleid(Movement, Booked)[!Movement & Booked]))

#1 4 6
#2 2 1

data

df1 <- structure(list(TimeDate = c("2017-01-02 09:00:13", "2017-01-02 09:01:13",
"2017-01-02 09:02:15", "2017-01-02 09:03:13", "2017-01-02 09:04:13",
"2017-01-02 09:05:13", "2017-01-02 09:06:13", "2017-01-02 09:07:13",
"2017-01-02 09:08:14", "2017-01-02 09:09:13"), Movement = c(FALSE,
FALSE, TRUE, TRUE, FALSE, FALSE, FALSE, FALSE, TRUE, FALSE),
    Booked = c(TRUE, TRUE, TRUE, TRUE, FALSE, FALSE, TRUE, TRUE,
    TRUE, TRUE)), class = "data.frame", row.names = c("1", "2",
"3", "4", "5", "6", "7", "8", "9", "10"))


问:
React.js: The value from variable was changed after checkbox but not written in <imput value={...}.../>
One of the form fields: address Actual and address of registration. With an active checkbox, the address Actual should be equal to the address of registration. But in fact, the address Actual is not recorded in a variable.
When checkbox is active, call checkBoxAddress:

```<code>    checkBoxAddress = () => {
        this.setState({checked: !this.state.checked});
    }
</code>```

Then check the checkbox before return(...), and change the value of the variable:

```<code>        if (this.state.checked) {
            addressActual = passportAddress;
            console.log(addressActual) // There are true value, worked. But not in form value

        }
</code>```

Form:

```<code>...
<div className="form-group col-md-12">
                                <input name="passportAddress" type="text" className="form-control"
                                       placeholder="Адрес прописки"
                                       value={passportAddress} onChange={this.changeHandler}/>
                            </div>
                        </div>
                        <div className="form-check">
                            <div className="form-group">
                                <input className="form-check-input" type="checkbox" id="gridCheck"
                                       onChange={this.checkBoxAddress}
                                       defaultChecked={this.state.checked}/>
                                <label className="form-check-label" htmlFor="gridCheck">
                                    Адрес прописки совпадает с почтовым
                                </label>
                            </div>
                        </div>
                        <div className="form-row">
                            <div className="form-group col-md-12">
                                <input name="addressActual" type="text" className="form-control"
                                       placeholder="Адрес почтовый"
                                       value={addressActual} onChange={this.changeHandler}/>
                            </div>
...
</code>```

In the input field, the address is substituted, but not stored in the variable:

pasportAddress="12345", addressActual=""

答:
```<code>    checkBoxAddress = () => {
        console.log(this.state.checked)
        this.setState({checked: !this.state.checked})
        if (!this.state.checked) {
            this.setState({addressActual: this.state.passportAddress})
        }
    };
</code>```

Now, it's worked.

Body:
One of the form fields: address Actual and address of registration. With an active checkbox, the address Actual should be equal to the address of registration. But in fact, the address Actual is not recorded in a variable.
When checkbox is active, call checkBoxAddress:

    checkBoxAddress = () => {
        this.setState({checked: !this.state.checked});
    }

Then check the checkbox before return(...), and change the value of the variable:

        if (this.state.checked) {
            addressActual = passportAddress;
            console.log(addressActual) // There are true value, worked. But not in form value

        }

Form:

...

                                <input name="passportAddress" type="text" className="form-control"
                                       placeholder="Адрес прописки"
                                       value={passportAddress} onChange={this.changeHandler}/>

                                <input className="form-check-input" type="checkbox" id="gridCheck"
                                       onChange={this.checkBoxAddress}
                                       defaultChecked={this.state.checked}/>

                                    Адрес прописки совпадает с почтовым

                                <input name="addressActual" type="text" className="form-control"
                                       placeholder="Адрес почтовый"
                                       value={addressActual} onChange={this.changeHandler}/>

...

In the input field, the address is substituted, but not stored in the variable:

pasportAddress="12345", addressActual=""

Body:
    checkBoxAddress = () => {
        console.log(this.state.checked)
        this.setState({checked: !this.state.checked})
        if (!this.state.checked) {
            this.setState({addressActual: this.state.passportAddress})
        }
    };

Now, it's worked.


问:
PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE concurrent monitoring
I'm working on a custom implementation on top of perf_event_open syscall.

The implementation aims to support various of PERF_TYPE_HARDWARE, PERF_TYPE_SOFTWARE and PERF_TYPE_HW_CACHE events for specific threads on any core.

In Intel® 64 and IA-32 Architectures Software Developer’s Manual vol 3B I see the following for my testing CPU (Kaby Lake):

To my understanding so far, one can monitor (theoretically) unlimited PERF_TYPE_SOFTWARE events concurrently but limited (without multiplexing) PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events concurrently since each event is measured by the limited (as can be seen on the manual above) number of counters of the CPU's PMU.

So for a quad-core Kaby Lake CPU with HyperThreading enabled I assume that up to 4 PERF_TYPE_HARDWARE/PERF_TYPE_HW_CACHE events can be monitored concurrently (or up to 8 if only 4 threads are used).

Experimenting with the above assumptions I found out that while I can successfully monitor up to 4 PERF_TYPE_HARDWARE events (for 8 threads) this is not the case for PERF_TYPE_HW_CACHE events where only up to 2 events can be monitored concurrently!

I also tried to use only 4 threads but the upper limit of concurrently monitored 'PERF_TYPE_HARDWARE' events remains 4. The same is happening with HyperThreading disabled!

One could ask: why do you need to avoid multiplexing. First of all, the implementation needs to be as much accurate as possible by avoiding the potential blind spots of multiplexing and secondly when the "upper limit" is exceeded all event values are 0...

The PERF_TYPE_HW_CACHE events I'm targeting are:

```<code>CACHE_LLC_READ(PERF_HW_CACHE_TYPE_ID.PERF_COUNT_HW_CACHE_LL.value  | PERF_HW_CACHE_OP_ID.PERF_COUNT_HW_CACHE_OP_READ.value << 8 | PERF_HW_CACHE_OP_RESULT_ID.PERF_COUNT_HW_CACHE_RESULT_ACCESS.value << 16),
CACHE_LLC_WRITE(PERF_HW_CACHE_TYPE_ID.PERF_COUNT_HW_CACHE_LL.value  | PERF_HW_CACHE_OP_ID.PERF_COUNT_HW_CACHE_OP_WRITE.value << 8 | PERF_HW_CACHE_OP_RESULT_ID.PERF_COUNT_HW_CACHE_RESULT_ACCESS.value << 16),
CACHE_LLC_READ_MISS(PERF_HW_CACHE_TYPE_ID.PERF_COUNT_HW_CACHE_LL.value  | PERF_HW_CACHE_OP_ID.PERF_COUNT_HW_CACHE_OP_READ.value << 8 | PERF_HW_CACHE_OP_RESULT_ID.PERF_COUNT_HW_CACHE_RESULT_MISS.value << 16),
CACHE_LLC_WRITE_MISS(PERF_HW_CACHE_TYPE_ID.PERF_COUNT_HW_CACHE_LL.value  | PERF_HW_CACHE_OP_ID.PERF_COUNT_HW_CACHE_OP_WRITE.value << 8 | PERF_HW_CACHE_OP_RESULT_ID.PERF_COUNT_HW_CACHE_RESULT_MISS.value << 16),
</code>```

all are implemented with the provided formula:

```<code>(perf_hw_cache_id) | (perf_hw_cache_op_id << 8) |
(perf_hw_cache_op_result_id << 16)
</code>```

and are manipulated as a group (the first is the group leader etc).

So, my questions are the following:

Which counters of the PMU are used for PERF_TYPE_HARDWARE and which for PERF_TYPE_HW_CACHE events and where can I find this information?

What is the difference between the PERF_TYPE_HARDWARE pre-defined events (such as PERF_COUNT_HW_CACHE_MISSES) and the PERF_TYPE_HW_CACHE events?

Any suggestions on how to monitor without multiplexing all listed PERF_TYPE_HW_CACHE events?

Any suggestions on how to monitor without multiplexing up to 8 PERF_TYPE_HARDWARE or/and PERF_TYPE_HW_CACHE events?

Thanks in advance!

答:
The PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events are mapped to two sets of registers involved in performance monitoring. The first set of MSRs are called IA32_PERFEVTSELx where x can vary from 0 to N-1, N being the total number of general purpose counters available. The PERFEVTSEL is a short for "performance event select", they specify various conditions on the fulfillment of which event counting will happen. The second set of MSRs are called IA32_PMCx, where x varies similarly as PERFEVTSEL. These PMC registers store the counts of performance monitoring events. Each PERFEVTSEL register is paired with a corresponding PMC register.

The mapping happens as follows-

At the initialization of the architecture specific portion of the kernel, a pmu for measuring hardware specific events is registered here with type PERF_TYPE_RAW. All PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events are mapped to PERF_TYPE_RAW events to identify the pmu, as can be seen here.

```<code>if (type == PERF_TYPE_HARDWARE || type == PERF_TYPE_HW_CACHE)
        type = PERF_TYPE_RAW;
</code>```

The same architecture specific initialization is responsible for setting up the addresses of the first/base registers of each of the aforementioned sets of performance monitoring event registers, here

```<code>    .eventsel       = MSR_ARCH_PERFMON_EVENTSEL0,
    .perfctr        = MSR_ARCH_PERFMON_PERFCTR0,
</code>```

The event_init function specific to the PMU identified, is responsible for setting up and "reserving" the two sets of performance monitoring registers, as well as checking for event constraints etc., here. The reservation happens here.

```<code>for (i = 0; i < x86_pmu.num_counters; i++) {
        if (!reserve_perfctr_nmi(x86_pmu_event_addr(i)))
            goto perfctr_fail;
    }

    for (i = 0; i < x86_pmu.num_counters; i++) {
        if (!reserve_evntsel_nmi(x86_pmu_config_addr(i)))
            goto eventsel_fail;
    }
</code>```

The value num_counters = number of general-purpose counters as identified by CPUID instruction.

In addition to this, there are a couple of extra registers that monitor offcore events (eg. the LLC-cache specific events).

In later versions of architectural performance monitoring, some of the hardware events are measured with the help of fixed-purpose registers, as seen here. These are the fixed-purpose registers -

```<code>#define MSR_ARCH_PERFMON_FIXED_CTR0 0x309
#define MSR_ARCH_PERFMON_FIXED_CTR1 0x30a
#define MSR_ARCH_PERFMON_FIXED_CTR2 0x30b
</code>```

The PERF_TYPE_HARDWARE pre-defined events are all architectural performance monitoring events. These events are architectural, since the behavior of each architectural performance event is expected to be consistent on all processors that support that event. All of the PERF_TYPE_HW_CACHE events are non-architectural, which means they are model-specific and may vary from one family of processors to another.

For an Intel Kaby Lake machine that I have, a total of 20 PERF_TYPE_HW_CACHE events are pre-defined. The event constraints involved, ensure that the 3 fixed-function counters available are mapped to 3 PERF_TYPE_HARDWARE architectural events. Only one event can be measured on each of the fixed-function counters, so we can discard them for our analysis. The other constraint is that only two events targeting the LLC-caches, can be measured at the same time, since there are only two OFFCORE RESPONSE registers. Also, the nmi-watchdog may pin an event to another counter from the family of general-purpose counters. If the nmi-watchdog is disabled, we are left with 4 general purpose counters.

Given the constraints involved, and the limited number of counters available, there is just no way to avoid multiplexing if all the 20 hardware cache events are measured at the same time. Some workarounds to measure all the events, without incurring multiplexing and its errors, are -

3.1.  Group all the PERF_TYPE_HW_CACHE events into groups of 4, such that all of the 4 events can be scheduled on each of the 4 general-purpose counters at the same time. Make sure there are no more than 2 LLC cache events in a group. Run the same profile and obtain the counts for each of the groups separately.

3.2.  If all the PERF_TYPE_HW_CACHE events are to be monitored at the same time, then some of the errors with multiplexing can be reduced, by decreasing the value of perf_event_mux_interval_ms. It can be configured via a sysfs entry called /sys/devices/cpu/perf_event_mux_interval_ms. This value cannot be lowered beyond a point, as can be seen here.

Monitoring upto 8 hardware or hardware-cache events would require hyperthreading to be disabled. Note that, the information about the number of general purpose counters available are retrieved using the CPUID instruction and the number of such counters are setup at the architecture initialization portion of the kernel startup via the early_initcall function. This can be seen here. Once the initialization is done, the kernel understands that only 4 counters are available, and any changes in hyperthreading capabilities later, do not make any difference.

Body:
I'm working on a custom implementation on top of perf_event_open syscall.

The implementation aims to support various of PERF_TYPE_HARDWARE, PERF_TYPE_SOFTWARE and PERF_TYPE_HW_CACHE events for specific threads on any core.

In Intel® 64 and IA-32 Architectures Software Developer’s Manual vol 3B I see the following for my testing CPU (Kaby Lake):

To my understanding so far, one can monitor (theoretically) unlimited PERF_TYPE_SOFTWARE events concurrently but limited (without multiplexing) PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events concurrently since each event is measured by the limited (as can be seen on the manual above) number of counters of the CPU's PMU.

So for a quad-core Kaby Lake CPU with HyperThreading enabled I assume that up to 4 PERF_TYPE_HARDWARE/PERF_TYPE_HW_CACHE events can be monitored concurrently (or up to 8 if only 4 threads are used).

Experimenting with the above assumptions I found out that while I can successfully monitor up to 4 PERF_TYPE_HARDWARE events (for 8 threads) this is not the case for PERF_TYPE_HW_CACHE events where only up to 2 events can be monitored concurrently!

I also tried to use only 4 threads but the upper limit of concurrently monitored 'PERF_TYPE_HARDWARE' events remains 4. The same is happening with HyperThreading disabled!

One could ask: why do you need to avoid multiplexing. First of all, the implementation needs to be as much accurate as possible by avoiding the potential blind spots of multiplexing and secondly when the "upper limit" is exceeded all event values are 0...

The PERF_TYPE_HW_CACHE events I'm targeting are:

CACHE_LLC_READ(PERF_HW_CACHE_TYPE_ID.PERF_COUNT_HW_CACHE_LL.value  | PERF_HW_CACHE_OP_ID.PERF_COUNT_HW_CACHE_OP_READ.value << 8 | PERF_HW_CACHE_OP_RESULT_ID.PERF_COUNT_HW_CACHE_RESULT_ACCESS.value << 16),
CACHE_LLC_WRITE(PERF_HW_CACHE_TYPE_ID.PERF_COUNT_HW_CACHE_LL.value  | PERF_HW_CACHE_OP_ID.PERF_COUNT_HW_CACHE_OP_WRITE.value << 8 | PERF_HW_CACHE_OP_RESULT_ID.PERF_COUNT_HW_CACHE_RESULT_ACCESS.value << 16),
CACHE_LLC_READ_MISS(PERF_HW_CACHE_TYPE_ID.PERF_COUNT_HW_CACHE_LL.value  | PERF_HW_CACHE_OP_ID.PERF_COUNT_HW_CACHE_OP_READ.value << 8 | PERF_HW_CACHE_OP_RESULT_ID.PERF_COUNT_HW_CACHE_RESULT_MISS.value << 16),
CACHE_LLC_WRITE_MISS(PERF_HW_CACHE_TYPE_ID.PERF_COUNT_HW_CACHE_LL.value  | PERF_HW_CACHE_OP_ID.PERF_COUNT_HW_CACHE_OP_WRITE.value << 8 | PERF_HW_CACHE_OP_RESULT_ID.PERF_COUNT_HW_CACHE_RESULT_MISS.value << 16),

all are implemented with the provided formula:

(perf_hw_cache_id) | (perf_hw_cache_op_id << 8) |
(perf_hw_cache_op_result_id << 16)

and are manipulated as a group (the first is the group leader etc).

So, my questions are the following:

Which counters of the PMU are used for PERF_TYPE_HARDWARE and which for PERF_TYPE_HW_CACHE events and where can I find this information?

What is the difference between the PERF_TYPE_HARDWARE pre-defined events (such as PERF_COUNT_HW_CACHE_MISSES) and the PERF_TYPE_HW_CACHE events?

Any suggestions on how to monitor without multiplexing all listed PERF_TYPE_HW_CACHE events?

Any suggestions on how to monitor without multiplexing up to 8 PERF_TYPE_HARDWARE or/and PERF_TYPE_HW_CACHE events?

Thanks in advance!

Body:
The PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events are mapped to two sets of registers involved in performance monitoring. The first set of MSRs are called IA32_PERFEVTSELx where x can vary from 0 to N-1, N being the total number of general purpose counters available. The PERFEVTSEL is a short for "performance event select", they specify various conditions on the fulfillment of which event counting will happen. The second set of MSRs are called IA32_PMCx, where x varies similarly as PERFEVTSEL. These PMC registers store the counts of performance monitoring events. Each PERFEVTSEL register is paired with a corresponding PMC register.

The mapping happens as follows-

At the initialization of the architecture specific portion of the kernel, a pmu for measuring hardware specific events is registered here with type PERF_TYPE_RAW. All PERF_TYPE_HARDWARE and PERF_TYPE_HW_CACHE events are mapped to PERF_TYPE_RAW events to identify the pmu, as can be seen here.

if (type == PERF_TYPE_HARDWARE || type == PERF_TYPE_HW_CACHE)
        type = PERF_TYPE_RAW;

The same architecture specific initialization is responsible for setting up the addresses of the first/base registers of each of the aforementioned sets of performance monitoring event registers, here

    .eventsel       = MSR_ARCH_PERFMON_EVENTSEL0,
    .perfctr        = MSR_ARCH_PERFMON_PERFCTR0,

The event_init function specific to the PMU identified, is responsible for setting up and "reserving" the two sets of performance monitoring registers, as well as checking for event constraints etc., here. The reservation happens here.

for (i = 0; i < x86_pmu.num_counters; i++) {
        if (!reserve_perfctr_nmi(x86_pmu_event_addr(i)))
            goto perfctr_fail;
    }

    for (i = 0; i < x86_pmu.num_counters; i++) {
        if (!reserve_evntsel_nmi(x86_pmu_config_addr(i)))
            goto eventsel_fail;
    }

The value num_counters = number of general-purpose counters as identified by CPUID instruction.

In addition to this, there are a couple of extra registers that monitor offcore events (eg. the LLC-cache specific events).

In later versions of architectural performance monitoring, some of the hardware events are measured with the help of fixed-purpose registers, as seen here. These are the fixed-purpose registers -

#define MSR_ARCH_PERFMON_FIXED_CTR0 0x309
#define MSR_ARCH_PERFMON_FIXED_CTR1 0x30a
#define MSR_ARCH_PERFMON_FIXED_CTR2 0x30b

The PERF_TYPE_HARDWARE pre-defined events are all architectural performance monitoring events. These events are architectural, since the behavior of each architectural performance event is expected to be consistent on all processors that support that event. All of the PERF_TYPE_HW_CACHE events are non-architectural, which means they are model-specific and may vary from one family of processors to another.

For an Intel Kaby Lake machine that I have, a total of 20 PERF_TYPE_HW_CACHE events are pre-defined. The event constraints involved, ensure that the 3 fixed-function counters available are mapped to 3 PERF_TYPE_HARDWARE architectural events. Only one event can be measured on each of the fixed-function counters, so we can discard them for our analysis. The other constraint is that only two events targeting the LLC-caches, can be measured at the same time, since there are only two OFFCORE RESPONSE registers. Also, the nmi-watchdog may pin an event to another counter from the family of general-purpose counters. If the nmi-watchdog is disabled, we are left with 4 general purpose counters.

Given the constraints involved, and the limited number of counters available, there is just no way to avoid multiplexing if all the 20 hardware cache events are measured at the same time. Some workarounds to measure all the events, without incurring multiplexing and its errors, are -

3.1.  Group all the PERF_TYPE_HW_CACHE events into groups of 4, such that all of the 4 events can be scheduled on each of the 4 general-purpose counters at the same time. Make sure there are no more than 2 LLC cache events in a group. Run the same profile and obtain the counts for each of the groups separately.

3.2.  If all the PERF_TYPE_HW_CACHE events are to be monitored at the same time, then some of the errors with multiplexing can be reduced, by decreasing the value of perf_event_mux_interval_ms. It can be configured via a sysfs entry called /sys/devices/cpu/perf_event_mux_interval_ms. This value cannot be lowered beyond a point, as can be seen here.

Monitoring upto 8 hardware or hardware-cache events would require hyperthreading to be disabled. Note that, the information about the number of general purpose counters available are retrieved using the CPUID instruction and the number of such counters are setup at the architecture initialization portion of the kernel startup via the early_initcall function. This can be seen here. Once the initialization is done, the kernel understands that only 4 counters are available, and any changes in hyperthreading capabilities later, do not make any difference.


问:
R: projection robin in tmap
I am trying to create map with projection "robin", but I get an error:

```<code>library(sp)
library(tmap)
data(World)
tm_shape(World, projection = "robin")+tm_borders()+tm_fill(col = 'pop_est_dens')+tm_grid(n.x = 11, n.y = 11)
Error in st_crs.character(x[[shape.id[masterID]]]$projection) :
  invalid crs: robin
</code>```

What shall I do to get the desired projection?

答:
You may consider using the PROJ4 string to specify the projection; this can be found with a little googling

```<code>library(sp)
library(tmap)
data(World)

robin <- "+proj=robin +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs"

tm_shape(World, projection = robin) + tm_borders() +
  tm_fill(col = 'pop_est_dens') +
  tm_grid(n.x = 11, n.y = 11)
</code>```

The map itself seems to be projected okay (Alaska and Australia look very robinsonish) but the grid does not seem quite right I am afraid...

Title: R: projection robin in tmap

Body:
I am trying to create map with projection "robin", but I get an error:

library(sp)
library(tmap)
data(World)
tm_shape(World, projection = "robin")+tm_borders()+tm_fill(col = 'pop_est_dens')+tm_grid(n.x = 11, n.y = 11)
Error in st_crs.character(x[[shape.id[masterID]]]$projection) :
  invalid crs: robin

What shall I do to get the desired projection?

Body:
You may consider using the PROJ4 string to specify the projection; this can be found with a little googling

library(sp)
library(tmap)
data(World)

robin <- "+proj=robin +lon_0=0 +x_0=0 +y_0=0 +ellps=WGS84 +datum=WGS84 +units=m +no_defs"

tm_shape(World, projection = robin) + tm_borders() +
  tm_fill(col = 'pop_est_dens') +
  tm_grid(n.x = 11, n.y = 11)

The map itself seems to be projected okay (Alaska and Australia look very robinsonish) but the grid does not seem quite right I am afraid...


问:
Using JQuery Sortable toArray with connectWith; how to get IDs of all items?
Given n lists of m items that are Sortable across lists, how can I get an overall list of IDs for all items in all lists from top to bottom (including both sort participants and non-participants)?

My list of items looks something like this:

```<code>Category 1
- (id=a) Item 1A
- (id=b) Item 1B
Category 2
- (id=c) Item 2A
- (id=d) Item 2B
Category 3
- (id=e) Item 3A
- (id=f) Item 3B
</code>```

These items are Sortable across categories and connected via connectWith (see below for code).  Item 2B, for example, may be dragged and dropped into Category 1.

I can get as far as using update and toArray to get IDs for the section being dragged from and the section being dragged to, but is there a clean way to access all of them?  Do I need to build it manually somehow? How can I get a list of all the IDs in their current sort order?

If I drag Item 2B to the first position in Category 1, for example, I can get two array lists for two fires of the update, the first containing "d" (the list I moved from) and the second containing "c, a, b" (the list I moved to).  How can I get a list containing "c, a, b, d, e, f"?  This is happening on a button click, so I'm not restricted to update, if there's a better mechanism through which to get the information.

Here's a JFIDDLE demo of what I have so far.

答: There doesn't seem to be a native function to do this, so I created a demo that iterates all the .sortable elements to create one array and log to the console in Chrome or Firefox with Firebug. I also changed the update to receive so it only fires once (see jQueryUI sortable documentation).

ContentLicense: CC BY-SA 3.0

Body:
Given n lists of m items that are Sortable across lists, how can I get an overall list of IDs for all items in all lists from top to bottom (including both sort participants and non-participants)?

My list of items looks something like this:

Category 1
- (id=a) Item 1A
- (id=b) Item 1B
Category 2
- (id=c) Item 2A
- (id=d) Item 2B
Category 3
- (id=e) Item 3A
- (id=f) Item 3B

These items are Sortable across categories and connected via connectWith (see below for code).  Item 2B, for example, may be dragged and dropped into Category 1.

I can get as far as using update and toArray to get IDs for the section being dragged from and the section being dragged to, but is there a clean way to access all of them?  Do I need to build it manually somehow? How can I get a list of all the IDs in their current sort order?

If I drag Item 2B to the first position in Category 1, for example, I can get two array lists for two fires of the update, the first containing "d" (the list I moved from) and the second containing "c, a, b" (the list I moved to).  How can I get a list containing "c, a, b, d, e, f"?  This is happening on a button click, so I'm not restricted to update, if there's a better mechanism through which to get the information.

Here's a JFIDDLE demo of what I have so far.


问:
How to pass extra context to Django Rest Framework serializers
It is my first project using Django rest framework and i'm struggling to get this right. I know that in mainstream Django framework, if i need to add extra contexts to a class-based view, i will do something like this:

```<code>class PostDetail(DetailView):
    model = Post

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        # Add in a QuerySet of all the books by a certain author John
        context['john_books'] = Book.objects.filter(author=john)
        return context
</code>```

then, i will be able to access the context 'john_books' in my template. Now i need to do same by passing extra contexts to my PostViewSets. On the detail view, i want to access the list of post authored by that post author in my api endpoint (something like 'Posts from the same author'). I have read about get_serializer_context but still can't figure out how to implement it. This is what i have so far:

```<code>class PostViewSets(viewsets.ModelViewSet):
    queryset = Post.objects.all()
    serializer_class = PostSerializer

     def get_serializer_context(self):
        context = super(PostViewSets, self).get_serializer_context()
        author = self.get_object.author
        author_posts = self.get_queryset().filter(author=author)
        context.update({'author_posts': author_posts})
        return context
</code>```

i get this error:

AttributeError at /posts/ 'function' object has no attribute 'author'

My Post Model:

```<code>class Post(models.Model):
    title = models.CharField(max_length=100, unique=True)
    body = models.TextField()
    is_featured = models.BooleanField(default=True)
    viewcount = models.IntegerField(default=0)
    author = models.ForeignKey(User, on_delete=models.CASCADE)
    created = models.DateTimeField(auto_now_add=True)
</code>```

and my PostSerializer class:

```<code>class PostSerializer(serializers.ModelSerializer):
    author = UserSerializer()

    class Meta:
        model = Post
        fields = ['id', 'title', 'body', 'author', 'viewcount', 'is_featured', 'created']
</code>```

答:
1. The Error Message

AttributeError at /posts/ 'function' object has no attribute 'author'

explicitly explains what and where is the problem:

author = self.get_object.author

I guess you tried to do something like this

```<code>author = self.get_object().author
</code>```

2. A DRF ViewSet

responses with data serialized by corresponding Serializer. So you don't need to change the ViewSet, but update the Serializer with something like:

```<code>class PostListSerializer(serializers.ModelSerializer):
    ... some fields ...

    class Meta:
        model = Post
        fields = [ ... some fields ... ]

class PostDetailsSerializer(serializers.ModelSerializer):
    ... some fields ...

    author_posts = PostListSerializer(source="author.post_set", many=True, read_only=True)

    class Meta:
        model = Post
        fields = [ ... some fields ... , 'author_posts']
</code>```

or with SerializerMethodField

```<code>    class PostDetailsSerializer(serializers.ModelSerializer):
    ... some fields ...

    author_posts = serializers.SerializerMethodField()

    def get_author_posts(self, obj):
        return PostListSerializer(instance=obj.post_set.all(), many=True).data

    class Meta:
        model = Post
        fields = [ ... some fields ... , 'author_posts']
</code>```

I didn't try this exact code, but this is the main idea.

Body:
It is my first project using Django rest framework and i'm struggling to get this right. I know that in mainstream Django framework, if i need to add extra contexts to a class-based view, i will do something like this:

class PostDetail(DetailView):
    model = Post

    def get_context_data(self, **kwargs):
        context = super().get_context_data(**kwargs)
        # Add in a QuerySet of all the books by a certain author John
        context['john_books'] = Book.objects.filter(author=john)
        return context

then, i will be able to access the context 'john_books' in my template. Now i need to do same by passing extra contexts to my PostViewSets. On the detail view, i want to access the list of post authored by that post author in my api endpoint (something like 'Posts from the same author'). I have read about get_serializer_context but still can't figure out how to implement it. This is what i have so far:

class PostViewSets(viewsets.ModelViewSet):
    queryset = Post.objects.all()
    serializer_class = PostSerializer

     def get_serializer_context(self):
        context = super(PostViewSets, self).get_serializer_context()
        author = self.get_object.author
        author_posts = self.get_queryset().filter(author=author)
        context.update({'author_posts': author_posts})
        return context

i get this error:

AttributeError at /posts/ 'function' object has no attribute 'author'

My Post Model:

class Post(models.Model):
    title = models.CharField(max_length=100, unique=True)
    body = models.TextField()
    is_featured = models.BooleanField(default=True)
    viewcount = models.IntegerField(default=0)
    author = models.ForeignKey(User, on_delete=models.CASCADE)
    created = models.DateTimeField(auto_now_add=True)

and my PostSerializer class:

class PostSerializer(serializers.ModelSerializer):
    author = UserSerializer()

    class Meta:
        model = Post
        fields = ['id', 'title', 'body', 'author', 'viewcount', 'is_featured', 'created']

Body:
1. The Error Message

AttributeError at /posts/ 'function' object has no attribute 'author'

explicitly explains what and where is the problem:

author = self.get_object.author

I guess you tried to do something like this

author = self.get_object().author

2. A DRF ViewSet

responses with data serialized by corresponding Serializer. So you don't need to change the ViewSet, but update the Serializer with something like:

class PostListSerializer(serializers.ModelSerializer):
    ... some fields ...

    class Meta:
        model = Post
        fields = [ ... some fields ... ]

class PostDetailsSerializer(serializers.ModelSerializer):
    ... some fields ...

    author_posts = PostListSerializer(source="author.post_set", many=True, read_only=True)

    class Meta:
        model = Post
        fields = [ ... some fields ... , 'author_posts']

or with SerializerMethodField

    class PostDetailsSerializer(serializers.ModelSerializer):
    ... some fields ...

    author_posts = serializers.SerializerMethodField()

    def get_author_posts(self, obj):
        return PostListSerializer(instance=obj.post_set.all(), many=True).data

    class Meta:
        model = Post
        fields = [ ... some fields ... , 'author_posts']

I didn't try this exact code, but this is the main idea.


问:
Problem comparing a tensor with binary values
I have a problem where there is an image and a given question. The answer is of the form [False, True]. During training I try to find whether the prediction is right or wrong.

def build_loss(logits, labels):
    # Cross-entropy loss
    loss = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels)

    # Classification accuracy
    correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(labels, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    return tf.reduce_mean(loss), accuracy

logits = C(self.img, self.q, scope='Classifier')
self.all_preds = tf.nn.softmax(logits)
self.loss, self.accuracy = build_loss(logits, self.a)
```

Assuming the original answer is [0, 0] and the predicted answer is [0, 1], the tf.argmax for both, logits and labels would return a value of 1. Is there a way to avoid this ugly comparison? I tried replacing tf.argmax with tf.reduce_max but then accuracy always comes to be 0.

Or is there any other way to change the model, so that, rather than having 2 neurons in output layer I could get it done in 1 neuron. Currently, I am using tf.nn.softmax for finding the predictions and tf.nn.sigmoid_cross_entropy_with_logits as my loss function.

答:
Yes, you can use only one neuron at the end and use sigmoid activation in it.</p>

```<code>model.add(Dense(1, activation='sigmoid'))
</code>```

Body:
I have a problem where there is an image and a given question. The answer is of the form [False, True]. During training I try to find whether the prediction is right or wrong.

def build_loss(logits, labels):
    # Cross-entropy loss
    loss = tf.nn.sigmoid_cross_entropy_with_logits(logits=logits, labels=labels)

    # Classification accuracy
    correct_prediction = tf.equal(tf.argmax(logits, 1), tf.argmax(labels, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
    return tf.reduce_mean(loss), accuracy

logits = C(self.img, self.q, scope='Classifier')
self.all_preds = tf.nn.softmax(logits)
self.loss, self.accuracy = build_loss(logits, self.a)

Assuming the original answer is [0, 0] and the predicted answer is [0, 1], the tf.argmax for both, logits and labels would return a value of 1. Is there a way to avoid this ugly comparison? I tried replacing tf.argmax with tf.reduce_max but then accuracy always comes to be 0.

Or is there any other way to change the model, so that, rather than having 2 neurons in output layer I could get it done in 1 neuron. Currently, I am using tf.nn.softmax for finding the predictions and tf.nn.sigmoid_cross_entropy_with_logits as my loss function.

Body:
Yes, you can use only one neuron at the end and use sigmoid activation in it.

model.add(Dense(1, activation='sigmoid'))


问:
Scheduler with blocking=False
The Python scheduler works well with default value blocking=True:

```<code>import sched, time
def print_time(a):
    print("From print_time", time.time(), a)
s = sched.scheduler()
s.enter(2, priority=1, action=print_time, argument=(1,))
s.enter(2, priority=2, action=print_time, argument=(2,))
print(time.time())
s.run()  # blocks until the last scheduled task is finished
print('finished')
</code>```

but I can't make it work with blocking=False.

Indeed with:

```<code>s.run(block=False)
print('finished')
</code>```

the script stops immediately so obviously it does not work (this is normal behaviour). But with:

```<code>s.run(block=False)
time.sleep(10)     # sleeping to make sure the script does not terminate immediately,
print('finished')  # but in real situation, there would be other tasks here
</code>```

it does not work either: the tasks are strangely not executed.

I don't really understand the documentation:

If blocking is false executes the scheduled events due to expire soonest (if any) and then return the deadline of the next scheduled call in the scheduler (if any).

How to use scheduler in blocking=False mode?

答:
I found the solution: the blocking=False mode allows to do other things in the meantime and poll the new events every now and then. This works:

```<code>import sched, time

def print_time(a):
    print("From print_time", time.time(), a)

s = sched.scheduler()
s.enter(2, priority=1, action=print_time, argument=(1,))
s.enter(2, priority=2, action=print_time, argument=(2,))

for _ in range(50):
   s.run(blocking=False)
   time.sleep(0.1)  # in a real situation, this would be some other tasks

print('finished')
</code>```

Title: Scheduler with blocking=False

Body:
The Python scheduler works well with default value blocking=True:

import sched, time
def print_time(a):
    print("From print_time", time.time(), a)
s = sched.scheduler()
s.enter(2, priority=1, action=print_time, argument=(1,))
s.enter(2, priority=2, action=print_time, argument=(2,))
print(time.time())
s.run()  # blocks until the last scheduled task is finished
print('finished')

but I can't make it work with blocking=False.

Indeed with:

s.run(block=False)
print('finished')

the script stops immediately so obviously it does not work (this is normal behaviour). But with:

s.run(block=False)
time.sleep(10)     # sleeping to make sure the script does not terminate immediately,
print('finished')  # but in real situation, there would be other tasks here

it does not work either: the tasks are strangely not executed.

I don't really understand the documentation:

If blocking is false executes the scheduled events due to expire soonest (if any) and then return the deadline of the next scheduled call in the scheduler (if any).

How to use scheduler in blocking=False mode?

Body:
I found the solution: the blocking=False mode allows to do other things in the meantime and poll the new events every now and then. This works:

import sched, time

def print_time(a):
    print("From print_time", time.time(), a)

s = sched.scheduler()
s.enter(2, priority=1, action=print_time, argument=(1,))
s.enter(2, priority=2, action=print_time, argument=(2,))

for _ in range(50):
   s.run(blocking=False)
   time.sleep(0.1)  # in a real situation, this would be some other tasks

print('finished')


问:
How to efficiently XML parse in R
I use R to parse XML data from a website. I have list of 20,000 rows with URLs from which I need to extract data. I have a code which gets the job done using a for loop, but it's very slow (takes approx. 12 hours). I thought of using parallel processing (I have access to several CPUs) to speed it up, but I cannot make it work properly. Would it be more efficient using a data table instead of a data frame? Is there any way to speed the process up? Thanks!</p>

```<code>for (i in  1:nrow(list)) {
  t <- xmlToDataFrame(xmlParse(read_xml(list$path[i]))) #Read the data into a file
  t$ID <- list$ID[i]
  emptyDF <- bind_rows(all, t) #Bind all into one file
  if (i / 10 == floor(i / 10)) {
    print(i)
  }  #print every 10th value to monitor progress of the loop
}
</code>```

答:
This script should point you in the correct direction:

```<code>t<-list()

for (i in  1:nrow(list)) {
   tempdf <- xmlToDataFrame(xmlParse(list$path[i])) #Read the data into a file
   tempdf$ID <- list$ID[i]
   t[[i]]<-tempdf

   if (i %% 10 == 0) {
      print(i)
   }  #print every 10th value to monitor progress of the loop
}

answer <- bind_rows(t) #Bind all into one file
</code>```

Instead of a for loop, an lapply would also work here.  Without any sample data, this is untested.

Title: How to efficiently XML parse in R

Body:
I use R to parse XML data from a website. I have list of 20,000 rows with URLs from which I need to extract data. I have a code which gets the job done using a for loop, but it's very slow (takes approx. 12 hours). I thought of using parallel processing (I have access to several CPUs) to speed it up, but I cannot make it work properly. Would it be more efficient using a data table instead of a data frame? Is there any way to speed the process up? Thanks!

for (i in  1:nrow(list)) {
  t <- xmlToDataFrame(xmlParse(read_xml(list$path[i]))) #Read the data into a file
  t$ID <- list$ID[i]
  emptyDF <- bind_rows(all, t) #Bind all into one file
  if (i / 10 == floor(i / 10)) {
    print(i)
  }  #print every 10th value to monitor progress of the loop
}

Body:
This script should point you in the correct direction:

t<-list()

for (i in  1:nrow(list)) {
   tempdf <- xmlToDataFrame(xmlParse(list$path[i])) #Read the data into a file
   tempdf$ID <- list$ID[i]
   t[[i]]<-tempdf

   if (i %% 10 == 0) {
      print(i)
   }  #print every 10th value to monitor progress of the loop
}

answer <- bind_rows(t) #Bind all into one file

Instead of a for loop, an lapply would also work here.  Without any sample data, this is untested.


问:
Swift 4 How to get all events from calendar?
I am using Swift 4.1. And I want to write a function which will collect all events from all calendars in Calendar app of iOS. Thanks to this answer on stackoverflow: How to get all Events out of a Calendar (Swift) I was able to write my own class and call it Cale. Please, look at this:

```<code>import UIKit
import EventKit

class Cale {

    private func createDate(year: Int) -> Date {
        var components = DateComponents()
        components.year = year
        components.timeZone = TimeZone(secondsFromGMT: 0)

        return Calendar.current.date(from: components)!
    }

    private let eventStore = EKEventStore()

    private func get() {
        let calendars = eventStore.calendars(for: .event)

        for calendar in calendars {
            // This checking will remove Birthdays and Hollidays callendars
            guard calendar.allowsContentModifications else {
                continue
            }

            let start = createDate(year: 2016)
            let end = createDate(year: 2025)

            print("start: \(start)")
            print("  end: \(end)")

            let predicate = eventStore.predicateForEvents(withStart: start, end: end, calendars: [calendar])

            print("predicate: \(predicate)")

            let events = eventStore.events(matching: predicate)

            for event in events {
                print("    title: \(event.title!)")
                print("startDate: \(event.startDate!)")
                print("  endDate: \(event.endDate!)")
            }
        }
    }

    func checkStatusAndGetAllEvents() {
        let currentStatus = EKEventStore.authorizationStatus(for: EKEntityType.event)

        switch currentStatus {
        case .authorized:
            //print("authorized")
            self.get()
        case .notDetermined:
            //print("notDetermined")
            eventStore.requestAccess(to: .event) { accessGranted, error in
                if accessGranted {
                    self.get()
                } else {
                    print("Change Settings to Allow Access")
                }
            }
        case .restricted:
            print("restricted")
        case .denied:
            print("denied")
        }
    }
}
</code>```

Quite simple class, you can use it, it is working but with one exception.
The main function there is get() In this function I am creating predicate based on two dates: start and end. As you can see start date is:

2016-01-01 00:00:00 +0000

and end date is:

2025-01-01 00:00:00 +0000

But if we run the program we will see that predicate will be like this:

CADEventPredicate start:01/01/2016, 03:00; end:01/01/2020, 03:00;
  cals:(
      2 )

Only from 2016 to 2020, 4 years! I have tested it on different dates, but I could get predicate with 4 years interval maximum. It means, it will not give me all events! So question is: How to get all events from calendar? If it possible, without using dates!

Thank you for any future help or advice!

答:
In case anyone is still wondering, Apple limited this to four years intentionally.

From predicateForEvents(withStart:end:calendars:):

For performance reasons, this method matches only those events within a four year time span. If the date range between startDate and endDate is greater than four years, it is shortened to the first four years.

If you want a range longer than that, I guess you'll have to wrap this in your own function to split it into four year chunks.

Body:
I am using Swift 4.1. And I want to write a function which will collect all events from all calendars in Calendar app of iOS. Thanks to this answer on stackoverflow: How to get all Events out of a Calendar (Swift) I was able to write my own class and call it Cale. Please, look at this:

import UIKit
import EventKit

class Cale {

    private func createDate(year: Int) -> Date {
        var components = DateComponents()
        components.year = year
        components.timeZone = TimeZone(secondsFromGMT: 0)

        return Calendar.current.date(from: components)!
    }

    private let eventStore = EKEventStore()

    private func get() {
        let calendars = eventStore.calendars(for: .event)

        for calendar in calendars {
            // This checking will remove Birthdays and Hollidays callendars
            guard calendar.allowsContentModifications else {
                continue
            }

            let start = createDate(year: 2016)
            let end = createDate(year: 2025)

            print("start: \(start)")
            print("  end: \(end)")

            let predicate = eventStore.predicateForEvents(withStart: start, end: end, calendars: [calendar])

            print("predicate: \(predicate)")

            let events = eventStore.events(matching: predicate)

            for event in events {
                print("    title: \(event.title!)")
                print("startDate: \(event.startDate!)")
                print("  endDate: \(event.endDate!)")
            }
        }
    }

    func checkStatusAndGetAllEvents() {
        let currentStatus = EKEventStore.authorizationStatus(for: EKEntityType.event)

        switch currentStatus {
        case .authorized:
            //print("authorized")
            self.get()
        case .notDetermined:
            //print("notDetermined")
            eventStore.requestAccess(to: .event) { accessGranted, error in
                if accessGranted {
                    self.get()
                } else {
                    print("Change Settings to Allow Access")
                }
            }
        case .restricted:
            print("restricted")
        case .denied:
            print("denied")
        }
    }
}

Quite simple class, you can use it, it is working but with one exception.
The main function there is get() In this function I am creating predicate based on two dates: start and end. As you can see start date is:

2016-01-01 00:00:00 +0000

and end date is:

2025-01-01 00:00:00 +0000

But if we run the program we will see that predicate will be like this:

CADEventPredicate start:01/01/2016, 03:00; end:01/01/2020, 03:00;
  cals:(
      2 )

Only from 2016 to 2020, 4 years! I have tested it on different dates, but I could get predicate with 4 years interval maximum. It means, it will not give me all events! So question is: How to get all events from calendar? If it possible, without using dates!

Thank you for any future help or advice!


问:
Warning from gcc: braces around scalar initializer, how to fix?
At rewriting kernel driver I got this warning:

```<code>msm-cirrus-playback.c:545:2: warning: braces around scalar initializer
</code>```

Read that this warning appears when I am declaring one structure's field in {}:

```<code>struct random_struct test = {
    { .name = "StackOverflow" },
    { .name = "StackExchange" },
};
</code>```

But my structure have 2-3 fields in {}:

```<code>static struct device_attribute *opalum_dev_attr = {
    {
    .attr->name = "temp-acc",
    .show = opsl_temp_acc_show,
    .store = opsl_temp_acc_store,
    },
    {
    .attr->name = "count",
    .show = opsl_count_show,
    .store = opsl_count_store,
    },
    {
    .attr->name = "ambient",
    .show = opsl_ambient_show,
    .store = opsl_ambient_store,
    },
    {
    .attr->name = "f0",
    .show = opsl_f0_show,
    .store = opsl_f0_store,
    },
    {
    .attr->name = "pass",
    .show = opsl_pass_show,
    },
    {
    .attr->name = "start",
    .show = opsl_cali_start,
    },
};
</code>```

This structure:

```<code>struct device_attribute {
    struct attribute    attr;
    ssize_t (*show)(struct device *dev, struct device_attribute *attr,
            char *buf);
     ssize_t (*store)(struct device *dev, struct device_attribute *attr,
             const char *buf, size_t count);
};
</code>```

How can I fix this warning? Qualcomm kernels are building with -Werror flag, so this warning is critical.

答:
static struct device_attribute *opalum_dev_attr means declare opalum_dev_attr as static pointer to struct device_attribute

Your code is trying to initialize a static array of struct device_attribute

What you want is: static struct device_attribute opalum_dev_attr[] which means declare opalum_dev_attr as static array of struct device_attribute

Body:
At rewriting kernel driver I got this warning:

msm-cirrus-playback.c:545:2: warning: braces around scalar initializer

Read that this warning appears when I am declaring one structure's field in {}:

struct random_struct test = {
    { .name = "StackOverflow" },
    { .name = "StackExchange" },
};

But my structure have 2-3 fields in {}:

static struct device_attribute *opalum_dev_attr = {
    {
    .attr->name = "temp-acc",
    .show = opsl_temp_acc_show,
    .store = opsl_temp_acc_store,
    },
    {
    .attr->name = "count",
    .show = opsl_count_show,
    .store = opsl_count_store,
    },
    {
    .attr->name = "ambient",
    .show = opsl_ambient_show,
    .store = opsl_ambient_store,
    },
    {
    .attr->name = "f0",
    .show = opsl_f0_show,
    .store = opsl_f0_store,
    },
    {
    .attr->name = "pass",
    .show = opsl_pass_show,
    },
    {
    .attr->name = "start",
    .show = opsl_cali_start,
    },
};

This structure:

struct device_attribute {
    struct attribute    attr;
    ssize_t (*show)(struct device *dev, struct device_attribute *attr,
            char *buf);
     ssize_t (*store)(struct device *dev, struct device_attribute *attr,
             const char *buf, size_t count);
};

How can I fix this warning? Qualcomm kernels are building with -Werror flag, so this warning is critical.


问:
Drop rows within dataframe based on condition pandas python
I have a dataframe like this

```<code>import pandas as pd
data = {'Index Title'  : ["Company1", "Company1", "Company2", "Company3"],
    'BusinessType'     : ['Type 1', 'Type 2', 'Type 1', 'Type 2'],
    'ID1'     : ['123', '456', '789', '012']
        }
df = pd.DataFrame(data)
df.index = df["Index Title"]
del df["Index Title"]
print(df)
</code>```

Dataframe

where Index Title is a company name.
For Company 1 I have two types - Type 1 and Type 2.

For Company 2 I have only Type 1
And for Company 3 I have only Type 2.

I would like to drop those rows where there is only one type - Type 1 or Type 2.

So in this case it should drop Company 2 and Company 3.

Could you please help me what is the best way to do that?

答:
This is one way:
- you group by Index Title
- filter if there is at least one Type 1 & one Type 2

```<code>df = (
    df.groupby('Index Title')
    .filter(lambda x: (x['BusinessType']=='Type 1').any() &
                      (x['BusinessType']=='Type 2').any())
    .reset_index()
)
</code>```

Update if you are looking for two or more types regardless if they are Type 1 & Type 2

```<code>df = (
    df.groupby('Index Title')
    .filter(lambda x: x['BusinessType'].nunique() > 1)
    .reset_index()
)
</code>```

In this case @cs95's answer is the cleaner one, which you should use.

Body:
I have a dataframe like this

import pandas as pd
data = {'Index Title'  : ["Company1", "Company1", "Company2", "Company3"],
    'BusinessType'     : ['Type 1', 'Type 2', 'Type 1', 'Type 2'],
    'ID1'     : ['123', '456', '789', '012']
        }
df = pd.DataFrame(data)
df.index = df["Index Title"]
del df["Index Title"]
print(df)

Dataframe

where Index Title is a company name.
For Company 1 I have two types - Type 1 and Type 2.

For Company 2 I have only Type 1
And for Company 3 I have only Type 2.

I would like to drop those rows where there is only one type - Type 1 or Type 2.

So in this case it should drop Company 2 and Company 3.

Could you please help me what is the best way to do that?

Body:
This is one way:
- you group by Index Title
- filter if there is at least one Type 1 & one Type 2

df = (
    df.groupby('Index Title')
    .filter(lambda x: (x['BusinessType']=='Type 1').any() &
                      (x['BusinessType']=='Type 2').any())
    .reset_index()
)

Update if you are looking for two or more types regardless if they are Type 1 & Type 2

df = (
    df.groupby('Index Title')
    .filter(lambda x: x['BusinessType'].nunique() > 1)
    .reset_index()
)

In this case @cs95's answer is the cleaner one, which you should use.


问:
Multiple relations from many entities to one entitiy in SQL Server (EF core)
Can anybody help me? I want to set constraint to my database for example:

First entity is Triangle (Triangle_Id, Name, Side_size)

Second entity is Square (Square_Id, Name, Side_size)

Third entity is Area (Area_Id, ObjectId, Area_value)

Fourth entity is Circle (Circle_Id, Name, Radius)

How can I add constraints from entities Triangle and Square to Area that I am unable to write any other type of objects in ObjectId column except triangle or square?

I tried to write come FK constraints, but have had an error.

FK_Area_Triangle_ObjectId (area_ObjectId == triangle_Id)

FK_Area_Square_ObjectId (area_ObjectId == square_Id)

答:
Foreign key is defined between two tables. It cannot be made between a table and a view, which would solve your task directly.

What you can do, if to create a scalar function checking for records in both tables and returning 1 if found and 0, if not. And then put a constraint that the function should be 1.


问:
OS X: Determine Trash location for a given path
Simply moving the file to ~/.Trash/ will not work, as if the file os on an external drive, it will move the file to the main system drive..

Also, there are other conditions, like files on external drives get moved to /Volumes/.Trash/501/ (or whatever the current user's ID is)

Given a file or folder path, what is the correct way to determine the trash folder? I imagine the language is pretty irrelevant, but I intend to use Python

答:
A better way is NSWorkspaceRecycleOperation, which is one of the operations you can use with -[NSWorkspace performFileOperation:source:destination:files:tag:]. The constant's name is another artifact of Cocoa's NeXT heritage; its function is to move the item to the Trash.

Since it's part of Cocoa, it should be available to both Python and Ruby.

OwnerDisplayName: dbr

ContentLicense: CC BY-SA 2.5

OwnerDisplayName: Peter Hosey


问:
Can I open UNIX POSIX ports from Python?
Can I open UNIX POSIX ports from Python 2.7 ( I don't need IP port, just UNIX POSIX) ? Does anybody have experience with this ?</p>

答:
Yes, you can.</p>

```<code>s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(path)
</code>```

Body:
Yes, you can.

s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(path)


答:
How about this? For each line check that you get the correct sequence and then use rle.

```<code>xy <- read.table(text = "FALSE   TRUE
FALSE   TRUE
TRUE    TRUE
TRUE    TRUE
FALSE   FALSE
FALSE   FALSE
FALSE   TRUE
FALSE   TRUE
TRUE    TRUE
FALSE   TRUE", header = FALSE)
colnames(xy) <- c("Movement", "Booked")

out <- rle(apply(xy, MARGIN = 1, FUN = function(x) !isTRUE(x[1]) & isTRUE(x[2])))
out$lengths[out$values == TRUE]

[1] 2 2 1
</code>```

Another more simple option would be

```<code>out <- rle(xy$Movement == FALSE & xy$Booked == TRUE)
out$lengths[out$values == TRUE]
</code>```

Title: Get a vector with the count of consecutive specific values in R

Body:
I have a dataframe that looks like this:

    (TimeDate,Movement,Booked)

    1   2017-01-02 09:00:13 FALSE   TRUE
    2   2017-01-02 09:01:13 FALSE   TRUE
    3   2017-01-02 09:02:15 TRUE    TRUE
    4   2017-01-02 09:03:13 TRUE    TRUE
    5   2017-01-02 09:04:13 FALSE   FALSE
    6   2017-01-02 09:05:13 FALSE   FALSE
    7   2017-01-02 09:06:13 FALSE   TRUE
    8   2017-01-02 09:07:13 FALSE   TRUE
    9   2017-01-02 09:08:14 TRUE    TRUE
    10  2017-01-02 09:09:13 FALSE   TRUE

It has a minute by minute log with two booleans corresponding to movement detected and room booked. What I want to obtain is a vector that counts the consecutive occurrences of (FALSE,TRUE) for Movement and Booked comumns, so in this case I would want a resulting vector like this one:

    c(2,2,1)

Where the first element is the count of lines 1 and 2, the second for lines 7 and 8 and the last one for line 10.

Honestly I don't even know how to start with this.
Thanks!

Body:
How about this? For each line check that you get the correct sequence and then use rle.

xy <- read.table(text = "FALSE   TRUE
FALSE   TRUE
TRUE    TRUE
TRUE    TRUE
FALSE   FALSE
FALSE   FALSE
FALSE   TRUE
FALSE   TRUE
TRUE    TRUE
FALSE   TRUE", header = FALSE)
colnames(xy) <- c("Movement", "Booked")

out <- rle(apply(xy, MARGIN = 1, FUN = function(x) !isTRUE(x[1]) & isTRUE(x[2])))
out$lengths[out$values == TRUE]

[1] 2 2 1

Another more simple option would be

out <- rle(xy$Movement == FALSE & xy$Booked == TRUE)
out$lengths[out$values == TRUE]


答:
I assume you are talking about Unix domain sockets (I'm not sure what you mean by "UNIX POSIX port"... IP sockets have ports, Unix sockets don't).  The standard system calls are available through a thin wrapper.

```<code>import socket
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.bind("/path/to/socket")
s.listen(100)
while True:
    conn, addr = s.accept()
    do_something(conn, addr)
</code>```

Or for connecting to an existing Unix socket...

```<code>s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect("/path/to/socket")
</code>```

Title: Can I open UNIX POSIX ports from Python?

Body: Can I open UNIX POSIX ports from Python 2.7 ( I don't need IP port, just UNIX POSIX) ? Does anybody have experience with this ?

Body:
I assume you are talking about Unix domain sockets (I'm not sure what you mean by "UNIX POSIX port"... IP sockets have ports, Unix sockets don't).  The standard system calls are available through a thin wrapper.

import socket
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.bind("/path/to/socket")
s.listen(100)
while True:
    conn, addr = s.accept()
    do_something(conn, addr)

Or for connecting to an existing Unix socket...

s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect("/path/to/socket")


问:
How to Add a "Go Back" Toggle
I have a code where it allows me to go to new text if I press "next" but I also want to create another button where it would allow me to go back to the previous text I was reading.

I know I just write Previous but I don't know the javascript to go back.

$(function() {

    $('.js-button').click(function() {
      if ($('div.active').next().hasClass('hidemsg')) {
        $('div.active').next('div').addClass('active').end().removeClass('active');
      }else{
        alert('Sorry, there is no next entry');
      }
    });
});```
<pre class="snippet-code-css lang-css prettyprint-override"><code>.hidemsg {
 display: none;
  }

  .hidemsg.active {
 display: block;
  }</code>```

<!DOCTYPE html>

  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1">

      1) Info here

       2) Info Here

       3) Info Here

       4) Info Here

       5) Info Here

Next```

答:
replace your javascript with following lines

```<code>$(function() {
    $('#next').click(function() {
        $('.hide-default').show();
        if ($('div.active').next().hasClass('hidemsg')) {
            $('div.active').next('div').addClass('active').end().removeClass('active');
        } else{
            alert('Sorry, there is no next entry');
        }
    });
    $('#prev').click(function() {
        //$('.hide-default').show();
        if ($('div.active').prev().hasClass('hidemsg')) {
            $('div.active').prev('div').addClass('active').end().removeClass('active');
        } else{
            alert('Sorry, there is no previous entry');
        }
    });
});
</code>```

add following line of css at end of your css

```<code>.hide-default{
    display: none;
}
</code>```

remove the following line from your html

```<code><a href="#" type="button" class="btn btn-default js-button">Next</a>
</code>```

and add these lines to your html

```<code><a id="prev" href="#" type="button" class="btn btn-default hide-default js-button">Previous</a>
<a id="next" href="#" type="button" class="btn btn-default js-button">Next</a>
</code>```

Title: How to Add a "Go Back" Toggle

Body:
I have a code where it allows me to go to new text if I press "next" but I also want to create another button where it would allow me to go back to the previous text I was reading.

I know I just write Previous but I don't know the javascript to go back.

$(function() {

    $('.js-button').click(function() {
      if ($('div.active').next().hasClass('hidemsg')) {
        $('div.active').next('div').addClass('active').end().removeClass('active');
      }else{
        alert('Sorry, there is no next entry');
      }
    });
});

.hidemsg {
 display: none;
  }

  .hidemsg.active {
 display: block;
  }

<!DOCTYPE html>

  <meta charset="UTF-8" />
  <meta name="viewport" content="width=device-width, initial-scale=1">

      1) Info here

       2) Info Here

       3) Info Here

       4) Info Here

       5) Info Here

Next

Body:
replace your javascript with following lines

$(function() {
    $('#next').click(function() {
        $('.hide-default').show();
        if ($('div.active').next().hasClass('hidemsg')) {
            $('div.active').next('div').addClass('active').end().removeClass('active');
        } else{
            alert('Sorry, there is no next entry');
        }
    });
    $('#prev').click(function() {
        //$('.hide-default').show();
        if ($('div.active').prev().hasClass('hidemsg')) {
            $('div.active').prev('div').addClass('active').end().removeClass('active');
        } else{
            alert('Sorry, there is no previous entry');
        }
    });
});

add following line of css at end of your css

.hide-default{
    display: none;
}

remove the following line from your html

<a href="#" type="button" class="btn btn-default js-button">Next</a>

and add these lines to your html

<a id="prev" href="#" type="button" class="btn btn-default hide-default js-button">Previous</a>
<a id="next" href="#" type="button" class="btn btn-default js-button">Next</a>


问:
Transform C# List<KeyValuePair<int,string> to TypeScript Object that can be used in Angular 8 DropDownList using ngFor
I have a simple list of KeyValuePair that is being created in MVC Controller and passed to an Angular component.ts. Within Angular I can manually create a key value pair to use in dropdown list as:

```<code>object: { [key: number]: string } = { 2: 'foo', 1: 'bar' };
</code>```

This is what I have in the component.html:

```<code> <option *ngFor="let program of object | keyvalue " value={{program.key}}>{{program.value}}</option>
</code>```

With this method, I get exactly what I expect -- a drop down list. If I use the KeyValyePair that I received from MVC controller the drop down list displays [ object Object] instead of the value I am looking for.
So my question is, how to I transform my KeyValuePair list into the { [key: number], string} format the TypeScript uses?

This is my first attempt at using Angular in DotNet Core 3.1 MVC App and unfortunately at this point I have only a basic knowledge of TypeScript.

答:
The solution offered by Askash did not work 100% in my case but it lead me to the right answer. When I return the KeyValuePair from MVC to the component.ts I ended  up with an array of KeyValuePair that looked like Key: 1 Value: "Whatever".

I used the code from Askash with slight changes and it worked

```<code><ng-container *ngFor="let program of object">
  <option [value]="program.Key">{{program.Value}}</option>
</ng-container>
</code>```

I did not need the two keyvalue pipes and program.key changed to program.Key and the same with value -- value became Value as reflected by the KeyValuePair object.

The key for me was wrapping the <option></option> in a ng-container.

Thanks

Body:
I have a simple list of KeyValuePair that is being created in MVC Controller and passed to an Angular component.ts. Within Angular I can manually create a key value pair to use in dropdown list as:

object: { [key: number]: string } = { 2: 'foo', 1: 'bar' };

This is what I have in the component.html:

 <option *ngFor="let program of object | keyvalue " value={{program.key}}>{{program.value}}</option>

With this method, I get exactly what I expect -- a drop down list. If I use the KeyValyePair that I received from MVC controller the drop down list displays [ object Object] instead of the value I am looking for.
So my question is, how to I transform my KeyValuePair list into the { [key: number], string} format the TypeScript uses?

This is my first attempt at using Angular in DotNet Core 3.1 MVC App and unfortunately at this point I have only a basic knowledge of TypeScript.

Body:
The solution offered by Askash did not work 100% in my case but it lead me to the right answer. When I return the KeyValuePair from MVC to the component.ts I ended  up with an array of KeyValuePair that looked like Key: 1 Value: "Whatever".

I used the code from Askash with slight changes and it worked

<ng-container *ngFor="let program of object">
  <option [value]="program.Key">{{program.Value}}</option>
</ng-container>

I did not need the two keyvalue pipes and program.key changed to program.Key and the same with value -- value became Value as reflected by the KeyValuePair object.

The key for me was wrapping the <option></option> in a ng-container.

Thanks


问:
nodemon not found in npm
I have a problem: nodemon does not run off the npm script (e.g. npm start),

but if nodemon is called on the command line outside the npm script, nodemon runs as normal.

```<code>$ nodemon server.js
14 Feb 22:59:51 - [nodemon] v1.3.7
14 Feb 22:59:51 - [nodemon] to restart at any time, enter `rs`
14 Feb 22:59:51 - [nodemon] watching: *.*
14 Feb 22:59:51 - [nodemon] starting `node server.js`
</code>```

How it is called in npm script:

```<code>package.json

{
...
  "scripts": {
    "start": "nodemon server.js"
  }
}
</code>```

When npm start script is run:

```<code>$ npm start
> aaa@0.0.1 start /home/akul/Documents/aaa
> nodemon server.js

sh: 1: nodemon: not found

npm ERR! Linux 3.13.0-45-generic
npm ERR! argv "node" "/home/akul/npm-global/bin/npm" "start"
npm ERR! node v0.12.0
npm ERR! npm  v2.5.0
npm ERR! code ELIFECYCLE
npm ERR! aaa@0.0.1 start: `nodemon server.js`
npm ERR! Exit status 127
npm ERR!
npm ERR! Failed at the aaa@0.0.1 start script 'nodemon server.js'.
npm ERR! This is most likely a problem with the aaa package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     nodemon server.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls aaa
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/akul/Documents/aaa/npm-debug.log
</code>```

I've been looking for a solution, but have not found one.

答:
for linux try

```<code>sudo npm install -g nodemon
</code>```

for windows
open powershell or cmd as administration

```<code>npm install -g nodemon
</code>```

Title: nodemon not found in npm

Body:
I have a problem: nodemon does not run off the npm script (e.g. npm start),

but if nodemon is called on the command line outside the npm script, nodemon runs as normal.

$ nodemon server.js
14 Feb 22:59:51 - [nodemon] v1.3.7
14 Feb 22:59:51 - [nodemon] to restart at any time, enter `rs`
14 Feb 22:59:51 - [nodemon] watching: *.*
14 Feb 22:59:51 - [nodemon] starting `node server.js`

How it is called in npm script:

package.json

{
...
  "scripts": {
    "start": "nodemon server.js"
  }
}

When npm start script is run:

$ npm start
> aaa@0.0.1 start /home/akul/Documents/aaa
> nodemon server.js

sh: 1: nodemon: not found

npm ERR! Linux 3.13.0-45-generic
npm ERR! argv "node" "/home/akul/npm-global/bin/npm" "start"
npm ERR! node v0.12.0
npm ERR! npm  v2.5.0
npm ERR! code ELIFECYCLE
npm ERR! aaa@0.0.1 start: `nodemon server.js`
npm ERR! Exit status 127
npm ERR!
npm ERR! Failed at the aaa@0.0.1 start script 'nodemon server.js'.
npm ERR! This is most likely a problem with the aaa package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     nodemon server.js
npm ERR! You can get their info via:
npm ERR!     npm owner ls aaa
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /home/akul/Documents/aaa/npm-debug.log

I've been looking for a solution, but have not found one.

Body:
for linux try

sudo npm install -g nodemon

for windows
open powershell or cmd as administration

npm install -g nodemon


答:
For such problems we usually consider groupby and transform based filtering as it is pretty fast.

```<code>df[df.groupby(level=0)['BusinessType'].transform('nunique') > 1]

            BusinessType  ID1
Index Title
Company1          Type 1  123
Company1          Type 2  456
</code>```

The first step is to determine the groups/rows which are associated with more than one type:

```<code>df.groupby(level=0)['BusinessType'].transform('nunique')

Index Title
Company1    2
Company1    2
Company2    1
Company3    1
Name: BusinessType, dtype: int64
</code>```

From here, we remove all companies whose # unique types associated with are == 1.

Title: Drop rows within dataframe based on condition pandas python

Body:
For such problems we usually consider groupby and transform based filtering as it is pretty fast.

df[df.groupby(level=0)['BusinessType'].transform('nunique') > 1]

            BusinessType  ID1
Index Title
Company1          Type 1  123
Company1          Type 2  456

The first step is to determine the groups/rows which are associated with more than one type:

df.groupby(level=0)['BusinessType'].transform('nunique')

Index Title
Company1    2
Company1    2
Company2    1
Company3    1
Name: BusinessType, dtype: int64

From here, we remove all companies whose # unique types associated with are == 1.