问:
Ubuntu 16.04 - wireless connectivity randomly drops and driver must be restarted
I am using Ubuntu 16.04 and have experienced a known issue where the wifi randomly fails, with the icon still showing connectivity but no connection available. This wasn't previously a problem and started suddenly occurring frequently, with no obvious cause.

If someone can find a solution for this set up I would hugely appreciate it, as this problem is making my system unusable.

So far I have attempted to solve the problem by:

Running 'sudo apt-get dist-upgrade'

Creating a file '/etc/pm/config.d/config' with:
(http://grenangen.se/node/86)

```<code> SUSPEND_MODULES="rtl8821ae"
</code>```

Creating a file '/etc/pm/config.d/config' with:

```<code> service network-manager restart
</code>```

Creating a file '/etc/pm/sleep.d/10_resume_wifi' with:
(https://askubuntu.com/questions/452826/wireless-networking-not-working-after-resume-in-ubuntu-14-04)

```<code>#!/bin/sh

case "${1}" in
        resume|thaw)
        nmcli nm sleep false
                ;;
esac
</code>```

OR

```<code>#!/bin/sh

 case "${1}" in
   resume|thaw)
     nmcli r wifi off && nmcli r wifi on ;;
esac
</code>```

Creating a file '/etc/systemd/system/wifi-resume.service' with:
(https://askubuntu.com/questions/761180/wifi-doesnt-work-after-suspend-after-16-04-upgrade)

```<code>#/etc/systemd/system/wifi-resume.service
#sudo systemctl enable wifi-resume.service
[Unit]
Description=Restart networkmanager at resume
After=suspend.target
After=hibernate.target
After=hybrid-sleep.target

[Service]
Type=oneshot
 ExecStart=/bin/systemctl restart network-manager.service

[Install]
WantedBy=suspend.target
WantedBy=hibernate.target
WantedBy=hybrid-sleep.target
</code>```

Creating a file '/etc/modprobe.d/rtl8821ae.conf' with:

```<code>options rtl8821ae fwlps=N
options rtl8821ae swenc=1
</code>```

Running the following in via the terminal

```<code>sudo apt-get install linux-headers-generic build-essential git
git clone http://github.com/lwfinger/rtlwifi_new.git
cd rtlwifi_new
make
sudo make install
</code>```

The following is my wireless info:

```########## wireless info START ##########

Report from: 08 May 2017 13:02 ICT +0700

Booted last: 08 May 2017 00:00 ICT +0700

Script from: 25 Mar 2017 07:04 UTC +0000

##### release ###########################

Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial

##### kernel ############################

Linux 4.4.0-75-generic #96-Ubuntu SMP Thu Apr 20 09:56:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Parameters: ro, quiet, splash, vt.handoff=7

##### desktop ###########################

Ubuntu

##### lspci #############################

01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 10)  Subsystem: Lenovo RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [17aa:384c]     Kernel driver in use: r8169

02:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. RTL8821AE 802.11ac PCIe Wireless Network Adapter [10ec:8821]     Subsystem: Lenovo RTL8821AE 802.11ac PCIe Wireless Network Adapter [17aa:a814]  Kernel driver in use: rtl8821ae

##### lsusb #############################

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 005: ID 0bda:0821 Realtek Semiconductor Corp.  Bus 001 Device 004: ID 0bda:0129 Realtek Semiconductor Corp. RTS5129 Card Reader Controller Bus 001 Device 003: ID 04f2:b57e Chicony Electronics Co., Ltd  Bus 001 Device 002: ID 046d:c077 Logitech, Inc. M105 Optical Mouse Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

##### PCMCIA card info ##################

##### rfkill ############################

0: ideapad_wlan: Wireless LAN   Soft blocked: no    Hard blocked: no 1: ideapad_bluetooth: Bluetooth    Soft blocked: no    Hard blocked: no 2: hci0: Bluetooth     Soft blocked: no    Hard blocked: no 6: phy0: Wireless LAN  Soft blocked: no    Hard blocked: no

##### lsmod #############################

rtl8821ae             225280  0 btcoexist              53248  1 rtl8821ae rtl_pci                28672  1 rtl8821ae rtlwifi            77824  2 rtl_pci,rtl8821ae mac80211              737280  3 rtl_pci,rtlwifi,rtl8821ae cfg80211              565248  2 mac80211,rtlwifi ideapad_laptop         24576  0 sparse_keymap         16384  1 ideapad_laptop mxm_wmi                16384  1 nouveau wmi    20480  3 ideapad_laptop,mxm_wmi,nouveau video                  40960  3 i915_bpo,ideapad_laptop,nouveau

##### interfaces ########################

auto lo iface lo inet loopback

##### ifconfig ##########################

docker0   Link encap:Ethernet  HWaddr <MAC 'docker0' [IF1]>
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

enp1s0    Link encap:Ethernet  HWaddr <MAC 'enp1s0' [IF2]>
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:3771 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3771 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:366844 (366.8 KB)  TX bytes:366844 (366.8 KB)

wlp2s0    Link encap:Ethernet  HWaddr <MAC 'wlp2s0' [IF3]>
          inet addr:192.168.1.51  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::5f5:68c6:8ccf:2c5c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1686 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2943 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:152100 (152.1 KB)  TX bytes:4142898 (4.1 MB)

##### iwconfig ##########################

lo        no wireless extensions.

enp1s0    no wireless extensions.

docker0   no wireless extensions.

wlp2s0    IEEE 802.11abgn  ESSID:"ROUND ABOUT"
          Mode:Managed  Frequency:5.805 GHz  Access Point: <MAC 'ROUND ABOUT' [AN4]>
          Bit Rate=150 Mb/s   Tx-Power=20 dBm
          Retry short limit:7   RTS thr=2347 B   Fragment thr:off
          Power Management:off
          Link Quality=64/70  Signal level=-46 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:32   Missed beacon:0

##### route #############################

Kernel IP routing table Destination     Gateway         Genmask        Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    600    0        0 wlp2s0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 docker0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp2s0

##### resolv.conf #######################

nameserver 127.0.1.1

##### network managers ##################

Installed:

    NetworkManager

Running:

root       859     1  0 12:45 ?        00:00:03 /usr/sbin/NetworkManager --no-daemon

##### NetworkManager info ###############

GENERAL.DEVICE:                         docker0 GENERAL.TYPE:          bridge GENERAL.NM-TYPE:                        NMDeviceBridge GENERAL.VENDOR:                          GENERAL.PRODUCT:              GENERAL.DRIVER:                         bridge GENERAL.DRIVER-VERSION:
2.3 GENERAL.FIRMWARE-VERSION:               N/A GENERAL.HWADDR:                         <MAC 'docker0' [IF1]> GENERAL.MTU:                            1500 GENERAL.STATE:                          100 (connected) GENERAL.REASON:                         0 (No reason given) GENERAL.UDI:                            /sys/devices/virtual/net/docker0 GENERAL.IP-IFACE:                     docker0 GENERAL.IS-SOFTWARE:                    yes GENERAL.NM-MANAGED:                     yes GENERAL.AUTOCONNECT:       yes GENERAL.FIRMWARE-MISSING:               no GENERAL.NM-PLUGIN-MISSING:              no GENERAL.PHYS-PORT-ID:
-- GENERAL.CONNECTION:                     docker0 GENERAL.CON-UUID:                       87779905-948b-4923-90a9-a9fdeb019c8c GENERAL.CON-PATH:                 /org/freedesktop/NetworkManager/ActiveConnection/1 GENERAL.METERED:    no (guessed) CAPABILITIES.CARRIER-DETECT:            yes CAPABILITIES.SPEED:                     unknown CAPABILITIES.IS-SOFTWARE:               yes BRIDGE.SLAVES:             CONNECTIONS.AVAILABLE-CONNECTION-PATHS: /org/freedesktop/NetworkManager/Settings/{23} CONNECTIONS.AVAILABLE-CONNECTIONS[1]:   87779905-948b-4923-90a9-a9fdeb019c8c | docker0 IP4.ADDRESS[1]:
172.17.0.1/16 IP4.GATEWAY:                             IP4.ROUTE[1]:                           dst = 169.254.0.0/16, nh = 0.0.0.0, mt = 1000 IP6.GATEWAY:

GENERAL.DEVICE:                         wlp2s0 GENERAL.TYPE:           wifi GENERAL.NM-TYPE:                        NMDeviceWifi GENERAL.VENDOR:                         Realtek Semiconductor Co., Ltd. GENERAL.PRODUCT:                        RTL8821AE 802.11ac PCIe Wireless Network Adapter GENERAL.DRIVER:                         rtl8821ae GENERAL.DRIVER-VERSION:                 4.4.0-75-generic GENERAL.FIRMWARE-VERSION:               N/A GENERAL.HWADDR:            <MAC 'wlp2s0' [IF3]> GENERAL.MTU:                            1500 GENERAL.STATE:                          100 (connected) GENERAL.REASON:                         0 (No reason given) GENERAL.UDI:                            /sys/devices/pci0000:00/0000:00:1c.5/0000:02:00.0/net/wlp2s0 GENERAL.IP-IFACE:                       wlp2s0 GENERAL.IS-SOFTWARE:    no GENERAL.NM-MANAGED:                     yes GENERAL.AUTOCONNECT:    yes GENERAL.FIRMWARE-MISSING:               no GENERAL.NM-PLUGIN-MISSING:              no GENERAL.PHYS-PORT-ID:
-- GENERAL.CONNECTION:                     ROUND ABOUT GENERAL.CON-UUID:                       0b9cd60a-1403-4e3f-88b2-05becd106bf0 GENERAL.CON-PATH:                 /org/freedesktop/NetworkManager/ActiveConnection/4 GENERAL.METERED:    no (guessed) CAPABILITIES.CARRIER-DETECT:            no CAPABILITIES.SPEED:                     150 Mb/s CAPABILITIES.IS-SOFTWARE:               no WIFI-PROPERTIES.WEP:        yes WIFI-PROPERTIES.WPA:                    yes WIFI-PROPERTIES.WPA2:  yes WIFI-PROPERTIES.TKIP:                   yes WIFI-PROPERTIES.CCMP:  yes WIFI-PROPERTIES.AP:                     yes WIFI-PROPERTIES.ADHOC: yes WIFI-PROPERTIES.2GHZ:                   yes WIFI-PROPERTIES.5GHZ:  yes CONNECTIONS.AVAILABLE-CONNECTION-PATHS: /org/freedesktop/NetworkManager/Settings/{3} CONNECTIONS.AVAILABLE-CONNECTIONS[1]:   0b9cd60a-1403-4e3f-88b2-05becd106bf0 | ROUND ABOUT IP4.ADDRESS[1]:
192.168.1.51/24 IP4.GATEWAY:                            192.168.1.1 IP4.DNS[1]:                             203.144.206.29 IP4.DNS[2]:
203.144.206.49 DHCP4.OPTION[1]:                        requested_subnet_mask = 1 DHCP4.OPTION[2]:                        requested_rfc3442_classless_static_routes = 1 DHCP4.OPTION[3]:         subnet_mask = 255.255.255.0 DHCP4.OPTION[4]:                        domain_name_servers = 203.144.206.29 203.144.206.49 DHCP4.OPTION[5]:   ip_address = 192.168.1.51 DHCP4.OPTION[6]:                        requested_static_routes = 1 DHCP4.OPTION[7]:                        dhcp_server_identifier = 192.168.1.1 DHCP4.OPTION[8]:                  default_ip_ttl = 64 DHCP4.OPTION[9]:                        requested_time_offset = 1 DHCP4.OPTION[10]:                       time_offset = 0 DHCP4.OPTION[11]:                       broadcast_address = 192.168.1.255 DHCP4.OPTION[12]:                    requested_interface_mtu = 1 DHCP4.OPTION[13]:                       requested_domain_name_servers = 1 DHCP4.OPTION[14]:                    dhcp_rebinding_time = 75600 DHCP4.OPTION[15]:                       dhcp_message_type = 5 DHCP4.OPTION[16]:                       requested_broadcast_address = 1 DHCP4.OPTION[17]:                      routers = 192.168.1.1 DHCP4.OPTION[18]:                       dhcp_renewal_time = 43200 DHCP4.OPTION[19]:                       requested_domain_name = 1 DHCP4.OPTION[20]:                       requested_routers = 1 DHCP4.OPTION[21]:                       expiry = 1494309725 DHCP4.OPTION[22]:                       requested_wpad = 1 DHCP4.OPTION[23]:                       requested_netbios_scope = 1 DHCP4.OPTION[24]:                       requested_ms_classless_static_routes = 1 DHCP4.OPTION[25]:             requested_netbios_name_servers = 1 DHCP4.OPTION[26]:                   network_number = 192.168.1.0 DHCP4.OPTION[27]:                       requested_domain_search = 1 DHCP4.OPTION[28]:                       requested_ntp_servers = 1 DHCP4.OPTION[29]:                       next_server = 192.168.1.1 DHCP4.OPTION[30]:                       requested_host_name = 1 DHCP4.OPTION[31]:                       dhcp_lease_time = 86400 IP6.ADDRESS[1]:                         fe80::5f5:68c6:8ccf:2c5c/64 IP6.GATEWAY:

GENERAL.DEVICE:                         enp1s0 GENERAL.TYPE:           ethernet GENERAL.NM-TYPE:                        NMDeviceEthernet GENERAL.VENDOR:                         Realtek Semiconductor Co., Ltd. GENERAL.PRODUCT:                        RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller GENERAL.DRIVER:                    r8169 GENERAL.DRIVER-VERSION:                 2.3LK-NAPI GENERAL.FIRMWARE-VERSION:                GENERAL.HWADDR:               <MAC 'enp1s0' [IF2]> GENERAL.MTU:                            1500 GENERAL.STATE:                          20 (unavailable) GENERAL.REASON:                         2 (Device is now managed) GENERAL.UDI:                            /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/net/enp1s0 GENERAL.IP-IFACE:                        GENERAL.IS-SOFTWARE:          no GENERAL.NM-MANAGED:                     yes GENERAL.AUTOCONNECT:    yes GENERAL.FIRMWARE-MISSING:               no GENERAL.NM-PLUGIN-MISSING:              no GENERAL.PHYS-PORT-ID:
-- GENERAL.CONNECTION:                     -- GENERAL.CON-UUID:                       -- GENERAL.CON-PATH:                       -- GENERAL.METERED:                        unknown CAPABILITIES.CARRIER-DETECT:            yes CAPABILITIES.SPEED:                     unknown CAPABILITIES.IS-SOFTWARE:               no WIRED-PROPERTIES.CARRIER:   off CONNECTIONS.AVAILABLE-CONNECTION-PATHS:

SSID                     BSSID              MODE   CHAN  FREQ      RATE       SIGNAL  BARS  SECURITY   ACTIVE  *  RA Workspace            <MAC 'RA Workspace' [AN1]>  Infra  11    2462 MHz  54 Mbit/s  100     ▂▄▆█  WPA1 WPA2  no         ROUND ABOUT 2            <MAC 'ROUND ABOUT 2' [AN2]>  Infra  11    2462 MHz  54 Mbit/s  94      ▂▄▆█  WPA1 WPA2  no         ASIA                     <MAC 'ASIA' [AN3]>  Infra  2     2417 MHz  54 Mbit/s  80      ▂▄▆_  WPA2       no         ROUND ABOUT   <MAC 'ROUND ABOUT' [AN4]>  Infra  161   5805 MHz  54 Mbit/s  76      ▂▄▆_  WPA1 WPA2  yes     *  SMART 4G 5.5 Enterprise  <MAC 'SMART 4G
5.5 Enterprise' [AN5]>  Infra  6     2437 MHz  54 Mbit/s  47      ▂▄__  WPA2       no         KaKA-HOME                <MAC 'KaKA-HOME' [AN6]> Infra  9     2452 MHz  54 Mbit/s  47      ▂▄__  WPA2       no         Kem3bb                   <MAC 'Kem3bb' [AN7]>  Infra  6     2437 MHz  54 Mbit/s  44      ▂▄__  WPA1 WPA2  no

##### NetworkManager.state ##############

[main] NetworkingEnabled=true WirelessEnabled=true WWANEnabled=true

##### NetworkManager.conf ###############

[main] plugins=ifupdown,keyfile,ofono dns=dnsmasq

[ifupdown] managed=false

##### NetworkManager profiles ###########

[[/etc/NetworkManager/system-connections/Pineapple3]] (600 root) [connection] id=Pineapple3 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Pineapple3 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Anna's iPhone]] (600 root) [connection] id=Anna's iPhone | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Anna's iPhone [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Travellers Hostel and Cafe]] (600 root) [connection] id=Travellers Hostel and Cafe | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Travellers Hostel and Cafe [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/9101809875 5G]] (600 root) [connection] id=9101809875 5G | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=9101809875 5G [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Pineapple 4]] (600 root) [connection] id=Pineapple 4 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Pineapple 4 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/unfashion]] (600 root) [connection] id=unfashion | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=unfashion [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/rockyrooster 2]] (600 root) [connection] id=rockyrooster 2 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=rockyrooster 2 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/CS-QR20]] (600 root) [connection] id=CS-QR20 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=CS-QR20 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Steps2]] (600 root) [connection] id=Steps2 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Steps2 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/thonglorartspace2014]] (600 root) [connection] id=thonglorartspace2014 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=thonglorartspace2014 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/rockyrooster 1]] (600 root) [connection] id=rockyrooster 1 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=rockyrooster 1 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Darkwonder II]] (600 root) [connection] id=Darkwonder II | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Darkwonder II [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/9101809875]] (600 root) [connection] id=9101809875 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=9101809875 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/the password is password]] (600 root) [connection] id=the password is password | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=the password is password [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/AP013904N002]] (600 root) [connection] id=AP013904N002 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=AP013904N002 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/ROUND ABOUT]] (600 root) [connection] id=ROUND ABOUT | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=ROUND ABOUT [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Merkey Merkel's Sex Dungeon]] (600 root) [connection] id=Merkey Merkel's Sex Dungeon | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Merkey Merkel's Sex Dungeon [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Sabaijai]] (600 root) [connection] id=Sabaijai | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Sabaijai [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/true_home2G_RmQ]] (600 root) [connection] id=true_home2G_RmQ | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=true_home2G_RmQ [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Melange]] (600 root) [connection] id=Melange | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Melange [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Haleeva SunShine 1]] (600 root) [connection] id=Haleeva SunShine 1 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Haleeva SunShine 1 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/the password is password 1]] (600 root) [connection] id=the password is password 1 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=the password is password [ipv4] method=auto [ipv6] method=auto

##### iw reg get ########################

Region: Asia/Bangkok (based on set time zone)

country 00: DFS-UNSET   (2402 - 2472 @ 40), (N/A, 20), (N/A)    (2457 - 2482 @ 40), (N/A, 20), (N/A), NO-IR     (2474 - 2494 @ 20), (N/A, 20), (N/A), NO-OFDM, NO-IR    (5170 - 5250 @ 80), (N/A, 20), (N/A), NO-IR     (5250 - 5330 @ 80), (N/A, 20), (0 ms), DFS, NO-IR   (5490 - 5730 @ 160), (N/A, 20), (0 ms), DFS, NO-IR  (5735 - 5835 @ 80), (N/A, 20), (N/A), NO-IR     (57240 - 63720 @ 2160), (N/A, 0), (N/A)

##### iwlist scan #######################

lo        Interface doesn't support scanning.

enp1s0    Interface doesn't support scanning.

docker0   Interface doesn't support scanning.

wlp2s0    Interface doesn't support scanning : Device or resource busy

##### module infos ######################

[rtl8821ae] filename:       /lib/modules/4.4.0-75-generic/kernel/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/rtl8821ae.ko firmware:       rtlwifi/rtl8821aefw.bin description:    Realtek 8821ae
802.11ac PCI wireless license:        GPL author:         Realtek WlanFAE   <wlanfae@realtek.com> srcversion:     0E3A4E69E52B6033CEE5893 depends:        rtlwifi,rtl_pci,btcoexist,mac80211 intree:         Y vermagic:       4.4.0-75-generic SMP mod_unload modversions  parm:     swenc:Set to 1 for software crypto (default 0)  (bool) parm:           ips:Set to 0 to not use link power save (default 1)  (bool) parm:      swlps:Set to 1 to use SW control power save (default 0)  (bool) parm:  fwlps:Set to 1 to use FW control power save (default 1)  (bool) parm:  msi:Set to 1 to use MSI interrupts mode (default 1)  (bool) parm:      debug:Set debug level (0-5) (default 0) (int) parm:           disable_watchdog:Set to 1 to disable the watchdog (default 0)  (bool) parm:           int_clear:Set to 0 to disable interrupt clear before set (default 1)  (bool)

[rtl_pci] filename:       /lib/modules/4.4.0-75-generic/kernel/drivers/net/wireless/realtek/rtlwifi/rtl_pci.ko description:    PCI basic driver for rtlwifi license:        GPL author:         Larry Finger    <Larry.FInger@lwfinger.net> author:       Realtek WlanFAE   <wlanfae@realtek.com> author:         lizhaoming    <chaoming_li@realsil.com.cn> srcversion:     3B33A271D8BDBE9755FC279 depends:        mac80211,rtlwifi intree:       Y vermagic:       4.4.0-75-generic SMP mod_unload modversions

[rtlwifi] filename:       /lib/modules/4.4.0-75-generic/kernel/drivers/net/wireless/realtek/rtlwifi/rtlwifi.ko description:    Realtek 802.11n PCI wireless core license:        GPL author:         Larry Finger   <Larry.FInger@lwfinger.net> author:       Realtek WlanFAE   <wlanfae@realtek.com> author:         lizhaoming    <chaoming_li@realsil.com.cn> srcversion:     F5F46436A12E44A7B798C34 depends:        mac80211,cfg80211 intree:      Y vermagic:       4.4.0-75-generic SMP mod_unload modversions

[mac80211] filename:       /lib/modules/4.4.0-75-generic/kernel/net/mac80211/mac80211.ko license: GPL description:    IEEE 802.11 subsystem srcversion:     0EDE2F31518E91FFBB1E4EC depends:        cfg80211 intree:         Y vermagic:       4.4.0-75-generic SMP mod_unload modversions  parm:     minstrel_vht_only:Use only VHT rates when VHT is supported by sta. (bool) parm:           max_nullfunc_tries:Maximum nullfunc tx tries before disconnecting (reason 4). (int) parm:           max_probe_tries:Maximum probe tries before disconnecting (reason 4). (int) parm:           beacon_loss_count:Number of beacon intervals before we decide beacon was lost. (int) parm:           probe_wait_ms:Maximum time(ms) to wait for probe response before disconnecting (reason 4). (int) parm:           ieee80211_default_rc_algo:Default rate control algorithm for mac80211 to use (charp)

[cfg80211] filename:       /lib/modules/4.4.0-75-generic/kernel/net/wireless/cfg80211.ko description:    wireless configuration support license:        GPL author:         Johannes Berg srcversion:     48C96882CCDC964173BE1D1 depends:         intree:         Y vermagic:       4.4.0-75-generic SMP mod_unload modversions  parm:           bss_entries_limit:limit to number of scan BSS entries (per wiphy, default 1000) (int) parm:       ieee80211_regdom:IEEE 802.11 regulatory domain code (charp) parm:      cfg80211_disable_40mhz_24ghz:Disable 40MHz support in the 2.4GHz band (bool)

##### module parameters #################

[rtl8821ae] debug: 0 disable_watchdog: N fwlps: N int_clear: Y ips: Y msi: Y swenc: Y swlps: N

[mac80211] beacon_loss_count: 7 ieee80211_default_rc_algo: minstrel_ht max_nullfunc_tries: 2 max_probe_tries: 5 minstrel_vht_only: Y probe_wait_ms: 500

[cfg80211] bss_entries_limit: 1000 cfg80211_disable_40mhz_24ghz: N ieee80211_regdom: 00

##### /etc/modules ######################

##### modprobe options ##################

[/etc/modprobe.d/blacklist-ath_pci.conf] blacklist ath_pci

[/etc/modprobe.d/blacklist.conf] blacklist evbug blacklist usbmouse blacklist usbkbd blacklist eepro100 blacklist de4x5 blacklist eth1394 blacklist snd_intel8x0m blacklist snd_aw2 blacklist i2c_i801 blacklist prism54 blacklist bcm43xx blacklist garmin_gps blacklist asus_acpi blacklist snd_pcsp blacklist pcspkr blacklist amd76x_edac

[/etc/modprobe.d/blacklist-rare-network.conf] alias net-pf-3 off alias net-pf-6 off alias net-pf-9 off alias net-pf-11 off alias net-pf-12 off alias net-pf-19 off alias net-pf-21 off alias net-pf-36 off

[/etc/modprobe.d/iwlwifi.conf] remove iwlwifi \ (/sbin/lsmod | grep -o
-e ^iwlmvm -e ^iwldvm -e ^iwlwifi | xargs /sbin/rmmod) \ && /sbin/modprobe -r mac80211

[/etc/modprobe.d/mlx4.conf] softdep mlx4_core post: mlx4_en

[/etc/modprobe.d/r8169.conf] options r8169 fwlps=N

[/etc/modprobe.d/rtl8821ae.conf] options rtl8821ae fwlps=N options rtl8821ae swenc=1

##### rc.local ##########################

exit 0

##### pm-utils ##########################

[/etc/pm/config.d/config] (644 root) service network-manager restart

[/etc/pm/sleep.d/10_resume_wifi] (755 root) case "${1}" in
    resume|thaw) nmcli nm sleep false ;; esac

##### udev rules ########################
```

答:
for < = ubuntu 20.4

```<code>echo "options iwlwifi 11n_disable=1" | sudo tee /etc/modprobe.d/iwlwifi.conf
sudo modprobe -rfv iwldvm
sudo modprobe -rfv iwlwifi
sudo modprobe -v iwlwifi
</code>```

or

```<code>sudo apt-get purge bcmwl-kernel-source
mkdir bcmwl && cd bcmwl
search to google ==>bcmwl-kernel-source_6.30.223.271+bdcom-0ubuntu7_amd64.deb
sudo dpkg -i bcmwl-kernel-source_6.30.223.271+bdcom-0ubuntu7_amd64.deb
</code>```

Body:
I am using Ubuntu 16.04 and have experienced a known issue where the wifi randomly fails, with the icon still showing connectivity but no connection available. This wasn't previously a problem and started suddenly occurring frequently, with no obvious cause.

If someone can find a solution for this set up I would hugely appreciate it, as this problem is making my system unusable.

So far I have attempted to solve the problem by:

Running 'sudo apt-get dist-upgrade'

Creating a file '/etc/pm/config.d/config' with:
(http://grenangen.se/node/86)

 SUSPEND_MODULES="rtl8821ae"

Creating a file '/etc/pm/config.d/config' with:

 service network-manager restart

Creating a file '/etc/pm/sleep.d/10_resume_wifi' with:
(https://askubuntu.com/questions/452826/wireless-networking-not-working-after-resume-in-ubuntu-14-04)

#!/bin/sh

case "${1}" in
        resume|thaw)
        nmcli nm sleep false
                ;;
esac

OR

#!/bin/sh

 case "${1}" in
   resume|thaw)
     nmcli r wifi off && nmcli r wifi on ;;
esac

Creating a file '/etc/systemd/system/wifi-resume.service' with:
(https://askubuntu.com/questions/761180/wifi-doesnt-work-after-suspend-after-16-04-upgrade)

#/etc/systemd/system/wifi-resume.service
#sudo systemctl enable wifi-resume.service
[Unit]
Description=Restart networkmanager at resume
After=suspend.target
After=hibernate.target
After=hybrid-sleep.target

[Service]
Type=oneshot
 ExecStart=/bin/systemctl restart network-manager.service

[Install]
WantedBy=suspend.target
WantedBy=hibernate.target
WantedBy=hybrid-sleep.target

Creating a file '/etc/modprobe.d/rtl8821ae.conf' with:

options rtl8821ae fwlps=N
options rtl8821ae swenc=1

Running the following in via the terminal

sudo apt-get install linux-headers-generic build-essential git
git clone http://github.com/lwfinger/rtlwifi_new.git
cd rtlwifi_new
make
sudo make install

The following is my wireless info:

########## wireless info START ##########

Report from: 08 May 2017 13:02 ICT +0700

Booted last: 08 May 2017 00:00 ICT +0700

Script from: 25 Mar 2017 07:04 UTC +0000

##### release ###########################

Distributor ID: Ubuntu Description: Ubuntu 16.04.2 LTS Release: 16.04 Codename: xenial

##### kernel ############################

Linux 4.4.0-75-generic #96-Ubuntu SMP Thu Apr 20 09:56:33 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

Parameters: ro, quiet, splash, vt.handoff=7

##### desktop ###########################

Ubuntu

##### lspci #############################

01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 10)  Subsystem: Lenovo RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [17aa:384c]     Kernel driver in use: r8169

02:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. RTL8821AE 802.11ac PCIe Wireless Network Adapter [10ec:8821]     Subsystem: Lenovo RTL8821AE 802.11ac PCIe Wireless Network Adapter [17aa:a814]  Kernel driver in use: rtl8821ae

##### lsusb #############################

Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 001 Device 005: ID 0bda:0821 Realtek Semiconductor Corp.  Bus 001 Device 004: ID 0bda:0129 Realtek Semiconductor Corp. RTS5129 Card Reader Controller Bus 001 Device 003: ID 04f2:b57e Chicony Electronics Co., Ltd  Bus 001 Device 002: ID 046d:c077 Logitech, Inc. M105 Optical Mouse Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

##### PCMCIA card info ##################

##### rfkill ############################

0: ideapad_wlan: Wireless LAN   Soft blocked: no    Hard blocked: no 1: ideapad_bluetooth: Bluetooth    Soft blocked: no    Hard blocked: no 2: hci0: Bluetooth     Soft blocked: no    Hard blocked: no 6: phy0: Wireless LAN  Soft blocked: no    Hard blocked: no

##### lsmod #############################

rtl8821ae             225280  0 btcoexist              53248  1 rtl8821ae rtl_pci                28672  1 rtl8821ae rtlwifi            77824  2 rtl_pci,rtl8821ae mac80211              737280  3 rtl_pci,rtlwifi,rtl8821ae cfg80211              565248  2 mac80211,rtlwifi ideapad_laptop         24576  0 sparse_keymap         16384  1 ideapad_laptop mxm_wmi                16384  1 nouveau wmi    20480  3 ideapad_laptop,mxm_wmi,nouveau video                  40960  3 i915_bpo,ideapad_laptop,nouveau

##### interfaces ########################

auto lo iface lo inet loopback

##### ifconfig ##########################

docker0   Link encap:Ethernet  HWaddr <MAC 'docker0' [IF1]>
          inet addr:172.17.0.1  Bcast:0.0.0.0  Mask:255.255.0.0
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

enp1s0    Link encap:Ethernet  HWaddr <MAC 'enp1s0' [IF2]>
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:3771 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3771 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:366844 (366.8 KB)  TX bytes:366844 (366.8 KB)

wlp2s0    Link encap:Ethernet  HWaddr <MAC 'wlp2s0' [IF3]>
          inet addr:192.168.1.51  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: fe80::5f5:68c6:8ccf:2c5c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1686 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2943 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:152100 (152.1 KB)  TX bytes:4142898 (4.1 MB)

##### iwconfig ##########################

lo        no wireless extensions.

enp1s0    no wireless extensions.

docker0   no wireless extensions.

wlp2s0    IEEE 802.11abgn  ESSID:"ROUND ABOUT"
          Mode:Managed  Frequency:5.805 GHz  Access Point: <MAC 'ROUND ABOUT' [AN4]>
          Bit Rate=150 Mb/s   Tx-Power=20 dBm
          Retry short limit:7   RTS thr=2347 B   Fragment thr:off
          Power Management:off
          Link Quality=64/70  Signal level=-46 dBm
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:0  Invalid misc:32   Missed beacon:0

##### route #############################

Kernel IP routing table Destination     Gateway         Genmask        Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    600    0        0 wlp2s0
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 docker0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlp2s0

##### resolv.conf #######################

nameserver 127.0.1.1

##### network managers ##################

Installed:

    NetworkManager

Running:

root       859     1  0 12:45 ?        00:00:03 /usr/sbin/NetworkManager --no-daemon

##### NetworkManager info ###############

GENERAL.DEVICE:                         docker0 GENERAL.TYPE:          bridge GENERAL.NM-TYPE:                        NMDeviceBridge GENERAL.VENDOR:                          GENERAL.PRODUCT:              GENERAL.DRIVER:                         bridge GENERAL.DRIVER-VERSION:
2.3 GENERAL.FIRMWARE-VERSION:               N/A GENERAL.HWADDR:                         <MAC 'docker0' [IF1]> GENERAL.MTU:                            1500 GENERAL.STATE:                          100 (connected) GENERAL.REASON:                         0 (No reason given) GENERAL.UDI:                            /sys/devices/virtual/net/docker0 GENERAL.IP-IFACE:                     docker0 GENERAL.IS-SOFTWARE:                    yes GENERAL.NM-MANAGED:                     yes GENERAL.AUTOCONNECT:       yes GENERAL.FIRMWARE-MISSING:               no GENERAL.NM-PLUGIN-MISSING:              no GENERAL.PHYS-PORT-ID:
-- GENERAL.CONNECTION:                     docker0 GENERAL.CON-UUID:                       87779905-948b-4923-90a9-a9fdeb019c8c GENERAL.CON-PATH:                 /org/freedesktop/NetworkManager/ActiveConnection/1 GENERAL.METERED:    no (guessed) CAPABILITIES.CARRIER-DETECT:            yes CAPABILITIES.SPEED:                     unknown CAPABILITIES.IS-SOFTWARE:               yes BRIDGE.SLAVES:             CONNECTIONS.AVAILABLE-CONNECTION-PATHS: /org/freedesktop/NetworkManager/Settings/{23} CONNECTIONS.AVAILABLE-CONNECTIONS[1]:   87779905-948b-4923-90a9-a9fdeb019c8c | docker0 IP4.ADDRESS[1]:
172.17.0.1/16 IP4.GATEWAY:                             IP4.ROUTE[1]:                           dst = 169.254.0.0/16, nh = 0.0.0.0, mt = 1000 IP6.GATEWAY:

GENERAL.DEVICE:                         wlp2s0 GENERAL.TYPE:           wifi GENERAL.NM-TYPE:                        NMDeviceWifi GENERAL.VENDOR:                         Realtek Semiconductor Co., Ltd. GENERAL.PRODUCT:                        RTL8821AE 802.11ac PCIe Wireless Network Adapter GENERAL.DRIVER:                         rtl8821ae GENERAL.DRIVER-VERSION:                 4.4.0-75-generic GENERAL.FIRMWARE-VERSION:               N/A GENERAL.HWADDR:            <MAC 'wlp2s0' [IF3]> GENERAL.MTU:                            1500 GENERAL.STATE:                          100 (connected) GENERAL.REASON:                         0 (No reason given) GENERAL.UDI:                            /sys/devices/pci0000:00/0000:00:1c.5/0000:02:00.0/net/wlp2s0 GENERAL.IP-IFACE:                       wlp2s0 GENERAL.IS-SOFTWARE:    no GENERAL.NM-MANAGED:                     yes GENERAL.AUTOCONNECT:    yes GENERAL.FIRMWARE-MISSING:               no GENERAL.NM-PLUGIN-MISSING:              no GENERAL.PHYS-PORT-ID:
-- GENERAL.CONNECTION:                     ROUND ABOUT GENERAL.CON-UUID:                       0b9cd60a-1403-4e3f-88b2-05becd106bf0 GENERAL.CON-PATH:                 /org/freedesktop/NetworkManager/ActiveConnection/4 GENERAL.METERED:    no (guessed) CAPABILITIES.CARRIER-DETECT:            no CAPABILITIES.SPEED:                     150 Mb/s CAPABILITIES.IS-SOFTWARE:               no WIFI-PROPERTIES.WEP:        yes WIFI-PROPERTIES.WPA:                    yes WIFI-PROPERTIES.WPA2:  yes WIFI-PROPERTIES.TKIP:                   yes WIFI-PROPERTIES.CCMP:  yes WIFI-PROPERTIES.AP:                     yes WIFI-PROPERTIES.ADHOC: yes WIFI-PROPERTIES.2GHZ:                   yes WIFI-PROPERTIES.5GHZ:  yes CONNECTIONS.AVAILABLE-CONNECTION-PATHS: /org/freedesktop/NetworkManager/Settings/{3} CONNECTIONS.AVAILABLE-CONNECTIONS[1]:   0b9cd60a-1403-4e3f-88b2-05becd106bf0 | ROUND ABOUT IP4.ADDRESS[1]:
192.168.1.51/24 IP4.GATEWAY:                            192.168.1.1 IP4.DNS[1]:                             203.144.206.29 IP4.DNS[2]:
203.144.206.49 DHCP4.OPTION[1]:                        requested_subnet_mask = 1 DHCP4.OPTION[2]:                        requested_rfc3442_classless_static_routes = 1 DHCP4.OPTION[3]:         subnet_mask = 255.255.255.0 DHCP4.OPTION[4]:                        domain_name_servers = 203.144.206.29 203.144.206.49 DHCP4.OPTION[5]:   ip_address = 192.168.1.51 DHCP4.OPTION[6]:                        requested_static_routes = 1 DHCP4.OPTION[7]:                        dhcp_server_identifier = 192.168.1.1 DHCP4.OPTION[8]:                  default_ip_ttl = 64 DHCP4.OPTION[9]:                        requested_time_offset = 1 DHCP4.OPTION[10]:                       time_offset = 0 DHCP4.OPTION[11]:                       broadcast_address = 192.168.1.255 DHCP4.OPTION[12]:                    requested_interface_mtu = 1 DHCP4.OPTION[13]:                       requested_domain_name_servers = 1 DHCP4.OPTION[14]:                    dhcp_rebinding_time = 75600 DHCP4.OPTION[15]:                       dhcp_message_type = 5 DHCP4.OPTION[16]:                       requested_broadcast_address = 1 DHCP4.OPTION[17]:                      routers = 192.168.1.1 DHCP4.OPTION[18]:                       dhcp_renewal_time = 43200 DHCP4.OPTION[19]:                       requested_domain_name = 1 DHCP4.OPTION[20]:                       requested_routers = 1 DHCP4.OPTION[21]:                       expiry = 1494309725 DHCP4.OPTION[22]:                       requested_wpad = 1 DHCP4.OPTION[23]:                       requested_netbios_scope = 1 DHCP4.OPTION[24]:                       requested_ms_classless_static_routes = 1 DHCP4.OPTION[25]:             requested_netbios_name_servers = 1 DHCP4.OPTION[26]:                   network_number = 192.168.1.0 DHCP4.OPTION[27]:                       requested_domain_search = 1 DHCP4.OPTION[28]:                       requested_ntp_servers = 1 DHCP4.OPTION[29]:                       next_server = 192.168.1.1 DHCP4.OPTION[30]:                       requested_host_name = 1 DHCP4.OPTION[31]:                       dhcp_lease_time = 86400 IP6.ADDRESS[1]:                         fe80::5f5:68c6:8ccf:2c5c/64 IP6.GATEWAY:

GENERAL.DEVICE:                         enp1s0 GENERAL.TYPE:           ethernet GENERAL.NM-TYPE:                        NMDeviceEthernet GENERAL.VENDOR:                         Realtek Semiconductor Co., Ltd. GENERAL.PRODUCT:                        RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller GENERAL.DRIVER:                    r8169 GENERAL.DRIVER-VERSION:                 2.3LK-NAPI GENERAL.FIRMWARE-VERSION:                GENERAL.HWADDR:               <MAC 'enp1s0' [IF2]> GENERAL.MTU:                            1500 GENERAL.STATE:                          20 (unavailable) GENERAL.REASON:                         2 (Device is now managed) GENERAL.UDI:                            /sys/devices/pci0000:00/0000:00:1c.0/0000:01:00.0/net/enp1s0 GENERAL.IP-IFACE:                        GENERAL.IS-SOFTWARE:          no GENERAL.NM-MANAGED:                     yes GENERAL.AUTOCONNECT:    yes GENERAL.FIRMWARE-MISSING:               no GENERAL.NM-PLUGIN-MISSING:              no GENERAL.PHYS-PORT-ID:
-- GENERAL.CONNECTION:                     -- GENERAL.CON-UUID:                       -- GENERAL.CON-PATH:                       -- GENERAL.METERED:                        unknown CAPABILITIES.CARRIER-DETECT:            yes CAPABILITIES.SPEED:                     unknown CAPABILITIES.IS-SOFTWARE:               no WIRED-PROPERTIES.CARRIER:   off CONNECTIONS.AVAILABLE-CONNECTION-PATHS:

SSID                     BSSID              MODE   CHAN  FREQ      RATE       SIGNAL  BARS  SECURITY   ACTIVE  *  RA Workspace            <MAC 'RA Workspace' [AN1]>  Infra  11    2462 MHz  54 Mbit/s  100     ▂▄▆█  WPA1 WPA2  no         ROUND ABOUT 2            <MAC 'ROUND ABOUT 2' [AN2]>  Infra  11    2462 MHz  54 Mbit/s  94      ▂▄▆█  WPA1 WPA2  no         ASIA                     <MAC 'ASIA' [AN3]>  Infra  2     2417 MHz  54 Mbit/s  80      ▂▄▆_  WPA2       no         ROUND ABOUT   <MAC 'ROUND ABOUT' [AN4]>  Infra  161   5805 MHz  54 Mbit/s  76      ▂▄▆_  WPA1 WPA2  yes     *  SMART 4G 5.5 Enterprise  <MAC 'SMART 4G
5.5 Enterprise' [AN5]>  Infra  6     2437 MHz  54 Mbit/s  47      ▂▄__  WPA2       no         KaKA-HOME                <MAC 'KaKA-HOME' [AN6]> Infra  9     2452 MHz  54 Mbit/s  47      ▂▄__  WPA2       no         Kem3bb                   <MAC 'Kem3bb' [AN7]>  Infra  6     2437 MHz  54 Mbit/s  44      ▂▄__  WPA1 WPA2  no

##### NetworkManager.state ##############

[main] NetworkingEnabled=true WirelessEnabled=true WWANEnabled=true

##### NetworkManager.conf ###############

[main] plugins=ifupdown,keyfile,ofono dns=dnsmasq

[ifupdown] managed=false

##### NetworkManager profiles ###########

[[/etc/NetworkManager/system-connections/Pineapple3]] (600 root) [connection] id=Pineapple3 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Pineapple3 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Anna's iPhone]] (600 root) [connection] id=Anna's iPhone | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Anna's iPhone [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Travellers Hostel and Cafe]] (600 root) [connection] id=Travellers Hostel and Cafe | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Travellers Hostel and Cafe [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/9101809875 5G]] (600 root) [connection] id=9101809875 5G | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=9101809875 5G [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Pineapple 4]] (600 root) [connection] id=Pineapple 4 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Pineapple 4 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/unfashion]] (600 root) [connection] id=unfashion | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=unfashion [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/rockyrooster 2]] (600 root) [connection] id=rockyrooster 2 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=rockyrooster 2 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/CS-QR20]] (600 root) [connection] id=CS-QR20 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=CS-QR20 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Steps2]] (600 root) [connection] id=Steps2 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Steps2 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/thonglorartspace2014]] (600 root) [connection] id=thonglorartspace2014 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=thonglorartspace2014 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/rockyrooster 1]] (600 root) [connection] id=rockyrooster 1 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=rockyrooster 1 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Darkwonder II]] (600 root) [connection] id=Darkwonder II | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Darkwonder II [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/9101809875]] (600 root) [connection] id=9101809875 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=9101809875 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/the password is password]] (600 root) [connection] id=the password is password | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=the password is password [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/AP013904N002]] (600 root) [connection] id=AP013904N002 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=AP013904N002 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/ROUND ABOUT]] (600 root) [connection] id=ROUND ABOUT | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=ROUND ABOUT [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Merkey Merkel's Sex Dungeon]] (600 root) [connection] id=Merkey Merkel's Sex Dungeon | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Merkey Merkel's Sex Dungeon [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Sabaijai]] (600 root) [connection] id=Sabaijai | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Sabaijai [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/true_home2G_RmQ]] (600 root) [connection] id=true_home2G_RmQ | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=true_home2G_RmQ [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Melange]] (600 root) [connection] id=Melange | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Melange [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/Haleeva SunShine 1]] (600 root) [connection] id=Haleeva SunShine 1 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=Haleeva SunShine 1 [ipv4] method=auto [ipv6] method=auto

[[/etc/NetworkManager/system-connections/the password is password 1]] (600 root) [connection] id=the password is password 1 | type=wifi | permissions=user:conor:; [wifi] mac-address=<MAC 'wlp2s0' [IF3]> | mac-address-blacklist= | ssid=the password is password [ipv4] method=auto [ipv6] method=auto

##### iw reg get ########################

Region: Asia/Bangkok (based on set time zone)

country 00: DFS-UNSET   (2402 - 2472 @ 40), (N/A, 20), (N/A)    (2457 - 2482 @ 40), (N/A, 20), (N/A), NO-IR     (2474 - 2494 @ 20), (N/A, 20), (N/A), NO-OFDM, NO-IR    (5170 - 5250 @ 80), (N/A, 20), (N/A), NO-IR     (5250 - 5330 @ 80), (N/A, 20), (0 ms), DFS, NO-IR   (5490 - 5730 @ 160), (N/A, 20), (0 ms), DFS, NO-IR  (5735 - 5835 @ 80), (N/A, 20), (N/A), NO-IR     (57240 - 63720 @ 2160), (N/A, 0), (N/A)

##### iwlist scan #######################

lo        Interface doesn't support scanning.

enp1s0    Interface doesn't support scanning.

docker0   Interface doesn't support scanning.

wlp2s0    Interface doesn't support scanning : Device or resource busy

##### module infos ######################

[rtl8821ae] filename:       /lib/modules/4.4.0-75-generic/kernel/drivers/net/wireless/realtek/rtlwifi/rtl8821ae/rtl8821ae.ko firmware:       rtlwifi/rtl8821aefw.bin description:    Realtek 8821ae
802.11ac PCI wireless license:        GPL author:         Realtek WlanFAE   <wlanfae@realtek.com> srcversion:     0E3A4E69E52B6033CEE5893 depends:        rtlwifi,rtl_pci,btcoexist,mac80211 intree:         Y vermagic:       4.4.0-75-generic SMP mod_unload modversions  parm:     swenc:Set to 1 for software crypto (default 0)  (bool) parm:           ips:Set to 0 to not use link power save (default 1)  (bool) parm:      swlps:Set to 1 to use SW control power save (default 0)  (bool) parm:  fwlps:Set to 1 to use FW control power save (default 1)  (bool) parm:  msi:Set to 1 to use MSI interrupts mode (default 1)  (bool) parm:      debug:Set debug level (0-5) (default 0) (int) parm:           disable_watchdog:Set to 1 to disable the watchdog (default 0)  (bool) parm:           int_clear:Set to 0 to disable interrupt clear before set (default 1)  (bool)

[rtl_pci] filename:       /lib/modules/4.4.0-75-generic/kernel/drivers/net/wireless/realtek/rtlwifi/rtl_pci.ko description:    PCI basic driver for rtlwifi license:        GPL author:         Larry Finger    <Larry.FInger@lwfinger.net> author:       Realtek WlanFAE   <wlanfae@realtek.com> author:         lizhaoming    <chaoming_li@realsil.com.cn> srcversion:     3B33A271D8BDBE9755FC279 depends:        mac80211,rtlwifi intree:       Y vermagic:       4.4.0-75-generic SMP mod_unload modversions

[rtlwifi] filename:       /lib/modules/4.4.0-75-generic/kernel/drivers/net/wireless/realtek/rtlwifi/rtlwifi.ko description:    Realtek 802.11n PCI wireless core license:        GPL author:         Larry Finger   <Larry.FInger@lwfinger.net> author:       Realtek WlanFAE   <wlanfae@realtek.com> author:         lizhaoming    <chaoming_li@realsil.com.cn> srcversion:     F5F46436A12E44A7B798C34 depends:        mac80211,cfg80211 intree:      Y vermagic:       4.4.0-75-generic SMP mod_unload modversions

[mac80211] filename:       /lib/modules/4.4.0-75-generic/kernel/net/mac80211/mac80211.ko license: GPL description:    IEEE 802.11 subsystem srcversion:     0EDE2F31518E91FFBB1E4EC depends:        cfg80211 intree:         Y vermagic:       4.4.0-75-generic SMP mod_unload modversions  parm:     minstrel_vht_only:Use only VHT rates when VHT is supported by sta. (bool) parm:           max_nullfunc_tries:Maximum nullfunc tx tries before disconnecting (reason 4). (int) parm:           max_probe_tries:Maximum probe tries before disconnecting (reason 4). (int) parm:           beacon_loss_count:Number of beacon intervals before we decide beacon was lost. (int) parm:           probe_wait_ms:Maximum time(ms) to wait for probe response before disconnecting (reason 4). (int) parm:           ieee80211_default_rc_algo:Default rate control algorithm for mac80211 to use (charp)

[cfg80211] filename:       /lib/modules/4.4.0-75-generic/kernel/net/wireless/cfg80211.ko description:    wireless configuration support license:        GPL author:         Johannes Berg srcversion:     48C96882CCDC964173BE1D1 depends:         intree:         Y vermagic:       4.4.0-75-generic SMP mod_unload modversions  parm:           bss_entries_limit:limit to number of scan BSS entries (per wiphy, default 1000) (int) parm:       ieee80211_regdom:IEEE 802.11 regulatory domain code (charp) parm:      cfg80211_disable_40mhz_24ghz:Disable 40MHz support in the 2.4GHz band (bool)

##### module parameters #################

[rtl8821ae] debug: 0 disable_watchdog: N fwlps: N int_clear: Y ips: Y msi: Y swenc: Y swlps: N

[mac80211] beacon_loss_count: 7 ieee80211_default_rc_algo: minstrel_ht max_nullfunc_tries: 2 max_probe_tries: 5 minstrel_vht_only: Y probe_wait_ms: 500

[cfg80211] bss_entries_limit: 1000 cfg80211_disable_40mhz_24ghz: N ieee80211_regdom: 00

##### /etc/modules ######################

##### modprobe options ##################

[/etc/modprobe.d/blacklist-ath_pci.conf] blacklist ath_pci

[/etc/modprobe.d/blacklist.conf] blacklist evbug blacklist usbmouse blacklist usbkbd blacklist eepro100 blacklist de4x5 blacklist eth1394 blacklist snd_intel8x0m blacklist snd_aw2 blacklist i2c_i801 blacklist prism54 blacklist bcm43xx blacklist garmin_gps blacklist asus_acpi blacklist snd_pcsp blacklist pcspkr blacklist amd76x_edac

[/etc/modprobe.d/blacklist-rare-network.conf] alias net-pf-3 off alias net-pf-6 off alias net-pf-9 off alias net-pf-11 off alias net-pf-12 off alias net-pf-19 off alias net-pf-21 off alias net-pf-36 off

[/etc/modprobe.d/iwlwifi.conf] remove iwlwifi \ (/sbin/lsmod | grep -o
-e ^iwlmvm -e ^iwldvm -e ^iwlwifi | xargs /sbin/rmmod) \ && /sbin/modprobe -r mac80211

[/etc/modprobe.d/mlx4.conf] softdep mlx4_core post: mlx4_en

[/etc/modprobe.d/r8169.conf] options r8169 fwlps=N

[/etc/modprobe.d/rtl8821ae.conf] options rtl8821ae fwlps=N options rtl8821ae swenc=1

##### rc.local ##########################

exit 0

##### pm-utils ##########################

[/etc/pm/config.d/config] (644 root) service network-manager restart

[/etc/pm/sleep.d/10_resume_wifi] (755 root) case "${1}" in
    resume|thaw) nmcli nm sleep false ;; esac

##### udev rules ########################

Body:
for < = ubuntu 20.4

echo "options iwlwifi 11n_disable=1" | sudo tee /etc/modprobe.d/iwlwifi.conf
sudo modprobe -rfv iwldvm
sudo modprobe -rfv iwlwifi
sudo modprobe -v iwlwifi

or

sudo apt-get purge bcmwl-kernel-source
mkdir bcmwl && cd bcmwl
search to google ==>bcmwl-kernel-source_6.30.223.271+bdcom-0ubuntu7_amd64.deb
sudo dpkg -i bcmwl-kernel-source_6.30.223.271+bdcom-0ubuntu7_amd64.deb


来源: stackexchange

问:
Improve PostgreSQL query performance
When running this query in my server it's very slow, and I can't understand why. Can anyone help me figure it out?

Query:

```<code>SELECT
    "t_dat"."t_year" AS "c0",
    "t_dat"."t_month" AS "c1",
    "t_dat"."t_week" AS "c2",
    "t_dat"."t_day" AS "c3",
    "t_purs"."p_id" AS "c4",
    sum("t_purs"."days") AS "m0",
    sum("t_purs"."timecreated") AS "m1"
FROM "t_dat", "t_purs"
WHERE "t_purs"."created" = "t_dat"."t_key"
  AND "t_dat"."t_year" = 2013
  AND "t_dat"."t_month" = 3
  AND "t_dat"."t_week" = 9
  AND "t_dat"."t_day" IN (1,2)
  AND "t_purs"."p_id" IN (
      '4','15','18','19','20','29',
      '31','35','46','56','72','78')
GROUP BY
    "t_dat"."t_year",
    "t_dat"."t_month",
    "t_dat"."t_week",
    "t_dat"."t_day",
    "t_purs"."p_id"
</code>```

Explain Analyze:

```
HashAggregate  (cost=12252.04..12252.04 rows=1 width=28) (actualtime=10212.374..10212.384 rows=10 loops=1)
  ->  Nested Loop  (cost=0.00..12252.03 rows=1 width=28) (actual time=3016.006..10212.249 rows=14 loops=1)
        Join Filter: (t_dat.t_key = t_purs.created)
        ->  Seq Scan on t_dat  (cost=0.00..129.90 rows=1 width=20) (actual time=0.745..2.040 rows=48 loops=1)
              Filter: ((t_day = ANY ('{1,2}'::integer[])) AND (t_year = 2013) AND (t_month = 3) AND (t_week = 9))
        ->  Seq Scan on t_purs  (cost=0.00..12087.49 rows=9900 width=16) (actual time=0.018..201.630 rows=14014 loops=48)
              Filter: (p_id = ANY ('{4,15,18,19,20,29,31,35,46,56,72,78}'::integer[]))
Total runtime: 10212.470 ms
```

答:
Your query is having sequential scans on t_purs and t_dat. Creating the appropriate indexes will help you make this query faster and avoid sequential scans.

```<code>create index index_name on t_purs(created) where created is not null;
create index index_name on t_dat using btree(t_key, t_year, t_month, t_week, t_day)
</code>```

Run explain analyze after running the above two queries. You'll see the plan time and execution time will be reduced.

ContentLicense: CC BY-SA 3.0

Title: Improve PostgreSQL query performance

Body:
When running this query in my server it's very slow, and I can't understand why. Can anyone help me figure it out?

Query:

SELECT
    "t_dat"."t_year" AS "c0",
    "t_dat"."t_month" AS "c1",
    "t_dat"."t_week" AS "c2",
    "t_dat"."t_day" AS "c3",
    "t_purs"."p_id" AS "c4",
    sum("t_purs"."days") AS "m0",
    sum("t_purs"."timecreated") AS "m1"
FROM "t_dat", "t_purs"
WHERE "t_purs"."created" = "t_dat"."t_key"
  AND "t_dat"."t_year" = 2013
  AND "t_dat"."t_month" = 3
  AND "t_dat"."t_week" = 9
  AND "t_dat"."t_day" IN (1,2)
  AND "t_purs"."p_id" IN (
      '4','15','18','19','20','29',
      '31','35','46','56','72','78')
GROUP BY
    "t_dat"."t_year",
    "t_dat"."t_month",
    "t_dat"."t_week",
    "t_dat"."t_day",
    "t_purs"."p_id"

Explain Analyze:

HashAggregate  (cost=12252.04..12252.04 rows=1 width=28) (actualtime=10212.374..10212.384 rows=10 loops=1)
  ->  Nested Loop  (cost=0.00..12252.03 rows=1 width=28) (actual time=3016.006..10212.249 rows=14 loops=1)
        Join Filter: (t_dat.t_key = t_purs.created)
        ->  Seq Scan on t_dat  (cost=0.00..129.90 rows=1 width=20) (actual time=0.745..2.040 rows=48 loops=1)
              Filter: ((t_day = ANY ('{1,2}'::integer[])) AND (t_year = 2013) AND (t_month = 3) AND (t_week = 9))
        ->  Seq Scan on t_purs  (cost=0.00..12087.49 rows=9900 width=16) (actual time=0.018..201.630 rows=14014 loops=48)
              Filter: (p_id = ANY ('{4,15,18,19,20,29,31,35,46,56,72,78}'::integer[]))
Total runtime: 10212.470 ms

Body:
Your query is having sequential scans on t_purs and t_dat. Creating the appropriate indexes will help you make this query faster and avoid sequential scans.

create index index_name on t_purs(created) where created is not null;
create index index_name on t_dat using btree(t_key, t_year, t_month, t_week, t_day)

Run explain analyze after running the above two queries. You'll see the plan time and execution time will be reduced.

ContentLicense: CC BY-SA 4.0


答:
Here is the code I wrote. It's somewhat long but it works. May be you (or someone) can modify and make it shorter.

# Transforming factors_df
df = factors_df.pivot(columns=["category", "option"])
df.reset_index(inplace=True)
# Renaming column names for each combination of option and category
df.columns = [s3 + s2 + s1 for (s1, s2, s3) in df.columns.to_list()]
df.drop(columns=["index"], inplace=True)
# Flattening to a single row to be able to apply formula
df = pd.DataFrame(df.max()).T
# Merging input with transformed factors data
input_df["tmp"] = 1
df["tmp"] = 1
df = pd.merge(input_df, df, on="tmp", how="left")
df.drop("tmp", axis=1, inplace=True)

# Calculating values for opt_1 using the formula
df["opt_1_value"] = (
    df["l"]
    * df["N_cars"]
    * (
        (
            df["opt_1Acategory %"]
            / 100
            * (
                df["opt_1Aa"] * df["v"] * df["v"]
                + df["opt_1Ab"] * df["v"]
                + df["opt_1Ac"]
                + df["opt_1Ad"] / df["v"]
            )
            / (
                (
                    df["opt_1Ae"] * df["v"] * df["v"]
                    + df["opt_1Af"] * df["v"]
                    + df["opt_1Ag"]
                )
                * (1 - df["opt_1Ah"])
            )
        )
        + (
            df["opt_1Bcategory %"]
            / 100
            * (
                df["opt_1Ba"] * df["v"] * df["v"]
                + df["opt_1Bb"] * df["v"]
                + df["opt_1Bc"]
                + df["opt_1Bd"] / df["v"]
            )
            / (
                (
                    df["opt_1Be"] * df["v"] * df["v"]
                    + df["opt_1Bf"] * df["v"]
                    + df["opt_1Bg"]
                )
                * (1 - df["opt_1Bh"])
            )
        )
        + (
            df["opt_1Ccategory %"]
            / 100
            * (
                df["opt_1Ca"] * df["v"] * df["v"]
                + df["opt_1Cb"] * df["v"]
                + df["opt_1Cc"]
                + df["opt_1Cd"] / df["v"]
            )
            / (
                (
                    df["opt_1Ce"] * df["v"] * df["v"]
                    + df["opt_1Cf"] * df["v"]
                    + df["opt_1Cg"]
                )
                * (1 - df["opt_1Ch"])
            )
        )
    )
)

# Calculating values for opt_2 using the formula
df["opt_2_value"] = (
    df["l"]
    * df["N_cars"]
    * (
        (
            df["opt_2Acategory %"]
            / 100
            * (
                df["opt_2Aa"] * df["v"] * df["v"]
                + df["opt_2Ab"] * df["v"]
                + df["opt_2Ac"]
                + df["opt_2Ad"] / df["v"]
            )
            / (
                (
                    df["opt_2Ae"] * df["v"] * df["v"]
                    + df["opt_2Af"] * df["v"]
                    + df["opt_2Ag"]
                )
                * (1 - df["opt_2Ah"])
            )
        )
        + (
            df["opt_2Bcategory %"]
            / 100
            * (
                df["opt_2Ba"] * df["v"] * df["v"]
                + df["opt_2Bb"] * df["v"]
                + df["opt_2Bc"]
                + df["opt_2Bd"] / df["v"]
            )
            / (
                (
                    df["opt_2Be"] * df["v"] * df["v"]
                    + df["opt_2Bf"] * df["v"]
                    + df["opt_2Bg"]
                )
                * (1 - df["opt_2Bh"])
            )
        )
        + (
            df["opt_2Ccategory %"]
            / 100
            * (
                df["opt_2Ca"] * df["v"] * df["v"]
                + df["opt_2Cb"] * df["v"]
                + df["opt_2Cc"]
                + df["opt_2Cd"] / df["v"]
            )
            / (
                (
                    df["opt_2Ce"] * df["v"] * df["v"]
                    + df["opt_2Cf"] * df["v"]
                    + df["opt_2Cg"]
                )
                * (1 - df["opt_2Ch"])
            )
        )
    )
)

# Calculating values for opt_3 using the formula
df["opt_3_value"] = (
    df["l"]
    * df["N_cars"]
    * (
        (
            df["opt_3Acategory %"]
            / 100
            * (
                df["opt_3Aa"] * df["v"] * df["v"]
                + df["opt_3Ab"] * df["v"]
                + df["opt_3Ac"]
                + df["opt_3Ad"] / df["v"]
            )
            / (
                (
                    df["opt_3Ae"] * df["v"] * df["v"]
                    + df["opt_3Af"] * df["v"]
                    + df["opt_3Ag"]
                )
                * (1 - df["opt_3Ah"])
            )
        )
        + (
            df["opt_3Bcategory %"]
            / 100
            * (
                df["opt_3Ba"] * df["v"] * df["v"]
                + df["opt_3Bb"] * df["v"]
                + df["opt_3Bc"]
                + df["opt_3Bd"] / df["v"]
            )
            / (
                (
                    df["opt_3Be"] * df["v"] * df["v"]
                    + df["opt_3Bf"] * df["v"]
                    + df["opt_3Bg"]
                )
                * (1 - df["opt_3Bh"])
            )
        )
        + (
            df["opt_3Ccategory %"]
            / 100
            * (
                df["opt_3Ca"] * df["v"] * df["v"]
                + df["opt_3Cb"] * df["v"]
                + df["opt_3Cc"]
                + df["opt_3Cd"] / df["v"]
            )
            / (
                (
                    df["opt_3Ce"] * df["v"] * df["v"]
                    + df["opt_3Cf"] * df["v"]
                    + df["opt_3Cg"]
                )
                * (1 - df["opt_3Ch"])
            )
        )
    )
)

# Removing unnecessary columns
df_final = df[["l", "N_cars", "v", "opt_1_value", "opt_2_value", "opt_3_value"]]
print(df_final)
```
<p>Output:</p>
<pre class="lang-py prettyprint-override"><code>      l  N_cars    v  opt_1_value  opt_2_value  opt_3_value
0   3.5    1000  100  1496.002370  1420.656629  1534.748740
1   5.7     500  110   750.997279   710.944885   767.411691
2  10.0     367  110   551.157686   521.754019   562.906668
3  11.1    1800   95  2685.551348  2554.477141  2756.164589
4   2.8     960  105  1439.467965  1364.815604  1475.082027
5   4.7     800  120  1206.116125  1138.614075  1229.225287
6  10.4     103  111   154.744048   146.445615   157.990346
7  20.1    1950  115  2933.825622  2773.297776  2990.828374
</code>```

Title: How to make a python program that calculates a result for each row of the input table?

Body:
I am trying to make a Python program that will calculate a result based on a formula, given factors and an input dataframe.

I have a number of cars (N_cars) on a given length of the road (l) and their average speed (v):

input_columns = ['l', 'N_cars', 'v']
input_data = [[3.5, 1000, 100], [5.7, 500, 110],
              [10, 367, 110], [11.1, 1800, 95],
              [2.8, 960, 105], [4.7, 800, 120],
              [10.4, 103, 111], [20.1, 1950, 115]]

input_df = pd.DataFrame(input_data, columns=input_columns)
input_df

      l  N_cars    v
0   3.5    1000  100
1   5.7     500  110
2  10.0     367  110
3  11.1    1800   95
4   2.8     960  105
5   4.7     800  120
6  10.4     103  111
7  20.1    1950  115

I also know the factors needed for the formula for each category of car, and I know the percentage of each category. I also have different options for each category (3 options that I have here are just an example, there are many more options).

factors_columns = ['category', 'category %', 'option', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h']
factors_data = [['A', 58, 'opt_1', 0.000011, 0.23521, 0.93847, 0.39458, 0.00817, 0.24566, 0.0010, 0],
                ['A', 58, 'opt_2', 0.000011, 0.23521, 0.93145, 0.39458, 0.00467, 0.24566, 0.0010, 0],
                ['A', 58, 'opt_3', 0.000011, 0.23521, 0.93145, 0.39458, 0.00467, 0.24566, 0.0010, 0],
                ['B', 22, 'opt_1', 0.002452, 0.48327, 0.83773, 0.92852, 0.00871, 0.29568, 0.0009, 0.02],
                ['B', 22, 'opt_2', 0.002899, 0.49327, 0.83773, 0.92852, 0.00871, 0.30468, 0.0009, 0.02],
                ['B', 22, 'opt_3', 0.002452, 0.48327, 0.83773, 0.92852, 0.00771, 0.29568, 0.0119, 0.01],
                ['C', 17, 'opt_1', 0.082583, 0.39493, 0.02462, 0.82714, 0.00918, 0.28572, 0.0012, 0],
                ['C', 17, 'opt_2', 0.072587, 0.35493, 0.02852, 0.82723, 0.00912, 0.29572, 0.0018, 0],
                ['C', 17, 'opt_3', 0.082583, 0.39493, 0.02852, 0.82714, 0.00962, 0.28572, 0.0012, 0.01],
                ['D', 3, 'opt_1', 0.018327, 0.32342, 0.82529, 0.92752, 0.00988, 0.21958, 0.0016, 0],
                ['D', 3, 'opt_2', 0.014427, 0.32342, 0.82729, 0.92752, 0.00968, 0.22558, 0.0026, 0],
                ['D', 3, 'opt_3', 0.018327, 0.32342, 0.82729, 0.94452, 0.00988, 0.21258, 0.0016, 0]]

factors_df = pd.DataFrame(factors_data, columns=factors_columns)
factors_df

   category  category % option         a        b        c        d        e        f       g     h
0         A          58  opt_1  0.000011  0.23521  0.93847  0.39458  0.00817  0.24566  0.0010  0.00
1         A          58  opt_2  0.000011  0.23521  0.93145  0.39458  0.00467  0.24566  0.0010  0.00
2         A          58  opt_3  0.000011  0.23521  0.93145  0.39458  0.00467  0.24566  0.0010  0.00
3         B          22  opt_1  0.002452  0.48327  0.83773  0.92852  0.00871  0.29568  0.0009  0.02
4         B          22  opt_2  0.002899  0.49327  0.83773  0.92852  0.00871  0.30468  0.0009  0.02
5         B          22  opt_3  0.002452  0.48327  0.83773  0.92852  0.00771  0.29568  0.0119  0.01
6         C          17  opt_1  0.082583  0.39493  0.02462  0.82714  0.00918  0.28572  0.0012  0.00
7         C          17  opt_2  0.072587  0.35493  0.02852  0.82723  0.00912  0.29572  0.0018  0.00
8         C          17  opt_3  0.082583  0.39493  0.02852  0.82714  0.00962  0.28572  0.0012  0.01
9         D           3  opt_1  0.018327  0.32342  0.82529  0.92752  0.00988  0.21958  0.0016  0.00
10        D           3  opt_2  0.014427  0.32342  0.82729  0.92752  0.00968  0.22558  0.0026  0.00
11        D           3  opt_3  0.018327  0.32342  0.82729  0.94452  0.00988  0.21258  0.0016  0.00

For each option (opt_1, opt_2, opt_3), I have to calculate the result based on this formula (factors are taken from the factors table, but v is coming from the input table):

formula = ( (a*v*v) + (b*v) + c + (d/v) )  /  ( (e*v*v) + (f*v) + g) * (1 - h)
result = l * N_cars * formula

However, I have to take into account the percentage of each category of car. For each row of the input_df I have to perform the calculations three times, once for each of the three options. For example, for the index 0 of input_df, I have N_cars=1000, v=100 and l=3.5, the output should be something like this:

# for opt_1:
result = 3.5 * 1000 * ((58% of category A {formula for index 0 of factors_df}) +
                       (22% of category B {formula for index 3 of factors_df) +
                       (17% of category C {formula for index 6 of factors_df}) +
                       (3% of category D {formula for index 9 of factors_df}) )

# for opt_2:
result = 3.5 * 1000 * ((58% of category A {formula for index 1 of factors_df}) +
                       (22% of category B {formula for index 4 of factors_df) +
                       (17% of category C {formula for index 7 of factors_df}) +
                       (3% of category D {formula for index 10 of factors_df}) )

# for opt_3:
result = 3.5 * 1000 * ((58% of category A {formula for index 2 of factors_df}) +
                       (22% of category B {formula for index 5 of factors_df) +
                       (17% of category C {formula for index 8 of factors_df}) +
                       (3% of category D {formula for index 11 of factors_df}) )

So, as an output, for each of the rows in input_df, I should have three results, one for each of the three options.

I can do the calculation manually for each step, but what I am having troubles with is to make a loop that does it automatically for each input row and all 3 options and then passes to the next input row and so on until the last input row.

Body:
Here is the code I wrote. It's somewhat long but it works. May be you (or someone) can modify and make it shorter.

# Transforming factors_df
df = factors_df.pivot(columns=["category", "option"])
df.reset_index(inplace=True)
# Renaming column names for each combination of option and category
df.columns = [s3 + s2 + s1 for (s1, s2, s3) in df.columns.to_list()]
df.drop(columns=["index"], inplace=True)
# Flattening to a single row to be able to apply formula
df = pd.DataFrame(df.max()).T
# Merging input with transformed factors data
input_df["tmp"] = 1
df["tmp"] = 1
df = pd.merge(input_df, df, on="tmp", how="left")
df.drop("tmp", axis=1, inplace=True)

# Calculating values for opt_1 using the formula
df["opt_1_value"] = (
    df["l"]
    * df["N_cars"]
    * (
        (
            df["opt_1Acategory %"]
            / 100
            * (
                df["opt_1Aa"] * df["v"] * df["v"]
                + df["opt_1Ab"] * df["v"]
                + df["opt_1Ac"]
                + df["opt_1Ad"] / df["v"]
            )
            / (
                (
                    df["opt_1Ae"] * df["v"] * df["v"]
                    + df["opt_1Af"] * df["v"]
                    + df["opt_1Ag"]
                )
                * (1 - df["opt_1Ah"])
            )
        )
        + (
            df["opt_1Bcategory %"]
            / 100
            * (
                df["opt_1Ba"] * df["v"] * df["v"]
                + df["opt_1Bb"] * df["v"]
                + df["opt_1Bc"]
                + df["opt_1Bd"] / df["v"]
            )
            / (
                (
                    df["opt_1Be"] * df["v"] * df["v"]
                    + df["opt_1Bf"] * df["v"]
                    + df["opt_1Bg"]
                )
                * (1 - df["opt_1Bh"])
            )
        )
        + (
            df["opt_1Ccategory %"]
            / 100
            * (
                df["opt_1Ca"] * df["v"] * df["v"]
                + df["opt_1Cb"] * df["v"]
                + df["opt_1Cc"]
                + df["opt_1Cd"] / df["v"]
            )
            / (
                (
                    df["opt_1Ce"] * df["v"] * df["v"]
                    + df["opt_1Cf"] * df["v"]
                    + df["opt_1Cg"]
                )
                * (1 - df["opt_1Ch"])
            )
        )
    )
)

# Calculating values for opt_2 using the formula
df["opt_2_value"] = (
    df["l"]
    * df["N_cars"]
    * (
        (
            df["opt_2Acategory %"]
            / 100
            * (
                df["opt_2Aa"] * df["v"] * df["v"]
                + df["opt_2Ab"] * df["v"]
                + df["opt_2Ac"]
                + df["opt_2Ad"] / df["v"]
            )
            / (
                (
                    df["opt_2Ae"] * df["v"] * df["v"]
                    + df["opt_2Af"] * df["v"]
                    + df["opt_2Ag"]
                )
                * (1 - df["opt_2Ah"])
            )
        )
        + (
            df["opt_2Bcategory %"]
            / 100
            * (
                df["opt_2Ba"] * df["v"] * df["v"]
                + df["opt_2Bb"] * df["v"]
                + df["opt_2Bc"]
                + df["opt_2Bd"] / df["v"]
            )
            / (
                (
                    df["opt_2Be"] * df["v"] * df["v"]
                    + df["opt_2Bf"] * df["v"]
                    + df["opt_2Bg"]
                )
                * (1 - df["opt_2Bh"])
            )
        )
        + (
            df["opt_2Ccategory %"]
            / 100
            * (
                df["opt_2Ca"] * df["v"] * df["v"]
                + df["opt_2Cb"] * df["v"]
                + df["opt_2Cc"]
                + df["opt_2Cd"] / df["v"]
            )
            / (
                (
                    df["opt_2Ce"] * df["v"] * df["v"]
                    + df["opt_2Cf"] * df["v"]
                    + df["opt_2Cg"]
                )
                * (1 - df["opt_2Ch"])
            )
        )
    )
)

# Calculating values for opt_3 using the formula
df["opt_3_value"] = (
    df["l"]
    * df["N_cars"]
    * (
        (
            df["opt_3Acategory %"]
            / 100
            * (
                df["opt_3Aa"] * df["v"] * df["v"]
                + df["opt_3Ab"] * df["v"]
                + df["opt_3Ac"]
                + df["opt_3Ad"] / df["v"]
            )
            / (
                (
                    df["opt_3Ae"] * df["v"] * df["v"]
                    + df["opt_3Af"] * df["v"]
                    + df["opt_3Ag"]
                )
                * (1 - df["opt_3Ah"])
            )
        )
        + (
            df["opt_3Bcategory %"]
            / 100
            * (
                df["opt_3Ba"] * df["v"] * df["v"]
                + df["opt_3Bb"] * df["v"]
                + df["opt_3Bc"]
                + df["opt_3Bd"] / df["v"]
            )
            / (
                (
                    df["opt_3Be"] * df["v"] * df["v"]
                    + df["opt_3Bf"] * df["v"]
                    + df["opt_3Bg"]
                )
                * (1 - df["opt_3Bh"])
            )
        )
        + (
            df["opt_3Ccategory %"]
            / 100
            * (
                df["opt_3Ca"] * df["v"] * df["v"]
                + df["opt_3Cb"] * df["v"]
                + df["opt_3Cc"]
                + df["opt_3Cd"] / df["v"]
            )
            / (
                (
                    df["opt_3Ce"] * df["v"] * df["v"]
                    + df["opt_3Cf"] * df["v"]
                    + df["opt_3Cg"]
                )
                * (1 - df["opt_3Ch"])
            )
        )
    )
)

# Removing unnecessary columns
df_final = df[["l", "N_cars", "v", "opt_1_value", "opt_2_value", "opt_3_value"]]
print(df_final)

Output:

      l  N_cars    v  opt_1_value  opt_2_value  opt_3_value
0   3.5    1000  100  1496.002370  1420.656629  1534.748740
1   5.7     500  110   750.997279   710.944885   767.411691
2  10.0     367  110   551.157686   521.754019   562.906668
3  11.1    1800   95  2685.551348  2554.477141  2756.164589
4   2.8     960  105  1439.467965  1364.815604  1475.082027
5   4.7     800  120  1206.116125  1138.614075  1229.225287
6  10.4     103  111   154.744048   146.445615   157.990346
7  20.1    1950  115  2933.825622  2773.297776  2990.828374


问:
Github Action create a new file for each value in an array
I'm trying to create a new file for each value in an array:

This is what I have so far and it successfully creates a file based on a GET and pushes it into my repo. Now I would like it to do it for multiple files at once. How do I get it to iterate over the result of myRequest if it were an array and create a file for each value and then finally commit and push them all?

```<code>name: Manual workflow
on:
  workflow_dispatch:
jobs:
  makefiles:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    - uses: actions/checkout@v2
    - name: Getting
      uses: fjogeleit/http-request-action@master
      id: myRequest
      with:
        url: 'https://domain/api/file'
        method: 'GET'
    - name: Show File
      run: echo ${{ steps.myRequest.outputs.response }}
    - name: Create A File
      uses: 1arp/create-a-file-action@0.2
      with:
        path: 'src'
        file: 'foo.bar'
        content: ${{steps.myRequest.outputs.response}}
    - name: final commit
      uses: zwaldowski/git-commit-action@v1
      id: git_commit
    - name: show
      run: echo "${{ steps.git_commit.outputs.sha }}"
</code>```

The output of my request is an encoded JSON:

```<code>[
{
  title: "foo",
  body: "bar"
},
{
  title: "foo",
  body: "bar"
},
{
  title: "foo",
  body: "bar"
}
]
</code>```

答:
If your response is an Array, you can pass it to next step using toJson, fromJSON.
This example should be helpful:

```<code>name: build
on: push
jobs:
  job1:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - id: set-matrix
      run: |
        JSON=$(cat ./your.json)
        echo "::set-output name=matrix::${JSON//'%'/'%25'}"

  job2:
    needs: job1
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{fromJSON(needs.job1.outputs.matrix)}}
    steps:
      - run: build
</code>```

I don't know exact format of your file and how those files are written down - but as long as you can make Array out of them, it should be easy to use toJSON to get proper output format.

One way is do use jq to generate JSON in bash, or even better make your https://domain/api/file script to generate JSON as an output format - easiest solution imho.

If you don't want to use it directly into matrix, you can read it also like this:

```<code>job2:
    needs: job1
    runs-on: ubuntu-latest
    steps:
      - run: |
         echo "${{ fromJSON(needs.job1.outputs.matrix).jsonKey}}"
</code>```

For bash iterations, use a bash syntax:

```<code>for i in "${array[@]}"
do:
done
</code>```

Body:
I'm trying to create a new file for each value in an array:

This is what I have so far and it successfully creates a file based on a GET and pushes it into my repo. Now I would like it to do it for multiple files at once. How do I get it to iterate over the result of myRequest if it were an array and create a file for each value and then finally commit and push them all?

name: Manual workflow
on:
  workflow_dispatch:
jobs:
  makefiles:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest

    # Steps represent a sequence of tasks that will be executed as part of the job
    steps:
    - uses: actions/checkout@v2
    - name: Getting
      uses: fjogeleit/http-request-action@master
      id: myRequest
      with:
        url: 'https://domain/api/file'
        method: 'GET'
    - name: Show File
      run: echo ${{ steps.myRequest.outputs.response }}
    - name: Create A File
      uses: 1arp/create-a-file-action@0.2
      with:
        path: 'src'
        file: 'foo.bar'
        content: ${{steps.myRequest.outputs.response}}
    - name: final commit
      uses: zwaldowski/git-commit-action@v1
      id: git_commit
    - name: show
      run: echo "${{ steps.git_commit.outputs.sha }}"

The output of my request is an encoded JSON:

[
{
  title: "foo",
  body: "bar"
},
{
  title: "foo",
  body: "bar"
},
{
  title: "foo",
  body: "bar"
}
]

Body:
If your response is an Array, you can pass it to next step using toJson, fromJSON.
This example should be helpful:

name: build
on: push
jobs:
  job1:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - id: set-matrix
      run: |
        JSON=$(cat ./your.json)
        echo "::set-output name=matrix::${JSON//'%'/'%25'}"

  job2:
    needs: job1
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{fromJSON(needs.job1.outputs.matrix)}}
    steps:
      - run: build

I don't know exact format of your file and how those files are written down - but as long as you can make Array out of them, it should be easy to use toJSON to get proper output format.

One way is do use jq to generate JSON in bash, or even better make your https://domain/api/file script to generate JSON as an output format - easiest solution imho.

If you don't want to use it directly into matrix, you can read it also like this:

job2:
    needs: job1
    runs-on: ubuntu-latest
    steps:
      - run: |
         echo "${{ fromJSON(needs.job1.outputs.matrix).jsonKey}}"

For bash iterations, use a bash syntax:

for i in "${array[@]}"
do:
done


问:
Spider grabs 1 item from each page and not all items
My spiders seems to only grab about 1 job-listing from each webpage. When I remove parse_jobs, and load_item() in parse I can extract all the job-listings for each page. So the issue is likely when it goes to parse_jobs and loads items, however I cannot seem to figure out the issue.

Here's what I have tried:

from scrapy.item import Field
import scrapy
from scrapy.loader import ItemLoader
from scrapy.crawler import CrawlerProcess
from itemloaders.processors import TakeFirst, MapCompose, Join
import pandas as pd
from collections import defaultdict
from scrapy_splash import SplashRequest

headers = {
    'authority': 'api2.branch.io',
    'sec-ch-ua': '" Not;A Brand";v="99", "Google Chrome";v="97", "Chromium";v="97"',
    'sec-ch-ua-mobile': '?0',
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36',
    'sec-ch-ua-platform': '"macOS"',
    'content-type': 'application/x-www-form-urlencoded',
    'accept': '*/*',
    'origin': 'https://www.reed.co.uk',
    'sec-fetch-site': 'cross-site',
    'sec-fetch-mode': 'cors',
    'sec-fetch-dest': 'empty',
    'referer': 'https://www.reed.co.uk/',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
}

class ReedItem(scrapy.Item):
    category = Field(output_processor = TakeFirst())
    salary = Field(output_processor = TakeFirst())
    title =  Field(output_processor = TakeFirst())
    organisation = Field(output_processor = TakeFirst())
    region = Field(input_processor = MapCompose(str.strip),
        output_processor = TakeFirst())
    items = Field(output_processor = TakeFirst())
    post = Field(input_processor = MapCompose(),
    output_processor = Join(" "))
    page_no = Field(output_processor = TakeFirst())

class ReedSpider(scrapy.Spider):
    name = 'reed'
    degrees={'upper': ['Finance','Accounting','Aeronautical-Engineering','Manufacturing-Engineering'],
            'degrees_entry': ['degree-finance-entry','degree-accounting-entry','degree-aeronautical-engineering-entry','degree-manufacturing-engineering-entry'],
            'degrees_graduate': ['degree-finance-graduate','degree-accounting-graduate','degree-aeronautical-engineering-graduate','degree-manufacturing-engineering-graduate'],
            'degrees': ['degree-finance','degree-accounting','degree-aeronautical-engineering','degree-manufacturing-engineering'],
            'graduate_entry': ['graduate-finance-entry','graduate-accounting-entry','graduate-aeronautical-engineering-entry','graduate-manufacturing-engineering-entry'],
            'graduate': ['graduate-finance','graduate-accounting','graduate-aeronautical-engineering','graduate-manufacturing-engineering'],
            'sector': ['Accountancy_Finance','Accountancy_Finance','Engineering_Manufacturing','Engineering_Manufacturing'],
            'degree_type': ['Accountancy_finance','Accountancy_finance','Aeronautical_Engineering','Manufacturing_Engineering']}
    degree = pd.DataFrame(degrees)
    start_urls = defaultdict(list)

    custom_settings = {
        'USER_AGENT':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
        'DOWNLOAD_DELAY':0.5,
        #'LOG_LEVEL':'INFO',

    }

    def start_requests(self):
        for degrees, degrees_entry,degrees_graduate, graduate_entry, graduate,sector  in zip(self.degree.degrees,self.degree.degrees_entry,self.degree.degrees_graduate,self.degree.graduate_entry,self.degree.graduate, self.degree.sector):
            self.start_urls[sector].append(f'https://www.reed.co.uk/jobs/{degrees}-jobs')
            self.start_urls[sector].append(f'https://www.reed.co.uk/jobs/{degrees_entry}-jobs')
            self.start_urls[sector].append(f'https://www.reed.co.uk/jobs/{graduate_entry}-jobs')
            self.start_urls[sector].append(f'https://www.reed.co.uk/jobs/{degrees_graduate}-jobs')
            self.start_urls[sector].append(f'https://www.reed.co.uk/jobs/{graduate}-jobs')

        for items, urls in self.start_urls.items():
            for url in urls:
                yield scrapy.Request(
                    url = url,
                    headers=headers,
                    callback = self.parse,
                    cb_kwargs = {
                        'items':items,
                        'page_no':0
                    }
            )
    def parse(self, response, items, page_no):
        container = response.xpath("//div[@class='row search-results']")
        for lists in container:
            page_no += 1
            loader = ItemLoader(ReedItem(), selector = lists)
            loader.add_value('items', items)
            loader.add_xpath('region', ".//div[@class='metadata']//ul//li[@class='location']//text()")
            loader.add_value('page_no', page_no)
            loader.add_xpath('category', "//div[@class='col-sm-11 col-xs-12 page-title']//h1/text()")
            loader.add_xpath('title', './/h3[@class="title"]/a/@title')
            loader.add_xpath('salary', './/li[@class="salary"]/text()')
            loader.add_xpath('organisation', './/a[@class="gtmJobListingPostedBy"]/text()')
            links = response.xpath('.//h3[@class="title"]/a/@href').get()
            yield response.follow(
                response.urljoin(links),
                callback = self.parse_jobs,
                cb_kwargs = {
                    'loader':loader
                })
        next_page = response.xpath('//a[@id="nextPage"]/@href').get()
        if next_page:
            yield response.follow(
                    url = next_page,
                    callback = self.parse,
                    headers=headers,
                    cb_kwargs = {
                        'items':items,
                        'page_no':page_no
                    })
    def parse_jobs(self, response, loader):
        loader.add_value('post',response.xpath('(//span[@itemprop="description"]/p/text()) | (//span[@itemprop="description"]/p//text())  | (//span[@itemprop="description"]/ul//li/text())').getall())
        yield loader.load_item()

process = CrawlerProcess(
    settings = {
        'FEED_URI':'reed_jobs_post.jl',
        'FEED_FORMAT':'jsonlines'
    }
)
process.crawl(ReedSpider)
process.start()
```

答:
I commented out the lines I changed.

    def parse(self, response, items, page_no):
        # container = response.xpath("//div[@class='row search-results']")
        container = response.xpath("//div[@class='row search-results']//article")
        page_no += 1
        for lists in container:
            # page_no += 1
            loader = ItemLoader(ReedItem(), selector=lists)
            loader.add_value('items', items)
            loader.add_xpath('region', ".//div[@class='metadata']//ul//li[@class='location']//text()")
            loader.add_value('page_no', page_no)
            loader.add_xpath('category', "//div[@class='col-sm-11 col-xs-12 page-title']//h1/text()")
            loader.add_xpath('title', './/h3[@class="title"]/a/@title')
            loader.add_xpath('salary', './/li[@class="salary"]/text()')
            loader.add_xpath('organisation', './/a[@class="gtmJobListingPostedBy"]/text()')
            # links = response.xpath('.//h3[@class="title"]/a/@href').get()
            links = lists.xpath('.//h3[@class="title"]/a/@href').get()
            yield response.follow(
                response.urljoin(links),
                callback=self.parse_jobs,
                cb_kwargs={
                    'loader': loader
                }
            )
        next_page = response.xpath('//a[@id="nextPage"]/@href').get()
        if next_page:
            yield response.follow(
                url=next_page,
                callback=self.parse,
                headers=headers,
                cb_kwargs={
                    'items': items,
                    'page_no': page_no
                })
```


问:
Built up string causing a syntax issue
I have a c# function that builds a string which in turn is used as a hyperlink to another page.  However, with some strings with single quotes it is causing a javascript error as shown here:

I'm calling the javascript function in the code behind as so

```<code>linkFullMatch.NavigateUrl = "javascript:showFullMatches(" + sb.ToString() + ")";
</code>```

the javascript is on the aspx function as so:

```<code><script>
function showFullMatches(url) {
    window.open(url, "_blank", "height=344,width=1100,scrollbars=yes,resizable=yes,toolbar=no,location=no,status=no,menubar=no,left=580,top=194");
}
</code>```

Any help would be greatly appreciated.  Any string that doesn't have a single quote in works fine and the page link opens as requested.

Rob

答:
You need to add an additional layer of quote marks to make the sb.ToString() value an JS string. Adjust your call like:

```<code>linkFullMatch.NavigateUrl = "javascript:showFullMatches('" + sb.ToString() + "')";
</code>```

Note the additional ' marks.

Title: Built up string causing a syntax issue

Body:
I have a c# function that builds a string which in turn is used as a hyperlink to another page.  However, with some strings with single quotes it is causing a javascript error as shown here:

I'm calling the javascript function in the code behind as so

linkFullMatch.NavigateUrl = "javascript:showFullMatches(" + sb.ToString() + ")";

the javascript is on the aspx function as so:

Body:
You need to add an additional layer of quote marks to make the sb.ToString() value an JS string. Adjust your call like:

linkFullMatch.NavigateUrl = "javascript:showFullMatches('" + sb.ToString() + "')";

Note the additional ' marks.


问:
Implementing generic digits function
I wanted to implement digits function which returns a vector of individual digits for a given number. For example, digits(12345) should return Vec[1, 2, 3, 4, 5].

After struggling with some small issues, I ended up with the following code:

```<code>fn digits<T, R>(mut n: T) -> Vec<R> where
    T: Num + Rem + DivAssign + FromPrimitive + Copy,
    R: Num + TryFrom<T>
{
    let zero = T::zero();
    let ten: T = T::from_i8(10).unwrap();
    let mut acc: Vec<R> = Vec::new();

    while n != zero {
        let d: R = R::try_from(n % ten).ok().unwrap();
        acc.push(d);
        n /= ten;
    }
    acc.reverse();
    acc
}
</code>```

And this seems to working fine with most primitive integer types. The problem is that the input type T should implement Copy, I can't use this function for BigInt or BigUint as Copy is not implemented for them.

If I remove Copy trait from T, the compilation fails due to various errors with such messages as bowwor of moved value: n or use of moved value: ten.

How can fix this so it can work with primitive integers as well as BigInt and BigUint?

答:
Anything that implements Copy does Clone too, also BigInt and BigUint implements Clone. So you could change the constraint to Clone instead and explicitly add any clone that you may need.

fn digits<T, R>(mut n: T) -> Vec<R> where
    T: Num + Rem + DivAssign + FromPrimitive + Clone,
    R: Num + TryFrom<T>
{
...
}
```

Playground

Title: Implementing generic digits function

Body:
I wanted to implement digits function which returns a vector of individual digits for a given number. For example, digits(12345) should return Vec[1, 2, 3, 4, 5].

After struggling with some small issues, I ended up with the following code:

fn digits<T, R>(mut n: T) -> Vec<R> where
    T: Num + Rem + DivAssign + FromPrimitive + Copy,
    R: Num + TryFrom<T>
{
    let zero = T::zero();
    let ten: T = T::from_i8(10).unwrap();
    let mut acc: Vec<R> = Vec::new();

    while n != zero {
        let d: R = R::try_from(n % ten).ok().unwrap();
        acc.push(d);
        n /= ten;
    }
    acc.reverse();
    acc
}

And this seems to working fine with most primitive integer types. The problem is that the input type T should implement Copy, I can't use this function for BigInt or BigUint as Copy is not implemented for them.

If I remove Copy trait from T, the compilation fails due to various errors with such messages as bowwor of moved value: n or use of moved value: ten.

How can fix this so it can work with primitive integers as well as BigInt and BigUint?

Body:
Anything that implements Copy does Clone too, also BigInt and BigUint implements Clone. So you could change the constraint to Clone instead and explicitly add any clone that you may need.

fn digits<T, R>(mut n: T) -> Vec<R> where
    T: Num + Rem + DivAssign + FromPrimitive + Clone,
    R: Num + TryFrom<T>
{
...
}

Playground


问:
How to connect mongodb instance region wise from my node js application server
I have one master instance of mongo db and 3 read only instances configured on different regions. Now whenever a request lands on to my node js application server, i want to connect to near by db instance to get data.</p>

答:
Option 1: ( better, since you dont care where the application is , the driver find the nearest members automatically)
To achieve this behaviour you need to set the read-preference to  nearest.
That way your driver will route your read request to much closest member depending on periodical latency check.

Option 2:
You define custom tags for members known in advance are in the application region and use those tags when read request are sent.