text: scheduler falsely reports share service down
  Bug Description
  Hi,
  with a low/default service_down_time in config and a high number (I've seen it with 5) of manila-share services it can happen that a service gets wrongly reported as down in scheduling.
  I believe this is because
  https:/
  collects the heartbeat data first and then loops over the services.
  E.g. the last service in the loop may be reached after the service_down_time time has passed, the service normally should have received a new heartbeat in the meantime, but the loop operates on old data and does not know.
  I propose to let service_is_up do a live check against the database each time, or at least make it configurable for the caller.
  I hope my explanation is understandable.
  Cheers,
  Maurice
  Hi Maurice,
  I started investigating this bug. Tried to reproduce this issue, but no success.
  I deployed a DevStack, then configured 8 LVMShareDriver backends and 7 NetAppDriver backends.
  Running `manila service-list` a couple seconds after restarting m-shr, I can see all the services up.
  The service_down_time is not set, so it's using the default value.
  Could you add some details about the environment and the steps you used to find this bug?
  Regards,
  Lucio
  Hi Lucio,
  thanks for investigating.
  Maybe it needs actual payload to be visible - I've seen it with 2000 shares and reporting those takes a while.
  Now that I think again, I remember especially the server_
  BR,
  Maurice
  Thanks for the details, Maurice.
  I managed to reproduce the issue with 500 shares using Dummy driver.
  Now I'll verify if your approach doing a live check would fix the issue.
  Cheers,
  Lucio
  Seems that `_update_
  While creating 500 shares, the manila-scheduler log starts printing "Share service is down." several times. However, running `manila service-list` still shows the manila-share service as `up`.
  But when I restart manila-share service, running `manila service-list` shows manila-share service as `down` while it's exporting its 500 shares.
  Actually, even modifying `_upodate_
  def _update_
  # Get resource usage across the available share nodes:
  topic = CONF.share_topic
  for service in share_services:
  # Get an updated state of the service
  host = updated_
  # Warn about down services and remove them from host_state_map
  if (not utils.service_
  ...
  But despite the warning message, the shares are being created successfully, and `manila service-list` showing manila share-service as `up`.
  It's only shown as `down` while exporting the shares, upon manila-share service restart.
  Sorry, please ignore the comments #4-#6.
  It's normal to manila-share service be shown as `down` for a while until re-exporting all the shares.
  If it still remains `down` after a long time after restarting, it should be another issue to be addressed in a new bug report.
  So I didn't manage to reproduce the issue yet.
  Hello Team,
  I believe I was able to reproduce this issue in my env.
  - I added 9 NetApp Share Backends to Manila(Backend details shown in the Session 5th output).
  - I am trying to create manila shares continuously in 3 different sessions.
  - In an another parallel session I am grepping the logs to see if any Backend is reported as down because of the above bug. -- 4th session
  - In an another session I am running "manila service-list | grep ontap | grep down" continuously. -- 5th session
  Note : The services were not restarted during this time frame.
  The services were restarted at least 20 mins before running this activity.
  I have captured the data for a 10 second timeframe (Fri Apr 12 06:20:39 EDT 2019) to Fri Apr 12 06:20:48 EDT 2019.
  Here is the session output from the 5th session.
  #######
  root@25-
  Fri Apr 12 06:20:39 EDT 2019
  root@25-
  +----+-
  | Id | Binary | Host | Zone | Status | State | Updated_at |
  +----+-
  | 1 | manila-share | 25-nareshtwo@london | manila-zone-0 | enabled | down | 2019-04-
  | 2 | manila-share | 25-nareshtwo@paris | manila-zone-1 | enabled | down | 2019-04-
  | 3 | manila-scheduler | 25-nareshtwo | nova | enabled | up | 2019-04-
  | 4 | manila-data | 25-nareshtwo | nova | enabled | up | 2019-04-
  | 5 | manila-share | 25-nareshtwo@ontap2 | nova | enabled | up | 2019-04-
  | 6 | manila-share | 25-nareshtwo@ontap6 | nova | enabled | up | 2019-04-
  | 7 | manila-share | 25-nareshtwo@
  | 8 | manila-share | 25-nareshtwo@
  | 9 | manila-share | 25-nareshtwo@
  | 10 | manila-share | 25-nareshtwo@ontap3 | nova | enabled | up | 2019-04-
  | 11 | manila-share | 25-nareshtwo@
  | 12 | manila-share | 25-nareshtwo@ontap4 | nova | enabled | up | 2019-04-
  | 13 | manila-share | 25-nareshtwo@
  +----+-
  root@25-
  root@25-
  root@25-
  This bug will be discussed under the Edge PTG topic