title: Rabbitmq nodes unable to restart : Job for rabbitmq-server.service failed because the control process exited with error code
body: <p>I am configuring ansible scripts for creating HA setup for rabbitmq with 3 nodes, after the whole script is executed it runs handlers to restart rabbitmq server, but it is failing to restart the nodes. Initially when i was running the script without copying the erlang cookie in the other nodes, it was running well, but when i copied erlang cookie in the other nodes, it is not restarting. </p>
  
  <p><strong>tasks/main.yml</strong></p>
  
  <pre><code>---
  - name: Install prerequisites
    apt:
      name: "{{ item }}"
      update_cache: yes
      state: present
    with_items:
     # - apt-transport-https
     # - ca-certificates
     # - python3-pip
     - curl
     - software-properties-common
  
  #- name: packages - install erlang
  #  apt:
  #    pkg: erlang
  #    update_cache: yes
  #    cache_valid_time: 3600
  #    state: latest
  
  - name: "add the official rabbitmq repository's key"
    apt_key:
      url: "https://www.rabbitmq.com/rabbitmq-release-signing-key.asc"
      state: present
    when: not rabbitmq_os_package
  
  - name: Add RabbitMQ APT repository
    apt_repository:
      repo: 'deb https://dl.bintray.com/rabbitmq-erlang/debian bionic erlang-22.x'
      state: present
  
  - name: Install RabbitMQ
    apt:
      deb: "https://github.com/rabbitmq/rabbitmq-server/releases/download/v3.8.3/rabbitmq-server_3.8.3-1_all.deb"
  
  - name: enable rabbitmq plugins
    rabbitmq_plugin:
      names: rabbitmq_management,rabbitmq_tracing,rabbitmq_federation,rabbitmq_shovel,rabbitmq_shovel_management
      state: enabled
    notify:
    - rabbitmq restart
  
  - name: Capturing Erlang Cookie On Master
    command: "cat {{ rabbitmq_erlang_cookie_file }}"
    become: true
    register: "rabbitmq_erlang_cookie"
    when: inventory_hostname == "rabbit-node1"
  
  - name: Check if rabbitmq_conf_vm_memory_high_watermark is under the recommended range
    fail:
      msg: "The recommended vm_memory_high_watermark range is 0.4 to 0.66."
    when: rabbitmq_conf_vm_memory_high_watermark &lt; 0.4 or rabbitmq_conf_vm_memory_high_watermark &gt; 0.66
  
  - name: copy the rabbitmq configuration file
    template:
      src=rabbitmq.config.j2
      dest={{ rabbitmq_config_file_path }}
      owner={{ rabbitmq_config_file_owner }}
      group={{ rabbitmq_config_file_group }}
      mode={{ rabbitmq_config_file_mode }}
      backup=yes
  
  #- name: generate rabbitmq environment-variables file
  #  template:
  #    src=rabbitmq-env.conf.j2
  #    dest={{ rabbitmq_env_variables_file_path }}
  #    owner={{ rabbitmq_config_file_owner }}
  #    group={{ rabbitmq_config_file_group }}
  #    mode={{ rabbitmq_config_file_mode }}
  #    backup=yes
  #  when: rabbitmq_conf_env is defined
  
  - name: alter number of system's open file if rabbitmq_system_number_open_files is set
    block:
      - name: check if value is under 500k
        fail:
          msg: "The number of open files in the system should not exceed 500K."
        when: rabbitmq_system_number_open_files &gt; 500000
  
      - name: make sure service.d exists
        file:
          path: "{{ rabbitmq_service_d_path }}"
          state: directory
  
      - name: change limit of system's number of open file
        template:
          src: limits.conf.j2
          dest: "{{ rabbitmq_service_d_path }}/limits.conf"
          backup: true
    when: rabbitmq_system_number_open_files is defined
  
  - name: Capturing Erlang Cookie On Master
    command: "cat {{ rabbitmq_erlang_cookie_file }}"
    become: true
    register: "rabbitmq_erlang_cookie"
    when: inventory_hostname == "rabbit-node1"
  
  - name: Setting Erlang Cookie Of Master on Non-Master
    set_fact:
      rabbitmq_erlang_cookie: "{{ hostvars['rabbit-node1']['rabbitmq_erlang_cookie']['stdout'] }}"
    when: inventory_hostname != "rabbit-node1"
  
  - name: Copy erlang cookie
    template:
      src: erlang.cookie.j2
      dest: "{{ rabbitmq_erlang_cookie_file }}"
      owner: rabbitmq
      group: rabbitmq
      mode: 0400
      # backing up in case the need to recover
      backup: yes
    become: true
    when: inventory_hostname != "rabbit-node1"
    notify:
    - rabbitmq restart
  
  - name: ensure the users are removed
    rabbitmq_user:
      user={{ item }}
      state=absent
    with_items: "{{ rabbitmq_users_remove }}"
  
  - name: ensure users exist
    rabbitmq_user:
      user: "{{ item }}"
      password: 
      vhost: /
      configure_priv: .*
      read_priv: .*
      write_priv: .*
      tags: "{{ rabbitmq_administrator_tag }}"
      state: present
    with_items: "{{ rabbitmq_users }}"
  
  - name: ensure rabbitmq server is running
    service:
      name: rabbitmq-server
      daemon_reload: true #changed systemd conf file
      state: started 
  </code></pre>
  
  <p>Here is the handlers file which runs at the end of the script.</p>
  
  <p><strong>handlers/main.yml</strong></p>
  
  <pre><code>---
  
  - name: rabbitmq restart
    service:
      name: rabbitmq-server
      state: restarted
  </code></pre>
  
  <p><strong>error</strong></p>
  
  <pre><code>fatal: [rabbit-node2]: FAILED! =&gt; {"changed": false, "msg": "Unable to restart service rabbitmq-server: Job for rabbitmq-server.service failed because the control process exited with error code.\nSee \"systemctl status rabbitmq-server.service\" and \"journalctl -xe\" for details.\n"}                                                                     
  
  fatal: [rabbit-node3]: FAILED! =&gt; {"changed": false, "msg": "Unable to restart service rabbitmq-server: Job for rabbitmq-server.service failed because the control process exited with error code.\nSee \"systemctl status rabbitmq-server.service\" and \"journalctl -xe\" for details.\n"}
  </code></pre>
  
  <p>By checking systemctl status in nodes, it displays the following message:</p>
  
  <pre><code>● rabbitmq-server.service - RabbitMQ broker
  Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/rabbitmq-server.service.d
  └─limits.conf
  Active: activating (start) since Thu 2020-04-16 09:27:18 UTC; 889ms ago
  Process: 929 ExecStop=/usr/lib/rabbitmq/bin/rabbitmqctl shutdown (code=exited
  status=78)
  Main PID: 7865 (rabbitmq-server)
  Tasks: 23 (limit: 4703)
  CGroup: /system.slice/rabbitmq-server.service
  ├─7865 /bin/sh /usr/lib/rabbitmq/bin/rabbitmq-server
  ├─7981 /usr/lib/erlang/erts-10.7.1/bin/beam.smp -- -root /usr/lib/erlang -progname erl -- -home /var/lib/rabbitmq -- -pa /usr/lib/rabbitmq/lib/rabbitmq_server-3.8.3/ebin -boot start_clean -noshell -no
  ├─7991 /usr/lib/erlang/erts-10.7.1/bin/epmd -daemon
  └─7995 erl_child_setup 65535 
  </code></pre>
  
  <p>By applying journalctl -xe command i got the below message:</p>
  
  <pre><code>rabbit-node2 rabbitmq-server[16137]: Crash dump is being written to: /var/log/rabbitmq/erl_crash.dump...done
  rabbit-node2 systemd[1]: rabbitmq-server.service: Main process exited, code=exited, status=1/FAILURE
  rabbit-node2 systemd[1]: rabbitmq-server.service: Failed with result 'exit-code'.
  rabbit-node2 systemd[1]: Failed to start RabbitMQ broker.
  </code></pre>