SWITCH Cloud Blog

Starting 1000 instances on SWITCHengines



openstack server create \
--image "Ubuntu Xenial 16.04 (SWITCHengines)" \
--flavor c1.small \
--network demo-network \
--user-data cloud-init.txt \
--key-name mykey \
--min 100 \
--max 100 test



post: [ hostname, fqdn ]




openstack console log show


We changed our nova.conf as follows:



This command is called everytime a neutron port is created:

real 0m0.048s
user 0m0.000s
sys 0m0.016s


default via proto kernel scope link src
real 0m0.713s
user 0m0.472s
sys 0m0.172s

We had to change the agent section of neutron.conf

root_helper = sudo neutron-rootwrap /etc/neutron/rootwrap.conf



6 thoughts on “Starting 1000 instances on SWITCHengines

  1. Cool stuff – congrats!


  2. “For each created port the dhcp agent had to add a corresponding line to the file /var/lib/neutron/dhcp//host”

    Is there something missing for the line you added to that file?


    • You are right. I fixed the cut and paste error. The right path is /var/lib/neutron/dhcp/UUID/host and you have to replace UUID with the actual value of the UUID of the network you are working on. For example: /var/lib/neutron/dhcp/44a3aeb9-bb19-4f53-9ebf-c9318c4f2ebd/host


  3. Which release was this against? Pike, Ocata, older? Also, are you just running a single compute node or multiple and if so, how many? I’m assuming you’re using the libvirt driver too but which Neutron backend, ML2 OVS?

    Thanks for sharing this. I’m really interested in finding out how people are overcoming scale issues with scheduling and server create times in Nova, especially what improvements or regressions we’re making with cells v2 sharding and the Placement service starting in Ocata and further enhanced in Pike.


  4. Hello Matt,
    this work is with Openstack Newton. We are running 48 compute nodes. The neutron backend ML2 OVS and the virtualization is KVM/Libvirt.
    One important bit of information is that we are in Newton, but we did not yet create the Cell0. We are working just with 2 nova databases. The main DB and the nova api DB. Before upgrading to Ocata we will introduce the third Cell0 DB and I will be able to tell you if we see any performance regression.


    • Cool. FWIW cell0 shouldn’t have any impact during scheduling and creation of guests. It’s purpose is a place to put “dead” instances that never got scheduled properly, i.e. NoValidHost during scheduling. Those instances get put into ERROR state and shoved into cell0. For instances that get a host chosen by the scheduler, those go into cell1 (your normal nova database). When listing / showing instances from the API, we’ll pull them from nova_api (the build_requests table for in-progress instances that are being scheduled and not yet in a cell), cell0 (instances in ERROR state that never got scheduled successfully) and cell1 (the ‘normal’ nova database you’re used to).