Fix a CloudStack instance stuck in an incorrect state

In Apache CloudStack it is possible for the database to become out of sync with what is actually happening. For instance if there is a network issue preventing CloudStack from correctly connecting to the hypervisor or virtual machine and you try to shutdown or reboot a server, the action may not actually take place despite the state in the CloudStack database being modified.

This can cause the instance to show as being in the ‘starting’ or ‘stopping’ states for instance, however the virtual machine may already be fully booted or completely powered off. To fix this we can manually update the management database to the correct state.

In this example, our virtual machine is powered off. This was confirmed by checking through XenServer. CloudStack saw the state as being ‘stopping’, and while in this state the start button is not available preventing you from starting the stopped instance. To get the start button back through the user interface, we will modify the database.

Modifying the Apache CloudStack database

First log in to the database server where your Apache CloudStack management database is running, connect to MySQL and select the CloudStack management database as shown below.

mysql> use cloud;
Database changed

Now that the database is selected, we can view the information specific to instance 50 which is our example that is stuck in the stopping state in Apache CloudStack. The instance number should be available through the CloudStack user interface in the ‘Internal name’ field.

mysql> select * from vm_instance where id='50'\G
*************************** 1. row ***************************
                 id: 50
               name: server01
               uuid: f37a2528-68c5-69cb-3dca-e3ad2a48937e
      instance_name: i-11-50-VM
              state: Stopping
     vm_template_id: 1093
        guest_os_id: 99
private_mac_address: 05:00:7d:e0:04:09
 private_ip_address: 10.0.0.10
             pod_id: 2
     data_center_id: 2
            host_id: 25
       last_host_id: 25
           proxy_id: 6313
  proxy_assign_time: 2015-02-05 23:48:15
       vnc_password: 
         ha_enabled: 1
      limit_cpu_use: 0
       update_count: 18
        update_time: 2015-04-02 13:08:12
            created: 2015-02-05 23:46:05
            removed: NULL
               type: User
            vm_type: User
         account_id: 1
          domain_id: 1
service_offering_id: 1
     reservation_id: 59656330-19ce-41ef-87ef-a5771f637bac
    hypervisor_type: XenServer
1 row in set (0.00 sec)

As seen above the state is “Stopping”. Normally this will show while the virtual machine is in the process of stopping. In this instance, we checked Citrix XenCenter and confirmed that there are no running virtual machines listed as instance 50, confirming that this virtual machine is already in the stopped state.

While the virtual machine was actually stopped, the Apache CloudStack status did not update from stopping to stopped after about 30 minutes, so it was time to manually edit the database as shown below.

mysql> update vm_instance set state = 'stopped' where id = 50;

Now if you run your ‘select * from vm_instance’ query again you should see that the state is now stopped.

Once this has been completed, have a look at the instance through Apache CloudStack again and you should see that it now correctly shows as stopped which will bring the start button back. Now you can simply click the start button and the virtual machine should correctly power up. The option to start the instance up would not have been available while CloudStack thought the instance was in the process of shutting down.

Conclusion

With this method you can quickly repair any database inconsistencies that Apache CloudStack may have. If CloudStack thinks the instance is running, stopped, starting, or stopping but that is not the case you can update the value as required to match the actual state if you know what that is. This sort of action is only recommended if the state is not correctly updating and you have confirmed that it is not changing properly, I would suggest waiting for quite a while normally to see if the database will update correctly on it’s own before making your own manual changes.

Leave a Comment

NOTE - You can use these HTML tags and attributes:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>