Openstack Nova – InstanceNotFound – Modify the HardReboot function to recreate domain from XML if it doesnt exist

Ive not finished writing this article… If you are interested in discussing this further feel free to hit me up cory@hawkless.id.au

So in the process of managing an OpenStack cluster where you have Pet VM’s that are not treated like Cattle (Read about Pets vs Cattle here, TL;DR – Pets = Import VM that cant die, Cattle, you can easily kill the VM and replace it with another fresh;y built one) You almost certainly come across a situation where a VM for whatever reason wil get stuck into an error state.
So the VM existins in the database, you can see it in the console, but when you go to power it on it just fails.
You check the host that it’s assigned to and you find an error like this ‘..InstanceNotFound..’ UGH!? WHY?
I’m still not sure WHY OpenStack gets itself into this tangle and up until now Ive not been sure how to get it out either. The only solution has been to delete the Instance definition and create it from scratch, which can be a giant pain in the ass.

 

 

 

2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server [req-4e2f492f-e9a4-4bcf-82d9-34746b93a074 f2800cf724264988aab44aa21bf1dae4 40c2b46fb47c4ee7ac076b259c4e0814 - default default] Exception during message handling: InstanceNotFound: Instance ec88fe77-ce03-4cfb-a377-deef13cde2cc could not be found.
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server function_name, call_dict, binary)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 186, in decorated_function
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server "Error: %s", e, instance=instance)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 156, in decorated_function
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/utils.py", line 976, in decorated_function
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 202, in decorated_function
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4222, in resize_instance
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server self._revert_allocation(context, instance, migration)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server self.force_reraise()
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4219, in resize_instance
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server instance_type, clean_shutdown)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 4257, in _resize_instance
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server timeout, retry_interval)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 8032, in migrate_disk_and_power_off
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server disk_info = self._get_instance_disk_info(instance, block_device_info)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/driver.py", line 7809, in _get_instance_disk_info
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server guest = self._host.get_guest(instance)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 526, in get_guest
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server return libvirt_guest.Guest(self._get_domain(instance))
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/dist-packages/nova/virt/libvirt/host.py", line 546, in _get_domain
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server raise exception.InstanceNotFound(instance_id=instance.uuid)
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server InstanceNotFound: Instance ec88fe77-ce03-4cfb-a377-deef13cde2cc could not be found.
2020-05-25 15:19:11.240 79840 ERROR oslo_messaging.rpc.server
2020-05-25 15:19:15.655 79840 INFO nova.compute.resource_tracker [req-4e2f492f-e9a4-4bcf-82d9-34746b93a074 f2800cf724264988aab44aa21bf1dae4