KVM\Qemu\Openstack – Manage a live migration

virsh qemu-monitor-command {VMNAME} --pretty '{"execute":"migrate_cancel"}'

Allow Virsh more downtime(If it cant keepup with RAM utilization)

virsh migrate-setmaxdowntime VMNAME 2500

 

Check migration status

virsh domjobinfo instance-000002ac
Job type: Unbounded
Operation: Outgoing migration
Time elapsed: 1307956 ms
Data processed: 118.662 GiB
Data remaining: 9.203 MiB
Data total: 8.005 GiB
Memory processed: 118.662 GiB
Memory remaining: 9.203 MiB
Memory total: 8.005 GiB
Memory bandwidth: 41.294 MiB/s
Dirty rate: 35040 pages/s
Page size: 4096 bytes
Iteration: 197
Constant pages: 1751031
Normal pages: 31041965
Normal data: 118.416 GiB
Expected downtime: 3314 ms
Setup time: 70 ms

 

https://www.redhat.com/archives/libvirt-users/2014-January/msg00007.html
https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/abort-live-migration.html

 

https://www.server24.eu/private-cloud/complete-live-migration-vms-high-load/

 

Openstack – Manually edit VM

Find the host the VM is running on and the instance ID(Use console view to get instance ID)

cp /etc/libvirt/qemu/instance-0000030a.xml .
edit instance-0000030a.xml to be what you need it to be

While the VM is running (Warning, will crash the VM)

virsh destroy instance-0000030a
virsh undefine instance-0000030a
virsh define instance-0000030a.xml
virsh start instance-0000030a

Openstack Queens – Nova: Error creating new key pair

Ina fresh install of Oepnstack Queens I was having an issue generating new keypairs after a reboot f the Nova node.

The following error was logged in nova-api.log

A simple restart of the nova-api service resolved this issue for me

 

 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi [req-78070ded-b2cd-4e1a-8a51-8c082b50e0a4 f2800cf724264988aab44aa21bf1dae4 0d280551fd45414eace8211d6ac154c0 - default 252047b9b8504489a11d9230a8f9bf55] Unexpected exception in API method: InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=2147897744L, lib=128, func=101, reason=400, reason_text='error:80065190:lib(128):osrandom_rand_bytes:getrandom() initialization failed.'), _OpenSSLErrorWithText(code=67637251L, lib=4, func=129, reason=3, reason_text='error:04081003:rsa routines:RSA_BUILTIN_KEYGEN:BN lib')])
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 788, in wrapped
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108, in wrapper
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108, in wrapper
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/keypairs.py", line 112, in create
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return self._create(req, body)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/keypairs.py", line 133, in _create
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi context, user_id, name, key_type)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi function_name, call_dict, binary)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi self.force_reraise()
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi six.reraise(self.type_, self.value, self.tb)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return f(self, context, *args, **kw)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 5287, in create_key_pair
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi user_id, key_type)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 5322, in _generate_key_pair
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return crypto.generate_key_pair()
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/crypto.py", line 132, in generate_key_pair
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi key = paramiko.RSAKey.generate(bits)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/paramiko/rsakey.py", line 156, in generate
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi public_exponent=65537, key_size=bits, backend=default_backend()
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/primitives/asymmetric/rsa.py", line 119, in generate_private_key
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return backend.generate_rsa_private_key(public_exponent, key_size)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 362, in generate_rsa_private_key
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi self.openssl_assert(res == 1)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 106, in openssl_assert
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return binding._openssl_assert(self._lib, ok)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 75, in _openssl_assert
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi errors_with_text
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=2147897744L, lib=128, func=101, reason=400, reason_text='error:80065190:lib(128):osrandom_rand_bytes:getrandom() initialization failed.'), _OpenSSLErrorWithText(code=67637251L, lib=4, func=129, reason=3, reason_text='error:04081003:rsa routines:RSA_BUILTIN_KEYGEN:BN lib')])

Cinder – Resize root volume

To resize the root volume of an instance can be tricky, this worked for me

Note some commands, particularly “cinder reset-state” returned a HTTP 403, i had to run those as admin

 

[root@vm-os-ks01 ~]# source admin.rc
[root@vm-os-ks01 ~]# cinder reset-state --state available b35e676a-7593-458d-b6f4-cc60e075cfd4
[root@vm-os-ks01 ~]# nova volume-list
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| ID                                   | Status    | Display Name | Size | Volume Type | Attached to                          |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| e1072658-7399-4505-a563-a5b6ed062325 | in-use    |              | 10   | sata        | 11684d86-5e17-4760-a32b-26ed6c47f1c1 |
| b5ac679f-76ae-45aa-b6ae-c2d4d87a97d0 | in-use    |              | 10   | sata        | 65717578-a520-4c63-a3b4-63537bce2332 |
| c3c88bc1-a6b8-4bdc-b1b1-d399b0696be8 | in-use    |              | 10   | sata        | 815043a2-44fa-487e-9912-ecc015bf6a46 |
| bef780ff-ac9f-492d-9504-11062e30acdf | in-use    |              | 40   | sata        | 76d7a922-3b5e-4154-ab61-cfca7810369d |
| b35e676a-7593-458d-b6f4-cc60e075cfd4 | available |              | 40   | sata        | 933899fe-b218-4eff-b2b1-658ae8068350 |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
[root@vm-os-ks01 ~]# cinder extend b35e676a-7593-458d-b6f4-cc60e075cfd4 80
[root@vm-os-ks01 ~]# nova volume-list
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| ID                                   | Status    | Display Name | Size | Volume Type | Attached to                          |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
| e1072658-7399-4505-a563-a5b6ed062325 | in-use    |              | 10   | sata        | 11684d86-5e17-4760-a32b-26ed6c47f1c1 |
| b5ac679f-76ae-45aa-b6ae-c2d4d87a97d0 | in-use    |              | 10   | sata        | 65717578-a520-4c63-a3b4-63537bce2332 |
| c3c88bc1-a6b8-4bdc-b1b1-d399b0696be8 | in-use    |              | 10   | sata        | 815043a2-44fa-487e-9912-ecc015bf6a46 |
| bef780ff-ac9f-492d-9504-11062e30acdf | in-use    |              | 40   | sata        | 76d7a922-3b5e-4154-ab61-cfca7810369d |
| b35e676a-7593-458d-b6f4-cc60e075cfd4 | available |              | 80   | sata        | 933899fe-b218-4eff-b2b1-658ae8068350 |
+--------------------------------------+-----------+--------------+------+-------------+--------------------------------------+
[root@vm-os-ks01 ~]# cinder reset-state --state in-use b35e676a-7593-458d-b6f4-cc60e075cfd4
[root@vm-os-ks01 ~]# nova volume-list
+--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
| ID                                   | Status | Display Name | Size | Volume Type | Attached to                          |
+--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
| e1072658-7399-4505-a563-a5b6ed062325 | in-use |              | 10   | sata        | 11684d86-5e17-4760-a32b-26ed6c47f1c1 |
| b5ac679f-76ae-45aa-b6ae-c2d4d87a97d0 | in-use |              | 10   | sata        | 65717578-a520-4c63-a3b4-63537bce2332 |
| c3c88bc1-a6b8-4bdc-b1b1-d399b0696be8 | in-use |              | 10   | sata        | 815043a2-44fa-487e-9912-ecc015bf6a46 |
| bef780ff-ac9f-492d-9504-11062e30acdf | in-use |              | 40   | sata        | 76d7a922-3b5e-4154-ab61-cfca7810369d |
| b35e676a-7593-458d-b6f4-cc60e075cfd4 | in-use |              | 80   | sata        | 933899fe-b218-4eff-b2b1-658ae8068350 |
+--------------------------------------+--------+--------------+------+-------------+--------------------------------------+
[root@vm-os-ks01 ~]#