Useful gnocchi commmands

resource update -a customerID:"customer1" 729389e687504dbf8b2c4aead14a607c --type my_consumable_resource

resource show 4f42c5e60813488ea166206c5bfc10cd --type my_consumable_resource

resource search project_id=729389e687504dbf8b2c4aead14a607c

KVM\Qemu\Openstack – Manage a live migration

virsh qemu-monitor-command {VMNAME} --pretty '{"execute":"migrate_cancel"}'

Allow Virsh more downtime(If it cant keepup with RAM utilization)

virsh migrate-setmaxdowntime VMNAME 2500

 

Check migration status

virsh domjobinfo instance-000002ac
Job type: Unbounded
Operation: Outgoing migration
Time elapsed: 1307956 ms
Data processed: 118.662 GiB
Data remaining: 9.203 MiB
Data total: 8.005 GiB
Memory processed: 118.662 GiB
Memory remaining: 9.203 MiB
Memory total: 8.005 GiB
Memory bandwidth: 41.294 MiB/s
Dirty rate: 35040 pages/s
Page size: 4096 bytes
Iteration: 197
Constant pages: 1751031
Normal pages: 31041965
Normal data: 118.416 GiB
Expected downtime: 3314 ms
Setup time: 70 ms

 

https://www.redhat.com/archives/libvirt-users/2014-January/msg00007.html
https://specs.openstack.org/openstack/nova-specs/specs/mitaka/implemented/abort-live-migration.html

 

https://www.server24.eu/private-cloud/complete-live-migration-vms-high-load/

 

Oepnsatck Rocky – Keystone – Requesting a token scoped as a different project

Because yet again I found the documentation to be wrong on the Openstack site, here is what I have FINALLY managed to determine is the correct request to get a token issued to an admin user an another project. In my case i need this to create volume backups because the backup API does not allow you to project a project ID when creating a backup

http://KeystoneIP:5000/v3/auth/tokens

{
"auth": {
"scope": {
"project": {
"domain": {
"name": "Default"
},
"name": "OtherProjectName"
}
},
"identity": {
"password": {
"user": {
"domain": {
"name": "Default"
},
"password": "password",
"name": "admin"
}
},
"methods": [
"password"
]
}
}
}

Metadata service in DHCP namespace

Some gold info
http://kimizhang.com/metadata-service-in-dhcp-namespace/

Openstack – Manually edit VM

Find the host the VM is running on and the instance ID(Use console view to get instance ID)

cp /etc/libvirt/qemu/instance-0000030a.xml .
edit instance-0000030a.xml to be what you need it to be

While the VM is running (Warning, will crash the VM)

virsh destroy instance-0000030a
virsh undefine instance-0000030a
virsh define instance-0000030a.xml
virsh start instance-0000030a

Kolla commands

Build just one container and send it to a private registry

kolla-build -b ubuntu kolla-toolbox --registry localhost:443 --push

Build one thing and publish it to a custom namespace

sudo kolla-build horizon -b ubuntu --push --registry m2-kolla-deploy:443 -n cory

Push a single ‘project’ out – Unconfirmed

kolla-ansible deploy keystone -i all-in-one

 

kolla-ansible prechecks -e 'ansible_become=true' -e 'ansible_become_method=sudo' -i all-in-one

 

Private registry with self signed certificates

root@m3-kolla-dockerhost1:/etc/docker/certs.d/m2-kolla-deploy:443# ll
total 16
drwxr-xr-x 2 root root 4096 Dec 14 16:02 ./
drwxr-xr-x 3 root root 4096 Dec 14 16:02 ../
-rw-r--r-- 1 root root 757 Dec 14 15:04 client.cert
-rw-r--r-- 1 root root 887 Dec 14 15:04 client.key

root@m3-kolla-dockerhost1:~# cat /etc/docker/daemon.json
{
"insecure-registries" : ["m2-kolla-deploy:443"]
}

Starting the registry container

sudo docker run -d --restart=always --name registry2 -v /opt/registry/:/certs -e REGISTRY_HTTP_ADDR=0.0.0.0:443 -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/registry.nexulon.crt -e REGISTRY_HTTP_TLS_KEY=/certs/registry.nexulon.key -p 443:443 registry:2

 

Build with overrides

sudo kolla-build kolla-toolbox -b ubuntu --push --registry m2-kolla-deploy:443 -n cory --template-override kolla-toolbox_override.j2

cory@m3-kolla-deploy:~$ cat kolla-toolbox_override.j2
{% extends parent_template %}

# Horizon
{% set kolla_toolbox_packages_append = ['python-openstackclient'] %}

Openstack Queens – Nova: Error creating new key pair

Ina fresh install of Oepnstack Queens I was having an issue generating new keypairs after a reboot f the Nova node.

The following error was logged in nova-api.log

A simple restart of the nova-api service resolved this issue for me

 

 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi [req-78070ded-b2cd-4e1a-8a51-8c082b50e0a4 f2800cf724264988aab44aa21bf1dae4 0d280551fd45414eace8211d6ac154c0 - default 252047b9b8504489a11d9230a8f9bf55] Unexpected exception in API method: InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=2147897744L, lib=128, func=101, reason=400, reason_text='error:80065190:lib(128):osrandom_rand_bytes:getrandom() initialization failed.'), _OpenSSLErrorWithText(code=67637251L, lib=4, func=129, reason=3, reason_text='error:04081003:rsa routines:RSA_BUILTIN_KEYGEN:BN lib')])
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi Traceback (most recent call last):
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 788, in wrapped
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return f(*args, **kwargs)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108, in wrapper
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/validation/__init__.py", line 108, in wrapper
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return func(*args, **kwargs)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/keypairs.py", line 112, in create
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return self._create(req, body)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/keypairs.py", line 133, in _create
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi context, user_id, name, key_type)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 76, in wrapped
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi function_name, call_dict, binary)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi self.force_reraise()
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi six.reraise(self.type_, self.value, self.tb)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/exception_wrapper.py", line 67, in wrapped
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return f(self, context, *args, **kw)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 5287, in create_key_pair
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi user_id, key_type)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 5322, in _generate_key_pair
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return crypto.generate_key_pair()
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/nova/crypto.py", line 132, in generate_key_pair
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi key = paramiko.RSAKey.generate(bits)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/paramiko/rsakey.py", line 156, in generate
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi public_exponent=65537, key_size=bits, backend=default_backend()
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/primitives/asymmetric/rsa.py", line 119, in generate_private_key
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return backend.generate_rsa_private_key(public_exponent, key_size)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 362, in generate_rsa_private_key
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi self.openssl_assert(res == 1)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 106, in openssl_assert
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi return binding._openssl_assert(self._lib, ok)
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi File "/usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/openssl/binding.py", line 75, in _openssl_assert
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi errors_with_text
 2018-05-02 13:25:41.101 1351 ERROR nova.api.openstack.wsgi InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=2147897744L, lib=128, func=101, reason=400, reason_text='error:80065190:lib(128):osrandom_rand_bytes:getrandom() initialization failed.'), _OpenSSLErrorWithText(code=67637251L, lib=4, func=129, reason=3, reason_text='error:04081003:rsa routines:RSA_BUILTIN_KEYGEN:BN lib')])

Openstack – Layer2 Gateway(VXLAN -> Real world bridge)

This article is the culmination of 100’s of hours of work, I hope it can save others some time.

Here are some super useful articles that got me across the line

https://networkop.co.uk/blog/2016/05/21/neutron-l2gw/

https://wiki.openstack.org/wiki/Ovs-flow-logic

http://kimizhang.com/neutron-l2-gateway-hp-5930-switch-ovsdb-integration/

https://drive.google.com/file/d/0Bx8nDIFktlzBRm0tV3pmYURnZ3M/view

https://github.com/openstack/networking-l2gw

 

Setting up and Openvswitch VTEP

Step1 – Kill all Openvswitch processes

use ps ax | grep ovs to find any ovs processes that are running and kill them all

Step 2 – Bring up Openvswitch as a VTEP

Now configure the script below to suit

This process will kill any OVS config you have in place, if you like your config… well… do something else!

Here we use ens4 as out ‘trunk port’ and the name of our ‘physical switch'(Actually OpenvSwitch running on a server\VM) is switch-l2gw02

172.0.0.170 is the IP of the machine running OVS(Presumable the machine running this script)

#!/bin/bash
modprobe openvswitch
ip link set up dev ens4
rm etc/openvswitch/*
ovsdb-tool create /etc/openvswitch/vtep.db /usr/share/openvswitch/vtep.ovsschema
ovsdb-tool create /etc/openvswitch/vswitch.db /usr/share/openvswitch/vswitch.ovsschema
mkdir /var/run/openvswitch/
ovsdb-server --pidfile --detach --log-file --remote ptcp:6632:172.0.0.170 \
 --remote punix:/var/run/openvswitch/db.sock --remote=db:hardware_vtep,Global,managers \
 /etc/openvswitch/vswitch.db /etc/openvswitch/vtep.db
ovs-vswitchd --log-file --detach --pidfile unix:/var/run/openvswitch/db.sock
ovs-vsctl add-br switch-l2gw02
vtep-ctl add-ps switch-l2gw02
vtep-ctl set Physical_Switch switch-l2gw02 tunnel_ips=172.0.0.170
ovs-vsctl add-port switch-l2gw02 ens4
vtep-ctl add-port switch-l2gw02 ens4
/usr/share/openvswitch/scripts/ovs-vtep \
 --log-file=/var/log/openvswitch/ovs-vtep.log \
 --pidfile=/var/run/openvswitch/ovs-vtep.pid \
 --detach switch-l2gw02

Install and Configure the Neutron L2 Agent

For me the l2agent was available in the APT repo, so the installation was nice and simple.

This is a configuration agent, it doesn’t move any packets itself but it just orchestrates the required changes to the VTEP’s, So i run this on the same VM as my Neutron server services

Set the following line in l2gateway_agent.ini

ovsdb_hosts ='l2gw01:172.0.0.169:6632,l2gw02:172.0.0.170:6632'

 

 

Inbound ARP bug

This is a biggie and it’s a giant PIA
Inbound ARP requests will hit your VTEP but will not be forwarded on, even if the VTEP was to forward them on, it does have a ovs table suitable for sending broadcast packets(That is a table that speciifies an output port of every VXLan endpoint)

So to achieve this we use a bit of a workaround, first set a kind of ‘failover’ for all multicast packets on the VTEP to forward these unknown packets(inbound ARP requests) to one of the ‘network nodes’ that is a Neutron node that’s got a line in "ovs-vfctl dump-flows br-tun" that looks like this
table=22, n_packets=15, n_bytes=1030, idle_age=11531, priority=1,dl_vlan=9 actions=strip_vlan,load:0x3fa->NXM_NX_TUN_ID[],output:9,output:2,output:4,output:13
This is a boradcast rule, anything that hits it will be sent to all relevant VXLAN endpoints.
(I say relevant because it seems that it out outputs to ports that have devices on the other end on the same VXLAN, E.g If you have a compute node that doesn’t have any VM’s using that VXLAN network, the output port entry for that vxlan tunnel wont appear)

To configure this ‘failover’ run
sudo vtep-ctl add-mcast-remote 818b4779-645c-49bb-ae4a-aa9340604019 unknown-dst 10.0.3.10
Where the UUID is the result of vtep-ctl list-ls and the IP address is the IP of the neutron network node with table22 in place

Helpful hint: To find the names and numbers of the ports use this command
ovs-vsctl -f table -- --columns=name,ofport,external-ids,options list interface

Ok, so now the ARP packets are heading to the Network node but we aren’t quite done, we need to convince the network node to shunt all ARP requests out to table 22 and table 10(See here for a more detailed explanation from someone who actually knows what they are talking about https://networkop.co.uk/blog/2016/05/21/neutron-l2gw/ under the heading “Programming Network Node as BUM replication service node“)

To achieve this we need to add the following rule to br-tun on table 4
table=4,arp,tun_id=0x3fa,priority=2,actions=mod_vlan_vid:9,resubmit(,10),resubmit(,22)

Where 0x3fa is the segmentation id of our network in HEX format and vlan9 is the vlan used on THAT node for processing, you can find this ID by running ovs-ofctl dump-flows br-tun | grep 0x3fa
You’ll see a few entries and they’ll all share then same vlan_id.. That’s what we are after for our custom rule
When you have all the info run
ovs-ofctl add-flow br-tun "table=4,arp,tun_id=0x3fa,priority=2,actions=mod_vlan_vid:9,resubmit(,10),resubmit(,22)"

And that’s it!, not he ARP requests form the outside world hit the VTEP, overflow to the network node which kindly broadcasts them out to the VXLAN endpoints for us.

I hope this has helped you in some way to join your Openstack VXLAN networks to the real world.

Openstack Queens\Rocky – Glance: Create an image from an existing RBD volume

Seems like something that should be simple!

Step1 – Create the empty image

glance image-create --container-format=bare --disk-format=raw  --min-ram 2048 --name="Windows Server test"

+------------------+--------------------------------------+
| Property | Value |
+------------------+--------------------------------------+
| checksum | None |
| container_format | bare |
| created_at | 2018-04-26T04:08:37Z |
| disk_format | raw |
| id | 763a2ca2-e8f8-4bf9-974f-98d7020e200b |
| locations | [] |
| min_disk | 0 |
| min_ram | 0 |
| name | Windows Server test |
| owner | 40c2b46fb47c4ee7ac076b259c4e0814 |
| protected | False |
| size | None |
| status | queued |
| tags | [] |
| updated_at | 2018-04-26T04:08:37Z |
| virtual_size | None |
| visibility | shared |
+------------------+--------------------------------------+

Step 2 – Update the ‘location’ attribute

glance location-add --url "rbd://b42a82f3-f493-49f4-98e0-2d355bbe8ee3/saspool/image-Windows2016v1/snap" 763a2ca2-e8f8-4bf9-974f-98d7020e200b

OR if using the Openstack cli

openstack image set --property direct_url='rbd://cbb8d45d-79ee-4cf5-9ace-eeb145c89fd2/pool/image-pfsense235/snap' ee52d217-d2c8-4be4-b576-7766373de9e7

You’ll obviously need to ensure you have a protected snapshow of your image like so

:~# rbd snap create saspool/image-Windows2016v1@snap

:~# rbd snap protect saspool/image-Windows2016v1@snap

:~# rbd snap ls saspool/image-Windows2016v1

SNAPID NAME   SIZE TIMESTAMP

18 snap 150 GB Thu Apr 26 13:24:30 2018

You’ll also need to ensure that show_multiple_locations = true is set in glance-api.conf

Openstack Queens – Cinder: Deleting volumes using Mysql

Sometimes things get so broken you just have to hack stuff out of the database because the API’s are in a stuck state.

Here is what I did to successfully remove the volume entries for some stuck volumes after a MySQL server crash.

Note, You’ll also need to adjust the quotas table too to decrement the number of volumes used by that users

set @id :="b9840e99-f9f6-4a80-9dcc-dc51898d9cb3"; 
update cinder.volumes set status="available",attach_status="detached" where id=@id;
update cinder.volume_attachment set attach_status="detached",detach_time="2018-04-22 09:16:33",deleted=1,deleted_at="2018-04-23 08:16:33",updated_at="2018-04-23 08:16:33" where volume_id=@id;

Another little trick is to manually set volumes into an error state if they get stuck in a “Creating” state

update cinder.volumes set status="error" where status="creating";