Openstack – Layer2 Gateway(VXLAN -> Real world bridge)

This article is the culmination of 100’s of hours of work, I hope it can save others some time.

Here are some super useful articles that got me across the line


Setting up and Openvswitch VTEP

Step1 – Kill all Openvswitch processes

use ps ax | grep ovs to find any ovs processes that are running and kill them all

Step 2 – Bring up Openvswitch as a VTEP

Now configure the script below to suit

This process will kill any OVS config you have in place, if you like your config… well… do something else!

Here we use ens4 as out ‘trunk port’ and the name of our ‘physical switch'(Actually OpenvSwitch running on a server\VM) is switch-l2gw02 is the IP of the machine running OVS(Presumable the machine running this script)

modprobe openvswitch
ip link set up dev ens4
rm etc/openvswitch/*
ovsdb-tool create /etc/openvswitch/vtep.db /usr/share/openvswitch/vtep.ovsschema
ovsdb-tool create /etc/openvswitch/vswitch.db /usr/share/openvswitch/vswitch.ovsschema
mkdir /var/run/openvswitch/
ovsdb-server --pidfile --detach --log-file --remote ptcp:6632: \
 --remote punix:/var/run/openvswitch/db.sock --remote=db:hardware_vtep,Global,managers \
 /etc/openvswitch/vswitch.db /etc/openvswitch/vtep.db
ovs-vswitchd --log-file --detach --pidfile unix:/var/run/openvswitch/db.sock
ovs-vsctl add-br switch-l2gw02
vtep-ctl add-ps switch-l2gw02
vtep-ctl set Physical_Switch switch-l2gw02 tunnel_ips=
ovs-vsctl add-port switch-l2gw02 ens4
vtep-ctl add-port switch-l2gw02 ens4
/usr/share/openvswitch/scripts/ovs-vtep \
 --log-file=/var/log/openvswitch/ovs-vtep.log \
 --pidfile=/var/run/openvswitch/ \
 --detach switch-l2gw02

Install and Configure the Neutron L2 Agent

For me the l2agent was available in the APT repo, so the installation was nice and simple.

This is a configuration agent, it doesn’t move any packets itself but it just orchestrates the required changes to the VTEP’s, So i run this on the same VM as my Neutron server services

Set the following line in l2gateway_agent.ini

ovsdb_hosts ='l2gw01:,l2gw02:'



Inbound ARP bug

This is a biggie and it’s a giant PIA
Inbound ARP requests will hit your VTEP but will not be forwarded on, even if the VTEP was to forward them on, it does have a ovs table suitable for sending broadcast packets(That is a table that speciifies an output port of every VXLan endpoint)

So to achieve this we use a bit of a workaround, first set a kind of ‘failover’ for all multicast packets on the VTEP to forward these unknown packets(inbound ARP requests) to one of the ‘network nodes’ that is a Neutron node that’s got a line in "ovs-vfctl dump-flows br-tun" that looks like this
table=22, n_packets=15, n_bytes=1030, idle_age=11531, priority=1,dl_vlan=9 actions=strip_vlan,load:0x3fa->NXM_NX_TUN_ID[],output:9,output:2,output:4,output:13
This is a boradcast rule, anything that hits it will be sent to all relevant VXLAN endpoints.
(I say relevant because it seems that it out outputs to ports that have devices on the other end on the same VXLAN, E.g If you have a compute node that doesn’t have any VM’s using that VXLAN network, the output port entry for that vxlan tunnel wont appear)

To configure this ‘failover’ run
sudo vtep-ctl add-mcast-remote 818b4779-645c-49bb-ae4a-aa9340604019 unknown-dst
Where the UUID is the result of vtep-ctl list-ls and the IP address is the IP of the neutron network node with table22 in place

Helpful hint: To find the names and numbers of the ports use this command
ovs-vsctl -f table -- --columns=name,ofport,external-ids,options list interface

Ok, so now the ARP packets are heading to the Network node but we aren’t quite done, we need to convince the network node to shunt all ARP requests out to table 22 and table 10(See here for a more detailed explanation from someone who actually knows what they are talking about under the heading “Programming Network Node as BUM replication service node“)

To achieve this we need to add the following rule to br-tun on table 4

Where 0x3fa is the segmentation id of our network in HEX format and vlan9 is the vlan used on THAT node for processing, you can find this ID by running ovs-ofctl dump-flows br-tun | grep 0x3fa
You’ll see a few entries and they’ll all share then same vlan_id.. That’s what we are after for our custom rule
When you have all the info run
ovs-ofctl add-flow br-tun "table=4,arp,tun_id=0x3fa,priority=2,actions=mod_vlan_vid:9,resubmit(,10),resubmit(,22)"

And that’s it!, not he ARP requests form the outside world hit the VTEP, overflow to the network node which kindly broadcasts them out to the VXLAN endpoints for us.

I hope this has helped you in some way to join your Openstack VXLAN networks to the real world.

HPACUCLI – Mapping a disk to a linux mount point /dev/sdx

Found this little nugget by accident but here is the command to use hpacucli on linux to determine which logical disk is mapped to which mount point

[root@ceph1 ~]# /usr/sbin/uname26 hpacucli controller slot=2 ld 3 show

Smart Array P410 in Slot 2

array C

Logical Drive: 3
 Size: 3.6 TB
 Fault Tolerance: 0
 Heads: 255
 Sectors Per Track: 32
 Cylinders: 65535
 Strip Size: 256 KB
 Full Stripe Size: 256 KB
 Status: OK
 Caching: Enabled
 Unique Identifier: 600508B1001C953FDF746410876C6824
 Disk Name: /dev/sdb
 Mount Points: /var/lib/ceph/osd/ceph-58 3.6 TB
 OS Status: LOCKED
 Logical Drive Label: A9874C30PACCRID105108LP4D16
 Drive Type: Data

[root@adlwest-ceph70 ~]# /usr/sbin/uname26 hpacucli controller slot=2 ld 4 show

Smart Array P410 in Slot 2

array D

Logical Drive: 4
 Size: 3.6 TB
 Fault Tolerance: 0
 Heads: 255
 Sectors Per Track: 32
 Cylinders: 65535
 Strip Size: 256 KB
 Full Stripe Size: 256 KB
 Status: Failed
 Caching: Enabled
 Unique Identifier: 600508B1001C9435EB0B8689EAB537B1
 Mount Points: None
 Logical Drive Label: A9875C36PACCRID105108LP839F
 Drive Type: Data

Openstack Queens\Rocky – Glance: Create an image from an existing RBD volume

Seems like something that should be simple!

Step1 – Create the empty image

glance image-create --container-format=bare --disk-format=raw  --min-ram 2048 --name="Windows Server test"
| Property         | Value                                |
| checksum         | None                                 |
| container_format | bare                                 |
| created_at       | 2020-07-14T03:37:29Z                 |
| disk_format      | raw                                  |
| id               | ca9bf831-df20-4c7c-884e-08e355e5a012 |
| locations        | []                                   |
| min_disk         | 0                                    |
| min_ram          | 2048                                 |
| name             | 3CX - July 2020                      |
| os_hash_algo     | None                                 |
| os_hash_value    | None                                 |
| os_hidden        | False                                |
| owner            | ccd547784c804de99a63dd17dfb7ff15     |
| protected        | False                                |
| size             | None                                 |
| status           | queued                               |
| tags             | []                                   |
| updated_at       | 2020-07-14T03:37:29Z                 |
| virtual_size     | None                                 |
| visibility       | shared                               |

Step 2 – If you haven’t already done so, upload the image into ceph – Note the image must be in RAW format. Use qemu-img convert to convert the image from QCOW2 to RAW if required

To convert and import in one pass

qemu-img convert -f qcow2 -O raw inputfile.cqow2 rbd:images/outputfile.raw

To import an existing RAW image file

rbp import -p images myimg.raw

Step 2 – Update the ‘location’ attribute

glance location-add --url "rbd://b42a82f3-f493-49f4-98e0-2d355bbe8ee3/saspool/image-Windows2016v1/snap" 763a2ca2-e8f8-4bf9-974f-98d7020e200b

You’ll obviously need to ensure you have a protected snapshot of your image like so

:~# rbd snap create saspool/image-Windows2016v1@snap

:~# rbd snap protect saspool/image-Windows2016v1@snap

:~# rbd snap ls saspool/image-Windows2016v1


18 snap 150 GB Thu Apr 26 13:24:30 2018

You’ll also need to ensure that show_multiple_locations = true is set in glance-api.conf or you’ll see something like this

403 Forbidden: It's not allowed to add locations if locations are invisible. (HTTP 403)

Openstack Queens – Cinder: Deleting volumes using Mysql

Sometimes things get so broken you just have to hack stuff out of the database because the API’s are in a stuck state.

Here is what I did to successfully remove the volume entries for some stuck volumes after a MySQL server crash.

Note, You’ll also need to adjust the quotas table too to decrement the number of volumes used by that users

set @id :="b9840e99-f9f6-4a80-9dcc-dc51898d9cb3"; 
update cinder.volumes set status="available",attach_status="detached" where id=@id;
update cinder.volume_attachment set attach_status="detached",detach_time="2018-04-22 09:16:33",deleted=1,deleted_at="2018-04-23 08:16:33",updated_at="2018-04-23 08:16:33" where volume_id=@id;

Another little trick is to manually set volumes into an error state if they get stuck in a “Creating” state

update cinder.volumes set status="error" where status="creating";

Openstack Queens – Horizon: Custom SVG Logos

This is what worked for me, it’s NOT what the documentation says but hey!


Using the material theme as the default, this should replace the logo on the heading bar in Horizon.

No need to run ./ collectstatic or anything, just run this and hit F5 in your browser

cp /home/myadmin/logo.svg /usr/share/openstack-dashboard/openstack_dashboard/themes/material/templates/material/openstack-one-color.svg


The logo on the login page is a little different

cp /home/myadmin/logo-splash.svg /usr/share/openstack-dashboard/openstack_dashboard/static/dashboard/img/
python /usr/share/openstack-dashboard/ collectstatic