Whats inside an APT package?

sudo dpkg –listfiles docker-ce

Generate OTP keys in linux – Extracting FreeOTP keys

apt install oathtool
oathtool --totp -b -d 6 KY3OUPMUYWCKS53F

Linux: TOTP Password Generator

https://github.com/philipsharp/FreeOTPDecoder
Enable USB debugging – https://www.kingoapp.com/root-tutorials/how-to-enable-usb-debugging-mode-on-android.htm
Backup the FreeOTP app – adb backup -f ~/freeotp.ab -noapk org.fedorahosted.freeotp
Decompress the backup – dd if=freeotp.ab bs=1 skip=24 | python -c "import zlib,sys;sys.stdout.write(zlib.decompress(sys.stdin.read()))" | tar -xvf -
Decode the keys – https://github.com/philipsharp/FreeOTPDecoder

Openstack – Manually edit VM

Find the host the VM is running on and the instance ID(Use console view to get instance ID)

cp /etc/libvirt/qemu/instance-0000030a.xml .
edit instance-0000030a.xml to be what you need it to be

While the VM is running (Warning, will crash the VM)

virsh destroy instance-0000030a
virsh undefine instance-0000030a
virsh define instance-0000030a.xml
virsh start instance-0000030a

Ceph scrubbing performance

Original article here – http://sudomakeinstall.com/linux-systems/ceph-scrubbing

Ceph’s default IO priority and class for behind the scene disk operations should be considered required vs best efforts. For those of us who actually utilize our storage for services that require performance will quickly find that deep scrub grinds even the most powerful systems to a halt.

Below are the settings to run the scrub as the lowest possible priority. This REQUIRES CFQ as the scheduler for the spindle disk. Without CFQ you cannot prioritize IO. Since only 1 service utilizes these disk CFQ performance will be comparable to deadline and noop.

Show the current scheduler

for file in /sys/block/sd*; do
echo ${file}
cat ${file}/queue/scheduler
echo “”
done

Set all disks to CFQ

for file in /sys/block/sd*; do
echo cfq > ${file}/queue/scheduler
cat ${file}/queue/scheduler
echo “”
done

Inject the new settings for the existing OSD:
ceph tell osd.* injectargs '--osd_disk_thread_ioprio_priority 7'
ceph tell osd.* injectargs '--osd_disk_thread_ioprio_class idle'

Edit your ceph.conf on your storage nodes to automatically set the the priority at runtime.
#Reduce impact of scrub.
osd_disk_thread_ioprio_class = "idle"
osd_disk_thread_ioprio_priority = 7

You can go a step further and setup redhats optimizations for the system charactistics.
tuned-adm profile latency-performance
This information referenced from multiple sources.

Reference documentation.
http://dachary.org/?p=3268

Disable scrubbing in realtime to determine its impact on your running cluster.
http://dachary.org/?p=3157

A detailed analysis of the scrubbing io impact.
http://blog.simon.leinen.ch/2015/02/ceph-deep-scrubbing-impact.html

OSD Configuration Reference
http://ceph.com/docs/master/rados/configuration/osd-config-ref/

Redhat system tuning.
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Performance_Tuning_Guide/sect-Red_Hat_Enterprise_Linux-Performance_Tuning_Guide-Tool_Reference-tuned_adm.html

List pools and their crush rule

ceph osd pool ls | while read line; do echo $line && ceph osd pool get $line crush_rule; done

error on subcontainer ‘ia_addr’ insert (-1)

If you use SNMP on Ubuntu you might see this annoying log entry repeat itself in syslog
Sep 15 17:29:34 myServer snmpd[3609]: error on subcontainer 'ia_addr' insert (-1)
Easy fix is
sed -i 's/Lsd/LS6d/g' /etc/default/snmpd
service snmpd restart

Dell S6000-ON Fan-out port configuration

Using Openswitch 2.3.1 on the Dell S6000-ON I was able to configure the fan-out \ breakout port configuration but the link was not coming up on my 10G switch or on the 4 sub interfaces generated by OPX. Use the following steps to solve that issue.

Don’t quote me on this but I believe that on the Dell S6000-ON ports 13-16 and 29-32 can NOT be enabled for fan out, which makes sense because the manual says upto 96 x 10G ports using fanout. 32 ports minus those 8 leaves 24. 24×4=96

Sept 1 – Configure the 4 way port split

Note, this works if you are using a QSFP+ to 4 x SFP+ cable or a QSFP+ to QSFP+ cable or a QSFP+ to SFP+ adapter.

opx-config-fanout e101-001-0 4x1 10g

 

Step 2 – Bring up the 10G channel
ip link set up e101-001-1

 

Step 3 – Disable Auto-negotiate
opx-ethtool -s e101-001-1 autoneg off

 

Step 4 – Configure link speed to 10GB/s
opx-ethtool -s e101-001-1 speed 10000

 

Check the link is up by running ip a

ip a
...
37: e101-001-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f4:8e:38:47:a4:79 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f68e:38ff:fe47:a479/64 scope link
valid_lft forever preferred_lft forever

If you’d like to have this persist a reboot then edit /etc/network/interfaces to include the following

auto e101-001-1 allow-hotplug e101-001-1 iface e101-001-1 inet manual pre-up sudo ip link set up $IFACE pre-up sudo opx-ethtool -s $IFACE autoneg off pre-up sudo opx-ethtool -s $IFACE speed 10000

and edit /etc/opx/dn_nas_fanout_init_config.xml to include

<interfaces> <interface name="e101-001-0" fanout="4x1" speed="10G"/> </interfaces>

Bootleg HP iLO Firmware

This absolute champion is keeping the most recent firmware for most HP iLO’s available to normal humans without a super secret super expensive support contract available online.

http://pingtool.org/latest-hp-ilo-firmwares/

Docker fun

root@m2-kolla-deploy:~# curl -X GET https://localhost/v2/_catalog?n=2000 -k
{"repositories":["kolla/ubuntu-binary-barbican-base","kolla/ubuntu-binary-barbican-keystone-listener","kolla/ubuntu-binary-base","kolla/ubuntu-binary-fluentd","kolla/ubuntu-binary-keystone","kolla/ubuntu-binary-keystone-base","kolla/ubuntu-binary-keystone-fernet","kolla/ubuntu-binary-keystone-ssh","kolla/ubuntu-binary-openstack-base"]}
root@m2-kolla-deploy:~# curl -X GET https://localhost/v2/kolla/ubuntu-binary-fluentd/tags/list -k
{"name":"kolla/ubuntu-binary-fluentd","tags":["5.0.0"]}

Working with self signed certificates? Add this.

root@m2-kolla-control01:/etc/docker/certs.d/m2-kolla-deploy:443# cat /etc/docker/daemon.json
{ "insecure-registries":["m2-kolla-deploy:443"] }
root@m2-kolla-control01:/etc/docker/certs.d/m2-kolla-deploy:443#

 

 

root@m2-kolla-control01:/etc/docker/certs.d/m2-kolla-deploy:443# ll
total 16
drwxr-xr-x 2 root root 4096 Dec 11 15:26 ./
drwxr-xr-x 3 root root 4096 Dec 11 15:25 ../
-rw-r--r-- 1 root root 757 Dec 11 15:26 client.cert
-rw-r--r-- 1 root root 887 Dec 11 15:26 client.key

 

Delete all images

docker stop $(docker ps -a -q)

docker rm $(docker ps -a -q)

 

Get a bash prompt in running container

sudo docker exec -i -t mariadb /bin/bash

 

Watch console of a container

docker logs -t -f dockerstuff_scriptrunner_1

 

Attach to running container

docker exec -i -t dockerstuff_php_1 /bin/bash

 

Check port mapping

docker port dockerstuff_maint_1 22

 

Linux static routes & fixing “Error: either “to” is duplicate, or gw is a garbage.”

I was having troubles adding a route using ip route, I was getting “Error: either “to” is duplicate, or “172.16.1.254” is a garbage.”

[root@adlwest-nms1 ~]# ip route add 172.16.103.0/24 172.16.1.254 dev eth1
Error: either "to" is duplicate, or "172.16.1.254" is a garbage.

But running ip route add subnet/netmask via gw worked

[root@adlwest-nms1 ~]# ip route add 172.16.103.0/24 via 172.16.1.254
[root@adlwest-nms1 ~]# ping 172.16.103.41
PING 172.16.103.41 (172.16.103.41) 56(84) bytes of data.
64 bytes from 172.16.103.41: icmp_seq=1 ttl=62 time=1.04 ms
64 bytes from 172.16.103.41: icmp_seq=2 ttl=62 time=0.992 ms