Whats inside an APT package?

sudo dpkg –listfiles docker-ce

Generate OTP keys in linux – Extracting FreeOTP keys

apt install oathtool
oathtool --totp -b -d 6 KY3OUPMUYWCKS53F

Linux: TOTP Password Generator

Enable USB debugging – https://www.kingoapp.com/root-tutorials/how-to-enable-usb-debugging-mode-on-android.htm
Backup the FreeOTP app – adb backup -f ~/freeotp.ab -noapk org.fedorahosted.freeotp
Decompress the backup – dd if=freeotp.ab bs=1 skip=24 | python -c "import zlib,sys;sys.stdout.write(zlib.decompress(sys.stdin.read()))" | tar -xvf -
Decode the keys – https://github.com/philipsharp/FreeOTPDecoder

Openstack – Manually edit VM

Find the host the VM is running on and the instance ID(Use console view to get instance ID)

cp /etc/libvirt/qemu/instance-0000030a.xml .
edit instance-0000030a.xml to be what you need it to be

While the VM is running (Warning, will crash the VM)

virsh destroy instance-0000030a
virsh undefine instance-0000030a
virsh define instance-0000030a.xml
virsh start instance-0000030a

Ceph scrubbing performance

Original article here – http://sudomakeinstall.com/linux-systems/ceph-scrubbing

Ceph’s default IO priority and class for behind the scene disk operations should be considered required vs best efforts. For those of us who actually utilize our storage for services that require performance will quickly find that deep scrub grinds even the most powerful systems to a halt.

Below are the settings to run the scrub as the lowest possible priority. This REQUIRES CFQ as the scheduler for the spindle disk. Without CFQ you cannot prioritize IO. Since only 1 service utilizes these disk CFQ performance will be comparable to deadline and noop.

Show the current scheduler

for file in /sys/block/sd*; do
echo ${file}
cat ${file}/queue/scheduler
echo “”

Set all disks to CFQ

for file in /sys/block/sd*; do
echo cfq > ${file}/queue/scheduler
cat ${file}/queue/scheduler
echo “”

Inject the new settings for the existing OSD:
ceph tell osd.* injectargs '--osd_disk_thread_ioprio_priority 7'
ceph tell osd.* injectargs '--osd_disk_thread_ioprio_class idle'

Edit your ceph.conf on your storage nodes to automatically set the the priority at runtime.
#Reduce impact of scrub.
osd_disk_thread_ioprio_class = "idle"
osd_disk_thread_ioprio_priority = 7

You can go a step further and setup redhats optimizations for the system charactistics.
tuned-adm profile latency-performance
This information referenced from multiple sources.

Reference documentation.

Disable scrubbing in realtime to determine its impact on your running cluster.

A detailed analysis of the scrubbing io impact.

OSD Configuration Reference

Redhat system tuning.

List pools and their crush rule

ceph osd pool ls | while read line; do echo $line && ceph osd pool get $line crush_rule; done

error on subcontainer ‘ia_addr’ insert (-1)

If you use SNMP on Ubuntu you might see this annoying log entry repeat itself in syslog
Sep 15 17:29:34 myServer snmpd[3609]: error on subcontainer 'ia_addr' insert (-1)
Easy fix is
sed -i 's/Lsd/LS6d/g' /etc/default/snmpd
service snmpd restart

Dell S6000-ON Fan-out port configuration

Using Openswitch 2.3.1 on the Dell S6000-ON I was able to configure the fan-out \ breakout port configuration but the link was not coming up on my 10G switch or on the 4 sub interfaces generated by OPX. Use the following steps to solve that issue.

Don’t quote me on this but I believe that on the Dell S6000-ON ports 13-16 and 29-32 can NOT be enabled for fan out, which makes sense because the manual says upto 96 x 10G ports using fanout. 32 ports minus those 8 leaves 24. 24×4=96

Sept 1 – Configure the 4 way port split

Note, this works if you are using a QSFP+ to 4 x SFP+ cable or a QSFP+ to QSFP+ cable or a QSFP+ to SFP+ adapter.

opx-config-fanout e101-001-0 4x1 10g


Step 2 – Bring up the 10G channel
ip link set up e101-001-1


Step 3 – Disable Auto-negotiate
opx-ethtool -s e101-001-1 autoneg off


Step 4 – Configure link speed to 10GB/s
opx-ethtool -s e101-001-1 speed 10000


Check the link is up by running ip a

ip a
37: e101-001-1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether f4:8e:38:47:a4:79 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f68e:38ff:fe47:a479/64 scope link
valid_lft forever preferred_lft forever

If you’d like to have this persist a reboot then edit /etc/network/interfaces to include the following

auto e101-001-1 allow-hotplug e101-001-1 iface e101-001-1 inet manual pre-up sudo ip link set up $IFACE pre-up sudo opx-ethtool -s $IFACE autoneg off pre-up sudo opx-ethtool -s $IFACE speed 10000

and edit /etc/opx/dn_nas_fanout_init_config.xml to include

<interfaces> <interface name="e101-001-0" fanout="4x1" speed="10G"/> </interfaces>

Bootleg HP iLO Firmware

This absolute champion is keeping the most recent firmware for most HP iLO’s available to normal humans without a super secret super expensive support contract available online.


Docker fun

root@m2-kolla-deploy:~# curl -X GET https://localhost/v2/_catalog?n=2000 -k
root@m2-kolla-deploy:~# curl -X GET https://localhost/v2/kolla/ubuntu-binary-fluentd/tags/list -k

Working with self signed certificates? Add this.

root@m2-kolla-control01:/etc/docker/certs.d/m2-kolla-deploy:443# cat /etc/docker/daemon.json
{ "insecure-registries":["m2-kolla-deploy:443"] }



root@m2-kolla-control01:/etc/docker/certs.d/m2-kolla-deploy:443# ll
total 16
drwxr-xr-x 2 root root 4096 Dec 11 15:26 ./
drwxr-xr-x 3 root root 4096 Dec 11 15:25 ../
-rw-r--r-- 1 root root 757 Dec 11 15:26 client.cert
-rw-r--r-- 1 root root 887 Dec 11 15:26 client.key


Delete all images

docker stop $(docker ps -a -q)

docker rm $(docker ps -a -q)


Get a bash prompt in running container

sudo docker exec -i -t mariadb /bin/bash


Watch console of a container

docker logs -t -f dockerstuff_scriptrunner_1


Attach to running container

docker exec -i -t dockerstuff_php_1 /bin/bash


Check port mapping

docker port dockerstuff_maint_1 22


Linux static routes & fixing “Error: either “to” is duplicate, or gw is a garbage.”

I was having troubles adding a route using ip route, I was getting “Error: either “to” is duplicate, or “” is a garbage.”

[root@adlwest-nms1 ~]# ip route add dev eth1
Error: either "to" is duplicate, or "" is a garbage.

But running ip route add subnet/netmask via gw worked

[root@adlwest-nms1 ~]# ip route add via
[root@adlwest-nms1 ~]# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=62 time=1.04 ms
64 bytes from icmp_seq=2 ttl=62 time=0.992 ms