Blog of iam8up

Linux shell make csv files readable

vim ~/.bashrc

#add this function

readcsv() {
cat “$1” | sed ‘s/,,/, ,/g’ | column -s’,’ -t
}

Average numbers echo’ed in bash

I’m tired of figuring this out again and again. This will average your output.

cat *latency.csv |cut -f5 -d,| awk '{sum+=$1; count++} END {if (count > 0) print "Average:", sum/count}'

Identify what hardware your Pi is

cat /sys/firmware/devicetree/base/model;echo

I love my Pi hardware and this is the easiest way I found to identify what version I have. Be it to figure out compatibility, what CPU/power it has, or you just want to know without cracking open the case.

Thanks to lbro100 for this tip!

Updating Centos 7.9 firmware for Raspbery Pi 4 Model B

I purchased a Raspberry Pi 4 Model B 2019 Quad Core 64 Bit WiFi Bluetooth (4GB). Every other Pi I use in production uses Centos 7 so of course I go to load it on a microSD and boot. I’m then halted with progress by this error:

start4.elf is not compatible this board requires newer software

The fix is very simple. Download this archive:
*http://bay.uchicago.edu/centos-altarch/7/isos/aarch64/images/CentOS-Userland-7-aarch64-RaspberryPI-Minimal-4-2009-sda.raw.xz
and then extract the raw image file.

Then run upd-image-fw from:
*https://forums.raspberrypi.com/viewtopic.php?p=1714655#p1714655
*(direct link to program) https://forums.raspberrypi.com/download/file.php?id=41880&sid=0a370ba73c65275426888fdfcf0b01f0

and then simply run:

sudo ./upd-image-fw CentOS-Userland-7-aarch64-RaspberryPI-Minimal-4-2009-sda.raw

I like to use balenaEtcher to write the image to SD.

Unbound stats graphed with Xymon

Refer to this overall guide for a master overview and below is an example hopefully others can find useful. This is for the Unbound stats we wanted to monitor initially and it may change.

Script to collect data

On the Xymon client:

touch ~xymon/ext/unboundstats.sh && chown xymon:xymon unboundstats.sh && chmod 744 unboundstats.sh

unboundstatsh.sh:

#!/bin/sh

/usr/sbin/unbound-control stats | \
grep -E “^total.num.queries|^total.num.queries_ip_ratelimited|^total.num.cachehits|^total.num.cachemiss|^total.recursion.time.avg|num.query.tcp|^unwanted.queries|^unwanted.replies” | \
awk -F= -v OFS=: ‘{gsub(/\./, “”, $1); print}’ > /tmp/unbound.stats

#sed would remove decimal in the recursion time value
#sed s,=,:, | sed s,\\.,,g > /tmp/unbound.stats

$XYMON $XYMSRV “status $MACHINE.unboundstats green `date`

`cat /tmp/unbound.stats`
”

exit 0

You need to restart the Xymon server! Generally this is systemctl restart xymon or /etc/init.d/xymonlaunch.cfg restart

I’m not sure if this is the right way of going about the permissions issue, but the Xymon user needs to be able to read the SSL key/pem…

chgrp xymon /etc/unbound/unbound_control.pem /etc/unbound/unbound_control.key /etc/unbound/unbound_server.pem /etc/unbound/unbound_server.key

Now make the Xymon client run the above script in ~xymon/etc/clientlaunch.cfg:

#this collects unbound stats
[unboundstats]
ENVFILE $XYMONCLIENTHOME/etc/xymonclient.cfg
CMD $XYMONCLIENTHOME/ext/unboundstats.sh
INTERVAL 5m

On the Xymon server you will want to turn this test submission data into RRD. In ~xymon/server/etc/xymonserver.cfg:

TEST2RRD=”cpu=la,defaultdefaultdefault,unboundstats=ncv”

SPLITNCV_unboundstats=”totalnumqueries:GAUGE,totalnumqueries_ip_ratelimited:GAUGE,totalnumcachehits:GAUGE,totalnumcachemiss:GAUGE,totalrecursiontimeavg:GAUGE,numquerytcp:GAUGE,numquerytcpout:GAUGE,unwantedqueries:GAUGE,unwantedreplies:GAUGE”

GRAPHS=”defaultdefaultdefault”

GRAPHS_unboundstats=”unboundstats,unboundstats1″ #note this requires xymon 4.3.20 https://lists.xymon.com/archive/2017-April/044560.html

After a few minutes you should see RRD files

# ls /var/lib/xymon/rrd/HOST.FOOBAR.COM/unboundstats,*
/var/lib/xymon/rrd/HOST.FOOBAR.COM/unboundstats,numquerytcpout.rrd
…

Add the graph definition (this translates from the RRD files to pretty graphs and once the file is edited/saved you can refresh the test page on Xymon/unboundstats for live viewing) in ~xymon/server/etc/graphs.cfg:

[unboundstats]
TITLE Unbound Stats
YAXIS Num
DEF:numqueries=unboundstats,totalnumqueries.rrd:lambda:AVERAGE
DEF:numqueriesiprate=unboundstats,totalnumqueries_ip_ratelimited.rrd:lambda:AVERAGE
DEF:numcachehits=unboundstats,totalnumcachehits.rrd:lambda:AVERAGE
DEF:totalnumcachemiss=unboundstats,totalnumcachemiss.rrd:lambda:AVERAGE

LINE2:numqueries#00CCCC:Total Queries
LINE2:numqueriesiprate#00FF00:Queries IP Rate
LINE2:numcachehits#FF0000:Total Cache Hits
LINE2:totalnumcachemiss#0000FF:Total Cache Miss

COMMENT:\n
GPRINT:numqueries:LAST:Total Queries \: %5.1lf%s (cur)
GPRINT:numqueries:MAX: \: %5.1lf%s (max)
GPRINT:numqueries:MIN: \: %5.1lf%s (min)
GPRINT:numqueries:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numqueriesiprate:LAST:IP Rate Queries \: %5.1lf%s (cur)
GPRINT:numqueriesiprate:MAX: \: %5.1lf%s (max)
GPRINT:numqueriesiprate:MIN: \: %5.1lf%s (min)
GPRINT:numqueriesiprate:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numcachehits:LAST:Cache Hits\: %5.1lf%s (cur)
GPRINT:numcachehits:MAX: \: %5.1lf%s (max)
GPRINT:numcachehits:MIN: \: %5.1lf%s (min)
GPRINT:numcachehits:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:totalnumcachemiss:LAST:Cache Miss\: %5.1lf%s (cur)
GPRINT:totalnumcachemiss:MAX: \: %5.1lf%s (max)
GPRINT:totalnumcachemiss:MIN: \: %5.1lf%s (min)
GPRINT:totalnumcachemiss:AVERAGE: \: %5.1lf%s (avg)\n

[unboundstats1]
TITLE Unbound Stats
YAXIS Num
DEF:totalrecursiontimeavg=unboundstats,totalrecursiontimeavg.rrd:lambda:AVERAGE
DEF:numquerytcp=unboundstats,numquerytcp.rrd:lambda:AVERAGE
DEF:numquerytcpout=unboundstats,numquerytcpout.rrd:lambda:AVERAGE
DEF:unwantedqueries=unboundstats,unwantedqueries.rrd:lambda:AVERAGE
DEF:unwantedreplies=unboundstats,unwantedreplies.rrd:lambda:AVERAGE

LINE2:totalrecursiontimeavg#AAA1AA:Average Recusion Time
LINE2:numquerytcp#C6913B:Queries w TCP
LINE2:numquerytcpout#8BBFFF:Queries out w TCP
LINE2:unwantedqueries#FF69B4:Unwanted Queries
LINE2:unwantedreplies#FFFF00:Unwanted Replies

COMMENT:\n
GPRINT:totalrecursiontimeavg:LAST:Recursion Time Avg\: %5.1lf%s (cur)
GPRINT:totalrecursiontimeavg:MAX: \: %5.1lf%s (max)
GPRINT:totalrecursiontimeavg:MIN: \: %5.1lf%s (min)
GPRINT:totalrecursiontimeavg:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numquerytcp:LAST:TCP Queries\: %5.1lf%s (cur)
GPRINT:numquerytcp:MAX: \: %5.1lf%s (max)
GPRINT:numquerytcp:MIN: \: %5.1lf%s (min)
GPRINT:numquerytcp:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:numquerytcpout:LAST:TCP Queries Out\: %5.1lf%s (cur)
GPRINT:numquerytcpout:MAX: \: %5.1lf%s (max)
GPRINT:numquerytcpout:MIN: \: %5.1lf%s (min)
GPRINT:numquerytcpout:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:unwantedqueries:LAST:Unwanted Queries\: %5.1lf%s (cur)
GPRINT:unwantedqueries:MAX: \: %5.1lf%s (max)
GPRINT:unwantedqueries:MIN: \: %5.1lf%s (min)
GPRINT:unwantedqueries:AVERAGE: \: %5.1lf%s (avg)\n

GPRINT:unwantedreplies:LAST:Unwanted Replies\: %5.1lf%s (cur)
GPRINT:unwantedreplies:MAX: \: %5.1lf%s (max)
GPRINT:unwantedreplies:MIN: \: %5.1lf%s (min)
GPRINT:unwantedreplies:AVERAGE: \: %5.1lf%s (avg)\n

Extract one column from a CSV

Assuming your CSV looks like this

“ID”,”DeviceName”,”IPAddress”,”LastStatusChange”
“211”,”customer name here”,”172.16.10.19″,”2018-08-1309:51:28″
“339”,”fue bah”,”10.13.0.1″,”2018-09-0619:28:30″

We can use Notepadd++ (or other tool) and some regex to extract the IP address.

Find what: ^(.*?),(.*?),(.*?),(.*?)$
Replace with: $3

Be sure to use regex if you’re on Windows/Notepad++! This works with four “column” documents, to adjust it for more or less be sure to mirror the number of commas and use one – (.*?) – for each column.

CentOS 6 with Python 2.7

https://danieleriksson.net/2017/02/08/how-to-install-latest-python-on-centos/

# Start by making sure your system is up-to-date:
yum update
# Compilers and related tools:
yum groupinstall-y”development tools”
# Libraries needed during compilation to enable all features of Python:
yum install-yzlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel expat-devel
# If you are on a clean “minimal” install of CentOS you also need the wget tool:
yum install-ywget

wget http://python.org/ftp/python/2.7.14/Python-2.7.14.tar.xz
tar xf Python-2.7.14.tar.xz
cdPython-2.7.14
./configure–prefix=/usr/local–enable-unicode=ucs4–enable-shared LDFLAGS=”-Wl,-rpath /usr/local/lib”
make&&makealtinstall

wget https://bootstrap.pypa.io/get-pip.py

# Then execute it using Python 2.7 and/or Python 3.6:
python2.7 get-pip.py

# With pip installed you can now do things like this:
pip2.7 install [packagename]
pip2.7 install –upgrade [packagename]
pip2.7 uninstall [packagename]

Clean Config Output

One of the best alias ideas I've ever seen!  This strips all the comments and such out of a config file so you can see, generally, what it looks like to the program reading it.

# alias
alias cleanconfig='sed -e '\''s/#.*//'\'' -e '\''s/[ ^I]*$//'\'' -e '\''/^$/ d'\'''

# cleanconfig /etc/ntp.conf
driftfile /var/lib/ntp/drift
restrict default kod nomodify notrap nopeer noquery

RRDTool dump and restore

This issue typically comes up when moving data from a 32bit to a 64bit system. The commands are very easy to work with for one file, but with many files and multiple directories (subdirectories) it can be cumbersome. I found this post in a forum and found it extremely useful and powerful.

SRC: http://www.linuxquestions.org/questions/programming-9/script-to-dump-and-restore-rrd-files-608407/

This should be a foo.sh file and you should run it in /usr/share/app/data where the files are within data or below that. It will convert and rename the file from rrd to xml. Transfer the data over to the new system and run the same command (either no argument or look at the bottom for ‘help’).

#!/bin/sh

ECHO="echo -e"

rrdump ()
{
	for rrd in `find . -type f -name "*.rrd"`
		do
			xml=`echo $rrd | sed 's/.rrd//g'`
			rrdtool dump $rrd > $xml.xml
			rm $rrd
		done
}	

xmlimport ()
{
	for xml in `find . -type f -name "*.xml"`
        do
                rrd=`echo $xml | sed 's/.xml//g'`
                rrdtool restore $xml $rrd.rrd
                rm $xml
        done
}


case "$1" in
        dump)
                rrdump
		;;
        import)
                xmlimport
                ;;
        *)
                $ECHO "$0 takes one of two arguments\n"
		$ECHO "$0 dump -- dumps contents to xml\n"
		$ECHO "$0 import -- imports xml files to rrd\n"
                ;;
esac

Network throughput on Linux seen on CLI

There are many options – http://www.binarytides.com/linux-commands-monitor-network/

nload is probably the simplest and what most people want though iftop is pretty cool.