A Chef that doesn’t know how to cook with systemd

Here is some good advice to handle systemd anxiety from adamo.

Configuration management / orchestration software already knows how to handle systemd and they do it well. Take advantage of that.

Unfortunately for me, there is a Chef server in our infra so old that doesn’t know how to handle systemd, and with the release of CentOS 7 (1511) it finally broke.

================================================================
Error executing action `restart` on resource 'service[iptables]'
================================================================

Chef::Exceptions::Service
-------------------------
service[iptables]: unable to locate the init.d script!

It expects that all services have a script in /etc/init.d, so here is my quick hack around it.

$ ls -l /etc/init.d/iptables
lrwxrwxrwx 1 root root 32 Dec 17 17:34 /etc/init.d/iptables -> /usr/local/bin/systemctl-wrapper

$ cat /usr/local/bin/systemctl-wrapper
#!/bin/bash
# workaround for old school SysV persons
# stsimb dec 2015

SERVICE="$(basename $0)"
exec /bin/systemctl "$@" ${SERVICE}.service

Now we can spend more time migrating to our new Chef, that knows how to handle systemd.

troubleshooting dnssec

If you’re looking for DNSSEC troubleshooting tools, Carsten Strotmann’s article Delve deep into DNSSEC presents a new cli tool that will be available in BIND 9.10 called delv.

delv works very much like the already know dig. It is like dig with special DNSSEC validation powers. delv checks the DNSSEC validation chain using the same code that is used by the BIND 9 DNS server itself.

But keep in mind that delv, as any standard unix tool, will use the resolvers in /etc/resolv.conf to perform all lookups. If all your resolvers listed  are DNSSEC validating, delv will not be able to lookup a non-dnssec validating RR, and will not help you debug the problem.

[stsimb@jumbo ~]$ delv www.spbet.eu. +multi +rtrace
 ;; fetch: www.spbet.eu/A
 ;; resolution failed: failure

You need to have access to some non-DNSSEC validating resolver, to kick start the trace..

[stsimb@jumbo ~]$ delv www.spbet.eu. @4.2.2.1 +multi +rtrace
 ;; fetch: www.spbet.eu/A
 ;; fetch: eu/DS
 ;; fetch: ./DNSKEY
 ;; fetch: spbet.eu/DS
 ;; fetch: eu/DNSKEY
 ;; fetch: eu/DS
 ;; fetch: ./DNSKEY
 ;; fetch: www.spbet.eu/DS
 ;; fetch: eu/DS
 ;; fetch: ./DNSKEY
 ;; fetch: spbet.eu/DS
 ;; fetch: eu/DNSKEY
 ;; fetch: eu/DS
 ;; fetch: ./DNSKEY
 ;; no valid RRSIG resolving 'www.spbet.eu/DS/IN': 4.2.2.1#53
 ;; no valid DS resolving 'www.spbet.eu/A/IN': 4.2.2.1#53
 ;; resolution failed: no valid DS

Personally I don’t consider this a bug. It’s expected behaviour, a unix tool to use the resolvers in /etc/resolv.conf for resolution.

delv cannot determine for sure if it’s your resolver’s fault, and then try another one (and if it did, which one should it choose?). It’s your job to have this in mind, and ask some other non-dncsec validating resolver that you have access to.

Nevertheless, this situation may come up many times. Troubleshooting broken trust chains is one of the reasons someone will use delv, and gives you a reason to keep a non dnssec-validating resolver available somewhere.

Update: Thanks to Peter van Dijk‘s tweet I discovered that +cdflag works in delv (although it’s not listed in delv -h).

Zimbra mailbox size per folder

It looks like Zimbra has an easy way to report message count per folder (zmmailbox getAllFolders), but not folder size.

Size information is buried in the database, and needs a few queries per folder to dig..

Here is a script to do it for you.. zimbra-size.sh

[zimbra@zimbra ~]$ /tmp/zimbra-size.sh [email protected]
[email protected]'s max mailbox size = 20480 MB, current mailbox size = 343.02 MB.

size (MB) msgcount unread folder
--------- --------- ---------- ----------------------------
0 0 0 /Chats
0 0 0 /Drafts
12 221 221 /Inbox
15 1001 1001 /Inbox/Tickets
310 7158 7158 /Inbox/Syslog
3 78 78 /Inbox/Wiki
0 0 0 /Junk
3 21 0 /Sent
...

increase ldap search results per user

By default, openldap limits search results to 500 for all users except root.

If you don’t want to increase it globally using sizelimit, then you can do it for specific users.

All you have to do is add something like the following in slapd.conf.

limits dn.exact="uid=serviceuser,ou=users,ou=domain,ou=com" size=unlimited

(manual reference)

sudoedit vs rvim

Recently I inherited some systems running Chef client, with a script that has to be called via sudo to edit some root owned Chef provisioned config files (e.g. munin.conf)..

/etc/sudoers contents looked like this

%sudoers hostname=(root) NOPASSWD: /usr/bin/chef-man-provision

and it actually executed rvim

rvim -u /root/.rvimrc /etc/munin/munin.conf

Contents of /root/.rvimrc look like this

" all following options are for securing vi for chef-man-provision script. 
" Security option: disable file opening with `:r` command
cmap r <Nop>
cmap o <Nop>

This «security option» had a pretty nasty effect… We couldn’t use characters «r» and «o» in vi command mode, even in simple search strings

Enter sudoedit! To quote the sudoedit(8) manpage

-e‘ The -e (edit) option indicates that, instead of running a command, the user wishes to edit one or more files. In lieu of a command, the string «sudoedit» is used when consulting the security policy. If the user is authorized by the policy, the following steps are taken:

1. Temporary copies are made of the files to be edited with the owner set to the invoking user.

2. The editor specified by the policy is run to edit the temporary files. The sudoers policy uses the SUDO_EDITOR, VISUAL and EDITOR environment variables (in that order). If none of SUDO_EDITOR, VISUAL or EDITOR are set, the first program listed in the editor sudoers(5) option is used.

3. If they have been modified, the temporary files are copied back to their original location and the temporary versions are removed.

If the specified file does not exist, it will be created. Note that unlike most commands run by sudo, the editor is run with the invoking user’s environment unmodified. If, for some reason, sudo is unable to update a file with its edited version, the user will receive a warning and the edited copy will remain in a temporary file.

So, /etc/sudoers was modified to look like this

%sudoers hostname=(root) NOPASSWD: sudoedit /etc/munin/munin.conf

We now type «sudoedit /etc/munin/munin.conf» and everything works as it should work in the first place, i.e. securely and we can use all the letters!

cvi going distributed

cvi was published in an earlier post: version control for important files

Going distributed instead of local can be tricky, because somebody else may have changed the file in the master repo.. but here is a cruel attempt..

#!/bin/sh
# https://stsimb.irc.gr/2013/04/07/cvi-going-distributed/

if [ -z "$1" ] ; then
  echo edit a file stored in a git repo
  echo usage = $0 filename
  exit 1
fi

git pull origin master || exit $?
vi $1
git add $1
git commit
git push