Comparing Puppet and CFEngine in recursive file handling
Tuesday, October 22nd, 2013 - 10:36 am - CFEngine
So, we all know that ruby’s memory management is sketchy at best, and the Puppet is generally slow. But how can we quantify that? One of the metrics which is important to my usage is that of verifying the permissions on a large number of files. To that end, I wrote a simple script to compare the performance of ensuring that the contents of a large directory of files are owned by a specific group. Before each test, I remove a temp directory, create a set of sequentially-named files with the wrong group ownership, and then correct the ownership. I then run the same command again to see how quickly it can verify the permissions – which should be the common case.
For the baseline, I use “find | xargs chgrp”, which is slightly slower than “chgrp -R”, but not much slower (and, in my mind, slightly more fair). I then use a simple CFEngine policy and a simple Puppet policy to do the same thing. The summary? Puppet is dog slow at file recursion, while CFEngine is nearly as fast as pure find. CFEngine actually uses less memory than the shell when you get to many files (probably due to the pipe to xargs), and Puppet wastes memory like it’s been surfing the web for weeks using an old version of Firefox.
Here’s the CFEngine policy file (using local bodies rather than the standard library, to be fair):
bundle agent do_perms {
files:
"/srv/gluster/sec/tmp"
depth_search => recurse( "inf" ),
perms => g( "security" );
}
body depth_search recurse(d) {
depth => "$(d)";
}
body perms g(G) {
groups => { "$(G)" };
}
Here’s the Puppet manifest (yeah, it’s slightly simpler):
class{ "test": }
class test {
file { "/srv/gluster/sec/tmp/test":
recurse => "inf",
group => "security",
}
}
and here’s the test script:
#!/usr/bin/ksh
################################################################################
# compare CFEngine, Puppet, and find
################################################################################
# note that $D will get deleted between runs
D=/srv/gluster/sec/tmp/test
#S=$(basename "$0")
S="test_file"
CF=$( rpm -q --qf 'CFEngine %{VERSION}\n' cfengine-community )
PU=$( rpm -q --qf 'Puppet %{VERSION}\n' puppet )
function clean {
rm -fr "$D"
}
function create {
mkdir -p "$D"
seq 1 $1 | sed "s?^?$D/?; s?\$?.$S?" | xargs touch
}
alias t="/usr/bin/time -f '%E, %MKB, %U user, %S kernel'"
for N in 1 10 100 1000 2000 5000 10000 20000 30000 50000 100000 200000 500000
do
clean
create $N
# system test
print -n "$N, system set, "
t \
sh -c "find '$D' -type f -not -group security -print0 | xargs -0 chgrp security"
print -n "$N, system check, "
t \
sh -c "find '$D' -type f -not -group security -print0 | xargs -0 chgrp security 2>/dev/null"
# CFEngine test
clean
create $N
print -n "$N, $CF set, "
t /var/cfengine/bin/cf-agent -K -b do_perms -f ./cfengine_v_puppet.cf
print -n "$N, $CF check, "
t /var/cfengine/bin/cf-agent -K -b do_perms -f ./cfengine_v_puppet.cf
if [[ $N -le 10000 ]] # it's just too slow
then
# Puppet test
clean
create $N
print -n "$N, $PU set, "
t /usr/bin/puppet apply --logdest /dev/null cfengine_v_puppet.pp
print -n "$N, $PU check, "
t /usr/bin/puppet apply --logdest /dev/null cfengine_v_puppet.pp
fi
done
clean
The results are clear. CFEngine can verify permissions on a half million files in about the amount of time it takes for Puppet to start up, and can set perms on a quarter million files before Puppet can even start up to check one. Further CFEngine’s memory consumption remains constant while it’s doing this.
$ ./cfengine_v_puppet.ksh 1, system set, 0:00.00, 5520KB, 0.00 user, 0.00 kernel 1, system check, 0:00.00, 5488KB, 0.00 user, 0.00 kernel 1, CFEngine 3.3.4 set, 0:00.02, 17536KB, 0.00 user, 0.01 kernel 1, CFEngine 3.3.4 check, 0:00.02, 17520KB, 0.00 user, 0.01 kernel 1, Puppet 2.7.16 set, 0:06.20, 374752KB, 5.14 user, 0.91 kernel 1, Puppet 2.7.16 check, 0:06.22, 374528KB, 5.11 user, 0.97 kernel 10, system set, 0:00.00, 5504KB, 0.00 user, 0.00 kernel 10, system check, 0:00.00, 5488KB, 0.00 user, 0.00 kernel 10, CFEngine 3.3.4 set, 0:00.02, 17568KB, 0.00 user, 0.01 kernel 10, CFEngine 3.3.4 check, 0:00.02, 17520KB, 0.01 user, 0.00 kernel 10, Puppet 2.7.16 set, 0:06.28, 378336KB, 5.14 user, 0.96 kernel 10, Puppet 2.7.16 check, 0:06.19, 378192KB, 5.09 user, 0.94 kernel 100, system set, 0:00.00, 5520KB, 0.00 user, 0.00 kernel 100, system check, 0:00.00, 5520KB, 0.00 user, 0.00 kernel 100, CFEngine 3.3.4 set, 0:00.03, 17568KB, 0.01 user, 0.01 kernel 100, CFEngine 3.3.4 check, 0:00.02, 17552KB, 0.01 user, 0.01 kernel 100, Puppet 2.7.16 set, 0:07.77, 379184KB, 6.53 user, 1.10 kernel 100, Puppet 2.7.16 check, 0:07.18, 382912KB, 5.97 user, 1.05 kernel 1000, system set, 0:00.01, 5504KB, 0.00 user, 0.01 kernel 1000, system check, 0:00.00, 5504KB, 0.00 user, 0.00 kernel 1000, CFEngine 3.3.4 set, 0:00.05, 17680KB, 0.02 user, 0.02 kernel 1000, CFEngine 3.3.4 check, 0:00.03, 17664KB, 0.01 user, 0.01 kernel 1000, Puppet 2.7.16 set, 0:23.47, 610880KB, 20.60 user, 2.65 kernel 1000, Puppet 2.7.16 check, 0:17.62, 515184KB, 15.31 user, 2.11 kernel 2000, system set, 0:00.02, 6592KB, 0.00 user, 0.02 kernel 2000, system check, 0:00.00, 6560KB, 0.00 user, 0.00 kernel 2000, CFEngine 3.3.4 set, 0:00.07, 17664KB, 0.03 user, 0.03 kernel 2000, CFEngine 3.3.4 check, 0:00.04, 17648KB, 0.01 user, 0.02 kernel 2000, Puppet 2.7.16 set, 0:45.38, 668560KB, 40.77 user, 4.29 kernel 2000, Puppet 2.7.16 check, 0:32.01, 626832KB, 28.34 user, 3.46 kernel 5000, system set, 0:00.06, 9968KB, 0.01 user, 0.05 kernel 5000, system check, 0:00.01, 9936KB, 0.00 user, 0.01 kernel 5000, CFEngine 3.3.4 set, 0:00.14, 17664KB, 0.05 user, 0.07 kernel 5000, CFEngine 3.3.4 check, 0:00.07, 17648KB, 0.04 user, 0.02 kernel 5000, Puppet 2.7.16 set, 2:08.18, 1337632KB, 118.14 user, 9.51 kernel 5000, Puppet 2.7.16 check, 1:31.71, 1191152KB, 84.26 user, 6.95 kernel 10000, system set, 0:00.12, 15600KB, 0.02 user, 0.12 kernel 10000, system check, 0:00.03, 15552KB, 0.00 user, 0.02 kernel 10000, CFEngine 3.3.4 set, 0:00.24, 17664KB, 0.11 user, 0.11 kernel 10000, CFEngine 3.3.4 check, 0:00.12, 17648KB, 0.07 user, 0.04 kernel 10000, Puppet 2.7.16 set, 5:36.38, 2457280KB, 316.83 user, 18.17 kernel 10000, Puppet 2.7.16 check, 4:20.40, 1766560KB, 246.25 user, 12.97 kernel 20000, system set, 0:00.25, 26832KB, 0.03 user, 0.27 kernel 20000, system check, 0:00.06, 26800KB, 0.00 user, 0.05 kernel 20000, CFEngine 3.3.4 set, 0:00.45, 17664KB, 0.22 user, 0.22 kernel 20000, CFEngine 3.3.4 check, 0:00.22, 17648KB, 0.14 user, 0.07 kernel 30000, system set, 0:00.43, 38080KB, 0.06 user, 0.41 kernel 30000, system check, 0:00.08, 38048KB, 0.02 user, 0.06 kernel 30000, CFEngine 3.3.4 set, 0:00.71, 17664KB, 0.28 user, 0.38 kernel 30000, CFEngine 3.3.4 check, 0:00.31, 17648KB, 0.17 user, 0.13 kernel 50000, system set, 0:00.75, 60592KB, 0.12 user, 0.73 kernel 50000, system check, 0:00.14, 60560KB, 0.02 user, 0.11 kernel 50000, CFEngine 3.3.4 set, 0:01.17, 17680KB, 0.51 user, 0.61 kernel 50000, CFEngine 3.3.4 check, 0:00.52, 17648KB, 0.32 user, 0.19 kernel 100000, system set, 0:01.30, 116832KB, 0.22 user, 1.36 kernel 100000, system check, 0:00.28, 116800KB, 0.05 user, 0.22 kernel 100000, CFEngine 3.3.4 set, 0:02.25, 17680KB, 0.98 user, 1.21 kernel 100000, CFEngine 3.3.4 check, 0:01.01, 17648KB, 0.62 user, 0.35 kernel 200000, system set, 0:02.87, 229328KB, 0.45 user, 2.86 kernel 200000, system check, 0:00.58, 229296KB, 0.09 user, 0.47 kernel 200000, CFEngine 3.3.4 set, 0:04.64, 17664KB, 1.95 user, 2.46 kernel 200000, CFEngine 3.3.4 check, 0:02.01, 17648KB, 1.25 user, 0.74 kernel 500000, system set, 0:07.48, 566832KB, 1.09 user, 7.45 kernel 500000, system check, 0:01.44, 566800KB, 0.26 user, 1.16 kernel 500000, CFEngine 3.3.4 set, 0:11.75, 17664KB, 4.97 user, 6.24 kernel 500000, CFEngine 3.3.4 check, 0:05.01, 17648KB, 3.19 user, 1.80 kernel
And again with CFEngine 3.5.1p3, the most current release as of this writing:
1, system set, 0:00.00, 5520KB, 0.00 user, 0.00 kernel 1, system check, 0:00.00, 5504KB, 0.00 user, 0.00 kernel 1, CFEngine 3.5.1 set, 0:00.02, 17920KB, 0.01 user, 0.01 kernel 1, CFEngine 3.5.1 check, 0:00.02, 17936KB, 0.01 user, 0.00 kernel 10, system set, 0:00.00, 5504KB, 0.00 user, 0.00 kernel 10, system check, 0:00.00, 5488KB, 0.00 user, 0.00 kernel 10, CFEngine 3.5.1 set, 0:00.03, 17952KB, 0.01 user, 0.01 kernel 10, CFEngine 3.5.1 check, 0:00.02, 17920KB, 0.01 user, 0.01 kernel 100, system set, 0:00.00, 5520KB, 0.00 user, 0.00 kernel 100, system check, 0:00.00, 5504KB, 0.00 user, 0.00 kernel 100, CFEngine 3.5.1 set, 0:00.03, 17936KB, 0.01 user, 0.01 kernel 100, CFEngine 3.5.1 check, 0:00.03, 17920KB, 0.01 user, 0.01 kernel 1000, system set, 0:00.01, 5520KB, 0.00 user, 0.01 kernel 1000, system check, 0:00.00, 5504KB, 0.00 user, 0.00 kernel 1000, CFEngine 3.5.1 set, 0:00.08, 18016KB, 0.04 user, 0.03 kernel 1000, CFEngine 3.5.1 check, 0:00.05, 17984KB, 0.03 user, 0.01 kernel 2000, system set, 0:00.03, 6592KB, 0.00 user, 0.02 kernel 2000, system check, 0:00.00, 6560KB, 0.00 user, 0.00 kernel 2000, CFEngine 3.5.1 set, 0:00.11, 18000KB, 0.06 user, 0.04 kernel 2000, CFEngine 3.5.1 check, 0:00.07, 18000KB, 0.04 user, 0.02 kernel 5000, system set, 0:00.06, 9968KB, 0.01 user, 0.05 kernel 5000, system check, 0:00.01, 9952KB, 0.00 user, 0.00 kernel 5000, CFEngine 3.5.1 set, 0:00.22, 18016KB, 0.13 user, 0.08 kernel 5000, CFEngine 3.5.1 check, 0:00.13, 17984KB, 0.09 user, 0.03 kernel 10000, system set, 0:00.12, 15584KB, 0.01 user, 0.12 kernel 10000, system check, 0:00.03, 15552KB, 0.00 user, 0.01 kernel 10000, CFEngine 3.5.1 set, 0:00.40, 18000KB, 0.24 user, 0.15 kernel 10000, CFEngine 3.5.1 check, 0:00.24, 18000KB, 0.17 user, 0.06 kernel 20000, system set, 0:00.26, 26832KB, 0.04 user, 0.26 kernel 20000, system check, 0:00.05, 26800KB, 0.01 user, 0.04 kernel 20000, CFEngine 3.5.1 set, 0:00.79, 18016KB, 0.50 user, 0.27 kernel 20000, CFEngine 3.5.1 check, 0:00.47, 18000KB, 0.33 user, 0.13 kernel 30000, system set, 0:00.38, 38080KB, 0.06 user, 0.39 kernel 30000, system check, 0:00.08, 38048KB, 0.01 user, 0.07 kernel 30000, CFEngine 3.5.1 set, 0:01.21, 18016KB, 0.74 user, 0.44 kernel 30000, CFEngine 3.5.1 check, 0:00.69, 17984KB, 0.53 user, 0.14 kernel 50000, system set, 0:00.70, 60592KB, 0.09 user, 0.70 kernel 50000, system check, 0:00.14, 60560KB, 0.02 user, 0.11 kernel 50000, CFEngine 3.5.1 set, 0:01.94, 18000KB, 1.21 user, 0.70 kernel 50000, CFEngine 3.5.1 check, 0:01.15, 18000KB, 0.86 user, 0.25 kernel 100000, system set, 0:01.32, 116832KB, 0.22 user, 1.35 kernel 100000, system check, 0:00.28, 116816KB, 0.04 user, 0.23 kernel 100000, CFEngine 3.5.1 set, 0:03.93, 18016KB, 2.45 user, 1.38 kernel 100000, CFEngine 3.5.1 check, 0:02.23, 18000KB, 1.62 user, 0.59 kernel 200000, system set, 0:02.70, 229328KB, 0.42 user, 2.71 kernel 200000, system check, 0:00.56, 229296KB, 0.10 user, 0.44 kernel 200000, CFEngine 3.5.1 set, 0:07.96, 18016KB, 5.00 user, 2.77 kernel 200000, CFEngine 3.5.1 check, 0:04.56, 17984KB, 3.38 user, 1.14 kernel 500000, system set, 0:07.17, 566832KB, 1.14 user, 7.07 kernel 500000, system check, 0:01.38, 566800KB, 0.26 user, 1.11 kernel 500000, CFEngine 3.5.1 set, 0:20.63, 18016KB, 12.95 user, 7.11 kernel 500000, CFEngine 3.5.1 check, 0:11.09, 18000KB, 8.41 user, 2.63 kernel 1000000, system set, 0:15.29, 1129328KB, 2.29 user, 15.13 kernel 1000000, system check, 0:02.96, 1129296KB, 0.56 user, 2.38 kernel 1000000, CFEngine 3.5.1 set, 0:42.48, 18016KB, 25.03 user, 16.32 kernel 1000000, CFEngine 3.5.1 check, 0:22.29, 17984KB, 16.60 user, 5.62 kernel
