Some thoughts on Solar Power

This is something I wrote back in 2014 – thought it needed a wider audience.

The Solar Power Economy

What amazes me is that it is not mandatory for any new build house or other building with a south facing roof not to have it covered in solar panels as an intrinsic non negotiable part of the design. There is a whole estate of new housing going up near me (on the site of an old hospital) NOT ONE solar panel to be seen.

Yes, sunny deserts like the Sahara would be the most efficient for generation, and one could argue that unlike a lot of human interventions a solar array would actually enhance the local ecology by providing abundant shade. Clever design would also help the panels to help condense morning mists into usable water too. HOWEVER long distance power transit is another big cost. Letting people have local control over their solar power is a far more practical idea. Also many maximally sunny parts of the world are not the most politically stable so a HUGE solar array investment that people thousands of miles away depend on may be risky.

Giving up viable farm land to solar is not ideal (we need to eat too) so the next stage after roofing is the areas that have already been lost to agriculture i.e. roads, footpaths and other paved spaces.

IF this catches the public mood that we get momentum of people that want this and are willing to deploy it on their own land the costs will inevitably fall.

The pitch for the Solar “Freakin” Roadway is that a panel on the roof has one function, to generate power. There is NO other utility to it (it is putting extra weight on your roof, something to worry about in bad weather etc etc.) The road panels have so many secondary uses too.

The really important factor here is that they are ones that still have value even if mass solar deployment pushes the resale price of the power towards zero (which will have a HUGE positive effect on the prospects for the human race as a whole).

Remember it is the FEAR of this affecting their profits that has made the big energy corporations drag their feet over alternative energy for DECADES. They love the people to have the idea that bills only ever go up! WHY should this be the case?

3D VR for a pound!

I just popped into my local Poundland store today for some crisps and I saw these for sale

3d Viewer for £1!

I have seen relatively cheap(ish) “VR headsets” of the sort that take your mobile phone on sale for a while now but like a lot of people did not want to spend out on something I might not have any long term use for. The packaging says “Smartphone not included” – come on Poundland you disappoint me ūüėČ

The Viewer is mostly cardboard, plus some foam, Velcro and a couple of plastic magnifying lenses, held in place by the 3 layers of cardboard sandwich. There were clear instructions.

What you get in the bag, considerable assembly required!

And about 10 minutes later I had the finished article.

The completed 3D glasses. The phone fits in the box.

I have a Samsung Galaxy S4 phone that fitted snugly in the box. If you have a phone considerably larger than this then you may need to look for something more upmarket to house it.

The viewer claimed to be compatible with Google Cardboard this being a standard for base level VR hardware and software that works with it. However the one aspect of the official Google design missing is any way to press a button, thus unless you want to spend a little more money on a fancier handheld controller like this one, you are limited to experiences that involve just changing your view (i.e turning your head) rather than interacting (like playing a game). There is I am sure you are pleased to hear a very simple and cheap solution to this. If you have a USB mouse and one of those OTG usb adapters handy – the sort that lets you plug full size USB keys, keyboards etc. etc into a mobile phone you can use this with Cardboard. The support for mice in Android is such that a button click is the same as a touch event on the screen. Cardboard does not care where the touch event is on the screen, just that there is one, so a mouse connected to the phone and held in the hand does the job! If you have a Bluetooth connected mouse, or keyboard with track-pad ability, then you do not even have to have the trailing wire coming out of your phone – a slight damage risk to your expensive phone if you are blundering about with a VR headset on! However if you have to buy one you may as well get one of the purpose made Bluetooth VR controllers, designed to be used by touch alone and only a few pounds.

Apart from the lack of a button to press the ultra cheap headset works well. I am short sighted but without my glasses the screen image through the magnifier is clear, although with the magnifying lens unavoidably slightly pixelated. Looking at the specs of more expensive headsets what you get is the ability to adjust focus, but for me at least the default seemed fine. Getting a picture less pixelated would require a higher pixel density on the screen of the phone itself. In fact this application is the only real practical use for super high resolutions on a phone screen.

I do not know if prolonged use would result in headaches. It is all rather unnatural for our brain so common sense on how long you use this for in one session would be needed!

There is already quite a bit of software available in the App Store for Google Cardboard, as it a simple baseline for VR expecting just a single click control. I was rather limited not having any way to interact but still managed to enjoy a simulated roller-coaster ride and watching the fish swim by in an aquarium. I could tell my now 3 generations old Galaxy S4 was struggling a tiny bit with doing the graphics. VR is an area where an up to date phone would show the extra value.

I did feel rather vulnerable to real world dangers (like tripping over) while using the headset. Attempting to use it standing up I felt my knees buckling slightly as my balance and sense of motion was being messed with. To feel secure while experiencing a VR world the safest arrangement would be a sturdy swivel char positioned well away from desks or any other obstructions. For people with more space (and money) something like this would be great fun!

Just seen this story on BBC news that 3D TV is pretty much a bust now – interest from manufacturers has all but vanished. The public just did not go mad about it. VR shares some characteristics with 3D cinema in that the trick of giving each eye a slightly offset image trick our brain into conveying depth to a scene. However this whole ‘binocular vision’ thing has evolved for predators catching prey – requiring very keen spatial awareness in the space directly in front of them.

The problem with 3D cinema is this binocular vision trick is the sum total of what can be provided. the far stronger effect is that if we turn or nod our head the scene changes in a way we have evolved to expect. This gives our brains information about what objects are nearer than others, without needing binocular vision even. A prepared film cannot do this, the entire frame moves as one. It is like going through life with your head locked into position with a neck brace. The binocular effect on its own is not very satisfying.

The big experience value of VR through a headset like this is that the image occupies almost all the available field of view and responds to head movement. Without the latter is is nothing more than a very big 3D cinema screen you can carry with you, so in the end just as unsatisfying. The fact that you have to loose visual contact with the ‘real’ world limit the places where it can be used. Maybe what is needed is a harness so that the person with headset on can either stand up, but not able to wander into danger, or float freely away from any obstructions?


Docker thoughts

What is Docker?

A feature that has been part of Unix systems pretty much forever is the concept of the ‘chroot’. This is a special system call that makes the process from then on see the new root of the file-system being the directory the chroot is run from. If various files are in expected places, such as libraries, then software can still run within this smaller view of the world. Chroot is a very useful feature. First there is the obvious security applications such as running potentially exploitable network services inside ‘chroot jails’ – these are deliberately limited environments that contain enough files for a service to run without it having any access to the wider machine. Another useful trick with chroot that many Linux system owners may be need of from time to time is that a system can be recovered and run even if the bootstrap or kernel it is configured to use is corrupt in some way. See my article¬†Xtra-PC in depth for a practical example of how this is used.

This system call is (well is supposed to be) irreversible for the life of the process and any of its children. The only way supposed to be out of a ‘chroot jail’ is for all the jailed processes to die.

Cgroups¬†is a feature that has been in Linux kernels for nearly a decade now that extends the old chroot idea in lots of different other directions. As well as only seeing part of the file-systems, Cgroups or Control Groups allow groups of processes to be set up so that they cannot see each other, have a limit placed on allowed memory usage or share of CPU time and can treat things that can be shared such as network cards (physical and ‘virtual’) as if they are the only ones that see them. If this seems like virtualisation to you then yes, it is very close. The big difference however is that only a single Linux kernel is in place underneath it all. This means that this ‘containerised’ approach to partitioning off resources is much lighter weight than full blown virtualisation. With full virtualisation a complete kernel (Operating System) has to be set up onto of virtual hardware. This imposes costs in terms of memory and startup time but does mean you can run e.g. Windows programs under a Linux host (or vica-versa).

Docker is one of a number of technologies that leverage the Cgroups idea and further extends it to provide the tools to build “do one job well” microservices, scaling over multiple physical machines.

The problem that Docker solves is that in today’s Internet enabled software architecture the sensible way to build big things is out of smaller things until the individual components get down to a size where they can be easily understood, tested, debugged and generally inspire confidence. What is not needed when building a large and complex piece of software is side effects between those components. Many of these components in a modern system are built using languages like Java, Python and Ruby and Perl that themselves rely on a myriad of other libraries from various sources.

Although programmers when improving their software try not to break existing programs it often does happen. A bug may need to be fixed for package A to work reliably but forcing that change on package B may cause it to behave in strange ways. The Docker approach is to start from an appropriate base image (in effect a chroot tree of all the libraries and other support files expected in e.g. a base distribution of Debian Jessie) and then using a build script known as a Dockerfile to install either exact or just ‘latest’ versions of just what the specific aim of the container is.

Docker avoids what used to be called on Windows computers “DLL Hell” where it was impossible to have 2 different applications installed on the same machine because their shared library needs were incompatible. Linux machines have traditionally solved this issue by having a custom compiled ¬†version of the Perl or Python or whatever needed by a fussy application installed in a non standard location and overriding the default PATH & LD_LIBRARY_PATH settings for that application. This is a little messy and error prone. Docker improves on it as within each container everything can live in the expected place.

Splitting individual parts of the system up into microservice chunks forces the whole design to only use communication methods that are visible and explicit. Normally this would be via TCP or UDP protocol on either internal or external network interfaces but Docker does also allow areas of the host filesystem to be shared between one or more of the containerised services, either read/write or read only also.

How does this all not result in huge amounts of storage usage? The answer to that is Docker’s use of another Unix/Linux technology that has been around a long time. The Union filesystem. One of the earliest practical uses of the Union filesystem in Linux was the 1995 distribution Linux-FT. This was a ‘Live CD’ that cached any programs actually used to a much smaller file living on the host Windows hard disk. This allowed the system to start fast, get faster as it was used and only use an absolute minimum of the then very expensive hard disk space. At that time a 600MB CD was much bigger than most available hard disks! This trick was all done using a Union filesystem of the read only CD and a writeable filesystem inside a single file on the Windows disk.

Docker takes use of Union filesystems to a whole new level. Each action taken to customise a docker image to make a new image results in a new filesystem layer. These layers are only as large as they need to be to contain the  files that the new operation has changed e.g. installation of a specific version of a package. Even docker images that are downloaded from the Hub may consist of dozens of these layers Рas can be seen from the download process.

This extreme stratification gives opportunities for layer caching and reuse and provides an audit trail that should keep the security team happy.

A Docker handy commands crib


Installing docker depends on which flavour of Linux you are using. For example for Centos 7 the following needs to go into /etc/yum.repos.d/docker.repo

name=Docker Repository

Then install & start it with

sudo yum install docker-engine
sudo systemctl enable docker.service
sudo systemctl start docker

Note only the root user is permitted to start docker containers by default. To enable other users:

sudo groupadd docker
sudo usermod -aG docker yourusername

Basic Container Management

docker ps

Shows any current docker sessions, extra -l option shows just the latest created container (2 lines out output at the most).

docker ps -a

Shows all docker containers both active and stopped.

docker stop

Suspends execution of a current container – remember this is one of the neat tricks that Cgroups gives you.

docker start

Restarts a container that was stopped. Think of a laptop being woken from suspend.

docker rm

Removes a stopped container, forgets about it.

docker rm -f

Removes a container even while it is running! – think of the equally powerful rm -f command.


docker inspect image or container

This spits out details about the configuration of the image or the container as a block of JSON code. This has the advantage this it is both easy for humans to read and immediately usable from any programming language that has a JSON parser.

Okay but how do I get containers onto my computer?

Docker images come from the centralised repository called the Docker Hub. If you have registered for a Docker ID you will be able to contribute your own images to this resource but without that you can still choose to consume other peoples containers (and add more layers to them to make them your own).

docker search term

Looks on the docker hub for containers matching your search term. For example Alpine Linux is a very simple and minimal Linux layout ideally suited for use inside containers. Core container images officially supported by Docker have names of just one word. Images contributed by third parties are in the form author/container. Puppet uses a very similar convention for its Forge too.

docker search -f is-official=true alpine
alpine A minimal Docker image based on Alpine Lin... 1759 [OK]

The¬†-f is-official=true filter we are using here limits the output to just the base container that Docker officially sanctions. Other matches mentioning ‘alpine’ may be (most probably) based on this but have the hard work of adding other software expressed as those nice union filesystem layers. An image is a collection of such layers – an image that has other images that are derived from it cannot be removed unless those images are also removed.

docker run

docker run -it alpine /bin/sh
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
0a8490d0dfd3: Pull complete 
Digest: sha256:dfbd4a3a8ebca874ebd2474f044a0b33600d4523d03b0df76e5c5986cb02d7e8
Status: Downloaded newer image for alpine:latest
/ #

What this is doing is running a chosen command (/bin/sh) within the image called ‘alpine’ using an interactive terminal. If the alpine image is int already downloaded, or the one on the hub is newer and we state we want ‘latest’ rather than a specific version then it is downloaded. If there are several union layers that make up the image they all get downloaded simultaneously.

We show this by now repeating the run command for one of the other alpine based images on the hub:

docker run -it mritd/alpine   /bin/sh
Unable to find image 'mritd/alpine:latest' locally
latest: Pulling from mritd/alpine
0a8490d0dfd3: Already exists 
f22eacae62c9: Pull complete 
Digest: sha256:98023aa60f8398432f5d62ccd16cea02279bb9efc109244d93d441dd09100e18
Status: Downloaded newer image for mritd/alpine:latest
/ # 

This shows clearly that ‘mritd/alpine’ is based on the alpine files we have already downloaded plus an extra layer of some customisation or other adaptation that contributor felt worth sharing (not important for the purpose of this discussion).

Chunks of storage called volumes ¬†on the host can be added (mounted in) to the container with the -v argument to run. If a path in the container is mentioned on its own then the attached storage will be allocated by docker – but still exist outside of the union filesystem ‘layer cake’ of the image. If a colon separated pair of paths is used the host path is joined into the image as the destination path.

docker run -v `pwd`:/tmp/x -it alpine /bin/sh

Will run that super simple Alpine Linux image but if in the shell if you cd to /tmp/x you will find whatever was in the current directory on the host. If you would like the container to be able to look but not touch the data you are sharing add a :ro to the end.

-v `pwd`:/tmp/x:ro

Adding volume mounts means that the data lives outside of the life-cycle of that particular Docker image, and is not subject to size constraints of the original filesystem container – usually only a few GB. Volumes can also be created using the docker volume create command and can live on shared storage systems making them independent of any one host. See this documentation on flocker opening up to Docker ‘swarms’ operating over large numbers of hosts. This allows a software service to be rolled out in the modern fault tolerant “shared nothing” architecture without needing lots of dedicated physical machines, or even dedicated virtual machines.

Going in the other direction it is possible to narrow sharing down to an individual file level too:

docker run --rm -it -v ~/.bash_history:/root/.bash_history ubuntu /bin/bash

This example allows the shell within a ubuntu container to share your bash history and even add commands to it while running as root within that container. A handy trick while you are still experimenting with what need to go into that Dockerfile….

Lastly if a container has mounted volumes you can create a new container that mounts the same volumes – in effect shares the same data areas by using the¬†–volumes-from flag to run. This together with the shared storage paradigm allows large groups of distributed services all sharing the same underlying data to be created.

docker image

This shows what images we have locally (contrast this with docker ps for running and suspended instances) . This consists of both images that have been downloaded from the Hub and those we have made ourselves by applying a Dockerfile to the base of an existing image.

docker rmi

Rmi stands for remove image. Eventually you could end up with lots and lots of images you no longer have any use for. The rmi command will remove any image that is not referenced by another. For the above case if we want to docker rmi alpine we would need to add a -f (force) flag because we have another image dependent on it:

docker rmi -f alpine
Untagged: alpine:latest
Untagged: alpine@sha256:dfbd4a3a8ebca874ebd2474f044a0b33600d4523d03b0df76e5c5986cb02d7e8
Deleted: sha256:88e169ea8f46ff0d0df784b1b254a15ecfaf045aee1856dca1ec242fdd231ddd

If  dependencies are more complex it is not possible to use the -f Рsome scripting will be needed to find all the dependant images and remove them first.

docker build

If you want to make custom changes to an image create a directory with a Dockerfile text file plus any custom content you wish to go into an image (e.g. content for a web server). You can give the resulting image any name you like that is not already in use by another image. With the Dockerfile and command line options you can control aspects such as what network ports are accessible within the image, how much memory and CPU time it is permitted to use, and what parts of the host filesystem are exposed to it. Volume mappings as explained above in docker run can alos be specified in the Dockerfile. When the built images are run with docker run it is common for ports in the image to be remapped so that they can be visible from the outside world. For example you can have several web services configured within the container to operate on port 80 but from the outside world accessible on ports 8081, 8082 etc. Whole sets of micro-servers can be orchestrated using Docker Compose in a similar manner to how Amazon Cloud Formation or OpenStack HEAT templates work.

docker login

If you have signed up for a hub account you can use those credentials to log into it and then docker push mages you have created with docker build to it. This allows containerised infrastructure to be developed and then stored centrally for deployment.

This is not an exhaustive set of what Docker can do, what I have tried to do is give a useful summary and pointers for further study.


Some musings on minimal program size and startup overhead

I have been playing a little with Google’s Go language – the report that they are working a way of translating CPython code into Go for better ability to run in massively parallel systems got my interest. A project called Grumpy.

I have never used Go so intrigued by the introductory video available at the Go site I installed it on my Centos 7 Linux machine.

Here is that ubiquitous “Hello World” program in Go:

package main

import "fmt"

func main() {
    fmt.Printf("hello, world\n")

Unlike languages like Python and Perl, Go is very fussy where your code is before it will compile it for you.

In order to use the go compiler (simply) you need to set up a shell variable called GOPATH to where you are working. You also need to create a src directory under this then a directory for your project that finally contains the above code in a hello.go file

go install hello

Will if there are no errors quickly create a bin/hello program under the same directory specified by $GOPATH.

NOTE! You will need to be in the directory you have defined as GOPATH for this to work. Not the src sub directory or the directory containing the hello.go file itself. This took a little getting used to.

This blog is not about how to program with Go however. What interested me to take time out from my own learning of Go to write this is the comparative size and execution speeds of different languages.

Google’s motivation to undertake the work of running existing Python code is one of speed. Many key Google sites such as YouTube rely heavily on Python code so a faster and more scale-able way to run it (free from the Global Interpreter Lock) is obviously of great value to them.

The super simple Go program weighs in at 1.6M in size! When stripped it goes down to just under 1MB. I have been programming long enough to have considered this wildly extravagant in my youth ūüėČ

The equivalent absolutely minimal C program:

main() {
    puts("hello world\n");

when similarly stripped weighs in at just 6240 bytes. This figure may differ slightly depending on compiler version and options but it will always be a LOT less than 1MB.

OK what gives? The first thing I tested was if this program is genuinely compiled or has any outside dependencies. I did this by temporarily renaming the whole tree that the Go compiler had been installed into. The compiled program still ran without any issues – proving it is a true compiler. The complied program has everything it needs to function within that 1MB code budget. It could be deployed to another computer and run easily – a true compiler.

Is there any speed penalty for carrying around this code?

Well the super simple C version executes in 0.002 seconds – the resolution of the Linux time command only goes down to 0.001 second so this 0.002 seconds represents the absolute overhead for loading a program (any program) from a standing start.

The Go version? Only an extra thousandth of a second! That is not bad considering the program is so much larger. In terms of Linux system calls invoked (which can be found with the strace program) the Go version needs to make 146 syscalls compared to just 30 from the minimal C version.

How do other languages do?

Perl has long been used for quick fix scripts and it’s “hello world” is pretty impressive too!

time perl -e 'print "Hello World\n"'
Hello World

real 0m0.006s
user 0m0.002s
sys 0m0.004s

So only twice as slow as the compiled Go version – but only 30 bytes long

234 syscalls were required which goes some of the way to explaining why it is slower.


time python -c 'print "Hello World\n"'
Hello World

real 0m0.070s
user 0m0.063s
sys 0m0.007s

Considerably slower! Lets look at the number of syscalls needed:

strace python -c 'print "Hello World\n"' 2>&1 | wc -l

Well that’s more than 4 times as many syscalls which explains why things take longer.

These numbers will vary depending on just how much each work each language system has to do on startup. It may be that much more code is available to the Python compared to the Perl on my machine. Search paths that have to be initialised as the program starts to run.

Lastly Ruby.

Ruby is a pure interpreter. This gives the flexibility to write Ruby code on the fly within the program as it executes. This power in one way has a price to pay in pure speed however:

time ruby -e 'puts "Hello World\n"'
Hello World

real	0m0.100s
user	0m0.085s
sys	0m0.015s

A whole tenth of a second. The number of syscalls used (1106) is actually lower than for Python so the slowness cannot be blamed on latency in the OS.


Which language is best for writing “Hello World” ūüėČ

The Ruby version is some 500 times slower than the pure C version. But still.. a 10th of a second is fast enough! In any of these languages a program would have to get considerably more complex (such as doing lots and lots of looping) before the run time would go up much. There is a relatively large penalty for starting up a new process in the first place. This is why time critical tasks inside computer systems are served by pools of ready loaded and waiting for action processes.

Go seems a worth-while language to learn. The per compiled program 1MB space overhead may be an issue in some very tight on space environments but in general purpose computers 1MB is nothing now (but remember this was more than the entire address space of MS-DOS & the original IBM PC – how times have changed!)


This is a handy bit of Perl code I wrote years ago

It will quickly and efficiently tell you which of a group of machines you specify are ‘up’ – in the case of this code it is that they are listening on the port for ssh but it is trivial to change the code to look for something else e.g. http server on port 80.

The trick to the speed is that all machines are pinged at once and only then are the answers collected.

The other nice feature is that it will expand ranges in the form name[01..10] will ping machines called name01, name02 etc. This part can be adapted to suit whatever naming convention you have on your site. As we are using the Time::HiRes module we get detailed stats about how long it takes to resolve the name of each machine.

# general machine existance test. Martin Houston.
# checks that ssh service is running on a set of machines
# could be adapted to test another port e.g. 80 for http.
use Net::Ping;
use Data::Dumper;
use Time::HiRes qw( usleep 
 ualarm gettimeofday tv_interval nanosleep
 clock_gettime clock_getres clock_nanosleep clock stat );

# Take the names of the systems to ping on the command line
my (@systems) = @ARGV;
# or piped as a list
@systems = (<STDIN>) unless scalar(@systems) > 0;
# remove newlines
chomp for @systems;

@systems = expand_ranges(@systems);

my %trial = ();
my %lookuptimes = ();

# first we get how long it takes to look each system up 

for(@systems) {
    my $before = [gettimeofday];
    $lookuptimes{$_} = tv_interval ($before);
# allow 2 sec response
my $p = Net::Ping->new('syn',2);
# we check they are listening on the ssh port
$p->{port_num} = getservbyname('ssh', 'tcp');
# just send syn packets
map { $p->ping($_) } (@systems);
# collect all acks that arrive within 2 secs
while(my($host,$rtt,$ip) = $p->ack) {
    $trial{$host} = "'$ip' : $lookuptimes{$host} : $rtt";
# these machines have a ssh (port 22) service running
my @live = grep { defined $trial{$_} } @systems;
print join("\n", map { "$trial{$_} : '$_'," } @live), "\n";

# This works with computer names with numbers
# name[1..10] will expand to name01 name02 name03...
sub expand_ranges {
    my @res = ();
    for my $sys (@_) {
        if($sys =~ /\[/) {
            my $pre = $`;
            my $pat = '[' . $';
            # get Perl to do the hard work on pattern expansion
            my ($numbers) = eval($pat);
            for(@{$numbers}) {
                # 01 not 1, change to suit your conventions
                $_ = '0' . $_ if $_ < 10;
                push @res, "$pre$_";
       else {
           # not special
           push @res, $sys;
   return @res;

Please take and adapt this to what suits your site. A handy adaptation is to make a notpingable script that instead returns just the list of systems that are not responding. This, combined with testing various ports can be the basis of a very lightweight system monitor.

Python function decorators

I am refreshing my knowledge about Docker at the moment (may be the subject of another post in the near future) and one of the samples in the Docker course I was doing deployed, as part of a larger collection of services, a simple web app using the Python Flask micro-framework. I have done quite a bit of Python but the past but had not yet come across a construct like Flask uses (or rather I have seen them before in programs but never had the need to pry):

@app.route("/", methods=['POST','GET'])
def somefunc():
    return "Stuff to go to the web page"

Turns out the @ sign turns out to be something called a function decorator. In this case it takes a function that just spits out text for a web page and sets it up bound to a specific page on the site (in this case the root) for the Flask framework to serve on request. This is a neat abstraction as it means the page can be designed but what the url leading to that page can be easily changed in one place.

Function decorators is one of those concepts that seems rather daunting at first (a function returning a function returning a function HELP!) but is actually not that bad once you grasp what it is trying to do.

Think of a pipeline. A function decorator is something that can take the output of another function and change it in some way – such as upper casing any text or adding top & tail tag pairs such as are used in HTML or XML for example.

Of course you could do this the long way round by providing the output of one function as an argument to the function doing further processing, but this could get a bit messy and long-winded.

The @ sign notation is ‘syntactic sugar’ that allows such pipelines to be built up.

Decorator functions have to be defined to return a function that in turn does the actual work. This is needed as the decorator is bound to the function when it is defined, not every time it is used. Being able to do this although ‘syntactic sugar’ saves a lot of complex and potentially confusing coding if that function is called a lot.

There is a helper function from functools called wraps which is itself a decorator that aids your debugging efforts by normalising function names in stack back traces. I have added it in this example for that purpose. If the code does not need debugging you can leave it out.

from functools import wraps
def tags(tagn,opts = None):
    ''' decorate an output producing function with a tag '''
    def tags_decorator(func):
        # optionally itself decorated
        def func_wrapper(name):
            if opts is None:
                return "<{0}>{1}</{0}>".format(tagn, func(name))
                return "<{0} {2}>{1}</{0}>".format(tagn, func(name), opts)
       # back in tags_decorator
       return func_wrapper
   # back in tags return the function - NOT the result of calling it
   return tags_decorator
   # here we return a function(tags) that itself returns
   # a function(tags_decorator) that returns a function(func_wrapper)!

# now we use it
def h1(text)
    # note all the work done in the decorator in this case
    return text

def p(text)
   # note all the work done in the decorator in this case
   return text

print h1("This is a title")+p("And some centred text")

Note that the decorated function definition spans multiple lines.

It is not valid Python syntax to say @decorator() def decorated(): Remember in Python the choice and amount of white-space is significant. Cutting & pasting example code from websites may not always work because of this – check the logic of what you end up with. Another important tip on the syntax is that comments are permitted between the decorator and decorated function. This means that you could have several decorators and just un-comment the ones you want to use – easy trial of alternative presentation styles perhaps?

In conclusion function decorators allow processing to be split up pipeline fashion to keep logic in neat compartments. The above example is very simple but real uses could be something like substituting real values for placeholders that the inner function deals with. This makes it a good tool for imposing local configuration that the inner core of logic does not need to know about.

Linux 20 years ago – RedHat 4.1 on the PCW magazine coverdisk

20 years ago virtually nobody had fast internet. If you were outside of Government, big Corporations or Academia your Internet experience was most likely just a slow dial-up modem. This was great for exploring the early WWW or sending emails but asking someone to download enough software to leave the confines of DOS/Windows and start to explore Linux was daunting.

As explained in my other article vital to the growth of Linux back then were the itself relatively new technology of computer magazines coming with first CD-ROMS and then DVDs attached to their cover. You still see cover DVDs on many magazines with with widespread high speed broadband it is more convenience that necessity nowadays.

Continue reading “Linux 20 years ago – RedHat 4.1 on the PCW magazine coverdisk”

A problem with solid state storage (and a suggested solution)

This article also appears at so if you are on linked-in you can choose to comment there instead..

There are several computers inside an Ocado Warehouse robot, each controlling some aspect of the operation. The single ARM CPU Linux based computer within each robot I was working on had two jobs, one to manage the in situ upgrading and if needed downgrading (change back-out) for the firmware on the other control computers. This was only occasionally used so the relatively meagre performance of the hardware was not an issue. The other role was considerably more demanding. That of collecting all the low level debug information logs from the rest of the robot and delivering them for analysis when uploaded every time the robot docked for battery charging. While working out on the warehouse grid the robots were controlled by WiFi signals but there was not enough bandwidth available to send back a fire-hose of detailed status & debug information in real time. While the robots were under active development the volume of debug data grew and grew and the only place to put it until it could be uploaded was the micro SD card inside the control computer board. This micro SD card had a maximum write speed of about 10MB/s. Having spent several months looking after hundreds of computers behaving in this way I have a new respect for the reliability and longevity of standard consumer grade microSD cards but nothing was going to change the fact that they were too slow for the job of handling a potentially limitless appetite for debug information from developers trying to nail illusive problems.

Writing the logs to memory was much, much faster than to SD card but the control computer had only been specified with 512MB of RAM, as it was never envisaged that such a large volume of data would need to be collected during robot development. I did some research and found that with the fallocate system call it is also possible to punch holes in a file at the beginning as well as the usual file truncation action of freeing blocks at the end. What you are left with is a ‘sparse’ file where sections that have nothing stored on disk read as full of nulls. I found that if you truncate the beginning of a log file the processes writing that file simply do not care. It is still possible to append and also to ‘tail’ the file to read back the most recent contents. The file can grow and grow to whatever the size limit is of the underlying file-system but only occupy a small amount of actual space. This discovery allowed me to collect logs into ram-disk instead of direct to the slow SD card. I used inotify system calls to watch a whole tree of log files being written alerting a process which collected all but the last few KB of each file and produced a compressed multiplexed version of all of them. The actions of compressing and multiplexing increased the effective write rate of the SD card enough to cope with much higher rates of logging activity, in effect kicking the can far enough down the road that the developers could have what logging they liked. The way SD cards work is that if 4MB of data is all written at once in a single file stream the write is much more efficient as that is the size of the erase block in the SD technology. I thought I had a prefect solution! However when fully stress tested I found it was missing one component, and one that would be non trivial to write.

I was able to run tests with simulated log data writing into the ram-disk that eventually overran the ability of the inotify driven background process to catch up. It took many minutes but slowly the ram-disk would fill up completely forcing the writing of log-file data to fail ‚Äď implying missing, possibly vital, log information. What would be nice, I thought, was if the system calls writing data to a file-system that was in danger of getting full could be slowed in some way, just by a few microseconds, that would give the background process time to catch up.

An old mechanical hard disk would in a way do this. The furious search for free blocks would increase IO wait time so that writing blocks to a nearly full file-system would indeed take longer. However regressing back to mechanical disks is no solution as the forced head movements would also hamper processes that would be reading and consuming the data too!

What the Linux kernel needs is some way to simulate the slowing effect of writes that a nearly full file-system has on processes wanting to make the situation worse, and in a gradually increasing severity, but no corresponding penalty to readers (and removers) of data. I knew this would solve my immediate problem and then realised that it would have highly beneficial effects for data storage in the Enterprise too. Having file-systems and filers which exhibited this behaviour would get an early warning on file systems filling up, but most importantly a way to delay the inevitable full file-system crisis. With fancy enough monitoring it would be possible to isolate the issue to a single ‚Äúway too chatty‚ÄĚ application. The rate of log writing just for that process could be slowed so that the team responsible would have time to sort out what is going wrong. The fallocate trick I had found for dealing with the robot logs would also come in handy here – if a log has been discovered that has been growing for months or even years (a failure to implement log rotation), then a fallocate punched hole could be used to archive or just dispose off all data too old to be interesting without having to disrupt the running process at all.

Even if the rate for a single process had to be slowed to effectively a full stop it is definitely ‚Äúthe lesser of two evils‚ÄĚ than allowing the file-system it is writing to to fill up. That would more likely have collateral damage to other well behaved parts of the infrastructure that were using that portion of storage space responsibly. The normal panic mode thing that system admins have to do in such a situation, on filers which have that luxury, is to give that file-system more (expensive) storage. This is a costly way to “fix” the problem and it does nothing to address the reasons why that file-system got full in the first place.

This was several months ago now and at the time I did a search so see if any such feature already exists in the kernel but drew a blank. As I had seen enough email circulars of how Ocado was keen on maximising their IP I put forward a proposal to do this seemingly missing part of Linux kernel capability as a project. My request was turned down (even though it was the missing piece needed to solve the log collecting issue with the robots). I was told I was welcome to do the work ‚Äúin my own time‚ÄĚ. Now I no longer work there here is my chance to ask people 1. does this technology already exist or 2. anyone fancy giving me a hand in writing it?

Puppet commands mini crib

Just a list of useful Puppet commands and key concepts

Not intended to really be a tutorial… If you find it useful I am glad, but I am learning Puppet myself and I always find the best way to really understand something is finding a really simple way to explain it. This is my attempt to distil Puppet down to the bare essentials that need to be grasped. It is a work in progress.

puppet help

Come on this one should be obvious!
The puppet help command on its own gives a list of all available sub commands. Including one of them after the word help gives detailed help on how to use it – use that to expand on what you learn from this crib sheet! The puppet man command gives a full manual page about the command – an expansion on the summary that puppet help gives.

puppet describe –list

This gives a list of all the different sorts of things that the puppet you have installed and configured knows how to control – i.e. that puppet code has been written to manage. This of course will vary with your Puppet set-up. If you are trying to apply a manifest containing resources your Puppet has no clue how to manage you are not going to get very far are you? You may want to pipe this through your favourite pager as the list may be long!

puppet describe <something from the list>

Gives the equivalent ¬†of a “manual page” for that particular “thing” – what options are available for controlling it. This is a definition of what should be put in the manifest for that sort of resource. For example here are the options for “mount”

Manages mounted filesystems, including putting mount
information into the mount table. The actual behavior depends
on the value of the 'ensure' parameter.
**Refresh:** `mount` resources can respond to refresh events (via
`notify`, `subscribe`, or the `~>` arrow). If a `mount` receives an event
from another resource **and** its `ensure` attribute is set to `mounted`,
Puppet will try to unmount then remount that filesystem.
**Autorequires:** If Puppet is managing any parents of a mount resource ---
that is, other mount points higher up in the filesystem --- the child
mount will autorequire them.


- **atboot**
    Whether to mount the mount at boot.  Not all platforms
support this.

- **blockdevice**
    The device to fsck.  This is property is only valid
    on Solaris, and in most cases will default to the correct

- **device**
    The device providing the mount.  This can be whatever
    device is supporting by the mount, including network
    devices or devices specified by UUID rather than device
    path, depending on the operating system.

- **dump**
    Whether to dump the mount.  Not all platform support this.
    Valid values are `1` or `0` (or `2` on FreeBSD). Default is `0`.
    Values can match `/(0|1)/`.

- **ensure**
    Control what to do with this mount. Set this attribute to
    `unmounted` to make sure the filesystem is in the filesystem table
    but not mounted (if the filesystem is currently mounted, it will be
    unmounted).  Set it to `absent` to unmount (if necessary) and remove
    the filesystem from the fstab.  Set to `mounted` to add it to the
    fstab and mount it. Set to `present` to add to fstab but not change
    mount/unmount status.
    Valid values are `defined` (also called `present`), `unmounted`,
    `absent`, `mounted`. 

- **fstype**
    The mount type.  Valid values depend on the
    operating system.  This is a required option.

- **name**
    The mount path for the mount.

- **options**
    Mount options for the mounts, comma-separated as they would appear
    in the fstab on Linux. AIX options other than dev, nodename, or vfs may
    be defined here. If specified, AIX options of account, boot, check,
    mount, size, type, vol, log, and quota must be alphabetically sorted at
    the end of the list.

- **pass**
    The pass in which the mount is checked.

- **remounts**
    Whether the mount can be remounted  `mount -o remount`.  If
    this is false, then the filesystem will be unmounted and remounted
    manually, which is prone to failure.
Valid values are `true`, `false`. 

- **target**
    The file in which to store the mount table.  Only used by
    those providers that write to disk.


Note that the puppet man command gives a very similar detailed output for puppet commands themselves.

puppet resource

This is a way to “hand crank” Puppet – to get Puppet to set the options for a resource the same way as if a manifest was to be applied. As an extra bonus option you get emitted a manifest entry that would result in the same puppet resource (Note as we are doing something to change the system ¬†sudo is needed):

sudo puppet resource user bogus ensure=present

Notice: /User[bogus]/ensure: created
user { 'bogus':
  ensure => 'present',

And we can see that the user has indeed been created:

grep bogus /etc/passwd

And remove it again:

sudo puppet resource user bogus ensure=absent
Notice: /User[bogus]/ensure: removed
user { 'bogus':
  ensure => 'absent',

And the user is no longer in the /etc/passwd file.

Note if you just want the emitting manifest syntax side effect use the –noop flag ¬†at the end of the command – so that actions are not actually done. The resource command still has to be one puppet would be able to do (correct parameters for the resource type etc).

Summary is that puppet resource is an opportunity to get puppet to do individual actions. This is both a handy way to do low level code testing and gives syntactically correct code to paste into manifests for automated repetition. Puppet manifest syntax (Puppet DSL) takes a bit of getting used to as it is rather richer in meaning than it looks at first glance to be.

Without the option=value parts puppet resource can be used to query the current state of managed resources, again emitting the result as syntax suitable for a manifest file (Note sudo still required – this data may well be sensitive!):

 sudo puppet resource user root

user { 'root':
  ensure           => 'present',
  comment          => 'root',
  gid              => '0',
  home             => '/root',
  password         => '$6$Fzx4UD.89YeLcYry$ph.N6w1t.dSzmn/dycQ0sGTGL/gGsgI8JTo94nwqapffTbruqDhrSgKE.G132RmJ.q03lT.MEMLDahG7XH.',
  password_max_age => '99999',
  password_min_age => '0',
  shell            => '/bin/bash',
  uid              => '0',

All the above options could be supplied¬†to recreate a root account as it exists. Many of them however are defaults – such as our ‘bogus’ user got a bash shell without having to explicitly specify it.

puppet apply

The blocks of Puppet DSL emitted by puppet resource, or new creations with the same syntax, can serve as input to the next stage up the food chain – puppet apply

Using the same example as before – create a bogus user:

sudo puppet apply --execute "user { 'bogus': ensure => 'present',}"
Notice: Compiled catalog for in environment production in 0.44 seconds
Notice: /Stage[main]/Main/User[bogus]/ensure: created
Notice: Finished catalog run in 1.82 seconds

Note that although the end effect is the same, the output messages from puppet apply are rather different and more like what you get to see on a ‘real’ puppet run. Note -e can be used as shorthand for –execute.

Using nothing more than puppet apply it is possible to develop new pieces of Puppet DSL and test them in situ before tested manifest files get installed on a puppet master.

If instead of being squashed onto 1 line for use with –execute, that DSL for creating the bogus user was simply saved in a file called bogus.pp (Puppet manifest files by convention end in .pp) then

puppet apply bogus.pp

would have the same effect of creating the user.

Now we have this action captured in a file it becomes easy to start adding parameters, like giving the user a password and other options appropriate to the user resource. Remember “puppet describe user” will give us full details on what options are possible for this resource.

puppet config print

This will print out the current config that puppet is using on this machine as a long list of name=value pairs. Changes to the config can be achieved with puppet config set name value changes this way end up in /etc/puppet/puppet.conf for the Open Source version and /etc/puppetlabs/puppet/puppet.conf for the Puppetlabs Enterprise version. These files can also be edited by hand for more complex config (with comments). A particular point of limitation with using puppet config set is that the variables only end up in the [main] section so could be overridden if thee has been a set variable in a more specific section for [agent].

sudo puppet config print confdir will show where the configuration data for your puppet is set (/etc/puppet for root in the Open Source version). Note that if you run this without the sudo you will get the answer $HOME/.puppet Рthis means that  you can work with and tests some aspects of Puppet without needing root, however it is a trap for the unwary as configuration will differ depending on if root is used or not!

Enabling a development environment

If $confdir is /etc/puppet then we need to set up some environments under there where Puppet will expect them to be.

sudo mkdir -p /etc/puppet/environments/development

sudo puppet config set environment development

This is saying that the ‘bootstrap’ or ‘root’ for Puppet configuration to be applied to this machine while it is in ‘development’ status starts with a site.pp file located under the manifests directory here. This structure becomes particularly important when a machine becomes ¬†a puppet master, storing the configuration for many different machines that are configured to take their setup from different states. Some machines may be in ‘development’ state, others in ‘preprod’ or ‘production’. In big Puppet deployments a version control system like git and a formal change control procedure are used to make sure that Puppet code does not reach critical production machines unless it has been properly tested with less critical ones.

puppet module generate

In order to tailor puppet to our system in non trivial ways we will need to generate one or more modules:

puppet module generate yournamemodname –skip-interview

This generates the whole expected boilerplate structure for a module in the current directory under a directory called yourname-modname. Your first action should be to rename that directory to just modname as we are just using self created modules to describe the setup of this site at a high level. The yourname bit is just to give some namespace separation if you later decide to share your modules, or are making modules to be shared, so in this case is simply not needed, the top level config is really going to be what is unique about your site.

The two really critical files here are manifests/init.pp and tests/init.pp. The former defines the meat of the module, you will probably want several .pp files but init.pp is the entry point (in the same way that site.pp is the entry point for an entire puppet run) and the latter exercises it for test purposes. Modules have to be invoked from somewhere in order to do anything – the tests section gives an opportunity for unit testing of the Puppet code being written.

Here is a simple example of this:

Go to the directory you have defined as development (e.g. etc/puppet/environments/development)

Create modules directory and change to it.

puppet module generate nevermind-donothing –skip-interview

mv nevermind-donothing donothing

Now edit donothing/init.pp and add this to the empty donothing class:

notify { ‘Applying class donothing’:

puppet apply donothing/tests/init.pp # note we have not edited this – its default behaviour of invoking the class is enough for this.
Notice: Compiled catalog for in environment development in 0.11 seconds
Notice: Applying class donothing
Notice: /Stage[main]/Donothing/Notify[Applying class donothing]/message: defined ‘message’ as ‘Applying class donothing’
Notice: Finished catalog run in 1.76 seconds

Because we are working in the place puppet expects things to be we do not need a –modulepath=./ that would be required if you want to develop module code in arbitrary other places

puppet module search & install

The other sort of modules are ones that are installed from the puppet forge. These are just tared up archive trees the same as created originally by module generate.  One of the things that makes Puppet so powerful is that when sysadmins solve a problem about how to orchestrate something with Puppet the most often contribute the code back for others to use and improve. This is the whole essence of why the Open Source idea is so powerful.

puppet module search for a named module confirms that it exists and offers possible alternatives.

puppet module search puppetlabs-apache
Notice: Searching ...
puppetlabs-apache Installs, configur... @puppetlabs web ssl 
dploeger-pagespeed Installs the Apach... @dploeger 
mayflower-php Generic PHP module... @mayflower php fpm 
maestrodev-rvm A puppet module fo... @maestrodev ruby rvm 
jgazeley-django Deploy a Django ap... @jgazeley 
hetzner-roundcube Roundcube webmail ... @hetzner webmail 
puppet-puppetboard Install and config... @puppet puppetdb 
saw-reviewboard Install and contro... @saw trac 
jhoblitt-awstats Manages the AWStat... @jhoblitt 
nibalizer-puppetboard Install and config... @nibalizer redhat 
spotify-puppetexplorer Manage the Puppet ... @spotify puppetdb 
pulp-pulp This module can be... @pulp 
pltraining-dockeragent Manages Puppet Lab... @pltraining 
landcareresearch-amazon_s3 Manages mounting S... @landcareresearch 
thejandroman-grafana Installs and confi... @thejandroman 
thejandroman-kibana3 Installs and confi... @thejandroman kibana 
gajdaw-symfony Puppet module to i... @gajdaw php app 
jgazeley-mod_auth_cas Configure mod_auth... @jgazeley httpd cas 
42ways-railsapp Basic Rails Server... @42ways rails ruby
jgazeley-speedtest Install Ookla Spee... @jgazeley

puppet module install actually installs it e.g.

puppet module install puppetlabs-apache
Notice: Preparing to install into /etc/puppet/environments/development/modules ...
Notice: Downloading from ...
Notice: Installing -- do not interrupt ...
‚ĒĒ‚ĒÄ‚Ē¨ puppetlabs-apache (v1.11.0)
  ‚Ēú‚ĒÄ‚ĒÄ puppetlabs-concat (v2.2.0)
  ‚ĒĒ‚ĒÄ‚ĒÄ puppetlabs-stdlib (v4.14.0)

Note as with creating your own modules a –modulepath= option will allow you to install modules somewhere other than the standard place Puppet is going to be looking (under the development branch in our case). Note that there is dependency following going on here, just like with Linux packages. As well as the puppetlabs-apache we wanted we also get two modules it in turn needs.

These modules also get installed under


so you can see the rest of how modules need to be constructed by looking at other people’s examples. Lower level modules will tend to have actual Ruby code as well as Puppet DSL declarations.

Fun with facts

The facter command without parameters will list all the custom facts available to Puppet on this system. If names of facts are listed on the command line only those as name => value pairs will be retuned. If an unknown fact is asked for like ‘wibble’ it gets mapped to nil.

facter is_virtual operatingsystem wibble
is_virtual => true
operatingsystem => CentOS
wibble => nil

It is possible to define custom facts so that anything that Puppet needs to be able to make decisions about configuration of systems.

There are 2 places facts are used within the Puppet DSL. Firstly within Templates – the .erb files. A fact can be embedded using this syntax:

 <%= @operatingsystem %>

Actually facts are not really local variables (this same syntax would be used for showing those in a template too) so a little bit more correct (and if you had the misfortune to have a local variable name conflicting with a fact:

<%= scope['::operatingsystem'] %>

Within Puppet DSL in the .pp files a $:: prefix is used (normal local variables just have a $ prefix within .pp files.

$OS = $::operatingsystem

Classes and defined types

We have already seen defined types as built into puppet as e.g. the user.


You can define your own types however with default values for any parameters that can then be overridden. Remember that donothing class we created earlier? Extend it like this:

class donothing {
 notify { 'Applying class donothing':
 bogus { 'suspect':
   flavour => 'suspect'

 bogus { 'plain': }

define bogus ($flavour='stinky') {
 notify { "Seen a $flavour bogus type": }

puppet apply -e "include donothing" 
Notice: Compiled catalog for in environment development in 0.08 seconds
Notice: Seen a stinky bogus type
Notice: /Stage[main]/Donothing/Bogus[plain]/Notify[Seen a stinky bogus type]/message: defined 'message' as 'Seen a stinky bogus type'
Notice: Seen a suspect bogus type
Notice: /Stage[main]/Donothing/Bogus[suspect]/Notify[Seen a suspect bogus type]/message: defined 'message' as 'Seen a suspect bogus type'
Notice: Applying class donothing
Notice: /Stage[main]/Donothing/Notify[Applying class donothing]/message: defined 'message' as 'Applying class donothing'
Notice: Finished catalog run in 1.82 seconds

Note a class does not have this ability – they are intended to be top level containers – a file with the same name as the class and ending in .pp contains the class.

To include such a file (found on the search path) within another class we just use include – like

include apache

or as we did above:

puppet apply -e "include donothing"

This assumes that the donothing class can be found on your search path.
Add –search-path=./ if need to set it for explicit test.

Note that if a defined type is specific to a particular module the convention is to prefix it with the module name and double colon e.g. apache::vhost

Classes can also have parameters – which allows some aspects of how that class is used to be changed when the class is invoked. The syntax is just a little different to defined types:

In a file withparams.pp:
class withparams($param_one='this',$param_two='other') {
# body does not matter...

# and on the command line
puppet apply -e  "class { 'withparams': param_one => 'that' }"

This will result in the ‘withparams’ class getting used but with the $param_one variable set to ‘that’ and $param_two with the default value of ‘other’ Note that we are being absolutely explicit here that a class is being called with non standard arguments. Calling a class without overriding any of the arguments is done with just ‘include’ as we have seen before. Remember puppet is NOT procedural. A class calling another class is just establishing a relationship of things that have to be done. The order they are done in needs to be separately controlled if indeed it is important. Remember apart from init.pp which is the special top level the class name must match the file name it is in as this is how classes are found when other classes request to include them.

puppet apply -e “class { ‘withparams’:}”

is exactly the same as

puppet apply -e “include withparams”

require/notify and the arrow notation -> ~>

Stating in the Puppet DSL that for specifying that other things have to be in place before the service will run (like a configuration file for example) or another resource can be created Рthis is done with the  require keyword.

This code:

file {'myconffile':
# commands to populate the file e.g. from a template
service {'myservice':
  require File['myconffile']
  # other stuff

Could instead be written as 
file {'myconffile':
# commands to populate the file e.g. from a template
} ->
service {'myservice':


There is an equivalent when you have the file but want to say what service is affected by it which is called  notify. The other way to state this is with a ~> arrow Рwith the tilde character. Having -> and ~> means that you can add the appropriate require or notify without having to change the DSL source of the other component, which is handy as it might be a contributed module written by someone else.

Note that the -> or ~> is used as a joiner between the two text blocks. For require in particular long chains can be implied that first thing needs to be done first then second then third.. e.g create a directory, then create a sub directory then create some files in it. Using explicit requires in this case would look messy.

Puppet master

Although puppet code can be developed and run on any host directly a complex system where servers interrelate to each others requires one machine to be set up as a puppet master. If the puppet master is down all that is lost is the ability for changes to be made to a deployed configuration. Therefor there is no special need for the puppet master to be designed as fault tolerant. Even production networks can tolerate short outages of the puppet master. What is VITAL however is that the puppet master is secure as gaining root on it is potentially compromising as root access to any of the machines being controlled by that master. This is because the master machine controls exactly what happens to the agent machines by deploying puppet code to them. This is why it is called Puppet.

Systems to be controlled register with the puppet master by using puppet agent mode. Communication is authenticated using SSL so each agent needs to issue a SSL certificate and have it signed by the master. Signing of certificates by the master can be automated but if security is important is best left as manual as new systems also have to have decisions made to them about which puppet environment (development or production) is controlling them and what role the have within that environment. This determines how that machine is to be configured by the puppet master e.g. as a web server.

For trivial testing and small deployments there is a simple, single threaded web server built directly into puppet so that the puppet master can be easily started and stopped and debugged etc.

puppet master --debug --verbose --no-daemonize --logdest console

This¬†will run in the foreground and emit verbose output direct to the console (have a nice big scroll-able terminal to run it in).¬†However for larger deployments using the puppet master in conjunction with a proper multi-threaded web server like Apache or there is a puppetserver package also available which comes from puppetlabs and is a puppetserver written in jruby (therefore running under the java jvm) for performance. This will usually be available as a package called ‘puppetserver’.

“bootstrapping” machines to use a puppet master

As stated earlier it is easy just to use puppet locally and invoke specific pieces of the Puppet DSM by hand using puppet apply to test them. However the whole point of puppet is to enable the control of complex setups that span over dozens to thousands of machines. In order for this to happen there needs to be a central place where all the machine configurations are decided and actions issued to get machine configurations into line with what they should be.

In order for a machine to be joined into the Puppet infrastructure these things need to happen.

  1. Enough of puppet code needs to be present on the machine to allow the puppet agent function to be performed. This would be the responsibility of initial system build from a technology like kickstart or cloning a basic system image when a new VM is created.
  2. That machine needs to know where to find the puppetmaster – this will either be an alias in whatever DNS domain the newly created machine has been given (possibly handed out by a DHCP server) or, more simply an entry in the local hosts file put there by whatever initial build process is in place.
  3. Command puppet config set certname with a FQDN of a name that you want this node to be know to the puppet master as. This name is vital as it must correspond to an entry in the manifests/site.pp file on the puppetmaster.
  4. Command puppet agent -t run to present this certificate to the puppet master for signing.
  5. Unless the puppet master has been set to auto-sign all certificate requests (unwise) the sysadm now must take action to sign the certificate.
  6. A further puppet agent command will start regular background communication with the master. Note that the master does not  itself need 100% availability. Loss of puppet connectifity only affects ability to change configuration, not actual running of the machine itself.
  7. Now communication is established it is possible to use the node classifier on the master to assign this node into whatever group is desired. The site.pp within that group then acts as a starting point for the Puppet configuration and therefore the role that machine will then take.


Child Abuse in Football: The FA is not Matchfit

An urgent communication from the Whiteflowers Campaign Рreproduced here as received by me just with a bit of tidying from email to blog form: See also 19th December update at the end.

The Football Association was forced to replace the head of its inquiry into child abuse in football within ten days. At least the toppled Kate Gallafent QC can claim that she lasted longer than then-Home Secretary Theresa May’s first child abuse inquiry appointee, Judge Butler Sloss.

The bookies may yet offer odds on how long Gallafent‚Äôs replacement, Clive Sheldon QC will last. If the FA wishes to get to the bottom of what happened then it may appear strange to many survivors when they appoint a man whom the Jewish Chronicle recently highlighted as having a ‚Äúrecord of defending the establishment‚ÄĚ

Continue reading “Child Abuse in Football: The FA is not Matchfit”