Docker thoughts

What is Docker?

A feature that has been part of Unix systems pretty much forever is the concept of the ‘chroot’. This is a special system call that makes the process from then on see the new root of the file-system being the directory the chroot is run from. If various files are in expected places, such as libraries, then software can still run within this smaller view of the world. Chroot is a very useful feature. First there is the obvious security applications such as running potentially exploitable network services inside ‘chroot jails’ – these are deliberately limited environments that contain enough files for a service to run without it having any access to the wider machine. Another useful trick with chroot that many Linux system owners may be need of from time to time is that a system can be recovered and run even if the bootstrap or kernel it is configured to use is corrupt in some way. See my article Xtra-PC in depth for a practical example of how this is used.

This system call is (well is supposed to be) irreversible for the life of the process and any of its children. The only way supposed to be out of a ‘chroot jail’ is for all the jailed processes to die.

Cgroups is a feature that has been in Linux kernels for nearly a decade now that extends the old chroot idea in lots of different other directions. As well as only seeing part of the file-systems, Cgroups or Control Groups allow groups of processes to be set up so that they cannot see each other, have a limit placed on allowed memory usage or share of CPU time and can treat things that can be shared such as network cards (physical and ‘virtual’) as if they are the only ones that see them. If this seems like virtualisation to you then yes, it is very close. The big difference however is that only a single Linux kernel is in place underneath it all. This means that this ‘containerised’ approach to partitioning off resources is much lighter weight than full blown virtualisation. With full virtualisation a complete kernel (Operating System) has to be set up onto of virtual hardware. This imposes costs in terms of memory and startup time but does mean you can run e.g. Windows programs under a Linux host (or vica-versa).

Docker is one of a number of technologies that leverage the Cgroups idea and further extends it to provide the tools to build “do one job well” microservices, scaling over multiple physical machines.

The problem that Docker solves is that in today’s Internet enabled software architecture the sensible way to build big things is out of smaller things until the individual components get down to a size where they can be easily understood, tested, debugged and generally inspire confidence. What is not needed when building a large and complex piece of software is side effects between those components. Many of these components in a modern system are built using languages like Java, Python and Ruby and Perl that themselves rely on a myriad of other libraries from various sources.

Although programmers when improving their software try not to break existing programs it often does happen. A bug may need to be fixed for package A to work reliably but forcing that change on package B may cause it to behave in strange ways. The Docker approach is to start from an appropriate base image (in effect a chroot tree of all the libraries and other support files expected in e.g. a base distribution of Debian Jessie) and then using a build script known as a Dockerfile to install either exact or just ‘latest’ versions of just what the specific aim of the container is.

Docker avoids what used to be called on Windows computers “DLL Hell” where it was impossible to have 2 different applications installed on the same machine because their shared library needs were incompatible. Linux machines have traditionally solved this issue by having a custom compiled  version of the Perl or Python or whatever needed by a fussy application installed in a non standard location and overriding the default PATH & LD_LIBRARY_PATH settings for that application. This is a little messy and error prone. Docker improves on it as within each container everything can live in the expected place.

Splitting individual parts of the system up into microservice chunks forces the whole design to only use communication methods that are visible and explicit. Normally this would be via TCP or UDP protocol on either internal or external network interfaces but Docker does also allow areas of the host filesystem to be shared between one or more of the containerised services, either read/write or read only also.

How does this all not result in huge amounts of storage usage? The answer to that is Docker’s use of another Unix/Linux technology that has been around a long time. The Union filesystem. One of the earliest practical uses of the Union filesystem in Linux was the 1995 distribution Linux-FT. This was a ‘Live CD’ that cached any programs actually used to a much smaller file living on the host Windows hard disk. This allowed the system to start fast, get faster as it was used and only use an absolute minimum of the then very expensive hard disk space. At that time a 600MB CD was much bigger than most available hard disks! This trick was all done using a Union filesystem of the read only CD and a writeable filesystem inside a single file on the Windows disk.

Docker takes use of Union filesystems to a whole new level. Each action taken to customise a docker image to make a new image results in a new filesystem layer. These layers are only as large as they need to be to contain the  files that the new operation has changed e.g. installation of a specific version of a package. Even docker images that are downloaded from the Hub may consist of dozens of these layers – as can be seen from the download process.

This extreme stratification gives opportunities for layer caching and reuse and provides an audit trail that should keep the security team happy.

A Docker handy commands crib

Installing…

Installing docker depends on which flavour of Linux you are using. For example for Centos 7 the following needs to go into /etc/yum.repos.d/docker.repo

[dockerrepo]
name=Docker Repository
baseurl=https://yum.dockerproject.org/repo/main/centos/7/
enabled=1
gpgcheck=1
gpgkey=https://yum.dockerproject.org/gpg

Then install & start it with

sudo yum install docker-engine
sudo systemctl enable docker.service
sudo systemctl start docker

Note only the root user is permitted to start docker containers by default. To enable other users:

sudo groupadd docker
sudo usermod -aG docker yourusername

Basic Container Management

docker ps

Shows any current docker sessions, extra -l option shows just the latest created container (2 lines out output at the most).

docker ps -a

Shows all docker containers both active and stopped.

docker stop

Suspends execution of a current container – remember this is one of the neat tricks that Cgroups gives you.

docker start

Restarts a container that was stopped. Think of a laptop being woken from suspend.

docker rm

Removes a stopped container, forgets about it.

docker rm -f

Removes a container even while it is running! – think of the equally powerful rm -f command.

 

docker inspect image or container

This spits out details about the configuration of the image or the container as a block of JSON code. This has the advantage this it is both easy for humans to read and immediately usable from any programming language that has a JSON parser.

Okay but how do I get containers onto my computer?

Docker images come from the centralised repository called the Docker Hub. If you have registered for a Docker ID you will be able to contribute your own images to this resource but without that you can still choose to consume other peoples containers (and add more layers to them to make them your own).

docker search term

Looks on the docker hub for containers matching your search term. For example Alpine Linux is a very simple and minimal Linux layout ideally suited for use inside containers. Core container images officially supported by Docker have names of just one word. Images contributed by third parties are in the form author/container. Puppet uses a very similar convention for its Forge too.

docker search -f is-official=true alpine
NAME DESCRIPTION STARS OFFICIAL AUTOMATED
alpine A minimal Docker image based on Alpine Lin... 1759 [OK]

The -f is-official=true filter we are using here limits the output to just the base container that Docker officially sanctions. Other matches mentioning ‘alpine’ may be (most probably) based on this but have the hard work of adding other software expressed as those nice union filesystem layers. An image is a collection of such layers – an image that has other images that are derived from it cannot be removed unless those images are also removed.

docker run

docker run -it alpine /bin/sh
Unable to find image 'alpine:latest' locally
latest: Pulling from library/alpine
0a8490d0dfd3: Pull complete 
Digest: sha256:dfbd4a3a8ebca874ebd2474f044a0b33600d4523d03b0df76e5c5986cb02d7e8
Status: Downloaded newer image for alpine:latest
/ #

What this is doing is running a chosen command (/bin/sh) within the image called ‘alpine’ using an interactive terminal. If the alpine image is int already downloaded, or the one on the hub is newer and we state we want ‘latest’ rather than a specific version then it is downloaded. If there are several union layers that make up the image they all get downloaded simultaneously.

We show this by now repeating the run command for one of the other alpine based images on the hub:

docker run -it mritd/alpine   /bin/sh
Unable to find image 'mritd/alpine:latest' locally
latest: Pulling from mritd/alpine
0a8490d0dfd3: Already exists 
f22eacae62c9: Pull complete 
Digest: sha256:98023aa60f8398432f5d62ccd16cea02279bb9efc109244d93d441dd09100e18
Status: Downloaded newer image for mritd/alpine:latest
/ # 

This shows clearly that ‘mritd/alpine’ is based on the alpine files we have already downloaded plus an extra layer of some customisation or other adaptation that contributor felt worth sharing (not important for the purpose of this discussion).

Chunks of storage called volumes  on the host can be added (mounted in) to the container with the -v argument to run. If a path in the container is mentioned on its own then the attached storage will be allocated by docker – but still exist outside of the union filesystem ‘layer cake’ of the image. If a colon separated pair of paths is used the host path is joined into the image as the destination path.

docker run -v `pwd`:/tmp/x -it alpine /bin/sh

Will run that super simple Alpine Linux image but if in the shell if you cd to /tmp/x you will find whatever was in the current directory on the host. If you would like the container to be able to look but not touch the data you are sharing add a :ro to the end.

-v `pwd`:/tmp/x:ro

Adding volume mounts means that the data lives outside of the life-cycle of that particular Docker image, and is not subject to size constraints of the original filesystem container – usually only a few GB. Volumes can also be created using the docker volume create command and can live on shared storage systems making them independent of any one host. See this documentation on flocker opening up to Docker ‘swarms’ operating over large numbers of hosts. This allows a software service to be rolled out in the modern fault tolerant “shared nothing” architecture without needing lots of dedicated physical machines, or even dedicated virtual machines.

Going in the other direction it is possible to narrow sharing down to an individual file level too:

docker run --rm -it -v ~/.bash_history:/root/.bash_history ubuntu /bin/bash

This example allows the shell within a ubuntu container to share your bash history and even add commands to it while running as root within that container. A handy trick while you are still experimenting with what need to go into that Dockerfile….

Lastly if a container has mounted volumes you can create a new container that mounts the same volumes – in effect shares the same data areas by using the –volumes-from flag to run. This together with the shared storage paradigm allows large groups of distributed services all sharing the same underlying data to be created.

docker image

This shows what images we have locally (contrast this with docker ps for running and suspended instances) . This consists of both images that have been downloaded from the Hub and those we have made ourselves by applying a Dockerfile to the base of an existing image.

docker rmi

Rmi stands for remove image. Eventually you could end up with lots and lots of images you no longer have any use for. The rmi command will remove any image that is not referenced by another. For the above case if we want to docker rmi alpine we would need to add a -f (force) flag because we have another image dependent on it:

docker rmi -f alpine
Untagged: alpine:latest
Untagged: alpine@sha256:dfbd4a3a8ebca874ebd2474f044a0b33600d4523d03b0df76e5c5986cb02d7e8
Deleted: sha256:88e169ea8f46ff0d0df784b1b254a15ecfaf045aee1856dca1ec242fdd231ddd

If  dependencies are more complex it is not possible to use the -f – some scripting will be needed to find all the dependant images and remove them first.

docker build

If you want to make custom changes to an image create a directory with a Dockerfile text file plus any custom content you wish to go into an image (e.g. content for a web server). You can give the resulting image any name you like that is not already in use by another image. With the Dockerfile and command line options you can control aspects such as what network ports are accessible within the image, how much memory and CPU time it is permitted to use, and what parts of the host filesystem are exposed to it. Volume mappings as explained above in docker run can alos be specified in the Dockerfile. When the built images are run with docker run it is common for ports in the image to be remapped so that they can be visible from the outside world. For example you can have several web services configured within the container to operate on port 80 but from the outside world accessible on ports 8081, 8082 etc. Whole sets of micro-servers can be orchestrated using Docker Compose in a similar manner to how Amazon Cloud Formation or OpenStack HEAT templates work.

docker login

If you have signed up for a hub account you can use those credentials to log into it and then docker push mages you have created with docker build to it. This allows containerised infrastructure to be developed and then stored centrally for deployment.

This is not an exhaustive set of what Docker can do, what I have tried to do is give a useful summary and pointers for further study.

 

Linux 20 years ago – RedHat 4.1 on the PCW magazine coverdisk

20 years ago virtually nobody had fast internet. If you were outside of Government, big Corporations or Academia your Internet experience was most likely just a slow dial-up modem. This was great for exploring the early WWW or sending emails but asking someone to download enough software to leave the confines of DOS/Windows and start to explore Linux was daunting.

As explained in my other article vital to the growth of Linux back then were the itself relatively new technology of computer magazines coming with first CD-ROMS and then DVDs attached to their cover. You still see cover DVDs on many magazines with with widespread high speed broadband it is more convenience that necessity nowadays.

Continue reading “Linux 20 years ago – RedHat 4.1 on the PCW magazine coverdisk”

A problem with solid state storage (and a suggested solution)

This article also appears at https://www.linkedin.com/pulse/problem-solid-state-storage-suggested-solution-martin-houston so if you are on linked-in you can choose to comment there instead..

There are several computers inside an Ocado Warehouse robot, each controlling some aspect of the operation. The single ARM CPU Linux based computer within each robot I was working on had two jobs, one to manage the in situ upgrading and if needed downgrading (change back-out) for the firmware on the other control computers. This was only occasionally used so the relatively meagre performance of the hardware was not an issue. The other role was considerably more demanding. That of collecting all the low level debug information logs from the rest of the robot and delivering them for analysis when uploaded every time the robot docked for battery charging. While working out on the warehouse grid the robots were controlled by WiFi signals but there was not enough bandwidth available to send back a fire-hose of detailed status & debug information in real time. While the robots were under active development the volume of debug data grew and grew and the only place to put it until it could be uploaded was the micro SD card inside the control computer board. This micro SD card had a maximum write speed of about 10MB/s. Having spent several months looking after hundreds of computers behaving in this way I have a new respect for the reliability and longevity of standard consumer grade microSD cards but nothing was going to change the fact that they were too slow for the job of handling a potentially limitless appetite for debug information from developers trying to nail illusive problems.

Writing the logs to memory was much, much faster than to SD card but the control computer had only been specified with 512MB of RAM, as it was never envisaged that such a large volume of data would need to be collected during robot development. I did some research and found that with the fallocate system call it is also possible to punch holes in a file at the beginning as well as the usual file truncation action of freeing blocks at the end. What you are left with is a ‘sparse’ file where sections that have nothing stored on disk read as full of nulls. I found that if you truncate the beginning of a log file the processes writing that file simply do not care. It is still possible to append and also to ‘tail’ the file to read back the most recent contents. The file can grow and grow to whatever the size limit is of the underlying file-system but only occupy a small amount of actual space. This discovery allowed me to collect logs into ram-disk instead of direct to the slow SD card. I used inotify system calls to watch a whole tree of log files being written alerting a process which collected all but the last few KB of each file and produced a compressed multiplexed version of all of them. The actions of compressing and multiplexing increased the effective write rate of the SD card enough to cope with much higher rates of logging activity, in effect kicking the can far enough down the road that the developers could have what logging they liked. The way SD cards work is that if 4MB of data is all written at once in a single file stream the write is much more efficient as that is the size of the erase block in the SD technology. I thought I had a prefect solution! However when fully stress tested I found it was missing one component, and one that would be non trivial to write.

I was able to run tests with simulated log data writing into the ram-disk that eventually overran the ability of the inotify driven background process to catch up. It took many minutes but slowly the ram-disk would fill up completely forcing the writing of log-file data to fail – implying missing, possibly vital, log information. What would be nice, I thought, was if the system calls writing data to a file-system that was in danger of getting full could be slowed in some way, just by a few microseconds, that would give the background process time to catch up.

An old mechanical hard disk would in a way do this. The furious search for free blocks would increase IO wait time so that writing blocks to a nearly full file-system would indeed take longer. However regressing back to mechanical disks is no solution as the forced head movements would also hamper processes that would be reading and consuming the data too!

What the Linux kernel needs is some way to simulate the slowing effect of writes that a nearly full file-system has on processes wanting to make the situation worse, and in a gradually increasing severity, but no corresponding penalty to readers (and removers) of data. I knew this would solve my immediate problem and then realised that it would have highly beneficial effects for data storage in the Enterprise too. Having file-systems and filers which exhibited this behaviour would get an early warning on file systems filling up, but most importantly a way to delay the inevitable full file-system crisis. With fancy enough monitoring it would be possible to isolate the issue to a single “way too chatty” application. The rate of log writing just for that process could be slowed so that the team responsible would have time to sort out what is going wrong. The fallocate trick I had found for dealing with the robot logs would also come in handy here – if a log has been discovered that has been growing for months or even years (a failure to implement log rotation), then a fallocate punched hole could be used to archive or just dispose off all data too old to be interesting without having to disrupt the running process at all.

Even if the rate for a single process had to be slowed to effectively a full stop it is definitely “the lesser of two evils” than allowing the file-system it is writing to to fill up. That would more likely have collateral damage to other well behaved parts of the infrastructure that were using that portion of storage space responsibly. The normal panic mode thing that system admins have to do in such a situation, on filers which have that luxury, is to give that file-system more (expensive) storage. This is a costly way to “fix” the problem and it does nothing to address the reasons why that file-system got full in the first place.

This was several months ago now and at the time I did a search so see if any such feature already exists in the kernel but drew a blank. As I had seen enough email circulars of how Ocado was keen on maximising their IP I put forward a proposal to do this seemingly missing part of Linux kernel capability as a project. My request was turned down (even though it was the missing piece needed to solve the log collecting issue with the robots). I was told I was welcome to do the work “in my own time”. Now I no longer work there here is my chance to ask people 1. does this technology already exist or 2. anyone fancy giving me a hand in writing it?

Puppet commands mini crib

Just a list of useful Puppet commands and key concepts

Not intended to really be a tutorial… If you find it useful I am glad, but I am learning Puppet myself and I always find the best way to really understand something is finding a really simple way to explain it. This is my attempt to distil Puppet down to the bare essentials that need to be grasped. It is a work in progress.

puppet help

Come on this one should be obvious!
The puppet help command on its own gives a list of all available sub commands. Including one of them after the word help gives detailed help on how to use it – use that to expand on what you learn from this crib sheet! The puppet man command gives a full manual page about the command – an expansion on the summary that puppet help gives.

puppet describe –list

This gives a list of all the different sorts of things that the puppet you have installed and configured knows how to control – i.e. that puppet code has been written to manage. This of course will vary with your Puppet set-up. If you are trying to apply a manifest containing resources your Puppet has no clue how to manage you are not going to get very far are you? You may want to pipe this through your favourite pager as the list may be long!

puppet describe <something from the list>

Gives the equivalent  of a “manual page” for that particular “thing” – what options are available for controlling it. This is a definition of what should be put in the manifest for that sort of resource. For example here are the options for “mount”

mount
=====
Manages mounted filesystems, including putting mount
information into the mount table. The actual behavior depends
on the value of the 'ensure' parameter.
**Refresh:** `mount` resources can respond to refresh events (via
`notify`, `subscribe`, or the `~>` arrow). If a `mount` receives an event
from another resource **and** its `ensure` attribute is set to `mounted`,
Puppet will try to unmount then remount that filesystem.
**Autorequires:** If Puppet is managing any parents of a mount resource ---
that is, other mount points higher up in the filesystem --- the child
mount will autorequire them.


Parameters
----------

- **atboot**
    Whether to mount the mount at boot.  Not all platforms
support this.

- **blockdevice**
    The device to fsck.  This is property is only valid
    on Solaris, and in most cases will default to the correct
value.

- **device**
    The device providing the mount.  This can be whatever
    device is supporting by the mount, including network
    devices or devices specified by UUID rather than device
    path, depending on the operating system.

- **dump**
    Whether to dump the mount.  Not all platform support this.
    Valid values are `1` or `0` (or `2` on FreeBSD). Default is `0`.
    Values can match `/(0|1)/`.

- **ensure**
    Control what to do with this mount. Set this attribute to
    `unmounted` to make sure the filesystem is in the filesystem table
    but not mounted (if the filesystem is currently mounted, it will be
    unmounted).  Set it to `absent` to unmount (if necessary) and remove
    the filesystem from the fstab.  Set to `mounted` to add it to the
    fstab and mount it. Set to `present` to add to fstab but not change
    mount/unmount status.
    Valid values are `defined` (also called `present`), `unmounted`,
    `absent`, `mounted`. 

- **fstype**
    The mount type.  Valid values depend on the
    operating system.  This is a required option.

- **name**
    The mount path for the mount.

- **options**
    Mount options for the mounts, comma-separated as they would appear
    in the fstab on Linux. AIX options other than dev, nodename, or vfs may
    be defined here. If specified, AIX options of account, boot, check,
    free,
    mount, size, type, vol, log, and quota must be alphabetically sorted at
    the end of the list.

- **pass**
    The pass in which the mount is checked.

- **remounts**
    Whether the mount can be remounted  `mount -o remount`.  If
    this is false, then the filesystem will be unmounted and remounted
    manually, which is prone to failure.
Valid values are `true`, `false`. 

- **target**
    The file in which to store the mount table.  Only used by
    those providers that write to disk.

Providers
---------
    parsed

Note that the puppet man command gives a very similar detailed output for puppet commands themselves.

puppet resource

This is a way to “hand crank” Puppet – to get Puppet to set the options for a resource the same way as if a manifest was to be applied. As an extra bonus option you get emitted a manifest entry that would result in the same puppet resource (Note as we are doing something to change the system  sudo is needed):

sudo puppet resource user bogus ensure=present

Notice: /User[bogus]/ensure: created
user { 'bogus':
  ensure => 'present',
}

And we can see that the user has indeed been created:

grep bogus /etc/passwd
bogus:x:1001:1001::/home/bogus:/bin/bash

And remove it again:

sudo puppet resource user bogus ensure=absent
Notice: /User[bogus]/ensure: removed
user { 'bogus':
  ensure => 'absent',
}

And the user is no longer in the /etc/passwd file.

Note if you just want the emitting manifest syntax side effect use the –noop flag  at the end of the command – so that actions are not actually done. The resource command still has to be one puppet would be able to do (correct parameters for the resource type etc).

Summary is that puppet resource is an opportunity to get puppet to do individual actions. This is both a handy way to do low level code testing and gives syntactically correct code to paste into manifests for automated repetition. Puppet manifest syntax (Puppet DSL) takes a bit of getting used to as it is rather richer in meaning than it looks at first glance to be.

Without the option=value parts puppet resource can be used to query the current state of managed resources, again emitting the result as syntax suitable for a manifest file (Note sudo still required – this data may well be sensitive!):

 sudo puppet resource user root

user { 'root':
  ensure           => 'present',
  comment          => 'root',
  gid              => '0',
  home             => '/root',
  password         => '$6$Fzx4UD.89YeLcYry$ph.N6w1t.dSzmn/dycQ0sGTGL/gGsgI8JTo94nwqapffTbruqDhrSgKE.G132RmJ.q03lT.MEMLDahG7XH.',
  password_max_age => '99999',
  password_min_age => '0',
  shell            => '/bin/bash',
  uid              => '0',
}

All the above options could be supplied to recreate a root account as it exists. Many of them however are defaults – such as our ‘bogus’ user got a bash shell without having to explicitly specify it.

puppet apply

The blocks of Puppet DSL emitted by puppet resource, or new creations with the same syntax, can serve as input to the next stage up the food chain – puppet apply

Using the same example as before – create a bogus user:

sudo puppet apply --execute "user { 'bogus': ensure => 'present',}"
Notice: Compiled catalog for asrock1.deluxetech.co.uk in environment production in 0.44 seconds
Notice: /Stage[main]/Main/User[bogus]/ensure: created
Notice: Finished catalog run in 1.82 seconds

Note that although the end effect is the same, the output messages from puppet apply are rather different and more like what you get to see on a ‘real’ puppet run. Note -e can be used as shorthand for –execute.

Using nothing more than puppet apply it is possible to develop new pieces of Puppet DSL and test them in situ before tested manifest files get installed on a puppet master.

If instead of being squashed onto 1 line for use with –execute, that DSL for creating the bogus user was simply saved in a file called bogus.pp (Puppet manifest files by convention end in .pp) then

puppet apply bogus.pp

would have the same effect of creating the user.

Now we have this action captured in a file it becomes easy to start adding parameters, like giving the user a password and other options appropriate to the user resource. Remember “puppet describe user” will give us full details on what options are possible for this resource.

puppet config print

This will print out the current config that puppet is using on this machine as a long list of name=value pairs. Changes to the config can be achieved with puppet config set name value changes this way end up in /etc/puppet/puppet.conf for the Open Source version and /etc/puppetlabs/puppet/puppet.conf for the Puppetlabs Enterprise version. These files can also be edited by hand for more complex config (with comments). A particular point of limitation with using puppet config set is that the variables only end up in the [main] section so could be overridden if thee has been a set variable in a more specific section for [agent].

sudo puppet config print confdir will show where the configuration data for your puppet is set (/etc/puppet for root in the Open Source version). Note that if you run this without the sudo you will get the answer $HOME/.puppet – this means that  you can work with and tests some aspects of Puppet without needing root, however it is a trap for the unwary as configuration will differ depending on if root is used or not!

Enabling a development environment

If $confdir is /etc/puppet then we need to set up some environments under there where Puppet will expect them to be.

sudo mkdir -p /etc/puppet/environments/development

sudo puppet config set environment development

This is saying that the ‘bootstrap’ or ‘root’ for Puppet configuration to be applied to this machine while it is in ‘development’ status starts with a site.pp file located under the manifests directory here. This structure becomes particularly important when a machine becomes  a puppet master, storing the configuration for many different machines that are configured to take their setup from different states. Some machines may be in ‘development’ state, others in ‘preprod’ or ‘production’. In big Puppet deployments a version control system like git and a formal change control procedure are used to make sure that Puppet code does not reach critical production machines unless it has been properly tested with less critical ones.

puppet module generate

In order to tailor puppet to our system in non trivial ways we will need to generate one or more modules:

puppet module generate yournamemodname –skip-interview

This generates the whole expected boilerplate structure for a module in the current directory under a directory called yourname-modname. Your first action should be to rename that directory to just modname as we are just using self created modules to describe the setup of this site at a high level. The yourname bit is just to give some namespace separation if you later decide to share your modules, or are making modules to be shared, so in this case is simply not needed, the top level config is really going to be what is unique about your site.

The two really critical files here are manifests/init.pp and tests/init.pp. The former defines the meat of the module, you will probably want several .pp files but init.pp is the entry point (in the same way that site.pp is the entry point for an entire puppet run) and the latter exercises it for test purposes. Modules have to be invoked from somewhere in order to do anything – the tests section gives an opportunity for unit testing of the Puppet code being written.

Here is a simple example of this:

Go to the directory you have defined as development (e.g. etc/puppet/environments/development)

Create modules directory and change to it.

puppet module generate nevermind-donothing –skip-interview

mv nevermind-donothing donothing

Now edit donothing/init.pp and add this to the empty donothing class:

notify { ‘Applying class donothing’:
}

puppet apply donothing/tests/init.pp # note we have not edited this – its default behaviour of invoking the class is enough for this.
Notice: Compiled catalog for asrock1.deluxetech.co.uk in environment development in 0.11 seconds
Notice: Applying class donothing
Notice: /Stage[main]/Donothing/Notify[Applying class donothing]/message: defined ‘message’ as ‘Applying class donothing’
Notice: Finished catalog run in 1.76 seconds

Because we are working in the place puppet expects things to be we do not need a –modulepath=./ that would be required if you want to develop module code in arbitrary other places

puppet module search & install

The other sort of modules are ones that are installed from the puppet forge. These are just tared up archive trees the same as created originally by module generate.  One of the things that makes Puppet so powerful is that when sysadmins solve a problem about how to orchestrate something with Puppet the most often contribute the code back for others to use and improve. This is the whole essence of why the Open Source idea is so powerful.

puppet module search for a named module confirms that it exists and offers possible alternatives.

puppet module search puppetlabs-apache
Notice: Searching https://forgeapi.puppetlabs.com ...
NAME DESCRIPTION AUTHOR KEYWORDS 
puppetlabs-apache Installs, configur... @puppetlabs web ssl 
dploeger-pagespeed Installs the Apach... @dploeger 
mayflower-php Generic PHP module... @mayflower php fpm 
maestrodev-rvm A puppet module fo... @maestrodev ruby rvm 
jgazeley-django Deploy a Django ap... @jgazeley 
hetzner-roundcube Roundcube webmail ... @hetzner webmail 
puppet-puppetboard Install and config... @puppet puppetdb 
saw-reviewboard Install and contro... @saw trac 
jhoblitt-awstats Manages the AWStat... @jhoblitt 
nibalizer-puppetboard Install and config... @nibalizer redhat 
spotify-puppetexplorer Manage the Puppet ... @spotify puppetdb 
pulp-pulp This module can be... @pulp 
pltraining-dockeragent Manages Puppet Lab... @pltraining 
landcareresearch-amazon_s3 Manages mounting S... @landcareresearch 
thejandroman-grafana Installs and confi... @thejandroman 
thejandroman-kibana3 Installs and confi... @thejandroman kibana 
gajdaw-symfony Puppet module to i... @gajdaw php app 
jgazeley-mod_auth_cas Configure mod_auth... @jgazeley httpd cas 
42ways-railsapp Basic Rails Server... @42ways rails ruby
jgazeley-speedtest Install Ookla Spee... @jgazeley

puppet module install actually installs it e.g.

puppet module install puppetlabs-apache
Notice: Preparing to install into /etc/puppet/environments/development/modules ...
Notice: Downloading from https://forgeapi.puppetlabs.com ...
Notice: Installing -- do not interrupt ...
/etc/puppet/environments/development/modules
└─┬ puppetlabs-apache (v1.11.0)
  ├── puppetlabs-concat (v2.2.0)
  └── puppetlabs-stdlib (v4.14.0)

Note as with creating your own modules a –modulepath= option will allow you to install modules somewhere other than the standard place Puppet is going to be looking (under the development branch in our case). Note that there is dependency following going on here, just like with Linux packages. As well as the puppetlabs-apache we wanted we also get two modules it in turn needs.

These modules also get installed under

/etc/puppet/environments/development/modules

so you can see the rest of how modules need to be constructed by looking at other people’s examples. Lower level modules will tend to have actual Ruby code as well as Puppet DSL declarations.

Fun with facts

The facter command without parameters will list all the custom facts available to Puppet on this system. If names of facts are listed on the command line only those as name => value pairs will be retuned. If an unknown fact is asked for like ‘wibble’ it gets mapped to nil.

facter is_virtual operatingsystem wibble
is_virtual => true
operatingsystem => CentOS
wibble => nil

It is possible to define custom facts so that anything that Puppet needs to be able to make decisions about configuration of systems.

There are 2 places facts are used within the Puppet DSL. Firstly within Templates – the .erb files. A fact can be embedded using this syntax:

 <%= @operatingsystem %>

Actually facts are not really local variables (this same syntax would be used for showing those in a template too) so a little bit more correct (and if you had the misfortune to have a local variable name conflicting with a fact:

<%= scope['::operatingsystem'] %>

Within Puppet DSL in the .pp files a $:: prefix is used (normal local variables just have a $ prefix within .pp files.

$OS = $::operatingsystem

Classes and defined types

We have already seen defined types as built into puppet as e.g. the user.

 

You can define your own types however with default values for any parameters that can then be overridden. Remember that donothing class we created earlier? Extend it like this:

class donothing {
 notify { 'Applying class donothing':
 }
 bogus { 'suspect':
   flavour => 'suspect'
 }

 bogus { 'plain': }

}
define bogus ($flavour='stinky') {
 notify { "Seen a $flavour bogus type": }
}

puppet apply -e "include donothing" 
Notice: Compiled catalog for asrock1.deluxetech.co.uk in environment development in 0.08 seconds
Notice: Seen a stinky bogus type
Notice: /Stage[main]/Donothing/Bogus[plain]/Notify[Seen a stinky bogus type]/message: defined 'message' as 'Seen a stinky bogus type'
Notice: Seen a suspect bogus type
Notice: /Stage[main]/Donothing/Bogus[suspect]/Notify[Seen a suspect bogus type]/message: defined 'message' as 'Seen a suspect bogus type'
Notice: Applying class donothing
Notice: /Stage[main]/Donothing/Notify[Applying class donothing]/message: defined 'message' as 'Applying class donothing'
Notice: Finished catalog run in 1.82 seconds

Note a class does not have this ability – they are intended to be top level containers – a file with the same name as the class and ending in .pp contains the class.

To include such a file (found on the search path) within another class we just use include – like

include apache

or as we did above:

puppet apply -e "include donothing"

This assumes that the donothing class can be found on your search path.
Add –search-path=./ if need to set it for explicit test.

Note that if a defined type is specific to a particular module the convention is to prefix it with the module name and double colon e.g. apache::vhost

Classes can also have parameters – which allows some aspects of how that class is used to be changed when the class is invoked. The syntax is just a little different to defined types:

In a file withparams.pp:
class withparams($param_one='this',$param_two='other') {
# body does not matter...
}

# and on the command line
puppet apply -e  "class { 'withparams': param_one => 'that' }"

This will result in the ‘withparams’ class getting used but with the $param_one variable set to ‘that’ and $param_two with the default value of ‘other’ Note that we are being absolutely explicit here that a class is being called with non standard arguments. Calling a class without overriding any of the arguments is done with just ‘include’ as we have seen before. Remember puppet is NOT procedural. A class calling another class is just establishing a relationship of things that have to be done. The order they are done in needs to be separately controlled if indeed it is important. Remember apart from init.pp which is the special top level the class name must match the file name it is in as this is how classes are found when other classes request to include them.

puppet apply -e “class { ‘withparams’:}”

is exactly the same as

puppet apply -e “include withparams”

require/notify and the arrow notation -> ~>

Stating in the Puppet DSL that for specifying that other things have to be in place before the service will run (like a configuration file for example) or another resource can be created – this is done with the  require keyword.

This code:

file {'myconffile':
# commands to populate the file e.g. from a template
}
service {'myservice':
  require File['myconffile']
  # other stuff
}

Could instead be written as 
file {'myconffile':
# commands to populate the file e.g. from a template
} ->
service {'myservice':
}


 

There is an equivalent when you have the file but want to say what service is affected by it which is called  notify. The other way to state this is with a ~> arrow – with the tilde character. Having -> and ~> means that you can add the appropriate require or notify without having to change the DSL source of the other component, which is handy as it might be a contributed module written by someone else.

Note that the -> or ~> is used as a joiner between the two text blocks. For require in particular long chains can be implied that first thing needs to be done first then second then third.. e.g create a directory, then create a sub directory then create some files in it. Using explicit requires in this case would look messy.

Puppet master

Although puppet code can be developed and run on any host directly a complex system where servers interrelate to each others requires one machine to be set up as a puppet master. If the puppet master is down all that is lost is the ability for changes to be made to a deployed configuration. Therefor there is no special need for the puppet master to be designed as fault tolerant. Even production networks can tolerate short outages of the puppet master. What is VITAL however is that the puppet master is secure as gaining root on it is potentially compromising as root access to any of the machines being controlled by that master. This is because the master machine controls exactly what happens to the agent machines by deploying puppet code to them. This is why it is called Puppet.

Systems to be controlled register with the puppet master by using puppet agent mode. Communication is authenticated using SSL so each agent needs to issue a SSL certificate and have it signed by the master. Signing of certificates by the master can be automated but if security is important is best left as manual as new systems also have to have decisions made to them about which puppet environment (development or production) is controlling them and what role the have within that environment. This determines how that machine is to be configured by the puppet master e.g. as a web server.

For trivial testing and small deployments there is a simple, single threaded web server built directly into puppet so that the puppet master can be easily started and stopped and debugged etc.

puppet master --debug --verbose --no-daemonize --logdest console

This will run in the foreground and emit verbose output direct to the console (have a nice big scroll-able terminal to run it in). However for larger deployments using the puppet master in conjunction with a proper multi-threaded web server like Apache or there is a puppetserver package also available which comes from puppetlabs and is a puppetserver written in jruby (therefore running under the java jvm) for performance. This will usually be available as a package called ‘puppetserver’.

“bootstrapping” machines to use a puppet master

As stated earlier it is easy just to use puppet locally and invoke specific pieces of the Puppet DSM by hand using puppet apply to test them. However the whole point of puppet is to enable the control of complex setups that span over dozens to thousands of machines. In order for this to happen there needs to be a central place where all the machine configurations are decided and actions issued to get machine configurations into line with what they should be.

In order for a machine to be joined into the Puppet infrastructure these things need to happen.

  1. Enough of puppet code needs to be present on the machine to allow the puppet agent function to be performed. This would be the responsibility of initial system build from a technology like kickstart or cloning a basic system image when a new VM is created.
  2. That machine needs to know where to find the puppetmaster – this will either be an alias in whatever DNS domain the newly created machine has been given (possibly handed out by a DHCP server) or, more simply an entry in the local hosts file put there by whatever initial build process is in place.
  3. Command puppet config set certname with a FQDN of a name that you want this node to be know to the puppet master as. This name is vital as it must correspond to an entry in the manifests/site.pp file on the puppetmaster.
  4. Command puppet agent -t run to present this certificate to the puppet master for signing.
  5. Unless the puppet master has been set to auto-sign all certificate requests (unwise) the sysadm now must take action to sign the certificate.
  6. A further puppet agent command will start regular background communication with the master. Note that the master does not  itself need 100% availability. Loss of puppet connectifity only affects ability to change configuration, not actual running of the machine itself.
  7. Now communication is established it is possible to use the node classifier on the master to assign this node into whatever group is desired. The site.pp within that group then acts as a starting point for the Puppet configuration and therefore the role that machine will then take.

 

Xtra-PC in depth

The review I did of the Xtra-PC I did a few weeks ago is by far the most read (and commented on) piece on my blog. So I thought it is time for a follow on. This includes instructions on how to clone your XtraPC onto another USB stick or onto the internal hard disk in your PC (if that is still working) – so if you want to know how do do that (a commonly asked question on the original review), read on!

Continue reading “Xtra-PC in depth”

How to make your computer faster

My that is a super vague title isn’t it?

Everyone would like a faster computer would’t they? For some things, such as live broadcasting, a computer that is not quite fast enough can be next to useless, nobody enjoys watching badly lagging and stuttering video and audio. However a screamingly fast computer is not all advantage. They consume more power so over their lifetime cost more to run and tend to be less reliable being “bleeding edge”.  Also they are harder to keep cool so tend to generate more noise while in use. The very fastest chips that gaming enthusiasts use require water cooling to get the very most out of them, a hobby known as “Overclocking”. This eliminates some of the noise but can bring its own set of problems associated with the possibility of leakage as systems get older and get worn. Water and electricity do not mix very well.

Continue reading “How to make your computer faster”

Urgent action needed on the Snoopers Charter

Last week the UK got potentially the most draconian surveillance powers in the world. Only Royal Assent is now needed before the Investigatory Powers Act becomes law.

In a dangerous world, the only way we can protect our civil liberties is by sacrificing them
Thanks to Martin Shovel https://twitter.com/MartinShovel for permission to use this

Teresa May has tried several times to get her “Snoopers Charter” past, each time beaten back by huge public pressure. Now with everything else going on they have managed to sneak it into law.

What can you do?

Firstly there is this petition which has nearly 150,000 signatures and rising fast calling for the repeal of this ridiculous law.

Secondly everyone who uses the Internet can start taking measures to protect your privacy e.g. with a VPN to outside of the UK, or alternatively swamp the system with random web visits etc. making it impossible for them to collect the volume of data if millions of us do it. Maybe this can be a new area of software development? Software that sits on your computer and visit lots and lots of random sites full of rubbish, so you don’t have to.

This strategy was mentioned in this UK COLUMN broadcast

Update for 1/12/16 – the inventor of the Web Sir Tim Berners-Lee has called this measure “A Security Nightmare” and The Register has weighed in too.

This is an email I sent my MP Alistair Burt back in March while this was still just under debate:

I am one of your constituents and I was also one of the organisers of the protest against ACTA back in 2012.

That vile treaty was defeated by massive coordinated action right across Europe.
Do you REALLY want to cement in place the infrastructure of a tyrannical police state Mr Burt? To be used by whoever manages to seize power? Kim Jong Un will be so envious!
Because this is exactly what putting into law the dirty tricks that the out of control and unaccountable spying agencies have already been using.
If you allow them these powers what do you think they will try to get away with next, under the cover of the gargantuan data flows coming to them?
The fuss between Apple and the FBI at the moment also illustrates the colossal commercial damage this will do to our Internet industry – people will not be able to trust any products that UK companies have any hand in for fear of back doors. Once a back door exists other dark actors will soon find ways of opening them – it is a path of true insanity so do not tread down it.
Needless to say that fell on deaf ears and we now have this very sorry state of affairs. Sign the petition, write to your MP and tell them what a grave mistake you think they have made. Give your full support to those within the Internet industry that refuse to cooperate with this attempt at tyranny.

This just out from swilliamism:

And here, if you need reminding, is the sort of turnout we had against ACTA in 2012

Needs to happen again and again.

Data inflation

Some 25 years ago now I did a spell of work for BG Transco. These are the gas pipeline equivalent of the National Grid – they are responsible for delivering gas supplies form sources such as North Sea and LPG tankers coming from far away to every home and business in the UK that is connected to the gas main. As you can imagine with all that pipeline and all those customers to look after the used a lot of computers. The main bulk of the file storage was on a massive EMC Storage Area Network – or SAN. Physically this was very impressive, jet black cabinet after cabinet with lots and lots of flashing lights. Disk drives were much smaller back then. 1GB was considered very large! I once asked how much useable storage this giant split site EMC storage array had and was told “About a Terabyte”. A Terabyte back then would be 1,000 1GB sized disk drives or more if the drives were not quite that large. This was sufficient capacity to look after all the gas pipes, all the work being do to maintain them and the millions of customers. Let that sink in.

About a 8 years ago I was working for one of the big London Hedge Funds. They had a Hitachi SAN system that supplied disk space to computers over Fibre Channel. This arrangement allowed computers to be configured to see as much or as little disk space as they needed to do their job. Being a SAN the storage was redundant and backups were all taken care of by the team that ran the storage system. There was an internal price set to the business for this disk space of £20,000 per Terabyte. Clearly this was a lot cheaper than a Terabyte of storage had cost just over a decade area when it would have required millions of pounds worth of hardware to provide! However I remember at the time there was some grumbling from internal customers as 1TB disks for fitting inside PCs were getting available and affordable – more like £100 than £20,000!

This sudden explosion of ultra cheap storage has been seen by many as a good thing, it opens up the world of “Big Data” where the sort of incidental data such as the browsing habits of millions of computer users can be analysed for things that might give “business advantage”.

Call me pessimistic if you like but I for one do not share this boundless enthusiasm for ever increasing amounts of storage. In this article I am setting out some of the reasons I think it is not a very good idea. Fee free to debate & disagree with me in the comments.

  1. Disks have got bigger, fast but have not got that much faster. Yes there is the very worthwhile speed boost you get for going to SSD storage, especially for random access storage needed for things like databases. However the more you store without clever indexing the longer it is going to take to find it by brute force searching. In fact past a certain point brute force searching becomes unfeasible. If you have no index entry to data then that data is in effect lost.
  2. The apparent low cost of storage has made individuals and companies deeply complacent about “Data Hygiene”. Many times in my career as a System Administrator I have come across situations where large amounts of data that are causing a problem in some way (such as tens of thousands of files in a single directory) but no owner and nobody willing to take the responsibility to delete can be found! There was a program on British TV a few years ago called “How Clean is Your House” where people who have let their housework lapse (often to a truly disgusting extent) get a deep clean blitz and gentle advice not to let things slip back into the same state. The insides of many peoples computer storage systems equally horrifying (well maybe without the maggots and rats). Storage being cheap means the path of least resistance to running out is just to “buy some more”, not to put real work into finding which data is actually useful and which is just dead weight.
  3. Computer and storage vendors have no incentive to help users out of their “storage addiction” – in fact the incentives are all the other way with literally millions to be made out of Petabyte sized Enterprise grade storage systems. This is a problem that the users have to recognise that they themselves have, and take steps themselves to deal with it.
  4. Not really knowing which of the data you are keeping is important now, was important once but not anymore, or was never important is particularly acute when it comes to Disaster Recovery. If disaster strikes and the data has to be loaded into new computers from some off site silo would you know which to load first? I would bet that creeping data growth has made the DR plans of most organisations inoperable by now. Volume is a big issue in that the rate at which data can be reloaded is finite. If it has to come from remote storage across the internet it is limited to the speed of your incoming Internet connection.

Do you have an over indulgence in data problem? Need some help analysing the nature of the problem and taking steps to “slim down”? Give me a call.

Here is a little example of the sort of steps that can be taken. A few years ago I did some work for an insurance company that had several Terabytes of archived correspondence to customers. These were all stored as PDF files. Some savings were made at the expense of instant access by archiving up related PDFs into zip files but still the sheer volume of documents was formidable. I did some analysis and found that the documents were in fact just the result of several dozen different standard letters. Also around half of the space in each letter was taken up with a single element – that was the scanned signature one of a dozen or so people who were the senders of the various form letters.

Converting the PDF form letters into SVG allowed the sections of boilerplate that always remained the same to be changed to include file directives, with by far the greatest saving from including the correct signature block. Rendering the PDF back from the SVG equivalent was trivial to do if the PDF (and a printout of it) was ever needed. The important thing about this approach is that it only used about 5% of the space that the original archive had done. This has practical implications far beyond just the saving of a few pounds of disk storage. 5% of the original size means that in a DR situation that data would be available twenty times faster. This is why Data Hygiene needs to be treated much more seriously.

 

Selling Unix in the 1990s

Back in the days before Linux was invented there was already a much better choice compared to DOS and the primitive versions of Windows available then. Back in 1989 I started out in business as a re-seller for SCO Unix, having worked side by side with SCO porting their Unix onto computers at Apricot. I have scanned some of the original documents back then – it is interesting how far things have come, but how some concerns have stayed the same.

houston_technology_1989 This is a pdf of a scan of the original press release back in 1989. It it looks a bit quirky it is because it was done on a daisy wheel printer. That was the technology at the time for a smart looking letter. It was slow, it was noisy, but produced something far better than the other technology at the time, the dot matrix printer. Laser printers were just about there but were themselves rather primitive and prone to go wrong. So they only tended to be used by larger companies and specialised copy shops.

Also from 1989 an early ‘flyer’ that I produced – this time on a dot matrix printer as can be seen form the lower quality of the typeface (but more flexability such as simple graphics) houston_technology_1989_infosheet

The Free Software Foundation itself was in its infancy in 1990, the two main products at the time were the GCC compiler and the Emacs editor. I managed to obtain source code for these and cross compiled them using my SCO Unix system. I found that even then the compiler was much better than the Unix standard one. Here is an article that I had published about my findings at the time: houston_technology_free_software_foundation1990

And here is a collection of advertising in the Chamber of Commerce directory, photography of the machines at the time and a marketing flyer: houston_technology_1990

Did the small business people of the Midlands flock to buy Unix systems? Sadly no. As the Unix software itself was not free complete systems with enough memory and a couple of terminals added up to  a lot more than customers were willing to part with.

The advent of Linux where there is no longer a charge for the software has changed the dynamic of this though. Also the computer power you can get for your £ has increased beyond all comparison. Back then most computer software ran in text mode, sometimes with simple graphics (like Lotus 123).

I thought people as old as me would like to remember what the computer industry used to be like 🙂 Not the “good old days” by any means  but it is important that what was already got right by then should not be forgotten, thrown out just because computers are now thousands of times faster.

Persuading a Windows 10 machine to take to Linux

I had a customer yesterday who wanted a Linux install on his laptop.

The laptop had tricked him into an unwanted upgrade from Windows 8 to Windows 10. As it was a machine with only 2 cores and 4GB of memory he was not happy with the performance of Windows 10 and wanted to move over to using the machine purely for Linux.

I already had an external SSD set up with Ubuntu 16.10 “Yackety Kak” release so I could demonstrate what we would be getting before the install. The computer had other ideas! Every time it was restarted it eventually resulted in the Windows 10 sign in screen. There was no prompt during boot to press a key to select anything else.

After about 4 times round this loop, including taking the battery out in case the ‘shutdown’ was really just a sleep it was time to get drastic. Luckily I had my small bladed screwdriver with me so I was able to remove in internal SSD in the laptop.

Now with nowhere to boot to the machine was finally forced to show me a BIOS configuration screen. Humanity 1 Machines 0!

The problem was the default boot option called something like “Windows boot mode”. I did not take photos or notes, but that is probably just as well as the details may differ from your brand of PC anyway.

The settings I changed was to disable “Windows boot mode” and changed from UEFI to ‘Legacy’ boot mode. I also explicitly turned on the option to boot from USB which was also off by default. Note that some Linux distributions can operate with UEFI boot mode but it means that only kernels that are signed by a party that the computer maker trusts are permitted to boot. This may be good for security but is also a huge curtailment of the freedom of owning a general purpose computer (as opposed to a games console or some other piece of consumer electronics).

These changes were enough to persuade the laptop to boot to my external SSD! Impressively fast too, compared to what Windows 10 with who knows what bloatware was managing. As an aside here: if your computer is new enough to have USB3 ports on the main-board you will most likely be able to boot from them. USB3 is only slightly slower than the internal SATA3 disk interface so if you want the convenience of being able to take your hard disk away and lock it in a safe overnight this is a good way to achieve that.

The next stage, once the direction to boot Windows right away has been removed,  was to replace the disk and boot the computer again from a Ubuntu Linux installation USB key – this is just the DVD image copied into an empty USB stick. This allowed the install to happen in the normal way and 20 minutes or so later the customer was happy playing with Linux on his previously stuck with Windows laptop.

So the moral of this is: If a computer BIOS has been set so that it seems impossible to stop it booting to Windows, try taking the disk where windows resides out, at least temporarily.