Configure Chef to Install Packages from a Custom Repo

Going Nuts

This was driving me completely nuts last week. I could write Chef recipes to install packages from standard repos but I could not get Chef set up so that the recipe would add a new repo and then install packages using the Chef package configuration.

Truth be told I could do this in a really crude way. I could add a repo by writing a file on the node, and then run a command to install a package directly. It just wouldn't be managed by Chef.

The wrong way

I first created a new cookbook called myinstaller. Then in the templates directory I created a file called custom.repo.erb (The Chef template format) with the following contents:


[custom]
name=MyPackages
baseurl=http://chef.vb:8088/yum/Redhat/6/x86_64
enabled=1
gpgcheck=0

Note that the baseurl parameter is pointing to a yum repository I have created on one of the virtual machines running on my virtual network.

I then edited the recipe file recipes/default.rb
and added the following:


template "custom.repo" do
  path "/etc/yum.repos.d/custom.repo"
  source "custom.repo.erb"
end

execute "installjdk" do

    command "yum -y --disablerepo='*' --enablerepo='bmchef' install jdk.x86_64"

end

This works. But it is crude because I am not using Chef to manage the packages that are installed.

The right way

When you search around into this topic you will come across pages like this one that talk about adding packages with the yum_package command. However if you change the install section above to use this command it will not work. It seems to be related to the fact that simply adding a file to the yum repos directory on the node (while recognized on the machine itself) is not recognized by Chef.

I dug deeper and tried many different versions of adding that repo configuration and I eventually started finding references to the command 'yum_repository'. However, if you try to whack this command into your recipe it doesn't bloody work. It turns out that this is because it is not a command that is built into Chef (unlike 'package' and 'yum_package') it is in fact a command that comes from this open source cookbook for installing yum packages.

If you do not want to use this entire cookbook the critical files to grab are as follows

yum/resources/repository.rb

yum/providers/repository.rb

yum/templates/default/*

(I took the three files from this last directory, which may not be strictly necessary).

Now before you can use the command there are a couple of gotchas. 

1) If you copy all of this to a new cookbook called myyum, then the repository command will now be 'myyum_repository'

2) You will need to edit the file yum/providers/repository.rb 

 go to the bottom where the repo config is being written and change the line:

cookbook "yum"

So that the name of your cookbook appears there instead.

You will now be able to add a repository by putting the following in a recipe

myyum_repository "custom" do

  description "MyRepo"

  url "http://chef.vb:8088/yum/Redhat/6/x86_64"

  repo_name "custom"

  action :add

end

yum_package "mypackage" do

  action :install

  flush_cache [:before]

end

Just upload your new cookbook: 

sudo knife cookbook upload myyum

Add the recipe to your node: 

knife node run_list add node2.vb 'recipe[myyum::default]'

And execute: 

knife ssh name:node2.vb -x <USER> -P <PASSWORD> "sudo chef-client"

Amazing

Configuration Management with Chef

Configuration Management with Chef

Have you ever been through a long tedious process of setting up a server, messing with configuration files all over the machine trying to get some poorly documented piece of software working. Finally it works, but you have no idea what you did. This is a constant frustration for loads of people. Configuration Management (maybe) the answer.

Stated simply, you write scripts that will configure the server. Need to change something, you modify the script and and rerun. Branch the script and use it in version control so that you keep a track of multiple experiments. All of the advantages of managed code are brought over to managing a server.

We are currently investigating using Chef, which as brilliant as it appears to be, is sorely lacking in straightforward, complete and accurate tutorials. What I need with every new tool  I use is a bare bones get up and running walk-through. I don't need to see a highly branched and complete set of instructions designed to tutor people who already know what they are doing. This blog post is my attempt at a bare bones attack.

So here we go.

Configuration

In this walk-through we are creating an entire networked Chef system using virtual machines. To do this we need to set up a local DNS server that will map names to IP addresses on the local virtual network. <<THIS PART IS ASSUMED>>

DNS server config

Once you have the DNS server set up you need to make a few modifications.
Set it so that it will forward unknown names to your Gateway DNS server
Change the named configuration to forward to <<Gateway DNS>>

Add entries for all components of the Chef networks. The following are assumed to exist

dns.vb (DNS server) 192.168.56.200

chef.vb (Server) 192.168.56.199

node.vb (Node) 192.168.56.101

Host Configuration

Configure your host machine

1) Add dns.vb to /etc/hosts
2) Disable wireless (or other connections)
3) Fix the DNS server in your wired connection
       - In the IPV4 setting tab add the IP address of dns.vb

Virtual Machine Config

Once this is done you will need to configure each and every machine that is added to the system (Chef Server and Nodes )

1) Give the machine a name
A) Edit /etc/sysconfig/network file and change the HOSTNAME: field
B) Run the command hostname <myhostname>

2) Give the machine a static IP address and set the DNS server

vi /etc/sysconfig/network-scripts/ifcfg-eth1
BOOTPROTO=none
IPADDR=<<IP ADDRESS>>
NETWORK=255.255.255.0
DNS1=192.168.56.200


3) Add the machine's IP address and hostname combination to the DNS server
A) Edit the file  /etc/named/db.vb and add a line at the bottom for each hostname IP combination
B) Restart  the DNS server :  service named restart

4) Prevent the eth0 connectuon from setting the DNS
vi /etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO=dhcp
PEERDNS=no

Set up a Server

This blog entry pretty much covers it
http://www.opscode.com/blog/2013/03/11/chef-11-server-up-and-running/

You basically grab the right version of Chef Server
 wget https://opscode-omnibus-packages.s3.amazonaws.com/el/6/x86_64/chef-server-11.0.8-1.el6.x86_64.rpm

Install it
sudo yum localinstall chef-server-11.0.8-1.el6.x86_64.rpm --nogpgcheck

Configure and start
sudo chef-server-ctl reconfigure

Set up a Workstation

The workstation is your host machine, where you will write recipes and from which you will deploy them to the nodes.

I started following the instructions here: http://docs.opscode.com/install_workstation.html
But that got confusing and inaccurate pretty quickly.

In summary, what I did was:

Start up a new virtuals machine (configure network settings as above), then:
sudo curl -L https://www.opscode.com/chef/install.sh | bash

When that is finished check the install with

chef-client -v

There are three config files the workstation needs

knife.rb

knife configure --initial

admin.pem
scp root@chef.vb:/etc/chef-server/admin.pem ~/.chef
chef-validator.pem
scp root@chef.vb:/etc/chef-server/chef-validator.pem ~/.chef

Set up a Node

A Node is a machine for which  you will manage the configuration using Chef.
Start up a new virtual machine (configure network settings as above), then:
install the Chef client onto the Node using the bootstrap process.
To do this run the command on the workstation:

knife bootstrap node1.vb -x <username> -P <password> --sudo

Once this is done you can add recipes to the node and deploy them.

Create your first Cookbook

Create your first cookbook using the following command on your workstation:

sudo knife cookbook create mytest

There will now be a cookbook in the following location
/var/chef/cookbooks/mytest

You can go in and edit the default recipe file:
/var/chef/cookbooks/mytest/recipes/default.rb

Add something simple, for example we will write out a file from a template.

template "test.template" do
  path "~/test.txt"
  source "test.template.erb"
end


Then create the template file
/var/chef/cookbooks/mytest/templates/default/test.template.erb

add whatever text you like to the file.

Applying the Cookbook

First thing to do is upload the cookbook to the server

sudo knife cookbook upload mytest

Then add the cookbook to the node

knife node run_list add mynode 'recipe[mytest]'

Then use Knife to apply the cookbook using the Chef-client on the node

knife ssh name:mynode  -x <username> -P <password> "sudo chef-client"

Done!!!!