Creating a Jenkins Node on LXC Container
What is a container
A container is a virtualization method for isolating the applications (or even operating systems) from each other.
Why do we need a container for Jenkins node
In Jenkins, node is a location where our jobs run. One user can use one node for all of his processes and multiple users may also use one node for all of their processes. In Jenkins freestyle project, we can use bash shell or windows command shell due to which there is a possibility to navigate anywhere in the server machine. This possibility can lead to various security and integrity issues for server administrators and also for other users using that webserver. So one must isolate each node and allocate each node to each user separately.
What is LXC Container
LXC stands for Linux Containers. LXC is a package for linux operating systems and provides linux users with containers which may contain a whole linux operating system while also being lightweight than a virtual machine. More information regarding LXC can be found at https://linuxcontainers.org/lxc/introduction/
On ubuntu 20.04 one can install LXC using command
[sudo] apt-get install lxc
Creating a container with LXC
NOTE: Throughout this document, the name of the container will be my-container
and the name of the user will be user1
. So wherever my-container is written, one may change the name to whatever one wants to give it.
Pre-requisites
Before proceeding, it is important to mention that at the point of writing this document following are the specifications for linux kernel and distribution:
Linux Kernel: 5.15.0-46-generic (can be checked on ubuntu by command uname -r
)
Distribution: Ubuntu focal (20.04) (can be checked on ubuntu by command lsb_release -a
)
By default linux users are not allowed to create any network device on the machine. For doing that, one must add uid
and gid
in the /etc/lxc/lxc-usernet
. The uid and gid of the user you want to use can be found in the files /etc/uid
and /etc/gid
respectively.
After getting the gid
and uid
of the user you want to allow for creating the network devices, one must go to /etc/lxc/lxc-usernet
and add the uid
and gid
in the following format:
<username> veth lxcbr0 10
e.g.
jenkins veth lxcbr0 10
In above example jenkins
is the username, veth
is the command used for creating the bridges between virtual network devices and physical network devices (you will be able to see that ethernet device in our container will be lxcbr0
) and lxbr0
is the network device we want to create. ‘10’ represents the number of devices we want to create using our specified user.
According lxc documentation, in ubuntu 20.04 an additional command is required before creating lxc container:
export DOWNLOAD_KEYSERVER="hkp://keyserver.ubuntu.com"
Creating image
After this, one can create container using following command
systemd-run --unit=my-unit --user --scope -p "Delegate=yes" -- lxc-create -t download -n my-container
This runs lxc container with unit name my-unit
, container name my-container
and delegates a control group (also known as the cgroup
) which is needed for resource allocation for processes in container.
This will output the list of available linux distributions in which one may want to run the container and will prompt for Distribution as shown in the following image.
After selecting the suitable distribution, release and architecture (which is also mentioned in the table), the container may be created as shown in the image below:
Next thing is to start a container which will change its state from STOPPED to RUNNING using the following command.
lxc-start -n my-container
Above command will have no output if it succeeds.
The state of the container can be checked using the following command.
lxc-info -n my-container
After starting the container, its state will be set as running and is just like turning a linux machine ON.
From this point onwards, if one wants to use the machine in the terminal then use the following command and this will switch the terminal to the root of the container.
lxc-attach my-container
Now the container is ready to be used and is completely isolated from the host machine.
Using SSH to access container with username and password
The above mentioned method can be used to attach the host machine terminal to the container and this can be used to access the container. But if one wants to access the machine remotely then one possible and well-known method will be to configure and use SSH on the container.
As it is an out of the box linux distro and only the root user is present, so first create another user using the following command and then manage its permissions for /home folder
.
#Considering you remain the root user for execution of all the following #commands
useradd user1
cd /home
mkdir user1 #Creating home directory for user1
chown user1:user1 user1 #Giving ownership of home directory to new user
#For adding the same shell and bashrc configurations for new user, use #following
#command, otherwise the shell will be very basic for new user and will be #very inconvenient to use.
usermod -s /bin/bash user1
For the sake of simplicity of this document the name used for the new user is user1
here.
(You may want to set the password for user1
by executing the passwd
command in root.)
At this point user1
is not in sudoers. For adding it to the sudoers, it must be added in the sudo group which can be done by using the following command.
usermod -a -G sudo user1
Now switch to the user1 using the following command.
su - user
Now install openssh-server for configuring the ssh on user1.
sudo apt install openssh-server
After that one must find the ip of the container we are using, for this either run following command while in container with user1,
sudo apt-install net-tools
#Because ifconfig is part of net-tools which are by default not installed on new #container
Ifconfig -a
OR open a new terminal in the host machine and execute the following command.
lxc-info my-container -iH
So, the ip of the container is 10.0.3.127
.
The command for establishing an ssh connection to a remote machine is mentioned below and it will ask for the password of the remote machine which is actually the container in our case.
ssh user1@10.0.3.127
After entering the password, terminal will switch to the container’s user1 as can be seen in the following image.
Using SSH to access Jenkins agents on the container
First install some initial dependencies (git, jdk, jre) on the containers for running agents on the container.
sudo apt update
sudo apt install default-jdk default-jre git maven
Now login to jenkins with administrator privileges and create a node in it from Dashboard > Manage Jenkins > Nodes
and press + New Node
. Enter a name for the node and select the desirable node type.
For this documentation, the node will be a permanent type and the name will be temp_node
.
After this click on Create
which will display the configuration page of the node.
- Write the description of the node as desired.
- Number of executors means number of threads running at a time (it will be better to set it to the number of processors present on the machine which is running this node).
- Remote root directory will be the directory where the jobs will run by default on the node. In our case this will be a specified directory inside the container.
- Labels indicate that this node will run only when a job with specified labels is run, otherwise this node will not be used (also depends on the usage method in the next option). If the purpose is to use the node by default for every job, then leave it empty.
- Select a desired usage option.
- In launch methods, select “Launch agents via SSH”
- In “Host” enter the ip address of the container which is 10.0.3.127 in our case.
- In “Credentials”, click Add and this will open another sub dialog for entering credentials information.
- Select the kind as “Username wih password”.
- Leave other options as is and write the username and password of the container user, in our case the username will be user1 and password will be the password which was set for user1.
- “ID” and “Description” are optional.
- Click on “Add”.
- Now as the credentials are added, click on the dropdown menu and select the username you just added. In our case it is user1 as the username added was user1.
- After this rest of the options need not to be changed if this node is going to be a default node.
- Click on “Save”.
After complete setup, the configuration for this node will look something like this.
If no issue is encountered during this whole setup, jenkins will take us to the log and after sometime (when the ssh connection is established) we can see “Agent successfully connected and online” at the bottom of the log as can be seen in the screenshot below.
After this point, node will be able to run jobs from the container directory.
Reference Links
Documentation for LXC containers can be found at: https://linuxcontainers.org
Details regarding Jenkins ssh agents can be found at: https://acloudguru.com/blog/engineering/adding-a-jenkins-agent-node