Running TeamCity on EC2

 

One of Headspring’s core mantras is to outsource everything that is not part of our core business. We are true believers on running the business in the cloud. We are even giving a talk about it. We use many cloud service providers on a daily basis. We use Google Apps for Email, Calendar, Voice and Intranet. We use Salesforce as our CRM. We use BitBucket to host our source code.

We use all this services to do things that are necessary in order to run the company but that we don’t want to manage or build ourselves. So, we decided to do the same thing with our build system. As we were using TeamCity as our CI server and it has integration with EC2 out of the box we decided to go with EC2.

First, TeamCity is composed of two pieces. The build server, that runs TeamCity’s dashboard and the build agents, that actually run the builds. As TeamCity Server is a Java application, a small Linux instance on EC2 is enough to run it. The server itself won’t leverage the Elastic features on EC2, but having the server running on the same region and availability zone as the agents is important in order to avoid Data Transfer charges.

 

Server

 

To install TeamCity server on Linux, follow this steps:

1) Get TeamCity from JetBrains.

2) Unpack the file Teamcity<version number>.tar.gz as documented in the docs.

3) Elevate your privileges to root:

> sudo su

This is needed because we want the server to listen to port 80 and ports below 1024 can only be listened by the root user.

4) Install Sun/Oracle JDK.

EC2 instance is a RedHat distribution, that uses OpenJDK by default. To the cloud integration on TeamCity to work properly, it needs Sun/Oracle’s JDK.

5) With Sun/Oracle’s JDK installed, set the JAVA_HOME environment variable to use it, instead of OpenJDK:

> JAVA_HOME=/usr/java/jdk<version>

6) Change TeamCity server port to 80 (if you need to)

7) Start the Server. On TeamCity’s bin directory, run:

> ./teamcity-server.sh start

That’s it. The server should be up and running. If you hit instance’s public URL on your browser, after the server finishes starting up you should get a screen to accept TeamCity’s EULA.

TeamCIty_EULA

After accepting the EULA, you will be prompted to create an Administrator Account. With the Administrator Account created, TeamCity server is installed and running properly.

Installing the Build Agent

 

With the Server up and running it’s time to create the agents that will run the builds.

To do so, first launch a instance ( I’m going to use a Windows one as we are doing .NET here) on EC2 and configure it to work as a TeamCity agent. In our case, we chose to use a Large Windows x64 instance.

As it’s necessary to remote into the instance to configure it to run as a TeamCity agent, wait approximately 15 minutes after the instance is launched to Remote into it, otherwise you won’t be able to get the password from AWS.

Wait_for_instance[1]

With the instance credentials, follow this steps:

1) Remote into the instance

2) Open the browser and go to TeamCity Server URL. After authenticating, go to the Agents tab and download MS Windows Installer.

Agents_Tab

3) Run the installer. After the agent is installed, a form to “Configure Build Agent Properties” will be prompted. Change the serverUrl property to point to the correct URI.

4) TeamCity agent-server communication uses port 9090 by default, unless you’ve changed it when configuring Build Agent properties on the previous step. So, in order to enable the communication between agent and server, create a inbound rule on Windows firewall to allow communication trough port 9090.

5) If everything is configured properly, you should have a Unauthorized agent showing up on TeamCity Server.

unauthorized_agent

6) Authorize the agent and check that it’s capable of running properly your builds.

 

Making the Build Agent Elastic

 

Right now we have a EC2 based TeamCity server and a TeamCity Agent running on a EC2 instance but it’s not leveraging the Elastic features of TeamCity yet. To enable Elastic, Cloud Based, build agents one has to create a AMI image of the instance  running the Agent.

1) On AWS, select the running instances and right-click on the instance running the Build Agent. On the context menu, select “Create Image (EBS AMI)”

Create_Image_EBS[4]

2) When the AMI becomes available on aws, go to Cloud Tab under TeamCity Agents page:

cloud

3) On the configuration page, create a new Cloud Profile. As of now, the only option of Cloud Type is Amazon EC2. Choose your location and availability zone to match the ones the server instance is running in order to avoid extra Data Transfer costs.

cloud_profile[5]

 

That’s it. TeamCity is configured to use Elastic Build Agents.

When a build is triggered, if there are no agents available to run it, the build will be placed on the Build Queue. A instance will be requested to EC2. It will take approximately 10 minutes for the instance to be running and authorized as a Build Agent on TeamCity.

instance_starting

My goal with this post was to document the process of migrating Headspring’s build system to leverage EC2 features. I hope this helps not only me.

One last thing, can you do me a favor and vote for this TeamCity feature request?

  • When you first create an instance of a machine, it only has a single disk attached – the disk that the boot image is on. TeamCity should not use that disk for its TEAMCITY_DATA_PATH. So we need to create a new EBS (Elastic Block Store) disk, attach it, partition it, format it, mount it, and then configure TeamCity to use it. Here are the steps:

    Create a new EBS volume
    Log into the AWS Management Console using our AWS credentials.
    Select the EC2 tab at the top.
    On the left side menu, select Elastic Block Store, Volumes
    Create a new volume, You’ll have to specify the size, and that it is not based on any snapshot. I made mine 30GB
    Attach the volume to the Instance
    Select the newly created volume in the AWS Management Console, the click the “Attach Volume” button above the list of volumes.
    When asked what Instance to attach it to, select the Instance that is running the TeamCity server
    When asked for what device name to use, you can just use the default (/dev/sdf)
    Partition the volume
    SSH to your TeamCity running instance. Ensure you are root by running sudo su
    type fdisk to show all the attached disks. You should see something like /dev/sda1 (this is the Instance disk – leave it alone) and /dev/xsdf (note it is not the same as what you picked in step 2.3 above)
    now type fdisk /dev/xsdf, which enters the fdisk partition program. It has its own commands. Type m for help.
    type n to create a new partition. It will be a primary partition, partition 1. Then just accept the defaults for block size, etc.
    type w to write the new partition table out
    type q to exit fdisk
    Format the volume
    ext3 is a reasonable file system to use, so type mkfs.ext3 /dev/xsdf
    Mount the disk so you can use it right now
    mkdir /data (or whatever you want the directory to be where this is mounted)
    mount /dev/xsdf /data
    Edit the /etc/fstab file so that the disk will be mounted the next time the machine boots
    {favorite editor} /etc/fstab – I use nano, others like vi, vim, emacs…
    add this line at the end: /dev/xsdf /data ext3 defaults 1 2
    Finally, configure TeamCity to use the new data directory
    {favorite editor} ~/.bashrc
    add this line: export TEAMCITY_DATA_PATH=/data/.BuildServer
    To set the environment variables right now, run source ~/.bashrc
    Start the TeamCity server as described above. Check your work by browsing to the TeamCity server, going to the administration section, then Server Configuration. You should see /data/.BuildServer as the Data Directory.
    Celebrate!

  • Paul Ryan

    Since you’ve been running this for a couple of years, mind giving us an idea of the general cost per month for this kind of setup (with the approximate number of builds per month).

    Thank for the great tutorial.