Ansible – build with maven to AWS

Ansible build maven AWS: Maven builds automation with AWS product placement.

With the professional process of developing and distributing a product and its new functions, the issue of reducing the daily identical operations with the maximum elimination of the manual component of the development process becomes critical over time. This is natural and convenient and allows you to devote more time to improving the product itself and, as a result, improving its quality.

This has led to the development of the following automated process as one of many ways of using the following tools and their combinations.

Several tools and cloud services can be used to implement such a task. Now, taking into account specific requirements, they are used most simply:

• Ubuntu,
• Ansible,
• Python,
• boto 2.47.0,
• boto3-1.4.4-py2,
• bash shell,
• AWS SDK,
• AWS CLI,
• elasticsearch,
• kibana,
• Jenkins.

The process uses VPN and DNS, with the necessary level of isolation of the tool at the global network level.

The process is executed as Jenkins 4 jobs: 1 main (starting) and three auxiliary. Each of the extra jobs performs its stage of the process. The main job – starts the additional, in a specific order with the condition that each previous job completes. In each of the Jenkins jobs, notifications to Slack are used, as much as possible, for complete logging of the execution of each operation, with the definition of their process timestamps on the dedicated log server. The dedicated log server simplifies the analysis of the process, possible errors in its flow, including and after the completion of jobs, and the removal of their results for various reasons.

After debugging the process with Jenkins, the process runs on schedule.

The diagram shows the process graphically.

Jenkins jobs:

0. The main one, “Run the workflow: Ansible build Maven AWS” is to start the process. As a result, AMI images of the product are created and put into working order:

1. Maven Build –> TAR: synchronization with the control system version, the assembly of the product in the TAR archive,
2. Ansible –> preparation of an AWS environment: The pre-removal potential of temporary facilities, the creation of a pair of RSA keys, VPC, security group,
3. Ansible –> build VM, AMI, transfer to DC: AWS creation of the basic intermediate virtual machine, assembly from it AMI image, its distribution by regions.

When debugging a process or only part of the process needs to be executed, jobs can be run in any order, with the necessary parameters and components generated by the previous position.

The subordinate jobs are described below.

1. Maven Build –> TAR: Ansible build maven AWS

Job for building servers. To run it, you only need to specify one input parameter, the “product version” – which you must prepare for the final image with the operating system. In the parameter value, there is a “version” by default if a regular version is built regularly.

This is where bash scripts are run sequentially:

• The source code of the product is extracted from the version control system (GIT, SVN),
• The associated documentation is extracted in PDF format,
• MAVEN and its plugins perform JAVA compilation, build source code with specific build profiles,
• Build installers particular to each type of operating system (Windows, Mac, Linux),
• Additional helper tools are collected,
• The product is assembled, archived, and placed in a specific location for the next task.

To perform this task, the environment should be configured appropriately: Ubuntu, JDK, Innosetup, Wine, SVN and GIT clients.

Once the job is complete, the packaged product is ready to be uploaded to the AWS EC2 instance. It is then stored in the AMI and cached in the Jenkins file system. It’s used in the “Ansible –> build VM, AMI, transfer to DC” job.

2. Ansible –> prepare an AWS environment: Ansible build maven AWS – the pre-removal potential of temporary facilities, the creation of a pair of RSA-keys, VPC, security group

The job is cleaning up the temporary resource from AWS that was created after a previous build process ran.

Jobs clean up previous resources, create a temporary instance with needed parameters, upload prebuilt, prepare an instance for AMI registration, create AMI with used SVN revision in its name, copy created AMI to predefined locations, and finish preparing model.
Incoming parameters:
– a key pair, its default value – SSH key for remote access to Amazon instance. It should be created in AWS before running the job and placed in the Jenkins filesystem.
– hdd2, the default value is “10”, Gb – a required size of the hard disk volume to store the server and its content.
– an initial region, the default value is “eu-west-1” – the area where the starting instance and its first created AMI are located.
– region dst1, the default value is “eu-central-1” – the region where the created AMI will be copied.
– region dst2, the default value is “us-east-1” – the region to which the created AMI will be copied.

The inbound parameter is defined to run the job: the starting region for the instance and AMI creation. The default value is Ireland == “EU-west-1”, where the initial model and its first created AMI are located. The description defines the list of possible values:

Code || Name
us-east-1 || US East (N. Virginia)
us-east-2 || US East (Ohio)
us-west-1 || US West (N. California)
us-west-2 || US West (Oregon)
ca-central-1 || Canada (Central)
eu-west-1 || EU (Ireland)
eu-central-1 || EU (Frankfurt)
eu-west-2 || EU (London)
ap-northeast-1 || Asia Pacific (Tokyo)
ap-northeast-2 || Asia Pacific (Seoul)
ap-southeast-1 || Asia Pacific (Singapore)
ap-southeast-2 || Asia Pacific (Sydney)
ap-south-1 || Asia Pacific (Mumbai)
sa-east-1 || South America (São Paulo)

During the execution of the workflow, the instance_id created is stored in the properties files for the subsequent use of the Ansible playbook.
Given the instance_id and region, the job will drop the instance and associated security group created in the area.
Instances created before the AMI and their snapshots will not be deleted in the region where they were made or in other areas.
If it’s mandatory to delete the AMIs from AWS EC2 – a user should do it manually, using the AWS web panel or some other way.

An SSH public/private key pair is used to access the temporary instance, which should be stored in the Jenkins file system (with a defined default value) and described as a job parameter.

To run the workflow, the AWS credentials (KEY and KEY_ID) should be stored in /var/lib/jenkins/.boto, and the user should be granted permissions in the AWS web panel.

The job will execute an Ansible playbook with the described scenario of actions.

3. Ansible –> build VM, AMI, transfer to DC: Ansible build maven AWS – creation AWS the basic intermediate virtual machine, assembly from it AMI-image, its distribution by regions

By default, the temporary instance (t2.micro) and AMI are prepared on the AWS location “eu-west-1|| EU (Ireland)”. For temporary instance two volumes are linked: OS and working product server. The size of the second working disk volume is defined as one of the job parameters and is 10 GB by default.

For the correct product to run in an instance with the Linux Ubuntu in the AMI image, the TCP settings section should be modified to include the following settings in /etc/sysctl.conf:

net.ipv4.tcp_keepalive_time=200
net.ipv4.tcp_keepalive_intvl=200
net.ipv4.tcp_keepalive_probes=8

0. Execute the workflow: Ansible build maven AWS

The job is to start the total build and deploy process.

As parameters should be defined in 2 destination regions to transfer final AMI: default values are “eu-central-1 || EU (Frankfurt)” and “us-east-1 || US East (N. Virginia)”.

As inbound parameters:
– RELEASE VERSION, its default value, and what should be built.
– The key pair, its default value – SSH key for remotely accessing Amazone build template instance. It should be created in AWS before the job execution and placed in the Jenkins filesystem.
– hdd2, the default value is 10 GB – a needed size of HDD volume to store the product server and its content.
– an initial region, the default value is “eu-west-1” – area, where the starting instance and its first created AMI are located.
– region dst1, the default value is “eu-central-1” – the region where the created AMI will be copied.
– region dst2, the default value is “us-east-1” – the region to which the created AMI will be copied.

The main job, “Execute the workflow”, executes the subsidiary processes.
After the job is finished, the instance and AMI are placed in Ireland; the AMI is also transferred to “eu-central-1 || EU (Frankfurt)” and “us-east-1 || US East (N. Virginia)”.

It is worth mentioning that in the YAML playbook of Ansible, all syntax is counted by every symbol. Even every space should be in its place. Otherwise, an error will occur during execution, and often without a precisely defined area of a typo.

The following commands were used in the Ansible playbooks:

• ec2
• ec2_ami
• ec2_copy
• ec2_group
• ec2_tag
• shell
• file
• copy
• apt
• unarchive