Drupal deployments in AWS ECS with AWS Copilot

Jan 22, 2021 AWS DRUPAL

Share on:

Example of a stateful application deployment in ECS Fargate using AWS Copilot CLI from a docker-compose setup

Copilot is a new released tool from AWS to allow developers deploying containerized applications in AWS Fargate and aditional resources like s3 buckets easily.

In this tutorial I will deploy a complete drupal application with this architecture using copilot:

made with cloudcraft

TL;DR

Copilot is a great cli but still in early stages. I dont want my dev teams to debug cloudformation stacks neither grant them permissions in prod accounts.

What did I miss from copilot?

More use cases for jobs/tasks: event-scheduled tasks or let reusing service manifest specs (credentials, task roles, etc.) in a simple way.
Cross accounts functionalities: being able to promote artifacts from develop to production accounts and follow good AWS practices.
Works in conjunction with another tools like AWS CDK or Hashicorp terraform.

The full sample code is in github (v1.1.0 branch)

Motivation

Test new cloud tools to accelerate development.
Learn about Cloudformation and look for alternatives to our current deployment workflows with terraform.

Requisites

Some knowledge about drupal or similar stacks with docker containers.
AWS account with a s3 bucket to store drupal static files and mysql backups. Remember to secure your bucket access, for instance: https://docs.aws.amazon.com/AmazonS3/latest/dev/example-bucket-policies.html#example-bucket-policies-use-case-3
Install copilot: https://aws.github.io/copilot-cli/docs/getting-started/install/
A Dockerfile and some code

1. Setup Drupal

The drupal aplication is started in localhost from a docker-compose.yaml with the dockerhub images of nginx, drupal, mysql and minio extended with the gomplate template renderer to configure the services at runtime. This will allow to use drush commands like database migrations and manage drupal without access to drupal itself. There is an open issue in aws/containers-roadmap you should follow if using ECS/Copilot with stacks like Drupal, Django, etc. who needs a CLI for administration tasks.

Another use case of gomplate that is not covered is the ability to directly use AWS secret manager or parameters store among others inside containers.

After Starting a new drupal project installing the umami demonstration drupal project I configured the S3FS drupal module to use a S3 bucket instead local volumes. Backup your database, upload your static files to your secured S3 and you're ready to go.

2. Create the Copilot application

The first step is to specify which AWS CLI profile to use. Configure your AWS CLI credentials file and set the correct profile, otherwise the default profile is used:

1export AWS_DEFAULT_PROFILE=kammin

Next create the first app resources by running:

1copilot app init drupal

Copilot will create the required AWS Identity and Access Management roles to manage the infrastructure and the sub-directory copilot/ for the required manifests.

3. Create the first AWS environment

A environment is the common shared resources used by the services of the app consisting of a new or existing VPC with public and private subnets across two availability zones, IAM roles, an ECS cluster and a Cloud Map Namespace for service discovery.

1copilot env init --app drupal --name dev --profile kammin --default-config

3.1 Create a Drupal service

The Copilot "Backend Service" will create a ECS service from the selected Dockerfile in the private subnet of the VPC only accessible from other services. If the manifest already exists is not overwriting but the command have to be launched to create the ECR repository and the required values in AWS Parameter Store.

1copilot svc init --name drupal --svc-type "Backend Service" \
2                 --port 9000 --dockerfile ./docker-images/drupal/Dockerfile

The build options can be extended to use build context, build args or a target in a multibuild stage to name a few.

Copilot comes with some additional AWS resources creation in the cli (more to come yet). A s3 bucket attached to drupal is created with this command to save Drupal static files:

1copilot storage init --name static-files --storage-type S3 --workload drupal

In the created file under copilot/drupal/addons/static-files.yaml I will manually edit the managed policy to grant acces to the backend service to another s3 bucket to sync files from.

To create additional AWS resources check this guide. The custom cloudformation template needs a IAM ManagedPolicy with the generated ARN in the Outputs to inject the value to the attached service.

There is also a sample RDS CloudFormation template in the repository with fixed values for username and password. Do not reuse this example in producction workloads. You can use secrets following the documentation.

After customizing our resources deploy the backend service, the s3 bucket and the RDS database with this command:

  1$ copilot svc deploy --name drupal --env dev
  2Environment dev is already on the latest version v1.1.0, skip upgrade.
  3Sending build context to Docker daemon  226.1MB
  4Step 1/17 : FROM drupal:9.0-fpm-alpine
  5 ---> fec11100cd3e
  6Step 2/17 : COPY --from=hairyhenderson/gomplate:v3.8.0-slim /gomplate /bin/gomplate
  7 ---> Using cache
  8 ---> 4955d04a4757
  9Step 3/17 : RUN apk --update add --no-cache fcgi mysql-client bash
 10 ---> Using cache
 11 ---> 38bcf8c995fd
 12Step 4/17 : RUN curl -o /usr/local/bin/php-fpm-healthcheck https://raw.githubusercontent.com/renatomefi/php-fpm-healthcheck/master/php-fpm-healthcheck &&    chmod +x /usr/local/bin/php-fpm-healthcheck
 13 ---> Using cache
 14 ---> e6d257728fb1
 15Step 5/17 : COPY docker-images/drupal/conf/php-fpm-status.conf /usr/local/etc/php-fpm.d/status.conf
 16 ---> Using cache
 17 ---> 7b0128d021d8
 18Step 6/17 : COPY docker-images/drupal/conf/php.ini /usr/local/etc/php/php.ini
 19 ---> Using cache
 20 ---> 32c08698cb9c
 21Step 7/17 : COPY docker-images/drupal/conf/drush.yml /root/.drush/drush.yml
 22 ---> Using cache
 23 ---> d869a642f327
 24Step 8/17 : COPY docker-images/drupal/conf/alias.site.yml /root/.drush/sites/alias.site.yml
 25 ---> Using cache
 26 ---> ee2222b5a4c9
 27Step 9/17 : HEALTHCHECK --interval=5s --timeout=10s --start-period=5s --retries=3 CMD [ "php-fpm-healthcheck" ]
 28 ---> Using cache
 29 ---> c32abc7ba4ba
 30Step 10/17 : COPY docker-images/drupal/assets/ /assets
 31 ---> 91876c727481
 32Step 11/17 : COPY src/ /var/www/html/
 33 ---> 3f5aca81bb81
 34Step 12/17 : RUN composer install
 35 ---> Running in c8b24abc10e3
 36Loading composer repositories with package information
 37Installing dependencies (including require-dev) from lock file
 38Nothing to install or update
 39Package doctrine/reflection is abandoned, you should avoid using it. Use roave/better-reflection instead.
 40Generating autoload files
 4128 packages you are using are looking for funding.
 42Use the `composer fund` command to find out more!
 43  * Homepage: https://www.drupal.org/project/drupal
 44  * Support:
 45    * docs: https://www.drupal.org/docs/user_guide/en/index.html
 46    * chat: https://www.drupal.org/node/314178
 47Removing intermediate container c8b24abc10e3
 48 ---> 67093b83ec33
 49Step 13/17 : RUN chown -R www-data:www-data web/sites/default
 50 ---> Running in eb7f359f2b59
 51Removing intermediate container eb7f359f2b59
 52 ---> 6143b056e60b
 53Step 14/17 : RUN ln -s /var/www/html/vendor/drush/drush/drush /usr/local/bin/drush
 54 ---> Running in 9a25817f3fb6
 55Removing intermediate container 9a25817f3fb6
 56 ---> d672dfea1318
 57Step 15/17 : EXPOSE 9000
 58 ---> Running in e8baf8c678bb
 59Removing intermediate container e8baf8c678bb
 60 ---> 16ea15d8308f
 61Step 16/17 : ENTRYPOINT ["/bin/bash","/assets/bin/docker-entrypoint.sh"]
 62 ---> Running in 8be54fcc4b0d
 63Removing intermediate container 8be54fcc4b0d
 64 ---> ae9504fc024c
 65Step 17/17 : CMD ["php-fpm","--fpm-config","/usr/local/etc/php-fpm.conf","-c","/usr/local/etc/php/php.ini"]
 66 ---> Running in f57fa9581629
 67Removing intermediate container f57fa9581629
 68 ---> 9625be037456
 69Successfully built 9625be037456
 70Successfully tagged 164569299119.dkr.ecr.eu-west-1.amazonaws.com/drupal/drupal:9c587264
 71WARNING! Your password will be stored unencrypted in /home/steran/.docker/config.json.
 72Configure a credential helper to remove this warning. See
 73https://docs.docker.com/engine/reference/commandline/login/#credentials-store
 74
 75Login Succeeded
 76The push refers to repository [164569299119.dkr.ecr.eu-west-1.amazonaws.com/drupal/drupal]
 77c66c9b92c834: Pushed 
 780b89619b8195: Pushed 
 79da92061480e6: Pushed 
 809deb8b49c259: Pushed 
 815141c71b26ca: Pushed 
 82d549a5c3cc23: Pushed 
 8351391cc9f0d9: Pushed 
 847c3559086a98: Pushed 
 85bd2706928b8b: Pushed 
 8685f0bd8fd97d: Pushed 
 87f44ba2e6e89c: Pushed 
 8852324637408c: Pushed 
 899c3da75c664e: Pushed 
 90739fdd2386a4: Pushed 
 91f43c6165c459: Pushed 
 927b8497bfba0f: Pushed 
 934d94a9b9e286: Pushed 
 9479d9a1461370: Pushed 
 9598556f763a25: Pushed 
 96add721aa4e5c: Pushed 
 9744fbeaef347a: Pushed 
 98bc11fe48042d: Pushed 
 9924e52497c24f: Pushed 
10086d905c1f58e: Pushed 
10122573737ba76: Pushed 
102777b2c648970: Pushed 
1039c587264: digest: sha256:5f4a4b9ff3cac43f40be614206f8d55e6fa563965ced94b02449eff143b3f94b size: 5752
104
105✔ Deployed drupal, its service discovery endpoint is drupal.drupal.local:9000.

3.2 Create nginx service

The nginx webserver will be accessible from internet using an Application Load Balancer.

1copilot svc init --name nginx --svc-type "Load Balanced Web Service" \
2                 --port 80 --dockerfile ./docker-images/nginx/Dockerfile

This service will also need access to the static files in the S3 bucket. I will attach the created task role in the drupal service with access to S3 to the nginx service using a cloud formation file in the addons directory.

Deploy the webserver by running:

 1$ copilot svc deploy --name nginx --env dev
 2Environment dev is already on the latest version v1.1.0, skip upgrade.
 3Sending build context to Docker daemon  226.1MB
 4Step 1/7 : FROM kammin/copilot-drupal as src
 5 ---> b86a80e9e240
 6Step 2/7 : FROM nginx:1.19.6
 7 ---> ae2feff98a0c
 8Step 3/7 : COPY --from=hairyhenderson/gomplate:v3.8.0-slim /gomplate /bin/gomplate
 9 ---> Using cache
10 ---> 47fed42c1182
11Step 4/7 : COPY docker-images/nginx/assets/ /assets
12 ---> Using cache
13 ---> 850dfebe524a
14Step 5/7 : COPY --from=src /var/www/html/web /var/www/html/web
15 ---> Using cache
16 ---> aa2a289596db
17Step 6/7 : ENTRYPOINT ["/bin/bash","/assets/bin/docker-entrypoint.sh"]
18 ---> Using cache
19 ---> f50a5fd9d908
20Step 7/7 : CMD ["nginx", "-g", "daemon off;"]
21 ---> Using cache
22 ---> 8970911cf339
23Successfully built 8970911cf339
24Successfully tagged 164569299119.dkr.ecr.eu-west-1.amazonaws.com/drupal/nginx:9c587264
25WARNING! Your password will be stored unencrypted in /home/steran/.docker/config.json.
26Configure a credential helper to remove this warning. See
27https://docs.docker.com/engine/reference/commandline/login/#credentials-store
28
29Login Succeeded
30The push refers to repository [164569299119.dkr.ecr.eu-west-1.amazonaws.com/drupal/nginx]
319d5ab819b891: Pushed 
3293e8457b8e86: Pushed 
3302c85958d14e: Pushed 
344eaf0ea085df: Pushed 
352c7498eef94a: Pushed 
367d2b207c2679: Pushed 
375c4e5adc71a8: Pushed 
3887c8a1d8f54f: Pushed 
399c587264: digest: sha256:8b20830a31c3943e130762bf92c388556c1e6ca1681867da2344372b44e538a3 size: 1993
40
41
42✔ Deployed nginx, you can access it at http://drupa-publi-kxncfk7jyr9z-339478161.eu-west-1.elb.amazonaws.com/.

3.3 Sync static files to s3 and database to AWS RDS

To run the migration script I will parse the info provided by 'copilot svc show'

 1S3_BUCKET=$(copilot svc show --name drupal --json | jq -r '.variables[] | select(.name=="S3_BUCKET") | .value')
 2
 3S3_BUCKET_BACKUP=$(copilot svc show --name drupal --json | jq -r '.variables[] | select(.name=="S3_BUCKET_BACKUP") | .value')
 4
 5MYSQL_HOST=$(copilot svc show --name drupal --json | jq -r '.variables[] | select(.name=="MYSQL_HOST") | .value')
 6
 7TASK_DEFINITION=$(copilot svc status --name drupal --json | jq -r .Service.taskDefinition)
 8
 9TASK_ROLE=$(aws ecs describe-task-definition --task-definition ${TASK_DEFINITION} --query 'taskDefinition.taskRoleArn' --output text | cut -d '/' -f2)
10
11copilot task run -n drupalmigrate --app drupal --env dev --follow \
12  --task-role ${TASK_ROLE} \
13  --dockerfile ./docker-images/drush/Dockerfile \
14  --env-vars S3_BUCKET=${S3_BUCKET},S3_BUCKET_BACKUP=${S3_BUCKET_BACKUP},S3_REGION=eu-west-1,MYSQL_HOST=${MYSQL_HOST}
15
16✔ Successfully provisioned task resources.
17
18Sending build context to Docker daemon  4.608kB
19Step 1/4 : FROM kammin/copilot-drupal
20 ---> b86a80e9e240
21Step 2/4 : RUN apk --update add --no-cache aws-cli
22 ---> Using cache
23 ---> 230df5257330
24Step 3/4 : COPY assets/ /assets
25 ---> Using cache
26 ---> ba88af3c3d76
27Step 4/4 : CMD ["/assets/scripts/db_migrate.sh"]
28 ---> Using cache
29 ---> d857db40fc4c
30Successfully built d857db40fc4c
31Successfully tagged 164569299119.dkr.ecr.eu-west-1.amazonaws.com/copilot-drupalmigrate:latest
32WARNING! Your password will be stored unencrypted in /home/steran/.docker/config.json.
33Configure a credential helper to remove this warning. See
34https://docs.docker.com/engine/reference/commandline/login/#credentials-store
35
36Login Succeeded
37The push refers to repository [164569299119.dkr.ecr.eu-west-1.amazonaws.com/copilot-drupalmigrate]
387a3251d8fb97: Layer already exists 
3994ae59e3f0e1: Layer already exists 
4006f2e1371265: Layer already exists 
41f5a9a0202fd7: Layer already exists 
428e1fc8982109: Layer already exists 
434f7485f057d9: Layer already exists 
4479ac02142b14: Layer already exists 
4562d90a23a5d9: Layer already exists 
46d933dd23c7bf: Layer already exists 
471fed3837e243: Layer already exists 
48bd2706928b8b: Layer already exists 
4985f0bd8fd97d: Layer already exists 
50f44ba2e6e89c: Layer already exists 
5152324637408c: Layer already exists 
529c3da75c664e: Layer already exists 
53739fdd2386a4: Layer already exists 
54f43c6165c459: Layer already exists 
557b8497bfba0f: Layer already exists 
564d94a9b9e286: Layer already exists 
5779d9a1461370: Layer already exists 
5898556f763a25: Layer already exists 
59add721aa4e5c: Layer already exists 
6044fbeaef347a: Layer already exists 
61bc11fe48042d: Layer already exists 
6224e52497c24f: Layer already exists 
6386d905c1f58e: Layer already exists 
6422573737ba76: Layer already exists 
65777b2c648970: Layer already exists 
66latest: digest: sha256:b5f70f1ed70e2657e40833916d389876abf0bc954626f228f87b1574a8bb0b1d size: 6171
67✔ Successfully updated image to task.
68
69✔ Task drupalmigrate is running.
70
71copilot-task/drupalmigrat + source /assets/bin/entrypoint.functions
72copilot-task/drupalmigrat + process_templates
73copilot-task/drupalmigrat + gomplate -f /assets/templates/settings.local.php.tmpl -o /var/www/html/web/sites/default/settings.local.php
74copilot-task/drupalmigrat + gomplate -f /assets/templates/default.settings.php.tmpl -o /var/www/html/web/sites/default/settings.php
75copilot-task/drupalmigrat + exec /assets/scripts/db_migrate.sh
76copy: s3://tetestetestetes-drupal-s3fs/s3fs-public/crema-catalana-umami.jpg to s3://drupal-dev-drupal-static-files/s3fs-public/crema-catalana-umami.jpg
77copy: s3://tetestetestetes-drupal-s3fs/s3fs-public/chili-sauce-umami.jpg to s3://drupal-dev-drupal-static-files/s3fs-public/chili-sauce-umami.jpg
78....
79copy: s3://tetestetestetes-drupal-s3fs/s3fs-public/vegan-chocolate-nut-brownies.jpg to s3://drupal-dev-drupal-static-files/s3fs-public/vegan-chocolate-nut-brownies.jpg
80copy: s3://tetestetestetes-drupal-s3fs/s3fs-public/watercress-soup-umami.jpg to s3://drupal-dev-drupal-static-files/s3fs-public/watercress-soup-umami.jpg
81copy: s3://tetestetestetes-drupal-s3fs/s3fs-public/styles/medium_3_2_2x/public/watercress-soup-umami.jpg to s3://drupal-dev-drupal-static-files/s3fs-public/styles/medium_3_2_2x/public/watercress-soup-umami.jpg
82copy: s3://tetestetestetes-drupal-s3fs/s3fs-public/translations/.htaccess to s3://drupal-dev-drupal-static-files/s3fs-public/translations/.htaccess
83download: s3://tetestetestetes-drupal-s3fs/mysql-backups/drupal_umami_202101112017.sql to ./mysql_dump.sql
84copilot-task/drupalmigrat  [info] Executing: command -v mysql [0.79 sec, 9.52 MB]
85copilot-task/drupalmigrat  [info] Executing: mysql --defaults-file=/tmp/drush_lidJBj --database=drupal --host=dr1e5h2wuan1rsf.cqrlqadfjarc.eu-west-1.rds.amazonaws.com --port=3306 --silent -A < /opt/drupal/mysql_dump.sql [0.89 sec, 9.64 MB]
86Task has stopped.

Check website

After the sync job the site is up.

Delete app

 1$ copilot app delete --yes --name drupal
 2✔ Deleted service drupal from environment dev.
 3✔ Deleted resources of service drupal from application drupal.
 4✔ Deleted service drupal from application drupal.
 5✔ Deleted service nginx from environment dev.
 6✔ Deleted resources of service nginx from application drupal.
 7✔ Deleted service nginx from application drupal.
 8✔ Deleted environment dev from application drupal.
 9✔ Cleaned up deployment resources.
10✔ Deleted application resources.
11✔ Deleted application configuration.
12✔ Deleted local .workspace file.

References:

https://aws.github.io/copilot-cli/
https://maartenbruntink.nl/blog/2020/08/16/deploying-containers-with-the-aws-copilot-cli-part-1/
https://github.com/drupalwxt
https://nathanpeck.com/speeding-up-amazon-ecs-container-deployments/
https://youtu.be/WOhm_YgrGwY