# 6. Cloudify - Complex App in AWS

In this tutorial, we will use a Docker image of the premium version of the Cloudify Manager to develop a blueprint of a cloud application on AWS. We will instantiate IAM users, roles, policies, a Lambda function and an RDS instance, along with the supporting network services like VPC, Gateways and Subnets.

### Prerequisites

Note: This tutorial is tested with Cloudify 5.0.0

### Warm Up

• Open a terminal and start the Docker image with the premium version of the Cloudify Manager.
sudo docker start cfy_manager_local

cfy_manager_local

• Open your browser and go to localhost:80. Login in the Cloudify console (credentials are admin admin).
• Click on the Blueprints widget over the Upload Blueprint blue button and then on the Cloudify Composer blue button. Cloudify Composer is the Cloudify graphical editor for designing cloud applications and packaging them in blueprints.
• On the top-left corner, under the Blueprints panel, click on Plugins and add the AWS plugin by clicking on Add in the cloudify-aws-plugin widget. Finally, go back to the Cloudify Composer Tab.
• On the top-left corner, go back to the Topology to start designing the cloud application.

### Blueprint AWS Nodes Configuration

Now we start developing the YAML blueprint for deploying our cloud application. Under the hood, Cloudify invokes AWS Boto SDK for Python. Boto3 exposes APIs to create, configure, and manage AWS services such as EC2, S3 and Lambda functions. Therefore, syntax, arguments and parameters are very similar to what the AWS Boto3 documentation describes.

• From the Built-in Types panel on the left, create (i.e. drag)
• 1 node of type:
• cloudify.nodes.aws.iam.Role
• cloudify.nodes.aws.lambda.Function
• cloudify.nodes.aws.s3.BucketObject
• cloudify.nodes.aws.rds.Instance
• cloudify.nodes.aws.rds.ParameterGroup
• cloudify.nodes.aws.ec2.SecurityGroup
• cloudify.nodes.aws.ec2.SecurityGroupRuleIngress
• cloudify.nodes.aws.rds.SubnetGroup
• 2 nodes of type:
• cloudify.nodes.aws.iam.User
• cloudify.nodes.aws.ec2.Subnet
• cloudify.nodes.aws.ec2.RouteTable
• cloudify.nodes.aws.ec2.Route
• 3 nodes of type:
• cloudify.nodes.aws.s3.Bucket
• 4 nodes of type:
• cloudify.nodes.aws.iam.Policy
• Each node has a client_config property which stores cloud account credentials. Click on each node and past the following code in the client_config field.
aws_access_key_id: { get_secret: aws_access_key_id }
aws_secret_access_key: { get_secret: aws_secret_access_key }
region_name: { get_input: aws_region_name }

• We already configured the AWS access and secret keys in the previous tutorial. Now, we have to define the aws_region_name input. On the left panel, click on Inputs & Outputs. Add the input aws_region_name and write a brief description. Put as default your AWS region (e.g., eu-central-1) and as type string. Finally, click on the + button.

#### Configure S3 Bucket

Amazon Simple Storage Service (Amazon S3) is an object storage service that allows you to store data in the cloud. In our cloud application, we will set up three S3 buckets.

• Click on each of the AWS S3 buckets and set the number of instances to 1.
• In the resource_config field, you can add the following parameters (see example below).
• Bucket: String. The bucket name.
• ACL: String. The canned ACL to apply to the bucket. We will apply access control policies to IAM users directly, so we will not use this. For a detailed comparison about buckets and IAM policies, check the documentation.
• CreateBucketConfiguration: Map.
• LocationConstraint: String. Specifies the region where the bucket will be created. If you don’t specify a region, the bucket is created in the US East (N. Virginia) Region (us-east-1).
Bucket: cloudify-lambda-bucket
CreateBucketConfiguration:
LocationConstraint: { get_input: aws_region_name }

• The S3 buckets do not have any outgoing relationship.

#### Configure S3 Bucket Object

AWS S3 bucket objects represent files stored in AWS S3 buckets. In our blueprint, we will use a bucket object to represent the code of the Lambda function.

• Click on the S3 Bucket Object node and set the number of instances to 1.
• In the resource_config field, we configure the following parameters (see example below):
• Bucket: String. The name of the S3 bucket in which we will upload the object.
• Key: String. The key or name that the object will have in the bucket.
• ACL: String. The canned ACL to apply to the bucket. We will apply access control policies to IAM users directly, so we will not use this.
Bucket: cloudify-lambda-bucket
Key: LambdaCode.zip

• In the source_type field, we insert the source type of the object that needs to be upload to the S3. The object can come either from a remote URL, a local URL within the blueprint or it can be a sequence of bytes. These bytes should be specified inside resource_config. In our case, we set this field to local
• We need to provide the code to the Lambda. Therefore, we add the zip archive containing the code of the Lambda in the Resources folder (top-left panel) of the blueprint. Then, insert in the path field the path (i.e. name) of the archive.

Note: you have to omit "./Resources/" from the path of the file.

• We should indicate now that the S3 bucket object should be uploaded after the S3 bucket has been created. To do so, we specify a relationship: draw a line from the S3 object to the bucket. Then, click on the new connection and change the connection type to cloudify.relationships.depends_on. This relationship describes a node that depends on another node. For example, the creation of a new subnet depends on the creation of a new network. Usually, we specify this relationship when a certain node should be created before or after another for the sake of ordering

Note: do now write or copy-paste the relationship type, since it seems that the Cloudify Composer does not accept it. Instead, browse the relationship type among the available ones.

#### Configure IAM Role

An IAM role is an IAM identity with specific permissions. An IAM role is similar to an IAM user, in that it is an AWS identity with permission policies that determine what the identity can and cannot do in AWS. However, instead of being uniquely associated with one person, a role is intended to be assumable by anyone who needs it, like a Lambda function.

• Click on the IAM Role node and set the number of instances to 1.
• In resource_config of the IAM role, we configure the following parameters (see example below):
• AssumeRolePolicyDocument: String. The trust relationship policy document that grants an entity (e.g., the Lambda) permission to assume the role.
• RoleName: String. The name of the role to create.
• Path: String. The path to the role.
RoleName: cloudifyRoleForLambda
Path: !!str /
AssumeRolePolicyDocument:
Version: !!str 2012-10-17
Statement:
- Effect: Allow
Principal:
Service: !!str lambda.amazonaws.com
Action: !!str sts:AssumeRole

• The IAM role node is connected to three of four IAM policy nodes. Therefore, draw a line from the IAM role to three policy nodes. Then, click on each of the new connections and change the connection type from cloudify.relationships.connected_to to cloudify.relationships.aws.iam.role.connected_to.

#### Configure Three IAM Policies

A policy is an object in AWS that, when associated with an identity or resource, defines its permissions. AWS evaluates these policies when a principal entity (user or role) makes a request. Permissions in the policies determine whether the request is allowed or denied. Most policies are stored in AWS as JSON documents. AWS supports six types of policies: identity-based policies, resource-based policies, permissions boundaries, Organizations SCPs, ACLs, and session policies.

We attach three (the ones you just associated with the IAM Role) of the four policies to the IAM Role. AWS already offers pre-defined policies collecting a set of permissions commonly distributed. In particular, we use the AmazonRDSFullAccess, AmazonS3FullAccess and AmazonVPCFullAccess policies.

• Click on three IAM Policy node and set the number of instances to 1.
• Since we are not creating a new AWS resource but instead using an already defined one, set the use_external_resource field to true
• In the resource_id, we need to specify the ARN of the policy. They are, respectively, arn:aws:iam::aws:policy/AmazonRDSFullAccess, arn:aws:iam::aws:policy/AmazonS3FullAccess and arn:aws:iam::aws:policy/AmazonVPCFullAccess.
• We need also to feed the ARN of the policies for the relationship with the IAM Role. Cloudify expects the ARN to be in the runtime properties of the IAM policies. Therefore, set the aws_resource_arn property in the cloudify.interfaces.lifecycle.create runtime properties of each of the IAM Policy nodes.
aws_resource_arn: 'arn:aws:iam::aws:policy/AmazonRDSFullAccess'

• In the resource_config of the IAM policy, we configure the following parameters (see example below). Actually, the parameters are ignored since we use an external resource. However, Cloudify still requires them
• PolicyName String. The friendly name of the policy.
• Description: String. A friendly description of the policy.
• Path: String. The path to the policy.
• PolicyDocument: String. The policy document.
PolicyName: CloudifyAmazonRDSFullAccessPolicy
Description: >-
AmazonRDSFullAccess policy
Path: !!str /
PolicyDocument:
Version: '2012-10-17'
Statement: []

• The IAM policy node has not outgoing relationships

#### Configure Lambda Function

AWS Lambda is a compute service that lets you run code without managing servers. AWS Lambda executes Python, Java, Go, Ruby, .NET or JS code only when needed and scales automatically. All you need to do is supply your code in one of the languages that AWS Lambda supports.

• Click on the Lambda node and set the number of instances to 1.
• In the Resource_config field, we configure the following parameters (see example below):
• FunctionName: String. The name of the Lambda function.
• Runtime: String. The runtime version for the function, i.e. the chosen programming language.
• Handler: String. The name of the method within your code that Lambda calls to execute your function. Usually is the path (i.e. package) dot the file name.
• Code: String. The code archive for the function. You can either provide the zip archive yourself or link an archive from an S3 bucket that has to be in the same region as the Lambda. In the first case, remember to add the zip archive in the Resources section (top-left panel) of your blueprint and then specify the ZipFile parameter. In the second case, specify the S3Bucket and S3Key parameters.
FunctionName: CloudifyLambdaFunction
Runtime: java8
Handler: package.className
Code:
ZipFile: Resources/LambdaCode.zip # only if you provide the code (not our case)
S3Bucket: cloudify-lambda-bucket # if the code is on an S3 bucket
S3Key: LambdaCode.zip      # if the code is on an S3 bucket
kwargs:
MemorySize: 128

• Lambda functions can stay in VPC (see Cloudify Manager Premium AWS Lambda Hello World) and must have an execution IAM role. Since our RDS instance will be publicly accessible, we do not need a VPC for the Lambda. Therefore, the Lambda does not need outgoing relationships. Actually, we should indicate now that the Lambda function should be instantiated after the upload of the S3 bucket object. As we did before, draw a line from the Lambda to the S3 object and change the connection type to cloudify.relationships.depends_on.

#### Configure User IAM Policy

We configure one IAM policy for the IAM user to allow him to invoke the Lambda function and provide limited access to the S3 buckets.

• Click on the AWS IAM Policy node and set the number of instances to 1.
• In the resource_config of the policy, we configure the following parameters (see example below).
• Description: String. A friendly description of the policy.
• PolicyName: String. The friendly name of the policy.
• Path: String. The path to the policy. For more information about paths, see the IAM Identifiers guide.
• PolicyDocument: String. The policy document.
PolicyName: cloudify-lambda-and-bucket-policy
Description: >-
Path: !!str /
PolicyDocument:
Version: '2012-10-17'
Statement:
- Sid: VisualEditor0
Effect: Allow
Action:
- s3:GetObject
- lambda:InvokeFunction
Resource:
- arn:aws:lambda:eu-central-1:117968532143:function:CloudifyLambdaFunction
- Sid: VisualEditor1
Effect: Allow
Action: s3:PutObject

• The user IAM Policy has not outgoing relationships

#### Configure IAM User

An AWS Identity and Access Management (IAM) user is an entity that you create in AWS to represent the person or application that uses it to interact with AWS. A user in AWS consists of a name and credentials. We configure a user that can invoke the Lambda function and access the S3 buckets.

• Click on the AWS IAM User node and set the number of instances to 1.
• In the resource_config field, we configure the following parameters (see example below).
• UserName: String. The name of the IAM user that the new key will belong to.
• Path: String. The path to the user. For more information about paths, see the IAM Identifiers guide.
• PermissionsBoundary: String. The ARN of the policy that is used to set the permissions boundary for the user. We will not use this
• Tags: List. A list of tags that you want to attach to the newly created user. Each tag consists of a key name and an associated value. For more information about tagging, see the Tagging IAM Identities guide.
UserName: CloudifyUser
Path: /

• We have to configure the relationships with the IAM policy. Draw a line from the IAM User to the Policy. Then, click on the connection and change the type from cloudify.relationships.connected_to to cloudify.relationships.aws.iam.user.connected_to

#### Configure RDS Instance

Amazon Relational Database Service (Amazon RDS) allows you to set up, operate, and scale a relational database in the cloud choosing among different RDBMS (e.g., MariaDB, MySQL, PostgreSQL).

• Click on the AWS RDS Instance node and set the number of instances to 1.
• In the resource_config field, add an ID like cloudify-rds-instance-id
• In the Interfaces => create => resource_config field, we configure the following parameters (see example below):
• DBInstanceClass: String. The compute and memory capacity of the DB instance.
• Engine: String. The name of the database engine to be used for this instance (e.g., Aurora, MariaDB, MySQL).
• EngineVersion*: String. The version number of the database engine to use.
• AvailabilityZone: String. The Availability Zone (AZ) where the database will be created.
• StorageType: String. Specifies the storage type to be associated with the DB instance (gp2 is General Purpose Storage).
• AllocatedStorage: Integer. The amount of storage (in gibibytes) to allocate for the DB instance. (for gp2 it be an integer from 20 to 65536)
• DBName: String. The name of the database to create when the DB instance is created.

Note: The meaning of this parameter differs according to the database engine you use.

• MasterUsername: String. The name for the master user.
• MasterUserPassword: String. The password for the master user. The password can include any printable ASCII character except “/”, “"”, or “@”.
• PubliclyAccessible: Boolean. A value that indicates whether the DB instance is publicly accessible. When the DB instance is publicly accessible, it is an Internet-facing instance with a publicly resolvable DNS name, which resolves to a public IP address. When the DB instance is not publicly accessible, it is an internal instance with a DNS name that resolves to a private IP address. PubliclyAccessible only applies to DB instances in a VPC. The DB instance must be part of a public subnet and PubliclyAccessible must be enabled for it to be publicly accessible.
DBInstanceClass: db.t2.micro
Engine: mysql
EngineVersion: 8.0.16
AvailabilityZone: eu-central-1a
StorageType: gp2
AllocatedStorage: 20
DBName: devdb
PubliclyAccessible: !!bool True
VpcSecurityGroupIds:
- get_property:
- CloudifySecurityGroup
- resource_id

• The last step is to configure the relationships with the other nodes. The RDS node should be connected to the subnet, parameter and security group. To create these relationships, draw a line from the RDS node to each of such nodes. Then, click on all the connections and change the connection type from cloudify.relationships.connected_to to cloudify.relationships.aws.rds.instance.connected_to

#### Configure Parameter Group

You manage your DB engine configuration by associating your DB instances with parameter groups. Amazon RDS defines parameter groups with default settings that apply to newly created DB instances. You can define your own parameter groups with customized settings. For instance, let’s see how we can enable the execution of triggers and stored functions in our database.

• Click on the Parameter Group node and set the number of instances to 1.
• In the Interfaces => configure => resource_config field, we can configure several parameters (see example below) As an example, we configure:
• log_bin_trust_function_creators: Basestring. Enforces restrictions on stored functions / triggers - logging for replication. Set to 1 to allow them, 0 to forbid their execution.
Parameters:
- ParameterName: log_bin_trust_function_creators
ParameterValue: '1'
ApplyMethod: immediate

• In resource_config of the Parameter Group, we configure the following parameters (see example below):
• kwargs
• DBParameterGroupName: String. The name of the DB parameter group.
• DBParameterGroupFamily: String. It provides the name of the DB parameter group family that this DB cluster parameter group is compatible with.
• Description: String. Provides the customer-specified description for this DB cluster parameter group.
kwargs:
DBParameterGroupName: cloudify-parameter-group
DBParameterGroupFamily: mysql8.0
Description: MySQL8.0 Parameter Group for Cloudify

• The Parameter Group node does not have outgoing relationships

#### Configure Security Group

A security group acts as a virtual firewall for the RDS instance to control inbound and outbound traffic. Security groups act at the instance level, not the subnet level. If you don’t specify a particular group at launch time, the instance is automatically assigned to the default security group for the VPC.

• Click on the Security Group node and set the number of instances to 1.
• In resource_config of the security group, we configure the following parameters (see example below):
• Description: String. Some arbitrary description.
• GroupName: String. A name for the group.
• VpcId: String. The ID of the VPC to create the group in.
GroupName: Cloudify RDS Security Group
Description: Cloudify RDS Security Group
VpcId:
get_attribute:
- CloudifyVpc
- aws_resource_id

• Finally, we have to configure the relationships with the other nodes. The security group node depends on the VPC node. Therefore, draw a line from the security group to the VPC. Then, click on the new connection and change the connection type from cloudify.relationships.connected_to to cloudify.relationships.depends_on

#### Configure Security Group Rule Ingress

For each security group, you add rules that control the inbound traffic to instances, and a separate set of rules that control the outbound traffic. By default, AWS blocks all incoming traffic and allow all outgoing traffic. Therefore, we just have to modify the inbound traffic.

• Click on the Security Group Rule Ingress node and set the number of instances to 1.
• Click on the Security Group Rule Ingress node and set the number of instances to 1.
• In resource_config of the security group, we configure the following parameters (see example below):
• IpPermissions: List. A list of IP Permissions.
IpPermissions:
- IpProtocol: "-1"
FromPort: -1
ToPort: -1
IpRanges:
- CidrIp: 0.0.0.0/0
UserIdGroupPairs: [ { GroupId: { get_attribute: [ CloudifySecurityGroup, aws_resource_id ] } } ]

• Finally, we have to configure the relationships with the other nodes. The security group rule ingress node is contained in a security group node. Therefore, drag the security group rule ingress node inside the security group node.

#### Configure Subnet Group

Unless you are working with a legacy DB instance, your DB instance is in a VPC. To configure internet accessibility, we need to associate the RDS instance with a subnet group.

• Click on the Subnet Group node and set the number of instances to 1.
• In the resource_config field, add an ID like cloudify-subnet-group-id
• In the resource_config field, we can configure a description (see example below).
kwargs:
DBSubnetGroupDescription: MySQL8.0 Cloudify Subnet Group

• Intuitively, the subnet group is connected to the two subnets we instantiated. To create these relationships, draw a line from the subnet group node to both the cloudify.nodes.aws.ec2.Subnet nodes. Then, click on all the connections and change the connection type from cloudify.relationships.connected_to to cloudify.relationships.aws.rds.subnet_group.connected_to

#### Configure Subnets

In a VPC, we can configure one or more subnets. When you create a subnet, you specify the CIDR block for the subnet, which is a subset of the VPC CIDR block.

• Click on each of the Subnet nodes and set the number of instances to 1.
• In resource_config of each of the subnets, we configure the following parameters (see examples below):
• AvailabilityZone: String. The Availability Zone for the subnet. For RDS instances, AWS requires at least two subnets with different availability zones.
• CidrBlock: String. The IPv4 network range for the subnet, in CIDR notation.
• VpcId: String. The ID of the VPC, if any. Alternately use a relationship, as we do.
CidrBlock: 10.0.0.0/24
AvailabilityZone: {concat: [{get_input: aws_region_name}, a]}

CidrBlock: 10.0.1.0/24
AvailabilityZone: {concat: [{get_input: aws_region_name}, b]}

• Finally, we have to configure the relationships with the other nodes. The Subnet node is created after the VPC node. Therefore, draw a line from each of the subnets to:
• cloudify.nodes.aws.ec2.Vpc. Then, click on both of the two new connections and change the connection type from cloudify.relationships.connected_to to cloudify.relationships.depends_on

#### Configure RouteTable

A route table contains a set of rules, called routes, that are used to determine where network traffic from your subnet is directed.

• Click on the RouteTable node and set the number of instances to 1.
• The route table does not need any configuration parameters.
• The RouteTable node should be connected to the subnet and contained in the VPC. Therefore, draw a line from the RouteTable node to the cloudify.nodes.aws.ec2.Subnet nodes and insert it in the VPC to obtain a cloudify.relationships.contained_in relationship.

### Configure Route

Routes determine where network traffic from your subnet is directed. To make our DB publicly accessible, we need to allow external traffic.

• Click on the Route node and set the number of instances to 1.
• In resource_config of the Route, we configure the following parameters (see example below):
• DestinationCidrBlock: String. The IPv4 network range to be redirected, in CIDR notation.
kwargs:
DestinationCidrBlock: 0.0.0.0/0

• The Route node should be connected to the internet gateway to redirect there the traffic and contained in the RouteTable. Therefore, draw a line from the Route node to the gateway and insert it in the RouteTable to obtain a cloudify.relationships.contained_in relationship.

Bug The first subnet was associated with this route table, while the second was associated with a route table which was created out of the blue. I don’t know why. To solve the problem, I created two distinct route tables and assigned each of them to each subnet.

#### Configure VPC

A virtual private cloud (VPC) is a virtual network dedicated to your AWS account. It is logically isolated from other virtual networks in the AWS Cloud. You can launch your AWS resources, such as Amazon EC2 instances, into your VPC. When you create a VPC, you must specify a range of IPv4 addresses for the VPC in the form of a Classless Inter-Domain Routing (CIDR) block; for example, 10.0.0.0/16. This is the primary CIDR block for your VPC.

• Click on the VPC node and set the number of instances to 1.
• In resource_config of the VPC, we configure the following parameters (see example below):
• CidrBlock: String. The IPv4 network range for the VPC, in CIDR notation.
   CidrBlock: 10.0.0.0/16

• To enable the DNS Hostnames resolution, in *Interfaces => configure create a new input of name modify_vpc_attribute_args. There, we configure the following parameters (see example below):
• EnableDnsSupport. Boolean. Indicates whether the DNS resolution is supported for the VPC. If enabled, queries to the Amazon-provided DNS server at the 169.254.169.253 IP address, or the reserved IP address at the base of the VPC network range “plus two” succeed. If disabled, the Amazon-provided DNS service in the VPC that resolves public DNS hostnames to IP addresses is not enabled.
• EnableDnsHostnames. Boolean. Indicates whether the instances launched in the VPC get DNS hostnames. If enabled, instances in the VPC get DNS hostnames; otherwise, they do not.
EnableDnsHostnames:
Value: true

• Our VPC node does not have outgoing relationships

#### Configure Internet Gateway

An internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in your VPC and the internet. It, therefore, imposes no availability risks or bandwidth constraints on your network traffic. An internet gateway serves two purposes: to provide a target in your VPC route tables for internet-routable traffic and to perform network address translation (NAT) for instances that have been assigned public IPv4 addresses. An internet gateway supports IPv4 and IPv6 traffic.

• Click on the Internet Gateway node and set the number of instances to 1.
• Configure the tags to describe the gateway
- Key: Name
Value: CloudifyInternetGateway

• Configure the relationships with the VPC by drawing a line from the Internet Gateway node to the cloudify.nodes.aws.ec2.Vpc node. Then, click on the connection and change the connection type from cloudify.relationships.connected_to to cloudify.relationships.depends_on.

Finally, you can download the .zip archive containing the blueprint.

### Deploy

• Running the example
1. Go to “Local Blueprints” menu and click the Upload button, providing the blueprint you just created.

Note: you may have to wait for one minute and a half and then see an An unexpected error has occurred: ESOCKETTIMEDOUT error. Unfortunately, this behaviour happens even though the blueprint was successfully uploaded. After the error, wait for another thirty seconds for the new blueprint to appear in the list of all blueprints.

3. In the blueprints table, click on the name of the blueprint.

4. Click on the Create Deployment button.

5. Type the name of the blueprint in the deployment name field.

6. Type the region (e.g, “eu-central-1") in the aws_region_name field.

7. Click Deploy.

8. Scroll down to the deployments table, click the hamburger icon on the right and select Install and then Execute. The install workflow will take a few minutes to complete.

### Uninstall

• Tear down the deployment

1. Go to the “Deployments” tab.
2. Click on the hamburger icon and choose Uninstall then Execute. This will take a few minutes to complete
• To remove the docker with the Cloudify Manager, run

sudo docker rm -f cfy_manager_local

cfy_manager_local


### Conclusion

In this tutorial, we developed a quite complex cloud application using Cloudify. We configured and deployed several AWS services, like RDS, S3, Lambda and IAM entities. As you may have noticed, the blueprint configuration effort is not negligible. Indeed, Cloudify better assists in the orchestration of IaaS cloud application. In next tutorials, we will see how to configure and orchestrate containers on a local installation of OpenStack.