Infrastructure as Code with Pulumi, AWS and Azure DevOps

 

Introduction

I still remember when I had to ask IT to have a new server. After that, I had to wait and pray for having the server with the corresponding permissions, software, etc. That is that Martin Fowler describes in his article about IaC as the Iron Age. Having lived that experience makes me feel a little old, but at the same time I feel lucky for having experienced the evolution, specially with the introduction of the cloud, or the Cloud Age

Since the cloud showed up, the dependencies between developers and IT have been reduced drastically, until the point that most of the work related to infrastructure and operations can be handled by developers.

Creating infrastructure on the cloud is really easy, compared to having to setup your on-premise machines. Most providers offer their UI, where you can select what you want to create and which parameters you want to configure. I guess this is how we all stated the first time we used the cloud. However, when your infrastructure grows and you need to maintain multiple environments, you have probably seen how tedious can be the manual approach to create and maintain cloud resources. 


Infrastructure-as-Code to the rescue

Imagine each member of the team creating resources manually as they are needed, or trying to replicate an environment. In summary...it can be a big mess.

Fortunately, Infrastructure-as-Code (IaC) came to the rescue. IaC is the infrastructure management using code and the same versioning as DevOps teams use for source code. Following the same principle that source code will generate always the same binaries, IaC will generates always the same infrastructure. IaC can be extremely powerful if we use it in conjunction with continuous delivery.

In fact, the value of IaC can be broken down into three measurable categories: cost (reduction), speed (faster execution) and risk (remove errors and security violations).

But there was a problem. To implement Infrastructure as Code....there was no code! There were templates. In particular YAML templates. YAML templates are good for small config files, because they are quite readable for small files with few levels of tabulation spaces. But infrastructure can require a significant amount of YAML that can be really hard to read. And that is not the worst part, the worst part is having to write that code.

I know YAML is trendy these days, but in my experience, it is really hard to create and maintain. I would even prefer JSON, because at least you won't have tabulation space errors, which are quite frequent in YAML.

Some of the most popular IaC products based on templates are Microsoft ARM, AWS CloudFormation and Terraform.

However, there are other more recent products that are trying to implement Infrastructure-as-Code with real coding languages. The most popular is Pulumi, and that is the technology I've been using to build my microservices architecture.

In addition, if you work with AWS, you have the possibility to use AWS CDK https://aws.amazon.com/cdk/. AWS CDK also allow you to work with nice programming languages like typescript or C#, but it will generate an AWS CloudFront template from your code. However, it is good to see that main cloud vendors are starting to develop real infrastructure as code tools, because that means they are being aware of the gap they have in their traditional IaC tools based on YAML.

Building Microservice Architectures with Pulumi on AWS

Pulumi is an open source project that supports multiple programming languages, such as: .Net Core (C#, VB, F#), Go, Node.js (JavaScript, TypeScript) and Python. In contrast to other template language based alternatives, Pulumi is not free for commercial purposes. In spite of that fact, I think it is a good product and paying for it is worthy, but bear that in mind.

If you are a developer, Pulumi will be really straightforward for you. You just need to choose your favourite language and start writing resources for your infrastructure as if they were classes in a normal program.

In this post, I'm not going to write any examples, because there is a great repository with many examples (https://github.com/pulumi/examples) and their documentation is great. However, I would like to mention some good practices that I found useful while I was developing our microservices architecture with Pulumi.

Use small stacks

In our microservices architecture we have multiple services per domain. We could have built one stack per domain or even one stack for the whole infrastructure. But having smaller stacks will be easier to maintain and operate, in case you have to destroy one of those stacks. For me the ideal unity for a stack is something that makes sense as logic unity. For example, if your microservice uses some Security Groups, a Load balancer and one ECS service, those three would belong to the same stack.

In order to organise better your stack, you can also user classes or creating cloud components. Then, you can decide whether you want to distribute them as an NPM/NuGet package or not.

Use dependent stacks

Pulumi has a cool feature called Inter-Stack Dependencies. This feature makes easier to share the resources you created in one stack with other dependent stacks. For example, you can share resources like your VPC in the stack "my-example/infra/dev":

 

Then you could use that vpcId you created in other stacks in this way:

I wouldn't overuse this feature if you don't want to end up with many dependencies, since it might be difficult to track and maintain. But it is a feature nice to have into consideration.

Use tags

 Tags are extremely useful to categorise resources on AWS, but it can also be really handy to organise your stacks, depending on your domains, team, environments, etc.

 In order to create a tag, you just need to write:

Use Pulumi secrets

Probably, you will have in our stack some variables that are secrets and you don't want to push into your repository as plain text. For those, Pulumi includes a build-in secrets management system. 

In order to work with secrets, you just have to use the flag --secret when you add new configuration.

That command will generate an encrypted string for you. Also, you will see it with asterisks in your pulumi management page and every time your program print it.

Pulumi also supports external encryption providers for the main cloud vendors: https://www.pulumi.com/docs/intro/concepts/config/#available-encryption-providers

Use pulumi-awsx package

If you are working with AWS, Pulumi provides two packages: pulumi/pulumi-aws and pulumi/pulumi-awsx. Pulumi-aws package contains all AWS components in a similar way they exist on AWS. However, AWSX is an upper layer that simplifies quite a lot the work, providing more high-level components, with many options configured out-of-the-box for you.

This package can be really convenient to save time and complexity, if you don't need to define all the details for every resource.

However, this package is not still available for all languages and that was one of the main reason why we decided to use Typescript over C#.

CI/CD with Azure DevOps and Pulumi

Once you have your infrastructure written with Pulumi, you will start to see the benefits. For example, in order to create a new environment, you just need to write your config for those stacks and execute "pulumi up". As simple as that. And with the guarantee that your infrastructure is well configured and no human errors were made.

But you will have the full potential of IaC when you combine it with Continuous Integration / Continuous Deployment (CI/CD). In that moment, creating new infrastructure will mean developing the new pieces in your Pulumi code, creating a PR and those changes will be made for you in your Cloud provider. In addition, the delivery of those changes will be promoted to your environments in the same way you normally deliver your application.

In order to offer integration with CI/CD, Pulumi supports most of the well-known vendors: https://www.pulumi.com/docs/guides/continuous-delivery/.

In my particular case, I use Azure DevOps for CI/CD. In order to support Azure DevOps easily, there is a task that you can use in your pipeline: https://marketplace.visualstudio.com/items?itemName=pulumi.build-and-release-task

When you use Pulumi task on Azure DevOps, you will need to create a token in our Pulumi account. Then, you have to set that token as environment variable with name: PULUMI_ACCESS_TOKEN.

 If you are working with AWS, you will need to setup your AWS credentials too. I normally do that step with bash task like this, where AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are my AWS credentials that I have as encrypted environment variables:

Finally, I can use my Pulumi task to deploy my IaC

 Conclusions

In this article we have seen why Pulumi can be great tool to build your IaC and some good practices to improve your IaC when you are working with this technology. In my opinion, Pulumi is a great tool that makes really simple writing infrastructure as code, compared to template language approaches. It supports some trendy programming languages and it is multi-cloud.



Comments

Post a Comment

Popular posts from this blog

Building Micro-Frontends with Single-Spa

Image
Introduction Micro-services architecture is becoming trendy since it is a suitable option many projects, specially those that are complex and we want to be able spilt our business logic into different services that can be built and deliver independently. In spite of the fact that micro-services architectures are more complex, they increase the resilience of your system and increase the time-to-market delivery, which can be crucial for many companies. But it is not all about services and backend. With the introduction of single page applications (SPAs), we can build fully-functional user interfaces that can work independently in the browser. And here we should ask ourselves the same questions as for backend applications: Is my SPA becoming a huge monolith? Would I like to deliver functional modules independently? If you using micro-services, is there a relation between my services and your UI? Do we want to have functional teams that can deliver new features (frontend+backend) independe...
Read more

AWS assuming role + MFA with pulumi (and other applications)

Image
Introduction In AWS, managing your IAM in a centralized account is a good practice, so then you can assume a role to access to other accounts without need of duplicating users. You can find more info about this approach in this article . In addition, for security reasons, it is highly recommended to enable MFA (Multi-factor Authentication), so you need a device to generate a temporary code to get access. In the image below it is showed how assume role with MFA works, but you also have an article about MFA in this link . This approach is really useful and most of the configuration can be automated with pulumi (or other providers like terraform or CloudFormation). And once it is configured, you can switch your role and get access to other accounts with a few clicks.  However, when it comes to external software such us pulumi, getting access assuming a role and using MFA might not be too straightforward. In this article we will see how this can be configured and how we can easily refr...
Read more

Managing snapshots for Amazon ElasticSearch with Dotnet Core Lambdas

Image
Introduction It is awesome to have some useful services like ElasticSearch managed by AWS , so you don't have to care about patching, monitoring, etc.  When it comes to backup management for indexes, Amazon ElasticSearch includes automated snapshots. Automated snapshots are only for cluster recovery. You can use them to restore your domain in the event of red cluster status or other data loss. Amazon ES stores automated snapshots in a preconfigured Amazon S3 bucket at no additional charge. In particular, if you are using a higher version than 5.3, Amazon ElasticSearch takes hourly automated snapshots and retains up to 336 of them for 14 days. However, this policy might not be enough if you need to keep your snapshots for a longer 15 days, or you need to take a snapshot to create a new cluster in another region, subnet, etc. If you have different requirements to manage your snapshots, you will have to work with manual snapshots. For this part, AWS does not offer a good managed exper...
Read more