How to Setup EventStoreDB on AWS EC2 with Pulumi IaC

Introduction 

EventStoreDB (ESDB) is an industrial-strength database technology used as the central data store for event-sourced systems. It is available open-source to run locally on most platforms or as SaaS through Event Store Cloud. Currently their SaaS version is under development, so if you want to run it on your cloud, you will have to set it up.

One option is docker, since ESDB is available as docker image on Docker Hub. However, as Greg Young explains in this issue, it might be not the best option when it comes to performance, due to the extra virtualization layer. For the same reason, we don't usually use docker images for other database engines like SQL Server, MongoDB, etc.

Then, if you want to run ESDB on cloud, in particular AWS cloud, EC2 + EBS is the most reasonable option. 

In this post, I would like to share how you can configure ESDB on EC2+EBS easily with Pulumi, which will allow you to automate this process following the best practices.

Creating EC2 with Pulumi

If you are not familiar with Pulumi, I recommend to start having a look at this documentation, where it is explained how to setup a new project for AWS.

Creating a new EC2 in Pulumi is really straightforward. You just need to define your instance like this:


In that code, you can specify the following parameters:

  • instanceType: type of EC2 instance that you want to use
  • keyName: key pair that you will use to connect with SSH
  • ami: AMI you want to use. For example AMI "ami-06fd8a495a537da8b" is to use Ubuntu Server 20.04 LTS
  • vpcSecurityGroupIds: the security group you are going to use. Here for ESDB you might need to enable ports 1113 and 2113. For SSH you will need to enable port 22.
  • userData: this is the script to setup ESDB. We will see this more in detail later.
  • tags: your AWS for this resource. It is highly recommended to tag all your AWS resources with meaningful tags.

Setup EventStoreDB on EC2

First of all, lets have a look at ESDB config file.

# Paths
Db: /data/eventstore/db
Index: /data/eventstore/index
Log: /data/eventstore/log

# Certificates configuration
CertificateFile: /data/certs/node1/node.crt
CertificatePrivateKeyFile: /data/certs/node1/node.key
TrustedRootCertificatesPath: /data/certs/ca

# Network configuration
HttpPort: 2113
ExtTcpPort: 1113
EnableExternalTcp: true
EnableAtomPubOverHTTP: true

# Cluster config
ClusterSize: 1

# Projections configuration
RunProjections: System

If you want to know more about ESDB, I recommend you to visit this guide, which is quite good. Here I normally configure ESDB data and certificates in folder /data, which is in an independent EBS, but I will skip that part in this post, in order to simplify.

Now, in our EC2 userData we need to follow these steps:

1) Install EventStore

In this step we install EventStore repository and we install the latest version, which currently is version 20.6.1.

    # Install EventStore
    curl -s https://packagecloud.io/install/repositories/EventStore/EventStore-OSS/script.deb.sh | sudo bash
    sudo apt-get install eventstore-oss=20.6.1-2

  2) Update EventStore Config

 In this step, we overwrite eventstore.conf file with the configuration we saw previously.

    # Update EventStore Config
    sudo echo "${getEventStoreConfig()}" >| /etc/eventstore/eventstore.conf

3) Create /data folder and add permissions to eventstore

Then, we create folder /data and we assign permissions to eventstore, so the service will be able to own that folder.

    # Create /data folder and add permissions to eventstore
    cd /
    mkdir data
    sudo chown eventstore data
    sudo chgrp eventstore data

4) Create certificates

 In this step, we create CA and Node certificates for SSL connection with es-gencert-cli, which is a software created by EventStore to simplify this process. You could also use OpenSSL if you wish. 

# Create certificates
    cd /data
    sudo mkdir certs
    cd certs/
    sudo wget -c https://github.com/EventStore/es-gencert-cli/releases/download/1.0.2/es-gencert-cli_1.0.2_Linux-x86_64.tar.gz
    sudo tar -xzvf es-gencert-cli_1.0.2_Linux-x86_64.tar.gz
    sudo ./es-gencert-cli create-ca
    sudo ./es-gencert-cli create-node -out ./node1 --dns-names *.example.com
    find . -type f  -name '*.crt' -o -name '*.key' -print0 | sudo xargs -0 chmod 666

5) Run EventStore

 Finally we enable and start eventstore service.

    # Run EventStore
    sudo systemctl enable eventstore
    sudo systemctl start eventstore

You can find the full example source code on this GitHub repository: https://github.com/albertocorrales/eventstoredb-aws-ec2

Conclusions

In this post, we have seen how we can setup EventStoreDB on AWS EC2 with Pulumi, as IaC framework. If you want to know more about these technologies, I suggest to visit EventStore documentation and Pulumi documentation.



Comments

Post a Comment

Popular posts from this blog

Building Micro-Frontends with Single-Spa

Image
Introduction Micro-services architecture is becoming trendy since it is a suitable option many projects, specially those that are complex and we want to be able spilt our business logic into different services that can be built and deliver independently. In spite of the fact that micro-services architectures are more complex, they increase the resilience of your system and increase the time-to-market delivery, which can be crucial for many companies. But it is not all about services and backend. With the introduction of single page applications (SPAs), we can build fully-functional user interfaces that can work independently in the browser. And here we should ask ourselves the same questions as for backend applications: Is my SPA becoming a huge monolith? Would I like to deliver functional modules independently? If you using micro-services, is there a relation between my services and your UI? Do we want to have functional teams that can deliver new features (frontend+backend) independe...
Read more

AWS assuming role + MFA with pulumi (and other applications)

Image
Introduction In AWS, managing your IAM in a centralized account is a good practice, so then you can assume a role to access to other accounts without need of duplicating users. You can find more info about this approach in this article . In addition, for security reasons, it is highly recommended to enable MFA (Multi-factor Authentication), so you need a device to generate a temporary code to get access. In the image below it is showed how assume role with MFA works, but you also have an article about MFA in this link . This approach is really useful and most of the configuration can be automated with pulumi (or other providers like terraform or CloudFormation). And once it is configured, you can switch your role and get access to other accounts with a few clicks.  However, when it comes to external software such us pulumi, getting access assuming a role and using MFA might not be too straightforward. In this article we will see how this can be configured and how we can easily refr...
Read more

Managing snapshots for Amazon ElasticSearch with Dotnet Core Lambdas

Image
Introduction It is awesome to have some useful services like ElasticSearch managed by AWS , so you don't have to care about patching, monitoring, etc.  When it comes to backup management for indexes, Amazon ElasticSearch includes automated snapshots. Automated snapshots are only for cluster recovery. You can use them to restore your domain in the event of red cluster status or other data loss. Amazon ES stores automated snapshots in a preconfigured Amazon S3 bucket at no additional charge. In particular, if you are using a higher version than 5.3, Amazon ElasticSearch takes hourly automated snapshots and retains up to 336 of them for 14 days. However, this policy might not be enough if you need to keep your snapshots for a longer 15 days, or you need to take a snapshot to create a new cluster in another region, subnet, etc. If you have different requirements to manage your snapshots, you will have to work with manual snapshots. For this part, AWS does not offer a good managed exper...
Read more