AWS

Getting started with AWS part two


Last year our colleague Christoffer Pozeus shared how to get started with AWS from a technical perspective covering the different alternatives such as AWS Landing Zone, AWS Control Tower and AWS Deployment framework (ADF). If you haven’t read his blog make sure to give it a read!

In this blog we would like to share some of our experience with some of our recent customers and also what we are seeing in the market and get up and running as quickly as possible with your new AWS environment.

What we are seeing are two main directions customers are taking, first group of customers are moving to the cloud, we are going to migrate from our data centers and go all in AWS. The second group has a lot on-prem, doing a huge lift & shift will not be cost effective and time efficient so they start small and grow from there.

If you are a company in situation one or two then you will need to start with building your foundation and AWS in this example there are many ways of doing this. What we are seeing is that many do this themselves and if it’s to build competence within AWS then that is great, but our recommendation is view it as just this, to build competence and not our future foundation that we will over time will move business critical applications which requires a high level of security, cost control and scalability.

Depending on your situation if you have an in-house team who you want to build this competence within or if you want someone else to manage this for you make sure that you receive some help. All from guiding principles, lessons learnt to recommendations. This will enable you and your team and set you up for success and of course avoiding costly troubles later down the road.

We have done this journey many times over the years with all different kind of customers in different shapes and sizes and the question we always get is how long will this take? The boring answer is of course it depends, but the realistic answer is from a few days to a few weeks.

We strongly believe that the best out-comes are when we do things together understanding you as a customer where you are, where you wantto be and how we can support and share our experience and learn together.

To summarize, make sure to set a solid foundation as early as possible which will be sustainable over time!

AWS

The introduction of Site Reliability Engineering (SRE)

The introduction of Site Reliability Engineering (SRE) teams in the structure of an organization, is becoming more and more popular in the IT industry and in the DevOps domains.
Let’s discover in this article the reason for SRE popularity and what are the differences and the common points between DevOps and SRE. 


In the last two decades we have witnessed a huge transformation in the way of building and delivering software. The Agile culture first and the DevOps revolution later, have transformed the structure of the tech organizations and they can be seen as a de facto standard in the IT industry.

As everyone knows, IT is a constantly evolving sector and recently we are seeing an increasing popularity of Site Reliability Engineering discipline, especially in the DevOps domains. But what is SRE? And what are the differences and the common points between SRE and DevOps? 

DevOps

DevOps culture has contributed to tearing down the wall between software development and the software operation, providing a set of ideas and practices that has led several benefits:

  • Better collaboration and communication between the parts;
  • Shorter release cycles;
  • Faster innovation;
  • Better systems reliability;
  • Reduced IT costs;
  • Higher software delivery performance.

Even though this sounds amazing, there are still quite a lot of companies that struggle with bringing the DevOps culture in their organization. The reason for this is that DevOps is an ideology and not a methodology or technology, which means it doesn’t say anything about how to successfully implement a good DevOps strategy. 

SRE 

Site Reliability Engineering (SRE) is a discipline that was born at Google in early 2000s to reduce the gap between software development and operations and that was completely independent by the DevOps movement. SRE uses software engineering approaches to solve operational problems.
SRE teams have main focus on:

  • Reliability;
  • Automation.

Let’s deepen these aspects.

Reliability

One of the main goals of the SRE is making and keeping systems up and running “no matter what”. In order to achieve this, it is important to keep in mind that failures and faults can happen. SRE discipline embraces them by focusing on:

  • observability;
  • system performance;
  • High availability (HA);
  • emergency response and disaster recovery; 
  • incidents management;
  • Learning from the past problems;
  • disaster mitigation and prevention.

Automation

Automating all the activities that are traditionally performed manually is another of the main goals of SRE.
Automation and software engineering are used to solve operational problems. 

Automation plays a fundamental role in SRE: it allows us to get rid of human errors present in the processes and the activities that regard the system. One could argue that automation introduces bugs in the system anyway and well, that is true but there is one big difference: one can test automated processes but cannot test processes that involve human activities. 

DevOps vs. SRE

As we have understood, both DevOps culture and SRE discipline aim to reduce the gap between software development and operations. Below we summarize them, describing their common goal first and where they differ the most. 

class SRE implements DevOps

As mentioned earlier DevOps doesn’t say anything about how to successfully bring the culture in the organization since it is an ideology. On the other hand, SRE can be seen as implementation of the DevOps philosophy. 
In fact, even though the origins of SRE are completely independent from DevOps, and the discipline provides additional practices that are not part of DevOps, SRE implements DevOps ideas.

Responsibility and Code ownership

SRE can be considered the next stage of DevOps because of the focus on code ownership: the SRE engineer accepts the responsibility of owning the code they develop, in production. A bit different from DevOps where the responsibilities are shared to achieve a shorter release cycle and to improve the collaborations.

Conclusions

The introduction of Site Reliability Engineering (SRE) teams in the structure of an organization, is becoming more and more popular in the IT industry and in the DevOps domains. 
The reason for its popularity can be found in the benefits that the discipline brings: 

  • Better collaboration and communication between the parts;
  • Shorter release cycles;
  • Faster innovation;
  • Better systems reliability;
  • Reduced IT costs;
  • Higher software delivery performance;
  • Reducing incidents in production;
  • Code ownership;
  • Automation of the processes;

As you could notice some of these benefits are exactly the same that you will experience bringing DevOps in your organization.

SRE can be considered an implementation of DevOps culture that has the goal of making and keeping services reliable.

AWS

Connect your Azure AD to AWS Single Sign-On (SSO)

In this blog post, I will provide a step-by-step guide for how to integrate your Azure AD with AWS Single Sign On (SSO). The integration will enable you to manage access to AWS accounts and applications centrally for single sign-on, and make use of automatic provisioning to reduce complexity when managing and using identities.

Organizations usually like to maintain a single identity across their range of applications and AWS cloud environments. Azure Active Directory (AD) is a common authentication method as Office 365 often is used among companies, and might be the hub of authentication as it often is integrated with other applications as well.

If you are using AWS Single Sign-On (SSO) you can leverage your existing identity infrastructure while enabling your users to access their AWS accounts and resources at one place. By connecting Azure AD with AWS SSO you can sign in to multiple AWS accounts and applications using your Azure AD identity with the possibility to enable automatic synchronization of Azure AD Users/Groups into AWS SSO.

This makes perfect sense and often improves your ability to further automate how you handle user-lifecycle and access to your AWS accounts as you might already have some identity manager connected to your HR system like Microsoft Identity Manager or Omada in place for example. You can also leverage your existing process for applying for access to different systems, ServiceNow or similar solution might already be connected to Azure AD in one way or another which then could be leveraged for applying for access to different AWS Accounts.
There are also other benefits such as levering your existing MFA solution if your organization has such a solution in place.

To the good stuff! I will in this blog-post demonstrate how you can connect your Azure AD to AWS SSO and take advantage of its capabilities.

Creating a new Azure Enterprise Application

Login to your Azure portal and open Azure Active Directory. Under the Manage section, click on Enterprise application.

Click New application and select a Non-gallery application, give your new application an appropriate name and click Add.

Once the Azure gods have created our new application, head into the Overview page and select Set up single sign-on and choose the SAML option.

Under section, SAML Signing Certificate click Download next to Federation Metadata XML.

Please keep this page open as we later need to upload the metadata XML file from AWS SSO.

Setup AWS SSO

Login to AWS management console and open AWS Single Sign-On, please ensure that you are in your preferred region. If you haven’t enabled AWS Single Sign-On already, you can enable it by clicking Enable AWS SSO as shown below.

Click Choose your identity source. You can also configure your custom User portal URL if you’d like but it is not required.

Select External identity provider. Upload the AD Federation Metadata XML file downloaded earlier inside the IdP SAML metadata section and download the AWS SAML metadata file.

In the Azure SAML page, click Upload metadata file and upload the AWS SSO SAML metadata file.

If you have configured a User portal URL earlier, you need to edit the Basic SAML Configuration section and match the Sign-on URL.

Setting up automatic provisioning

The connection between Azure AD and AWS SSO is now established, we can proceed to enable automatic provisioning to synchronise users/groups from Azure AD to AWS SSO.

Note that you can use Azure AD groups but not nested groups ie. groups that are into groups.

Head over to the Provisioning page and change the mode to Automatic. Please keep this page open as we will copy values from AWS SSO.

In the AWS SSO Settings page, click Enable automatic provisioning

Take note of both values given in the popup

In the Provisioning page in the Azure portal, expand the Admin Credentials section and insert the values from above. It is also recommended to add an email address for notification of failures.

SCIM endpoint > Tenant URL
Access token > Secret Token

Note that these tokens expire after 1 year and should be renewed for continuous connectivity.

Click Test Connection and it should result in a success message.

Expand the Mapping section and click Synchronize Azure Active Directory Users to customappsso

Which attributes you want to sync over depends on your setup, but default setups you can remove all attributes except:
userName
active
displayName
emails
name.givenName
name.familyName

You then create a new attribute mapping objectId with externalId.

Important to note is that you can modify the email attribute to use userPrincipalName over mail as not all users have Office365 licenses which leave that attribute null.

In the Provisioning page, you can now set the Status to On. It is recommended leaving Scope set to Sync only assigned users and groups.
Click Save, it should take about 5 minutes for it to start synchronizing.

Our AWS SSO and Azure AD connection is now fully set up, when you assign Azure Users/Groups to the enterprise app, they will then appear in AWS SSO Users/Groups within around 40 minutes.

Creation and assignments of AWS SSO Permission Sets

Using Permission Sets, we can assign permissions to synchronized Groups and Users, these permission sets will later create IAM roles in accounts which they are assigned.
You can create new Permission Sets based on AWS Identity and Access Management (IAM) managed policies or create your own custom policies.

To create a new Permission Set in the AWS Management console you can follow the below steps:

  1. Go to the AWS SSO management portal, in the navigation pane, choose AWS accounts and then the AWS organization tab.
  2. In AWS account, choose the account that you want to create a permission set for, and then choose Assign users.
  3. In Display name, choose the user name that you want to create the permission set for, and then choose Next: Permission sets.
  4. In Select permission sets, choose Create new permission set.
  5. In Create new permission set, choose Use an existing job function policy or Create a custom permission set depending on your needs, click Next Details, and then select a job function or create a custom policy or managed policy.
  6. You can then complete the guide and click Create.

You should then see the message “We have successfully configured your AWS account. Your users can access this AWS account with the permissions you assigned”.

If you are more comfortable with the AWS CLI you can use create-permission-set and create-account-assignment in the same way if you would like to.

The most preferred way is however to use Infrastructure as Code and keep this in version control to manage and deploy this.
If you want to use CloudFormation you can use the below template as a base to get started.
https://github.com/pozeus/aws-sso-management/blob/main/template.yml

But be careful on how you deploy these AWS SSO Permission Sets and assignments since it needs to be executed in the Master account. You should always follow the least privilege principle and should therefore carefully plan on which approach you use to deploy these Permission Sets and assignments.
If you want to automate assignments and creation of Permission Sets further, I suggest you go with an event-based approach and assign Permission Sets using Lambdas.

Summary

In this blog-post I showed how you can connect Azure AD to AWS Single Sign-On (SSO), you can now manage access to AWS accounts and applications centrally for single sign-on, and make use of automatic provisioning to reduce complexity when managing and using identities.
Azure AD can now act as a single source of truth for managing users, and users no longer need to manage additional identities and passwords to access their AWS accounts and applications.
Sign in is accomplished using the familiar Azure AD experience, and users will be able to choose the accounts and roles to assume in the AWS SSO portal.

You now also have the possibility to use your existing automation process on how you apply for access, grant and revoke access to systems.

If you have any questions or just want to get in contact with me or any of my colleagues, I’m reachable on any of the following channels.

Email: christoffer.pozeus@tiqqe.com

LinkedIn: Christoffer Pozeus

AWS

Continuous improvement – server access example

When you work with any of the big public cloud providers, one thing is for sure – there will be many changes and improvements to the services that they provide to you. Some changes may perhaps change the way you would architect and design a solution entirely, while others may bring smaller, but still useful improvements.

This post is a story of one of these smaller improvements. It is pretty technical, the gist of it is that with a mindset of continuous improvement, we can find nuggets to make life easier and better and it does not have to be completely new architectures and solutions.

A cloud journey

In 2012, before TIQQE existed, and when some of us at TIQQE started our journey in the public cloud, we created virtual machines in AWS to run the different solutions we built. It was a similar set-up to what we had used in on-premises data centres, and we used the EC2 service in AWS.

Using VPC (Virtual Private Cloud) we could set up different solutions isolated from each other. A regularly used pattern used back then was a single account per customer solution, with separate VPCs for test and production environments. These included both private and public (DMZ) subnets.

To login to a server (required in most cases, not so much immutable infrastructure back then) you needed credentials for the VPN server solution, with appropriate permissions set up. To log in to an actual server, you also needed a private SSH key. One such SSH key is the one which you select or create when you create the virtual machine, for the ec2-user user.

While this worked, it did provide some challenges in terms of security and management – which persons or roles should be able to connect to the VPNs, which VPCS should they be able to access? Of those users and roles, who should be able to SSH into a server and which servers?

There was a centrally managed secrets store solution for the SSH keys for the ec2-user user and different keys for different environments and purposes, but this was a challenge to maintain.

Serverless and Systems Manager

The serverless trend which kind of started with AWS Lambda removed some of these headaches since there were no server access or logins to consider – at least not where solution components can be AWS Lambda implementations. That was great – and still is!
Going serverless can provide other challenges, and it is not the answer to all problems either. There is a lot to say about benefits with serverless solutions. However, this story is focusing on when you still need to use servers.

AWS has another service, called Systems Manager, which is essentially a collection of tools and services to manage a fleet of servers. That service has steadily improved over the years, and a few years back it introduced a new feature called Session Manager. This feature allows a user to login to a server via the AWS Console or via the AWS CLI – no SSH keys necessary to be maintained and no ports to configure for SSH access. This feature also removes the need for a VPN solution for people who need direct access to the server for some reason.
Access control uses AWS Identity and access management (IAM) – no separate credential solution.

Some other major cloud providers already had similar features, so in this regard, AWS was doing some catch-up. It is good that they did!

A new solution to an old problem

For a solution that requires servers, there is a new access pattern to use. No VPN, no bastion hosts. Those that should have direct access to a server and login to that server can now login directly via the AWS Console in a browser tab. No VPN connections, no SSH keys to manage –  only select to connect log in to the server via the browser.  That is, assuming you have the IAM permissions to do so!


For those cases that the browser solution is not good enough, it is still possible to perform an SSH login from a local machine. In this case, it is possible with the help of the AWS CLI to make a connection to a server using Systems Manager Session Manager. The user can have their SSH key, which can be authorized temporarily for accessing a specific server.

Since it is then possible to use regular SSH software locally, it is then also possible to do port forwarding for example, so that the user can access some other resource (e.g. a database) that is only accessible via that server. AWS Systems Manager also allows for an audit trail of the access activities. 

Overall, I believe this approach is useful and helpful for situations where we need direct server access. The key here is though, with a mindset of continuous improvement – we can pick up ways to do better, both big and small.

AWS

AWS re:Invent Online 2020

Usually this time of the year we at TIQQE are getting prepared for re:Invent, traveling to Las Vegas and having our yearly Reinvent comes to you live streamed from our office in Örebro together with our friends, customers and employees. 

This year will of course be a little different but still the possibility to take part online! 

You are well on your way to the best few weeks of the year for Cloud. Make sure to join AWS re:Invent and learn about the latest trends, customers and partners. Followed by many excellent key notes, Break-out sessions, Tracks and not to forget all the possibilities to deepen your knowledge and be provided with training and certifications.

So, whether you are just getting started on the cloud or are an advanced user, come and learn something new at the AWS re:Invent Online 2020.

Make sure to register yourself on the link below and secure your place to re:Invent 2020! 

https://reinvent.awsevents.com/

Want to get started with AWS? At TIQQE, we have loads of experience and are an Advanced Partner to AWS. Contact us, we’re here to help.

AWS

Decide security posture for your Landing Zone.

This is the second post in the series “Where do I start with AWS?”. In this blog post, we will turn our focus on securing our data.

So, we have previously discussed best practises in regards to setting up and governing a new, secure multi-account AWS environment and a framework that is being used to deliver our Infrastructure-as-Code (IaC), as well as application code.

If you haven’t read this blogpost, you can find it here: Where do I start with AWS?

It’s now time to take care of securing our state of the art infrastructure and how we can start automating security incidents that might occur.

Blog post agenda:

  1. Fundamentals of security in your AWS environment
  2. Where to start with security practice?
  3. Introduction to AWS Security Hub
  4. How to start with AWS Security Hub

Fundamentals of security in your AWS environment

Amazon Web Services has a concept they call the Shared Responsibility Model. In this model the responsibility is, as the name implies, a responsibility between you – the customer, the consumer of the services and AWS themselves, the provider of the services.

Below picture describes the Shared Responsibility Model.

Source: aws.amazon.com

As the picture implies, you can say that “You are responsible for what is running ON the cloud. AWS is responsible for running the cloud.”

Let’s take an example related to configuration management stated on the compliance page related to the Shared Responsibility Model at AWS.

“Configuration Management: AWS maintains the configuration of its infrastructure devices, but businesses are responsible for configuring their own guest operating systems, databases, and applications.”

Source: aws.amazon.com

Above example is somewhat related to AWS IaaS services. AWS does however have a lot of other services, services which you don’t have to manage other than providing your code and some minor configuration related to that.

A service like that is AWS Lambda. The AWS Shared Responsibility Model for Lambda sees another level of layer peeled away. Instead of having to manage, maintain and run a EC2 Instance to run their code, or having to track software dependencies in a user-managed container, Lambda allows organizations to upload their code and let AWS figure out how to run it at scale.

Below is a picture of the AWS Shared Responsibility Model for Lambda.

Source: aws.amazon.com

To learn more about the Shared Responsibility Model, please visit the official compliance page at AWS: Shared Responsibility Model.

Where to start with security practice?

Now that we have learned about what we have to take in consideration and what we are responsible for when running workloads in AWS, it is time to actually start with our security practice.

Setting up a baseline

When setting up a baseline for your security practice it is important to first identify which data is important for your business. Classifying data does not only enable you to understand, categorize, label and protect data today, but also in the future when preparing new data structures, regulations and compliance frameworks etc. that might come in your way. Without proper classification, no proper protection.

Get to know what your organization’s “Sacred data”, i.e. the crown jewel data that is of greatest value to your business and would cause the most damage if compromised. Every organization has different needs, and will therefore also have different sacred data. This data will need the most restrictive controls applied to it and should be protected at all cost.

Organizations should create classification categories that make sense for its needs, basically classifying data for what it is worth.

A common method being used is a green, yellow and red color model. As you might think, the color-coded scale depends on the data value and the importance of it.

An example of this could be:

Green data: Likely related to data that is publicly available or confidential company records, not something that would impact a stock price for example but would be a minor reputation hit.

Yellow data: Would be something that is very concerning for your business, sensitive customer data leakage for example.

Red data: Big news event, extreme fines and loss of customer trust, something that might take your business to bankruptcy.

Next steps would be to secure your data according to its classification. When you are aware of the importance levels of your data, you can work backwards to employ security controls that are aligned with its criticality. In this way you can minimize the probability of breaches happening and ask appropriate questions related to your data.

Examples of those questions could be:

  • What systems is processing red data?
  • Does this data need encryption in-transit and at-rest?
  • Who has access to encryption keys?
  • Are there systems that inappropriately move red data over to systems with fewer security controls, such as systems built for green or yellow data?
  • Are you working with least privilege access to your data?

Data classification is a start with the goal of reaching compliance, but there are other things to take in consideration as well. Security is also a people process and needs ongoing collaborative dialog in your organization. You need to grow security awareness within your Operations and Cloud Development teams as a solid understanding and awareness of the implications of running software in the cloud is crucial. If you have a base set of guardrails it becomes important to train your developers to take the responsibility themselves. This could be considered as a “Trust but verify” approach, where you have baseline guardrails in place, but you also provide reviews to the teams to ensure they are compliant with the expectations. This can be a tough thing, and it’s important to work together with teams and be there to support them in succeeding, rather than to be a control mechanism that prevents progress (this is what you are, but it’s all in the attitude towards your coworkers).

Before continuing I would like to say that there are tons of security solutions that can accomplish the same tasks out there, and you might already have one in place that cover some of your needs. I will in this blogpost go through a AWS-based alternative related to security, which is a cost efficient alternative to other more license-heavy solutions out there.

Introduction to AWS Security Hub

At the General Availability announcement of AWS Security Hub, Dan Plastina, Vice President for External Security Services at AWS stated:

“AWS Security Hub is the glue that connects what AWS and our security partners do to help customers manage and reduce risk,” said Dan Plastina, Vice President for External Security Services at AWS. “By combining automated compliance checks, the aggregation of findings from more than 30 different AWS and partner sources, and partner-enabled response and remediation workflows, AWS Security Hub gives customers a simple way to unify management of their security and compliance.”

What is Dan Plastina really talking about here?

AWS Security Hub gives you a broad view of your security alerts and security aspect across your AWS accounts. Powerful security tools such as firewalls and endpoint protection to vulnerability and compliance scanners are all available in this single service which makes this a quite powerful one.

Security Hub is this neat single place that aggregates, organizes and prioritizes your security alerts and findings from AWS services such as Amazon GuardDuty, Amazon Inspector, Amazon Macie, AWS Identity and Access Management (IAM) Access Analyzer, AWS Firewall Manager and other AWS Partner solutions.

AWS Security Hub continuously monitors your environment using automated security checks based on the AWS best practices and industry standards that your organization follows.

You can also take action by using other AWS services such as Amazon Detective or sending findings to your own ticketing system/chat of choice using CloudWatch Event rules. If you are using your own incident management tools, Security Information and Event Management (SIEM) or Security Orchestration Automation and Response (SOAR) it is possible to act on Security Hub findings in these systems as well.

You do also have the ability to develop your own custom remediation actions called playbooks that act on events you define. AWS have released a solution for developing playbooks that remediates security events related to security standards defined as part of the CIS AWS Foundations Benchmark.

The solution is also all serverless, no servers to manage, which means more time for fika or other useful things 😉

You can find this solution at below link:

AWS Security Hub Automated Response and Remediation

I want AWS Security Hub! How and where do I deploy it?

Alright alright, easy now.. We do first need an AWS account suitable for this kind of service, it is after all a critical component and something you only want the Security Team or other people in similar roles to see.

The account that acts as the Security Hub master should be an account that is responsible for security tools. The same account should also be the aggregator account for AWS Config.

It is also important to note that after you have enabled Security Hub in your account that is acting as the Security Hub master, you would also need to enable Security Hub in the other member accounts and then, from the Security Hub that is acting the master, invite the other member accounts. You will then be able to see all security findings related to your accounts in one place i.e. the Security Hub master account.

There is also a script available for deploying security hub in an multi-account environment at below link:

AWS Security Hub Multiaccount Scripts

For those who are using Control Tower, the Security Hub master account would be the shared account named Audit.

There is also an version of above multi-account script that is modified to work with Control Tower that you can find at below link:

AWS Control Tower Security Hub Enabler

Conclusion

In essence Security Hub is a SIEM aggregator, with remediation tips thrown in too. You can make use of a lot of mature AWS services such as CloudWatch and Lambda etc. which makes it very flexible. It can help you understand activity happening in your AWS environment and take appropriate action on this, as well as understand and monitor critical components of your environment.

When integrated with other services such as Amazon GuardDuty, Amazon Macie and Amazon Detective you will have a great toolset to put you in great advantage in terms of your security posture.

Security Hub has a very competitive pricing model and is beneficial for companies looking to get further insight in their AWS workloads.

Security Hub does also have integrations with a lot of third-party providers and is like many other AWS services, developed in an impressive phase as new features are added regularly.

Below is an monthly pricing example of an organization that uses 2 regions and 20 accounts, a quite large organization in other words.

500 security checks per account/region/month

10,000 finding ingestion events per account/region/month

Monthly charges = 500 * $0.0010 * 2 * 20 (first 100,000 checks/account/region/month)

+ 10,000 * $0 * 2 * 20 (first 10,000 events/account/region/month)

= $20 per month

Besides pricing, Security Hub is simple to use and provides several frameworks ready for use out-of-the-box. Security Hub is getting traction among larger respected players such as Splunk, Rackspace, GoDaddy for these reasons and is by no doubt a great service.

Security should be one of the top priorities among organizations but that is not usually the case. When investing in security solutions one organization should first estimate how much a security breach will cost them and which implications it might have and then use this information to set aside a budget dedicated to this field. Classify your data and think about how this data is being processed or used in-transit and at-rest, this can lead to great insights and should not be underestimated.

You will probably have a hard time finding a solution that provides more bang for the buck than Security Hub in regards to securing your AWS resources.

If you have any questions or just want to get in contact with me or any of my colleagues, I’m reachable on any of the following channels.

Mail: christoffer.pozeus@tiqqe.com

LinkedIn: Christoffer Pozeus

AWS

Simply: AWS DynamoDB

My “Simply AWS” series is aimed at absolute beginners to quickly get started with the wonderful world of AWS, this one is about DynamoDB.

What is DynamoDB?

DynamoDB is a NoSQL, fully managed, key-value database within AWS.

Why should I use it?

When it comes to data storage, selecting what technology to use is always a big decisions, DynamoDB is like any other technology not a silver bullet but it does offer a lot of positives if you need a document based key-value storage.

Benefits of DynamoDB

How do I start?

First you’re gonna need an AWS account, follow this guide.

Setting up our first DynamoDB database

If you feel like it you can set your region on the top right corner of the AWS console, it should default to us-east-1 but you can select something closer to you, read more about regions here.

From the AWS console, head to Services and search for DynamoDB, select the first option.

The first time you open up DynamoDB you should see a blue button with the text Create Table, click it.

Now you’re presented with some options for creating your table, enter myFirstTable (this can be anything) in the Table name.

Primary key

A key in a database is something used to identify items in the table and as such it must always be unique for every item. In DynamoDB the key is built up by a Partion key and an option Sort key

  • Partition Key: As the tooltip in the AWS console describes the Partion key is used to partion data across hosts because of that for best practice you should use an attribute that has a wide range of values, for now we don’t need to worry much about this, the main thing to takeaway is that if the Partion key is used alone it must be unique
  • Sort key: if the optional sort key is included the partion key does not have to be unique (but the combination of partion key and sort key does) it allows us to sort within a partion.

Let’s continue, for this example I’m gonna say i’m creating something like a library system, so I’ll put Author as the Partion key and BookTitle as the sort Key.

Note that this is just one of many ways you could setup this type of table and choosing a good primary key is arguably one of the most important decisions when creating a DynamoDB table, what’s good about AWS is that we can create a table, try it out, change our minds and just create a new one with ease.

Next up are table settings, these are things like secondary indexesprovisioned capacityautoscalingencryption and such. It’s a good idea to eventually get a bit comfortable with these options and I highly recommend looking into on-demand read/write capacity mode, but as we just want to get going now the default settings are fine and will not cost you anything for what we are doing today.

Hit create and wait for your table to be created!

Now you should be taken to the tables view of DynamoDB and your newly created table should be selected, this can be a bit daunting as there is a lot of options and information, but let’s head over to the Items tab.

From here we could create an Item directly from the console (feel free to try it out if you want) but I think we can do one better and setup a lambda for interacting with the table.

Creating our first item

If you’ve never created an AWS lambda before I have written a similar guide to this one on the topic, you can find it here.

Create a lambda called DynamoDBInteracter

Make sure to select to create a new role from a template and search for the template role Simple microservice permissions (this will allow us to perform any actions agains DynamoDB).

After creating the lambda we can directly edit it in the AWS console, copy and paste this code.

const AWS = require('aws-sdk')
const client = new AWS.DynamoDB.DocumentClient();
exports.handler = async (event) => {
    try {
        console.log(event.action)
        console.log(event.options)

        let response;
        switch (event.action) {
            case 'put':
                response = await putItem(event.options);
                break;
            case 'get':
                response = await getItem(event.options);
                break;
        }
        return response;
    } catch (e) {
        console.error(e)
        return e.message || e
    }
};


let putItem = async (options) => {
    var params = {
      TableName : 'myFirstTable',
      Item: {
         Author: options.author,
         BookTitle: options.bookTitle,
         genre: options.genre
      }
    };

    return await client.put(params).promise();
}

let getItem = async (options) => {
    var params = {
      TableName : 'myFirstTable',
      Key: {
        Author: options.author,
        BookTitle: options.bookTitle
      }
    };


    return await client.get(params).promise();
}

hit Save then create a new test event like this.

{
    "action": "put",
    "options": {
        "author": "Douglas Adams",
        "bookTitle": "The Hitchhiker's Guide to the Galaxy",
        "genre": "Sci-fi"
    }
}

and run that test event.

Go back to DynamoDB and the Items tab and you should see your newly created Item!

Notice that we did not have to specify the genre attribute that is because DynamoDB is NoSQL it follows no schemea and any field + value can be added to any item irregardless of the other items composition as long as the primary key is valid.

Retrieving our item

Now let’s try to get that item, create another test event like this.

{
    "action": "get",
    "options": {
         "author": "Douglas Adams",
        "bookTitle": "The Hitchhiker's Guide to the Galaxy"
    }
}

and run it.

You can expand the execution results and you should see your response with the full data of the item.

Congratulations!

You’ve just configured your first DynamoDB table and performed calls against it with a lambda but don’t stop here the possibilities with just these two AWS services are endless, my next guide in this series will cover API-Gateway and how we can connect an API to our lambda that then communicates with our database table, stay tuned!

What’s next?

As I’m sure you understand we’ve just begun to scratch the surface of what DynamoDB has to offer, my goal with this series of guides is to get your foot in the door with some AWS services and show that although powerful and vast they are still easy to get started with, to check out more of what calls can be made with the DynamoDB API (such as more complicated queries, updates and scans as well as batch writes and reads) check this out, feel free to edit the code and play around.

I would also like to recommend this guide if you want even more in-depth look into DynamoDB, it covers most of what I have here but more detailed and also goes over some of the API actions mentioned above.

Contact me

Questions? Thoughts? Feedback?
Twitter: @tqfipe
Linkedin: Filip Pettersson

AWS

AWS Summit Online 2020

June 17, 2020 – You are well on your way to the best day of the year for cloud! Join the AWS Summit Online and deepen your knowledge with this free, virtual event if you are a technologist at any level. There is something for everybody.

Hear about the latest trends, customers and partners in EMEA, followed by the opening keynote with Werner Vogels, CTO, Amazon.com. All developers at TIQQE are always attending Werner’s keynotes.

After the keynote, dive deep in 55 breakout sessions across 11 tracks, including getting started, building advanced architectures, app development, DevOps and more. Tune in live to network with fellow technologists, have your questions answered in real-time by AWS Experts and claim your certificate of attendance.

So, whether you are just getting started on the cloud or are an advanced user, come and learn something new at the AWS Summit Online.

Want to get started with AWS? At TIQQE, we have loads of experience and are an Advanced Partner to AWS. Contact us, we’re here to help.

What to expect

AWS

Continuous delivery in AWS – tools overview

Continuous Delivery (CD) is a term that is used for a collection of practices that strive for enabling an organisation to provide both speed and quality in their software delivery process. It is a complex topic and in this article we will focus on one aspect, which is selecting tools for CD pipelines when deploying software in AWS.

Before we dive into various tooling options for continuous delivery though, let us define some scope, terminology and also talk a bit why we would bother with this in the first place.

Scope

Our scope for this overview is for delivering solutions that runs in AWS. Source code lives in a version control system and we assume that it is a hosted solution (not on-premise) and that it is Git. Git is currently the most common version control system in use. Some services mentioned may also work with other version control systems, such as Subversion, for example.

Continuous delivery tools can either be a software-as-a-service (SaaS) solution, or we can manage it ourselves on servers – in the cloud or on-premise. Our scope here is only for the SaaS solutions.

If you have a version control system that is hosted on-premise or on your own servers in some cloud, that typically works with continuous delivery software that you can host yourself – either on-premise or in the cloud. These options we will not cover here.

Terminology

First of all, there are a few terms and concepts to look at:

  • Pipeline – this generally refers to the process that starts with changing the code to the release and deployment of the updated software. This would be a mostly or even completely automated process in some cases.
  • Continuous integration (CI) – the first part of the pipeline, in which developers can perform code updates in a consistent and safe way with fast feedback loops. The idea is to do this often and that it should be quick, so any errors in changes can be caught and corrected quickly. Doing it often means that there are only small changes each time, which makes it easier to pinpoint and correct any errors. For CI to work well, it needs a version control system and a good suite of automated tests that can be executed when updates someone commits updates to a version control system.
  • Continuous Delivery (CD) – This refers to the whole process from code changes and CI to the point where a software release is ready for deployment. This includes everything in the continuous integration part and other steps that may be needed to make the software ready for release. Ideally this is also fully automated, although may include manual steps. Again, the goal is that this process is quick, consistent and safe, so that it would be possible to make a deployment “at the click of a button”, or similar simple procedure. But the deployment is not part of continuous delivery.
  • Continuous Deployment (CD) – Unfortunately the abbreviation is the same as for continuous delivery, but it is not the same. This is continuous delivery plus automated deployment. In practice, this is applicable for some solutions but not all. With serverless solutions it is generally easier to do this technically, but in many cases it is not a technology decision, but a business decision.

Why continuous delivery?

Speed and safety for the business/organisation – that is essentially what it boils down to. To be able to adapt and change based on market and changing business requirements and to do this in a way that minimises disruption of the business.

Depending on which stakeholders you look at, there are typically different aspects of this process that are of interest:

  • Business people’s interests are in speed and predictability of delivery of changed business requirements and that services continues to work satisfyingly for customers.
  • Operations people’s interests are in safety, simplicity and predictability of updates and that disruptions can be avoided.
  • Developers’ interest is in fast feedback on the work they do and that they can do changes without fear of messing things up for themselves and their colleagues. Plus that they can focus on solving problems and building useful or cool solutions.

It is a long process to reach continuous delivery Nirvana, and the world of IT a mess to various degrees – we are never done. A sane choice of tooling for continuous delivery can at least get us part of the way.

Continuous delivery tools

If we want a continuous delivery tool which targets AWS, uses git and runs as a SaaS solution, we have a few categories:

  • Services provided by AWS
  • Services provided by the managed version control system solution
  • Third party continuous delivery SaaS tools

Services provided by AWS

AWS has a number of services that is related to continuous delivery, which all have names that start with “Code” in them. This includes:

  • AWS CodeCommit
  • AWS CodePipeline
  • AWS CodeBuild
  • AWS CodeDeploy
  • AWS CodeGuru
  • AWS CodeStar

A key advantage with using AWS services is that credentials and access is the regular identity and access management (IAM) in AWS and encryption with key management service (KMS). There is no AWS secrets information that has to be stored elsewhere outside of AWS, since it all lives in AWS – assuming your CI/CD workflow goes all-in on AWS – or to a large extent at least.

A downside with these AWS services is that they are not the most user-friendly, plus there are a number of them. They can be used together to set up elaborate CI/CD workflows, but it requires a fair amount of effort to do so. CodeStar is a service here that was an attempt to set up an end-to-end development workflow with CI/CD.
I like the idea behind CodeStar and for some use cases it may be just fine. But it has not received so much love from AWS since it was launched.

You do not necessarily need all of these services to set up a CI/CD workflow – in its simplest form you just need a supported source code repository (CodeCommit/Github/Bitbucket) and CodeBuild. But things can quickly get more complicated, in particular once the number of repositories, developers and/or AWS accounts involved starts to grow. One project that tries to alleviate that pain is the AWS Deployment Framework.

Services provided by the managed version control system solution

Three of the more prominent version control system hosting services are Github, Gitlab and Bitbucket. They all have CI/CD services bundled with their hosted service offering. Both Bitbucket and Gitlab also provide on-premise/self-hosted versions of their source code repository software as well as continuous delivery tooling and other tools for the software lifecycle. The on-premise continuous delivery tooling for Bitbucket is Bamboo, while the hosted (cloud) version is Bitbucket Pipelines. For Gitlab the software is the same for both hosted and on-premise. We only cover the cloud options here.

On the surface the continuous delivery tooling is similar for all these three – a file in each repository which describes the CI/CD workflow(s) for that particular repository. They are all based on running Docker containers to execute steps in the workflow and can handle multiple branches and pipelines. They all have some kind of organisational and team handling capabilities.

Beyond the continuous delivery basics they start to deviate a bit in their capabilities and priorities. Bitbucket, being an Atlassian product/service, focus on good integration with Jira in particular, but also some 3rd party solutions. Gitlab prides itself on providing a one-stop solution/application for the whole software lifecycle – what features are enabled depends on which edition of the software that is used. Github, being perhaps the most well-known source code repository provider, has a well-established ecosystem for integration with various tools into their toolchain, provided by 3rd parties and community – more so than the other providers.

Github and Gitlab have the concept of runners that allow you to set up your own machines to run tasks in the pipelines.

So if you are already using other Atlassian products, Bitbucket and Bitbucket Pipelines might be a good fit. If you want an all-in-one solution then Gitlab can suite well. For a best-of-breed approach to pick different components, then Github is likely a good fit.

Third party continuous delivery SaaS tools

There are many providers which provide hosted continuous delivery tooling. Some of these providers have been in this space for a reasonably long time, before the managed version control system providers added their own continuous delivery tooling.

In this segment there may be providers that support specific use cases better, or are able to set up faster and/or parallel pipelines easily. They also tend to support multiple managed version control system solutions and multiple cloud provider targets. Some of them also provide self-hosted/on-premise versions of their software solutions. Thus this category of providers may be interesting for integrating with a diverse portfolio of existing solutions.

Some of the more popular SaaS providers in this space include:

Pricing models

Regardless of category, pretty much all the different providers mentioned here provide some kind of free tier and then one or more on-demand paid tiers.

For example: Github Actions, Bitbucket Pipelines, Gitlab CI/CD and AWS CodeBuild provide a number of free build minutes per month. This is however limited to certain machine sizes used in executing the tasks in the pipelines.

A simple price model of just counting build minutes is easy to grasp, but will also not allow flexibility in machine sizes, since larger machine will require more capacity from the provider. In AWS case with CodeBuild, you can select a number of different machine sizes – but you need to pay for anything larger than the smaller machines from the first minute.

The third party continuous delivery providers have slightly different free tier models, I believe partially in order to distinguish them from the offerings of the managed version control system providers. For example, CircleCI provides a number of free “credits” per week. Depending on machine capacity and feature, pipeline execution will cost different amounts of credits.

The number of parallel pipeline executions is typically also a factor for all the different providers – free tiers tend to have 1 pipeline that can execute at any time, while more parallel execution will cost more.

Many pricing models also a restriction on the number of users and there may be a price tag attached to each active user also. All in all, you pay for compute capacity, to save time on pipeline execution and to have more people utilize the continuous delivery pipelines.

AWS, with a number of services fulfilling various parts of the continuous delivery solution, may be a bit more complex to grasp initially what things will actually cost. Also, the machine sizes may not be identical across the different services either, so a build minute for one service may not necessarily be one build minute at another provider.

Trying to calculate the exact amount the continuous delivery solution will cost may be counterproductive at an early stage though. Look at features needed first and their importance, then consider pricing after that.

End notes

Selecting continuous delivery tooling can be a complex topic. The bottom line is that it is intended to deliver software faster, more secure and more consistently, with fewer problems – and with good insight into the workflow for different stakeholders. Do not loose sight of that goal and what your requirements are – beyond the simple cases. Most alternatives will be ok for the very simple cases. Do not be afraid to try out some of them, but time box the effort.

If you wish to discuss anything of the above, please feel free to contact me at erik.lundevall@tiqqe.com