AWS

The introduction of Site Reliability Engineering (SRE)

The introduction of Site Reliability Engineering (SRE) teams in the structure of an organization, is becoming more and more popular in the IT industry and in the DevOps domains.
Let’s discover in this article the reason for SRE popularity and what are the differences and the common points between DevOps and SRE. 


In the last two decades we have witnessed a huge transformation in the way of building and delivering software. The Agile culture first and the DevOps revolution later, have transformed the structure of the tech organizations and they can be seen as a de facto standard in the IT industry.

As everyone knows, IT is a constantly evolving sector and recently we are seeing an increasing popularity of Site Reliability Engineering discipline, especially in the DevOps domains. But what is SRE? And what are the differences and the common points between SRE and DevOps? 

DevOps

DevOps culture has contributed to tearing down the wall between software development and the software operation, providing a set of ideas and practices that has led several benefits:

  • Better collaboration and communication between the parts;
  • Shorter release cycles;
  • Faster innovation;
  • Better systems reliability;
  • Reduced IT costs;
  • Higher software delivery performance.

Even though this sounds amazing, there are still quite a lot of companies that struggle with bringing the DevOps culture in their organization. The reason for this is that DevOps is an ideology and not a methodology or technology, which means it doesn’t say anything about how to successfully implement a good DevOps strategy. 

SRE 

Site Reliability Engineering (SRE) is a discipline that was born at Google in early 2000s to reduce the gap between software development and operations and that was completely independent by the DevOps movement. SRE uses software engineering approaches to solve operational problems.
SRE teams have main focus on:

  • Reliability;
  • Automation.

Let’s deepen these aspects.

Reliability

One of the main goals of the SRE is making and keeping systems up and running “no matter what”. In order to achieve this, it is important to keep in mind that failures and faults can happen. SRE discipline embraces them by focusing on:

  • observability;
  • system performance;
  • High availability (HA);
  • emergency response and disaster recovery; 
  • incidents management;
  • Learning from the past problems;
  • disaster mitigation and prevention.

Automation

Automating all the activities that are traditionally performed manually is another of the main goals of SRE.
Automation and software engineering are used to solve operational problems. 

Automation plays a fundamental role in SRE: it allows us to get rid of human errors present in the processes and the activities that regard the system. One could argue that automation introduces bugs in the system anyway and well, that is true but there is one big difference: one can test automated processes but cannot test processes that involve human activities. 

DevOps vs. SRE

As we have understood, both DevOps culture and SRE discipline aim to reduce the gap between software development and operations. Below we summarize them, describing their common goal first and where they differ the most. 

class SRE implements DevOps

As mentioned earlier DevOps doesn’t say anything about how to successfully bring the culture in the organization since it is an ideology. On the other hand, SRE can be seen as implementation of the DevOps philosophy. 
In fact, even though the origins of SRE are completely independent from DevOps, and the discipline provides additional practices that are not part of DevOps, SRE implements DevOps ideas.

Responsibility and Code ownership

SRE can be considered the next stage of DevOps because of the focus on code ownership: the SRE engineer accepts the responsibility of owning the code they develop, in production. A bit different from DevOps where the responsibilities are shared to achieve a shorter release cycle and to improve the collaborations.

Conclusions

The introduction of Site Reliability Engineering (SRE) teams in the structure of an organization, is becoming more and more popular in the IT industry and in the DevOps domains. 
The reason for its popularity can be found in the benefits that the discipline brings: 

  • Better collaboration and communication between the parts;
  • Shorter release cycles;
  • Faster innovation;
  • Better systems reliability;
  • Reduced IT costs;
  • Higher software delivery performance;
  • Reducing incidents in production;
  • Code ownership;
  • Automation of the processes;

As you could notice some of these benefits are exactly the same that you will experience bringing DevOps in your organization.

SRE can be considered an implementation of DevOps culture that has the goal of making and keeping services reliable.

AWS

Connect your Azure AD to AWS Single Sign-On (SSO)

In this blog post, I will provide a step-by-step guide for how to integrate your Azure AD with AWS Single Sign On (SSO). The integration will enable you to manage access to AWS accounts and applications centrally for single sign-on, and make use of automatic provisioning to reduce complexity when managing and using identities.

Organizations usually like to maintain a single identity across their range of applications and AWS cloud environments. Azure Active Directory (AD) is a common authentication method as Office 365 often is used among companies, and might be the hub of authentication as it often is integrated with other applications as well.

If you are using AWS Single Sign-On (SSO) you can leverage your existing identity infrastructure while enabling your users to access their AWS accounts and resources at one place. By connecting Azure AD with AWS SSO you can sign in to multiple AWS accounts and applications using your Azure AD identity with the possibility to enable automatic synchronization of Azure AD Users/Groups into AWS SSO.

This makes perfect sense and often improves your ability to further automate how you handle user-lifecycle and access to your AWS accounts as you might already have some identity manager connected to your HR system like Microsoft Identity Manager or Omada in place for example. You can also leverage your existing process for applying for access to different systems, ServiceNow or similar solution might already be connected to Azure AD in one way or another which then could be leveraged for applying for access to different AWS Accounts.
There are also other benefits such as levering your existing MFA solution if your organization has such a solution in place.

To the good stuff! I will in this blog-post demonstrate how you can connect your Azure AD to AWS SSO and take advantage of its capabilities.

Creating a new Azure Enterprise Application

Login to your Azure portal and open Azure Active Directory. Under the Manage section, click on Enterprise application.

Click New application and select a Non-gallery application, give your new application an appropriate name and click Add.

Once the Azure gods have created our new application, head into the Overview page and select Set up single sign-on and choose the SAML option.

Under section, SAML Signing Certificate click Download next to Federation Metadata XML.

Please keep this page open as we later need to upload the metadata XML file from AWS SSO.

Setup AWS SSO

Login to AWS management console and open AWS Single Sign-On, please ensure that you are in your preferred region. If you haven’t enabled AWS Single Sign-On already, you can enable it by clicking Enable AWS SSO as shown below.

Click Choose your identity source. You can also configure your custom User portal URL if you’d like but it is not required.

Select External identity provider. Upload the AD Federation Metadata XML file downloaded earlier inside the IdP SAML metadata section and download the AWS SAML metadata file.

In the Azure SAML page, click Upload metadata file and upload the AWS SSO SAML metadata file.

If you have configured a User portal URL earlier, you need to edit the Basic SAML Configuration section and match the Sign-on URL.

Setting up automatic provisioning

The connection between Azure AD and AWS SSO is now established, we can proceed to enable automatic provisioning to synchronise users/groups from Azure AD to AWS SSO.

Note that you can use Azure AD groups but not nested groups ie. groups that are into groups.

Head over to the Provisioning page and change the mode to Automatic. Please keep this page open as we will copy values from AWS SSO.

In the AWS SSO Settings page, click Enable automatic provisioning

Take note of both values given in the popup

In the Provisioning page in the Azure portal, expand the Admin Credentials section and insert the values from above. It is also recommended to add an email address for notification of failures.

SCIM endpoint > Tenant URL
Access token > Secret Token

Note that these tokens expire after 1 year and should be renewed for continuous connectivity.

Click Test Connection and it should result in a success message.

Expand the Mapping section and click Synchronize Azure Active Directory Users to customappsso

Which attributes you want to sync over depends on your setup, but default setups you can remove all attributes except:
userName
active
displayName
emails
name.givenName
name.familyName

You then create a new attribute mapping objectId with externalId.

Important to note is that you can modify the email attribute to use userPrincipalName over mail as not all users have Office365 licenses which leave that attribute null.

In the Provisioning page, you can now set the Status to On. It is recommended leaving Scope set to Sync only assigned users and groups.
Click Save, it should take about 5 minutes for it to start synchronizing.

Our AWS SSO and Azure AD connection is now fully set up, when you assign Azure Users/Groups to the enterprise app, they will then appear in AWS SSO Users/Groups within around 40 minutes.

Creation and assignments of AWS SSO Permission Sets

Using Permission Sets, we can assign permissions to synchronized Groups and Users, these permission sets will later create IAM roles in accounts which they are assigned.
You can create new Permission Sets based on AWS Identity and Access Management (IAM) managed policies or create your own custom policies.

To create a new Permission Set in the AWS Management console you can follow the below steps:

  1. Go to the AWS SSO management portal, in the navigation pane, choose AWS accounts and then the AWS organization tab.
  2. In AWS account, choose the account that you want to create a permission set for, and then choose Assign users.
  3. In Display name, choose the user name that you want to create the permission set for, and then choose Next: Permission sets.
  4. In Select permission sets, choose Create new permission set.
  5. In Create new permission set, choose Use an existing job function policy or Create a custom permission set depending on your needs, click Next Details, and then select a job function or create a custom policy or managed policy.
  6. You can then complete the guide and click Create.

You should then see the message “We have successfully configured your AWS account. Your users can access this AWS account with the permissions you assigned”.

If you are more comfortable with the AWS CLI you can use create-permission-set and create-account-assignment in the same way if you would like to.

The most preferred way is however to use Infrastructure as Code and keep this in version control to manage and deploy this.
If you want to use CloudFormation you can use the below template as a base to get started.
https://github.com/pozeus/aws-sso-management/blob/main/template.yml

But be careful on how you deploy these AWS SSO Permission Sets and assignments since it needs to be executed in the Master account. You should always follow the least privilege principle and should therefore carefully plan on which approach you use to deploy these Permission Sets and assignments.
If you want to automate assignments and creation of Permission Sets further, I suggest you go with an event-based approach and assign Permission Sets using Lambdas.

Summary

In this blog-post I showed how you can connect Azure AD to AWS Single Sign-On (SSO), you can now manage access to AWS accounts and applications centrally for single sign-on, and make use of automatic provisioning to reduce complexity when managing and using identities.
Azure AD can now act as a single source of truth for managing users, and users no longer need to manage additional identities and passwords to access their AWS accounts and applications.
Sign in is accomplished using the familiar Azure AD experience, and users will be able to choose the accounts and roles to assume in the AWS SSO portal.

You now also have the possibility to use your existing automation process on how you apply for access, grant and revoke access to systems.

If you have any questions or just want to get in contact with me or any of my colleagues, I’m reachable on any of the following channels.

Email: christoffer.pozeus@tiqqe.com

LinkedIn: Christoffer Pozeus

Serverless

Choosing the right tool for your serverless journey.

What tools are available if I want to start building my own serverless applications? We will go over some of the most popular frameworks and tools available for the developer who wants to get started with AWS Lambda.

There are a lot of tools out there to use when building software that are powered by AWS Lambda. These tools aim to ease the process of coding, configuring and deploying the Lambdas themselves but also the surrounding AWS infrastructure. We will discuss the following alternatives:

  • AWS Cloud development kit (CDK)
  • Serverless Framework
  • AWS Serverless Application Model (SAM)
  • Terraform

About AWS CloudFormation

AWS CloudFormation is a service within AWS that lets you group resources (usually AWS resources) into stacks that you can deploy and continuously update. These CloudFormation templates are written in JSON or YAML and manually typing those can be very tedious. Especially when your stack grows to a lot of resources that reference each other in multiple ways. What lots of these frameworks and tools do is to provide an abstraction layer in front of CloudFormation so that the developer can more rapidly create and focus on the business value of the service they are building.

AWS Cloud development kit (CDK)

The AWS CDK went into general availability in the summer of 2019 and has been getting a lot of traction lately. It is an open source framework that lets you create your infrastructure as code instead of CloudFormation. You then generate CloudFormation templates from your code by running the command cdk synthesize.

You can choose from Python, TypeScript, JavaScript, .NET and Java to describe your resources instead of having to do it in pure CloudFormation. This gives you benefits such as code completion and being able to assign resources to variables, which helps you when a resource needs referencing to another one. Another great benefit is that it has helper functions in place to help with common developer use cases. For example setting up a NodeJs Lambda function or constructing ARNs.

The CDK code in the screeenshot will be synthesized to CloudFormation at build time. This example creates a Lambda function, a DynamoDB table and an API Gateway. The example also shows referencing between the resources: for example giving the Lambda function rights to read from the table.

Serverless Framework

This one has been around since 2015 and was called JAWS before quickly changing to its current name. As the very descriptive name says, it’s a framework for building serverless applications! The framework is easy to get started with and setting up an API with a few Lambdas require very little configuration for the developer as the framework takes care of the underlying CloudFormation template.

Because of its specific focus in serverless applications, the framework is not as broad as the CDK and that comes with pros and cons. You will get a lot of help if you are setting up Lambdas or events that trigger those Lambdas, but setting up the surrounding infrastructure such as queues, tables, kinesis streams and cognito user pools will often require you to write pure CloudFormation. At TIQQE, some of us like to create and deploy this surrounding infrastructure with the CDK, while developing the Lambdas  and the API gateway in Serverless Framework.

Serverless Framework is open source and multi-cloud. It’s also extendable with a wide range of plugins created by the community.

Shows the central configuration file in a Serverless Framework service: serverless.yml. When deployed, a Lambda function “myFirstLambda” is created and an API Gateway will be set up with a method to invoke the Lambda at the path /hello.

AWS Serverless Application Model (SAM)

AWS SAM is another framework similar to Serverless Framework that it let’s the developer write less code when building serverless applications. Unlike Serverless Framework, SAM is specific to AWS and its main configuration file template.yml is written with CloudFormation. So if you have previous experience with AWS and CloudFormation you will likely find it easy to get started with SAM. A neat feature in SAM is that it has support to deploy APIs using swagger out of the box. 

Template.yml in a AWS SAM project. Deploying this template will produce the same result as the Serverless Framework example above.

Terraform

This multi-cloud tool for infrastructure as code is worth a mention! It has been around since 2014 and is written in Go. For AWS it uses the aws-sdk to manage resources instead of CloudFormation, which gives the benefit of not having a resource limit of 200 that AWS impose for CloudFormation templates.

How do I choose which one to pick?

It comes down to some characteristics of your application, and a fair bit of personal preference! 

  • Are you building a service with an API endpoint and you have little or no previous experience in AWS or serverless architecture? We recommend you to check out Serverless Framework.
  • Are you not a fan of writing CloudFormation and your architecture needs a lot of infrastructure? Check out AWS CDK.
  • Are you familiar with CloudFormation and want to get started with serverless applications? AWS SAM could be the perfect match! 

There are countless forum posts and articles debating whether to go with AWS SAM or Serverless Framework. The frameworks are very similar and many times it comes down to personal taste. At TIQQE we have lots of experience working with Serverless Framework and some of us would debate that you get the job done in less lines of code with Serverless Framework. With that said, SAM does not have to worry about being generic for multiple clouds and that can be an edge if you are working only with AWS. SAM also defaults to giving Lambda functions least privilege access rights (AWS best practice), while Serverless Frameworks share a role between all Lambdas.

Terraform can be a good match if you are creating infrastructure as code across multiple clouds. While Terraform is capable of doing many things, it is not specialised in serverless technologies and you will have to write a lot of code to achieve the same results as the other frameworks described in this post. Not having a 200 resource limit is nice but should not be a problem that often if you are designing your systems in terms of microservices.

Do you have any comments or questions about this article? Please reach out!

johannes.uhr@tiqqe.com

Serverless

Simply: AWS Lambda

Why should I use AWS Lambda and how does it work? In this blog post I provide you with a practical hands-on guide of how to create your first AWS Lambda service and explain why you should use it to create awesome customer value.

What is AWS Lambda?

With AWS lambda we can write code and execute it without caring about configuring servers.

Why should I use it?

It enables you to quickly develop business relevant code and deliver value for your customers and stakeholders.

How do I start?

First you’re gonna need an AWS account, follow this guide.

Creating our first Lambda

From the AWS console head to Services and search for Lambda select the first option.

Click Create Function

Enter your name for the lambda and select runtime (I’m going with Node.js) Leave everything else default.

Writing code

When your lambda is created you’ll be taken to that lambdas page where you can see and setup lots of information and options about your lambda, let’s not worry too much about that right now and just scroll down to “Function Code”.

Using the inline editor (you are of course able to write code with any IDE you want and deploy it to AWS but I’ll cover that in another post) let’s enter some code, this is what I used.

exports.handler = async (event) => {
console.log('event', event); // initiate animals array
const animals = ['cat', 'dog', 'tardigrade']; // get input
const input = JSON.parse(event.body).input; // concatinate animals with input
concatAnimalsInput(animals, input) // create a response object and return it
const response = {
statusCode: 200,
body: JSON.stringify(animals),
};
return response;
};const concatAnimalsInput = (animals, input) => {
if(typeof input === 'string') {
animals.push(input);
} else {
animals = animals.concat(input);
}
}

Testing our code

At the top of the screen click configure test event and create an event to execute the function with.

The event in JSON format

Hit Create and finally click the “Test” button.

After its execution you’ll see the result and the output by clicking Details in the green result box, you can also click (logs) to enter CloudWatch Logs and get a better look into all executions of your lambda.

Good job!

You’ve just created a lambda and the possibilities with it are endless, in future posts I’ll discuss how we can connect an API to our lambda via API Gateway and how we can store our data in the NoSQL database DynamoDB.

Discussion: what about the price?

With Lambda the first million requests each month are alway free after that you pay $0.20 per 1M requests and $0.0000166667 for every GB-second, read more here. Lambda is usually used together with other AWS services that might also incur cost such as Cloudwatch logs which we touched upon in this post, Cloudwatch logs also offer a free tier, 5GB of Log Data Ingestion and 5GB of Log Data Archive, which means nothing we did in this post will result in any cost even if you do no cleanup.
Read more about the economics of cloud here “Cloud is expensive”

I don’t want to use the inline code editor!

Great, me neither, I suggest as a first step either looking into exporting your code to zip and uploading to the lambda

or exploring the Serverless framework, a tool that makes it easy to deploy serverless applications such as Lambda!

You’re welcome to contact me if you have any questions.

Mail: filip.pettersson@tiqqe.com
LinkedIn: Filip Pettersson

Read more articles in this series:

Simply: AWS DynamoDB