Deployments with Quality Gates

Describes how Keptn allows to deploy an artifact using automatic quality gates and blue/green deployments.

About this tutorial

When developing an application, sooner or later you need to update a service in a production environment. To conduct this in a controlled manner and without impacting end-user experience, the quality of the new service has to be ensured and adequate deployment strategies must be in place. For example, blue-green deployments are well-known strategies to roll out a new service version by also keeping the previous service version available if something goes wrong.

For this tutorial, we prepared a slow and a regular version of the carts service:

In this tutorial, we will deploy these two versions. During this deployment process, the versions have to pass a quality gate in the staging environment in order to get promoted to the production environment. This quality gate checks whether the average response time of the service is under 1 second. If the response time exceeds this threshold, the performance evaluation will be marked as failed.

In overview, we will conduct these two scenarios:

First, we will try to deploy the slow version of the carts service (v0.9.2). Therefore, Keptn will deploy this new version into the dev environment where functional tests will be executed. After passing these functional tests, Keptn will promote this service into the staging environment by releasing it as the blue or green version next to the previous version of the service. Then, Keptn will route traffic to this new version by changing the configuration of the virtual service (i.e., by setting weights for the routes between blue and green) and Keptn will start the defined performance test. Using the monitoring results of this performance test will allow Pitometer to evaluate the quality gate. This slow version will not pass the quality gate and, hence, the deployment will be rejected. Furthermore, Keptn will direct the requests to the service to the previous working deployment of the service.

Second, we will deploy the regular version of the carts service (v0.9.3). Therefore, Keptn will conduct the same steps as before except that this version will now pass the quality gate. Hence, this regular version will be promoted into the production environment.

Prerequisites

• Finish the Onboarding a Service tutorial.

Set up of monitoring for the carts service

Since this tutorial relies on the concept of quality gates, you will need to set up monitoring for your carts service. The Pitometer service will then evaluate the data coming from the monitoring solution to determine a score for the quality gate.

For using the quality gate, Pitometer requires a performance specification. This performance specification is described by two files, namely the service-indicators.yaml file, which describes the available types of metrics and their data sources (e.g., Prometheus or Dynatrace), and the service-objectives.yaml, which describe the desired values for those metrics. To learn more about the service-indicator, and service-objective file, click here Specifications for Site Reliability Engineering with Keptn.

In this tutorial, we will be using either the open source monitoring solution Prometheus or Dynatrace.

Option 1: Prometheus

Expand instructions

To set up the quality gates for the carts service, please navigate to the examples/onboarding-carts folder. This folder contains the files service-indicators.yaml and service-objectives-prometheus-only.yaml. To set the quality gates based on those files, upload it via the following command:

keptn add-resource --project=sockshop --service=carts --stage=staging --resource=service-indicators.yaml --resourceUri=service-indicators.yaml
keptn add-resource --project=sockshop --service=carts --stage=staging --resource=service-objectives-prometheus-only.yaml --resourceUri=service-objectives.yaml

Afterwards, execute the following command to set up the rules for the Prometheus Alerting Manager based on those quality gates:

keptn configure monitoring prometheus --project=sockshop --service=carts

To verify that the rules are correctly set up, you can access Prometheus by enabling port-forwarding for the prometheus-service:

kubectl port-forward svc/prometheus-service 8080 -n monitoring

It is then available on localhost:8080/targets where you can see the three targets for the carts service:

Option 2: Dynatrace

  keptn add-resource --project=sockshop --service=carts --stage=staging --resource=service-indicators.yaml --resourceUri=service-indicators.yaml

  keptn add-resource --project=sockshop --service=carts --stage=staging --resource=service-objectives-dynatrace-only.yaml --resourceUri=service-objectives.yaml

View carts service

• Get the URL for your carts service with the following commands in the respective namespaces:

  echo http://carts.sockshop-dev.$(kubectl get cm keptn-domain -n keptn -o=jsonpath='{.data.app_domain}')

  echo http://carts.sockshop-staging.$(kubectl get cm keptn-domain -n keptn -o=jsonpath='{.data.app_domain}')

  echo http://carts.sockshop-production.$(kubectl get cm keptn-domain -n keptn -o=jsonpath='{.data.app_domain}')

• Navigate to http://carts.sockshop-production.YOUR.DOMAIN for viewing the carts service in your production environment and you should receive an output similar to the following:

  

Deploy the slow carts version

1. Use the Keptn CLI to deploy a version of the carts service, which contains an artificial slowdown of 1 second in each request.

  keptn send event new-artifact --project=sockshop --service=carts --image=docker.io/keptnexamples/carts --tag=0.9.2

Quality gate in action

After triggering the deployment of the carts service in version v0.9.2, the following status is expected:

• Dev stage: The new version is deployed in the dev namespace and the functional tests passed.

  • To verify, open a browser and navigate to: http://carts.sockshop-dev.YOUR.DOMAIN

• Staging stage: In this stage, version v0.9.2 will be deployed and the performance test starts to run for about 10 minutes. After the test is completed, Keptn triggers the test evaluation and identifies the slowdown. Consequently, a roll-back to version v0.9.1 in this stage is conducted and the promotion to production is not triggered.

  • To verify, the Keptn’s Bridge shows the deployment of v0.9.2 and then the failed test in staging including the roll-back:

  

• Production stage: The slow version is not promoted to the production namespace because of the active quality gate in place. Thus, still version v0.9.1 is expected to be in production.

  • To verify, navigate to: http://carts.sockshop-production.YOUR.DOMAIN

Deploy the regular carts version

1. Use the Keptn CLI to send a new version of the carts artifact, which does not contain any slowdown.

   keptn send event new-artifact --project=sockshop --service=carts --image=docker.io/keptnexamples/carts --tag=0.9.3
   

2. This automatically changes the configuration of the service and automatically triggers the deployment.

3. In this case, the quality gate is passed and the service gets deployed in the production namespace.

4. To verify the deployment in production, open a browser an navigate to http://carts.sockshop-production.YOUR.DOMAIN. As a result, you see Version: v3.

5. Besides, you can verify the deployments in your Kubernetes cluster using the following commands:

   kubectl get deployments -n sockshop-production
   

   NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
   carts-db        1         1         1            1           63m
   carts-primary   1         1         1            1           98m
   

   kubectl describe deployment carts-primary -n sockshop-production
   

   ...
   Pod Template:
     Labels:  app=carts-primary
     Containers:
       carts:
         Image:      docker.io/keptnexamples/carts:0.9.3