Upgrading

This guide is for the platform team that runs Cloud Certainty Watchtower. It covers how releases work, how you find out about new ones, how to upgrade and roll back, and how to change plan.

#How versions work

• Semantic versions. Every release has a number MAJOR.MINOR.PATCH, for example 0.2.0:
  • PATCH releases (0.2.1) fix bugs and security issues. They never change parameters.
  • MINOR releases (0.3.0) add features. New parameters always have a safe default.
  • MAJOR releases (1.0.0) may need action from you. Their release notes say exactly what.
• One template pair per release, in the vendor's public bucket in us-east-1. The same URLs work in any Region:

  https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/pro/<version>/main.yaml
  https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/pro/<version>/spoke-role.yaml

  main.yaml pins the container image by digest, and the image carries the release's code (the API, the sync worker, the portal deployer, the update notifier, the web app, and the scanner with its Trivy binary), so a release can't change underneath you.
• Upgrading is a stack update. Update the stack with the newer template and keep every parameter value. There is no "version" parameter. (The Advanced parameters ArtifactVersionOverride and ImageUriOverride are for Cloud Certainty support only; leave them empty.)
• Published releases never change, so you can roll back with the same procedure.

#Which version am I running?

• Portal: Administration › Licence and about, or the version in the footer.
• Stack output ProductVersion:
  bash

  aws cloudformation describe-stacks --stack-name <stack> \
    --query "Stacks[0].Outputs[?OutputKey=='ProductVersion'].OutputValue" --output text

#How you hear about new releases

• In the portal: with UpdateChecks = Enabled (the default), the stack reads the public release manifest once a day. When a newer release exists, administrators see a banner on the Administration pages, and Licence and about shows the latest version, its summary, whether it is a security release, the release notes and how to upgrade, with the template URL and an AWS CLI command.
• By email or your own tooling: the stack publishes one message per new release to its SNS topic (output UpdateTopicArn). Set UpdateNotificationEmail to subscribe an address, or subscribe your own endpoints.
• Release notes: cloudcertainty.com/watchtower/releases.
• AWS Marketplace: each release is a new version of the listing.
• Release manifest: https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/latest.json names the latest release, whether it contains security fixes, the oldest supported version and the template URLs. You can poll it yourself.

The update check is a plain read of that public file. Nothing about you, your users or your account is sent. With UpdateChecks = Disabled the stack makes no outbound call to the vendor at all.

#Before you upgrade

• Read the release notes. A MAJOR release lists any action you need to take, including spoke role changes.
• Use the same permissions as for the install. The update can create, change or delete IAM roles (three have fixed names), Lambda functions, task definitions and so on.
• No backup step is needed. The DynamoDB table is never replaced by an upgrade, and it has point-in-time recovery. The findings index can always be rebuilt from Security Hub.

#Upgrade the hub

#AWS console

1. Open CloudFormation › Stacks, select the stack and choose Update stack › Make a direct update.
2. Choose Replace current template › Amazon S3 URL, paste the new main.yaml URL and choose Next.
3. Keep the current values of every parameter. New parameters appear with their defaults. Leave the Advanced (support use only) parameters empty. Choose Next twice.
4. Acknowledge "AWS CloudFormation might create IAM resources with custom names" and Submit. The update usually takes 5 to 10 minutes.

#AWS CLI

This keeps the current value of every parameter the new template still has; parameters that are new in this release get their defaults.

STACK=watchtower
REGION=eu-west-2
TEMPLATE_URL=https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/pro/<version>/main.yaml

NEW=$(aws cloudformation get-template-summary --region "$REGION" --template-url "$TEMPLATE_URL" \
  --query 'Parameters[].ParameterKey' --output text | tr '\t\r' '  ')
OLD=$(aws cloudformation describe-stacks --region "$REGION" --stack-name "$STACK" \
  --query 'Stacks[0].Parameters[].ParameterKey' --output text | tr '\t\r' '  ')
PARAMS=()
for k in $NEW; do
  case " $OLD " in *" $k "*) PARAMS+=("ParameterKey=$k,UsePreviousValue=true") ;; esac
done

aws cloudformation update-stack --region "$REGION" --stack-name "$STACK" \
  --template-url "$TEMPLATE_URL" --parameters "${PARAMS[@]}" \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM
aws cloudformation wait stack-update-complete --region "$REGION" --stack-name "$STACK"

To change a value in the same update, replace its UsePreviousValue=true entry with ParameterValue=<value>. To review the changes first, use create-change-set with the same arguments plus --change-set-name upgrade, then execute-change-set.

#What happens during an upgrade

• The new code is installed first. The update runs the one-off installer task from the new release's image. It checks the licence and copies the new code into the artifacts bucket under the new version's folder (about a minute). Only then are the Lambda functions, the scanner task definition and the web app updated. If the installer fails, the update stops and rolls back with nothing else changed. The previous version's folder is kept, so a rollback finds its code.
• The API switches to the new code within seconds; requests in flight complete.
• The web app is redeployed. Users get the new version when they reload the page.
• Data (settings, roles, connections, grants, saved views, tasks and comments, exceptions, integrations and their secrets, repositories, ingest tokens, the audit log) is kept, and so are the findings index, code indexes and scan results in the data bucket. If a release changes the index format, the sync worker rebuilds it with a full sync; the release notes say so.
• Syncs and scans carry on. A sync running during the update finishes with the old code and the next one uses the new code. A Trivy scan already running finishes with the old image; new scans use the new release's image.
• Nothing changes in Security Hub or on GitHub because of an upgrade.
• If the update fails, CloudFormation rolls the stack back to the previous release automatically.

#Upgrade the spoke roles

Spoke roles change rarely; the release notes say when (for example when a release reads a new Security Hub API). To update them, update each spoke stack (or the StackSet) with the new spoke-role.yaml URL, keeping the parameter values. For a single stack, run aws cloudformation update-stack with the same parameters in the delegated administrator account. For a StackSet:

aws cloudformation update-stack-set --region us-east-1 --stack-set-name watchtower-spoke \
  --template-url https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/pro/<version>/spoke-role.yaml \
  --parameters ParameterKey=HubApiRoleArn,UsePreviousValue=true ParameterKey=HubSyncRoleArn,UsePreviousValue=true \
    ParameterKey=ExternalId,UsePreviousValue=true ParameterKey=RoleName,UsePreviousValue=true \
    ParameterKey=AccessLevel,UsePreviousValue=true \
  --capabilities CAPABILITY_NAMED_IAM

Parameters that are new in a release must be left out of --parameters (they get their defaults); the release notes list them. The hub's role ARNs (ApiRoleArn and SyncRoleArn) are named after NamePrefix and the Region and don't change on upgrade, so existing spoke roles keep working.

#Rollback

To go back after a successful upgrade, upgrade "to" the previous release with the same steps and the previous release's template URL. Every release reads the data of the release before and after it, so rolling back by one release is safe. Rolling back further isn't supported; contact support.

#Version support

• Security fixes are released only as a new latest release, so upgrade promptly when the banner, the release notes or latest.json mark a release as a security release.
• Licence and about and latest.json name the oldest supported version.

#Changing plan

Your plan comes from your AWS Marketplace contract, which Watchtower reads from AWS License Manager. Moving between Team, Business and Enterprise needs no stack update: the new features and limits apply as soon as the new contract is active (Watchtower rechecks the licence every 10 minutes, and Licence and about can check it at once). While the agreement changes, the last active plan keeps working for up to 72 hours, so nobody loses access in between.

To move to a higher plan, or from Watchtower Starter to the Watchtower listing, contact [email protected]: we prepare an AWS Marketplace offer that credits the rest of your current contract, and tell you whether the move needs a stack update (moving between Team, Business and Enterprise never does). Your data, settings and grants stay as they are.

To move to a lower plan, contact us before your contract renews. Features the lower plan lacks then show a plan badge, and your data is kept. With a lower limit of monitored accounts, the accounts above it are skipped from the next sync on.

#Troubleshooting