Installation guide

This guide is for the platform team installing Cloud Certainty Watchtower in their AWS account. The install is one CloudFormation stack (the hub). The AWS account that holds your AWS Security Hub findings, normally the Security Hub delegated administrator, either runs the hub itself or gets a small role stack (the spoke). With IAM Identity Center or another SAML IdP, the hub takes two steps: deploy the stack, configure the SAML application, then update the stack. For the short version, see Getting started.

#Before you start

• Security Hub. Watchtower reads AWS Security Hub CSPM findings (the AWS Security Finding Format, ASFF). It works best with an organisation-wide setup: a delegated administrator account and cross-Region aggregation, so that one account and one Region (the aggregation Region) hold the findings of every member account and linked Region. The FAQ shows the setup. Watchtower also works with a single account, Region by Region.
• Region. Any Region with AWS Lambda, Amazon API Gateway (HTTP APIs), Amazon DynamoDB, Amazon S3, Amplify Hosting, Amazon Cognito, AWS License Manager and Amazon ECS on Fargate. The templates are always loaded from the vendor's public bucket in us-east-1 (CloudFormation accepts a template URL from any Region), and all product code comes from the container image. The hub's Region doesn't limit which Security Hub Regions Watchtower reads: list them in LocalRegions and in each connection.
• Account. Deploy the hub in the account that subscribed on AWS Marketplace: the licence is delivered to that account through AWS License Manager. To run the hub in another account of your organisation, distribute the licence to it with a License Manager grant first.
• Permissions. You need permission to create the resources in the stack: IAM roles (some have fixed names), Lambda, ECS, EC2 security groups, Cognito, API Gateway, Amplify, DynamoDB, S3, SNS, EventBridge, KMS and CloudWatch Logs.
• Network. A VPC with at least one subnet with outbound HTTPS access, for the one-off installer task and the scanner tasks (see Networking). Nothing else runs in your VPC.
• Identity. Decide how users will sign in: IAM Identity Center, another SAML 2.0 IdP (Entra ID, Okta and others), an OIDC IdP, or Cognito-managed users. SAML IdPs other than IAM Identity Center, and OIDC, need the Team plan or above.

#Choose where the hub runs

Watchtower needs one connection to the account and Region that hold your findings. There are two good layouts.

With layout A, the hub's own local connection also reads Security Hub in the hub account (in LocalRegions, or the stack's Region). If that account is a member of your organisation, its findings also arrive through the delegated administrator connection, so they appear twice for administrators. Give your users grants on the delegated administrator connection only. If Security Hub isn't enabled in the hub account, the local source reports "Security Hub is not enabled" under Administration › Sync; that is harmless.

You don't need spokes in member accounts: the delegated administrator's aggregation Region already holds their findings, and connecting both would index the same findings twice. Add other accounts only for Security Hub setups outside your organisation's aggregation.

#Read or Triage

Each connection's IAM role has an access level, set by the owner of the account: LocalAccessLevel on the hub stack, AccessLevel on a spoke stack. It is the ceiling for everyone, administrators included; grants in Watchtower narrow it per user or group but never widen it.

With Triage on the delegated administrator, a change made in Watchtower is a change in Security Hub for that member account's finding, visible in the Security Hub console too, and CloudTrail records the person who made it. Only users with a triage grant on the finding's account and Region can make it. Neither level allows any other write: Watchtower never changes your AWS resources.

#1. Subscribe

1. Open Watchtower in AWS Marketplace and choose Continue to Subscribe. Choose the plan and the contract length (Team is the base plan; Business and Enterprise are add-ons to it; Starter is the separate Watchtower Starter listing) and accept the offer. AWS License Manager now holds your licence.
2. Choose Continue to Configuration, select the fulfilment option and the Region, then open the CloudFormation launch link from the usage instructions.

#2. Launch the hub stack

In the CloudFormation console, fill in the parameters (see the reference). Acknowledge "AWS CloudFormation might create IAM resources with custom names" and create the stack. Creation takes about 10 minutes.

The quick-create link, with the release version filled in:

https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/pro/<version>/main.yaml&stackName=watchtower

From the AWS CLI (--region is the Region you deploy to):

aws cloudformation create-stack --region <your-region> --stack-name watchtower \
  --template-url https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/pro/<version>/main.yaml \
  --parameters file://params.json \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM

params.json lists the parameters, for example:

[
  {"ParameterKey": "VpcId", "ParameterValue": "vpc-0123456789abcdef0"},
  {"ParameterKey": "SubnetIds", "ParameterValue": "subnet-0123456789abcdef0,subnet-0fedcba9876543210"},
  {"ParameterKey": "BootstrapAdminEmail", "ParameterValue": "[email protected]"},
  {"ParameterKey": "IdentityProvider", "ParameterValue": "IAMIdentityCenter"},
  {"ParameterKey": "LocalAccessLevel", "ParameterValue": "Read"},
  {"ParameterKey": "LocalRegions", "ParameterValue": "eu-west-1"}
]

#What happens during creation

The product's code (the Lambda functions, the web app and the scanner) ships inside the subscription-protected container image, not as public downloads. Early in stack creation:

1. CloudFormation creates a private S3 bucket for the code in your account (output ArtifactsBucketName; all public access blocked, encrypted, TLS only) and a second private bucket for the findings index, code indexes and scan results (output DataBucketName).
2. The ArtifactInstall custom resource starts one Fargate task from the product image in your SubnetIds. The task checks your AWS Marketplace licence in AWS License Manager, verifies every file of the code bundle against the SHA-256 checksums in the image, and copies it into the artifacts bucket under the release's version. It runs for about a minute, with a read-only root filesystem and a security group without inbound rules.
3. The Lambda functions (the API, the sync worker and the update notifier) and the web app are then created from that bucket. An EventBridge rule starts the sync worker every 5 minutes.
4. With EnableScanner = Enabled (the default), the stack also creates the scanner's Fargate task definition and its security group (no inbound rules, outbound HTTPS only).

If the installer task fails, stack creation fails and rolls back. The error message gives the task's exit code and its log stream (log group /<NamePrefix>/installer); see Troubleshooting.

#3. Connect your identity provider

Until SamlMetadataUrl is set, the portal falls back to Cognito sign-in, which has no users. Nobody can sign in at that point, so the stack is safe in that state.

#4. First sign-in

Open the PortalUrl output and sign in as BootstrapAdminEmail. The first account that signs in with that (verified) email becomes the bootstrap administrator and stays bound to it (see the admin guide).

The local connection to the hub's own account exists from the start. It uses the stack's local role, /watchtower/<NamePrefix>-local-<Region>, which may do what LocalAccessLevel allows, in the Regions of LocalRegions. The sync worker reads its findings within 5 minutes. Administrators see every finding; other users see nothing until an administrator grants them access (admin guide).

#5. Connect the delegated administrator

Skip this step if the hub runs in the delegated administrator account (layout B).

spoke-role.yaml creates one role, /watchtower/<RoleName>, that only the hub's API and sync roles can assume, only with your external ID, and only with a source identity (the user's email, or watchtower-sync for the sync worker). Its template URL is the hub stack output SpokeTemplateUrl:

https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/pro/<version>/spoke-role.yaml

The role is the upper limit of what Watchtower can do in that account; grants narrow it per user or group but never widen it. It has the Security Hub read actions, organizations:ListAccounts and organizations:DescribeOrganization (for account names), and with Triage, securityhub:BatchUpdateFindings on the account's hub. Nothing else. IAM is global, so create the role once per account, in any Region. The stack's outputs are RoleArn (enter it in the connection), AccessLevel (choose the same level for the connection) and ExternalIdHint.

#Quick-create link

The hub's role ARNs can be filled in through the link:

https://console.aws.amazon.com/cloudformation/home#/stacks/quickcreate?templateURL=https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/pro/<version>/spoke-role.yaml&stackName=watchtower-spoke&param_HubApiRoleArn=<ApiRoleArn output>&param_HubSyncRoleArn=<SyncRoleArn output>

#AWS CLI

cat > spoke-params.json <<'EOF'
[
  {"ParameterKey": "HubApiRoleArn", "ParameterValue": "<ApiRoleArn output>"},
  {"ParameterKey": "HubSyncRoleArn", "ParameterValue": "<SyncRoleArn output>"},
  {"ParameterKey": "ExternalId", "ParameterValue": "<a secret of at least 16 characters>"},
  {"ParameterKey": "AccessLevel", "ParameterValue": "Read"}
]
EOF
aws cloudformation create-stack --region eu-west-1 --stack-name watchtower-spoke \
  --template-url https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/pro/<version>/spoke-role.yaml \
  --parameters file://spoke-params.json --capabilities CAPABILITY_NAMED_IAM

Run it with credentials for the delegated administrator account.

#Several accounts (StackSets)

If you need the role in several accounts (for example two organisations' delegated administrators, or standalone accounts outside your aggregation), deploy it with a StackSet. Use one external ID for the whole StackSet and enter that same external ID for every connection you create for those accounts.

# From the management account or a delegated StackSets administrator.
aws cloudformation create-stack-set --region us-east-1 --stack-set-name watchtower-spoke \
  --template-url https://cloudcertainty-watchtower-us-east-1.s3.us-east-1.amazonaws.com/pro/<version>/spoke-role.yaml \
  --parameters file://spoke-params.json \
  --capabilities CAPABILITY_NAMED_IAM \
  --permission-model SERVICE_MANAGED \
  --auto-deployment Enabled=true,RetainStacksOnAccountRemoval=false

# One Region is enough: IAM is global.
aws cloudformation create-stack-instances --region us-east-1 --stack-set-name watchtower-spoke \
  --deployment-targets OrganizationalUnitIds=ou-abcd-11111111 --regions eu-west-1

Service-managed StackSets don't deploy to the management account; use a normal stack there if Watchtower should reach it. To change AccessLevel later, update the stack or StackSet, then the access level of the connections.

#Add the connection

In Administration › Connections, add a connection with:

• Id: a short name used in links, for example security. It can't be changed later.
• Name and AWS account id.
• Access level: the same as the spoke stack's AccessLevel.
• Role ARN: the spoke stack's RoleArn output, arn:aws:iam::<account>:role/watchtower/<RoleName>. It must be in that account and under the /watchtower/ path.
• External ID: the same secret you gave the spoke stack. It is stored and never shown again.
• Hub Regions: the aggregation Region. Without cross-Region aggregation, list every Region where Security Hub runs in that account; each becomes its own source.

Then test it. For each hub Region, the test reports whether Security Hub is enabled, the aggregation Region and linked Regions when aggregation is set up, the enabled standards, a sample count of findings, or the AWS error. The sync worker reads the new source within 5 minutes. See the admin guide.

#Parameter reference

#Hub stack (main.yaml)

#Spoke stack (spoke-role.yaml)

HubApiRoleArn, HubSyncRoleArn, ExternalId, RoleName (default watchtower-spoke) and AccessLevel (default Read), described in Connect the delegated administrator.

#Networking

The Lambda functions don't run in your VPC: they reach STS, Security Hub, AWS Organizations, GitHub and SonarCloud over public endpoints. Only two kinds of one-off Fargate tasks run in SubnetIds, and nothing accepts inbound connections. Both security groups have no inbound rules and one outbound rule, TCP 443.

Use a NAT gateway, an internet gateway (public subnets with SubnetsArePublic = true) or VPC endpoints for the AWS services. GitHub and the Trivy database always need internet egress.

GitHub Enterprise Server must be reachable over HTTPS from the AWS Lambda functions (for indexing, pull requests and issues), which run outside your VPC, and from the scanner's subnets. A server reachable only inside a private network can't be used in this release.

#Encryption and KMS

Without KmsKeyArn, the DynamoDB tables, the log groups, the update SNS topic, the artifacts and data buckets and the integration secrets use AWS owned or AWS managed keys and SSE-S3.

With KmsKeyArn, your key encrypts all of them. The key must be a symmetric customer managed key in the stack's Region (give its ARN, not an alias), and its key policy must:

• let the account's IAM policies grant its use (the default key policy statement for the account root does this). The stack's roles use it only through DynamoDB, S3 and Secrets Manager;
• allow logs.<region>.amazonaws.com to use it for the stack's log groups;
• allow sns.amazonaws.com to use it for the update topic.

The identity that creates or updates the stack also needs kms:Decrypt on the key, because Lambda reads the function code from the encrypted artifacts bucket with the deploying identity's permissions.

The key protects Watchtower's own data. Security Hub keeps its own encryption.

#Stack outputs

#Updating

Each release is a new template; there is no version parameter. To upgrade, update the stack with the new release's template (Update stack › Replace current template) and keep every other parameter value. See the upgrading guide.

To change a setting such as LocalAccessLevel, LocalRegions, EnableScanner or SamlMetadataUrl, update the stack with the current template and change only that parameter.

#Uninstalling

1. Before you delete anything, revoke the exceptions that write back to Security Hub (Exceptions) and wait for the next sync: Watchtower then sets the findings they suppressed back to NEW. After the stack is gone, nothing will. Workflow statuses and notes written to Security Hub stay as they are either way.

2. Delete the hub stack. Two resources are retained on purpose:

   • the DynamoDB table <NamePrefix>-table, with its settings, roles, connections, grants, tasks, exceptions, saved views, integrations, repositories and the audit log. It has deletion protection; to delete it, turn that off, then delete the table. A new stack with the same NamePrefix can't be created while it exists;
   • the data bucket (output DataBucketName), with the findings index, code indexes and scan results. Everything in it can be rebuilt by a sync; empty and delete it when you no longer need it.

   Everything else is deleted: the cache table (with the trend history and usage counters), the Cognito user pool (Cognito-only users are lost), the artifacts bucket, the Lambda functions, the task definitions and the log groups.

3. Delete the integration secrets. Tokens and GitHub App keys you added under Administration › Integrations are Secrets Manager secrets named <NamePrefix>/integrations/<id>, created by Watchtower rather than by the stack. Delete them, and revoke the tokens or uninstall the GitHub App on the GitHub side.

4. Delete the spoke stacks or the StackSet in the other accounts.

5. Delete the SAML or OIDC application in your identity provider.

Branches and pull requests Watchtower opened in your repositories stay there; close them if you like.

#Troubleshooting