Accelerating software program supply with agentic QA automation utilizing Amazon Nova Act – Half 2

Manufacturing high quality assurance (QA) workflows require greater than particular person take a look at execution. You will need to manage exams into regression suites that run as a batch, and combine them into steady integration and steady supply (CI/CD) pipelines in order that take a look at outcomes gate deployments routinely.

In a earlier publish, we launched QA Studio, a reference answer for agentic QA automation constructed with Amazon Nova Act. We confirmed tips on how to outline particular person use instances in pure language, run them on demand by AI-powered visible navigation, and examine execution artifacts with full trajectory visibility.

On this publish, we lengthen that basis to exhibit how QA Studio addresses batch regression testing and pipeline integration by take a look at suites that manage and parallelize execution, and a command-line interface that brings agentic testing into automated CI/CD pipelines.

With QA Studio, you’ll be able to group particular person use instances, every validating a particular consumer journey, into collections known as take a look at suites that run collectively. These take a look at suites help structured regression testing throughout practical areas.

Suites execute as a batch with parallel processing: when a set runs, every use case executes independently by itself Amazon Elastic Container Service (Amazon ECS) on AWS Fargate employee process. As a result of every use case runs by itself Fargate employee process, a set of 20 exams can execute concurrently fairly than sequentially. This reduces whole suite length relative to serial execution.

You may manage suites by practical space, launch stage, or testing objective. Examples embody smoke exams that validate crucial paths on each deployment, regression suites that run throughout the complete utility, and integration exams that confirm cross-feature workflows earlier than launch.

Creating and managing take a look at suites

You create take a look at suites within the QA Studio internet interface by offering a reputation, description, and elective tags, then add current use instances to the suite. Every use case retains its personal configuration, together with beginning URL, variables, secrets and techniques, and headers. When the suite executes, these configurations apply independently to every use case.

Determine 1 — Take a look at suite element web page displaying use instances and execution historical past

Suite execution and outcomes

When a set executes, QA Studio creates particular person execution data for every use case and dispatches them to the employee queue. The suite execution web page offers an mixture view: what number of use instances succeeded, failed, or are nonetheless operating. You may drill into particular person execution outcomes to evaluate trajectory logs, screenshots, and session recordings for any failed take a look at.

Every suite maintains its personal execution historical past, providing you with a longitudinal view of regression stability. Constant passes construct confidence within the examined performance, whereas intermittent failures spotlight areas that want consideration.

Determine 2 — Suite execution outcomes displaying mixture standing and particular person use case outcomes

The QA Studio internet interface works properly for interactive take a look at creation and on-demand execution. CI/CD pipelines require a special interface: command-line execution with structured output, non-interactive authentication, and exit codes that combine with pipeline orchestrators.

The QA Studio CLI (qa-studio) offers this interface. It connects to the identical API backend as the net utility. However as a substitute of dispatching exams to Fargate staff, it runs them with Amazon Nova Act on the machine the place the CLI executes, corresponding to a CI/CD runner. Outcomes are reported again to the QA Studio deployment.

Putting in and authenticating

The QA Studio CLI is a part of the venture’s GitHub repository. Clone the repository, then set up the CLI as a Python package deal with its elective runner dependencies:

pip set up -e "./qa-studio-cli[runner]"

For CI/CD environments, the CLI helps OAuth 2.0 shopper credentials authentication. You create an OAuth shopper within the QA Studio internet interface with the required scopes (api/suite.learn, api/suite.write, api/executions.learn, api/executions.write, api/usecases.learn, api/usecases.execute), then configure the credentials as pipeline surroundings variables:

export OAUTH_CLIENT_ID="your-client-id"
export OAUTH_CLIENT_SECRET="your-client-secret"
export OAUTH_TOKEN_ENDPOINT="https://your-cognito-domain.auth.area.amazoncognito.com/oauth2/token"

The CLI routinely requests and caches entry tokens, refreshing them once they expire. No interactive browser login is required.

Working exams and suites

The qa-studio run command executes particular person use instances or total take a look at suites:

# Run a single take a look at
qa-studio run --usecase-id test-123

# Run a take a look at suite
qa-studio run --suite-id suite-456

Surroundings and variable overrides

CI/CD pipelines usually must run the identical exams towards totally different environments. The CLI helps a number of override mechanisms that modify take a look at conduct with out altering the take a look at definitions saved in QA Studio.

The --base-url flag replaces the area of the beginning URL whereas preserving the trail and question parameters. A single take a look at can then goal improvement, staging, or manufacturing environments:

# Run towards staging
qa-studio run --suite-id suite-456 --base-url https://staging.instance.com

# Run towards manufacturing
qa-studio run --suite-id suite-456 --base-url https://manufacturing.instance.com

The --var flag overrides template variables outlined within the use case. Variables referenced in take a look at steps utilizing {{VariableName}} syntax are substituted at runtime. This helps environment-specific configuration with out duplicating take a look at definitions:

# Override credentials for a particular surroundings
qa-studio run --usecase-id test-123 
    --var username=staging_user 
    --var password=staging_pass 
    --var api_key=staging_key_123

The --region flag controls which AWS Area the browser runs in, and --model-id selects the Amazon Nova Act mannequin model:

qa-studio run --usecase-id test-123 
    --region eu-central-1 
    --model-id nova-act-v1.0

Headers and secrets and techniques

Use instances can outline customized HTTP headers which can be despatched with each request throughout take a look at execution. That is helpful for authentication tokens, characteristic flags, or customized identifiers that the applying underneath take a look at requires. Headers are configured within the use case settings and utilized routinely throughout each internet interface and CLI execution.

Secrets and techniques present safe storage for delicate values like passwords, API keys, or tokens. Secrets and techniques are saved in AWS Secrets and techniques Supervisor, encrypted at relaxation. QA Studio is designed in order that secret values aren’t written to execution logs or historical past data. Take a look at steps reference secrets and techniques by identify, and the precise values are retrieved at runtime. This separation means CI/CD pipelines can execute exams that require credentials with out exposing these credentials in pipeline configuration or logs.

Exit codes and pipeline integration

The CLI makes use of exit codes that map on to pipeline success and failure states:

This three-state mannequin permits pipelines to tell apart between take a look at failures (exit code 1) and infrastructure issues (exit code 2). You may configure totally different notification or retry methods for every case.

The --format flag controls output formatting. The default json format offers structured output for programmatic consumption. The human format offers a readable abstract for pipeline logs:

qa-studio run --suite-id suite-456 --format human

Throughout execution, the CLI creates execution data in QA Studio, updates step statuses in actual time, and uploads artifacts together with trajectory logs and session recordings. You may monitor CLI-triggered executions from the net interface alongside manually triggered runs, sustaining a unified execution historical past no matter how exams have been initiated.

Determine 3 — CLI execution output displaying take a look at outcomes

The next examples exhibit QA Studio integration with frequent CI/CD instruments. Every instance assumes OAuth shopper credentials and AWS credentials are saved as pipeline secrets and techniques.

GitHub Actions

identify: QA Assessments

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  smoke-tests:
    runs-on: ubuntu-latest
    steps:
      - makes use of: actions/checkout@v4

      - identify: Arrange Python
        makes use of: actions/setup-python@v4
        with:
          python-version: '3.11'

      - identify: Set up QA Studio CLI
        run: pip set up -e "./qa-studio-cli[runner]"

      - identify: Configure AWS Credentials
        makes use of: aws-actions/configure-aws-credentials@v4
        with:
          aws-access-key-id: ${{ secrets and techniques.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets and techniques.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-east-1

      - identify: Run Smoke Assessments
        env:
          OAUTH_CLIENT_ID: ${{ secrets and techniques.OAUTH_CLIENT_ID }}
          OAUTH_CLIENT_SECRET: ${{ secrets and techniques.OAUTH_CLIENT_SECRET }}
          OAUTH_TOKEN_ENDPOINT: ${{ secrets and techniques.OAUTH_TOKEN_ENDPOINT }}
        run: |
          qa-studio run 
            --suite-id ${{ vars.SMOKE_TEST_SUITE_ID }} 
            --base-url https://staging.instance.com 
            --format human

      - identify: Add Artifacts
        if: all the time()
        makes use of: actions/upload-artifact@v3
        with:
          identify: test-artifacts
          path: ~/.qa-studio/artifacts/

The if: all the time() situation on the artifact add step verifies that take a look at recordings and logs are preserved even when exams fail, offering the debugging context that you must examine failures.

GitLab CI

phases:
  - take a look at

smoke-tests:
  stage: take a look at
  picture: python:3.11
  before_script:
    - pip set up -e "./qa-studio-cli[runner]"
  script:
    - |
      qa-studio run 
        --suite-id $SMOKE_TEST_SUITE_ID 
        --base-url https://staging.instance.com 
        --format human
  variables:
    AWS_DEFAULT_REGION: us-east-1
  artifacts:
    when: all the time
    paths:
      - ~/.qa-studio/artifacts/
    expire_in: 7 days
  solely:
    - primary
    - merge_requests

regression-tests:
  stage: take a look at
  picture: python:3.11
  before_script:
    - pip set up -e "./qa-studio-cli[runner]"
  script:
    - |
      qa-studio run 
        --suite-id $REGRESSION_SUITE_ID 
        --timeout 7200 
        --format human
  variables:
    AWS_DEFAULT_REGION: us-east-1
  artifacts:
    when: all the time
    paths:
      - ~/.qa-studio/artifacts/
    expire_in: 7 days
  solely:
    - schedules

GitLab CI variables (OAUTH_CLIENT_ID, OAUTH_CLIENT_SECRET, OAUTH_TOKEN_ENDPOINT, AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY) ought to be configured as protected and masked CI/CD variables within the venture settings. The regression-tests job makes use of the GitLab schedule set off, operating solely when triggered by a pipeline schedule fairly than a code push.

Jenkins

pipeline {
    agent any

    surroundings {
        AWS_DEFAULT_REGION = 'us-east-1'
    }

    phases {
        stage('Setup') {
            steps {
                sh '''
                python3.11 -m venv venv
                . venv/bin/activate
                pip set up -e "./qa-studio-cli[runner]"
                '''
            }
        }

        stage('Smoke Assessments') {
            steps {
                withCredentials([
                    string(credentialsId: 'oauth-client-id', variable: 'OAUTH_CLIENT_ID'),
                    string(credentialsId: 'oauth-client-secret', variable: 'OAUTH_CLIENT_SECRET'),
                    string(credentialsId: 'oauth-token-endpoint', variable: 'OAUTH_TOKEN_ENDPOINT'),
                    string(credentialsId: 'aws-access-key-id', variable: 'AWS_ACCESS_KEY_ID'),
                    string(credentialsId: 'aws-secret-access-key', variable: 'AWS_SECRET_ACCESS_KEY')
                ]) {
                    sh '''
                    . venv/bin/activate
                    qa-studio run 
                        --suite-id ${SMOKE_TEST_SUITE_ID} 
                        --base-url https://staging.instance.com 
                        --format human
                    '''
                }
            }
        }
    }

    publish {
        all the time {
            archiveArtifacts artifacts: '~/.qa-studio/artifacts/**/*',
                allowEmptyArchive: true
        }
    }
}

Jenkins makes use of the withCredentials block to inject secrets and techniques into the construct surroundings with out exposing them in console output. The publish.all the time block archives take a look at artifacts no matter construct consequence.

Take a look at suites and CI/CD integration lengthen QA Studio from an interactive take a look at creation device right into a platform for steady high quality assurance. Take a look at suites manage regression protection into manageable collections with parallel execution. The CLI brings agentic take a look at execution into automated pipelines with surroundings overrides, safe credential dealing with, and exit codes that map to pipeline success and failure states.

These capabilities construct on the inspiration described in our earlier publish: pure language take a look at definitions, AI-powered visible navigation, and end-to-end trajectory visibility. Collectively, they exhibit how agentic QA automation with Amazon Nova Act can combine into current software program supply workflows, offering automated high quality suggestions with out requiring you to take care of framework-specific take a look at code.

In a future publish, we plan to discover how agentic take a look at automation can lengthen to cell purposes.

The QA Studio reference answer, together with take a look at suites and CLI integration, is on the market on GitHub. For deployment directions and detailed documentation, see the venture README.

In regards to the authors