How to use QAmeleon

Getting started

What QAmeleon does

QAmeleon is a QA engineer that runs in your own CI. It checks your dev app every weekday morning and after every deploy, reviews the tickets waiting for QA, and raises what it finds on your board with steps to reproduce.

The morning and deploy checks run on your GitHub Actions. Reviews you start run on your machine. Both use your own Anthropic key, and your code, tickets and pages never reach us.

Getting started

Before you start

Have these ready and setup runs straight through in about ten minutes.

Getting started

Open setup

1. Install QAmeleon with the commands in your invite email: the first downloads QAmeleon with your access key, the second installs that file. They work with any npm, including npm 12. The key is yours alone, so keep it to yourself.
2. Open a terminal in your app's repository folder and run:
   $ qameleon setup
3. Your browser opens the setup page. It is served from your own machine only, with a one-time key, so nobody else can reach it.
4. Keep the terminal open beside it. When setup needs a password or token, an amber box says Your turn, in the terminal. Type it there, not in the browser.

The page has three parts: the steps on the left (Connect, Build, Loop), the current step in the middle, and QAmeleon on the right, who tells you what is happening and changes mood as it goes. A step unlocks once the one before it is done, and you can go back to any finished step from the left.

Setup

Welcome

The first screen says what is about to happen: connect your repo, your dev app and your board, then QAmeleon checks every build and reviews the tickets waiting for QA.

Three promises are on the screen, and they hold for the whole setup: it runs on your CI with your own Anthropic key, nothing is posted until you approve, and its writes are fenced so it never pays, publishes or deletes.

1. Press Let's start.

Setup, Connect 1 of 4

Git: pick your repo

This is where QAmeleon will live. It adds one workflow so it runs in your own CI.

1. Leave GitHub selected.
2. Pick your repository from the list. The list shows every repository your GitHub login can push to.
3. Read the box underneath: it lists the exact files QAmeleon will add in one pull request.
4. Press Continue.

Setup, Connect 2 of 4

Product: point QAmeleon at your dev app

QAmeleon works out how your app signs in and where its API is. You only confirm.

1. Type your dev app's address in Dev app address, for example https://dev.yourapp.com.
2. If your app has no sign-in, like a marketing site or a blog, tick My app has no sign-in. QAmeleon then tests it as a visitor, signed out, and the Accounts step has nothing to ask.
3. Press Have a look (or Enter). QAmeleon opens the app, finds the sign-in form and listens to the API calls.
4. Check the list it found: sign-in page, API address, how the session is kept, the build file and your time zone.
5. If something is missing, a red box says what, and a field appears for it: Sign-in page (for example /account/login) or API host (for example api.yourapp.com). Fill it in and press Have a look again.
6. When it is right, press Looks right.

Setup, Connect 3 of 4

Accounts: add test accounts

Two accounts in different customers let QAmeleon check one customer cannot reach another's data. A borrowed account with real data unblocks deeper review steps. If you ticked My app has no sign-in, this step says no test accounts are needed: press Continue.

The screen lists the accounts it wants: an owner, a second customer, and an optional borrowed account. For each one, choose how QAmeleon may use it:

1. Set Own data or Look only on each account. At least one account needs its own data.
2. Press Add passwords and check.
3. In the terminal, type each account's email and password when asked. Skip an optional account by pressing Enter.
4. QAmeleon signs in with each one to prove it works. Each account shows Signed in, Skipped or Not signed in.
5. When they all work, press Continue.

Setup, Connect 4 of 4

Board: connect your board

Tell QAmeleon where tickets wait for QA, where to put what it finds, and where verified passes go.

Jira

1. Choose Jira.
2. Fill in Jira address (for example yourcompany.atlassian.net), Jira account email and Project key (for example APP).
3. Press Load columns. In the terminal, type the API token for that email. Create one at id.atlassian.com, Security, API tokens.

Trello

1. Choose Trello.
2. Paste the Trello board address (for example https://trello.com/b/AbCd1234/my-board).
3. Give a Short key for cards, for example APP. Cards are then named APP-12, from their number on the board.
4. Press Load columns. In the terminal, type your Trello API key and then your token. Both come from trello.com/power-ups/admin, under your Power-Up's API key.

Notion

1. Choose Notion.
2. In Notion, open the database, choose ••• then Connections, and add your QAmeleon integration. Then paste the Notion database address.
3. Leave Short key for tickets empty to use the prefix of the database's Unique ID property, or type one, for example APP.
4. Press Load columns. In the terminal, type the integration secret from notion.so/my-integrations.

The database needs a Status property, whose options become the columns, and a Unique ID property, which names the tickets.

Pick the three columns

1. Press Continue. The token is saved to your GitHub secrets.

Setup, Build

First run: the dry run

QAmeleon checks everything and posts nothing. You approve the result as the baseline every later run is compared against.

1. Press Start dry run. A list shows each check as it runs, with a tick when done. It takes about a minute.
2. Read the four numbers: checks passed, serious findings, minor findings and test records left behind (this should be 0).
3. Read any findings underneath. Each has a severity and what QAmeleon saw.
4. Press Approve as baseline. Or Run again if you fixed something first.

Setup, Loop

Review and refine

The loop is on. This screen shows what now runs on its own, and offers to review the tickets already waiting.

Review the first batch

1. Check the list of tickets waiting and the cost estimate under it. A review usually costs between $0.30 and $0.65 a ticket on your Anthropic key.
2. Press Review 3 tickets (the number is how many are waiting). The first time, the terminal asks for your Anthropic API key. It is saved for CI.
3. Watch each ticket go from Waiting to Reviewing to Reviewed. Each takes a few minutes, and the line under it says what QAmeleon is doing right now.
4. Read the result on each ticket, then press Approve and publish. Or Not now: nothing is touched.

Merge, then open your console

1. Press Merge pull request. If your repository does not have your QAmeleon access key yet, the terminal asks for it first, checks it and saves it as a secret. Merging adds the workflow and your settings to your main branch, which switches on the morning check. Nothing else in your code changes.
2. Rather merge it yourself on GitHub? Do that, then press I've merged it, check again.
3. Press Open your console. Setup is done.

The console

Opening the console

The console is home after setup. Like setup, it runs on your own machine and opens in your browser.

The menu on the left has eight pages: Today, Morning report, Queue, Waiting for you, History, Connections, Controls and Settings. Queue and Waiting for you show a count; the Waiting count turns amber when something needs you. QAmeleon sits on the right and says, in one line, what matters on the page you are on.

The console shares memory with CI, so a pause or an approval you make here reaches the next CI run.

Console

Today

The front page: is QAmeleon publishing, does anything need you, and how did this morning go.

What you see

• The banner says how publishing stands: QAmeleon publishes on its own (green), Approval mode or Tightened (amber), or Paused (red), with Go to Controls when publishing is paused or tightened.
• Four cards: Waiting for you, the number of tickets in your review column, the last 24 hours (published, held, bugs raised and cost), and the share of findings your team accepted.
• Rejected findings: a bar showing how many findings people rejected, with an amber mark at the limit. Above it, QAmeleon stops publishing on its own.
• This morning: the morning check's result with Read it, the build on dev and when it was deployed, and the last batch.

What to do

Click a card to go to that page. If Waiting for you is above 0, start there.

Console

Morning report

The latest morning or deploy check, wherever it ran, plus the last runs.

• The title is the verdict: Nothing to do (green), Worth a look (amber) or Needs attention (red). The banner under it names the check, when it ran and the one-line summary.
• Recent runs lists the last seven, each with its colour and summary.
• The full report follows: builds, API changes, test results, speed, access checks, page scans, and what the run wrote on dev (always its own test data, with "nothing left behind").

Before the first morning check runs, the page says No report yet. See Reading the results for what each colour means.

Console

Queue

The tickets in your review column. Pick some and QAmeleon reviews them on dev.

1. Press Refresh if you moved cards in since the page was read.
2. Pick the tickets to review, or press Pick all. Up to 10 go in one batch. Each ticket shows how many steps it has to verify (or that it will be reviewed against its description) and when it was last reviewed.
3. Check the estimated cost at the bottom, worked out from your board's past reviews.
4. Press Review 3 tickets (the number is how many you picked). Each ticket runs in its own session behind the write fence, and the list shows Waiting, Reviewing and Reviewed.
5. When the batch is done, the page shows what was published by QAmeleon (with any bugs raised and tickets moved) and what was held for a person, with the cost.
6. If anything was held, press Go to Waiting for you. Otherwise Back to the queue.

Console

Waiting for you

Every review a guard held, each with the reason and what accepting it means. Guards cannot be switched off, only accepted one review at a time, by name.

What each card shows

• The ticket, and a chip for each guard that held it.
• A summary, how many steps passed, failed or were blocked, when it ran and what it cost.
• What the guard means in plain words, the findings with their evidence, and any flagged lines from the ticket, shown as text and never followed.

What to do

1. Type Your name. Approvals are recorded on the ticket under it.
2. Read the card. Tick each box it asks for, for example "I have checked the evidence and agree it is critical".
3. Press the card's button:
   • Approve and publish puts the review on the board.
   • Run again appears when nothing was reviewed yet, for example when the app was down.
   • Sign off and move appears when a review passed but a hold label asked for a person to sign it off.
4. If reviews were held only by a pause or tightening that is now over, press Publish through the guards at the top. Anything another guard holds stays.

See What gets held, and why for every guard.

Console

History

Everything QAmeleon did and who approved what, newest first.

Each batch shows when it ran, who started it and what it cost, then a line per ticket: Published (with any bugs raised and their severity, and where the ticket moved) or Held (with the guards). Every pause, resume, reset, approval and sign-off is there too, with the person's name.

Console

Connections

Where the report goes. Every morning check and every deploy check is sent to each place switched on here, from your own CI.

Add a place to send it

1. Under Add a place to send it, choose Slack, Microsoft Teams, Email, Datadog or Any webhook.
2. Fill in any fields it asks for (email and Datadog have some; the others have none).
3. Press Save and type the secrets, then type the address or key in the terminal that runs the console. It goes straight to your repository's GitHub secrets.
4. Press Send a test on its row. Check it arrived.
5. Commit qa.config.json. The amber banner reminds you: until it is committed, the morning check in CI still sends to the old list.

Each row

A row shows the destination, where it posts, and a chip: On, Off or Needs a secret. Use Switch off or Switch on to pause one, Send a test to try it, and Remove (then Remove it to confirm) to delete it.

Step by step for each one: Slack, Teams, Email, Datadog, Any webhook.

Console

Controls

Stop, start and reset. Every change is recorded with a name.

1. Type Your name.
2. Kill switch. Press Pause publishing to stop all publishing at once. Reviews in flight still finish and wait for you. Press Resume publishing to turn it back on.
3. Automatic tightening. The panel shows how many recent findings were rejected. Above the limit, QAmeleon stops publishing on its own and asks first. Read what was rejected, then press Reset.
4. The guards. The list at the bottom shows every guard, always on, and whether publishing is in auto or approval mode.

Console

Settings

What QAmeleon is connected to, and how to change it.

• Repository, app and board: the repository, the board and project, and the apps it tests.
• Change what is connected: Open setup opens the setup wizard again, starting from what is connected now. Use it to connect another repository, point at a different app or API, change the test accounts, or move to another board. Changes reach CI through a pull request, as the first time.
• Where the report goes: Go to Connections.

Send the report

Slack

The one-screen summary in a channel, every morning and after every deploy.

1. Make a Slack webhook

1. Go to api.slack.com/apps and press Create New App, then From scratch. Call it QAmeleon and pick your workspace.
2. In the app's menu, open Incoming Webhooks and switch Activate Incoming Webhooks on.
3. Press Add New Webhook to Workspace, pick the channel (for example #qa) and press Allow.
4. Copy the webhook address. It starts with https://hooks.slack.com/services/. Treat it like a password: anyone with it can post to that channel.

2. Add it in QAmeleon

1. In the console, open Connections and press Slack.
2. Press Save and type the secrets.
3. In the terminal, paste the webhook address when asked. It is hidden as you type and saved as QA_SLACK_WEBHOOK_URL.
4. Press Send a test. A test message appears in the channel.
5. Commit qa.config.json.

Send the report

Microsoft Teams

The summary in a Teams channel, through a Workflows webhook.

1. Make a Teams workflow

1. In Teams, open the channel, press More options (the three dots) next to it and choose Workflows.
2. Pick the template Post to a channel when a webhook request is received, name it QAmeleon, and confirm the team and channel.
3. Copy the address it gives you at the end.

2. Add it in QAmeleon

1. In Connections, press Microsoft Teams, then Save and type the secrets.
2. Paste the address in the terminal. It is saved as QA_TEAMS_WEBHOOK_URL.
3. Press Send a test, then commit qa.config.json.

Send the report

Email

The summary and the full report by email, sent through your own mail server.

1. In Connections, press Email.
2. Fill in Send to (one or more addresses, separated by commas), Mail server (for example smtp.office365.com), and optionally From and Port (587 is usual).
3. Press Save and type the secrets. In the terminal, type the mail server's sign-in name, then its password (hidden). They are saved as QA_SMTP_USER and QA_SMTP_PASSWORD.
4. Press Send a test, then commit qa.config.json.

Send the report

Datadog

An event for every run, plus numbers to graph: tests failed, sign-in time and open findings.

1. In Datadog, open Organization Settings, then API Keys, and create a key called QAmeleon. Copy it.
2. In Connections, press Datadog.
3. Choose your Datadog site: the one in your Datadog address, for example datadoghq.eu. Optionally add Extra tags such as team:web, env:dev.
4. Press Save and type the secrets and paste the API key in the terminal. It is saved as QA_DATADOG_API_KEY.
5. Press Send a test, then commit qa.config.json.

Send the report

Any webhook

The full report as JSON, for Zapier, Make, n8n or your own tool.

1. Create a webhook trigger in your tool and copy its address. It must start with https://.
2. In Connections, press Any webhook, then Save and type the secrets.
3. Paste the address in the terminal. It is saved as QA_REPORT_WEBHOOK_URL.
4. Press Send a test and check your tool received it, then commit qa.config.json.

Reference

Reading the results

QAmeleon changes colour with the result, in the report, the review message and on your board. His face changes too, so you can read it at a glance in any colour.

Severity of a finding

QAmeleon sets the severity from what the problem does to the customer, so the same problem gets the same severity on every ticket. A bug an earlier review rated keeps that rating. A review that would change it waits for a person.

Step results in a review

Each "How to verify manually" step is marked pass, fail, blocked (it says exactly what would unblock it, often a test account or some data) or Needs a person (it needs access QAmeleon does not have, such as a test card or cloud logs). A ticket moves on only when every step passed. If any step failed, it goes back to your findings column. If steps were only blocked, it stays where it is until they can be checked.

Reference

What gets held, and why

QAmeleon publishes on its own unless one of these holds the review for a person. They are always on. You can add hold labels, not remove guards.

Things QAmeleon never does

• Pay, publish, send, invite or check out in your app. Those writes are always blocked.
• Write to a look-only account, or delete anything it did not create.
• Follow instructions written in a ticket.
• Move a ticket out of a column it was not reviewing, or into one you did not pick.
• Post a report with anything that looks like a password or token in it. If one is found, nothing goes out.

Reference

Command line

Everything the setup page and the console do, you can also do from the terminal. qa and qauto still work as short names.

Reference

Troubleshooting