Continuous QA from GitHub

Without GitHub, you start rounds of testing yourself. With a repository connected, Replay QA tests each new build as it goes live, attributes journeys and bugs to the build that introduced them, and reports on pull requests.

What you get

• A version for every tested build, listing what changed, the journeys created or updated, and the bugs found.
• Main-branch testing whenever a new deployment of the default branch is live.
• Pull request testing on each revision, with results posted to the pull request as a comment and a check.
• Optional issue filing so each bug lands in GitHub Issues, Linear, or Jira.

Prerequisites

• A GitHub repository for the app.
• The Replay QA GitHub App installed on that repository. Someone with admin rights on the repository has to approve the installation; the project's GitHub settings show the install link once a repository is set.
• A way for Replay QA to know when a build is live (below).

Connect the repository

Tell the project chat to connect a GitHub repository, or set it in the project settings. You provide the repository URL and confirm the default branch, then choose how main-branch and pull request testing should run.

Main-branch testing

Mode	What it does
None	The main branch is not tested automatically.
Detect main branch deployments	You give Replay QA the deployment URL. It watches that URL and starts a round when a new build goes live. Use this when your host does not report deployments to GitHub.
Listen to GitHub events	Replay QA starts a round when GitHub reports a deployment of the default branch. Vercel, Netlify, and other hosts that create GitHub Deployments work without extra setup.

Main-branch rounds may create, update, or remove the project's journeys, because the main branch is the app.

Pull request testing

Mode	What it does
None	Pull requests are not tested.
Listen to GitHub events	Each pull request revision is tested against its preview deployment, and the results are posted back to the pull request.

Replay QA learns the preview URL from the GitHub Deployment for the commit, or from a commit status that carries a preview URL, which is how Netlify reports previews. Hosts that do neither need to publish a GitHub Deployment with an environment URL.

What happens on a pull request

1. You push to the pull request. Replay QA records a new version and waits for the preview deployment.
2. When the preview is live, it runs the project's journeys against it. Pull request rounds may add journeys for the change; they do not modify the project's existing journeys. Journeys added on a pull request are shown with that pull request.
3. Replay QA posts one comment on the pull request and edits it in place as results come in, and reports a Replay QA check on the commit.
4. Another push supersedes the running round. Closing the pull request cancels it.

Bugs found on a pull request are tied to that version. When the pull request closes, its open bugs move to the pr-closed status so they do not linger in the project's bug list.

Require approval

The check's conclusion depends on the require approval setting.

• Off (default): the check reports failure when bugs are found and neutral otherwise. It never blocks a merge.
• On: the check reports success only for a clean run and failure when bugs are found. Mark it as a required status check in the repository's branch protection to block merges until Replay QA is satisfied.

Require approval is set through the REST API as the requireApproval field of the project's GitHub settings.

Send bugs to your issue tracker

Replay QA can file each confirmed bug as an issue. The filed issue contains the full bug report as Markdown and a link back to the bug in Replay QA.

There are two ways to file to GitHub Issues, and one each for Linear and Jira.

Issues on the connected repository. When you connect a repository during project setup and choose GitHub Issues as the bug destination, Replay QA files issues through the GitHub App installation. No extra credential is needed. This is the file GitHub issues setting on the GitHub connection (fileIssues in the API).

Any tracker, configured from the chat. Ask the project chat to set up an issue tracker. It opens a panel where you pick the provider, enter its target and credential, and test the connection:

Tracker	What you provide
GitHub Issues	The repository URL and a personal access token with repo (or Issues: read and write) scope on that repository
Linear	A Linear API key and the team ID to file into
Jira	The Jira site URL, the project key, the email of the filing account, and its API token

Bugs found from that point on are filed as they are confirmed. A rediscovery of a bug that is already on file is recorded against the existing bug and does not create a new issue.

Hosting notes

• Vercel and Netlify both report preview and production deployments to GitHub, so "Listen to GitHub events" works for both main-branch and pull request testing.
• Other hosts: if the host creates GitHub Deployments with an environment URL, use "Listen to GitHub events". If it does not, use "Detect main branch deployments" with the production URL for the main branch; pull request testing needs a deployment or status from the host.
• The deployment URL has to be reachable from the internet without a password in front of it. Application logins are fine; give them to Replay QA as test logins.

Limitations

GitHub is the only supported repository host. GitLab and Bitbucket are not supported.