Building With AI: Which Tests and Checks Do You Need?

If you are building software with AI, the AI writes the code and you are the one deciding whether it is good enough to ship, usually without reading every line of it. If you are a developer building software with a team, the same decision comes up when the code was written by someone else. The tests and checks in this article are how I make that decision on my own projects. As an example, I might ask the AI for a feature where about 50 columns are being added to a table, so the user can add whichever column they want to the table and then filter the data by that column. It is not practical to check that all 50 exist and filter correctly by hand, I would spot check a few, but the AI should also write a test that runs against the deployed code and confirms all 50 can be filtered. The words for tests, unit, integration, smoke and end-to-end, get used loosely, and really it is a mix of all of them, with manual QA still in the order. This article walks through what each one is and what it is good at, and where the other checks like linters and formatters fit.

A way to picture it

It helps to think of the software as a car being built. Some of the checks look at one part on its own, and the rest test the whole car. The quick checks are run first, so if the build does not start you find out early rather than waiting on the slow tests.

Unit tests

A unit test checks one part on its own, so a fuel gauge with an input of half a tank should point the needle at the halfway mark. It runs on the source code with example inputs, not against the deployed site, so it proves the part works and nothing more. In code, a function is given one input, with the expected answer set in the code to check. Each unit test is small and quick, so there are often many of these tests. When one fails, it points to the function that broke, which allows for finding the problem quickly. On my own site, the spam checks on the contact form are tested this way, by providing a message that should pass, and a message that should be blocked.

Unit tests don’t check how parts work together, instead they test that one part for failure.

Integration tests

An integration test checks a few parts together, such as a request going through the API to the database and back, or a component rendering with its real data underneath. These catch the bugs where parts are working by themselves, but failing when working together. They are slower than unit tests, so there are usually fewer of them.

The earlier example of the automated testing of the 50 columns and their filters would probably be an integration test, or even an end-to-end test. The test would check that each column queries the API and returns the field filtered on the deployed code. A parameterised test helps here, which is one test written once and then played 50 times over from a list of column names, so once for each column. Parameterised is a technique rather than a kind of test. The same list of column names could drive a unit test of the filter function, or an integration test that also queries the API and returns values, or an end-to-end browser test, depending on what is required.

Smoke tests

A smoke test is a small set of quick checks that the main things are working after a build or a deploy, like turning the key to see if the engine starts. For a website it can be as simple as asking for the home page and checking that it comes back and loads successfully, and a step further is checking that a login works and the main API answers. For an app it could be checking that it starts up and returns the main window with content, and that it can read its data. Smoke tests are run prior to the longer end-to-end tests, because if the website or app is not even loading, then there is no point running other tests that require it to be loaded. A smoke test doesn’t check correctness in detail, it only confirms that the main things are up and responding.

End-to-end tests

An end-to-end test uses the software how a customer would. I use a software tool called Playwright for these tests, as it opens a browser, clicks buttons, types in the form and checks what appears. The tests are programmed and target selectors on a page, and wait automatically until elements are ready to be used. These tests are the slowest to run, and often break when the UI is changed. They are best kept for a few main flows such as signing up, paying and submitting a form, rather than trying to cover every end-to-end process with a test.

Regression tests

A regression test checks that something which used to work still works after a change. Often it is any of the kinds above, written after a bug so that the bug is caught if it comes back. You don’t have to write a regression test for every bug you fix, but if the same issues keep returning, or the severity of the bug is high or critical, or you know that other upcoming changes could touch this code, it can be good to build in regression testing. My own example is a linter I use on my writing, which has a set of rules for phrasing I want to keep out of my articles. When a rule is added or changed, the regression test runs every rule over a folder of samples of my own writing, and if any rule flags one of my sentences the test fails, which blocks a rule change that is really a regression. This testing is being used to maintain consistency in my writing when I am using AI to assist.

Manual QA

Manual testing still has a place, with a different job. A person can judge things a script cannot, such as a flow that does not make sense, something that looks wrong on the page, or a button label that does not match what it does, and can explore in ways nobody scripted. Checking 50 things the same way every release is better done by the parameterised test. So the split I ask for is that automated tests cover the things that must be true every time, and manual QA covers judgment and exploration, with a spot check on the automated areas rather than a full pass. For my own apps the manual step is installing the new build over the old one and using it before it goes out.

The other gates

The tests above check that the software does the right thing. The checks below check that it is safe and tidy, and they run after the code is completed and is being presented for commit. If they pass, the tests are run afterwards. These are the ones I run on my projects, in roughly this order.

• Type check: reads the code without running it, to check that every part is handed what it expects, for example a function that should return a number never returns a word. In TypeScript projects this is the TypeScript compiler run in a check-only mode, which reports type errors without producing any files. For Astro sites I use the Astro check command.
• Linter: reads the code for known mistakes, for example an unused variable or a comparison that can never be true. I use ESLint for JavaScript and Stylelint for CSS, and I have one for my writing as well.
• HTML validation: checks the structure of the markup, for example unclosed tags, mismatched nesting and attributes that do not exist. Browsers will often still render the page, which means these issues can be missed, however they could be causing scripts, styles or elements to not display or function correctly. I use html-validate to catch them.
• Formatter: rewrites the appearance of a file, for example indentation, quotes and line length, to keep every file looking the same. I use Prettier. With formatting handled, the only differences between versions of a file are real changes.
• Secrets scan: reads what is about to be committed and stops it if it looks like a password, API key or token. I use Betterleaks locally, and gitleaks runs again on GitHub. If you don’t build in these checks, a key could be in committed code, which then remains in git history. It would then need to be cancelled and replaced, as removing it from the history is not enough.
• Dependency allowlist and SAST: the allowlist checks that each package in the project is one that has been agreed to. SAST, static application security testing, reads the code without running it for patterns that are known to be unsafe, such as SQL injection. I use Semgrep for SAST.
• Audit: checks the packages you depend on against the GitHub Advisory Database of known vulnerabilities, which npm audit queries through the npm registry. It only reports vulnerabilities that have been published. I use npm audit, and Socket to score new packages.
• Licence check: confirms that the licences of the packages in the project allow them to be used in the product, for example commercially.
• Build: compiles everything into the files that get deployed. If the build fails there is nothing to deploy, so the smoke and end-to-end tests, which run against the built site or app, cannot run. Unit tests often run on the source directly. The tests then run with the quickest first, unit tests, then smoke, then integration and then end-to-end.

Where the tests and checks live

In my projects the tests sit in a tests folder at the top of the project, with a file for each kind, so smoke.js, integration.js and a flows folder for the Playwright tests, and unit tests are named after the thing they test, for example contact.test.mjs for the contact form checks. The commands to run them are in package.json under scripts, with names like test:smoke, test:integration and test:flows, and npm test runs the main set. The checks each have a small config file at the top of the project, for example eslint.config.js for the linter, playwright.config.js for the flows and tsconfig.json for the type check, and the gates run them in order from one script. Naming varies between frameworks, so the useful habit is to ask the AI to show you the tests folder and the scripts list, and to run npm test yourself once so you have seen it pass.

The commit-time checks are tied to git with a pre-commit hook, so a commit cannot be made unless they pass, whether it is me or the AI making the commit. In Claude Code I also use its own hooks, which run while the AI is working and stop it from ending its turn until the gates have passed and the changelog is updated, so the checking happens before the code reaches me for review. I have written about setting up both kinds of hook.

Your First Git Hook: Stop Secrets Reaching a Commit

Your First Claude Code Hook: Block a Command Before It Runs

Why the order

The quick checks run first to ensure parts are working. The integration tests and end-to-end tests take longer to run as they contain multiple parts. On my projects I use AI to generate the code, so the commit-time checks run on every commit, and the full set runs before the AI presents the code for my review, and again on GitHub after it is pushed. If you are building with AI and setting this up for the first time, start with commit-time checks and a smoke test, and ask the AI to add them to the project for you. Unit tests can be added where bugs happen, followed by integration tests and end-to-end testing of the main flows.

I hope this helps if you are deciding which tests and checks to put in place.