testers.ai

How It Works

How It Works Illustration

Data Privacy

  • ✓ Fully Offline Operation: testers.ai can be run without sharing any data externally, operating completely offline.
  • ✓ Local Processing: All testing occurs locally on your machine with no data sent to testers.ai server.
  • ✓ By default testers.ai passes data collected from the page being tested via prompts to Google Gemini API, but can be configured to use local LLMs/Azure.

Analytics & Telemetry

  • ✓ Minimal Analytics: A single call to Google Analytics only logs that a run occurred and which command was executed.
  • ✓ No PII Collection: No values, personal information, or test content is collected.
  • ✓ Opt-Out Available: Analytics can be easily disabled with a simple command line flag.

LLM Usage Options

  • ✓ API Flexibility: Use testers.ai API keys that leverage Google Gemini and OpenAI public APIs.
  • ✓ Provider Choice: Use any LLM you prefer - public cloud, private cloud, or local models.
  • ✓ Local AI Options: Can run entirely offline with local LLM models.
  • ✓ Custom Solutions: Custom LLM integrations available upon request.

Network & Storage

  • ✓ Limited Internet Access: Only requires internet access to load the webpage being tested.
  • ✓ Browser Compatibility: All major browsers are supported.
  • ✓ Local Results Storage: All test results are saved on your local file system as HTML and JSON files.
  • ✓ Transparent Output: All output is stored in clear text, not encrypted, for easy review.

Output directory file structure

Name Type Purpose
videos Folder Videos of test case execution
images Folder Contains all report-related images including icons, profile pictures of AI testers and personas, test result screenshots, and other visual assets used in reports
replay.mp4 File Video replay of initial page bug detection and functional test execution and generation of report summary
report.pdf File PDF version of report showing issue found, summary of quality issues, persona feedback, and functional test case results/flows
full_report.png File Image version of report showing issue found, summary of quality issues, persona feedback, and functional test case results/flows
report.html File HTML version of report with images inline to make it easy to share showing issue found, summary of quality issues, persona feedback, and functional test case results/flows
report_pdf.html File PDF version of report but with images linked to be smaller in size and have full-resolution images
report.json File JSON version of the report containing all issues, functional test results, and persona feedback and quality summary
trends.json File JSON that indicates how things like number of bugs, performance and even what bugs have appeared, or disappeared over the past few builds
summary.json File JSON that has all the quality summary details including grade, overview, what is working well, and what needs help and recommendations
issues_reviewed.json File JSON list of all the issues after having been reviewed/triaged by the test manager agent
issues_deduped.json File JSON list of all the issues after having been deduped by the AI
issues_dupes.json File JSON list of all the issues that were identified as duplicates by the AI
panel_feedback_image.json File JSON full details of all feedback from personas across many qualitative aspects of quality and individual persona profile details and individual feedback and suggestions
ai_tests_page_features.json File JSON list of tests generated for this page based on features found by the AI
ai_tests.json File JSON list of all tests generated for this page by the AI
panel_feedback_image_ai.json File JSON list of all feedback from personas on page based on custom ai personas
custom_panel_feedback_image_0.json File JSON data from one of the Custom, user-defined AI user personas
custom_panel_feedback_image_1.json File JSON data from one of the Custom, user-defined AI user personas
custom_panel_feedback_image_2.json File JSON data from one of the Custom, user-defined AI user personas
custom_panel_feedback_image_3.json File JSON data from one of the Custom, user-defined AI user personas
bugs_usability.json File JSON list of bugs related to usability
bugs_missing.json File JSON list of bugs related to things that might be missing on the page--expected by the user or business, but not there
bugs_responsive.json File JSON list of bugs related to responsive / resizing of the content for different device dimensions
bugs_sec.json File JSON list of bugs related to security
bugs_acc.json File JSON list of bugs related to accessibility
bugs_content.json File JSON list of bugs related to content issues like grammar, spelling or factual correctness
bugs_pri.json File JSON list of bugs related to privacy
bugs_api.json File JSON list of bugs related to networking and /API traffic
bugs_console.json File JSON list of bugs related to console logs
page_data.json File JSON info related to the page tested: URL, load performance, important elements/DOM, networking and console activity, time executed, etc.
screenshot_iphone14.png File Image of page loaded with viewport/size similar to iphone14 used to test resizing/responsiveness for mobile
screenshot_full.png File Image of the page including all the content as if i was scrolled down
screenshot.png File Image of the page in a default / standard size
early_screenshot.png File Image of the page taken early in its page load progress
ai_exec_results Folder Functional test case results
Submit_empty_flight_search Folder Artifacts from a functional test case execution
ai_test_issues.json File Issues discovered during this specific test case execution
test_info.json File Test case metadata and configuration details
test_result_orig.json File Original raw test results before AI review and double checking
test_result.json File Processed and formatted test execution results
steps Folder Contains step-by-step execution data and screenshots
0 Folder Initial step artifacts and state
ai_test_issues.json File Issues identified during this step
conversation.txt File Log of AI agent's thought process and decisions during this step
processed.png File Annotated screenshot highlighting elements and actions for this step
raw.png File Original unmodified screenshot of the page at this step
test_agent_reasoning.json File Structured data of AI agent's analysis and decision-making for this step
1 Folder First step artifacts and state (contains same set of files as step 0)
ai_test_issues.json File Issues identified during this step
conversation.txt File Log of AI agent's thought process and decisions during this step
processed.png File Annotated screenshot highlighting elements and actions for this step
raw.png File Original unmodified screenshot of the page at this step
test_agent_reasoning.json File Structured data of AI agent's analysis and decision-making for this step
2 Folder Second step artifacts and state (contains same set of files as step 0)
ai_test_issues.json File Issues identified during this step
conversation.txt File Log of AI agent's thought process and decisions during this step
processed.png File Annotated screenshot highlighting elements and actions for this step
raw.png File Original unmodified screenshot of the page at this step
test_agent_reasoning.json File Structured data of AI agent's analysis and decision-making for this step
3 Folder Third step artifacts and state (contains same set of files as step 0)
ai_test_issues.json File Issues identified during this step
conversation.txt File Log of AI agent's thought process and decisions during this step
processed.png File Annotated screenshot highlighting elements and actions for this step
raw.png File Original unmodified screenshot of the page at this step
test_agent_reasoning.json File Structured data of AI agent's analysis and decision-making for this step