## app

Testers.AI app; testers / testers.exe is an advanced AI testing tool utilizing AI testers and personas to test and
provide feedback on any app.  The testers.ai app has a number of useful features broken down as commands that are
documented below.


## Installation Requirements

MacOS 15+ arm (Sequoia, Senoma); x86 (Sequoia)
Windows 11 (perhaps 10)
Ubuntu

**NOTE: For some MacOS installations please run the following two commands after unzipping the tool.  This will prevent
the OS blocking downloaded executables.

```
xattr -d com.apple.quarantine testers/testers
xattr -d com.apple.quarantine testers/ffmpeg
```


## Notes

1. On Windows please use testers.exe wherever "testers" is used below.
2. When providing a url, one may need to enclose it in double quotes; i.e. "https://testers.ai"
3. All parameters must be after the command
4. Most commands store files in the ~/.testers directory. Output data is normally stored next to the executable.


## Usage

The simplest way to run testers is the following, which will launch app and provide a check of the provided URL.

```
(free version)
testers test {url}
testers check {url}

# url defaults to https://www.bing.com
testers test
testers check


(paid version)
testers test {url} --team {team} --activation-code {code}
testers check {url} --team {team} --activation-code {code}

# url defaults to https://www.bing.com
testers test --team {team} --activation-code {code}
testers check --team {team} --activation-code {code}
```


If one wishes to control how testers segregates the data one can use these options:

```
--team     [default: free]   - The team around which output is centralized.
--datadir  [default: ./data] - The directory to store output.
--app-name [default: APP]    - All run output with the same app name will be put together
--label    [default: HOME]   - A label to identify the url tested

{datadir}/{team}/{app-name}/{time_stampe}/pages/{label}/...

Example:
    testers test --datadir storage --app-name MY_APP --label PAGE_NAME

    storage/free/MY_APP/1741369798.975136/pages/PAGE_NAME/...
```


If one wishes to use their own public LLM accounts then all one needs is their API key.  Currently testers.ai app
only supports

    -Anthropic
    -Gemini (Google AI Platform)
    -OpenAI
    -AzureOpenAI

Example flags:
```
--ai-model                      [Default: None]  The name of the model to use (defaults: gemini-2.0-flash)
--ai-temp                       [Default: None]  AI temperature (float) [0.0, 2.0] (default: AI api default)
--anthropic-key                 [Default: None]  Anthropic API_KEY; or from ENV
--azure-openai-deployment-name  [Default: None]  Azure OpenAI deployment name; or from ENV
--azure-openai-endpoint         [Default: None]  Azure OpenAI endpoint; i.e. url endpoint.  Only accepts endpoints within the
                                                 azure.com domain.
--azure-openai-key              [Default: None]  Azure OpenAI API_KEY; or from ENV
--azure-openai-api-version      [Default: None]  Azure OpenAI version
--gemini-key                    [Default: None]  Google Gemini API_KEY; or from ENV
--openai-key                    [Default: None]  OpenAI API_KEY; or from ENV
--print-ai-stats                [Default: False] Print AI info and token usage.
```

**Notes:
*If one doesn't supply their own key then we provide access to Google Gemini through the testers.ai account.
*API_KEY can also be provided through the standard environment variables instead of as an argument.
*If an API_KEY is provided but the model is not provided then the default model for that LLM is used.

              DEFAULT MODEL NAME         ENV KEY NAME
-Anthropic:   claude-3-7-sonnet-latest   ANTHROPIC_API_KEY
-AzureOpenAI: gpt-4o                     AZURE_OPENAI_API_KEY, AZURE_OPENAI_ENDPOINT, AZURE_OPENAI_VERSION
-Gemini:      gemini-2.0-flash           GEMINI_API_KEY
-OpenAI:      gpt-4o                     OPENAI_API_KEY


# Help

The testers.ai app has built in help subsystem where anyone can find the list of support commands and each commands
set of options.

```
List commands:
    testers --help

List given command's options:
    testers {command} --help
```

*By default, if there is no command provided then a small flask server is launched and a UI is launched in your
default browser.


# test / test-all

The test and test-all commands perform a series of analyses against a given url (or set of urls for test-all).
These commands have a free and paid mode.  By default, the test commands first perform a check then generates
a set of functional tests based on AI and then runs those tests.

```
testers test {url}
testers test-all

Options**:

--cache-dir {cache_dir}          [Default: ./cache] The cache directory is used to store tests that will be applied
                                                    to each subsequent run of functional testing based on app or
                                                    app/label.
--max-steps                      [Default: 20]      Maximum number of steps to execute per test case (0 means
                                                    unlimited steps)
--max-tests                      [Default: 3]       Maximum number of tests to execute
--setup-prompt                   [Default: None]    A prompt used to setup the browser prior to checks/tests.
--skip-cache                     [Default: False]   Do not use cached tests.  Instead generate fresh AI tests for
                                                    the current run only.  These tests are not cached.
--skip-check                     [Default: False]   Do not include the standard url check.
--test-file                      [Default: None]    JSON file containing tests to run.  Only these tests will be run.
                                                    If the JSON data is a dictionary then it is assumed to be testers.ai
                                                    AI test file: [area]: [AI test case].  If the JSON data is a list
                                                    then it examines each entry.  If it is a string then it will attempt
                                                    to convert it to an AI test case otherwise if it is a dictionary
                                                    it will assume it is a testers.ai test case.  Creates a
                                                    custom_tests.json.
--test-prompt                    [Default: *]       A free form test description that is executed atomically.  Creates
                                                    a custom_tests.json.
```

**Notes:
1. See check command options below for shared options.
2. See launch command options below for shared options.
3. These commands also support the AI options to specify using one's own API key and targeted LLM model.
4. An example of how these options combine to build an output directory structure, see the above Usage section.


# check / check-all

The check and check-all commands perform a series of analyses against a given url (or set of urls for check-all).
These commands have a free and paid mode.  The free model restricts the number of reported bugs to a max of three
per tester.

```
testers check {url}
testers check-all

Options**:

--team {team}               [Default: free]
--activation-code {code}    [Default: None]
--app-name {app_name}       [Default: APP]
--label {label}             [Default: HOME]
--datadir {directory}       [Default: ./data]
--auto-accept-cookies       [Default: False]  Attempts to click "accept" for cookie web prompts
--skip-mobile               [Default: False]  Disable taking of a screenshot with iPhone 14 dimensions.  This is
                                              used to generate responsive test.
--responsive                [Default: False]  Take screenshots of the webpage with different devices/dimensions
--headless                  [Default: False]  Run in headless mode
--custom-prompt "{prompt}"  [Default: None]   A prompt that can provide additional information, control and override
                                              to test execution.
--max-steps                 [Default: 20]     Maximum number of steps to execute per test case (0 means
                                              unlimited steps)
--setup-prompt              [Default: None]   A prompt used to setup the browser prior to checks/tests.
--target-name               [Default: None]   The name of the app being tested. (displayed in report)
--target-icon-url           [Default: None]   Icon url for the target app being tested. (displayed in report)
```

**Notes:
1. See launch command options below for shared options.
2. These commands also support the AI options to specify using one's own API key and targeted LLM model.
3. An example of how these options combine to build an output directory structure, see the above Usage section.


# gen-testers

The gen-testers command is a paid feature allowing one to provide a prompt to generate custom testers.  Custom testers
are loaded by the check commands to perform more targeted testing.  This command supports using the AI options to
specify using one's own API key and targeted LLM model.

```
testers gen-testers --team {team} --activation-code {code} "{prompt}"
```


# gen-personas

The gen-personas command is a paid feature allowing one to provide a prompt to generate custom personas.  Custom
personas are loaded by the check commands to perform more targeted feedback.  This command supports using the
AI options to specify using one's own API key and targeted LLM model.

```
testers gen-personas --team {team} --activation-code {code} "{prompt}"
```

# launch

In order to provide the ability to perform stateful tests, testers.ai app provides the launch function.  The command
lets one launch and manually use a browser in a stateless or stateful fashion; stateless by default.  The browser state
can be used to allow manually configuring a url or set of urls prior to testing.  For example, one may desire a url to
be in a logged in state prior to performing a test.

```
testers launch {url}

Options:

--as-device {device_name}             [Default: None] Use device-like configuration; ignores browser option.
--browser {browser_name}              [Default: chromium]
--browser-install-path {directory}    [Default: ~/.testers/browsers]
--disable-video-recording             [Default: False]
--header                              [Default: None] Allows zero or more of this flag.  All entries are composed into
                                                      a dictionary of headers that will be applied to all sites.
--save-video-dir {directory}          [Default: None]
--user-agent {string}                 [Default: None] *Uses default value
--user-data {directory}               [Default: None]
--viewport-dims {width},{height}      [Default: 1420,1080]
--window-pos {x},{y}                  [Default: None] *Does not work with WebKit / Firefox
```

The user-data option determines whether or not the browser is in stateless mode.  If a directory is provided then
the browser will save relevant browser state such as cookies to this directory.  When the browser is
launched again with the same directory, the browser will load the provided state.  This can also be used with the
check and test commands to allow the browser to launch in the given state.


# browsers

In order to provide control over what browsers are installed, testers.ai app provides the browsers command.

```
testers browsers install {browser_name} [--force]
testers browsers list
testers browsers uninstall

Common options:

--browser-install-path {directory}    [Default: ~/.testers/browsers]
```

*The uninstall command uninstalls ALL "managed" browsers; does not include Chrome / MS Edge.

*The install command allows one of these browsers to be installed:

    ```chromium,firefox,webkit,chrome,chrome-beta,msedge,msedge-beta,msedge-dev``

There are occasions where a newer version is available but it is already installed.  In those instances, the --force
option can be used to force it to install even if the "browser" is "already installed".  This most often occurs when
the testers.ai app is updated.


# analytics

As we explore how best to provide testers.ai services and tools, we measure how often different commands and options
are used through Google Analytics.  Each time one run the testers.ai app we compile the relevant data and send it
to Google Analytics.  To protect privacy, we do NOT save urls, prompts, API keys, etc.  For that kind of data we simply
record whether or not that option was used as a True/False value.  The analytics command allows one to check whether
the app is set to record information or not and the ability to enable/disable this functionality.

```
testers analytics
testers analytics --enable
testers analytics --disable
```


# bug-filters

The bug filters manage a list of words or phrases and any issues/bugs found matching these strings will be filtered
out.  The bug-filters command allows adding a strings and printing the current filters.

```
testers bug-filters
testers bug-filters {string}
```


# add-url / history

Whenever a url is checked or tested it is recorded for a specific app.  This list can then be used by check-all or
test-all to process all the urls sequentially.  To see what urls are part of this list one can use the history command.
To add urls without checking or testing the urls first, one can use the add-url command to add it to the list


```
testers add-url [--app-name app_name] url
testers history [--app-name app_name]
```


# gen-ai-tests

Use AI to generate tests for a specific application or application/label depending on whether the label option is
provided.  AI generated tests examine the provided url and create a series of test cases across a number of
areas: edge, ai_unique, long_flow, interesting, basic, core_actions, search, and forms.  Additionally, there is
a flag that enables the generation of feature tests.

```
--app-name {app_name}      [Default: APP]
--auto-accept-cookies      [Default: False]     Attempts to click "accept" for cookie web prompts
--cache-dir {cache_dir}    [Default: ./cache]   The cache directory is used to store tests that will be applied
                                                to each subsequent run of functional testing based on app or app/label.
--include-feature-tests    [Default: False]     Use AI to generate feature tests for the given url.
--custom-prompt "{prompt}" [Default: None]      A prompt that can provide additional information, control and override
                                                to test execution.
--label {label}            [Default: None]      Not required if test is intended for all labels within application.
                                                The normal default value is HOME.
--max-steps                [Default: 20]        Maximum number of steps to execute per test case (0 means
                                                unlimited steps)
--setup-prompt             [Default: None]      A prompt used to setup the browser prior to checks/tests.
--team {team}              [Default: free]
{url}                      [Default: None]      The url to analyze; defaults to https://www.bing.com
```

**Notes:
1. See launch command options below for shared options.
2. These commands also support the AI options to specify using one's own API key and targeted LLM model.


# import-tests

Allows for the conversion user defined tests into AI tests.  The tests must be associated with a team and application
(label is optional).  These values are used to determine what tests are executed during functional testing.

```
--app-name {app_name}      [Default: APP]
--cache-dir {cache_dir}    [Default: ./cache]   The cache directory is used to store tests that will be applied
                                                to each subsequent run of functional testing based on app or app/label.
--custom-prompt "{prompt}"  [Default: None]     A prompt that can provide additional information, control and override
                                                to test execution.
--label {label}            [Default: None]      Not required if test is intended for all labels within application.
                                                The normal default value is HOME.
--team {team}              [Default: free]
{import_dir}               [Default: None]      The import directory specifies where to load test cases.  If it is
                                                not provided then it automatically generate a sample test case for
                                                the Google home page.
```

**Notes:
1. These commands also support the AI options to specify using one's own API key and targeted LLM model.


# test-filter

The test-filter command is used to build and manage test cases used for AI tests during the functional testing phase.  There
are two modes: one mode builds up a series of filters and the other mode applies filters to a set of test cases creating a
new set of test cases.  The simpliest way to view the creation of new tests is to run the command with no filters, which
simply performs a copy from on file into another.  An important note about this command is that it works with the same
directory structure as the cache and data directories; i.e. <root>/<team>/<app_name> or <root>/<team>/<app_name>/<label>

A key concept when using this command.  Exclusions will act as filters for removal whereas inclusions are used to filter
for what to keep.  If both are used together then the exclusions are applied to the inclusions found in the source test file.

```
--app-name {app_name}      [Default: APP]
--label {label}            [Default: None]      Not required if test is intended for all labels within application.
                                                The normal default value is HOME.
--team {team}              [Default: free]
--force                    [Default: False]     Forces overwrite; i.e. the input and output file are the same.  By default,
                                                the command does not allow overwrite.
--exclusions               [Default: False]     Two modes if present: 1. string is provided then stores as filter, 2. no string
                                                provided then applies exclusions strings as filters creating new test file.
--inclusions               [Default: False]     Two modes if present: 1. string is provided then stores as filter, 2. no string
                                                provided then applies white strings as filters creating new test file.
--in-dir {out_dir}         [Default: ./cache]   The input root directory for the source file.  Defaults to the cache directory.
--out-dir {out_dir}        [Default: ./cache]   The output root directory for the new file.  Defaults to the cache directory.
{string}                   [Default: None]      When used in conjunction with exclusions and inclusions flags, this string
                                                will be stored and later used as filters.
```
