Sep 09, 2017. The two new things you will notice in this snippet of code is the TestClass and TestMethod tags, which certainly don’t just float around in normal code. These tags are what allow Visual Studio’s built in testing framework to recognize this particular class as a class that contains unit tests, and to treat the method TryShootBug as a test case, instead of just an ordinary method.
-->Azure Pipelines
Use this task to run unit and functional tests (Selenium, Appium, Coded UI test, and more)using the Visual Studio Test Runner. Other than MSTest-based tests, test frameworks that have aVisual Studio test adapter, such as xUnit, NUnit, Chutzpah, can also be executed.
![Mac Mac](https://i.ytimg.com/vi/OYokgJvJXlU/maxresdefault.jpg)
Tests that target the .NET core framework can be executed by specifying the appropriate target framework value.
Tests can be distributed on multiple agents using version 2 of this task. For more information, see Run tests in parallel using the Visual Studio Test task.
Check prerequisites
If you're using a Windows self-hosted agent, be sure that your machine has this prerequisite installed:
- .NET Framework 4.6.2 or a later version
Demands
The agent must have the following capability:
vstest
The vstest demand can be satisfied in two ways:
- Totalspaces 2 8 6 inch. Visual Studio is installed on the agent machine.
- By using the Visual Studio Test Platform Installer task in the pipeline definition.
Arguments
Argument | Description |
---|---|
testSelector Select tests using | (Required) Test assembly: Use this option to specify one or more test assemblies that contain your tests. You can optionally specify a filter criteria to select only specific tests. Test plan: Use this option to run tests from your test plan that have an automated test method associated with it. To learn more about how to associate tests with a test case work item, see Associate automated tests with test cases. Test run: Use this option when you are setting up an environment to run tests from test plans. This option should not be used when running tests in a continuous integration/continuous deployment (CI/CD) pipeline. Default value: testAssemblies |
testAssemblyVer2 Test files | (Required) Run tests from the specified files. Ordered tests and webtests can be run by specifying the .orderedtest and .webtest files respectively. To run .webtest , Visual Studio 2017 Update 4 or higher is needed. The file paths are relative to the search folder. Supports multiple lines of minimatch patterns. More InformationDefault value: ***test*.dlln!***TestAdapter.dlln!**obj** |
testPlan Test plan | (Required) Select a test plan containing test suites with automated test cases. |
testSuite Test suite | (Required) Select one or more test suites containing automated test cases. Test case work items must be associated with an automated test method. Learn more. |
testConfiguration Test configuration | (Required) Select Test Configuration. |
tcmTestRun Test Run | (Optional) Test run based selection is used when triggering automated test runs from test plans. This option cannot be used for running tests in the CI/CD pipeline. |
searchFolder Search folder | (Required) Folder to search for the test assemblies. |
testFiltercriteria Test filter criteria | (Optional) Additional criteria to filter tests from Test assemblies. For example: Priority=1|Name=MyTestMethod . More information |
runOnlyImpactedTests Run only impacted tests | (Optional) Automatically select, and run only the tests needed to validate the code change. More information |
runAllTestsAfterXBuilds Number of builds after which all tests should be run | (Optional) Number of builds after which to automatically run all tests. Test Impact Analysis stores the mapping between test cases and source code. It is recommended to regenerate the mapping by running all tests, on a regular basis. |
uiTests Test mix contains UI tests | (Optional) To run UI tests, ensure that the agent is set to run in interactive mode with autologon enabled. Setting up an agent to run interactively must be done before queueing the build/release. Checking this box does not configure the agent in interactive mode automatically. This option in the task is to only serve as a reminder to configure agent appropriately to avoid failures. Hosted Windows agents from the VS 2015 and 2017 pools can be used to run UI tests. |
vstestLocationMethod Select test platform using | (Optional) Specify which test platform should be used. |
vsTestVersion Test platform version | (Optional) The version of Visual Studio test to use. If latest is specified it chooses Visual Studio 2017 or Visual Studio 2015 depending on what is installed. Visual Studio 2013 is not supported. To run tests without needing Visual Studio on the agent, use the Installed by tools installer option in the UI or toolsInstaller in YAML. Be sure to include the ‘Visual Studio Test Platform Installer’ task to acquire the test platform from NuGet. |
vstestLocation Path to vstest.console.exe | (Optional) Specify the path to VSTest. |
runSettingsFile Settings file | (Optional) Path to runsettings or testsettings file to use with the tests.Starting with Visual Studio 15.7, it is recommended to use runsettings for all types of tests. To learn more about converting a .testsettings file to a .runsettings file, see this topic. |
overrideTestrunParameters Override test run parameters | (Optional) Override parameters defined in the TestRunParameters section of runsettings file or Properties section of testsettings file. For example: -key1 value1 -key2 value2 . Note: Properties specified in testsettings file can be accessed via the TestContext using Visual Studio 2017 Update 4 or higher |
pathtoCustomTestAdapters Path to custom test adapters | (Optional) Directory path to custom test adapters. Adapters residing in the same folder as the test assemblies are automatically discovered. |
runInParallel Run tests in parallel on multi-core machines | (Optional) If set, tests will run in parallel leveraging available cores of the machine. This will override the MaxCpuCount if specified in your runsettings file. Click here to learn more about how tests are run in parallel. |
runTestsInIsolation Run tests in isolation | (Optional) Runs the tests in an isolated process. This makes vstest.console.exe process less likely to be stopped on an error in the tests, but tests might run slower. This option currently cannot be used when running with the multi-agent job setting. |
codeCoverageEnabled Code coverage enabled | (Optional) Collect code coverage information from the test run. |
otherConsoleOptions Other console options | (Optional) Other console options that can be passed to vstest.console.exe , as documented here. These options are not supported and will be ignored when running tests using the Multi agent parallel setting of an agent job or when running tests using Test plan option. The options can be specified using a settings file instead. |
distributionBatchType Batch tests | (Optional) A batch is a group of tests. A batch of tests runs its tests at the same time and results are published for the batch. If the job in which the task runs is set to use multiple agents, each agent picks up any available batches of tests to run in parallel. Based on the number of tests and agents: Simple batching based on the number of tests and agents participating in the test run. Based on past running time of tests: This batching considers past running time to create batches of tests such that each batch has approximately equal running time. Based on test assemblies: Tests from an assembly are batched together.' Default value: basedOnTestCases |
batchingBasedOnAgentsOption Batch options | (Optional) Simple batching based on the number of tests and agents participating in the test run. When the batch size is automatically determined, each batch contains (total number of tests / number of agents) tests. If a batch size is specified, each batch will contain the specified number of tests. Default value: autoBatchSize |
customBatchSizeValue Number of tests per batch | (Required) Specify batch size Default value: 10 |
batchingBasedOnExecutionTimeOption Batch options | (Optional) This batching considers past running time to create batches of tests such that each batch has approximately equal running time. Quick running tests will be batched together, while longer running tests may belong to a separate batch. When this option is used with the multi-agent job setting, total test time is reduced to a minimum. Default value: autoBatchSize |
customRunTimePerBatchValue Running time (sec) per batch | (Required) Specify the running time (sec) per batch Default value: 60 |
dontDistribute Replicate tests instead of distributing when multiple agents are used in the job | (Optional) Choosing this option will not distribute tests across agents when the task is running in a multi-agent job. Each of the selected test(s) will be repeated on each agent. The option is not applicable when the agent job is configured to run with no parallelism or with the multi-config option. Default value: False |
testRunTitle Test run title | (Optional) Provide a name for the test run |
platform Build platform | (Optional) Build platform against which the tests should be reported. If you have defined a variable for platform in your build task, use that here. |
configuration Build configuration | (Optional) Build configuration against which the tests should be reported. If you have defined a variable for configuration in your build task, use that here. |
publishRunAttachments Upload test attachments | (Optional) Opt in/out of publishing run level attachments. Default value: true |
failOnMinTestsNotRun Fail the task if a minimum number of tests are not run | (Optional) Use this option to fail the task if a minimum number of tests are not run. This may be useful if any changes to task inputs or underlying test adapter dependencies lead to only a subset of the desired tests to be found. Default value: False |
minimumExpectedTests Minimum # of tests | (Optional) Specify the minimum # of tests that should be run for the task to succeed. Total tests run is calculated as the sum of passed, failed and aborted tests. Default value: 1 |
diagnosticsEnabled Collect advanced diagnostics in case of catastrophic failures | (Optional) Use this option to turn on collection of diagnostic data to troubleshoot catastrophic failures such as test crash. When this option is checked, a sequence XML file is generated and attached to the test run. The sequence file contains information about the sequence in which tests ran, so that a potentially culprit test can be identified. Default value: false |
collectDumpOn Collect process dump and attach to test run report | (Optional) Use this option to collect a mini-dump that can be used for further analysis. On abort only: mini-dump will be collected only when test run is aborted. Always: mini-dump will always be collected regardless of whether the test run completes or not. Never: mini-dump will not be collected regardless of whether the test run completes or not |
rerunFailedTests Rerun failed tests | (Optional) Selecting this option will rerun any failed tests until they pass or the maximum # of attempts is reached. Default value: False |
rerunType Do not rerun if test failures exceed specified threshold | (Optional) Use this option to avoid rerunning tests when failure rate crosses the specified threshold. This is applicable if any environment issues leads to massive failures.You can specify % failures with basedOnTestFailurePercentage or # of failed tests as a threshold with basedOnTestFailureCount.Default value: basedOnTestFailurePercentage |
rerunFailedThreshold % failure | (Optional) Use this option to avoid rerunning tests when failure rate crosses the specified threshold. This is applicable if any environment issues leads to massive failures Default value: 30 |
rerunFailedTestCasesMaxLimit # of failed tests | (Optional) Use this option to avoid rerunning tests when number of failed test cases crosses specified limit. This is applicable if any environment issues leads to massive failures and if rerunType is rerunFailedTestCasesMaxLimit. Default value: 5 |
rerunMaxAttempts Maximum # of attempts | (Optional) Specify the maximum # of times a failed test should be retried. If a test passes before the maximum # of attempts is reached, it will not be rerun further. Default value: 3 |
Open source
This task is open source on GitHub. Feedback and contributions are welcome.
Vs For Mac Mstest 2017
FAQ
How can I run tests that use TestCase as a data source?
To run automated tests that use TestCase as a data source, the following is needed:
- You must have Visual Studio 2017.6 or higher on the agent machine. Visual Studio Test Platform Installer task cannot be used to run tests that use TestCase as a data source.
- Create a PAT that is authorized for the scope “Work Items (full)”.
- Add a secure Build or Release variable called Test.TestCaseAccessToken with the value set to the PAT created in the previous step.
I am running into issues when running data-driven xUnit and NUnit tests with some of the task options. Are there known limitations?
Data-driven tests that use xUnit and NUnit test frameworks have some known limitations and cannot be used with the following task options:
- Rerun failed tests.
- Distributing tests on multiple agents and batching options.
- Test Impact Analysis
The above limitations are because of how the adapters for these test frameworks discover and report data-driven tests.
-->VSTest.Console.exe is the command-line tool to run tests. You can specify several options in any order on the command line. These options are listed in General command-line options.
Note
The MSTest adapter in Visual Studio also works in legacy mode (equivalent to running tests with mstest.exe) for compatibility. In legacy mode, it can't take advantage of the TestCaseFilter feature. The adapter can switch to legacy mode when a testsettings file is specified, forcelegacymode is set to true in a runsettings file, or by using attributes such as HostType.
To run automated tests on an ARM architecture-based machine, you must use VSTest.Console.exe.
Open a Developer Command Prompt to use the command-line tool, or you can find the tool in %Program Files(x86)%Microsoft Visual Studio<version><edition>common7ideCommonExtensions<Platform | Microsoft>.
General command-line options
The following table lists all the options for VSTest.Console.exe and short descriptions of them. You can see a similar summary by typing
VSTest.Console/?
at a command line.Option | Description |
---|---|
[test file names] | Run tests from the specified files. Separate multiple test file names with spaces. Examples: mytestproject.dll , mytestproject.dll myothertestproject.exe |
/Settings:[file name] | Run tests with additional settings such as data collectors. For more information, see Configure unit tests using a .runsettings file Example: /Settings:local.runsettings |
/Tests:[test name] | Run tests with names that contain the provided values. To provide multiple values, separate them by commas. Example: /Tests:TestMethod1,testMethod2 The /Tests command-line option cannot be used with the /TestCaseFilter command-line option. |
/Parallel | Specifies that the tests be executed in parallel. By default, up to all available cores on the machine can be used. You can configure the number of cores to use in a settings file. |
/Enablecodecoverage | Enables data diagnostic adapter CodeCoverage in the test run. Default settings are used if not specified using settings file. |
/InIsolation | Runs the tests in an isolated process. This isolation makes the vstest.console.exe process less likely to be stopped on an error in the tests, but tests might run slower. |
/UseVsixExtensions | This option makes the vstest.console.exe process use or skip the VSIX extensions installed (if any) in the test run. This option is deprecated. Starting from the next major release of Visual Studio this option may be removed. Move to consuming extensions made available as a NuGet package. Example: /UseVsixExtensions:true |
/TestAdapterPath:[path] | Forces the vstest.console.exe process to use custom test adapters from a specified path (if any) in the test run. Example: /TestAdapterPath:[pathToCustomAdapters] |
/Platform:[platform type] | Target platform architecture to be used for test execution. Valid values are x86, x64, and ARM. |
/Framework: [framework version] | Target .NET version to be used for test execution. Example values are Framework35 , Framework40 , Framework45 , FrameworkUap10 , .NETCoreApp,Version=v1.1 .TargetFrameworkAttribute is used to automatically detect this option from your assembly, and defaults to Framework40 when the attribute is not present. You must specify this option explicitly if you remove the TargetFrameworkAttribute from your .NET Core assemblies.If the target framework is specified as Framework35, the tests run in CLR 4.0 'compatibility mode'. Example: /Framework:framework40 |
/TestCaseFilter:[expression] | Run tests that match the given expression. <Expression> is of the format <property>=<value>[|<Expression>]. Example: /TestCaseFilter:'Priority=1' Example: /TestCaseFilter:'TestCategory=Nightly|FullyQualifiedName=Namespace.ClassName.MethodName' The /TestCaseFilter command-line option cannot be used with the /Tests command-line option. For information about creating and using expressions, see TestCase filter. |
/? | Displays usage information. |
/Logger:[uri/friendlyname] | Specify a logger for test results. Specify the parameter multiple times to enable multiple loggers. Example: To log results into a Visual Studio Test Results File (TRX), use /Logger:trx [;LogFileName=<Defaults to unique file name>] |
/ListTests:[file name] | Lists discovered tests from the given test container. |
/ListDiscoverers | Lists installed test discoverers. |
/ListExecutors | Lists installed test executors. |
/ListLoggers | Lists installed test loggers. |
/ListSettingsProviders | Lists installed test settings providers. |
/Blame | Runs the tests in blame mode. This option is helpful in isolating problematic tests that cause the test host to crash. When a crash is detected, it creates a sequence file in TestResults/<Guid>/<Guid>_Sequence.xml that captures the order of tests that were run before the crash. For more information, see Blame data collector. |
/Diag:[file name] | Writes diagnostic trace logs to the specified file. |
/ResultsDirectory:[path] | Test results directory will be created in specified path if not exists. Example: /ResultsDirectory:<pathToResultsDirectory> |
/ParentProcessId:[parentProcessId] | Process ID of the Parent Process responsible for launching current process. |
/Port:[port] | The Port for socket connection and receiving the event messages. |
/Collect:[dataCollector friendlyName] | Enables data collector for the test run. More information. |
Examples
The syntax for running vstest.console.exe is:
vstest.console.exe [TestFileNames] [Options]
The following command runs vstest.console.exe for the test library myTestProject.dll:
The following command runs vstest.console.exe with multiple test files. Separate test file names with spaces:
Visual Studio For Mac Mstest
The following command runs vstest.console.exe with several options. It runs the tests in the myTestFile.dll file in an isolated process and uses settings specified in the Local.RunSettings file. Additionally, it only runs tests marked 'Priority=1', and logs the results to a .trx file.
Vs For Mac Mstest X
The following command runs vstest.console.exe with the
/blame
option for the test library myTestProject.dll:If a test host crash happened, the sequence.xml file is generated. The file contains fully qualified names of the tests in their sequence of execution up to and including the specific test that was running at the time of the crash.
If there is no test host crash, the sequence.xml file will not be generated.
Vs For Mac Mstest Download
Example of a generated sequence.xml file: