To expand a bit on the answer by @nirmal-kumar. Here is an example of a YAML based VSTest task without continueOnError set, so it defaults to false. A failure will stop subsequent steps from running unless you explicitly set a condition on those steps.

- task: VSTest@3
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    codeCoverageEnabled : true

Here is the same VSTest task with with continueOnError set to true.

- task: VSTest@3
  continueOnError: true
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    codeCoverageEnabled : true

With continueOnError set to true, subsequent steps in the build pipeline will still run even if tests fail. The overall result status will no longer be Failed (), nor will it be Success (). Instead, the status will be Warning ().

As @mengdi-liang commented, continueOnError does not appear in the VSTest@3 documentation because it is a general control option that can apply to any task, much like condition. This is important because it means that continueOnError does not go under inputs like many task settings. It should be indented to a level where it is a sibling of inputs as shown above. If continueOnError is indented beneath inputs as shown below, it will not work.

- task: VSTest@3
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    codeCoverageEnabled : true
    continueOnError: true   # DOES NOT WORK INDENTED UNDER INPUTS!