Skip to content

VSTestBridge+MSTest: Use TestMethodIdentifierProperty and stop sending VSTest-specifics #5409

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 20 commits into
base: main
Choose a base branch
from

Conversation

Youssef1313
Copy link
Member

@Youssef1313 Youssef1313 commented Apr 7, 2025

Fixes #5406

@drognanar Any concerns?

  • MSTest won't use the capabilities implementation of the bridge so that it removes vstestProvider (we still keep it for NUnit and Expecto)
  • When a framework of the VSTestBridge uses vstestProvider, everything is the same as today.
  • When a framework of the VSTestBridge doesn't use vstestProvider (i.e, MSTest for now), we expect ManagedType/ManagedMethod to be present. In that case, we parse them via AdapterUtilities to construct TestMethodIdentifierProperty.
  • We stop serializing location.namespace completely. And include both the namespace and type name of TestMethodIdentifierProperty in location.type.

@Youssef1313 Youssef1313 requested a review from nohwnd as a code owner April 7, 2025 11:22
At least for MSTest, ManagedMethod has parameter types, while FullyQualifiedName doesn't. So, prefer ManagedMethod/ManagedType when they are set.
@Youssef1313 Youssef1313 marked this pull request as draft April 7, 2025 15:43
@Youssef1313 Youssef1313 force-pushed the dev/ygerges/testmethodidentifier branch from e21d711 to f6c9ff8 Compare April 16, 2025 09:56
@Youssef1313 Youssef1313 changed the title VSTestBridge: Use TestMethodIdentifierProperty and stop sending VSTest-specifics VSTestBridge+MSTest: Use TestMethodIdentifierProperty and stop sending VSTest-specifics Apr 16, 2025
@@ -17,17 +16,14 @@ namespace Microsoft.Testing.Extensions.VSTestBridge.UnitTests.ObjectModel;
[TestClass]
public sealed class ObjectModelConvertersTests
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially I deleted some tests here when I was removing vstestProvider completely. In the current PR status, some logic is brought back, and I need to plug the tests back per the updated logic (we add the vstest properties under vstestProvider named capability + JsonRpc server mode).

@Youssef1313 Youssef1313 force-pushed the dev/ygerges/testmethodidentifier branch from cc314ba to 16615b4 Compare April 18, 2025 20:44
@Youssef1313 Youssef1313 force-pushed the dev/ygerges/testmethodidentifier branch from 16615b4 to fb351b6 Compare April 18, 2025 20:47
@codecov-commenter
Copy link

codecov-commenter commented Apr 18, 2025

Codecov Report

Attention: Patch coverage is 92.55319% with 7 lines in your changes missing coverage. Please review.

Project coverage is 73.82%. Comparing base (caeb624) to head (049a619).
Report is 73 commits behind head on main.

Files with missing lines Patch % Lines
...ons.VSTestBridge/VSTestBridgedTestFrameworkBase.cs 0.00% 5 Missing ⚠️
...STestBridge/ObjectModel/FrameworkHandlerAdapter.cs 87.50% 1 Missing ⚠️
...Bridge/ObjectModel/TestCaseDiscoverySinkAdapter.cs 83.33% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #5409      +/-   ##
==========================================
- Coverage   76.24%   73.82%   -2.43%     
==========================================
  Files         603      602       -1     
  Lines       36420    36467      +47     
==========================================
- Hits        27770    26923     -847     
- Misses       8650     9544     +894     
Flag Coverage Δ
Debug 73.82% <92.55%> (-2.43%) ⬇️
integration 73.82% <92.55%> (-2.43%) ⬇️
production 73.82% <92.55%> (-2.43%) ⬇️
unit 73.82% <92.55%> (-2.43%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...latformAdapter/TestApplicationBuilderExtensions.cs 100.00% <100.00%> (ø)
...abilities/VSTestBridgeExtensionBaseCapabilities.cs 100.00% <ø> (ø)
....VSTestBridge/ObjectModel/ObjectModelConverters.cs 94.28% <100.00%> (+1.47%) ⬆️
...uests/VSTestDiscoverTestExecutionRequestFactory.cs 77.77% <100.00%> (-2.23%) ⬇️
...e/Requests/VSTestRunTestExecutionRequestFactory.cs 76.47% <100.00%> (-3.53%) ⬇️
...ft.Testing.Platform/Messages/TestNodeProperties.cs 92.38% <ø> (ø)
...t.Testing.Platform/ServerMode/JsonRpc/Json/Json.cs 89.55% <100.00%> (+0.09%) ⬆️
...Platform/ServerMode/JsonRpc/SerializerUtilities.cs 91.87% <100.00%> (+0.07%) ⬆️
...STestBridge/ObjectModel/FrameworkHandlerAdapter.cs 58.33% <87.50%> (ø)
...Bridge/ObjectModel/TestCaseDiscoverySinkAdapter.cs 79.48% <83.33%> (-0.52%) ⬇️
... and 1 more

... and 65 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Youssef1313 Youssef1313 marked this pull request as ready for review April 18, 2025 22:13
testNode.Properties.Add(methodIdentifierProperty);
}

CopyVSTestProperties(testCase.Properties, testNode, testCase, testCase.GetPropertyValue, isTrxEnabled, ShouldAddVSTestProviderProperties(serviceProvider));
Copy link
Member

@fhnaseer fhnaseer Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion:

if (TryGetMethodIdentifierProperty(testCase, out TestMethodIdentifierProperty? methodIdentifierProperty))
    testNode.Properties.Add(methodIdentifierProperty);
else if (ShouldAddVSTestProviderProperties(serviceProvider)) // check here about vstest properties rather than inside CopyVSTestProperties function
   CopyVSTestProperties(testCase.Properties, testNode, testCase, testCase.GetPropertyValue, isTrxEnabled)
else
  throw exception?,

Maybe a check as well. If test framework sends both managed and vstest properties, then we can stop or warn them,

Copy link
Member Author

@Youssef1313 Youssef1313 Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MSTestDiscoverer.TestCategory needs to be copied regardless. It could be separated out of CopyVSTestProperties though, but we do it while we are looping already

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A suggestion then. Maybe we can rename CopyVSTestProperties to something generic CopyProperties and then pass different flags ShouldAddVSTestProviderProperties, ShouldCopySomething.

@@ -44,12 +65,10 @@ public static TestNode ToTestNode(this TestCase testCase, bool isTrxEnabled, ICl
}

private static void CopyVSTestProperties(IEnumerable<TestProperty> testProperties, TestNode testNode, TestCase testCase, Func<TestProperty, object?> getPropertyValue,
bool isTrxEnabled, IClientInfo client)
bool isTrxEnabled, bool addVSTestProviderProperties)
Copy link
Member

@fhnaseer fhnaseer Apr 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not like the idea of having bool addVSTestProviderProperties here. By just looking at the name CopyVSTestProperties, this function should not do anything if addVSTestProviderProperties = false. So, why someone should call it with false?

I would prefer to have

if (addVstestProperties)
   CopyVSTestProperties(...);

rather than

CopyVSTestProperties(..., false);

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here.

MSTestDiscoverer.TestCategory needs to be copied regardless.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

VSTestBridge doesn't use TestMethodIdentifierProperty
4 participants