Skip to content

Enhance SWEBenchTestTool with Retry Mechanism and Better Error Handling #2169

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

chojuninengu
Copy link

This PR enhances the SWEBenchTestTool with improved error handling, retry capabilities, and better test result analysis. The changes make the test execution more robust and provide better visibility into test execution.

Changes

  • Added configurable retry mechanism for test execution
  • Added timeout configuration for test requests
  • Enhanced error handling with detailed error messages
  • Added execution time tracking
  • Improved logging with tracing
  • Added better documentation and tool descriptions
  • Added evaluation criteria and reward scales

Testing

The changes have been tested with:

  • Various timeout configurations
  • Different retry counts
  • Error scenarios
  • Successful test executions

Related Issues

Closes #2150

@chojuninengu
Copy link
Author

hello @treefrog00 , how about this one

Copy link
Contributor

agentfarmx bot commented Apr 17, 2025

No operation ID found for this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[agent_farm] testing
1 participant