🥅 Handle Anthropic RateLimitError

[lcombaldieu]

Simple change to catch and handle Anthropic RateLimitError

[zachary62]

There has already been a built-in mechanisms to retry after failure: https://the-pocket.github.io/PocketFlow/core_abstraction/node.html

https://github.com/The-Pocket/Tutorial-Codebase-Knowledge/blob/0f4152cc165bc941132a492cc8b786c247c795e8/flow.py#L17

Does this address the Anthropic RateLimitError?

[lcombaldieu]

I don't think so for some reason. Here is the error stack :

....
Fetched 64 files.
Identifying abstractions using LLM...
Identified 8 abstractions.
Analyzing relationships using LLM...
Analyzing relationships using LLM...
Analyzing relationships using LLM...
Analyzing relationships using LLM...
Analyzing relationships using LLM...
Traceback (most recent call last):
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/main.py", line 84, in <module>
    main()
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/main.py", line 81, in main
    tutorial_flow.run(shared)
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/pocketflow/__init__.py", line 16, in run
    return self._run(shared)
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/pocketflow/__init__.py", line 50, in _run
    def _run(self,shared): p=self.prep(shared); o=self._orch(shared); return self.post(shared,p,o)
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/pocketflow/__init__.py", line 48, in _orch
    while curr: curr.set_params(p); last_action=curr._run(shared); curr=copy.copy(self.get_next_node(curr,last_action))
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/pocketflow/__init__.py", line 13, in _run
    def _run(self,shared): p=self.prep(shared); e=self._exec(p); return self.post(shared,p,e)
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/pocketflow/__init__.py", line 33, in _exec
    if self.cur_retry==self.max_retries-1: return self.exec_fallback(prep_res,e)
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/pocketflow/__init__.py", line 28, in exec_fallback
    def exec_fallback(self,prep_res,exc): raise exc
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/pocketflow/__init__.py", line 31, in _exec
    try: return self.exec(prep_res)
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/nodes.py", line 286, in exec
    response = call_llm(prompt)
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/utils/call_llm.py", line 90, in call_llm
    response = client.messages.create(
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/anthropic/_utils/_utils.py", line 275, in wrapper
    return func(*args, **kwargs)
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/anthropic/resources/messages/messages.py", line 953, in create
    return self._post(
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/anthropic/_base_client.py", line 1336, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/anthropic/_base_client.py", line 1013, in request
    return self._request(
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/anthropic/_base_client.py", line 1102, in _request
    return self._retry_request(
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/anthropic/_base_client.py", line 1151, in _retry_request
    return self._request(
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/anthropic/_base_client.py", line 1102, in _request
    return self._retry_request(
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/anthropic/_base_client.py", line 1151, in _retry_request
    return self._request(
  File "/Users/lcombaldieu/Workspace/Tutorial-Codebase-Knowledge/venv/lib/python3.10/site-packages/anthropic/_base_client.py", line 1117, in _request
    raise self._make_status_error_from_response(err.response) from None
anthropic.RateLimitError: Error code: 429 - {'type': 'error', 'error': {'type': 'rate_limit_error', 'message': 'This request would exceed the rate limit for your organization (c0a6fc51-0945-42c8-9012-bcb5bc0f8399) of 20,000 input tokens per minute. For details, refer to: https://docs.anthropic.com/en/api/rate-limits. You can see the response headers for current usage. Please reduce the prompt length or the maximum tokens requested, or try again later. You may also contact sales at https://www.anthropic.com/contact-sales to discuss your options for a rate limit increase.'}}

Let me know if you think there is a more graceful way of handling this problem. My PR code fixed the issue

[lcombaldieu]

By the way thank you for the awesome repo. Here is the doc I generated for our open source project.
You can check it out and cite it as an example if you wish 😄

[zachary62]

Got it! I think what you can do is to increase the wait and retry time.
You can set it to something like: (max_retries=10, wait=70)
This will wait for 70s, and retry up to 10 times.

[zachary62]

Your project is very cool! Would you mind sharing it on https://github.com/The-Pocket/Tutorial-Codebase-Knowledge/discussions?