[🐛 Bug]: provisioned providers are not installed during first deploy

[aksenk]

Describe the bug
Hi!

I'm facing a provisioned provider issue.

I deploying new installation of keep with configured provisioned provider.
After deploying I cant find provider in UI or database (table provider is empty).
I found following error in backend logs:

	
{"worker_type": "uvicorn", "asctime": "2025-04-03 09:41:30,872", "message": "Failed to provision provider vms", "levelname": "ERROR", "name": "keep.providers.providers_service", "filename": "providers_service.py", "otelTraceID": null, "otelSpanID": null, "otelTraceSampled": null, "otelServiceName": null, "threadName": "MainThread", "process": 1, "module": "providers_service", "exc_info": "Traceback (most recent call last):\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1967, in _exec_single_context\n    self.dialect.do_execute(\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py\", line 941, in do_execute\n    cursor.execute(statement, parameters)\npsycopg2.errors.ForeignKeyViolation: insert or update on table \"provider\" violates foreign key constraint \"provider_tenant_id_fkey\"\nDETAIL:  Key (tenant_id)=(keep) is not present in table \"tenant\".\n\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  File \"/venv/lib/python3.11/site-packages/keep/providers/providers_service.py\", line 200, in install_provider\n    session.commit()\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py\", line 2028, in commit\n    trans.commit(_to_root=True)\n  File \"<string>\", line 2, in commit\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py\", line 139, in _go\n    ret_value = fn(self, *arg, **kw)\n                ^^^^^^^^^^^^^^^^^^^^\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py\", line 1313, in commit\n    self._prepare_impl()\n  File \"<string>\", line 2, in _prepare_impl\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py\", line 139, in _go\n    ret_value = fn(self, *arg, **kw)\n                ^^^^^^^^^^^^^^^^^^^^\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py\", line 1288, in _prepare_impl\n    self.session.flush()\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py\", line 4352, in flush\n    self._flush(objects)\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py\", line 4487, in _flush\n    with util.safe_reraise():\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py\", line 146, in __exit__\n    raise exc_value.with_traceback(exc_tb)\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py\", line 4448, in _flush\n    flush_context.execute()\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py\", line 466, in execute\n    rec.execute(self)\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/unitofwork.py\", line 642, in execute\n    util.preloaded.orm_persistence.save_obj(\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py\", line 93, in save_obj\n    _emit_insert_statements(\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/orm/persistence.py\", line 1048, in _emit_insert_statements\n    result = connection.execute(\n             ^^^^^^^^^^^^^^^^^^^\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1418, in execute\n    return meth(\n           ^^^^^\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/sql/elements.py\", line 515, in _execute_on_connection\n    return connection._execute_clauseelement(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1640, in _execute_clauseelement\n    ret = self._execute_context(\n          ^^^^^^^^^^^^^^^^^^^^^^\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1846, in _execute_context\n    return self._exec_single_context(\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1986, in _exec_single_context\n    self._handle_dbapi_exception(\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 2355, in _handle_dbapi_exception\n    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py\", line 1967, in _exec_single_context\n    self.dialect.do_execute(\n  File \"/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py\", line 941, in do_execute\n    cursor.execute(statement, parameters)\nsqlalchemy.exc.IntegrityError: (psycopg2.errors.ForeignKeyViolation) insert or update on table \"provider\" violates foreign key constraint \"provider_tenant_id_fkey\"\nDETAIL:  Key (tenant_id)=(keep) is not present in table \"tenant\".\n\n[SQL: INSERT INTO provider (id, tenant_id, name, description, type, installed_by, installation_time, configuration_key, \"validatedScopes\", consumer, pulling_enabled, last_pull_time, provisioned) VALUES (%(id)s, %(tenant_id)s, %(name)s, %(description)s, %(type)s, %(installed_by)s, %(installation_time)s, %(configuration_key)s, %(validatedScopes)s::JSON, %(consumer)s, %(pulling_enabled)s, %(last_pull_time)s, %(provisioned)s)]\n[parameters: {'id': 'd71288b5f40f47b587225b416c447e6f', 'tenant_id': 'keep', 'name': 'vms', 'description': None, 'type': 'victoriametrics', 'installed_by': 'system', 'installation_time': datetime.datetime(2025, 4, 3, 9, 41, 30, 855719, tzinfo=datetime.timezone.utc), 'configuration_key': 'keep_victoriametrics_d71288b5f40f47b587225b416c447e6f', 'validatedScopes': '{}', 'consumer': False, 'pulling_enabled': True, 'last_pull_time': None, 'provisioned': True}]\n(Background on this error at: https://sqlalche.me/e/20/gkpj)\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/venv/lib/python3.11/site-packages/keep/providers/providers_service.py\", line 400, in provision_providers_from_env\n    ProvidersService.install_provider(\n  File \"/venv/lib/python3.11/site-packages/keep/providers/providers_service.py\", line 212, in install_provider\n    raise HTTPException(\nfastapi.exceptions.HTTPException: 409: Provider already installed"}

Looks like problem in tenant table (not exist row tenant_id=keep).

After first login (not sure that first login is required) I restart backend pod and the provider appeared in the database.

To Reproduce
Steps to reproduce the behavior:

1. Deploy new installation of keep with configured provisioned providers

2. Try to find installed provider in UI and database

[dosubot]

The issue you're encountering is due to the tenant table being empty during the first deployment of Keep, which causes a foreign key constraint error when trying to install a provider. This happens because the tenant_id specified for the provider does not exist in the tenant table ^[1].

To resolve this, ensure that the tenant table is properly initialized during the deployment process. This can be done by executing the Alembic migration script that creates the tenant table with the necessary structure ^[2]. Additionally, you can use the try_create_single_tenant function to create a default tenant if it doesn't already exist ^[3].

After ensuring the tenant table is initialized, you should be able to deploy the providers without encountering the foreign key constraint error.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[talboren]

Can you check if you are running the latest version? Issue shouldve been fixed in #4309

[aksenk]

yes, it works! thanks!

[aksenk]

but old provider still exist in database, it is OK? (rename vms -> victoriametrics)

-[ RECORD 1 ]-----+------------------------------------------------------
validatedScopes   | {}
id                | cd4df4ec5ca4498abdae8f54f1ce4877
tenant_id         | keep
name              | vms
description       |
type              | victoriametrics
installed_by      | system
installation_time | 2025-04-03 14:11:03.198118
configuration_key | keep_victoriametrics_cd4df4ec5ca4498abdae8f54f1ce4877
consumer          | f
last_pull_time    | 2025-04-03 15:13:19.669075
provisioned       | t
pulling_enabled   | t
-[ RECORD 2 ]-----+------------------------------------------------------
validatedScopes   | {}
id                | 8c5ed459ec064699a2a8babc12142339
tenant_id         | keep
name              | victoriametrics
description       |
type              | victoriametrics
installed_by      | system
installation_time | 2025-04-03 16:48:57.705686
configuration_key | keep_victoriametrics_8c5ed459ec064699a2a8babc12142339
consumer          | f
last_pull_time    | 2025-04-03 16:50:31.975431
provisioned       | t
pulling_enabled   | t

[talboren]

but old provider still exist in database, it is OK? (rename vms -> victoriametrics)

-[ RECORD 1 ]-----+------------------------------------------------------
validatedScopes   | {}
id                | cd4df4ec5ca4498abdae8f54f1ce4877
tenant_id         | keep
name              | vms
description       |
type              | victoriametrics
installed_by      | system
installation_time | 2025-04-03 14:11:03.198118
configuration_key | keep_victoriametrics_cd4df4ec5ca4498abdae8f54f1ce4877
consumer          | f
last_pull_time    | 2025-04-03 15:13:19.669075
provisioned       | t
pulling_enabled   | t
-[ RECORD 2 ]-----+------------------------------------------------------
validatedScopes   | {}
id                | 8c5ed459ec064699a2a8babc12142339
tenant_id         | keep
name              | victoriametrics
description       |
type              | victoriametrics
installed_by      | system
installation_time | 2025-04-03 16:48:57.705686
configuration_key | keep_victoriametrics_8c5ed459ec064699a2a8babc12142339
consumer          | f
last_pull_time    | 2025-04-03 16:50:31.975431
provisioned       | t
pulling_enabled   | t

It's not ideal, but we're working on a fix for that: #4399