diff --git a/docs/my-website/docs/proxy/db_deadlocks.md b/docs/my-website/docs/proxy/db_deadlocks.md new file mode 100644 index 0000000000..39498a7ec1 --- /dev/null +++ b/docs/my-website/docs/proxy/db_deadlocks.md @@ -0,0 +1,48 @@ +import Image from '@theme/IdealImage'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# High Availability Setup (Resolve DB Deadlocks) + +Resolve any Database Deadlocks you see in high traffic by using this setup + +## What causes the problem? + +LiteLLM writes `UPDATE` and `UPSERT` queries to the DB. When using 10+ pods of LiteLLM, these queries can cause deadlocks since each pod could simultaneously attempt to update the same `user_id`, `team_id`, `key` etc. + +## How the high availability setup fixes the problem +- All pods will write to a Redis queue instead of the DB. +- A single pod will acquire a lock on the DB and flush the redis queue to the DB. + + +## How it works + +### Stage 1. Each pod writes updates to redis + +Each pod will accumlate the spend updates for a key, user, team, etc and write the updates to a redis queue. + + + + +### Stage 2. A single pod flushes the redis queue to the DB + +A single pod will acquire a lock on the DB and flush all elements in the redis queue to the DB. + + + + + +## Setup +- Redis +- Postgres + +```yaml showLineNumbers title="litellm proxy_config.yaml" +general_settings: + use_redis_transaction_buffer: true + +litellm_settings: + cache: True + cache_params: + type: redis + supported_call_types: [] +``` diff --git a/docs/my-website/img/deadlock_fix_1.png b/docs/my-website/img/deadlock_fix_1.png new file mode 100644 index 0000000000..3dff86a4d2 Binary files /dev/null and b/docs/my-website/img/deadlock_fix_1.png differ diff --git a/docs/my-website/img/deadlock_fix_2.png b/docs/my-website/img/deadlock_fix_2.png new file mode 100644 index 0000000000..345acff586 Binary files /dev/null and b/docs/my-website/img/deadlock_fix_2.png differ diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index 1542edceb5..1cb3e02707 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -53,7 +53,7 @@ const sidebars = { { type: "category", label: "Architecture", - items: ["proxy/architecture", "proxy/db_info", "router_architecture", "proxy/user_management_heirarchy", "proxy/jwt_auth_arch", "proxy/image_handling"], + items: ["proxy/architecture", "proxy/db_info", "proxy/db_deadlocks", "router_architecture", "proxy/user_management_heirarchy", "proxy/jwt_auth_arch", "proxy/image_handling"], }, { type: "link", diff --git a/litellm/proxy/proxy_config.yaml b/litellm/proxy/proxy_config.yaml index 17658df903..fe8d73d26a 100644 --- a/litellm/proxy/proxy_config.yaml +++ b/litellm/proxy/proxy_config.yaml @@ -4,3 +4,12 @@ model_list: model: openai/fake api_key: fake-key api_base: https://exampleopenaiendpoint-production.up.railway.app/ + +general_settings: + use_redis_transaction_buffer: true + +litellm_settings: + cache: True + cache_params: + type: redis + supported_call_types: [] \ No newline at end of file