Commit graph

17755 commits

Author SHA1 Message Date
Krish Dholakia
d46660ea0f
LiteLLM Minor Fixes & Improvements (09/18/2024) (#5772)
* fix(proxy_server.py): fix azure key vault logic to not require client id/secret

* feat(cost_calculator.py): support fireworks ai cost tracking

* build(docker-compose.yml): add lines for mounting config.yaml to docker compose

Closes https://github.com/BerriAI/litellm/issues/5739

* fix(input.md): update docs to clarify litellm supports content as a list of dictionaries

Fixes https://github.com/BerriAI/litellm/issues/5755

* fix(input.md): update input.md to include all message values

* fix(image_handling.py): follow image url redirects

Fixes https://github.com/BerriAI/litellm/issues/5763

* fix(router.py): Fix model key/base leak in error message

Fixes https://github.com/BerriAI/litellm/issues/5762

* fix(http_handler.py): fix linting error

* fix(azure.py): fix logging to show azure_ad_token being used

Fixes https://github.com/BerriAI/litellm/issues/5767

* fix(_redis.py): add redis sentinel support

Closes https://github.com/BerriAI/litellm/issues/4381

* feat(_redis.py): add redis sentinel support

Closes https://github.com/BerriAI/litellm/issues/4381

* test(test_completion_cost.py): fix test

* Databricks Integration: Integrate Databricks SDK as optional mechanism for fetching API base and token, if unspecified (#5746)

* LiteLLM Minor Fixes & Improvements (09/16/2024)  (#5723)

* coverage (#5713)

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Move (#5714)

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(litellm_logging.py): fix logging client re-init (#5710)

Fixes https://github.com/BerriAI/litellm/issues/5695

* fix(presidio.py): Fix logging_hook response and add support for additional presidio variables in guardrails config

Fixes https://github.com/BerriAI/litellm/issues/5682

* feat(o1_handler.py): fake streaming for openai o1 models

Fixes https://github.com/BerriAI/litellm/issues/5694

* docs: deprecated traceloop integration in favor of native otel (#5249)

* fix: fix linting errors

* fix: fix linting errors

* fix(main.py): fix o1 import

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view (#5730)

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view

Supports having `MonthlyGlobalSpend` view be a material view, and exposes an endpoint to refresh it

* fix(custom_logger.py): reset calltype

* fix: fix linting errors

* fix: fix linting error

* fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix: fix import

* Fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* DB test

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Coverage

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* progress

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix test name

Signed-off-by: dbczumar <corey.zumar@databricks.com>

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Krish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>

* test: fix test

* test(test_databricks.py): fix test

* fix(databricks/chat.py): handle custom endpoint (e.g. sagemaker)

* Apply code scanning fix for clear-text logging of sensitive information

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* fix(__init__.py): fix known fireworks ai models

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2024-09-19 13:25:29 -07:00
Ishaan Jaff
49b2766723
add gemma2 9b it (#5788) 2024-09-19 13:03:33 -07:00
Ishaan Jaff
cd90807807
fix use converse for all llama3 models (#5729) 2024-09-19 09:31:52 -07:00
Krish Dholakia
8497e2aa36
feat(prometheus_api.py): support querying prometheus metrics for all-up + key-level spend on UI (#5782)
enables getting aggregated view from prometheus api

Makes proxy UI reliable in prod
2024-09-18 22:39:15 -07:00
Ishaan Jaff
a22e473636 set timeout on predibase test 2024-09-18 17:13:13 -07:00
Ishaan Jaff
c60f6f496a bump: version 1.46.5 → 1.46.6 2024-09-18 16:45:46 -07:00
Ishaan Jaff
4399deab2e docs fallback/login 2024-09-18 16:43:19 -07:00
Ishaan Jaff
5480563281 docs add info on /fallback/login 2024-09-18 16:41:19 -07:00
Ishaan Jaff
eba76377ca
[Chore-Proxy] enforce jwt auth as enterprise feature (#5770)
* enforce prometheus as enterprise feature

* show correct error on prometheus metric when not enrterprise user

* docs promethues metrics enforced

* docs enforce JWT auth

* enforce JWT auth as enterprise feature

* fix merge conflicts
2024-09-18 16:28:37 -07:00
Ishaan Jaff
50cc7c0353
[Chore LiteLLM Proxy] enforce prometheus metrics as enterprise feature (#5769)
* enforce prometheus as enterprise feature

* show correct error on prometheus metric when not enrterprise user

* docs promethues metrics enforced

* fix enforcing
2024-09-18 16:28:12 -07:00
Ishaan Jaff
7e07c37be7
[Feat-Proxy] Add Azure Assistants API - Create Assistant, Delete Assistant Support (#5777)
* update docs to show providers

* azure - move assistants in it's own file

* create new azure assistants file

* add azure create assistants

* add test for create / delete assistants

* azure add delete assistants support

* docs add Azure to support providers for assistants api

* fix linting errors

* fix standard logging merge conflict

* docs azure create assistants

* fix doc
2024-09-18 16:27:33 -07:00
Ishaan Jaff
a109853d21
[Prometheus] track requested model (#5774)
* enforce prometheus as enterprise feature

* show correct error on prometheus metric when not enrterprise user

* docs promethues metrics enforced

* track requested model on prometheus

* docs prom metrics

* fix prom tracking failures
2024-09-18 12:46:58 -07:00
Ishaan Jaff
5aad3e6ea4
[Feat - GCS Bucket Logger] Use StandardLoggingPayload (#5771)
* docs update standard logging object

* GCSBucketLogger

* test gcs bucket logger
2024-09-18 11:37:52 -07:00
Krrish Dholakia
8600ec7704 fix(litellm_logging.py): fix merge conflict 2024-09-18 10:49:57 -07:00
Ishaan Jaff
84e813b0f4 update gcs bucket to use standard logging payload 2024-09-18 10:34:21 -07:00
Ishaan Jaff
a4549b5b6c docs update what gets logged on gcs buckets 2024-09-18 10:18:57 -07:00
Ishaan Jaff
aa84bcebaf docs update standard logging object 2024-09-18 10:17:09 -07:00
Ishaan Jaff
2987b14f3b docs clarify how virtual key is read from cache / db 2024-09-18 09:39:54 -07:00
Krrish Dholakia
920280155b docs(azure_ai.md): add rerank api endpoint to docs 2024-09-17 23:06:19 -07:00
Krrish Dholakia
388e946df0 bump: version 1.46.4 → 1.46.5 2024-09-17 23:02:27 -07:00
Krish Dholakia
9c8fdee068
Additional Fixes (09/17/2024) (#5759)
* fix(auth_checks.py): check if key has all model access via wildcard routing

Fixes issue where key with `openai/*` couldn't call gpt models

* fix(slack_alerting.py): expose flag for disabling failed spend tracking alerts
2024-09-17 23:02:12 -07:00
Krish Dholakia
98c335acd0
LiteLLM Minor Fixes & Improvements (09/17/2024) (#5742)
* fix(proxy_server.py): use default azure credentials to support azure non-client secret kms

* fix(langsmith.py): raise error if credentials missing

* feat(langsmith.py): support error logging for langsmith + standard logging payload

Fixes https://github.com/BerriAI/litellm/issues/5738

* Fix hardcoding of schema in view check (#5749)

* fix - deal with case when check view exists returns None (#5740)

* Revert "fix - deal with case when check view exists returns None (#5740)" (#5741)

This reverts commit 535228159b.

* test(test_router_debug_logs.py): move to mock response

* Fix hardcoding of schema

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>

* fix(proxy_server.py): allow admin to disable ui via `DISABLE_ADMIN_UI` flag

* fix(router.py): fix default model name value

Fixes 55db19a1e4 (r1763712148)

* fix(utils.py): fix unbound variable error

* feat(rerank/main.py): add azure ai rerank endpoints

Closes https://github.com/BerriAI/litellm/issues/5667

* feat(secret_detection.py): Allow configuring secret detection params

Allows admin to control what plugins to run for secret detection. Prevents overzealous secret detection.

* docs(secret_detection.md): add secret detection guardrail docs

* fix: fix linting errors

* fix - deal with case when check view exists returns None (#5740)

* Revert "fix - deal with case when check view exists returns None (#5740)" (#5741)

This reverts commit 535228159b.

* Litellm fix router testing (#5748)

* test: fix testing - azure changed content policy error logic

* test: fix tests to use mock responses

* test(test_image_generation.py): handle api instability

* test(test_image_generation.py): handle azure api instability

* fix(utils.py): fix unbounded variable error

* fix(utils.py): fix unbounded variable error

* test: refactor test to use mock response

* test: mark flaky azure tests

* Bump next from 14.1.1 to 14.2.10 in /ui/litellm-dashboard (#5753)

Bumps [next](https://github.com/vercel/next.js) from 14.1.1 to 14.2.10.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v14.1.1...v14.2.10)

---
updated-dependencies:
- dependency-name: next
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* [Fix] o1-mini causes pydantic warnings on `reasoning_tokens`  (#5754)

* add requester_metadata in standard logging payload

* log requester_metadata in metadata

* use StandardLoggingPayload for logging

* docs StandardLoggingPayload

* fix import

* include standard logging object in failure

* add test for requester metadata

* handle completion_tokens_details

* add test for completion_tokens_details

* [Feat-Proxy-DataDog] Log Redis, Postgres Failure events on DataDog  (#5750)

* dd - start tracking redis status on dd

* add async_service_succes_hook / failure hook in custom logger

* add async_service_failure_hook

* log service failures on dd

* fix import error

* add test for redis errors / warning

* [Fix] Router/ Proxy - Tag Based routing, raise correct error when no deployments found and tag filtering is on  (#5745)

* fix tag routing - raise correct error when no model with tag based routing

* fix error string from tag based routing

* test router tag based routing

* raise 401 error when no tags avialable for deploymen

* linting fix

* [Feat] Log Request metadata on gcs bucket logging (#5743)

* add requester_metadata in standard logging payload

* log requester_metadata in metadata

* use StandardLoggingPayload for logging

* docs StandardLoggingPayload

* fix import

* include standard logging object in failure

* add test for requester metadata

* fix(litellm_logging.py): fix logging message

* fix(rerank_api/main.py): fix linting errors

* fix(custom_guardrails.py): maintain backwards compatibility for older guardrails

* fix(rerank_api/main.py): fix cost tracking for rerank endpoints

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: steffen-sbt <148480574+steffen-sbt@users.noreply.github.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-17 23:00:04 -07:00
Ishaan Jaff
c5c64a6c04 bump: version 1.46.3 → 1.46.4 2024-09-17 20:42:47 -07:00
Ishaan Jaff
7f638cd60d bump: version 1.46.2 → 1.46.3 2024-09-17 20:42:43 -07:00
Ishaan Jaff
be96c79b3c update datadog docs 2024-09-17 20:42:36 -07:00
Ishaan Jaff
d3406c92aa
[Feat] Log Request metadata on gcs bucket logging (#5743)
* add requester_metadata in standard logging payload

* log requester_metadata in metadata

* use StandardLoggingPayload for logging

* docs StandardLoggingPayload

* fix import

* include standard logging object in failure

* add test for requester metadata
2024-09-17 20:25:39 -07:00
Ishaan Jaff
1bb1f70a47
[Fix] Router/ Proxy - Tag Based routing, raise correct error when no deployments found and tag filtering is on (#5745)
* fix tag routing - raise correct error when no model with tag based routing

* fix error string from tag based routing

* test router tag based routing

* raise 401 error when no tags avialable for deploymen

* linting fix
2024-09-17 20:24:28 -07:00
Ishaan Jaff
911230c434
[Feat-Proxy-DataDog] Log Redis, Postgres Failure events on DataDog (#5750)
* dd - start tracking redis status on dd

* add async_service_succes_hook / failure hook in custom logger

* add async_service_failure_hook

* log service failures on dd

* fix import error

* add test for redis errors / warning
2024-09-17 20:24:06 -07:00
Ishaan Jaff
7f4dfe434a
[Fix] o1-mini causes pydantic warnings on reasoning_tokens (#5754)
* add requester_metadata in standard logging payload

* log requester_metadata in metadata

* use StandardLoggingPayload for logging

* docs StandardLoggingPayload

* fix import

* include standard logging object in failure

* add test for requester metadata

* handle completion_tokens_details

* add test for completion_tokens_details
2024-09-17 20:23:14 -07:00
dependabot[bot]
d0425e7767
Bump next from 14.1.1 to 14.2.10 in /ui/litellm-dashboard (#5753)
Bumps [next](https://github.com/vercel/next.js) from 14.1.1 to 14.2.10.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v14.1.1...v14.2.10)

---
updated-dependencies:
- dependency-name: next
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2024-09-17 18:21:58 -07:00
Krish Dholakia
dd602753c0
Litellm fix router testing (#5748)
* test: fix testing - azure changed content policy error logic

* test: fix tests to use mock responses

* test(test_image_generation.py): handle api instability

* test(test_image_generation.py): handle azure api instability

* fix(utils.py): fix unbounded variable error

* fix(utils.py): fix unbounded variable error

* test: refactor test to use mock response

* test: mark flaky azure tests
2024-09-17 18:02:23 -07:00
Krrish Dholakia
8d4339c702 test(test_router_debug_logs.py): move to mock response 2024-09-17 11:38:47 -07:00
Ishaan Jaff
8de6e3d3ba
Revert "fix - deal with case when check view exists returns None (#5740)" (#5741)
This reverts commit 535228159b.
2024-09-17 09:04:22 -07:00
Ishaan Jaff
535228159b
fix - deal with case when check view exists returns None (#5740) 2024-09-17 08:38:19 -07:00
Krrish Dholakia
815d46f9e1 bump: version 1.46.1 → 1.46.2 2024-09-17 08:06:11 -07:00
Krish Dholakia
234185ec13
LiteLLM Minor Fixes & Improvements (09/16/2024) (#5723) (#5731)
* LiteLLM Minor Fixes & Improvements (09/16/2024)  (#5723)

* coverage (#5713)

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Move (#5714)

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(litellm_logging.py): fix logging client re-init (#5710)

Fixes https://github.com/BerriAI/litellm/issues/5695

* fix(presidio.py): Fix logging_hook response and add support for additional presidio variables in guardrails config

Fixes https://github.com/BerriAI/litellm/issues/5682

* feat(o1_handler.py): fake streaming for openai o1 models

Fixes https://github.com/BerriAI/litellm/issues/5694

* docs: deprecated traceloop integration in favor of native otel (#5249)

* fix: fix linting errors

* fix: fix linting errors

* fix(main.py): fix o1 import

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view (#5730)

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view

Supports having `MonthlyGlobalSpend` view be a material view, and exposes an endpoint to refresh it

* fix(custom_logger.py): reset calltype

* fix: fix linting errors

* fix: fix linting error

* fix: fix import

* test(test_databricks.py): fix databricks tests

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>
2024-09-17 08:05:52 -07:00
Ishaan Jaff
1e59395280 fix guardrail linting change 2024-09-16 20:12:54 -07:00
Ishaan Jaff
7e9dbcd1a3 fix gemini 1.5 flash test 2024-09-16 19:37:41 -07:00
Ishaan Jaff
8762b64b1d ci/cd run again 2024-09-16 18:26:53 -07:00
Ishaan Jaff
9f5a33015f fix linting 2024-09-16 18:07:48 -07:00
Ishaan Jaff
3b034224a0 bump: version 1.46.0 → 1.46.1 2024-09-16 18:06:34 -07:00
Ishaan Jaff
b6ae2204a8
[Feat-Proxy] Slack Alerting - allow using os.environ/ vars for alert to webhook url (#5726)
* allow using os.environ for slack urls

* use env vars for webhook urls

* fix types for get_secret

* fix linting

* fix linting

* fix linting

* linting fixes

* linting fix

* docs alerting slack

* fix get data
2024-09-16 18:03:37 -07:00
Ishaan Jaff
8103e2b2da
[Fix-Proxy] Azure Key Management - Secret Manager (#5728)
* fix azure key mgtm error

* add test for azure kms

* add test for azure kms
2024-09-16 18:01:40 -07:00
Ishaan Jaff
ca6d99e1ad fix gemini 1.5 flash supports_response_schema 2024-09-16 17:59:59 -07:00
Ishaan Jaff
ce7a937a53 fix test_all_model_config 2024-09-16 17:47:31 -07:00
Ishaan Jaff
4dcb092d12 fix test_all_model_configs 2024-09-16 17:44:48 -07:00
Ishaan Jaff
7b09591ca6
[Fix-Proxy] log exceptions from azure key vault on verbose_logger.exceptions (#5719)
* log exceptions from azure key vault

* fix error from azure key vault
2024-09-16 16:58:37 -07:00
Ishaan Jaff
8fbe2abb89
[Feat-Proxy] Add upperbound key duration param (#5727)
* add upperbound key duration param

* use upper bound values when None set

* docs upperbound params
2024-09-16 16:28:36 -07:00
Ishaan Jaff
3a5039e284
Warning fix for Pydantic 2.0 (#5679) (#5707)
* Fixed UserWarning: Valid config keys have changed in V2 underscore_attrs_are_private

* Trying different method

Co-authored-by: CyanideByte <cyanidebyte@hotmail.com>
2024-09-16 11:24:12 -07:00
David Manouchehri
aa64c34ce6
Add unsupported params. (#5722) 2024-09-16 09:43:50 -07:00