phoenix/litellm
1
0
Fork 0
forked from phoenix/litellm-mirror
Code Pull requests Activity
18061 commits 1603 branches 645 tags 320 MiB
Commit graph

2752 commits

Author SHA1 Message Date
yujonglee
4132a97787
bump (#6187) 2024-10-14 18:22:54 +05:30
Ishaan Jaff
4d1b4beb3d
(refactor) caching use LLMCachingHandler for async_get_cache and set_cache (#6208) 
* use folder for caching

* fix importing caching

* fix clickhouse pyright

* fix linting

* fix correctly pass kwargs and args

* fix test case for embedding

* fix linting

* fix embedding caching logic

* fix refactor handle utils.py

* fix test_embedding_caching_azure_individual_items_reordered
 2024-10-14 16:34:01 +05:30
Krrish Dholakia
806a1c4acc docs: make it easier to find anthropic/openai prompt caching doc 2024-10-13 18:34:13 -07:00
Krish Dholakia
15b44c3221
docs(configs.md): document all environment variables (#6185) 2024-10-13 09:57:03 -07:00
Krish Dholakia
2acb0c0675
Litellm Minor Fixes & Improvements (10/12/2024) (#6179) 
* build(model_prices_and_context_window.json): add bedrock llama3.2 pricing

* build(model_prices_and_context_window.json): add bedrock cross region inference pricing

* Revert "(perf) move s3 logging to Batch logging + async [94% faster perf under 100 RPS on 1 litellm instance] (#6165)"

This reverts commit 2a5624af47.

* add azure/gpt-4o-2024-05-13 (#6174)

* LiteLLM Minor Fixes & Improvements (10/10/2024)  (#6158)

* refactor(vertex_ai_partner_models/anthropic): refactor anthropic to use partner model logic

* fix(vertex_ai/): support passing custom api base to partner models

Fixes https://github.com/BerriAI/litellm/issues/4317

* fix(proxy_server.py): Fix prometheus premium user check logic

* docs(prometheus.md): update quick start docs

* fix(custom_llm.py): support passing dynamic api key + api base

* fix(realtime_api/main.py): Add request/response logging for realtime api endpoints

Closes https://github.com/BerriAI/litellm/issues/6081

* feat(openai/realtime): add openai realtime api logging

Closes https://github.com/BerriAI/litellm/issues/6081

* fix(realtime_streaming.py): fix linting errors

* fix(realtime_streaming.py): fix linting errors

* fix: fix linting errors

* fix pattern match router

* Add literalai in the sidebar observability category (#6163)

* fix: add literalai in the sidebar

* fix: typo

* update (#6160)

* Feat: Add Langtrace integration (#5341)

* Feat: Add Langtrace integration

* add langtrace service name

* fix timestamps for traces

* add tests

* Discard Callback + use existing otel logger

* cleanup

* remove print statments

* remove callback

* add docs

* docs

* add logging docs

* format logging

* remove emoji and add litellm proxy example

* format logging

* format `logging.md`

* add langtrace docs to logging.md

* sync conflict

* docs fix

* (perf) move s3 logging to Batch logging + async [94% faster perf under 100 RPS on 1 litellm instance] (#6165)

* fix move s3 to use customLogger

* add basic s3 logging test

* add s3 to custom logger compatible

* use batch logger for s3

* s3 set flush interval and batch size

* fix s3 logging

* add notes on s3 logging

* fix s3 logging

* add basic s3 logging test

* fix s3 type errors

* add test for sync logging on s3

* fix: fix to debug log

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Willy Douhard <willy.douhard@gmail.com>
Co-authored-by: yujonglee <yujonglee.dev@gmail.com>
Co-authored-by: Ali Waleed <ali@scale3labs.com>

* docs(custom_llm_server.md): update doc on passing custom params

* fix(pass_through_endpoints.py): don't require headers

Fixes https://github.com/BerriAI/litellm/issues/6128

* feat(utils.py): add support for caching rerank endpoints

Closes https://github.com/BerriAI/litellm/issues/6144

* feat(litellm_logging.py'): add response headers for failed requests

Closes https://github.com/BerriAI/litellm/issues/6159

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Willy Douhard <willy.douhard@gmail.com>
Co-authored-by: yujonglee <yujonglee.dev@gmail.com>
Co-authored-by: Ali Waleed <ali@scale3labs.com>
 2024-10-12 11:48:34 -07:00
Krish Dholakia
11f9df923a
LiteLLM Minor Fixes & Improvements (10/10/2024) (#6158) 
* refactor(vertex_ai_partner_models/anthropic): refactor anthropic to use partner model logic

* fix(vertex_ai/): support passing custom api base to partner models

Fixes https://github.com/BerriAI/litellm/issues/4317

* fix(proxy_server.py): Fix prometheus premium user check logic

* docs(prometheus.md): update quick start docs

* fix(custom_llm.py): support passing dynamic api key + api base

* fix(realtime_api/main.py): Add request/response logging for realtime api endpoints

Closes https://github.com/BerriAI/litellm/issues/6081

* feat(openai/realtime): add openai realtime api logging

Closes https://github.com/BerriAI/litellm/issues/6081

* fix(realtime_streaming.py): fix linting errors

* fix(realtime_streaming.py): fix linting errors

* fix: fix linting errors

* fix pattern match router

* Add literalai in the sidebar observability category (#6163)

* fix: add literalai in the sidebar

* fix: typo

* update (#6160)

* Feat: Add Langtrace integration (#5341)

* Feat: Add Langtrace integration

* add langtrace service name

* fix timestamps for traces

* add tests

* Discard Callback + use existing otel logger

* cleanup

* remove print statments

* remove callback

* add docs

* docs

* add logging docs

* format logging

* remove emoji and add litellm proxy example

* format logging

* format `logging.md`

* add langtrace docs to logging.md

* sync conflict

* docs fix

* (perf) move s3 logging to Batch logging + async [94% faster perf under 100 RPS on 1 litellm instance] (#6165)

* fix move s3 to use customLogger

* add basic s3 logging test

* add s3 to custom logger compatible

* use batch logger for s3

* s3 set flush interval and batch size

* fix s3 logging

* add notes on s3 logging

* fix s3 logging

* add basic s3 logging test

* fix s3 type errors

* add test for sync logging on s3

* fix: fix to debug log

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: Willy Douhard <willy.douhard@gmail.com>
Co-authored-by: yujonglee <yujonglee.dev@gmail.com>
Co-authored-by: Ali Waleed <ali@scale3labs.com>
 2024-10-11 23:04:36 -07:00
Ishaan Jaff
4e1c892dfc docs fix 2024-10-11 19:32:59 +05:30
Ali Waleed
7ec414a3cf
Feat: Add Langtrace integration (#5341) 
* Feat: Add Langtrace integration

* add langtrace service name

* fix timestamps for traces

* add tests

* Discard Callback + use existing otel logger

* cleanup

* remove print statments

* remove callback

* add docs

* docs

* add logging docs

* format logging

* remove emoji and add litellm proxy example

* format logging

* format `logging.md`

* add langtrace docs to logging.md

* sync conflict
 2024-10-11 19:19:53 +05:30
yujonglee
42174fde4e
update (#6160) 2024-10-11 19:18:56 +05:30
Willy Douhard
8b00d2a25f
Add literalai in the sidebar observability category (#6163) 
* fix: add literalai in the sidebar

* fix: typo
 2024-10-11 19:18:47 +05:30
Jacques Verré
4064bfc6dd
[Feat] Observability integration - Opik by Comet (#6062) 
* Added Opik logging and evaluation

* Updated doc examples

* Default tags should be [] in case appending

* WIP

* Work in progress

* Opik integration

* Opik integration

* Revert changes on litellm_logging.py

* Updated Opik integration for synchronous API calls

* Updated Opik documentation

---------

Co-authored-by: Douglas Blank <doug@comet.com>
Co-authored-by: Doug Blank <doug.blank@gmail.com>
 2024-10-10 18:27:50 +05:30
Ishaan Jaff
89506053a4
(feat) use regex pattern matching for wildcard routing (#6150) 
* use pattern matching for llm deployments

* code quality fix

* fix linting

* add types to PatternMatchRouter

* docs add example config for regex patterns
 2024-10-10 18:24:16 +05:30
Krrish Dholakia
60baa65e0e docs(configs.md): add litellm config / s3 bucket object info in configs.md 2024-10-09 09:07:43 -07:00
Ishaan Jaff
b35da5014b doc onboarding orgs 2024-10-09 19:11:36 +05:30
Ishaan Jaff
5da6863804 docs rbac 2024-10-09 16:46:26 +05:30
Ishaan Jaff
399f50d558 fix rbac doc 2024-10-09 16:44:46 +05:30
Ishaan Jaff
0e83a68a69 doc - move rbac under auth 2024-10-09 15:27:32 +05:30
Ishaan Jaff
1fd437e263
(feat proxy) [beta] add support for organization role based access controls (#6112) 
* track LiteLLM_OrganizationMembership

* add add_internal_user_to_organization

* add org membership to schema

* read organization membership when reading user info in auth checks

* add check for valid organization_id

* add test for test_create_new_user_in_organization

* test test_create_new_user_in_organization

* add new ADMIN role

* add test for org admins creating teams

* add test for test_org_admin_create_user_permissions

* test_org_admin_create_user_team_wrong_org_permissions

* test_org_admin_create_user_team_wrong_org_permissions

* fix organization_role_based_access_check

* fix getting user members

* fix TeamBase

* fix types used for use role

* fix type checks

* sync prisma schema

* docs - organization admins

* fix use organization_endpoints for /organization management

* add types for org member endpoints

* fix role name for org admin

* add type for member add response

* add organization/member_add

* add error handling for adding members to an org

* add nice doc string for oranization/member_add

* fix test_create_new_user_in_organization

* linting fix

* use simple route changes

* fix types

* add organization member roles

* add org admin auth checks

* add auth checks for orgs

* test for creating teams as org admin

* simplify org id usage

* fix typo

* test test_org_admin_create_user_team_wrong_org_permissions

* fix type check issue

* code quality fix

* fix schema.prisma
 2024-10-09 15:18:18 +05:30
Ishaan Jaff
d1760b1b04
(fix) clean up root repo - move entrypoint.sh and build_admin_ui to /docker (#6110) 
* fix move docker files to docker folders

* move check file length

* fix docker hub deploy

* fix clean up root

* fix circle ci config
 2024-10-08 11:34:43 +05:30
Krrish Dholakia
cc960da4b6 docs(azure.md): add o1 model support to config 2024-10-07 22:37:49 -07:00
Krish Dholakia
6729c9ca7f
LiteLLM Minor Fixes & Improvements (10/07/2024) (#6101) 
* fix(utils.py): support dropping temperature param for azure o1 models

* fix(main.py): handle azure o1 streaming requests

o1 doesn't support streaming, fake it to ensure code works as expected

* feat(utils.py): expose `hosted_vllm/` endpoint, with tool handling for vllm

Fixes https://github.com/BerriAI/litellm/issues/6088

* refactor(internal_user_endpoints.py): cleanup unused params + update docstring

Closes https://github.com/BerriAI/litellm/issues/6100

* fix(main.py): expose custom image generation api support

Fixes https://github.com/BerriAI/litellm/issues/6097

* fix: fix linting errors

* docs(custom_llm_server.md): add docs on custom api for image gen calls

* fix(types/utils.py): handle dict type

* fix(types/utils.py): fix linting errors
 2024-10-07 22:17:22 -07:00
Ishaan Jaff
ef815f3a84
(docs) add remaining litellm settings on configs.md doc (#6108) 
* docs add litellm settings configs

* docs langfuse tags on config
 2024-10-08 07:57:04 +05:30
Ishaan Jaff
2b370f8e9e
(docs) key based callbacks (#6107) 2024-10-08 07:12:01 +05:30
Pradyumna Singh Rathore
b7ba558b74
fix links due to broken list (#6103) 2024-10-07 15:47:29 -04:00
Ishaan Jaff
1bafbf8382
(feat proxy) add v2 maintained LiteLLM grafana dashboard (#6098) 
* add new grafana dashboard litellm

* add v2 grafana dashboard
 2024-10-07 18:11:20 +05:30
Ishaan Jaff
b2fbee3923 docs key logging 2024-10-06 13:49:27 +05:30
Ishaan Jaff
fd7014a326 correct use of healthy / unhealthy 2024-10-06 13:48:30 +05:30
Krish Dholakia
04e5963b65
Litellm expose disable schema update flag (#6085) 
* fix: enable new 'disable_prisma_schema_update' flag

* build(config.yml): remove setup remote docker step

* ci(config.yml): give container time to start up

* ci(config.yml): update test

* build(config.yml): actually start docker

* build(config.yml): simplify grep check

* fix(prisma_client.py): support reading disable_schema_update via env vars

* ci(config.yml): add test to check if all general settings are documented

* build(test_General_settings.py): check available dir

* ci: check ../ repo path

* build: check ./

* build: fix test
 2024-10-05 21:26:51 -04:00
Krish Dholakia
f2c0a31e3c
LiteLLM Minor Fixes & Improvements (10/05/2024) (#6083) 
* docs(prompt_caching.md): add prompt caching cost calc example to docs

* docs(prompt_caching.md): add proxy examples to docs

* feat(utils.py): expose new helper `supports_prompt_caching()` to check if a model supports prompt caching

* docs(prompt_caching.md): add docs on checking model support for prompt caching

* build: fix invalid json
 2024-10-05 18:59:11 -04:00
Ishaan Jaff
6e6d38841f docs fix 2024-10-05 15:25:25 +05:30
Ishaan Jaff
5ee1342d37
(docs) reference router settings general settings etc (#6078) 2024-10-05 15:01:28 +05:30
Ishaan Jaff
d2f17cf97c docs routing config table 2024-10-05 14:40:07 +05:30
Ishaan Jaff
530915da51 add o-1 to Azure docs 2024-10-05 14:23:54 +05:30
Ishaan Jaff
c84cfe977e
(feat) add /key/health endpoint to test key based logging (#6073) 
* add /key/health endpoint

* add /key/health endpoint

* fix return from /key/health

* update doc string

* fix doc string for /key/health

* add test for /key/health

* fix linting

* docs /key/health
 2024-10-05 11:56:55 +05:30
Krish Dholakia
2e5c46ef6d
LiteLLM Minor Fixes & Improvements (10/04/2024) (#6064) 
* fix(litellm_logging.py): ensure cache hits are scrubbed if 'turn_off_message_logging' is enabled

* fix(sagemaker.py): fix streaming to raise error immediately

Fixes https://github.com/BerriAI/litellm/issues/6054

* (fixes)  gcs bucket key based logging  (#6044)

* fixes for gcs bucket logging

* fix StandardCallbackDynamicParams

* fix - gcs logging when payload is not serializable

* add test_add_callback_via_key_litellm_pre_call_utils_gcs_bucket

* working success callbacks

* linting fixes

* fix linting error

* add type hints to functions

* fixes for dynamic success and failure logging

* fix for test_async_chat_openai_stream

* fix handle case when key based logging vars are set as os.environ/ vars

* fix prometheus track cooldown events on custom logger (#6060)

* (docs) add 1k rps load test doc  (#6059)

* docs 1k rps load test

* docs load testing

* docs load testing litellm

* docs load testing

* clean up load test doc

* docs prom metrics for load testing

* docs using prometheus on load testing

* doc load testing with prometheus

* (fixes) docs + qa - gcs key based logging  (#6061)

* fixes for required values for gcs bucket

* docs gcs bucket logging

* bump: version 1.48.12 → 1.48.13

* ci/cd run again

* bump: version 1.48.13 → 1.48.14

* update load test doc

* (docs) router settings - on litellm config  (#6037)

* add yaml with all router settings

* add docs for router settings

* docs router settings litellm settings

* (feat)  OpenAI prompt caching models to model cost map (#6063)

* add prompt caching for latest models

* add cache_read_input_token_cost for prompt caching models

* fix(litellm_logging.py): check if param is iterable

Fixes https://github.com/BerriAI/litellm/issues/6025#issuecomment-2393929946

* fix(factory.py): support passing an 'assistant_continue_message' to prevent bedrock error

Fixes https://github.com/BerriAI/litellm/issues/6053

* fix(databricks/chat): handle streaming responses

* fix(factory.py): fix linting error

* fix(utils.py): unify anthropic + deepseek prompt caching information to openai format

Fixes https://github.com/BerriAI/litellm/issues/6069

* test: fix test

* fix(types/utils.py): support all openai roles

Fixes https://github.com/BerriAI/litellm/issues/6052

* test: fix test

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
 2024-10-04 21:28:53 -04:00
Ishaan Jaff
6d1de8e1ee
(docs) router settings - on litellm config (#6037) 
* add yaml with all router settings

* add docs for router settings

* docs router settings litellm settings
 2024-10-04 18:59:01 +05:30
Ishaan Jaff
0c9c42915f update load test doc 2024-10-04 18:47:26 +05:30
Ishaan Jaff
e394ed1e5b
(fixes) docs + qa - gcs key based logging (#6061) 
* fixes for required values for gcs bucket

* docs gcs bucket logging
 2024-10-04 16:58:04 +05:30
Ishaan Jaff
2449d258cf
(docs) add 1k rps load test doc (#6059) 
* docs 1k rps load test

* docs load testing

* docs load testing litellm

* docs load testing

* clean up load test doc

* docs prom metrics for load testing

* docs using prometheus on load testing

* doc load testing with prometheus
 2024-10-04 16:56:34 +05:30
Krrish Dholakia
793593e735 docs(realtime.md): add new /v1/realtime endpoint 2024-10-03 22:44:02 -04:00
Krish Dholakia
5c33d1c9af
Litellm Minor Fixes & Improvements (10/03/2024) (#6049) 
* fix(proxy_server.py): remove spendlog fixes from proxy startup logic

Moves  https://github.com/BerriAI/litellm/pull/4794 to `/db_scripts` and cleans up some caching-related debug info (easier to trace debug logs)

* fix(langfuse_endpoints.py): Fixes https://github.com/BerriAI/litellm/issues/6041

* fix(azure.py): fix health checks for azure audio transcription models

Fixes https://github.com/BerriAI/litellm/issues/5999

* Feat: Add Literal AI Integration (#5653)

* feat: add Literal AI integration

* update readme

* Update README.md

* fix: address comments

* fix: remove literalai sdk

* fix: use HTTPHandler

* chore: add test

* fix: add asyncio lock

* fix(literal_ai.py): fix linting errors

* fix(literal_ai.py): fix linting errors

* refactor: cleanup

---------

Co-authored-by: Willy Douhard <willy.douhard@gmail.com>
 2024-10-03 18:02:28 -04:00
Ishaan Jaff
d92696a303
(feat) add nvidia nim embeddings (#6032) 
* nvidia nim support embedding config

* add nvidia config in init

* nvidia nim embeddings

* docs nvidia nim embeddings

* docs embeddings on nvidia nim

* fix llm translation test
 2024-10-03 17:12:14 +05:30
Ishaan Jaff
05df9cc6d0 docs prometheus metrics 2024-10-03 16:31:29 +05:30
Ishaan Jaff
21e05a0f3e
(feat proxy) add key based logging for GCS bucket (#6031) 
* init litellm langfuse / gcs credentials in litellm logging obj

* add gcs key based test

* rename vars

* save standard_callback_dynamic_params in model call details

* add working gcs bucket key based logging

* test_basic_gcs_logging_per_request

* linting fix

* add doc on gcs  bucket team based logging
 2024-10-03 15:24:31 +05:30
Krrish Dholakia
121b493fe8 docs(code_quality.md): add doc on litellm code qa 2024-10-02 11:20:15 -04:00
Krish Dholakia
d57be47b0f
Litellm ruff linting enforcement (#5992) 
* ci(config.yml): add a 'check_code_quality' step

Addresses https://github.com/BerriAI/litellm/issues/5991

* ci(config.yml): check why circle ci doesn't pick up this test

* ci(config.yml): fix to run 'check_code_quality' tests

* fix(__init__.py): fix unprotected import

* fix(__init__.py): don't remove unused imports

* build(ruff.toml): update ruff.toml to ignore unused imports

* fix: fix: ruff + pyright - fix linting + type-checking errors

* fix: fix linting errors

* fix(lago.py): fix module init error

* fix: fix linting errors

* ci(config.yml): cd into correct dir for checks

* fix(proxy_server.py): fix linting error

* fix(utils.py): fix bare except

causes ruff linting errors

* fix: ruff - fix remaining linting errors

* fix(clickhouse.py): use standard logging object

* fix(__init__.py): fix unprotected import

* fix: ruff - fix linting errors

* fix: fix linting errors

* ci(config.yml): cleanup code qa step (formatting handled in local_testing)

* fix(_health_endpoints.py): fix ruff linting errors

* ci(config.yml): just use ruff in check_code_quality pipeline for now

* build(custom_guardrail.py): include missing file

* style(embedding_handler.py): fix ruff check
 2024-10-01 19:44:20 -04:00
Krrish Dholakia
18a28ef977 docs(data_security.md): cleanup docs 2024-10-01 15:33:10 -04:00
Krrish Dholakia
e8a291b539 docs(data_security.md): update faq doc 2024-10-01 14:38:34 -04:00
Ishaan Jaff
045ecf3ffb
(feat proxy slack alerting) - allow opting in to getting key / internal user alerts (#5990) 
* define all slack alert types

* use correct type hints for alert type

* use correct defaults on slack alerting

* add readme for slack alerting

* fix linting error

* update readme

* docs all alert types

* update slack alerting docs

* fix slack alerting docs

* handle new testing dir structure

* fix config for testing

* fix testing folder related imports

* fix /tests import errors

* fix import stream_chunk_testdata

* docs alert types

* fix test test_langfuse_trace_id

* fix type checks for slack alerting

* fix outage alerting test slack
 2024-10-01 10:49:22 -07:00
Ishaan Jaff
2a7e1e970d
(docs) prometheus metrics document all prometheus metrics (#5989) 
* fix doc on prometheus

* (docs) clean up prometheus docs

* docs show what metrics are deprectaed

* doc clarify labels used for bduget metrics

* add litellm_remaining_api_key_requests_for_model
 2024-09-30 16:38:38 -07:00