Ishaan Jaff
89506053a4
(feat) use regex pattern matching for wildcard routing ( #6150 )
...
* use pattern matching for llm deployments
* code quality fix
* fix linting
* add types to PatternMatchRouter
* docs add example config for regex patterns
2024-10-10 18:24:16 +05:30
Krrish Dholakia
60baa65e0e
docs(configs.md): add litellm config / s3 bucket object info in configs.md
2024-10-09 09:07:43 -07:00
Ishaan Jaff
b35da5014b
doc onboarding orgs
2024-10-09 19:11:36 +05:30
Ishaan Jaff
5da6863804
docs rbac
2024-10-09 16:46:26 +05:30
Ishaan Jaff
399f50d558
fix rbac doc
2024-10-09 16:44:46 +05:30
Ishaan Jaff
0e83a68a69
doc - move rbac under auth
2024-10-09 15:27:32 +05:30
Ishaan Jaff
1fd437e263
(feat proxy) [beta] add support for organization role based access controls ( #6112 )
...
* track LiteLLM_OrganizationMembership
* add add_internal_user_to_organization
* add org membership to schema
* read organization membership when reading user info in auth checks
* add check for valid organization_id
* add test for test_create_new_user_in_organization
* test test_create_new_user_in_organization
* add new ADMIN role
* add test for org admins creating teams
* add test for test_org_admin_create_user_permissions
* test_org_admin_create_user_team_wrong_org_permissions
* test_org_admin_create_user_team_wrong_org_permissions
* fix organization_role_based_access_check
* fix getting user members
* fix TeamBase
* fix types used for use role
* fix type checks
* sync prisma schema
* docs - organization admins
* fix use organization_endpoints for /organization management
* add types for org member endpoints
* fix role name for org admin
* add type for member add response
* add organization/member_add
* add error handling for adding members to an org
* add nice doc string for oranization/member_add
* fix test_create_new_user_in_organization
* linting fix
* use simple route changes
* fix types
* add organization member roles
* add org admin auth checks
* add auth checks for orgs
* test for creating teams as org admin
* simplify org id usage
* fix typo
* test test_org_admin_create_user_team_wrong_org_permissions
* fix type check issue
* code quality fix
* fix schema.prisma
2024-10-09 15:18:18 +05:30
Ishaan Jaff
d1760b1b04
(fix) clean up root repo - move entrypoint.sh and build_admin_ui to /docker ( #6110 )
...
* fix move docker files to docker folders
* move check file length
* fix docker hub deploy
* fix clean up root
* fix circle ci config
2024-10-08 11:34:43 +05:30
Krrish Dholakia
cc960da4b6
docs(azure.md): add o1 model support to config
2024-10-07 22:37:49 -07:00
Krish Dholakia
6729c9ca7f
LiteLLM Minor Fixes & Improvements (10/07/2024) ( #6101 )
...
* fix(utils.py): support dropping temperature param for azure o1 models
* fix(main.py): handle azure o1 streaming requests
o1 doesn't support streaming, fake it to ensure code works as expected
* feat(utils.py): expose `hosted_vllm/` endpoint, with tool handling for vllm
Fixes https://github.com/BerriAI/litellm/issues/6088
* refactor(internal_user_endpoints.py): cleanup unused params + update docstring
Closes https://github.com/BerriAI/litellm/issues/6100
* fix(main.py): expose custom image generation api support
Fixes https://github.com/BerriAI/litellm/issues/6097
* fix: fix linting errors
* docs(custom_llm_server.md): add docs on custom api for image gen calls
* fix(types/utils.py): handle dict type
* fix(types/utils.py): fix linting errors
2024-10-07 22:17:22 -07:00
Ishaan Jaff
ef815f3a84
(docs) add remaining litellm settings on configs.md doc ( #6108 )
...
* docs add litellm settings configs
* docs langfuse tags on config
2024-10-08 07:57:04 +05:30
Ishaan Jaff
2b370f8e9e
(docs) key based callbacks ( #6107 )
2024-10-08 07:12:01 +05:30
Pradyumna Singh Rathore
b7ba558b74
fix links due to broken list ( #6103 )
2024-10-07 15:47:29 -04:00
Ishaan Jaff
1bafbf8382
(feat proxy) add v2 maintained LiteLLM grafana dashboard ( #6098 )
...
* add new grafana dashboard litellm
* add v2 grafana dashboard
2024-10-07 18:11:20 +05:30
Ishaan Jaff
b2fbee3923
docs key logging
2024-10-06 13:49:27 +05:30
Ishaan Jaff
fd7014a326
correct use of healthy / unhealthy
2024-10-06 13:48:30 +05:30
Krish Dholakia
04e5963b65
Litellm expose disable schema update flag ( #6085 )
...
* fix: enable new 'disable_prisma_schema_update' flag
* build(config.yml): remove setup remote docker step
* ci(config.yml): give container time to start up
* ci(config.yml): update test
* build(config.yml): actually start docker
* build(config.yml): simplify grep check
* fix(prisma_client.py): support reading disable_schema_update via env vars
* ci(config.yml): add test to check if all general settings are documented
* build(test_General_settings.py): check available dir
* ci: check ../ repo path
* build: check ./
* build: fix test
2024-10-05 21:26:51 -04:00
Krish Dholakia
f2c0a31e3c
LiteLLM Minor Fixes & Improvements (10/05/2024) ( #6083 )
...
* docs(prompt_caching.md): add prompt caching cost calc example to docs
* docs(prompt_caching.md): add proxy examples to docs
* feat(utils.py): expose new helper `supports_prompt_caching()` to check if a model supports prompt caching
* docs(prompt_caching.md): add docs on checking model support for prompt caching
* build: fix invalid json
2024-10-05 18:59:11 -04:00
Ishaan Jaff
6e6d38841f
docs fix
2024-10-05 15:25:25 +05:30
Ishaan Jaff
5ee1342d37
(docs) reference router settings general settings etc ( #6078 )
2024-10-05 15:01:28 +05:30
Ishaan Jaff
d2f17cf97c
docs routing config table
2024-10-05 14:40:07 +05:30
Ishaan Jaff
530915da51
add o-1 to Azure docs
2024-10-05 14:23:54 +05:30
Ishaan Jaff
c84cfe977e
(feat) add /key/health endpoint to test key based logging ( #6073 )
...
* add /key/health endpoint
* add /key/health endpoint
* fix return from /key/health
* update doc string
* fix doc string for /key/health
* add test for /key/health
* fix linting
* docs /key/health
2024-10-05 11:56:55 +05:30
Krish Dholakia
2e5c46ef6d
LiteLLM Minor Fixes & Improvements (10/04/2024) ( #6064 )
...
* fix(litellm_logging.py): ensure cache hits are scrubbed if 'turn_off_message_logging' is enabled
* fix(sagemaker.py): fix streaming to raise error immediately
Fixes https://github.com/BerriAI/litellm/issues/6054
* (fixes) gcs bucket key based logging (#6044 )
* fixes for gcs bucket logging
* fix StandardCallbackDynamicParams
* fix - gcs logging when payload is not serializable
* add test_add_callback_via_key_litellm_pre_call_utils_gcs_bucket
* working success callbacks
* linting fixes
* fix linting error
* add type hints to functions
* fixes for dynamic success and failure logging
* fix for test_async_chat_openai_stream
* fix handle case when key based logging vars are set as os.environ/ vars
* fix prometheus track cooldown events on custom logger (#6060 )
* (docs) add 1k rps load test doc (#6059 )
* docs 1k rps load test
* docs load testing
* docs load testing litellm
* docs load testing
* clean up load test doc
* docs prom metrics for load testing
* docs using prometheus on load testing
* doc load testing with prometheus
* (fixes) docs + qa - gcs key based logging (#6061 )
* fixes for required values for gcs bucket
* docs gcs bucket logging
* bump: version 1.48.12 → 1.48.13
* ci/cd run again
* bump: version 1.48.13 → 1.48.14
* update load test doc
* (docs) router settings - on litellm config (#6037 )
* add yaml with all router settings
* add docs for router settings
* docs router settings litellm settings
* (feat) OpenAI prompt caching models to model cost map (#6063 )
* add prompt caching for latest models
* add cache_read_input_token_cost for prompt caching models
* fix(litellm_logging.py): check if param is iterable
Fixes https://github.com/BerriAI/litellm/issues/6025#issuecomment-2393929946
* fix(factory.py): support passing an 'assistant_continue_message' to prevent bedrock error
Fixes https://github.com/BerriAI/litellm/issues/6053
* fix(databricks/chat): handle streaming responses
* fix(factory.py): fix linting error
* fix(utils.py): unify anthropic + deepseek prompt caching information to openai format
Fixes https://github.com/BerriAI/litellm/issues/6069
* test: fix test
* fix(types/utils.py): support all openai roles
Fixes https://github.com/BerriAI/litellm/issues/6052
* test: fix test
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
2024-10-04 21:28:53 -04:00
Ishaan Jaff
6d1de8e1ee
(docs) router settings - on litellm config ( #6037 )
...
* add yaml with all router settings
* add docs for router settings
* docs router settings litellm settings
2024-10-04 18:59:01 +05:30
Ishaan Jaff
0c9c42915f
update load test doc
2024-10-04 18:47:26 +05:30
Ishaan Jaff
e394ed1e5b
(fixes) docs + qa - gcs key based logging ( #6061 )
...
* fixes for required values for gcs bucket
* docs gcs bucket logging
2024-10-04 16:58:04 +05:30
Ishaan Jaff
2449d258cf
(docs) add 1k rps load test doc ( #6059 )
...
* docs 1k rps load test
* docs load testing
* docs load testing litellm
* docs load testing
* clean up load test doc
* docs prom metrics for load testing
* docs using prometheus on load testing
* doc load testing with prometheus
2024-10-04 16:56:34 +05:30
Krrish Dholakia
793593e735
docs(realtime.md): add new /v1/realtime endpoint
2024-10-03 22:44:02 -04:00
Krish Dholakia
5c33d1c9af
Litellm Minor Fixes & Improvements (10/03/2024) ( #6049 )
...
* fix(proxy_server.py): remove spendlog fixes from proxy startup logic
Moves https://github.com/BerriAI/litellm/pull/4794 to `/db_scripts` and cleans up some caching-related debug info (easier to trace debug logs)
* fix(langfuse_endpoints.py): Fixes https://github.com/BerriAI/litellm/issues/6041
* fix(azure.py): fix health checks for azure audio transcription models
Fixes https://github.com/BerriAI/litellm/issues/5999
* Feat: Add Literal AI Integration (#5653 )
* feat: add Literal AI integration
* update readme
* Update README.md
* fix: address comments
* fix: remove literalai sdk
* fix: use HTTPHandler
* chore: add test
* fix: add asyncio lock
* fix(literal_ai.py): fix linting errors
* fix(literal_ai.py): fix linting errors
* refactor: cleanup
---------
Co-authored-by: Willy Douhard <willy.douhard@gmail.com>
2024-10-03 18:02:28 -04:00
Ishaan Jaff
d92696a303
(feat) add nvidia nim embeddings ( #6032 )
...
* nvidia nim support embedding config
* add nvidia config in init
* nvidia nim embeddings
* docs nvidia nim embeddings
* docs embeddings on nvidia nim
* fix llm translation test
2024-10-03 17:12:14 +05:30
Ishaan Jaff
05df9cc6d0
docs prometheus metrics
2024-10-03 16:31:29 +05:30
Ishaan Jaff
21e05a0f3e
(feat proxy) add key based logging for GCS bucket ( #6031 )
...
* init litellm langfuse / gcs credentials in litellm logging obj
* add gcs key based test
* rename vars
* save standard_callback_dynamic_params in model call details
* add working gcs bucket key based logging
* test_basic_gcs_logging_per_request
* linting fix
* add doc on gcs bucket team based logging
2024-10-03 15:24:31 +05:30
Krrish Dholakia
121b493fe8
docs(code_quality.md): add doc on litellm code qa
2024-10-02 11:20:15 -04:00
Krish Dholakia
d57be47b0f
Litellm ruff linting enforcement ( #5992 )
...
* ci(config.yml): add a 'check_code_quality' step
Addresses https://github.com/BerriAI/litellm/issues/5991
* ci(config.yml): check why circle ci doesn't pick up this test
* ci(config.yml): fix to run 'check_code_quality' tests
* fix(__init__.py): fix unprotected import
* fix(__init__.py): don't remove unused imports
* build(ruff.toml): update ruff.toml to ignore unused imports
* fix: fix: ruff + pyright - fix linting + type-checking errors
* fix: fix linting errors
* fix(lago.py): fix module init error
* fix: fix linting errors
* ci(config.yml): cd into correct dir for checks
* fix(proxy_server.py): fix linting error
* fix(utils.py): fix bare except
causes ruff linting errors
* fix: ruff - fix remaining linting errors
* fix(clickhouse.py): use standard logging object
* fix(__init__.py): fix unprotected import
* fix: ruff - fix linting errors
* fix: fix linting errors
* ci(config.yml): cleanup code qa step (formatting handled in local_testing)
* fix(_health_endpoints.py): fix ruff linting errors
* ci(config.yml): just use ruff in check_code_quality pipeline for now
* build(custom_guardrail.py): include missing file
* style(embedding_handler.py): fix ruff check
2024-10-01 19:44:20 -04:00
Krrish Dholakia
18a28ef977
docs(data_security.md): cleanup docs
2024-10-01 15:33:10 -04:00
Krrish Dholakia
e8a291b539
docs(data_security.md): update faq doc
2024-10-01 14:38:34 -04:00
Ishaan Jaff
045ecf3ffb
(feat proxy slack alerting) - allow opting in to getting key / internal user alerts ( #5990 )
...
* define all slack alert types
* use correct type hints for alert type
* use correct defaults on slack alerting
* add readme for slack alerting
* fix linting error
* update readme
* docs all alert types
* update slack alerting docs
* fix slack alerting docs
* handle new testing dir structure
* fix config for testing
* fix testing folder related imports
* fix /tests import errors
* fix import stream_chunk_testdata
* docs alert types
* fix test test_langfuse_trace_id
* fix type checks for slack alerting
* fix outage alerting test slack
2024-10-01 10:49:22 -07:00
Ishaan Jaff
2a7e1e970d
(docs) prometheus metrics document all prometheus metrics ( #5989 )
...
* fix doc on prometheus
* (docs) clean up prometheus docs
* docs show what metrics are deprectaed
* doc clarify labels used for bduget metrics
* add litellm_remaining_api_key_requests_for_model
2024-09-30 16:38:38 -07:00
Ishaan Jaff
ca9c437021
add Azure OpenAI entrata id docs ( #5985 )
2024-09-30 12:17:58 -07:00
Ishaan Jaff
30aa04b8c2
add docs on privacy policy
2024-09-30 11:53:52 -07:00
Ishaan Jaff
50d1c864f2
fix grammar on health check docs ( #5984 )
2024-09-30 09:21:42 -07:00
Krrish Dholakia
7630680690
docs(response_headers.md): add response headers to docs
2024-09-28 23:33:50 -07:00
DAOUDI Soufian
bfa9553819
Fixed minor typo in bash command to prevent overwriting .env file ( #5902 )
...
Changed '>' to '>>' in the bash command to append the environment variable to the .env file instead of overwriting it.
2024-09-28 23:12:19 -07:00
Krrish Dholakia
c9d6925a42
docs(reliability.md): add tutorial on setting wildcard models as fallbacks
2024-09-28 21:08:15 -07:00
Ishaan Jaff
b817974c8e
docs clean up langfuse.md
2024-09-28 18:59:02 -07:00
Ishaan Jaff
0d0f46a826
[Feat Proxy] Allow using hypercorn for http v2 ( #5950 )
...
* use run_hypercorn
* add docs on using hypercorn
2024-09-28 15:03:50 -07:00
Ishaan Jaff
fd87ae69b8
[Vertex Multimodal embeddings] Fixes to work with Langchain OpenAI Embedding ( #5949 )
...
* fix parallel request limiter - use one cache update call
* ci/cd run again
* run ci/cd again
* use docker username password
* fix config.yml
* fix config
* fix config
* fix config.yml
* ci/cd run again
* use correct typing for batch set cache
* fix async_set_cache_pipeline
* fix only check user id tpm / rpm limits when limits set
* fix test_openai_azure_embedding_with_oidc_and_cf
* add InstanceImage type
* fix vertex image transform
* add langchain vertex test request
* add new vertex test
* update multimodal embedding tests
* add test_vertexai_multimodal_embedding_base64image_in_input
* simplify langchain mm embedding usage
* add langchain example for multimodal embeddings on vertex
* fix linting error
2024-09-27 18:04:03 -07:00
Khanh Le
71f68ac185
docs(vertex.md): fix codestral fim placement ( #5946 )
2024-09-27 17:21:34 -07:00
Ishaan Jaff
bbf4db79c1
docs - show correct rpm - > tpm conversion for Azure
2024-09-27 17:18:55 -07:00
