Krish Dholakia
2b9db05e08
feat(proxy_cli.py): add new 'log_config' cli param ( #6352 )
...
* feat(proxy_cli.py): add new 'log_config' cli param
Allows passing logging.conf to uvicorn on startup
* docs(cli.md): add logging conf to uvicorn cli docs
* fix(get_llm_provider_logic.py): fix default api base for litellm_proxy
Fixes https://github.com/BerriAI/litellm/issues/6332
* feat(openai_like/embedding): Add support for jina ai embeddings
Closes https://github.com/BerriAI/litellm/issues/6337
* docs(deploy.md): update entrypoint.sh filepath post-refactor
Fixes outdated docs
* feat(prometheus.py): emit time_to_first_token metric on prometheus
Closes https://github.com/BerriAI/litellm/issues/6334
* fix(prometheus.py): only emit time to first token metric if stream is True
enables more accurate ttft usage
* test: handle vertex api instability
* fix(get_llm_provider_logic.py): fix import
* fix(openai.py): fix deepinfra default api base
* fix(anthropic/transformation.py): remove anthropic beta header (#6361 )
2024-10-21 21:25:58 -07:00
Ishaan Jaff
610974b4fc
(code quality) add ruff check PLR0915 for too-many-statements ( #6309 )
...
* ruff add PLR0915
* add noqa for PLR0915
* fix noqa
* add # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* add # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
* # noqa: PLR0915
2024-10-18 15:36:49 +05:30
Ishaan Jaff
603299e3c8
(feat) prometheus have well defined latency buckets ( #6211 )
...
* fix prometheus have well defined latency buckets
* use a well define latency bucket
* use types file for prometheus logging
* add test for LATENCY_BUCKETS
2024-10-14 17:16:01 +05:30
Krish Dholakia
d57be47b0f
Litellm ruff linting enforcement ( #5992 )
...
* ci(config.yml): add a 'check_code_quality' step
Addresses https://github.com/BerriAI/litellm/issues/5991
* ci(config.yml): check why circle ci doesn't pick up this test
* ci(config.yml): fix to run 'check_code_quality' tests
* fix(__init__.py): fix unprotected import
* fix(__init__.py): don't remove unused imports
* build(ruff.toml): update ruff.toml to ignore unused imports
* fix: fix: ruff + pyright - fix linting + type-checking errors
* fix: fix linting errors
* fix(lago.py): fix module init error
* fix: fix linting errors
* ci(config.yml): cd into correct dir for checks
* fix(proxy_server.py): fix linting error
* fix(utils.py): fix bare except
causes ruff linting errors
* fix: ruff - fix remaining linting errors
* fix(clickhouse.py): use standard logging object
* fix(__init__.py): fix unprotected import
* fix: ruff - fix linting errors
* fix: fix linting errors
* ci(config.yml): cleanup code qa step (formatting handled in local_testing)
* fix(_health_endpoints.py): fix ruff linting errors
* ci(config.yml): just use ruff in check_code_quality pipeline for now
* build(custom_guardrail.py): include missing file
* style(embedding_handler.py): fix ruff check
2024-10-01 19:44:20 -04:00
Ishaan Jaff
7e69cdd1b4
(proxy prometheus) track api key and team in latency metrics ( #5966 )
...
* track api key and team in prom latency metric
* add test for latency metric
* test prometheus success metrics for latency
2024-09-28 19:04:42 -07:00
Ishaan Jaff
49ec40b1cb
(feat proxy prometheus) track virtual key, key alias, error code, error code class on prometheus ( #5968 )
...
* track api key and team in prom latency metric
* add test for latency metric
* test prometheus success metrics for latency
* track team and key labels for deployment failures
* add test for litellm_deployment_failure_responses_total
* fix checks for premium user on prometheus
* log_success_fallback_event and log_failure_fallback_event
* log original_exception in log_success_fallback_event
* track key, team and exception status and class on fallback metrics
* use get_standard_logging_metadata
* fix import error
* track litellm_deployment_successful_fallbacks
* add test test_proxy_fallback_metrics
* add log log_success_fallback_event
* fix test prometheus
2024-09-28 19:00:21 -07:00
Ishaan Jaff
8b6eec1951
track api key and alias in remaining tokens metric ( #5924 )
2024-09-26 18:01:03 -07:00
Ishaan Jaff
37be0088ff
fix prometheus track input and output tokens ( #5780 )
2024-09-23 08:19:22 -07:00
Ishaan Jaff
711932294c
[Feat] Add testing for prometheus failure metrics ( #5823 )
...
* prom - show status code and class type on prom
* log exception_class name on prometheus metrics
* prometheus track error code and status
* add bad model
* add prometheus failure metric test
* remove outdated file
* fix litellm_proxy_total_requests_metric
* add prometheus metrics testing
2024-09-21 11:36:29 -07:00
Ishaan Jaff
91e58d9049
[Feat] Add proxy level prometheus metrics ( #5789 )
...
* add Proxy Level Tracking Metrics doc
* update service logger
* prometheus - track litellm_proxy_failed_requests_metric
* use REQUESTED_MODEL
* fix prom request_data
2024-09-19 17:13:07 -07:00
Ishaan Jaff
50cc7c0353
[Chore LiteLLM Proxy] enforce prometheus metrics as enterprise feature ( #5769 )
...
* enforce prometheus as enterprise feature
* show correct error on prometheus metric when not enrterprise user
* docs promethues metrics enforced
* fix enforcing
2024-09-18 16:28:12 -07:00
Ishaan Jaff
a109853d21
[Prometheus] track requested model ( #5774 )
...
* enforce prometheus as enterprise feature
* show correct error on prometheus metric when not enrterprise user
* docs promethues metrics enforced
* track requested model on prometheus
* docs prom metrics
* fix prom tracking failures
2024-09-18 12:46:58 -07:00
Ishaan Jaff
c8eff2dc65
[Feat-Prometheus] Track exception status on litellm_deployment_failure_responses ( #5706 )
...
* add litellm_deployment_cooled_down
* track num cooldowns on prometheus
* track exception status
* fix linting
* docs prom metrics
* cleanup premium user checks
* prom track deployment failure state
* docs prometheus
2024-09-14 18:44:31 -07:00
Ishaan Jaff
6ab601432b
feat prometheus add metric for failure / model
2024-08-31 10:05:23 -07:00
Ishaan Jaff
443e1b3bba
prometheus - safe update start / end time
2024-08-28 16:13:56 -07:00
Ishaan Jaff
4ac78a0765
fix prom latency metrics
2024-08-23 06:59:19 -07:00
Ishaan Jaff
36b550b8db
update promtheus metric names
2024-08-22 14:03:00 -07:00
Ishaan Jaff
06a362d35f
track litellm_request_latency_metric
2024-08-22 13:58:10 -07:00
Ishaan Jaff
65c0626aa4
fix init correct prometheus metrics
2024-08-22 13:29:35 -07:00
Ishaan Jaff
feb8c3c5b4
Merge pull request #5259 from BerriAI/litellm_return_remaining_tokens_in_header
...
[Feat] return `x-litellm-key-remaining-requests-{model}`: 1, `x-litellm-key-remaining-tokens-{model}: None` in response headers
2024-08-17 12:41:16 -07:00
Ishaan Jaff
5985c7e933
feat - use commong helper for getting model group
2024-08-17 10:46:04 -07:00
Ishaan Jaff
d630f77b73
show correct metric
2024-08-17 10:12:23 -07:00
Ishaan Jaff
412d30d362
add litellm-key-remaining-tokens on prometheus
2024-08-17 10:02:20 -07:00
Krrish Dholakia
61f4b71ef7
refactor: replace .error() with .exception() logging for better debugging on sentry
2024-08-16 09:22:47 -07:00
Ishaan Jaff
acadabe6c9
use litellm_ prefix for new deployment metrics
2024-08-14 09:08:14 -07:00
Ishaan Jaff
cc3316104f
doc new prometheus metrics
2024-08-10 17:13:36 -07:00
Ishaan Jaff
d1eedb04bd
prometheus log_success_fallback_event
2024-08-10 14:05:18 -07:00
Ishaan Jaff
01f5456e3d
feat - track latency per llm deployment
2024-08-10 12:53:56 -07:00
Ishaan Jaff
83446926af
track llm_deployment_success_responses
2024-08-10 10:05:33 -07:00
Ishaan Jaff
05fba48cec
feat - refactor prometheus metrics
2024-08-10 09:14:38 -07:00
Ishaan Jaff
0b9ad45790
refactor prom metrics
2024-08-09 09:02:23 -07:00
Ishaan Jaff
958e0fdfab
show warning about prometheus moving to enterprise
2024-08-07 12:46:26 -07:00
Ishaan Jaff
579f09b334
fix logging cool down deployment
2024-08-07 11:27:05 -07:00
Ishaan Jaff
99ecde7744
allow setting outage metrics
2024-08-07 10:36:18 -07:00
Ishaan Jaff
9a8fa9f047
emit deployment_partial_outage on prometheus
2024-08-07 09:56:01 -07:00
Ishaan Jaff
0304513b67
rename to set_llm_deployment_success_metrics
2024-08-07 09:46:08 -07:00
Ishaan Jaff
13d1caf94a
add set_remaining_tokens_requests_metric
2024-08-07 09:43:35 -07:00
Ishaan Jaff
85ecc13458
feat - show key alias on prometheus metrics
2024-07-04 09:57:00 -07:00
Ishaan Jaff
c37d45d556
feat - prometheus log remaining headers
2024-07-01 20:00:47 -07:00
Ishaan Jaff
9b340fb2f8
feat - add remaining budget for key on prometheus
2024-06-13 14:37:02 -07:00
Ishaan Jaff
51157c148f
feat - add remaining team budget gauge
2024-06-13 14:28:25 -07:00
Bram van Meurs
270dad93d5
feat(prometheus): add api_team_alias to exported labels
2024-06-13 12:50:40 +02:00
Krrish Dholakia
6cca5612d2
refactor: replace 'traceback.print_exc()' with logging library
...
allows error logs to be in json format for otel logging
2024-06-06 13:47:43 -07:00
Krrish Dholakia
4a3b084961
feat(bedrock_httpx.py): moves to using httpx client for bedrock cohere calls
2024-05-11 13:43:08 -07:00
Krrish Dholakia
6575143460
feat(proxy_server.py): return litellm version in response headers
2024-05-08 16:00:08 -07:00
Krrish Dholakia
52d5e02d8b
fix(prometheus.py): fix user-id get for prometheus
2024-04-24 08:08:42 -07:00
Krrish Dholakia
6ac0dba5c2
fix(prometheus.py): add user tracking to prometheus
2024-04-22 15:14:38 -07:00
Krrish Dholakia
9379e3d047
fix(lowest_tpm_rpm_v2.py): use a combined tpm+rpm query in async get cache, to reduce redis client calls in high traffic
2024-04-20 16:13:11 -07:00
Krrish Dholakia
08cf77623b
fix(caching.py): remove url parsing logic - causing redis ssl connections to fail
...
this reverts a change that was causing redis url w/ ssl to fail. this also adds unit testing for this sc
enario, to prevent future regressions
2024-04-19 14:01:13 -07:00
Krrish Dholakia
d8d78bfb4f
fix(prometheus.py): fix metric name to be more accurate
...
change metric name from litellm_failed_requests_metric to litellm_llm_api_failed_requests_metric
2024-04-18 12:30:44 -07:00
