feat: added oci-s3 compatibility (#4374)
Some checks failed
SqlStore Integration Tests / test-postgres (3.13) (push) Failing after 0s
Integration Auth Tests / test-matrix (oauth2_token) (push) Failing after 1s
Test External Providers Installed via Module / test-external-providers-from-module (venv) (push) Has been skipped
Integration Tests (Replay) / generate-matrix (push) Successful in 3s
SqlStore Integration Tests / test-postgres (3.12) (push) Failing after 5s
API Conformance Tests / check-schema-compatibility (push) Successful in 14s
Python Package Build Test / build (3.12) (push) Successful in 16s
Python Package Build Test / build (3.13) (push) Successful in 17s
Test External API and Providers / test-external (venv) (push) Failing after 30s
Vector IO Integration Tests / test-matrix (push) Failing after 50s
UI Tests / ui-tests (22) (push) Successful in 1m1s
Unit Tests / unit-tests (3.12) (push) Failing after 1m39s
Unit Tests / unit-tests (3.13) (push) Failing after 1m43s
Integration Tests (Replay) / Integration Tests (, , , client=, ) (push) Failing after 2m47s
Pre-commit / pre-commit (22) (push) Successful in 3m42s

# What does this PR do?
The PR validates and allow access to OCI object-storage through the S3
compatibility API. Additional documentation for OCI is supplied, in
notebook form, as well.

## Test Plan
<!-- Describe the tests you ran to verify your changes with result
summaries. *Provide clear instructions so the plan can be easily
re-executed.* -->

---------

Co-authored-by: raghotham <rsm@meta.com>
This commit is contained in:
Robert Riley (OCI) 2025-12-11 17:13:55 -06:00 committed by GitHub
parent 805abf573f
commit 10c878d782
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
3 changed files with 1107 additions and 0 deletions

View file

@ -210,6 +210,35 @@ config:
aws_secret_access_key: minioadmin
```
### Using OCI Object Storage with S3 Compatibility
[Official Object Storage Amazon S3 Compatibility API Documentation](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm)
OCI Object Storage can be utilized through the OCI S3 Compatibility API. Simply Update the `config.yaml` and set the env-vars as below.
#### config.yaml
```yaml
provider_type: remote::s3
config:
bucket_name: "${env.S3_BUCKET_NAME}"
region: "${env.AWS_REGION:=us-east-1}"
aws_access_key_id: "${env.AWS_ACCESS_KEY_ID:=}"
aws_secret_access_key: "${env.AWS_SECRET_ACCESS_KEY:=}"
endpoint_url: "${env.S3_ENDPOINT_URL:=}"
metadata_store:
table_name: files_metadata
backend: sql_default
```
#### .env
```
AWS_ACCESS_KEY_ID=OCI_ACCESS_KEY
AWS_SECRET_ACCESS_KEY=OCI_SECRET_KEY
S3_BUCKET_NAME=OCI_BUCKET_NAME
S3_ENDPOINT_URL=https://<namespace>.compat.objectstorage.<region>.oci.customer-oci.com
AWS_REQUEST_CHECKSUM_CALCULATION=when_required
AWS_RESPONSE_CHECKSUM_VALIDATION=when_required
```
## Monitoring and Logging
The provider logs important operations and errors. For production deployments, consider: