boto3 object storage: custom S3 endpoint setup and a read-only test

1. Set up a separate credential profile

Install boto3 in your Python environment. The AWS CLI can create a named profile for the storage access key and secret. Enter them at its prompts; do not put secrets in source code or a command you will save in shell history.

Use credentials issued by the S3-compatible provider, not your AWS account keys. The example defaults to the profile named dotmoose. On a server, a protected credential file or your deployment secret manager must make that profile available to the application.

python -m pip install boto3
aws configure --profile dotmoose
aws configure set s3.addressing_style path --profile dotmoose

Boto3 credential configuration

2. Set the endpoint, region and bucket

Replace the example values with the details from your service. The endpoint is the HTTPS base address; do not append the bucket name. The bucket is a separate argument in the request.

These shell commands are for macOS or Linux. On Windows, configure equivalent environment variables in PowerShell or your application settings. The Python code is the same.

# Replace these with the values supplied by your storage provider.
export S3_ENDPOINT_URL='https://storage.example.com'
export S3_REGION='provider-region'
export S3_BUCKET='your-assigned-bucket'

3. Run this read-only Python check

Save this as s3_check.py and run python s3_check.py. It requires all three settings, uses Signature Version 4, keeps TLS certificate validation enabled and requests at most five objects from the assigned bucket.

Path-style addressing places the bucket in the request path. Use the addressing style supported by your provider; a service requiring virtual-hosted addressing also needs matching bucket-hostname DNS and certificates. This example uses path style.

import os
from urllib.parse import urlsplit

import boto3
from botocore.config import Config
from botocore.exceptions import BotoCoreError, ClientError


def list_sample(env=os.environ):
    required = ("S3_ENDPOINT_URL", "S3_REGION", "S3_BUCKET")
    missing = [name for name in required if not env.get(name, "").strip()]
    if missing:
        raise ValueError("Set " + ", ".join(missing))

    endpoint = env["S3_ENDPOINT_URL"].strip()
    url = urlsplit(endpoint)
    if (url.scheme != "https" or not url.hostname
            or url.username or url.password or url.query or url.fragment
            or url.path not in ("", "/")):
        raise ValueError("Use the provider's HTTPS endpoint, without a bucket or credentials in the URL")

    # A named profile keeps these keys separate from your default AWS account.
    session = boto3.Session(profile_name=env.get("S3_PROFILE", "dotmoose"))
    client = session.client(
        "s3",
        endpoint_url=endpoint,
        region_name=env["S3_REGION"].strip(),
        config=Config(
            signature_version="s3v4",
            s3={"addressing_style": "path"},
            connect_timeout=10,
            read_timeout=30,
            retries={"mode": "standard", "total_max_attempts": 3},
        ),
    )
    try:
        result = client.list_objects_v2(
            Bucket=env["S3_BUCKET"].strip(), MaxKeys=5
        )
        return len(result.get("Contents", []))
    finally:
        client.close()


if __name__ == "__main__":
    try:
        print(f"Listed {list_sample()} objects (at most 5); no files changed.")
    except ValueError as error:
        raise SystemExit(str(error)) from None
    except ClientError as error:
        # Do not dump credentials, signed requests or object names into logs.
        code = error.response.get("Error", {}).get("Code", "Unknown")
        raise SystemExit(f"S3 request failed: {code}") from None
    except BotoCoreError:
        raise SystemExit("Check the named profile, network, endpoint and TLS trust.") from None

Boto3 client endpoint and TLS options | Botocore addressing, timeout and retry settings

4. Interpret the result

A successful empty listing is valid for an empty bucket. This check tests listing access only; it does not prove upload, download, versioning, multipart or presigned-URL compatibility.

AccessDenied can mean the key is valid but lacks permission to list that bucket. NoSuchBucket usually means the bucket name or endpoint is wrong. For SignatureDoesNotMatch, check the key, signing region, endpoint, system clock and any proxy changing requests.

For a TLS error, fix the hostname or certificate trust. Do not make verify=False the production workaround. For a profile error, check that the application user can read the named profile.

5. Test the operations your application will use

Before migrating real data, use a separate test prefix for one small upload and download, compare the returned bytes, and then test larger multipart files when your application uses them. Clean up only the test objects you created.

DotMoose provides the endpoint, region, bucket and access keys with an Object Storage service. Compare the plans for storage capacity, or check the exact operations your software requires before buying. No AWS-only feature is implied by S3 compatibility.

Compare Canadian Object Storage plans | Use AWS CLI with the same endpoint | How S3 presigned URLs work