Skip to main content

TSZ AI Provider Guide

This document provides comprehensive information about AI providers supported by TSZ (Thyris Safe Zone), including configuration, usage, and best practices.

Table of Contents​

  1. Overview
  2. Provider Architecture
  3. Supported Providers
  4. Configuration
  5. Provider Comparison
  6. Migration Guide
  7. Troubleshooting
  8. Best Practices

Overview​

TSZ supports multiple AI providers through a unified abstraction layer. This allows you to:

  • Switch providers without changing application code
  • Use different providers for different environments (dev, staging, production)
  • Leverage provider-specific features while maintaining compatibility
  • Optimize for cost, latency, or compliance requirements

Supported Provider Types​

  1. chat-completions-compatible - Works with any chat-completions-compatible API
  2. managed model service - Native integration with managed model service service

Provider Architecture​

Abstraction Layer​

TSZ uses a ChatProvider interface defined in internal/ai/provider.go:

type ChatProvider interface {
    Name() string
    Chat(ctx context.Context, req ChatRequest) (*ChatResponse, error)
    ChatStream(ctx context.Context, req ChatRequest) (<-chan StreamEvent, <-chan error)
    SupportsStreaming() bool
}

Key Components​

  1. Provider Factory - Initializes the appropriate provider based on configuration
  2. Request/Response Translation - Converts between chat-completions provider format and provider-specific formats
  3. Error Handling - Unified error handling across providers
  4. Credential Management - Provider-specific authentication mechanisms

Data Flow​

Supported Providers​

1. chat-completions-compatible Provider​

Provider ID: CHAT_COMPLETIONS_COMPATIBLE

Description​

The chat-completions-compatible provider works with any service that implements the chat-completions provider Chat Completions API. This includes:

  • chat-completions provider - Official chat-completions provider API
  • managed chat provider - software provider's managed chat provider Service
  • local model runtime - Local LLM runtime
  • LM Studio - Local LLM development tool
  • vLLM - High-throughput LLM serving
  • Text Generation Inference - model hosting provider's inference server
  • Any custom chat-completions-compatible endpoint

Configuration​

AI_PROVIDER=CHAT_COMPLETIONS_COMPATIBLE
AI_MODEL_URL=https://api.model-provider.example/v1
AI_API_KEY=sk-your-api-key-here
AI_MODEL=gpt-4

Configuration Parameters​

ParameterRequiredDescriptionExample
AI_MODEL_URLYesBase URL of the API endpointhttps://api.model-provider.example/v1
AI_API_KEYYes*API key for authenticationsk-...
AI_MODELYesDefault model namegpt-4

*Not required for services like local model runtime that don't use authentication

Examples​

chat-completions provider:

AI_PROVIDER=CHAT_COMPLETIONS_COMPATIBLE
AI_MODEL_URL=https://api.model-provider.example/v1
AI_API_KEY=sk-proj-...
AI_MODEL=gpt-4

managed chat provider:

AI_PROVIDER=CHAT_COMPLETIONS_COMPATIBLE
AI_MODEL_URL=https://api.managed-model.example/v1
AI_API_KEY=your-cloud provider-key
AI_MODEL=gpt-4

local model runtime (Local):

AI_PROVIDER=CHAT_COMPLETIONS_COMPATIBLE
AI_MODEL_URL=http://localhost:11434/v1
AI_API_KEY=local model runtime
AI_MODEL=llama3.1:8b

local model runtime (Docker):

AI_PROVIDER=CHAT_COMPLETIONS_COMPATIBLE
AI_MODEL_URL=https://reference.example/resource
AI_API_KEY=local model runtime
AI_MODEL=llama3.1:8b

Features​

  • Yes Non-streaming requests
  • Yes Streaming requests (SSE)
  • Yes Custom headers
  • Yes Timeout configuration
  • Yes Automatic retries (via HTTP client)

Limitations​

  • Authentication is limited to Bearer token
  • No built-in support for cloud provider SigV4 signing
  • Requires network connectivity to the endpoint

2. managed model service Provider​

Provider ID: MANAGED_MODEL

Description​

The managed model service provider offers native integration with managed model service, cloud provider's fully managed service for foundation models. This provider uses the cloud provider SDK for Go v2 and supports all managed model service model families.

Configuration​

AI_PROVIDER=MANAGED_MODEL
AWS_BEDROCK_REGION=us-east-1
AWS_BEDROCK_MODEL_ID=model provider.managed model-3-sonnet-20240229-v1:0
AWS_BEDROCK_ENDPOINT_OVERRIDE=  # Optional

Configuration Parameters​

ParameterRequiredDescriptionExample
AWS_BEDROCK_REGIONYescloud provider region where managed model service is availableus-east-1
AWS_BEDROCK_MODEL_IDYesmanaged model service model identifiermodel provider.managed model-3-sonnet-20240229-v1:0
AWS_BEDROCK_ENDPOINT_OVERRIDENoCustom endpoint URL (for VPC endpoints)https://reference.example/resource model service-runtime.us-east-1.vpce.amazonaws.com

cloud provider Credentials​

managed model service uses the standard cloud provider credential chain:

  1. Environment Variables:

    AWS_ACCESS_KEY_ID=<cloud_provider-access-key-id>
    AWS_SECRET_ACCESS_KEY=<cloud_provider-secret-access-key>
    AWS_SESSION_TOKEN=...  # Optional, for temporary credentials

  2. Shared Credentials File:

    ~/.cloud_provider/credentials

  3. IAM Role:

    • Automatically used when running on EC2, ECS, Lambda, etc.

Required IAM Permissions​

TSZ needs IAM permissions to invoke managed model service models. You can configure this in several ways:

Option 1: IAM User with Access Keys (Development/Testing)​
  1. Create IAM Policy in cloud provider Console:
    • Go to IAM -> Policies -> Create Policy
    • Select JSON tab and paste:
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "managed model service:InvokeModel",
                "managed model service:InvokeModelWithResponseStream"
            ],
            "Resource": [
                "arn:cloud_provider:managed model service:*::foundation-model/model provider.managed model-*",
                "arn:cloud_provider:managed model service:*::foundation-model/cloud provider.titan-*",
                "arn:cloud_provider:managed model service:*::foundation-model/communication provider.open model*",
                "arn:cloud_provider:managed model service:*::foundation-model/model provider.*",
                "arn:cloud_provider:managed model service:*::foundation-model/model provider.*",
                "arn:cloud_provider:managed model service:*::foundation-model/chat-completions provider.*"
            ]
        }
    ]
}

  1. Name the policy: TSZ-managed model service-Access

  2. Create IAM User:

    • Go to IAM -> Users -> Create User
    • Name: tsz-managed model service-user
    • Attach the policy: TSZ-managed model service-Access

  3. Create Access Keys:

    • Select the user -> Security Credentials -> Create Access Key
    • Choose "Application running outside cloud provider"
    • Save the Access Key ID and Secret Access Key

  4. Configure TSZ:

    AWS_ACCESS_KEY_ID=<cloud_provider-access-key-id>
    AWS_SECRET_ACCESS_KEY=<cloud_provider-secret-access-key>
    AI_PROVIDER=MANAGED_MODEL
    AWS_BEDROCK_REGION=us-east-1
    AWS_BEDROCK_MODEL_ID=model provider.managed model-3-sonnet-20240229-v1:0
Option 2: IAM Role (Production - Recommended)​

For production deployments on cloud provider (EC2, ECS, EKS, Lambda):

  1. Create IAM Role:

    • Go to IAM -> Roles -> Create Role
    • Select trusted entity: EC2, ECS Task, or EKS
    • Attach the policy: TSZ-managed model service-Access
    • Name: TSZ-managed model service-Role

  2. Attach Role to Service:

    • EC2: Attach role to EC2 instance
    • ECS: Specify role in task definition
    • EKS: Use IRSA (IAM Roles for Service Accounts)

  3. Configure TSZ (no credentials needed):

    # No AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY needed
    # Role credentials are automatically loaded
    AI_PROVIDER=MANAGED_MODEL
    AWS_BEDROCK_REGION=us-east-1
    AWS_BEDROCK_MODEL_ID=model provider.managed model-3-sonnet-20240229-v1:0
Option 3: cloud provider CLI Profile (Local Development)​

  1. Configure cloud provider CLI:

    cloud_provider configure --profile tsz-managed model service
    # Enter your Access Key ID
    # Enter your Secret Access Key
    # Enter default region (e.g., us-east-1)

  2. Use Profile:

    AWS_PROFILE=tsz-managed model service docker compose -f deployment/docker/docker-compose.yml up -d
Minimum IAM Policy (Specific Model)​

For tighter security, restrict to specific models:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "managed model service:InvokeModel"
            ],
            "Resource": [
                "arn:cloud_provider:managed model service:us-east-1::foundation-model/model provider.managed model-3-sonnet-20240229-v1:0"
            ]
        }
    ]
}

Supported Model Families​

TSZ's managed model service provider supports the following model families:

  • managed model family - managed model 3 Opus, Sonnet, Haiku
  • managed model family - Titan Text Express, Lite
  • open model family - open model 3 (8B, 70B)
  • model provider - model provider 7B, Mixtral 8x7B
  • model provider - Command, Command Light
  • chat-completions provider - GPT-OSS 20B, GPT-OSS 120B (via managed model service)

For the latest model IDs and availability, refer to the managed model service Model IDs documentation.

Features​

  • Yes Non-streaming requests
  • Yes Multiple model families
  • Yes cloud provider IAM authentication
  • Yes VPC endpoint support
  • Yes cloud provider KMS encryption
  • Yes CloudTrail audit logging
  • Planned Streaming requests (planned for future release)

Limitations​

  • Streaming is not yet supported (non-streaming only). If a client sends stream=true while AI_PROVIDER=MANAGED_MODEL, TSZ returns an chat-completions-compatible 400 error with code streaming_not_supported.
  • Requires cloud provider credentials and permissions
  • Model availability varies by region
  • Some models require explicit enablement in cloud provider console

Regional Availability​

managed model service is available in the following regions:

  • us-east-1 (N. Virginia)
  • us-west-2 (Oregon)
  • ap-southeast-1 (Singapore)
  • ap-northeast-1 (Region A)
  • eu-central-1 (Frankfurt)
  • eu-west-1 (Ireland)
  • eu-west-3 (Example City)

Check managed model service documentation for the latest regional availability.

Security Features​

  1. Data Residency: All data stays within cloud provider boundaries
  2. Encryption:
    • In-transit: TLS 1.2+
    • At-rest: cloud provider KMS encryption
  3. Access Control: IAM policies and roles
  4. Audit: CloudTrail logging for all API calls
  5. Network Isolation: VPC endpoints for private connectivity

Configuration​

Environment-Based Configuration​

TSZ uses environment variables for provider configuration. This allows for:

  • Easy configuration management across environments
  • Secure credential handling
  • No code changes required to switch providers

Configuration Precedence​

  1. Environment variables
  2. .env file (if present)
  3. Default values (defined in code)

Validation​

TSZ validates configuration at startup:

  • Required parameters are checked
  • Credentials are validated (where possible)
  • Provider initialization is tested

If configuration is invalid, TSZ will:

  • Log detailed error messages
  • Fail fast (exit with error code)
  • Provide guidance on fixing the issue

Provider Comparison​

Feature Matrix​

Featurechat-completions-compatiblemanaged model service
Non-streamingYesYes
StreamingYesPlanned Planned
Multiple modelsYesYes
Custom endpointsYesYes (VPC)
AuthenticationBearer tokencloud provider IAM
EncryptionTLSTLS + KMS
Audit loggingApplication logsCloudTrail
CostVaries by providercloud provider pricing
LatencyDepends on endpointcloud provider network
Data residencyDepends on providercloud provider regions

Use Case Recommendations​

Use chat-completions-compatible When:​

  • You need maximum flexibility in provider choice
  • You're using local models (local model runtime, LM Studio)
  • You have existing chat-completions provider integrations
  • You need streaming support immediately
  • You're in development/testing phase

Use managed model service When:​

  • You need to keep data within cloud provider boundaries
  • You require cloud provider compliance certifications
  • You want to leverage cloud provider IAM for access control
  • You need VPC endpoint connectivity
  • You're already using cloud provider services
  • You need CloudTrail audit logging

Migration Guide​

From chat-completions-compatible to managed model service​

  1. Update Environment Variables:

    # Before
    AI_PROVIDER=CHAT_COMPLETIONS_COMPATIBLE
    AI_MODEL_URL=https://api.model-provider.example/v1
    AI_API_KEY=sk-...
    AI_MODEL=gpt-4

    # After
    AI_PROVIDER=MANAGED_MODEL
    AWS_BEDROCK_REGION=us-east-1
    AWS_BEDROCK_MODEL_ID=model provider.managed model-3-sonnet-20240229-v1:0

  2. Configure cloud provider Credentials:

    cloud_provider configure
    # or set environment variables
    export AWS_ACCESS_KEY_ID=...
    export AWS_SECRET_ACCESS_KEY=...

  3. Verify IAM Permissions:

    • Ensure your IAM user/role has managed model service:InvokeModel permission

  4. Test Configuration:

    # Restart TSZ
    docker-compose restart api

    # Check logs
    docker logs thyris_api

  5. Update Application Code (if needed):

    • No code changes required for gateway usage
    • Update model names in requests if needed

From managed model service to chat-completions-compatible​

  1. Update Environment Variables:

    # Before
    AI_PROVIDER=MANAGED_MODEL
    AWS_BEDROCK_REGION=us-east-1
    AWS_BEDROCK_MODEL_ID=model provider.managed model-3-sonnet-20240229-v1:0

    # After
    AI_PROVIDER=CHAT_COMPLETIONS_COMPATIBLE
    AI_MODEL_URL=https://api.model-provider.example/v1
    AI_API_KEY=sk-...
    AI_MODEL=gpt-4

  2. Remove cloud provider Credentials (optional):

    • If not using cloud provider for other services

  3. Test Configuration:

    docker-compose restart api
    docker logs thyris_api

Troubleshooting​

Common Issues​

1. "Failed to initialize AI provider"​

Symptoms:

2025/12/18 01:26:27 Warning: Failed to initialize AI provider: managed model service region is required

Solutions:

  • Check that all required environment variables are set
  • Verify environment variable names are correct
  • Ensure .env file is in the correct location

2. "Access Denied" (managed model service)​

Symptoms:

managed model service invoke failed: AccessDeniedException: User is not authorized to perform: managed model service:InvokeModel

Solutions:

  • Verify IAM permissions include managed model service:InvokeModel
  • Check that the model ARN in the policy matches the model you're using
  • Ensure cloud provider credentials are correctly configured

3. "Model not found" (managed model service)​

Symptoms:

managed model service invoke failed: ResourceNotFoundException: Could not find model

Solutions:

  • Verify the model ID is correct
  • Check that the model is available in your region
  • Ensure the model is enabled in managed model service console

4. "Connection refused" (chat-completions-compatible)​

Symptoms:

request failed: dial tcp 127.0.0.1:11434: connect: connection refused

Solutions:

  • Verify the endpoint URL is correct
  • Check that the service is running
  • For Docker: use host.docker.internal instead of localhost

Debug Mode​

Enable debug logging:

LOG_LEVEL=debug

This will provide detailed information about:

  • Provider initialization
  • Request/response payloads
  • Error details

Best Practices​

1. Environment-Specific Configuration​

Use different providers for different environments:

# Development
AI_PROVIDER=CHAT_COMPLETIONS_COMPATIBLE
AI_MODEL_URL=http://localhost:11434/v1

# Staging
AI_PROVIDER=MANAGED_MODEL
AWS_BEDROCK_REGION=us-east-1

# Production
AI_PROVIDER=MANAGED_MODEL
AWS_BEDROCK_REGION=us-east-1
AWS_BEDROCK_ENDPOINT_OVERRIDE=https://reference.example/resource

2. Credential Management​

  • Never commit credentials to version control
  • Use secret managers (cloud provider Secrets Manager, secret manager)
  • Rotate credentials regularly
  • Use IAM roles when possible (for managed model service)

3. Cost Optimization​

  • Use cheaper models for development/testing
  • Implement caching to reduce API calls
  • Monitor usage with provider-specific tools
  • Consider local models (local model runtime) for development

4. Performance Optimization​

  • Choose regions close to your users
  • Use VPC endpoints for managed model service (reduces latency)
  • Implement connection pooling
  • Monitor and optimize timeout settings

5. Monitoring and Observability​

  • Log all provider interactions
  • Track error rates by provider
  • Monitor latency metrics
  • Set up alerts for failures

6. Disaster Recovery​

  • Have a fallback provider configured
  • Test provider switching regularly
  • Document the switching process
  • Monitor provider health status

Additional Resources​

Support​

For issues or questions about AI providers:

For managed model service-specific issues: