Standard Operating Procedures

A guide to the standard operating procedures for Acacia Analytics.

FastAPI, MongoDB, and GCP Services

1. Project Structure & Organization

Mandatory Directory Layout

project_name/
├── app/
│   ├── __init__.py
│   ├── main.py              # FastAPI application entry point
│   ├── core/
│   │   ├── __init__.py
│   │   ├── config.py        # Configuration settings
│   │   ├── security.py      # Authentication & authorization
│   │   └── database.py      # Database connection
│   ├── api/
│   │   ├── __init__.py
│   │   ├── dependencies.py  # API dependencies
│   │   └── v1/
│   │       ├── __init__.py
│   │       ├── endpoints/
│   │       │   ├── __init__.py
│   │       │   ├── users.py
│   │       │   └── health.py
│   │       └── router.py
│   ├── models/
│   │   ├── __init__.py
│   │   ├── user.py          # Pydantic models
│   │   └── common.py
│   ├── schemas/
│   │   ├── __init__.py
│   │   ├── user.py          # Request/Response schemas
│   │   └── common.py
│   ├── services/
│   │   ├── __init__.py
│   │   ├── user_service.py  # Business logic
│   │   └── auth_service.py
│   └── utils/
│       ├── __init__.py
│       └── helpers.py
├── tests/
│   ├── __init__.py
│   ├── conftest.py          # Pytest configuration
│   ├── test_main.py
│   ├── test_api/
│   └── test_services/
├── requirements.txt         # Production dependencies
├── Dockerfile
├── cloudbuild.yaml
├── .env.example
├── .gitignore
└── README.md

Structure Enforcement

  • Use ruff for linting and formatting
  • Use isort for import organisation
  • Validate layout using company starter templates

2. Code Style & Naming Conventions

General Standards

  • Python: >=3.10 with type hints everywhere
  • Follow PEP8 with 88-character line limit
  • Use ruff for consistent formatting and linting

Naming Conventions

  • Functions/variables: snake_case
  • Classes: PascalCase
  • Constants: UPPER_CASE
  • Files: snake_case.py

Database & API Naming

  • Collections: plural kebab-case (users, invoices)
  • REST endpoints: /api/v1/resources (plural nouns)
  • GCP service names: kebab-case (user-service, data-ingestion-service)

3. Database (MongoDB) Standards

Connection Management

  • All DB connections via app/core/database.py
  • Use async Motor with connection pools
  • Store connection strings in environment variables only
  • Never expose MongoDB _id directly in APIs

Database Management

  • Segregate Test databases and collections accordingly
  • Connection Strings should reflect this Dev > Prod

Schema & Validation

  • Represent MongoDB documents with Pydantic models
  • Enforce strict field typing and validation
  • Use snake_case for all field names

Indexes & Performance

  • Maintain indexes in version-controlled migration scripts
  • Document all indexes with their purpose
  • Monitor query performance regularly

Security

  • Always require database authentication
  • Grant minimum required privileges
  • Use separate DB users for different environments

4. FastAPI Application Standards

Application Structure

from fastapi import FastAPI

def create_app() -> FastAPI:
    app = FastAPI(
        title="Acacia Analytics API",
        version="1.0.0"
    )
    # Register routers, middleware, etc.
    return app

Dependencies & Validation

  • Use FastAPI dependency injection for:
    • Database connections
    • Authentication/authorization
    • Request validation
  • All input/output validated via Pydantic schemas
  • Implement consistent error responses with correlation IDs

API Development Standards

HTTP Methods and Status Codes:

  • GET: Retrieve resources (200, 404)
  • POST: Create new resources (201, 400, 422)
  • PUT: Update entire resources (200, 404)
  • PATCH: Partial updates (200, 404)
  • DELETE: Remove resources (204, 404)

URL Conventions:

  • Use plural nouns: /api/v1/users not /api/v1/user
  • Avoid verbs in URLs: /api/v1/users not /api/v1/get-users
  • Use nested resources: /api/v1/users/{id}/orders

5. Testing Requirements

Test Structure

  • Unit tests for services and utilities
  • Integration tests for database operations
  • API endpoint tests
  • Minimum 80% test coverage

Test Configuration

  • Use pytest with async support
  • Configure test database separately
  • Use fixtures for common test data
  • Mock external dependencies

A/B Testing & Experimentation:

  • Feature flags framework - Simple hash-based implementation for user segmentation
  • Consistent experiment tracking - Using structured logging for metrics
  • Clear documentation requirements for experiments

Staging Environment Strategy:

  • Three-tier setup - Development (local) → Staging → Production
  • Branch-based deployments - develop branch auto-deploys to staging, main to production
  • Staging validation - Mirror production config and run smoke tests
  • Environment-specific secrets - Separate configurations for each environment

Testing Framework:

  • Focused on API testing - Smoke tests for critical endpoints rather than extensive unit testing

  • Practical approach - Test what matters most (endpoints, integrations)

  • Extensive A/B testing - in the Staging deployment, simulate edge cases, pressure test system with volume, etc.

    Database Integration:

    • Connection testing
    • CRUD operation validation
    • Real database interaction

    Service Integration Tests:

    • End-to-end business logic flows
    • Error handling and edge cases
    • Database interaction validation

    Smoke Tests:

    • Health check verification
    • Basic connectivity tests
    • Documentation accessibility

    Running Tests:

    • Various execution scenarios
    • Coverage reporting
    • Selective test running
    • Performance considerations

6. Security Standards

Authentication & Authorization

  • JWT/OAuth2 required for all endpoints except /health
  • Implement rate limiting on public endpoints
  • Use HTTPS everywhere in production

Input Security

  • Validate and sanitize all inputs
  • Use parameterized queries to prevent injection
  • Implement proper CORS configuration

Secrets Management

  • Store secrets in GCP Secret Manager only
  • Never commit secrets to version control
  • Use environment-specific configurations

7. Performance & Scalability

Design Principles

  • Stateless services (no in-memory state)
  • Async-only for all I/O operations (database, HTTP)
  • Use connection pooling everywhere
  • Implement proper error handling and timeouts

Monitoring

  • Use structured JSON logging
  • Include correlation IDs in all log entries
  • Integrate with GCP Cloud Monitoring
  • Monitor response times and error rates

8. Environment Strategy & Deployment

Environment Setup

Development Environment:

  • Use .env.development for local configuration
  • Connect to development MongoDB instance

Staging Environment:

  • Mirror production configuration
  • Use staging GCP project and Production MongoDB cluster
  • Deploy automatically from develop branch
  • Run smoke tests after deployment

Production Environment:

  • Deploy only from main branch after staging validation
  • Implement blue-green or rolling deployments
  • Monitor deployment health and rollback capability

Environment-Specific Configurations:

# cloudbuild-staging.yaml
steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/$SERVICE_NAME-staging:$COMMIT_SHA', '.']

- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/$PROJECT_ID/$SERVICE_NAME-staging:$COMMIT_SHA']

- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: gcloud
  args:
  - 'run'
  - 'deploy'
  - '$SERVICE_NAME-staging'
  - '--image'
  - 'gcr.io/$PROJECT_ID/$SERVICE_NAME-staging:$COMMIT_SHA'
  - '--region'
  - 'us-central1'
  - '--set-env-vars'
  - 'ENVIRONMENT=staging'
  - '--set-secrets'
  - 'MONGO_URI=MONGO_URI_STAGING:latest,SECRET_KEY=SECRET_KEY_STAGING:latest'

# Run smoke tests after deployment
- name: 'gcr.io/cloud-builders/curl'
  args: ['--fail', 'https://$SERVICE_NAME-staging-hash-uc.a.run.app/api/v1/health']

Containerization

FROM python:3.11-slim

WORKDIR /app

# Install system dependencies
RUN apt-get update && apt-get install -y \
    build-essential \
    curl \
    && rm -rf /var/lib/apt/lists/*

# Copy and install Python dependencies
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Copy application code
COPY . .

# Environment variables
ENV PORT=8080
ENV PYTHONUNBUFFERED=1

# Health check
HEALTHCHECK --interval=30s --timeout=10s --start-period=5s --retries=3 \
    CMD curl -f http://localhost:${PORT}/api/v1/health || exit 1

# Expose port
EXPOSE 8080

# Run application
CMD exec uvicorn app.main:app --host 0.0.0.0 --port ${PORT} --workers 2

Cloud Build Configuration

# cloudbuild.yaml
steps:
# Build the container image
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/$PROJECT_ID/$SERVICE_NAME:$COMMIT_SHA', '.']

# Push the container image
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/$PROJECT_ID/$SERVICE_NAME:$COMMIT_SHA']

# Deploy to Cloud Run
- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
  entrypoint: gcloud
  args:
  - 'run'
  - 'deploy'
  - '$SERVICE_NAME'
  - '--image'
  - 'gcr.io/$PROJECT_ID/$SERVICE_NAME:$COMMIT_SHA'
  - '--region'
  - 'us-central1'
  - '--platform'
  - 'managed'
  - '--allow-unauthenticated'
  - '--set-secrets'
  - 'MONGO_URI=MONGO_URI:latest,SECRET_KEY=SECRET_KEY:latest'

images:
- 'gcr.io/$PROJECT_ID/$SERVICE_NAME:$COMMIT_SHA'

options:
  logging: CLOUD_LOGGING_ONLY

Deployment Requirements

  • Create Google Cloud Build trigger linked to main branch
  • Store all secrets in GCP Secret Manager
  • Configure proper IAM roles for service accounts
  • Set up monitoring and alerting

9. Documentation Standards

Required Documentation

  • Clear README with setup instructions
  • API documentation (auto-generated from FastAPI)
  • Architecture diagrams for complex services
  • Deployment and troubleshooting guides

Documentation Maintenance

  • Keep docs up-to-date with code changes
  • Review documentation during code reviews
  • Include examples for all public APIs

10. Development Workflow

Code Quality

  • Use ruff for linting and formatting
  • Type check with mypy
  • Run tests before committing
  • Maintain 80% minimum test coverage

Version Control

  • Use conventional commits
  • Create feature branches from main
  • Require pull request reviews
  • Squash commits when merging

Backend Development Checklist

Project Setup

  • Directory structure follows SOP layout
  • .env.example file created with all required variables
  • README.md with clear setup instructions
  • .gitignore configured for Python projects

Code Quality

  • ruff configured and passing
  • mypy type checking enabled and passing
  • All functions have type hints
  • Code follows PEP8 naming conventions
  • No hardcoded secrets in code

Database

  • MongoDB connection via app/core/database.py
  • Ensure Test Database segragation and correct environment usage
  • All models use Pydantic with proper validation
  • DBs and Collections use kebab-case naming
  • Indexes documented and version-controlled
  • Database authentication configured

API Development

  • FastAPI app follows factory pattern
  • All endpoints use proper HTTP methods and status codes
  • URLs follow REST conventions (plural nouns)
  • Input/output validation with Pydantic schemas
  • Error responses include correlation IDs
  • Health check endpoint implemented

Security

  • JWT/OAuth2 authentication implemented
  • Rate limiting configured where needed
  • All inputs validated and sanitised
  • Secrets stored in GCP Secret Manager only
  • CORS properly configured
  • HTTPS enforced in production

Testing

  • pytest configured with async support
  • Comply with our Testing Framework

    • Practical approach - Test what matters most (endpoints, integrations)

    • Focused on API testing - Smoke tests for critical endpoints rather than extensive unit testing

      • Smoke Tests:
        • Health check verification
        • Basic connectivity tests
        • Documentation accessibility

    • Extensive A/B testing - in the Staging deployment, simulate edge cases, pressure test system with volume, etc.

    • Database Integration:

      • Connection testing
      • CRUD operation validation
      • Real database interaction

    • Service Integration Tests:

      • End-to-end business logic flows
      • Error handling and edge cases
      • Database interaction validation

    • Running Tests:

      • Various execution scenarios
      • Coverage reporting
      • Selective test running
      • Performance considerations

Performance

  • All I/O operations are async
  • Connection pooling implemented
  • Service is stateless
  • Proper error handling and timeouts
  • Response times < 200ms for key requests

Logging & Monitoring

  • Structured JSON logging configured
  • Correlation IDs in all log entries
  • GCP Cloud Monitoring integration
  • Health check endpoint monitoring

Deployment

  • Dockerfile follows SOP template
  • cloudbuild.yaml configured
  • Cloud Build trigger linked to main branch
  • Secrets properly configured in deployment
  • Health checks working
  • Service accessible and responding

Documentation

  • Documentation must be updated and in-line with our current docs
    • This must be clear and concise
    • Explain data flows in the system
    • Explain data dependencies, inputs and expectations
    • Clear Architectural explanations of how components, modules or classes link together are are called between themselves.
  • Deployment instructions documented
  • Architecture diagrams (Use Mermaid Diagrams)

Final Review

  • Code reviewed by team member (if needed)
  • All automated checks passing
  • Performance requirements met
  • Security review completed (if needed)
  • Documentation reviewed and updated

Compliance: All engineers must follow this SOP. Deviations require explicit team lead approval and must be documented. Use this checklist for every new service and major updates.