Singularity
Guides

Database Scripts

Write, track, and manage database migration scripts with rollback support, hash-based change detection, parallel execution with table locking, and full execution history.

Singularity provides a structured script system for database migrations, data patches, and maintenance operations. Every script extends BaseScript, which adds execution tracking, rollback support, content-hash change detection, and automatic re-execution when a script file is modified. Scripts can run on application startup or be triggered manually through the CLI.


Script Execution Lifecycle

The following state diagram shows how a script moves through discovery, execution, and change detection.

Key behaviors:

  • Hash-based change detection -- Every time a script runs successfully, its SHA-256 file hash is stored in script_run_tracker. On subsequent startups, the runner compares the current hash against the stored one. If they differ, the script re-executes automatically.
  • Verification -- After execute() finishes, the runner calls verify(). If verification fails, the script is marked as failed.
  • Suppressed errors -- Scripts with suppress_errors=True log failures as warnings and do not block application startup.

Parallel Lock Scheduling

When multiple auto-run scripts execute during startup, the ScriptRunner launches them in parallel using asyncio.gather. Each script declares which tables it affects, and the runner creates asyncio.Lock instances per table. Scripts acquire their locks (in alphabetical order to prevent deadlocks) before executing, allowing independent scripts to run concurrently while scripts that share tables are serialized.

In this example, create_users and create_products run in parallel because they affect different tables. seed_orders depends on users, products, and orders, so it waits until the first two finish.

If a script does not declare affected_tables, the runner infers dependencies by analyzing the script's source code with AST inspection, searching for SQL keywords like FROM, JOIN, UPDATE, and INTO in string literals.


The BaseScript Class

Every script extends BaseScript and implements at least the execute() method. The class provides tracking, rollback, verification, and CLI integration out of the box.

Constructor Parameters

ParameterTypeDefaultDescription
script_namestrrequiredUnique identifier used in script_run_tracker
auto_runboolFalseRun automatically on application startup
rerunboolFalseForce rerun on every startup (even if already succeeded)
deprecatedboolFalseSkip on startup but keep rollback available
affected_tableslist[str]NoneTables modified by this script (used for lock scheduling)
suppress_errorsboolFalseLog errors as warnings instead of raising (does not block startup)

Lifecycle Methods

MethodRequiredDescription
execute(db)YesMain script logic. db is whatever handle the active tracker yields (see Tracker backends below).
rollback(db)NoUndo changes made by execute().
verify(db)NoReturn True if execution was successful. Defaults to True.

Tracker Backends

Execution state lives in a pluggable tracker — every backend implements the ScriptTracker protocol from singularity.scripts.trackers:

Backenddb handle passed to execute()Extra requiredAuto-selected when
Postgres (default)sqlalchemy.ext.asyncio.AsyncSession[sql][sql] installed
Mongopymongo.asynchronous.database.AsyncDatabase[mongo]MONGO_URL + MONGO_DB_NAME set
NoopNoneneither extra installed

The default is automatic. Pin one explicitly by setting tracker_backend = "mongo" on a subclass, or by passing a tracker= instance to super().__init__().

All backends use a script_run_tracker table/collection and share the same schema: {script_name, status, error_message, content_hash, updated_at}. The runner's history, sync-hashes, and startup commands work uniformly regardless of which backend is active.


Writing Scripts

Basic Migration Script

A script that creates default product records in the database. It runs automatically on startup, affects the products and prices tables, and provides both rollback and verification.

# scripts/executable/create_stripe_products.py

from sqlalchemy import delete, select, text
from sqlalchemy.ext.asyncio import AsyncSession

from singularity.scripts.base_script import BaseScript
from singularity.common.logger import log as logger


class CreateStripeProductsScript(BaseScript):
    """Create default Stripe product records in the database."""

    def __init__(self):
        super().__init__(
            "create_stripe_products",
            auto_run=True,
            affected_tables=["products", "prices"],
        )

    async def execute(self, db: AsyncSession) -> None:
        logger.info("Creating default Stripe products...")

        await db.execute(
            text("""
                INSERT INTO products (name, stripe_id, active)
                VALUES ('Pro Plan', 'prod_xxx', true)
                ON CONFLICT (stripe_id) DO NOTHING
            """)
        )
        await db.commit()

    async def rollback(self, db: AsyncSession) -> None:
        await db.execute(
            text("DELETE FROM products WHERE stripe_id = 'prod_xxx'")
        )
        await db.commit()

    async def verify(self, db: AsyncSession) -> bool:
        result = await db.execute(
            text("SELECT COUNT(*) FROM products WHERE stripe_id = 'prod_xxx'")
        )
        count = result.scalar()
        return count is not None and count > 0


def main():
    script = CreateStripeProductsScript()
    script.main()

if __name__ == "__main__":
    script = CreateStripeProductsScript()
    script.run_cli()

Non-Critical Cleanup with Error Suppression

For maintenance tasks that should not block startup if they fail.

# scripts/executable/cleanup_orphaned_records.py

from datetime import datetime, timedelta
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession

from singularity.scripts.base_script import BaseScript
from singularity.common.logger import log as logger


class CleanupOrphanedRecordsScript(BaseScript):
    """Remove temp uploads older than 30 days."""

    def __init__(self):
        super().__init__(
            "cleanup_orphaned_records",
            auto_run=True,
            suppress_errors=True,       # Log failures as warnings, don't block startup
            affected_tables=["temp_uploads"],
        )

    async def execute(self, db: AsyncSession) -> None:
        cutoff = datetime.utcnow() - timedelta(days=30)
        result = await db.execute(
            text("DELETE FROM temp_uploads WHERE created_at < :cutoff"),
            {"cutoff": cutoff},
        )
        await db.commit()
        logger.info(f"Cleaned up {result.rowcount} orphaned records")


def main():
    script = CleanupOrphanedRecordsScript()
    script.main()

if __name__ == "__main__":
    script = CleanupOrphanedRecordsScript()
    script.run_cli()

Deprecated Script

Mark a script as deprecated to skip it on startup while preserving its rollback logic. This is useful for scripts that have been superseded but may still need to be rolled back.

class OldMigrationScript(BaseScript):
    """Legacy migration -- superseded by v2_migration."""

    def __init__(self):
        super().__init__(
            "old_migration_v1",
            deprecated=True,    # Skipped on startup, rollback still available
        )

    async def execute(self, db: AsyncSession) -> None:
        # Original migration logic (kept for reference)
        ...

    async def rollback(self, db: AsyncSession) -> None:
        # Rollback logic still works via CLI
        ...

When a deprecated script's hash changes, the runner logs a warning but does not auto-rerun it. You can still run it manually with scripts run old_migration_v1.


Execution Tracking

Every script execution is recorded in the script_run_tracker database table.

ColumnTypeDescription
script_nameVARCHARUnique script identifier (primary key)
statusVARCHARpending, success, failed, or rolled_back
error_messageTEXTError details when status is failed
content_hashVARCHARSHA-256 hash of the script file at last execution
executed_atTIMESTAMPWhen the record was first created
updated_atTIMESTAMPWhen the record was last updated

Hash-Based Change Detection

The runner uses this table for intelligent re-execution:

  1. On startup, the runner discovers all scripts in scripts/executable/.

  2. For each auto_run script, it checks script_run_tracker for an existing record.

  3. If the record shows status = 'success', it calculates the current file hash and compares it with the stored content_hash.

  4. If the hashes differ, the script is re-executed automatically.

  5. If the hashes match and the script already succeeded, it is skipped.


Startup Coordination

When running with multiple Gunicorn or Uvicorn workers, only one worker should execute startup scripts. The run_startup_scripts() function uses a PostgreSQL advisory lock to coordinate:

from singularity.scripts.runner import run_startup_scripts

# Inside your application lifespan
async def lifespan(app):
    await run_startup_scripts()
    yield

The first worker to acquire the advisory lock runs all auto-run scripts. Other workers detect the lock and skip execution, logging an informational message.


CLI Usage

The scripts CLI provides commands for the full script lifecycle.

Creating Scripts

Generate a new script with proper boilerplate using the create command:

# Create a script with a description
singularity generate script add_user_roles -d "Add default user roles to the database"

# Create and overwrite an existing script
singularity generate script add_user_roles -d "Updated roles" --overwrite

The generated file includes BaseScript inheritance, execute(), rollback(), and verify() stubs, path setup, and a CLI entry point.

Running Scripts

# Run a script by name (default command is "run")
scripts run add_user_roles

# Run with force (re-execute even if already succeeded)
scripts run add_user_roles run --force

# Check execution status
scripts run add_user_roles status

# Rollback a previously executed script
scripts run add_user_roles rollback

# Run a script by its numeric ID (from the list output)
scripts run 3
scripts run 3 status

Listing and History

# List all discovered scripts with detailed info
scripts list

# Show simplified list
scripts list --simple

# Display execution history from the database
scripts history

# Sync content hashes for scripts that predate hash tracking
scripts sync-hashes

# Run all auto-run scripts (same as application startup)
scripts startup

CLI Command Reference

CommandDescription
scripts listList all discovered scripts with auto-run status, affected tables, and descriptions
singularity generate script <name>Generate a new script file with boilerplate
scripts run <name|id>Execute a script (default), check status, or rollback
scripts run <name> rollbackRoll back a previously executed script
scripts run <name> statusCheck the execution status of a script
scripts historyShow the full execution history from the database
scripts sync-hashesPopulate missing content hashes for legacy scripts
scripts startupRun all auto-run scripts (used during app startup)

Best Practices

Always declare affected tables

Explicit affected_tables prevents the AST-based inference from missing a dependency. This ensures correct lock scheduling during parallel startup.

Implement verify()

A verify() method catches silent failures (e.g., the query ran but inserted zero rows). The runner marks the script as failed if verification returns False.

Use suppress_errors for non-critical work

Cleanup and maintenance scripts should set suppress_errors=True so a failure does not prevent the application from starting.

Keep rollback logic current

When you update a script's execute() method, update rollback() to match. Hash-based re-execution means the old rollback may no longer be correct.