Upgrade to v0.88.0 from earlier versions of SigNoz

SigNoz is migrating from the legacy underscore exporter to the new dot-metrics exporter. This guide will help you migrate your existing alerts and dashboards to the new system.

✅ Info

Note: This migration is a one-time process. Once completed successfully, your system will be fully transitioned to the new metrics exporter.

Background

As previously communicated, SigNoz is moving from running both the old and new exporters simultaneously to running only the new exporter.

Further Details about this migration can be seen here

Migration

Migration Script

We provide a comprehensive migration script that will help you:

Migrate all existing alerts and dashboards
Backfill historical data for high retention users
Ensure seamless transition to the new metrics system

Github - Migration Script

Pre-Requisite

Take the backup of your SQLite DB.
Take the count of total metrics, samples, and fingerprints, for checking the counts post migration.
The migration process uses various environment variables to configure database connections, performance settings, and migration behavior. Understanding these variables is crucial for successful migration.

Environment Variables

ClickHouse Connection Variables

Variable	Description	Default	Required
`CH_ADDR`	ClickHouse server address and port	`localhost:9001`	Yes
`CH_DATABASE`	ClickHouse database name for metrics	`default`	Yes
`CH_USER`	ClickHouse username for authentication	`default`	Yes
`CH_PASS`	ClickHouse password for authentication	`""` (empty)	Yes

ClickHouse Performance & Connection Pool Variables

Variable	Description	What we used / Default
`CH_MAX_OPEN_CONNS`	Maximum number of open database connections	`10` (metadata), `32` (data), `5` (default)
`CH_MAX_IDLE_CONNS`	Maximum number of idle connections in pool	`8`, `2` (default)
`CH_CONN_MAX_LIFETIME`	Maximum lifetime of a connection	`30m`, `10m` (default)
`CH_DIAL_TIMEOUT`	Connection timeout duration	`60s`, `5s` (default)
`CH_MAX_MEMORY_USAGE`	Maximum memory usage per query (bytes)	`8388608000` (8GB), `1048576000` (default 1GB)
`CH_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY`	Memory threshold for external GROUP BY operations	`524288000` (500MB), `104857600` (default 100MB)
`CH_MAX_BYTES_BEFORE_EXTERNAL_SORT`	Memory threshold for external sorting	`524288000` (500MB), `104857600` (default 100MB)
`CH_MAX_EXECUTION_TIME`	Maximum query execution time (seconds)	`300`, `90` (default)
`CH_MAX_THREADS`	Maximum threads for query processing	`50`, `10` (default)

Migration-Specific Variables

Variable	Description	Default	Usage
`MIGRATE_WORKERS`	Number of parallel migration workers	`12`	Data migration performance
`MIGRATE_MAX_OPEN_CONNS`	Maximum connections for migration process	`32`	Migration-specific connection limit

Metrics & Attributes Mapping Variables

These variables handle mappings for metrics and attributes that may have changed names during the migration, and their dot correspondent metrics and attribute name is not present in the DB.

Variable	Description	Example Value
`NOT_FOUND_METRICS_MAP`	Maps old metric names to new ones, it would be passed as string in this manner 'key1=value1,key2=value2'	`rpc_server_responses_per_rpc_bucket=rpc.server.responses_per_rpc.bucket`
`NOT_FOUND_ATTR_MAP`	Maps old attribute names to new ones, it would be passed as string in this manner 'key1=value1,key2=value2'	`http_scheme=http.scheme,net_peer_name=net.peer.name`
`SKIP_METRICS_MAP`	skip invalid metrics, it would be passed as string in this manner 'metricName1=true,metricName2=true'	`http_scheme=http.scheme,net_peer_name=net.peer.name`

Kubernetes

If you're using Helm charts for deployment, follow these steps:

Update your values.yaml to include the migration configuration
Run the migration commands using the same Docker approach but ensure your ClickHouse connection parameters match your Helm deployment
Verify the migration by checking your dashboards and alerts after completion

Step 1: Migrate Historical Data (For High Retention Users)

This job runs alongside your SigNoz pods and connects with ClickHouse to perform insert operations for migrating older data, (only for those users who have high retention periods)

apiVersion: batch/v1
kind: Job
metadata:
  name: signoz-data-migration-job
  namespace: your-namespace  # Replace with your namespace
spec:
  backoffLimit: 3
  template:
    spec:
      containers:
      - name: migration
        image: signoz/migrate:v0.70.5
        imagePullPolicy: IfNotPresent
        args: ["migrate-data", "--workers=$(MIGRATE_WORKERS)", "--max-open-conns=$(MIGRATE_MAX_OPEN_CONNS)"]
        env:
        - name: CH_ADDR
          value: "your-clickhouse-service:9000"  # Replace with your ClickHouse service
        - name: CH_DATABASE
          value: "signoz_metrics"
        - name: CH_USER
          value: "admin"  # Replace with your ClickHouse user
        - name: CH_PASS
          value: "your-password"  # Replace with your ClickHouse password
        - name: CH_MAX_OPEN_CONNS
          value: "32"
        - name: CH_MAX_MEMORY_USAGE
          value: "8388608000"
        - name: CH_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY
          value: "524288000"
        - name: CH_MAX_BYTES_BEFORE_EXTERNAL_SORT
          value: "524288000"
        - name: CH_DIAL_TIMEOUT
          value: "60s"
        - name: CH_CONN_MAX_LIFETIME
          value: "30m"
        - name: CH_MAX_IDLE_CONNS
          value: "8"
        - name: CH_MAX_EXECUTION_TIME
          value: "300"
        - name: CH_MAX_THREADS
          value: "50"
        - name: MIGRATE_WORKERS
          value: "12"
        - name: MIGRATE_MAX_OPEN_CONNS
          value: "32"
        - name: NOT_FOUND_METRICS_MAP
          value: "rpc_server_responses_per_rpc_bucket=rpc.server.responses_per_rpc.bucket"
        - name: NOT_FOUND_ATTR_MAP
          value: "http_scheme=http.scheme,net_peer_name=net.peer.name,net_peer_port=net.peer.port,net_protocol_name=net.protocol.name,net_protocol_version=net.protocol.version,rpc_grpc_status_code=rpc.grpc.status_code,rpc_method=rpc.method,rpc_service=rpc.service,rpc_system=rpc.system"
        resources:
          requests:
            memory: 116000Mi
            cpu: 29500m
      restartPolicy: Never
      tolerations:
      - effect: NoSchedule
        key: signoz.cloud/workload
        operator: Equal
        value: store
      - effect: NoSchedule
        key: signoz.cloud/deployment.tier
        operator: Equal
        value: premium

Apply the data migration job:

kubectl apply -f data-migration-job.yaml

# 1) List Jobs (verify signoz-data-migration-job exists and its status)
kubectl get jobs -n <your-namespace>

# 2) List Pods created by that Job
kubectl get pods -l job-name=signoz-data-migration-job -n <your-namespace> -o wide

To verify the migration is successful, you can check the logs of the migration container:

kubectl logs job/signoz-data-migration-job -c migration -n <your-namespace>
2025-06-25 18:30:12 {"level":"info","ts":1750856412.9032447,"caller":"migrate/main.go:659","msg":"query args","start":1749718800000,"end":1749722250683}
2025-06-25 18:30:12 {"level":"info","ts":1750856412.943157,"caller":"migrate/main.go:728","msg":"migration success for windows start: 1749718800000, and end: 1749722250683"}
2025-06-25 18:30:12 2025/06/25 13:00:12 Data migration completed [1749718800000…1749722250683]

Step 2: Migrate Alerts and Dashboards Using Init Container

For alerts and dashboards migration, you need to add an init container to your SigNoz deployment. This init container will run before the main SigNoz container starts and execute the necessary queries on the SQLite database to migrate alerts and dashboards.

Add the following init container to your SigNoz deployment manifest:

    initContainers:
      - name: migration
        image: signoz/migrate:v0.70.5
        imagePullPolicy: IfNotPresent
        env:
          - name: SQL_DB_PATH
            value: /var/lib/signoz/signoz.db
          - name: CH_ADDR
            value: "your-clickhouse-service:9000" # Replace with your ClickHouse service
          - name: CH_DATABASE
            value: signoz_metrics
          - name: CH_USER
            value: admin
          - name: CH_PASS
            value: "your-password"  # Replace with your ClickHouse password
          - name: CH_MAX_OPEN_CONNS
            value: "10"
          - name: SKIP_METRICS_MAP
            value: "dd_internal_stats_payload=true"
          - name: CH_MAX_MEMORY_USAGE
            value: "8388608000"                        # 8 GB
          - name: CH_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY
            value: "4194304000"                        # 4 GB
          - name: CH_MAX_BYTES_BEFORE_EXTERNAL_SORT
            value: "4194304000"                        # 4 GB
          - name: NOT_FOUND_METRICS_MAP
            value: |-
              rpc_server_responses_per_rpc_bucket=rpc.server.responses_per_rpc.bucket
          - name: NOT_FOUND_ATTR_MAP
            value: |-
              http_scheme=http.scheme,
        args:
          - migrate-meta
        resources: {}   # add limits/requests as needed
        volumeMounts:
          - name: signoz-db
            mountPath: /var/lib/signoz

# 1) List all Deployments in the Signoz namespace
kubectl get deploy -n <your-namespace>

# 2) List Pods (init containers show up as Init:<x>/<y> until they finish)
kubectl get pod -n <your-namespace> -o wide

To verify the migration is successful, you can check the logs of the migration init container:

kubectl logs <pod-name> -c migration -n <your-namespace>
✅ updated x dashboards in signoz.db
✅ updated y rules in /var/lib/signoz/signoz.db

Step 3: Enable New Metrics

You also need to pass environment variable DOT_METRICS_ENABLED to true while running the SigNoz container, to allow the platform to use dot metrics.

Docker Standalone Users

If you're running SigNoz using Docker, follow these steps:

Update your Docker Compose file to include the migration configuration
Run the migration commands using the provided Docker image
Verify the migration by checking your dashboards and alerts after completion

Step 1: Migrate Historical Data (For High Retention Users)

This job runs alongside your SigNoz container and connects with ClickHouse to perform insert operations for migrating older data, (only for those users who have high retention periods)

Add the following to your docker-compose.yml file, Kindly make sure that your Clickhouse container is running and accessible.

  migration-job:
    !!merge <<: *db-depend
    image: signoz/migrate:v0.70.5
    command: >
      migrate-data
      --workers=${MIGRATE_WORKERS:-12}
      --max-open-conns=${MIGRATE_MAX_OPEN_CONNS:-32}
    environment:
      # Point to the ClickHouse service defined in SigNoz's compose file
      CH_ADDR: signoz-clickhouse:9000
      CH_DATABASE: signoz_metrics
      CH_USER: default
      CH_PASS: ""

      CH_MAX_OPEN_CONNS: "32"
      CH_MAX_MEMORY_USAGE: "8388608000"
      CH_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY: "524288000"
      CH_MAX_BYTES_BEFORE_EXTERNAL_SORT: "524288000"
      CH_DIAL_TIMEOUT: "60s"
      CH_CONN_MAX_LIFETIME: "30m"
      CH_MAX_IDLE_CONNS: "8"
      CH_MAX_EXECUTION_TIME: "300"
      CH_MAX_THREADS: "50"

      MIGRATE_WORKERS: "12"
      MIGRATE_MAX_OPEN_CONNS: "32"

      NOT_FOUND_METRICS_MAP: >
        rpc_server_responses_per_rpc_bucket=rpc.server.responses_per_rpc.bucket
      NOT_FOUND_ATTR_MAP: >
        http_scheme=http.scheme,net_peer_name=net.peer.name,net_peer_port=net.peer.port,
        net_protocol_name=net.protocol.name,net_protocol_version=net.protocol.version,
        rpc_grpc_status_code=rpc.grpc.status_code,rpc_method=rpc.method,
        rpc_service=rpc.service,rpc_system=rpc.system

    restart: "no"

Apply the migration job:

docker-compose up -d migration-job

Check the logs to ensure the migration is running smoothly:

docker-compose logs -f migration-job

Step 2: Migrate Alerts and Dashboards

For alerts and dashboards migration, you need to run the migration script against your SQLite database. This can be done using the same Docker image.

Add the following service in your docker compose setup or run the container separately:

  migrate-signoz:
    !!merge <<: *db-depend
    image: signoz/migrate:v0.70.5
    command: migrate-meta         
    restart: "no"              
    environment:
      SQL_DB_PATH: /var/lib/signoz/signoz.db
      CH_ADDR: signoz-clickhouse:9000
      CH_DATABASE: signoz_metrics
      CH_USER: default
      CH_PASS: ""

      CH_MAX_OPEN_CONNS: "10"
      CH_MAX_MEMORY_USAGE: "8388608000"
      CH_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY: "4194304000"
      CH_MAX_BYTES_BEFORE_EXTERNAL_SORT: "4194304000"

      SKIP_METRICS_MAP: "dd_internal_stats_payload=true"
      NOT_FOUND_METRICS_MAP: "rpc_server_responses_per_rpc_bucket=rpc.server.responses_per_rpc.bucket"
      NOT_FOUND_ATTR_MAP: "http_scheme=http.scheme,"
    volumes:
      - sqlite:/var/lib/signoz

To make sure that migration runs before the main SigNoz container, you can use the depends_on directive in your docker-compose.yml:

  signoz:
    depends_on:
      - migrate-signoz

For checking whether the migration is successful or not, you can check the logs of the migrate-signoz container:

docker logs -f migrate-signoz
✅ updated x dashboards in signoz.db
✅ updated y rules in /var/lib/signoz/signoz.db

Step 3: Enable New Metrics

You also need to pass environment variable DOT_METRICS_ENABLED to true, to allow the platform to use dot metrics.

  signoz:
    environment:
      DOT_METRICS_ENABLED: "true"

Docker Swarm

(Steps are identical to Docker Standalone but executed with docker stack commands. Use the official upgrade guide and substitute the environment flag and migration script as above.)

Step 1: Migrate Historical Data (For High Retention Users)

You can run the migration job in your Docker Swarm setup by creating a service with the migration image. Use the same environment variables as mentioned above.

Below service will run as a job and will not restart after completion.

docker service create \
  --name signoz_migration_job \
  --mode replicated-job --replicas 1 \
  --restart-condition none \
  --network signoz-net \
  --mount type=volume,src=sqlite,dst=/var/lib/signoz \
  \
  --env CH_ADDR=signoz_clickhouse:9000 \
  --env CH_DATABASE=signoz_metrics \
  --env CH_USER=default \
  --env CH_PASS= \
  \
  --env CH_MAX_OPEN_CONNS=32 \
  --env CH_MAX_MEMORY_USAGE=8388608000 \
  --env CH_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY=524288000 \
  --env CH_MAX_BYTES_BEFORE_EXTERNAL_SORT=524288000 \
  --env CH_DIAL_TIMEOUT=60s \
  --env CH_CONN_MAX_LIFETIME=30m \
  --env CH_MAX_IDLE_CONNS=8 \
  --env CH_MAX_EXECUTION_TIME=300 \
  --env CH_MAX_THREADS=50 \
  \
  --env MIGRATE_WORKERS=12 \
  --env MIGRATE_MAX_OPEN_CONNS=32 \
  \
  --env NOT_FOUND_METRICS_MAP=rpc_server_responses_per_rpc_bucket=rpc.server.responses_per_rpc.bucket \
  --env NOT_FOUND_ATTR_MAP=http_scheme=http.scheme,net_peer_name=net.peer.name,net_peer_port=net.peer.port,net_protocol_name=net.protocol.name,net_protocol_version=net.protocol.version,rpc_grpc_status_code=rpc.grpc.status_code,rpc_method=rpc.method,rpc_service=rpc.service,rpc_system=rpc.system \
  \
  signoz/migrate:v0.70.5 \
  migrate-data --workers 12 --max-open-conns 32

After the service is created, you can check the status of the migration job:

docker service ls

To check the logs of the migration job, you can use:

docker service logs signoz_migration_job -f

signoz_migration_job    | {"level":"info","ts":1750878083.9140384,"caller":"migrate/main.go:659","msg":"query args","start":1750867200000,"end":1750869262833}
signoz_migration_job   | {"level":"info","ts":1750878084.1492937,"caller":"migrate/main.go:728","msg":"migration success for windows start: 1750867200000, and end: 1750869262833"}
signoz_migration_job  | 2025/06/25 19:01:24 Data migration completed [1750867200000…1750869262833]

Step 2: Migrate Alerts and Dashboards

For alerts and dashboards migration, you can create another service in your Docker Swarm setup using the migration image. Use the same environment variables as mentioned above.

docker service create \
  --name "signoz_meta-job" \
  --mode replicated-job --replicas 1 \
  --restart-condition none \
  --network "signoz-net" \
  --mount type=volume,src=signoz-sqlite,dst=/var/lib/signoz/ \
  \
  -e SQL_DB_PATH=/var/lib/signoz/signoz.db \
  -e CH_ADDR=signoz_clickhouse:9000 \
  -e CH_DATABASE=signoz_metrics \
  -e CH_USER=default \
  -e CH_PASS="" \
  -e CH_MAX_MEMORY_USAGE="8388608000" \
  -e CH_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY="4194304000" \
  -e CH_MAX_BYTES_BEFORE_EXTERNAL_SORT="4194304000" \
  -e CH_MAX_OPEN_CONNS="10" \
  -e SKIP_METRICS_MAP="dd_internal_stats_payload=true" \
  -e NOT_FOUND_METRICS_MAP="rpc_server_responses_per_rpc_bucket=rpc.server.responses_per_rpc.bucket" \
  -e NOT_FOUND_ATTR_MAP="http_scheme=http.scheme," \
  \
  signoz/migrate:v0.70.5 \
  migrate-meta

After the service is created, you can check the status of the migration job:

docker service ls

To check the logs of the migration job, you can use:

docker service logs signoz_meta-job -f
signoz_meta-job  | ✅ updated 3 dashboards in signoz.db
signoz_meta-job  | ✅ updated 0 rules in /var/lib/signoz/signoz.db

Step 3: Enable New Metrics

You also need to pass environment variable DOT_METRICS_ENABLED to true, to allow the platform to use dot metrics.

  signoz:
    environment:
      DOT_METRICS_ENABLED: "true"

Apply the updated stack configuration:

docker stack deploy -c docker-compose.yml signoz

Linux Users

If you're running SigNoz on a Linux server without Docker or Kubernetes, you can follow these steps:

Download the migration binary from the migration repository.
Run the migration binary from the command line, passing the necessary environment variables and command-line arguments according to your setup.

Step 1: Migrate Historical Data (For High Retention Users)

Run the migration command to backfill historical data:

#!/usr/bin/env bash
set -euo pipefail

trap 'echo "❌ Migration failed at line $LINENO."; exit 1' ERR


# 1) Export ClickHouse settings
export CH_ADDR="localhst:9000"
export CH_DATABASE="signoz_metrics"
export CH_USER="default"
export CH_PASS="password"

# 2) Export ClickHouse tuning knobs (optional—use what you need)
export CH_MAX_OPEN_CONNS="32"
export CH_MAX_MEMORY_USAGE="8388608000"
export CH_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY="524288000"
export CH_MAX_BYTES_BEFORE_EXTERNAL_SORT="524288000"
export CH_DIAL_TIMEOUT="60s"
export CH_CONN_MAX_LIFETIME="30m"
export CH_MAX_IDLE_CONNS="8"
export CH_MAX_EXECUTION_TIME="300"
export CH_MAX_THREADS="50"

export MIGRATE_WORKERS="12"
export MIGRATE_MAX_OPEN_CONNS="32"

export NOT_FOUND_METRICS_MAP="rpc_server_responses_per_rpc_bucket=rpc.server.responses_per_rpc.bucket"
export NOT_FOUND_ATTR_MAP="http_scheme=http.scheme,net_peer_name=net.peer.name,net_peer_port=net.peer.port,net_protocol_name=net.protocol.name,net_protocol_version=net.protocol.version,rpc_grpc_status_code=rpc.grpc.status_code,rpc_method=rpc.method,rpc_service=rpc.service,rpc_system=rpc.system"
export SKIP_METRICS_MAP="http_scheme=http.scheme,net_peer_name=net.peer.name"
# 3) Point to your local SQLite file (for metadata)
export SQL_DB_PATH="/var/lib/signoz/signoz.db"

# 4) Run metadata migration first (dashboards, users, etc.)
# echo ">> Migrating metadata (sqlite)…"
# migrate-linux migrate-meta

# 5) Then run the data migration (ClickHouse)
echo ">> Migrating metrics data (ClickHouse)…"
migrate_output=$(./migrate-linux migrate-data \
  --workers "${MIGRATE_WORKERS}" \
  --max-open-conns "${MIGRATE_MAX_OPEN_CONNS}" \
  2>&1)


# Print what the tool said
echo "$migrate_output"

# If any line contains `"level":"error"`, abort
if grep -q '"level":"error"' <<<"$migrate_output"; then
  echo "❌ Migration failed: errors detected in migrate-data output."
  exit 1
fi

echo "✅ Migration complete!"

If you encounter issues, ensure that your ClickHouse server is running and accessible, and that the environment variables are correctly set.

On successful completion, you should see a message indicating that the migration was successful:

✅ Migration complete!

Step 2: Migrate Alerts and Dashboards

Run the migration command to update alerts and dashboards:

#!/usr/bin/env bash
set -euo pipefail
trap 'echo "❌ Migration failed at line $LINENO."; exit 1' ERR

# 1) Export ClickHouse settings
export CH_ADDR="localhost:9000"
export CH_DATABASE="signoz_metrics"
export CH_USER="default"
export CH_PASS="password"

# 2) Export ClickHouse tuning knobs (optional—use what you need)
export CH_MAX_OPEN_CONNS="32"
export CH_MAX_MEMORY_USAGE="8388608000"
export CH_MAX_BYTES_BEFORE_EXTERNAL_GROUP_BY="4194304000"
export CH_MAX_BYTES_BEFORE_EXTERNAL_SORT="524288000"


export NOT_FOUND_METRICS_MAP="rpc_server_responses_per_rpc_bucket=rpc.server.responses_per_rpc.bucket"
export NOT_FOUND_ATTR_MAP="http_scheme=http.scheme,"
export SKIP_METRICS_MAP="dd_internal_stats_payload=true"
# 3) Point to your local SQLite file (for metadata)
export SQL_DB_PATH="/var/lib/signoz/signoz.d"

# 4) Run metadata migration first (dashboards, users, etc.)
echo ">> Migrating metadata (sqlite)…"
migrate_output=$(./migrate-linux migrate-meta \
  2>&1)
echo "$migrate_output"

# If any line contains `"level":"error"`, abort
if grep -q '"level":"error"' <<<"$migrate_output"; then
  echo "❌ Migration failed: errors detected in migrate-meta output."
  exit 1
fi

echo "✅ Migration complete!"

If you encounter issues, ensure that your SQLite database file is accessible and that the environment variables are correctly set.

On successful completion, you should see a message indicating that the migration was successful:

>> Migrating metadata (sqlite)…
✅ updated 0 dashboards in signoz.db
✅ updated 0 rules in /var/lib/signoz/signoz.db
✅ Migration complete!

Step 3: Enable New Metrics

To enable the new dot metrics, you need to set the environment variable DOT_METRICS_ENABLED to true in your SigNoz configuration. This can be done by adding the following line to your SigNoz configuration file or environment variables:

vim /opt/signoz/conf/systemd.env
# Add the following line to enable dot metrics
DOT_METRICS_ENABLED=true

Migration Failure

If you encounter issues during migration:

Check the SigNoz Documentation
Visit our GitHub Issues
Join our Community Slack for real-time support

Next Steps

Once migration is complete:

Monitor your system for a few days to ensure stability
Update any external integrations that might reference the old metrics format
Consider optimizing your retention policies based on the new exporter's capabilities

FAQ

Connection errors: Verify your ClickHouse connection parameters
Permission issues: Ensure the Docker container has access to your signoz.db file
Data inconsistencies: Run the migration script again if you notice missing data

Next Steps

Was this page helpful?

Migration Guides

Upgrade to v0.76