How ClickHouse Diagnostic Logs Filled My Uptrace Server

My self-hosted Uptrace server had used 96 GiB of a 154 GiB root filesystem. Uptrace was configured to retain project data for 28 days, so I expected old telemetry to disappear. Instead, ClickHouse’s own diagnostic tables had grown to 79.1 GiB. The Uptrace application tables occupied just 1.19 GiB.

After removing fully expired diagnostic partitions and limiting new diagnostic data, the filesystem used 23 GiB (15%). Uptrace’s 28-day project retention did not change. This article describes the investigation and the operational detail that mattered most: adding a TTL to a large existing table can start an expensive rewrite.

Where the disk space went #

I compared filesystem use with active ClickHouse parts, grouping by database and table:

SELECT
    database,
    table,
    formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE active AND database IN ('system', 'uptrace')
GROUP BY database, table
ORDER BY sum(bytes_on_disk) DESC;

The largest tables were all in the system database:

Those five tables alone accounted for almost 75 GiB. They record ClickHouse’s internal activity; they are separate from the spans and logs stored for an Uptrace project. Project retention therefore did not limit their size.

The write rate explained the growth. On one day, text_log received about 2.81 million Trace rows and 1.59 million Debug rows, compared with only 68 rows at Information or higher. trace_log received roughly 1.27 million Memory and 1.27 million MemoryPeak samples. The ClickHouse configuration used trace logging and a 4 MiB memory profiler step, and the large diagnostic tables had no TTL.

If you see a similar mismatch, inspect system.parts before assuming that your application’s retention policy is broken. For background on Uptrace and its application telemetry, see my Uptrace and OpenTelemetry guide .

Reduce the incoming diagnostic volume #

I set both the server logger and text_log to information, and disabled routine memory and query stack sampling in the default ClickHouse profile. This still keeps informational messages, warnings, errors, and other ClickHouse system logs. Profiling can be enabled again for a focused investigation.

The settings live in XML files mounted read-only into the ClickHouse container, so recreating the container preserves them. The profile override was:

<clickhouse>
  <profiles>
    <default>
      <memory_profiler_step>0</memory_profiler_step>
      <query_profiler_real_time_period_ns>0</query_profiler_real_time_period_ns>
      <query_profiler_cpu_time_period_ns>0</query_profiler_cpu_time_period_ns>
    </default>
  </profiles>
</clickhouse>

I also configured seven-day row TTLs for ten high-volume system logs. Here is the server override, stored as clickhouse-config.d/diagnostic-storage.xml next to the Compose file:

<clickhouse>
  <logger><level>information</level></logger>
  <trace_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></trace_log>
  <text_log>
    <level>information</level>
    <ttl>event_date + INTERVAL 7 DAY DELETE</ttl>
  </text_log>
  <query_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></query_log>
  <query_views_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></query_views_log>
  <part_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></part_log>
  <metric_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></metric_log>
  <asynchronous_metric_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></asynchronous_metric_log>
  <processors_profile_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></processors_profile_log>
  <query_metric_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></query_metric_log>
  <error_log><ttl>event_date + INTERVAL 7 DAY DELETE</ttl></error_log>
</clickhouse>

The Compose service mounts this file at /etc/clickhouse-server/config.d/diagnostic-storage.xml and the profile file at /etc/clickhouse-server/users.d/profiling.xml. I validated the XML and ran docker compose config --quiet before recreating only ClickHouse.

The XML covers tables ClickHouse creates in the future. Existing tables also needed their TTL definitions changed with ALTER TABLE.

Recover old space without rewriting everything at once #

First, I listed partitions and their sizes:

SELECT
    table,
    partition,
    formatReadableSize(sum(bytes_on_disk)) AS size
FROM system.parts
WHERE database = 'system' AND active
GROUP BY table, partition
ORDER BY table, partition;

These system logs were partitioned by month. On September 27, the April through August partitions were wholly outside the new seven-day retention window. I dropped only those confirmed system log partitions, using ClickHouse’s ALTER TABLE ... DROP PARTITION command. For example:

ALTER TABLE system.trace_log DROP PARTITION ID '202604';

I repeated that operation for the confirmed expired partitions in the ten affected tables: 45 partitions in total, holding 70.029 GiB. I did not delete files from the Docker volume or touch the uptrace database.

Next I added the TTL to existing tables. A small query_metric_log table served as a canary:

ALTER TABLE system.query_metric_log
    MODIFY TTL event_date + INTERVAL 7 DAY DELETE;

An immediate TTL rewrite on metric_log then exceeded this host’s 7 GiB ClickHouse memory limit. I stopped the failed mutation; its TTL definition remained in place. For the remaining large tables I used:

ALTER TABLE system.text_log
    MODIFY TTL event_date + INTERVAL 7 DAY DELETE
    SETTINGS materialize_ttl_after_modify = 0;

This avoids an immediate full materialization when changing the TTL. It also means old rows are not instantly removed. I left the current month’s diagnostic data for background TTL work and scheduled checks of the actual part sizes and oldest retained dates. I did not force OPTIMIZE FINAL on these large tables. ClickHouse documents that TTL deletion happens during merges , so a TTL definition alone is not proof that disk space has already been recovered.

With monthly partitions, a seven-day policy cannot simply drop the whole current month. I kept row-level TTL rather than ttl_only_drop_parts=1, which would retain rows until their entire part had expired. For a new installation, aligning partition granularity with the retention period can make expiration cheaper; changing this existing layout was outside this cleanup.

Verify the result, then keep watching #

After recreating ClickHouse, I checked its health, the effective logger and profiler settings, all ten table TTL definitions, unfinished mutations, and any newly created system.*_log_0 tables. Uptrace’s HTTP page returned 200, a query worked, and the newest stored span was five seconds old. No new Debug, Trace, or routine profiler rows appeared after the configuration took effect. The Uptrace project’s four 28-day retention values were unchanged.

These are immediate results from one ClickHouse 25.8.15.35 installation, not a measured steady state. Read-only checks are scheduled at 24 hours and one week to confirm that the remaining older diagnostic rows expire and that system storage stays small. If it starts growing quickly again, I will inspect the per-table write rates, TTL progress, and any renamed system log tables before changing retention further.

The lesson for this installation is simple: monitor ClickHouse’s system database separately from Uptrace project data. They have different retention controls, and the database’s own diagnostics can become the dominant disk user.

References #

• ClickHouse TTL behavior and partitioning guidance
• ClickHouse <code>DROP PARTITION</code> reference

Illustrative header photo by Denny Müller on Unsplash .