show_chunks() does not correspond to what DELETE complains about - compression

Question

This is really puzzling. I need to delete a date from a hypertable from timescaleDB 1.7:
DELETE FROM raw WHERE tm::date = '2020-11-06' -- the local date style is YYYY-MM-DD
Before doing that, I check what chunks I need to decompress, giving it one day margin, and receive two chunks:
SELECT show_chunks('raw', newer_than => '2020-11-05 00:00'::timestamp)
---
Result:
"_timescaledb_internal._hyper_1_19_chunk"
"_timescaledb_internal._hyper_1_21_chunk"
So I decompress these two. However when I run the DELETE command above, I still get an error about totally different chunk:
ERROR: cannot update/delete rows from chunk "_hyper_1_1_chunk" as it
is compressed SQL state: XX000
BTW this chunk is empty as far as I can see by looking at it in the pgAdmin. Any idea what's going on? Looks like a bug to me, but maybe I'm doing something wrong?
Thanks!
Edit:
Below is an excerpt from the result of EXPLAIN DELETE, as requested by #k_rus:
EXPLAIN DELETE FROM raw WHERE tm::date = '2020-11-06'
Result:
"Delete on raw (cost=0.00..719.63 rows=147 width=6)"
" Delete on raw"
" Delete on _hyper_1_1_chunk"
" Delete on _hyper_1_2_chunk"
...
" Delete on _hyper_1_22_chunk"
" -> Seq Scan on raw (cost=0.00..0.00 rows=1 width=6)"
" Filter: ((tm)::date = '2020-11-06'::date)"
" -> Custom Scan (CompressChunkDml) on _hyper_1_1_chunk (cost=0.00..27.40 rows=6 width=6)"
" -> Seq Scan on _hyper_1_1_chunk (cost=0.00..27.40 rows=6 width=6)"
" Filter: ((tm)::date = '2020-11-06'::date)"
" -> Custom Scan (CompressChunkDml) on _hyper_1_2_chunk (cost=0.00..27.40 rows=6 width=6)"
" -> Seq Scan on _hyper_1_2_chunk (cost=0.00..27.40 rows=6 width=6)"
" Filter: ((tm)::date = '2020-11-06'::date)"
...
" -> Custom Scan (CompressChunkDml) on _hyper_1_22_chunk (cost=0.00..27.40 rows=6 width=6)"
" -> Seq Scan on _hyper_1_22_chunk (cost=0.00..27.40 rows=6 width=6)"
" Filter: ((tm)::date = '2020-11-06'::date)"

Answer 1

Thank you for providing the explain. The explain shows that the DELETE statement is planned to touch all chunks of the hypertable and only in runtime the execution of the DELETE statement will realise that it is nothing to delete in many chunks:
EXPLAIN DELETE FROM raw WHERE tm::date = '2020-11-06'
Result:
"Delete on raw (cost=0.00..719.63 rows=147 width=6)"
" Delete on raw"
" Delete on _hyper_1_1_chunk"
" Delete on _hyper_1_2_chunk"
...
" Delete on _hyper_1_22_chunk"
" -> Seq Scan on raw (cost=0.00..0.00 rows=1 width=6)"
" Filter: ((tm)::date = '2020-11-06'::date)"
" -> Custom Scan (CompressChunkDml) on _hyper_1_1_chunk (cost=0.00..27.40 rows=6 width=6)"
" -> Seq Scan on _hyper_1_1_chunk (cost=0.00..27.40 rows=6 width=6)"
" Filter: ((tm)::date = '2020-11-06'::date)"
" -> Custom Scan (CompressChunkDml) on _hyper_1_2_chunk (cost=0.00..27.40 rows=6 width=6)"
" -> Seq Scan on _hyper_1_2_chunk (cost=0.00..27.40 rows=6 width=6)"
" Filter: ((tm)::date = '2020-11-06'::date)"
...
" -> Custom Scan (CompressChunkDml) on _hyper_1_22_chunk (cost=0.00..27.40 rows=6 width=6)"
" -> Seq Scan on _hyper_1_22_chunk (cost=0.00..27.40 rows=6 width=6)"
" Filter: ((tm)::date = '2020-11-06'::date)"
Since some chunks are compressed, TimescaleDB returns error on the deletes planned for compressed chunks.
The only way to not get the error is to have the selection condition to trigger chunk exclusion at the planning time. In the question the selection condition is tm::date = '2020-11-06', which first extracts date from the column tm and then compares with the constant. Thus the planner cannot decide if the chunk is filtered or not and instead push the filter down for runtime execution on every chunk.
To resolve this it is good to have a selection condition, which compares the time dimension column with constant or value, which can be calculated at planning time. Assuming tm is the time dimension column in the hypertable raw, I suggest to convert the constant date into timestamp, e.g., '2020-11-06'::timestamp and keep the column. You will need to specify the range of timestamps to cover all rows belonging to the targeted date.
For example, the DELETE statement can be:
DELETE FROM raw WHERE tm BETWEEN '2020-11-06 00:00' AND '2020-11-06 23:59'
Answers to questions:
show_chunks() does not correspond to what DELETE complains about
show_chunk statement and DELETE statement have different conditions and thus cannot be compared directly. show_chunk only shows chunks, which cover the time newer than the given constant. While DELETE is planned to check every chunk, thus it can complain on any chunk of the hypertable.
BTW this chunk is empty as far as I can see by looking at it in the pgAdmin. Any idea what's going on? Looks like a bug to me, but maybe I'm doing something wrong?
The compressed chunk stores data in a different internal chunk, thus no data can be seen in _hyper_1_1_chunk. TimescaleDB assumes that data are read through the hypertable, not directly from the chunks. Hypertable is an abstraction, which hides implementation details of TimescaleDB.

Answer 2

What specific version of 1.7? There was a bug on this, but should be fixed from 1.7.3 forward. https://github.com/timescale/timescaledb/pull/2092
If you’re on 1.7.3 or later and still seeing this, it’d be best to open an issue on the timescaledb GitHub repo.
You can check your version by connecting with psql and running \dx