Very slow ingestion to QuestDB when using Postgres wire protocol - questdb

Question

I have a problem with ingestion time when inserting rows into QuestDB table.
Table definition:
create table trade1 (
id symbol,
buy_order_id string,
currency string,
price float,
quantity float,
instrument_id symbol,
sell_order_id string,
status string,
subtype symbol,
"type" string,
transact_time timestamp,
buy_trader_id string,
sell_trader_id string);
) timestamp(transact_time) PARTITION BY DAY;
I have an ETL process which extracts data from CSV files and inserts data using JDBC Postgrs driver.
When I insert data on empty table from the first file - it takes ~60s for ~300k rows.
However for the second file it takes significantly longer - 180s.
Forth file is over 10 minutes.
All files are similar in number of rows.
Also when I keep only one symbol column it seems to be faster but speed is decreasing as more rows are inserted:
create table trade1 (
id string,
buy_order_id string,
currency string,
price float,
quantity float,
instrument_id symbol,
sell_order_id string,
status string,
subtype string,
"type" string,
transact_time timestamp,
buy_trader_id string,
sell_trader_id string);
) timestamp(transact_time) PARTITION BY DAY;
Insert time: 15s, 19s, 29s, 37s, 35s, 59s, 62s, 74s so it's continously growing.
It seems that ingestion time grows together with number of rows inserted but how is that possible when there is not even index defined?
server.conf:
data:
server.conf: |
cairo.sql.append.page.size = 256
pg.worker.affinity = 1,2,3,4
pg.worker.count = 4
shared.worker.count = 2
QuestDB is deployed on Kubernetes using Helm chart.
Am I missing some core concept?

Answer 1

Does it still exhibit the slowness if you change instrument_id from SYMBOL to STRING?
If you can ingest your source CSV "as is", the REST import is ridiculously fast.
In my benchmarks, using the PostgreSQL wire protocol was the slowest (taking 2-3x longer) compared to REST /imp and InfluxDB Line Protocol.