I am using Opencart.
10M Product SQL Table
SQL responsible very slowly.
My server info;
48 CPU
64GB Memory
PHP 7.3
Mariadb 10.3
Nginx + PHP-FPM
SELECT p.product_id, (SELECT AVG(rating) AS total FROM oc_review r1 WHERE r1.product_id = p.product_id AND r1.status = '1' GROUP BY r1.product_id) AS rating, (SELECT price FROM oc_product_discount pd2 WHERE pd2.product_id = p.product_id AND pd2.customer_group_id = '1' AND pd2.quantity = '1' AND ((pd2.date_start = '0000-00-00' OR pd2.date_start < NOW()) AND (pd2.date_end = '0000-00-00' OR pd2.date_end > NOW())) ORDER BY pd2.priority ASC, pd2.price ASC LIMIT 1) AS discount, (SELECT price FROM oc_product_special ps WHERE ps.product_id = p.product_id AND ps.customer_group_id = '1' AND ((ps.date_start = '0000-00-00' OR ps.date_start < NOW()) AND (ps.date_end = '0000-00-00' OR ps.date_end > NOW())) ORDER BY ps.priority ASC, ps.price ASC LIMIT 1) AS special FROM oc_product_to_category p2c LEFT JOIN oc_product p ON (p2c.product_id = p.product_id) LEFT JOIN oc_product_description pd ON (p.product_id = pd.product_id) LEFT JOIN oc_product_to_store p2s ON (p.product_id = p2s.product_id) WHERE pd.language_id = '1' AND p.status = '1' AND p.date_available <= NOW() AND p2s.store_id = '0' AND p2c.category_id = '18' GROUP BY p.product_id ORDER BY p.sort_order ASC, LCASE(pd.name) ASC LIMIT 0,48
Related
This Redshift query fails -
DELETE FROM TBL_1 stg
WHERE EXISTS (
WITH CCDA as (
SELECT
row_number() OVER (PARTITION BY emp_id,customer_id ORDER BY seq_num desc) rn
, *
FROM TBL_2
WHERE end_dt > (SELECT max(end_dt) FROM TBL_3)
)
SELECT emp_id,customer_id FROM CCDA WHERE rn = 1
AND stg.emp_id = CCDA.emp_id
AND stg.customer_id = CCDA.customer_id
);
Error: Invalid operation: syntax error at or near "stg"
However, the below query runs fine -
SELECT * FROM TBL_1 stg
WHERE EXISTS (
WITH CCDA as (
SELECT
row_number() OVER (PARTITION BY emp_id,customer_id ORDER BY seq_num desc) rn
, *
FROM TBL_2
WHERE end_dt > (SELECT max(end_dt) FROM TBL_3)
)
SELECT emp_id,customer_id FROM CCDA WHERE rn = 1
AND stg.emp_id = CCDA.emp_id
AND stg.customer_id = CCDA.customer_id
);
Am I missing something?
You cannot use an alias in a DELETE statement for the target table. "stg" cannot be used as the alias and this is why you are getting this error.
Also to reference other tables in a DELETE statement you need to use the USING clause.
See: https://docs.aws.amazon.com/redshift/latest/dg/r_DELETE.html
A quick stab of what this would look like (untested):
WITH CCDA as (
SELECT
row_number() OVER (PARTITION BY emp_id,customer_id ORDER BY seq_num desc) rn
, *
FROM TBL_2
WHERE end_dt > (SELECT max(end_dt) FROM TBL_3)
)
DELETE FROM TBL_1
USING CCDA
WHERE CCDA.rn = 1
AND TBL_1.emp_id = CCDA.emp_id
AND TBL_1.customer_id = CCDA.customer_id
;
What are some way I can debug through a sparksql query?
I have defined a dataframe with a sparpksql query and I have included show(1), but the query continues to run very long. Could anyone provide pointers? Thank you!
select t1.id
from table1 t1
join table2 t2 on t1.id = t2.cd
where t1.product = 'I'
and not exists
(select * from table3 t3
where t3.id = t1.id
and t3.year = t1.year
and a3.month = t1.month
and t3.day = t1.day
and t3.code in ('4321','5604'))
and not exists
(select MAX(status) from table4 t4
where t4.id = t1.id
and t4.year = t1.year
and t4.month = t1.month
and t4.day = t1.day
having max(status) > 3)
""").show(1)
list_of_id_df.createOrReplaceTempView("list_of_id")
list_of_id_df.show(1)
print("Done")
I know the sequence name but I don't know the table name and column name of it.
So please help me how to find it.
Slight improvisation using #a_horse_with_no_name's answer. The below query will also give the schema name and sequence name. So, this gives you the data with schema,table,column and sequence names as single result
select ts.nspname as object_schema,
tbl.relname as table_name,
col.attname as column_name,
s.relname as sequence_name
from pg_class s
join pg_namespace sn on sn.oid = s.relnamespace
join pg_depend d on d.refobjid = s.oid and d.refclassid='pg_class'::regclass
join pg_attrdef ad on ad.oid = d.objid and d.classid = 'pg_attrdef'::regclass
join pg_attribute col on col.attrelid = ad.adrelid and col.attnum = ad.adnum
join pg_class tbl on tbl.oid = ad.adrelid
join pg_namespace ts on ts.oid = tbl.relnamespace
where s.relkind = 'S'
-- and s.relname = 'your_sequence_name_here'
and d.deptype in ('a', 'n');
Assuming it's a sequence that is owned by a column e.g. because the column is defined as serial or identity then you can get that information by looking at pg_depend and join that to pg_class (and others)
select tbl.relname as table_name,
col.attname as column_name
from pg_class s
join pg_namespace sn on sn.oid = s.relnamespace
join pg_depend d on d.refobjid = s.oid and d.refclassid='pg_class'::regclass
join pg_attrdef ad on ad.oid = d.objid and d.classid = 'pg_attrdef'::regclass
join pg_attribute col on col.attrelid = ad.adrelid and col.attnum = ad.adnum
join pg_class tbl on tbl.oid = ad.adrelid
join pg_namespace ts on ts.oid = tbl.relnamespace
where s.relkind = 'S'
and s.relname = 'your_sequence_name_her'
and d.deptype in ('a', 'n');
Another option is to look at the default value of the columns and check if it contains the sequence name:
select tbl.relname as table_name,
col.attname as column_name
from pg_attrdef ad
join pg_attribute col on col.attrelid = ad.adrelid and col.attnum = ad.adnum
join pg_class tbl on tbl.oid = ad.adrelid
where pg_get_expr(ad.adbin, ad.adrelid) like '%your_sequence_name_here%'
This would also work for sequences that are not owned by a column.
Looking at the SQL queries run when user.has_perms(perms) is called, I see:
SELECT "auth_permission"."id",
"auth_permission"."name",
"auth_permission"."content_type_id",
"auth_permission"."codename",
"django_content_type"."id",
"django_content_type"."name",
"django_content_type"."app_label",
"django_content_type"."model"
FROM "auth_permission"
inner join "auth_user_user_permissions"
ON ( "auth_permission"."id" =
"auth_user_user_permissions"."permission_id" )
inner join "django_content_type"
ON ( "auth_permission"."content_type_id" =
"django_content_type"."id" )
WHERE "auth_user_user_permissions"."user_id" = %s
ORDER BY "django_content_type"."app_label" ASC,
"django_content_type"."model" ASC,
"auth_permission"."codename" ASC
and:
SELECT "django_content_type"."app_label",
"auth_permission"."codename"
FROM "auth_permission"
inner join "auth_group_permissions"
ON ( "auth_permission"."id" =
"auth_group_permissions"."permission_id" )
inner join "auth_group"
ON ( "auth_group_permissions"."group_id" = "auth_group"."id" )
inner join "auth_user_groups"
ON ( "auth_group"."id" = "auth_user_groups"."group_id" )
left outer join "django_content_type"
ON ( "auth_permission"."content_type_id" =
"django_content_type"."id" )
WHERE "auth_user_groups"."user_id" = %s
my questions are:
What exactly are these queries doing?
Why are they run on
every request? Is there some way to cache these results?
How do I do this in doctrine2 QB or DQL.
SELECT * FROM
(
select * from my_table order by timestamp desc
) as my_table_tmp
group by catid
order by nid desc
I think your query is the same as:
SELECT *
FROM my_table
GROUP BY catid
HAVING timestamp = MAX(timestamp)
ORDER BY nid DESC
;
If it is correct, then you should be able to do:
$qb->select('e')
->from('My\Entities\Table', 'e')
->groupBy('e.catid')
->having('e.timestamp = MAX(e.timestamp)')
->orderBy('nid', 'DESC')
;
Or, directly using DQL:
SELECT e
FROM My\Entities\Table e
GROUP BY e.catid
HAVING e.timestamp = MAX(e.timestamp)
ORDER BY e.nid DESC
;
Hope this helps and works! ;)