My serverless redshift has thousands of running queries. I think it is stuck. I'm not sure how to cancel all queries and start fresh. Is there a way to do it?
Since STV_RECENTS is not available in serverless endpoint, you can get all the running queries with
SELECT * FROM SYS_QUERY_HISTORY WHERE status = 'running'
, the session_id is the PID
Then you can cancel them with
CANCEL <session_id>
From CANCEL - Amazon Redshift:
Cancels a database query that is currently running.
CANCEL process_id [ 'message' ]
The CANCEL command requires the process ID of the running query and displays a confirmation message to verify that the query was cancelled.
To cancel a currently running query, first retrieve the process ID for the query that you want to cancel. To determine the process IDs for all currently running queries, type the following command:
select pid, starttime, duration,
trim(user_name) as user,
trim (query) as querytxt
from stv_recents
where status = 'Running';
See also:
PG_TERMINATE_BACKEND - Amazon Redshift that terminates a session
Terminate (kill) specific session on a server - Amazon Redshift Data Dictionary Queries
Related
Some jobs are remaining with pending pending state and I can't cancel them.
How do I cancel the job.
Web console shows like this.
"The graph is still being analyzed."
All logs are "No entries found matching current filter."
Job status: "Starting..."
There isn't appered a cancel button yet.
There are no instances in the Compute Engline tab.
What I did is below.
I created a streaming job. it was simple template job, Pubsub subscription to BigQuery. I set machineType as e2-micro because it was just a testing.
I also tried to drain and cancel by gcloud but it doesn't work.
$ gcloud dataflow jobs drain --region asia-northeast1 JOBID
Failed to drain job [...]: (...): Workflow modification failed. Causes: (...):
Operation drain not allowed for JOBID.
Job is not yet ready for draining. Please retry in a few minutes.
Please ensure you have permission to access the job and the `--region` flag, asia-northeast1, matches the job's
region.
This is jobs list
$ gcloud dataflow jobs list --region asia-northeast1
JOB_ID NAME TYPE CREATION_TIME STATE REGION
JOBID1 pubsub-to-bigquery-udf4 Streaming 2021-02-09 04:24:23 Pending asia-northeast1
JOBID2 pubsub-to-bigquery-udf2 Streaming 2021-02-09 03:20:35 Pending asia-northeast1
...other jobs...
Please let me know how to stop/cancel/delete these streaming jobs.
Job IDs:
2021-02-08_20_24_22-11667100055733179687
2021-02-08_20_24_22-11667100055733179687
WebUI:
https://i.stack.imgur.com/B75OX.png
https://i.stack.imgur.com/LzUGQ.png
As per personal experience some time few instance get stuck either they keep on running, or they cannot be canceled or you can not see thr graphical data flow pipelines. Best way to handle this kind of issue is to leave them in thr status, unless it is not impacting your solution by exceeding maximum concurrent runs at a moment. It will be canceled automatically or by Google team, since Dataflow is a google managed.
In GCP console Dataflow UI, if you have running Dataflow jobs, you will see the "STOP" button just like the below image.
Press the STOP button.
When you successfully stop your job, you will see the status like below. (I was too slow to stop the job with the first try, so I had to test it again. :) )
I created a new python google function that schedule a Query in BigQuery every 10 minutes, I test it and it works.
deployment works fine
testing give this error : Error: memory limit exceeded. logs not available ( but I can see that the Query did run as expected in BigQuery)
using http trigger in cloud scheduler, I got failure with this the error message status: 503, but again, I can see in BigQuery console, it is running as expected
edit : here the code for the function
from google.cloud import bigquery
def load(request):
client = bigquery.Client()
dataset_id = 'yyyyyyyy'
table_id='xxxxxxxxxxx'
job_config = bigquery.QueryJobConfig()
job_config.use_legacy_sql = False
table_ref = client.dataset(dataset_id).table(table_id)
job_config.destination = table_ref
job_config.write_disposition = bigquery.WriteDisposition.WRITE_TRUNCATE
sql = """
SELECT * FROM `xxxxxx.datastudio.today_view`;
"""
query_job = client.query(sql,location='asia-northeast1',job_config=job_config)
query_job.result()
print("Job finished.")
The BigQuery job is asynchronous. Your Cloud Function trigger it and wait up to completion. If the function fail between, it's not a problem, the 2 services aren't correlated.
If you do this by API, when you create a job (a query) you got immediately a JobID. Then you have to poll regularly this job ID to know its status. The client library do exactly the same!
Your out of memory issue come from the result which wait the completion and read the results. Set a page size or a max_result to limit the data returned.
But, you can also don't wait the end and exit immediately (skip the line query_job.result()). You will save Cloud Functions processing (useless wait) time, and thus money!
I am executing a external job using DBMS_SCHEDULER through apex page by clicking a button in below manner.(Dynamic action=>Execute PlSql)
dbms_scheduler.run_job(job_name => 'APEXDATA.myJobName', use_current_session=> TRUE);
Its executing the external job correctly.(taking 1-2 minutes).My issue is that, in between the time while its executing i can not able to access any other page or can not able to login with new session nothing.showing below error in every task i am performing.
**503 Service Unavailable
The connection pool named: |apex|| is not correctly configured, due to the following error(s):
Exception occurred while getting connection: oracle.ucp.UniversalConnectionPoolException:
All connections in the Universal Connection Pool are in use**
Is this the general or known issue?if yes how to resolve the issue,because in same time other user also has to perform any other task or other may login same time.
Thank You.
I think you're mixing 2 things that hard to combine:
Dynamic actions are designed to submit code from the page without a page submit so the user can continue to work on the page after he has done something (eg run pl/sql code)
Running a process in the database that takes up the database session until it is completed ( use_current_session=> TRUE). Your dbms_scheduler.run_job process will run in the current session and as long as that job is running no other operations can be run in that database session (the connection is in use as shown in the error message).
Solutions:
use_current_session=> FALSE so the job runs in the background
In the dynamic action, set "Wait for result" to true, so the user is forced to wait until the job completes.
Execute the job on page submit which will also force the user to wait for the job to be completed.
Since your job takes 1-2 mins to complete, options 2 and 3 are probably not feasible because the user experience is not optimal. If you execute the job in the background, then you probably need to write some additional code to prevent the user from clicking a couple of times and submitting the job multiple times. You could do that by checking if the job is running before you submit it and not submit it if it is currently running.
In a long-running management command I'd like to have two connections to the same DB. One connection will hold a transaction to lock a certain row (select for update) and the other connection will record some processing info. If the process crashes, a new run of the management command can use the processing info to skip/simplify some processing steps, hence the need to record it in a different connection.
How do I go about creating a 2nd connection in the same thread? My first thought was to add a default2 entry to DATABASES with the same connection info as default and use .using("default2") in one of the queries, but wasn't sure if that would cause issues in Django
If I submit a series of SQL statements (each with GO in sqlcmd) that I want to make an reasonable attempt to run on an Azure SQL Data Warehouse, I've found in sqlcmd how to ignore errors. But I've seen if I want to abort a statement in that sequence of statements with:
KILL "SIDxxxxxxx";
The whole session ends:
Msg 111202, Level 16, State 1, Server adws_database, Line 1
111202;Query QIDyyyyyyyyyy has been cancelled.
Is there a way to not end a query session in Azure SQL Data Warehouse? Similar to how postgres's
pg_cancel_backend()
works?
In postgres the
pg_terminate_backed(<pid>)
seems to be working similarly to the ADW
KILL 'SIDxxxx'
command.
Yes, a client can cancel a running request without aborting the whole session. In SSMS this is what the red square does during query execution.
Sqlcmd doesn't expose any way to cancel a running request, though. Other client interfaces do, like the .NET SqlClient you can use SqlCommand.Cancel()
David