Morning folks, could really use your help on this one.
I have a query from a table I need to send out as a CSV attachment on an email using oracle apex. that's the goal of all of this - happy to explore different options/sql to get the same result (through oracle apex)
Example query: select user, title, dept from user_db where dept = :DEPARTMENT
My query brings back a huge amount of data (intended)
Based on this, my idea was to make this query into a BLOB and save it to a different table (TBL_EMAIL_CONTENT),which I could then use the APEX_MAIL.ADD_ATTACHMENT function to bring into an email.
I took the query and created a CLOB, then used a separate function to change the CLOB to a BLOB
declare
CSV_CONTENT clob;
l_BLOB BLOB;
crlf varchar2(2) := chr(13) || chr(10);
CSVFILENAME varchar2(255);
begin
CSV_CONTENT := 'User,'||'Title,'||'Dept,'||crlf;
for c5 in (select user, title, dept from user_db where dept = :DEPARTMENT)
LOOP
CSV_CONTENT := CSV_CONTENT||c5.CONTENT_csv;
CSVFILENAME := 'Example.csv';
END LOOP;
L_BLOB := clob_to_blob(CSV_CONTENT);
for c6 in (select * from TBL_EMAILCONTENT where EMAIL_ID = 1)
LOOP
Update TBL_EMAILCONTENT
set
filename = CSVFILENAME||'.CSV',
CONTENT2 = csv_content,
CONTENT = L_BLOB;
END LOOP;
END;
The clob_to_blob function is as follows:
create or replace FUNCTION clob_to_blob(src_clob CLOB) RETURN BLOB IS
tgt_blob BLOB;
amount INTEGER := DBMS_LOB.lobmaxsize;
dest_offset INTEGER := 1;
src_offset INTEGER := 1;
blob_csid INTEGER := nls_charset_id('UTF8');
lang_context INTEGER := DBMS_LOB.default_lang_ctx;
warning INTEGER := 0;
begin
if src_clob is null then
return null;
end if;
DBMS_LOB.CreateTemporary(tgt_blob, true);
DBMS_LOB.ConvertToBlob(tgt_blob, src_clob, amount, dest_offset, src_offset, blob_csid, lang_context, warning);
return tgt_blob;
end clob_to_blob;
I would then send this out using :
APEX_MAIL.SEND(
p_to => 'UKSERVERPATCHING#VODAFONE.COM',
p_from => 'ukserverpatching#vodafone.com',
p_bcc => 'ukserverpatching#vodafone.com',
p_body => to_clob(' '),
p_body_html => l_body,
p_subj => l_subj);
for c8 in(
SELECT CONTENT,FILENAME,MIMETYPE
FROM TBL_EMAILCONTENT
where EMAIL_ID = 1)
loop
APEX_MAIL.ADD_ATTACHMENT(
p_mail_id => l_id,
p_attachment => c8.CONTENT,
p_filename => c8.FILENAME,
p_mime_type => c8.MIMETYPE);
end loop;
Unfortunately, this result makes a CSV that isnt readable with garbled characters in it. Other options ive tried online so far seem to run into problems where the query result is too long to be dealt with via a varchar value.
has anyone came across this before or can point me to an article online that looks to do this sort of thing?
Thanks in advance, really appreciated.
Related
Apex 20.1
I have an interactive report which has one clickable column that I used to download a file from Apex to the user's computer.
Originally I had the link call an application process which worked beautifully. However I found that larger files would take time to download, and there was no "spinning circle" to indicate that the user should keep waiting, or that anything was happening.
So I put the download code in an "After Submit" process with a "Request=" server-side condition.
The link from my interactive report now calls the process with a URL thus:
javascript:var submit =apex.submit({request:"FETCH_ATTACHMENT", showWait:true, set:{"P4_MESSAGE_ID":#MESSAGE_ID#, "P4_PART_ID":#PART_ID#, "P4_PART_NAME":"#PART_NAME#"}});
The "showWait" works beautifully now, I get a spinning circle, except now I have the opposite problem. After the download is finished, the page is still greyed-out with the spinning circle. The only way to bring it back is to refresh the page.
I would like the spinning circle to go away once the file is downloaded and control to be returned back to the page.
Here is the code, it's still in dev so excuse the debug untidiness!
DECLARE
vclob CLOB;
vclobtmp CLOB;
vblob BLOB;
vmimetype VARCHAR2(1000);
v_position INTEGER;
v_progress VARCHAR2(100);
v_error varchar2(1000);
FUNCTION Clobfromblob (p_blob BLOB)
RETURN CLOB
IS
l_clob CLOB;
l_dest_offsset INTEGER := 1;
l_src_offsset INTEGER := 1;
l_lang_context INTEGER := dbms_lob.default_lang_ctx;
l_warning INTEGER;
BEGIN
IF p_blob IS NULL THEN
RETURN NULL;
END IF;
dbms_lob.Createtemporary(lob_loc => l_clob, CACHE => FALSE);
dbms_lob.Converttoclob(dest_lob => l_clob, src_blob => p_blob,
amount => dbms_lob.lobmaxsize, dest_offset => l_dest_offsset,
src_offset => l_src_offsset, blob_csid => dbms_lob.default_csid,
lang_context => l_lang_context, warning => l_warning);
RETURN l_clob;
END;
BEGIN
--raise_application_error(-20000,'P4_MESSAGE_ID '||:P4_MESSAGE_ID||' P4_PART_ID '||:P4_PART_ID||' :P4_PART_NAME '||:P4_PART_NAME);
delete from apps.barry_debug;
insert into apps.barry_debug(time,debug) values (sysdate, 'start1'); commit;
dbms_lob.Createtemporary(lob_loc => vclob, CACHE => TRUE,
dur => dbms_lob.call);
dbms_lob.Createtemporary(lob_loc => vclobtmp, CACHE => TRUE,
dur => dbms_lob.call);
dbms_lob.Createtemporary(lob_loc => vblob, CACHE => TRUE,
dur => dbms_lob.call);
SELECT part_data,
Substr(part_type, 1, Instr(part_type, ';') - 1)
INTO vblob, vmimetype
FROM source_data
WHERE message_id = :P4_MESSAGE_ID --APEX_APPLICATION.g_x01
AND part_id = :P4_PART_ID --APEX_APPLICATION.g_x02
;
v_progress := 'blob to clob';
insert into apps.barry_debug(time,debug) values (sysdate, 'clob1'); commit;
vclobtmp := Clobfromblob(vblob);
--Strip the header from the attachment data
v_position := dbms_lob.Instr(vclobtmp, 'Content-Transfer-Encoding: base64',
1, 1
)
+ 33;
dbms_lob.Copy(vclob, vclobtmp, dbms_lob.Getlength(vclobtmp), 1, v_position);
v_progress := 'decodebase64';
insert into apps.barry_debug(time,debug) values (sysdate, 'decodebase'); commit;
vblob := apex_web_service.Clobbase642blob(vclob);
--vblob := apps.P_Base64.DecodeToBlob (vClob);
sys.htp.init;
v_progress := 'mime_header';
sys.owa_util.Mime_header(vmimetype /*'application/octet-stream'*/, FALSE
/*,'UTF-8' */);
sys.htp.P('Content-length: '
|| sys.dbms_lob.Getlength(vblob));
sys.htp.P('Content-Disposition: attachment; filename="'
||:P4_PART_NAME
||'"');
sys.owa_util.http_header_close;
v_progress := 'download_file';
insert into apps.barry_debug(time,debug) values (sysdate, 'download'); commit;
sys.wpg_docload.Download_file(vblob);
insert into apps.barry_debug(time,debug) values (sysdate, 'release1'); commit;
dbms_lob.Freetemporary (vblob); --do not forget!!
insert into apps.barry_debug(time,debug) values (sysdate, 'release2'); commit;
dbms_lob.Freetemporary (vclob); --do not forget!!
insert into apps.barry_debug(time,debug) values (sysdate, 'release3'); commit;
dbms_lob.Freetemporary (vclobtmp); --do not forget!!
apex_application.stop_apex_engine; --this is needed to download file in background. But it raises an exception below.
insert into apps.barry_debug(time,debug) values (sysdate, 'finished'); commit;
EXCEPTION
when apex_application.e_stop_apex_engine then
raise; -- raise again the stop Application Express engine exception
WHEN no_data_found THEN
NULL;
when others then
--raise_application_error(-20000,'Error-fetch-attachment-'||v_progress||' '||sqlerrm);
v_error := sqlerrm;
insert into apps.barry_debug(time,debug) values (sysdate, 'Error-fetch-attachment-'||v_progress||'*'||v_error);
END;
The Debugging table shows that processing ends after the engine is stopped. (If I take this out nothing downloads at all).
P.S I have tried to ask the question on Oracle community help and got one answer which was to
paste this code into the Function and Global Variable Declarations. No difference.
$(document).ajaxComplete(function(){
var overlay$ = $("#apex_wait_overlay"),
processing$ = $(".u-Processing");
if(overlay$.length || processing$.length){
overlay$.remove();
processing$.remove();
}
});
I have one question, I want to have one page view statistics and send them through REST.
here is my request:
select count (*)
from apex_activity_log l
where flow_id = 100
and time_stamp> = sysdate - (1/24/60/60 * 2419200)
and userid is not null
and step_id = 172
;
answer : 867
This query works great in "SQL Commands". But when I use this query in Packages I get 0. Although the answer should be completely different. How do I give Packages access to apex_activity_log. That the correct answer came back to me. Thanks)
My api in Packages :
PROCEDURE post_connect_service (
p_status OUT NUMBER,
p_blob IN BLOB
) IS
v_blob blob := p_blob;
v_clob CLOB;
tv apex_json.t_values;
v_id varchar2(1000);
v_number varchar2(1000);
v_date_last date;
v_count_v_pl int;
v_count_sum_v int;
BEGIN
v_clob := iot_general.blob_to_clob(v_blob);
apex_json.parse(tv,v_clob);
v_id := apex_json.get_varchar2(p_path => 'id', p_values => tv);
select count(*) into v_count_sum_v
from apex_activity_log;
p_status := 200;
apex_json.open_object;
apex_json.write('success', true);
apex_json.write('count_views', v_count_v_pl);
apex_json.write('count_sum_views', v_count_sum_v);
apex_json.close_object;
EXCEPTION
WHEN OTHERS THEN
p_status := 500;
apex_json.open_object;
apex_json.write('success', false);
apex_json.write('message', substr(sqlerrm,1,4000));
apex_json.close_object;
END post_connect_service;
The view apex_activity_log has data for the current apex context. If it is queried outside of an apex context, it will return no rows. Easiest way is to create an apex session before querying it.
koen>SELECT COUNT(*) FROM apex_activity_log;
COUNT(*)
___________
0
koen>DECLARE
2 BEGIN
3 apex_session.create_session (
4 p_app_id => 286,
5 p_page_id => 1,
6 p_username => 'JOHN.DOE');
7 END;
8* /
PL/SQL procedure successfully completed.
koen>SELECT COUNT(*) FROM apex_activity_log;
COUNT(*)
___________
9327
koen>
first, there is a pre-configured REST API for this within ORDS.
https://docs.oracle.com/en/database/oracle/oracle-database/21/dbrst/api-oracle-apex.html
To answer your actual question:
Does your procedure run within the same workspace as application 100 is? Or are you dependent on the APEX_ADMINISTRATOR_ROLE for this to return rows?
if the latter, make sure to create your package or procedure with AUTHID CURRENT_USER to make sure that it ...
a) runs with the privieges of the invoking user
b) have roles (such as the APEX_ADMINISTRATOR_ROLE) enabled during PL/SQL execution.
I'm trying to optimize my Glue/PySpark job by using push down predicates.
start = date(2019, 2, 13)
end = date(2019, 2, 27)
print(">>> Generate data frame for ", start, " to ", end, "... ")
relaventDatesDf = spark.createDataFrame([
Row(start=start, stop=end)
])
relaventDatesDf.createOrReplaceTempView("relaventDates")
relaventDatesDf = spark.sql("SELECT explode(generate_date_series(start, stop)) AS querydatetime FROM relaventDates")
relaventDatesDf.createOrReplaceTempView("relaventDates")
print("===LOG:Dates===")
relaventDatesDf.show()
flightsGDF = glueContext.create_dynamic_frame.from_catalog(database = "xxx", table_name = "flights", transformation_ctx="flights", push_down_predicate="""
querydatetime BETWEEN '%s' AND '%s'
AND querydestinationplace IN (%s)
""" % (start.strftime("%Y-%m-%d"), today.strftime("%Y-%m-%d"), ",".join(map(lambda s: str(s), arr))))
However it appears, that Glue still attempts to read data outside the specified date range?
INFO S3NativeFileSystem: Opening 's3://.../flights/querydestinationplace=12191/querydatetime=2019-03-01/part-00045-6cdebbb1-562c-43fa-915d-93b125aeee61.c000.snappy.parquet' for reading
INFO FileScanRDD: Reading File path: s3://.../flights/querydestinationplace=12191/querydatetime=2019-03-10/part-00021-34a13146-8fb2-43de-9df2-d8925cbe472d.c000.snappy.parquet, range: 0-11797922, partition values: [12191,17965]
WARN S3AbortableInputStream: Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
INFO S3NativeFileSystem: Opening 's3://.../flights/querydestinationplace=12191/querydatetime=2019-03-10/part-00021-34a13146-8fb2-43de-9df2-d8925cbe472d.c000.snappy.parquet' for reading
WARN S3AbortableInputStream: Not all bytes were read from the S3ObjectInputStream, aborting HTTP connection. This is likely an error and may result in sub-optimal behavior. Request only the bytes you need via a ranged GET or drain the input stream after use.
Notice the querydatetime=2019-03-01 and querydatetime=2019-03-10 its outside the specified range of 2019-02-13 - 2019-02-27. Is that why there's the next line "aborting HTTP connection" tho? It goes on to say "This is likely an error and may result in sub-optimal behavior" is something wrong?
I wonder if the problem is because it does not support BETWEEN inside the predicate or IN?
The table create DDL
CREATE EXTERNAL TABLE `flights`(
`id` string,
`querytaskid` string,
`queryoriginplace` string,
`queryoutbounddate` string,
`queryinbounddate` string,
`querycabinclass` string,
`querycurrency` string,
`agent` string,
`quoteageinminutes` string,
`price` string,
`outboundlegid` string,
`inboundlegid` string,
`outdeparture` string,
`outarrival` string,
`outduration` string,
`outjourneymode` string,
`outstops` string,
`outcarriers` string,
`outoperatingcarriers` string,
`numberoutstops` string,
`numberoutcarriers` string,
`numberoutoperatingcarriers` string,
`indeparture` string,
`inarrival` string,
`induration` string,
`injourneymode` string,
`instops` string,
`incarriers` string,
`inoperatingcarriers` string,
`numberinstops` string,
`numberincarriers` string,
`numberinoperatingcarriers` string)
PARTITIONED BY (
`querydestinationplace` string,
`querydatetime` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION
's3://pinfare-glue/flights/'
TBLPROPERTIES (
'CrawlerSchemaDeserializerVersion'='1.0',
'CrawlerSchemaSerializerVersion'='1.0',
'UPDATED_BY_CRAWLER'='pinfare-parquet',
'averageRecordSize'='19',
'classification'='parquet',
'compressionType'='none',
'objectCount'='623609',
'recordCount'='4368434222',
'sizeKey'='86509997099',
'typeOfData'='file')
One of the issue I can see with the code is that you are using "today" instead of "end" in the between clause. Though I don't see the today variable declared anywhere in your code, I am assuming it has been initialized with today's date.
In that case the range will be different and the partitions being read by glue spark is correct.
In order to push down your condition, you need to change the order of columns in your partition by clause of table definition
A condition having "in" predicate on first partition column can not be push down as you are expecting.
Let me if it helps.
Pushdown predicates in Glue DynamicFrame works fine with between as well as IN clause.
As long as you have correct sequence of partition columns defined in table definition and in query.
I have table with three level of partitions.
s3://bucket/flights/year=2018/month=01/day=01 -> 50 records
s3://bucket/flights/year=2018/month=02/day=02 -> 40 records
s3://bucket/flights/year=2018/month=03/day=03 -> 30 records
Read data in dynamicFrame
ds = glueContext.create_dynamic_frame.from_catalog(
database = "abc",table_name = "pqr", transformation_ctx = "flights",
push_down_predicate = "(year == '2018' and month between '02' and '03' and day in ('03'))"
)
ds.count()
Output:
30 records
So, you are gonna get the correct results, if sequence of columns is correctly specified. Also note, you need to specify '(quote) IN('%s') in IN clause.
Partition columns in table:
querydestinationplace string,
querydatetime string
Data read in DynamicFrame:
flightsGDF = glueContext.create_dynamic_frame.from_catalog(database = "xxx", table_name = "flights", transformation_ctx="flights",
push_down_predicate=
"""querydestinationplace IN ('%s') AND
querydatetime BETWEEN '%s' AND '%s'
"""
%
( ",".join(map(lambda s: str(s), arr)),
start.strftime("%Y-%m-%d"), today.strftime("%Y-%m-%d")))
Try to do the end as this
start = str(date(2019, 2, 13))
end = str(date(2019, 2, 27))
# Set your push_down_predicate variable
pd_predicate = "querydatetime >= '" + start + "' and querydatetime < '" + end + "'"
#pd_predicate = "querydatetime between '" + start + "' AND '" + end + "'" # Or this one?
flightsGDF = glueContext.create_dynamic_frame.from_catalog(
database = "xxx"
, table_name = "flights"
, transformation_ctx="flights"
, push_down_predicate=pd_predicate)
The pd_predicate will be a string that will work as a push_down_predicate.
Here is a nice read about it if you like.
https://aws.amazon.com/blogs/big-data/work-with-partitioned-data-in-aws-glue/
I am looking to speed up the following PL/SQL function. Right now it has run for over 2 hours with no sign of completing. We aborted that one and attempting it again with a EXIT WHEN of 20 and it still shows no signs of actually completing.
We are running these through SQLDeveloper 17.3, and each of the (4) tables has about 15k rows.
The goal is to grab all of the SSN's in our database and change the first character to an illegal char and the last 2 characters to a random A-Z combination. We then have to update that SSN in every table that uses it (4).
declare
v_random varchar2(2);
v_origin_ssn varchar2(100);
v_working_start varchar2(100);
v_working_middle varchar2(100);
v_new_ssn varchar2(100);
begin
for o in (
select distinct ssn --loop all rows in tbl_customer
from program_one.tbl_customer
)
loop
if regexp_like(o.ssn, '^[A-Za-z9].*[A-Z]$') then continue; --if this is already scrambled, skip
else
select dbms_random.string('U', 2) --create random 2 cap letters
into v_random
from dual;
v_origin_ssn := o.ssn; --set origin ssn with the existing ssn
if regexp_like(o.ssn, '^[A-Za-z]') --if first char is already alpha, leave it alone, otherwise 9
then v_working_start := substr(o.ssn, 1, 1);
else v_working_start := 9;
end if;
v_working_middle := substr(o.ssn, 2, 6); --set middle ssn with the unchanged numbers
v_new_ssn := v_working_start||v_working_middle||v_random; --create new sanitized ssn
update program_one.tbl_customer --update if exists in tbl_customer
set ssn = v_new_ssn
where ssn = v_origin_ssn;
commit;
update program_one.tbl_mhc_backup --update if exists ssn tbl_mhc_backup
set ssn = v_new_ssn
where ssn = v_origin_ssn;
commit;
update program_two.tbl_waiver --update if exists ssn tbl_waiver
set ssn = v_new_ssn
where ssn = v_origin_ssn;
commit;
update program_two.tbl_pers --update if exists in tbl_pers
set ssan = v_new_ssn
where ssan = v_origin_ssn;
commit;
end if;
--dbms_output.put_line(v_origin_ssn||' : '||v_new_ssn); --output test string to verify working correctly
end loop;
end;
I'd do it without a function in plain SQL:
Create a table with old and new ssn:
CREATE TABLE tmp_ssn AS
SELECT ssn, '9'||substr(ssn,2,6)||dbms_random.string('U',2) as new_ssn
FROM (SELECT distinct ssn FROM program_one.tbl_customer);
CREATE UNIQUE INDEX ui_tmp_ssn ON tmp_ssn(ssn, new_ssn);
EXEC DBMS_STATS.GATHER_TABLE_STATS(null,'tmp_ssn');
... and then update the tables one by one:
MERGE INTO program_one.tbl_customer z USING tmp_ssn q ON (z.ssn=q.ssn)
WHEN MATCHED THEN UPDATE z.ssn = q.new_ssn;
COMMIT;
MERGE INTO program_one.tbl_mhc_backup z USING tmp_ssn q ON (z.ssn=q.ssn)
WHEN MATCHED THEN UPDATE z.ssn = q.new_ssn;
COMMIT;
etc
If that is still to slow, I'd do
RENAME tbl_customer to tbl_customer_old;
CREATE TABLE tbl_customer as
SELECT s.new_ssn as ssn, t.col1, t.col2, ... , t.coln
FROM tbl_customer_old t JOIN tmp_ssn s USING(ssn);
DROP TABLE tbl_customer_old;
Hie all,
I'm trying to get eventlog entries using WMI and WQL.
I can get the right log with the right sourcename of itand so on, but i can make a select query to only get result for the 5 or 10 past minutes.
here is my query:
Here are a few snippets from a script of mine:
Dim dtmStart, dtmEnd, sDate, ...
I actually had an array of dates and I was looking for logon/off/unlock events for the entire day. So I built my complete start and end dates from that.
I won't put in the day month and year but, you could just define it, e.g. sDate = 10100608.
dtmStart = sDate + "000000.000000-420" '0hr on the date in question.
dtmEnd = sDate + "235900.000000-420" ' 23:59 on the date in question
(Note that the UTC offset is in minutes here -420 day light savings time North America.)
Set colEvents = oWMIService.ExecQuery _
("SELECT * FROM Win32_NTLogEvent WHERE Logfile = 'Security' AND " _
& "TimeWritten >= '" & dtmStart & "' AND TimeWritten < '" _
& dtmEnd & "' AND " _
& "(EventCode = '528' OR EventCode = '540' OR EventCode = '538')")
' Query for events during the time range we're looking for.
Mike,
Show me your query. Usually the time format is something like this
20100608100000.000000-300
see this for more details about DateTime formatting for WQL