i am using the following code to handle the insets: but it is taking too much time, how can i enhance it or make in work faster:
<cfspreadsheet action="read" excludeheaderrow="yes" headerrow="1" rows="2-65536" query="mySheet" src="#dest#\#newfile#" sheet="1">
<cftry>
<cfquery datasource="#request.dsn#" name="myInsert">
<cfoutput>
<cfloop query="mySheet">
INSERT INTO mytable(memberID,lastname,firstname,pid,pname,gender,dob,workphone,homephone,address1,address2,city,county,state,zip,marketValue)
values('#mySheet.mem_id##RandRange(1,100)#','#mySheet.MEM_LAST_NAME#','#mySheet.MEM_FIRST_NAME#','#mySheet.PCP_ID#','#mySheet.pcp_name#','#mySheet.gender#','#mySheet.dob#','#mySheet.WORKTELEPHONENUMBER#','#mySheet.HOMETELEPHONENUMBER#','#mySheet.ADDRESSLINE1#','#mySheet.ADDRESSLINE2#','#mySheet.CITY#','#mySheet.COUNTY#','#mySheet.state#','#mySheet.zip#','#mySheet.GROUPNAME#')
</cfloop>
</cfoutput>
</cfquery>
records in excel sheet are more than 50000
Here's one way to handle it:
Put the entire thing (cfspreadsheet and all inserts) into a threaded function so it can run in the background
Save the spreadsheet query (upload results) into a session variable
After the upload is complete, do your insert queries in batches; in each batch add any issues into a feedback array that is also in the session scope
On the user's end, have a page that tells the user that things are "Working..." and re-loads and displays the feedback issues as they come up (best to do with ajax)
Related
I am moving one of our applications from ColdFusion 9.01 to ColdFusion 11 and encountered a situation where I cannot get the date formatted the way I want it using "DateFormat". I read through the docs since things have changed in CF versions, but I honestly can't figure out why this isn't working. It worked beautifully in CF 9. I know it's probably something very easy, but I am just not seeing it.
The query (Oracle DB) provides me a list of the last 30 days and the loop is simply to reformat the date output from "2014-07-01 00:00:00.0" to a more friendly looking display of 01-Jul-2014 except that I cannot get it to format as "dd-mmm-yyyy" it just spits back the original output from the query. I hard coded the date where normally there would be a cfquerparam. Any ideas?
<cfquery name="qryDateArray" datasource="#request.db#">
select trunc(to_date('07/01/2014', 'mm/dd/yyyy') + 1 - rownum) as ref_date
from dual connect by rownum <= 30
</cfquery>
<cfloop from="1" to="#qryDateArray.recordcount#" index="j">
<cfset qryDateArray.ref_date[j] = DateFormat(qryDateArray.ref_date[j], "dd-mmm-yyyy")>
</cfloop>
<cfoutput>
<cfdump var="#qryDateArray#">
</cfoutput>
I could not test this on CF11 since I do not have it handy. I did verify that your code though returns results as you explained when I ran it on my CF10 environment here. So what you can do is add a column to the query object and define it as a varchar and add your formatted data to that. This in turn dumped out the formatted dates.
<cfquery name="qryDateArray" datasource="#request.db#">
select trunc(to_date('07/01/2014', 'mm/dd/yyyy') + 1 - rownum) as ref_date
from dual connect by rownum <= 30
</cfquery>
<cfset aryData = [] />
<cfloop from="1" to="#qryDateArray.recordcount#" index="j">
<cfset ArrayAppend(aryData, DateFormat(qryDateArray.ref_date[j], "dd-mmm-yyyy")) />
</cfloop>
<cfset QueryAddColumn(qryDateArray, "STRDATE", "VarChar", aryData) />
<cfoutput>
<cfdump var="#qryDateArray#">
</cfoutput>
If dependent on the query column names then could use something like Ben's method explained here to do some renaming of the columns: http://www.bennadel.com/blog/357-ask-ben-changing-coldfusion-query-column-names.htm
It'd be great if you'd given us a portable test case rather than one that relies on your database, but I suspect it is because ColdFusion has become more rigid with its type management of query columns.
So CF considers your ref_date column to be of type date, so when you try to put the formatted string back into the query column, CF tries (and succeeds) to convert the string back into a date.
Aside:
I have to wonder why you don't format the data string in the DB from the outset, and just return it the way you need it, rather than returning something else, then looping over the thing to adjust it..?
I'm using <cfspreadsheet> to output a database query to an Excel spreadsheet. When these spreadsheets are created manually it is possible to highlight all the cells and 'Format as table'. This means when the table header is clicked, the user can sort the table ascending and descending.
Is it possible to specify this formatting in the ColdFusion code when generating the Excel file?
If you are on Coldfusion 9, you can use SpreadSheetNew, then SpreadSheetAddRow, SpreadSheetFormat functions to style an excel spreadsheet from a data set
<cfset sObj = spreadsheetNew("myreport","yes")>
<cfset SpreadsheetAddRow(sObj, "Column_1, ... , Column27")>
<cfset SpreadsheetFormatRow(sObj, {bold=TRUE, alignment="center"}, 1)>
<cfset spreadsheetAddRows(sObj, qMyQuery)>
<cfheader name="content-disposition" value="attachment; filename=report_#Dateformat(NOW(),"MMDDYYYY")#.xlsx">
BE WARNED however, this can be extremely taxing to the JVM, I had a query i was creating an xls with, applying only two styles (bold, text-center) to the header row, and any query over 700 rows would shut down the entire server via JVM memory loss...here is my SO question about it, with related code/answer SpreadsheetAddRows failing on moderate size query
It has been documented with adobe as being a bug
We have a function to pull RSS feeds and display the post on a ColdFusion page using this code:
<cfset rssUrl = "rss1">
<cffeed action="read" source="#rssUrl#" query="fitness" properties="info">
<cfset rssUrl2 = "rss2">
<cffeed action="read" source="#rssUrl2#" query="nutrition" properties="info">
<cfif #fitness.PUBLISHEDDATE# gt #nutrition.PUBLISHEDDATE#>
<cfset entries="fitness">
<cfelse>
<cfset entries="nutrition">
</cfif>
Output done via:
<cfoutput query="#entries#">
Problem is, the RSS feed has several posts and we only want to show one. Any thoughts on how to get it to pull and display only the most recent post? (We want the feed to have multiple posts, so right now our non ideal solution is to set maximum posts per feed to 1)
cfoutput/query=".." will go over an entire query. If you only want to do the first row, use:
<cfoutput>
Title from row 1: #somequery.title[1]#
</cfoutput>
Basically - array notation on the column. Make sense?
There's nothing wrong with Ray's answer, but here are some other options.
<cfoutput query="#entries#" maxrows="1">
Offers the least disruption to your existing code and, should you decide to change the number of rows displayed (like, via a user setting) it's an easy change.
OR
If you copy the query object rather than the query name (which isn't actually a copy but a copy by reference)
<cfset entries = fitness>
instead of
<cfset entries = "fitness">
you can do this
<cfoutput>
#entries.columnName1#
#entries.columnName2#
<!--- etc. --->
</cfoutput>
which will, by default, display only the first row of the query.
I have the following ColdFusion 9 code:
<cfloop from="1" to="#arrayLen(tagArray)#" index="i">
<cfquery name="qryGetSPFAQs" datasource="#application.datasource#">
EXEC searchFAQ '#tagArray[i]#'
</cfquery>
</cfloop>
The EXEC executes a stored procedure on the database server, which returns rows of data, depending on what the parameter is. What I am trying to do is combine the queries into one query object. In other words, if it loops 3 times and each loop returns 4 rows, I want a query object that has all 12 rows in one object. How do I acheive this?
You might want to take a different approach (modify your stored procedure to accept multiple arguments or use a list and fnSplit) and return the dataset all at once. However, to directly answer your question, this is how you could combine the queries as you're asking to:
You can use UNION in a Query of Queries to combine all of the datasets.
<cfloop from="1" to="#arrayLen(tagArray)#" index="i">
<cfquery name="qryGetSPFAQs#i#" datasource="#application.datasource#">
EXEC searchFAQ '#tagArray[i]#'
</cfquery>
</cfloop>
<cfquery name="combined" dbtype="query">
<cfloop from="1" to="#arrayLen(tagArray)#" index="i">
select * from qryGetSPFAQs#i#
<cfif i lt arrayLen(tagArray)>UNION</cfif>
</cfloop>
</cfquery>
A more direct way might be something like this:
<cfset bigQ = queryNew("column")>
<cfloop from="1" to="#arrayLen(tagArray)#" index="i">
<cfquery name="qryGetSPFAQs" datasource="#application.datasource#">
EXEC searchFAQ '#tagArray[i]#'
</cfquery>
<cfset queryAddRow(bigQ)>
<cfset querySetCell(bigQ, "column". qryGetSPFAQs)>
</cfloop>
You will need a querySetCell() assignment for each column. Check out the query functions in the live docs for more information.
Here is an out of the box solution, abandoning the StoredProc for a SQL View (I'll explain).
Disclaimer: without seeing the SP source code, I can't tell if my solution fits. I'm assuming that the SP is fairly basic, and I admit I usually prefer the compiled execution of an SP over a view, but the one-time execution of a SQL View should outperform the looping of the SP x times.
First make a view that looks like the SELECT statement in the SP (minus the parameterization, of course -- you'll cover that in a WHERE clause within the CFQUERY of your new view.
Second, set up your loop to do no more than build a data set we're going to use for the WHERE clause. You'll need to use ArrayToList and a little bit of string manipulation to tidy it up, with the end product being a string stored in a single CF variable looking like this:
('ValueOfArrayElement1','ValueOfArrayElement2','Value_And_So_On')
Building the string is pretty easy, using the delimeter attribute of ArrayToList, and after the loop is complete, append a Left Parenthesis & Single Quote to the Left most position of the string; and append a Single Quote & Right Parenthesis to the Right most position in the string.
Now, write the CFQUERY statement to SELECT the columns you need from your view (instead of executing your SP). And instead of passing a parameter to the SP, you're going to put a WHERE clause in the CFQUERY.
Oh, BTW, I am stating you need a SQL View, but the entire SELECT could be built in CFQUERY. Personally, when I have a multi-table JOIN, I like to define that in a SQL View where it's executed more quickly than a JOIN in CFQUERY. Ultimately a StoredProc is even faster, but our WHERE clause is much friendlier to code and read like this than it would be to send into StoredProc without looping in and out of SQL.
It's a good goal to make only one trip out to the database and back if possible. That's why we looped through the array to write a string equating to all the values in the dataset. This way, we'll only execute one query, one time.
SELECT Col1, Col2, Col_etc
FROM SQL_View_Name
WHERE ColumnName in #BigStringWeMadeFromArrayToList#
when our CFQUERY is rendered, the clause will look just like this in SQL:
WHERE ColumnName in
('ValueOfArrayElement1','ValueOfArrayElement2','Value_And_So_On')
So there you have it. Like I said, this is nice because it makes only one trip to the DB, and since we are building a view, the performance will still be pretty good -- better than running a StoredProc 4+ times. (no offense)
I'll must repeat... without having seen the SP code, I'm not sure if this is do-able. Plus, it's kind of odd to abandon a StoredProc for a SQL View, a "lesser" entity in the RDBMS, but I'm sure we will achieve greater performance and I think it's pretty readable, too.
Why does this not work? My welcome message, it just doesn't show up:
<p>Welcome <cfoutput>#Recordset1.UserID#</cfoutput>.</p>
The session variable on the login page I created is:
<cflock timeout=999 scope="Session" type="Exclusive">
<cfset Session.IDUsers =''>
</cflock>
is this incorrect? On the index page where I'm trying to display my welcome message I have:
<cfquery name="Recordset1" datasource="cfGossip">
SELECT *
FROM users
WHERE users.IDUsers = <cfqueryparam value="#Session.IDUsers#">
</cfquery>
I'm not sure if this works, or is necessary?
If you set the userid stored in the session to be the empty string, when you query on it, you will only get users for whom the id is the empty string, which shouldn't be any of them. Therefore, the query is returning an empty set, and your page is (correctly) not displaying a user id.
How are you initially identifying the user? Are you querying against a database when they log in? Are you storing a cookie? Reading Tarot cards? For this to work, at some point, you have to store the correct userid, probably in the session. To do that, you need to first identify who the user is.
Also, if you are using CF6+, you probably do not need the cflock. It is now used to prevent race conditions, as CF is now thread-safe.
Looks like you're just starting with CF, welcome to the community.
My understanding of your code makes the structure look like the following, if I'm understanding you correctly:
<cfset session.idUsers = '' />
<cfquery datasource = "cfgossip" name = "recordset1">
SELECT * FROM USERS WHERE USERS.ID_USERS = <cfqueryparam cfsqltype = "cf_sql_integer" value = "#session.idUsers# />
</cfquery>
<cfoutput>Welcome #recordset1.userID#</cfoutput>
The reason this doesn't work is because your session.idUsers value is blank. Assuming you have a user in your database with an ID_USERS value of 1, you could change the CFSET to look like and it should return results.
Additionally, while it's great to see you using CFQUERYPARAM, I'd recommend including a CFSQLTYPE attribute in the tag whenever possible to provide an added line of defense against injection attacks. You can check out http://livedocs.adobe.com/coldfusion/8/htmldocs/help.html?content=Tags_p-q_18.html to see the list of available types.
Is there anywhere in your code where you set your session.IDUsers? You initialize it as a blank ''. Coldfusion does not populate it for you. The session scope is a place that will remember things for that user that you put there for a specified period of time inactivity, usually 20 minutes. So hopefully, somewhere before you run your query you have additional logic that fills that in, otherwise you are asking the database for a user named, ''.
This is just a point of style, but the following may work better for you:
<cfset Session.IDUsers =''>
<!--- Do something here to populate Session.IDUsers --->
<!--- Creates a blank query - not necessary, but can reduce errors later --->
<cfset Recordset1 = queryNew("UserID")>
<!--- Populate the query from the database --->
<cfquery name="Recordset1" datasource="cfGossip">
SELECT *
FROM users
WHERE users.IDUsers = <cfqueryparam value="#Session.IDUsers#">
</cfquery>
<!--- If the query has data, use it, otherwise show generic message --->
<cfoutput>
<cfif Recordset1.recordcount>
<p>Welcome #Recordset1.UserID#.</p>
<cfelse>
<p>Welcome new user!</p>
</cfif>
</cfoutput>
<!--- OR since we used queryNew("userID"), we can simplify without throwing an error. ---->
<cfoutput>
<p>Welcome <cfif len(Recordset1.userID)>#Recordset1.userID#.<cfelse>new user!</cfif></p>
</cfoutput>
Putting the cfoutput outside the paragraph block will make it easier if you have additional variables to insert into the text. (but will work either way)
Regardless of all that, unless you forgot to share a bit more of the code, I think the issue is that the session.IDUsers is blank and needs to be populated before the query. I hope this helps!