Speed up QoQ's or an alternative approach? - coldfusion

Question

I am building an application that performs a master query with many joins. This query data is then available to the whole application to play around with in a global variable. The query refreshes or gets the latest result set on each page refresh; so it's only in the same state for the life of the request.
In other parts of this application, I sometimes run 100's of QoQ's on this data - usually the result of recursive function calls. However, while QoQ is a great feature, it's not too fast and sometimes page loads can be between 3000 - 5000 ms on a bad day. It's just not fast enough.
Is there any kind of optimisation techniques I can do to make QoQ perform faster or perhaps an alternative method? I read an interesting article by Ben Nadel on Duplicate() function - is there any scope for using that and if so, how?
I would love to hear your thoughts.
Don't worry about crazy suggestions, this is a personal project so I'm willing to take risks. I'm running this on Railo compatible with CF8.
Many thanks,
Michael.

Answer 1

Without seeing the code and complexity of the QoQs it is hard to say for sure the best approach, however one thing you can do is use a struct to index the records outside of a QoQ. Much of the overhead of using QoQ is building new query objects, and using a struct write only approach is much more efficient than for example looping over the original query and making comparisons.
For example:
<!--- build up index --->
<cfset structindex = {} />
<cfset fields = "first,last,company" />
<cfloop list="#fields#" index="field">
<cfset key = "field:#field#,value:#q[field][currentrow]#" />
<!--- initialize each key (instead of using stuctkeyexists) --->
<cfloop query="q">
<cfset structindex[key] = "" />
</cfloop>
<cfloop query="q">
<!--- update each key with list of matching row indexes --->
<cfset structindex[key] = listappend(structindex[key], currentrow) />
</cfloop>
</cfloop>
<!--- save structindex to global variable --->
<!--- output rows matching index --->
<cfset key = "field:company,value:stackexchange" />
<cfoutput>
<cfloop list="#structindex[key]#" index="row">
#q.last[row]#, #q.first[row]# (#q.company[row]#)<br />
</cfloop>
</cfoutput>
If this doesn't match your need provide some examples of the QoQ statements and how many records are in the main query.

Answer 2

First, I would look at the time taken by the master query. If it can be cached for some mount of time and is taking a good chunk of the pageload time, I would cache it.
Next, I would look at the recursive calls. If they can be made iterative, that would probably speed things up. I realize this is not always possible. I would be surprised if this isn't your biggest time sink. without knowing more about what you are doing, though, it's hard to help you optimize this.
I might also consider writing some of the recursive QoQs s stored procedures on the DB server, which is designed to handle data quickly and slice and dice efficiently. CF is not -- QoQs are very useful, but not speed demons (as you've noted).
Finally, I would look for straightfoward filters, and not use QoQ. Rather, I would just run a loop over the master query in a standard cfoutput tag, and filter on the fly. This means you are looping over the master query once, rather than the master query once and the result query once.

Answer 3

There are two primary solutions here. First you could do something in CF with the records outside of QoQ. I posted my suggestion on this already. The other is to do everything in the db. One way I've found to do this is to use a subquery as a temp table. You can even keep the sql statement in a global variable and then reference it in the same places you are currently with the QoQ but doing a real query to the database. It may sound slower than one trip tothe DB and then many QoQ but in reality it probably isn't if indexed efficiently.
select *
from (
#sqlstring#
) as tmp
where company = 'stackexchange'
I have actually done this for system with complex criteria for both what records a user should have access to and then also what they can filter for in those records. Going with this approach means you always know the source of the inner records instead of trying to ensure every single query is pulling correctly.
Edit:
It is actually safer (and usually more efficient) to use queryparams when ever possible. I found this can be done by including a file of the sql statement...
select *
from (
<cfinclude template="master_subquery.cfm" />
) as tmp
where company = 'stackexchange'