I'm writing a subsystem that tables might be renamed from project to project.
Instead of asking the user of my subsystem to search & replace before using it, does this work?
<cfquery name="local.foo" datasource="#dsn#">
SELECT col1, col2, col3
FROM #tableName#
</cfquery>
Without <cfqueryparam>, will it become non-cacheable? or any other issues? (assume SQL-injection is not an issue)
I don't think I can use <cfqueryparam> for table name, right?
Thanks.
That'll work, sure. CF simply converts all variables to their values and sends the string to the database driver.
Be very, very careful, though. As you've implied, this could set you up for some nasty SQL injection.
Related
In BigQuery I can write:
SELECT * EXCEPT (col1, col2, ...) ...
Is there an equivalent for RedShift? I don't think there is, but I wanted to see if anyone had any bright ideas.
Incidentally, I find this to be very useful in BigQuery when writing multiple subqueries, each flowing into the next. I can include/exclude columns at the relevant part of the query without having it break something later on, which is very useful when developing a complex query.
Not to my knowledge.
The only EXCEPT is the normal SELECT functionality to subtract one relation from another.
I am looking for solutions or ideas how to speed the processing of large data sets in sas.
What would you recommend?
What is better data step or proc sql procedure?
Speeding up your data processing depends on where your data is saved.
Your data can be either in:
SAS Table,
Database Table (Miscrosfot SQL, Oracle, DB2, MYSQL, ..
etc.)
Use SAS Data Step when:
You are querying/processing SAS tables,
You want to do iterative
processing (ex. retaining values or using arrays).
Use Proc SQL when:
You are querying a large Database table,
You can do a SQL "Pass Through" where you send SQL code to be
executed on the DB server and only the output is sent to SAS (instead
of bringing the entire tables through the network to SAS and then filter it),
You want to query SAS Tables but prefer SQL joins to data step merges.
Another topic you should consider is efficiency programming; where you are optimising your query and look-ups.
I find Proc SQL to be better for my use cases. We may need some more specifics on the size and variety of data your trying to join/export etc.
Give us some info on that and we can try to help.
Tips:
Limit the fields your pulling over
Subset data
Anecdotally from my experience Proc SQL seems faster.
Here are two tips on speeding up queries with Proc SQL:
In general, you want to rule out as much data as possible when querying. If you are usingProc SQL, the order of the restrictions in the where clause matters. Put the most restrictive parts first.
For example, if I'm querying a database for teachers with the last name "JONES", that were hired after Jan 2005, I would structure my where clause like this: where last_name = 'JONES' and hire_date > 200501 I would do this because last name is likely to exclude more records than the hire date restriction.
When possible, don't use Select * instead, list out the specific columns that you need. Remember, even if you are doing a calculation with a column, you don't have to include that column in your select statement.
Here is a very useful resource for understanding how to use proc sql efficiently. I recommend reading it in it's entirety if you do a lot of work with large data sets in SAS.
http://www2.sas.com/proceedings/sugi29/127-29.pdf
The default behaviour when importing data from a database table (such as SQL Server) is to bring in all columns and then select which columns you would like to remove.
Is there a way to do the reverse? ie Select which columns you want from a table? Preferably without using a Native SQL solution.
M:
let
db = Sql.Databases("sqlserver.database.url"){[Name="DatabaseName"]}[Data],
Sales_vDimCustomer = db{[Schema="Sales",Item="vDimCustomer"]}[Data],
remove_columns = Table.RemoveColumns(Sales_vDimCustomer,{"Key", "Code","Column1","Column2","Column3","Column4","Column5","Column6","Column7","Column8","Column9","Column10"})
in
remove_columns
The snippet above shows the connection and subsequent removal.
Compared to the native SQL way way:
= Sql.Database("sqlserver.database.url", "DatabaseName", [Query="
SELECT Name,
Representative,
Status,
DateLastModified,
UserLastModified,
ExtractionDate
FROM Sales.vDimCustomer
"])
I can't see much documentation on the }[Data], value in the step so was hoping maybe that I could hijack that field to specify which fields from that data.
Any ideas would be great! :)
My first concern is that when this gets compiled down to SQL, it gets sent as two queries (as watched in ExpressProfiler).
The first query removes the selected columns and the second selects all columns.
My second concern is that if a column is added to or removed from the database then it could crash my report (additional columns in Excel Tables jump your structured table language formulas to the wrong column). This is not a problem using Native SQL as it just won't select the new column and would actually crash if the column was removed which is something I would want to know about.
Ouch that was actually easy after I had another think and a look at the docs.
let
db = Sql.Databases("sqlserver.database.url"){[Name="DatabaseName"]}[Data],
Sales_vDimCustomer = Table.SelectColumns(
(db{[Schema="Sales",Item="vDimCustomer"]}[Data],
{
"Name",
"Representative",
"Status",
"DateLastModified",
"UserLastModified",
"ExtractionDate"
}
)
in
Sales_vDimCustomer
This also loaded much faster than the other way and only generated one SQL requested instead of two.
I have some results that came from a relational database in a SAS data set. All of the variable names start with numbers, so I can't rename them or access them in a data step. Is there any way to rename them or access them without getting the data out of the RDBMS again?
options validvarname=any; will allow you to access them, and perhaps even use the dataset - you can enclose an "illegal" variable name in "variable name"n (quotes then an n afterwards) to make a name literal which is equivalent to a variable name (like in Oracle using "variable name").
If you want to make them easier to use, you can do something like
proc sql;
select catx(' ','rename',name,'=',cats('_',name,';')) into :renamelist separated by ' '
from dictionary.columns
where libname='WORK' and memname='DATASETNAME'; *perhaps AND ANYDIGIT(substr(name,1,1)) as well;
quit;
proc datasets lib=work;
modify datasetname;
&renamelist;
quit;
You could also try setting options validvarname=v7; before you connect to the RDBMS as it's possible SAS will do this for you (depending on the situation) if you have it set that way (and don't currently).
The answer given by Joe has some helpful information, but I actually discovered that SAS has a (somewhat automatic) method for handling this. When you query data from an RDBMS, SAS will actually replace any column names starting with numbers with an underscore for the first character. So 1994Q4 becomes _994Q4. Thus, you can simply access the data that way.
SAS will, however, preserve the original name from the RDBMS as the variable title, so it will display as 1994Q4 (or whatever) in table view mode.
I am using ColdFusion 8 and have query I built using SELECT Count(dog) as stuff. This works fine on one page but when I use it, changing the parameters of course, it boggs my server down. All I need is to count the items in the db tables, return how many dogs, cats, sheep, etc. and show the count and the name. Did it with the SELECT Count the first time but can't seem to get it to work this time. Is there a way I can do the same thing by not using the Count()? And forgot to mention I am using an old Sybase db...
can't you do the following...
<cfquery name="getpetstuff" datasource="mydatabase" timeout="30">
Select dogs, count(*) as mycount
from allpets
group by dogs
</cfquery>
Adding indexes to tables will solve many performance problems in MS-SQL which has it's roots in Sybase.
Try adding an index that just contains the dogs in the allpets table.