I am using the libodbc++ ODBC wrapper for C++, designed similar to JDBC. I have a prepared statement "INSERT INTO t1 (col1) VALUES (?)", where t1.col1 is defined as VARCHAR(500).
When I call statement->setString(1, s), the value of s is truncated to 255. I suspect the libodbc++ library, but as I am not very familiar with ODBC I'd like to be sure that the wrapper doesn't just expose a restriction of the underlying ODBC. The ODBC API reference is too complicated to be skimmed quickly and frankly I really don't want to do that, so pardon me for asking a basic question.
NOTE: an un-prepared and un-parameterized insert statement via the same library inserts a long value ok, so it isn't a problem of the MySql DB.
For long string, use PreparedStatement::setAsciiStream() instead of PreparedStatement::setString().
But when use stream, I often encounter error "HY104 Invalid Precision Value", which is annoying because I have no idea how to tackle it head on, however I work around it with following steps:
1, order the columns in SQL statement, non-stream columns go first;
2, if that doesn't work, split the statement to multiple ones, update or query a single column per statement.
But(again), in order to insert a row first and then update some columns in stream manner, one may have to get the last insert id, which turns out to be another challenge which I again failed to tackle head on for now...
I don't know libodbc++, but PreparedStatements available via ODBC API can store more characters. I use it with Delphi/Kylix ODBC wrapper.
Maybe there is some configuration in libodbc++ to set value length limit? I have such setting in my Delphi library. If you use PreparedStatement then you can allocate big chunk of memory, divide it into fields and show ODBC where block for each column starts and how long is it via SQLBindParameter() function.
Related
I have a port 'Number_1' in expression transformation in Informatica. I connected the Number_1 port to a target sql table. I need to generate number for this port 'Number_1' every time i run the mapping starting from 1 till 999. once it reaches 999 then again the value of the Number_1 should reset to 1. I'm aware there is sequence generator trans. but i need to call Sequence function from SQL server. how to achieve above?
Create a stored procedure in sql server then use stored procedure tranformation to call this from Informatica
You may call it using Stored Procedure transfomation. You may also use a stored procedure as SQL Override on Source Qualifier, however...
I hope you know what you're doing, as this is in general a very bad idea. Each call requires communication between Intergration Service and database - this will cause huge delays. Therefore it's much better to use Informatica's Sequence Generator or - even better perhaps, if all you need is an integer port with round robin - a simple expression variable.
While maciejg says makes a lot of sense performance wise - however I've known a fair few people who were more comfortable using the native database sequencer than the inbuilt one ( even some Informatica specialists ).
The thing with Informatica sequencer is how much flexibility they give and when they get set wrong it can lead to unexpected numbers being picked.
One example I have is for a sequencer being used to create unique keys in a table - if you persist the value between sessions then it works fine until you select the incorrect option while reimporting the mapping.
If you choose to lookup your previous ending value from a config file / table and add the value produced by the sequencer to this then one day when someone by mistake sets the sequencer to persist values between runs you'll all of a sudden skip that many numbers in the sequence each time the session is restarted. Native db sequencers are very basic making them very predictable and fool proof
When we fire a SQL query like
SELECT * FROM SOME_TABLE_NAME under ORACLE
What exactly happens internally? Is there any parser at work? Is it in C/C++ ?
Can any body please explain ?
Thanks in advance to all.
Short answer is yes, of course there is a parser module inside Oracle that interprets the statement text. My understanding is that the bulk of Oracle's source code is in C.
For general reference:
Any SQL statement potentially goes through three steps when Oracle is asked to execute it. Often, control is returned to the client between each of these steps, although the details can depend on the specific client being used and the manner in which calls are made.
(1) Parse -- I believe the first action is actually to check whether Oracle has a cached copy of the exact statement text. If so, it can save the work of parsing your statement again. If not, it must of course parse the text, then determine an execution plan that Oracle thinks is optimal for the statement. So conceptually at least there are two entities at work in this phase -- the parser and the optimizer.
(2) Execute -- For a SELECT statement this step would generally run just enough of the execution plan to be ready to return some rows to the client. Depending on the details of the plan, that might mean running the whole thing, or might mean doing just a very small fraction of the work. For any other kind of statement, the execute phase is when all of the work is actually done.
(3) Fetch -- This is when rows are actually returned to the client. Generally the client has a predetermined fetch array size which sets the maximum number of rows that will be returned by a single fetch call. So there may be many fetches made for a single statement. Of course if the statement is one that cannot return rows, then there is no fetch step necessary.
Manasi,
I think internally Oracle would have its own parser, which does parsing and tries compiling the query. Think its not related to C or C++.
But need to confirm.
-Justin Samuel.
I am trying to understand which ODBC functions to call and how to call
them in order to fetch rows in batches or insert rows in batches ( inserts that use bind variables not just an array of insert statements ).
I can fetch one row at a time by calling these functions in order
SQLBindParameter
SQLExecute
SQLFetch
Also if doing inserts / updates I can do one row at a time by calling these functions
SQLBindParameter
SQLExecute
What I don't know is what I need to change in these calls in order to:
1) Fetch rows in batches e.g. 150 rows per batch
2) Insert several rows per SQLExcecute call e.g. 150 rows per call
Short contained examples ( not necessarily compilable since ODBC progs tend to be long .. so ignore setup/initialization, ignore error checking ) demonstrating how this is done would be helpful. Or a pointer to a comprehensible open source code that is doing this sort of thing
The following article tells you how to send rows of parameters in one go:
http://www.easysoft.com/products/data_access/odbc_odbc_bridge/performance_white_paper.html#3_1_2
Basically, you need to search for SQLSetStmtAttr and SQL_ATTR_PARAMSET_SIZE.
To fetch multiple rows in one go see http://www.easysoft.com/developer/languages/c/odbc-tutorial-fetching-results.html
Search for SQL_ATTR_ROW_ARRAY_SIZE.
There are two self contained examples of how to perform array binding on this site
http://msdn.microsoft.com/en-us/library/ms709287(v=vs.85).aspx
Also DB2 client ships with several example code some of which show how to do array binding both for inserts and selects
Here is an excellent article from IBM Developerworks that might answer some of your questions about the ODBC architecture:
ODBC programming using Apache Derby (available via the Wayback Machine)
One of the main "tricks" for optimizing network traffic with an ODBC connection is how you define your cursor:
http://technet.microsoft.com/en-us/library/ms131453.aspx
'Hope that helps .. PSM
I noted that Sybase SQL Anywhere supports them, but can't find any documentation on ASE also doing so.
If it doesn't, what would be my best option for designing a recursive query? In SQL Server 2008 I'd do it with a CTE, but if that's not available? A function perhaps?
Sybase ASE 12.5 (and also 15.0) doesn't support CTE.
You can use a inner query to solve your problem.
A simple CTE like this:
WITH Sales_CTE (Folders_Id)
AS
-- Define the CTE query.
(
SELECT Folders_Id FROM Folders
)
SELECT Folders_Id FROM Sales_CTE
is the same as this:
SELECT aux.Folders_Id
FROM (SELECT Folders_Id FROM Folders) aux
For a litle more info, check this!
Since 1984, the Standard, and Sybase, allowed for full recursion. We generally perform recursion in a stored proc, so that depth is controlled, infinite loops are avoided, and execution is faster than uncompiled SQL, etc.
Stored procs have no limits to recursion, result set construction etc. Of course, defining the content of the brackets as a View would make it faster again (it is after all a real View, not one that we have to Materialise every time we need it).
Point being, if you were used to this method (recursion in the server, a proc coded for recursion), as I am, there is no need for CTEs, with its new syntax; uncompiled speeds; temp tables; work tables; a cursor "walking" the hierarchy; all resulting in horrendous performance.
A recursive proc reads only the data, and nothing but the data, and reads only those rows that qualify at each level of the recursion. It does not use a cursor. It does not "walk", it builds the hierarchy.
A second option is to use Dynamic SQL. Simply construct the SELECTs, one per level of the hierarchy, and keep adding the UNIONs until you run out of levels; then execute.
You can use a function to provide the facility of a CTE, but do not do so. A function is intended for a different, column-oriented purpose, the code is subject to those constraints. It is scalar, good for constructing a column value. Stored procs and CTEs are row-oriented.
I am using CODBCRecordset (a class found on CodeProject) to find a single record in a table with 39 columns. If no record is found then the call to CRecordset::Open is fine. If a record matches the conditions then I get an Out of Memory exception when CRecordset::Open is called. I am selecting all the columns in the query (if I change the query to select only one of the columns with the same where clause then no exception).
I assume this is because of some limitation in CRecordset, but I can't find anything telling me of any limitations. The table only has 39 columns.
Has anyone run into this problem? And if so, do you have a work around / solution?
This is a MFC project using Visual Studio 6.0 if it makes any difference.
Here's the query (formatted here so wold show up without a scrollbar):
SELECT `id`, `member_id`, `member_id_last_four`, `card_number`, `first_name`,
`mi`, `last_name`, `participant_title_id`, `category_id`, `gender`,
`date_of_birth`, `address_line_1`, `address_line_2`, `city`, `state`,
`zip`, `phone`, `work_phone`, `mobile_phone`, `fax`, `email`,
`emergency_name`, `emergency_phone`, `job_title`, `mail_code`,
`comments`, `contract_unit`, `contract_length`, `start_date`,
`end_date`, `head_of_household`, `parent_id`, `added_by`, `im_active`,
`ct_active`, `organization`, `allow_members`, `organization_category_id`,
`modified_date`
FROM `participants`
WHERE `member_id` = '27F7D0982978B470C5CF94B1B833CC93F997EE23'
Copying and pasting into my query browser gives me only one result.
More info:
Commented out each column in the select statement except for id. Ran the query and no exception.
Then I systematically went through and uncommented each column, one at a time, and re-ran query in between each uncomment.
When I uncomment the comment column then I get the error.
This is defined as the following (Using MySQL): LONGTEXT
Can we assume you mean you're calling CODBCRecordset::Open(), yes? Or more precisely, something like:
CDatabase db;
db.Open (NULL,FALSE,FALSE,"ODBC;",TRUE);
CODBCRecordSet rs (&db);
rs.Open ("select blah, blah, blah from ...");
EDIT after response:
There are some known bugs with various ODBC drivers that appear to be caused by retrieving invalid field lengths. See these links:
http://forums.microsoft.com/msdn/showpost.aspx?postid=2700779&siteid=1
https://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=296391
This particular one seems to have been because CRecordset will allocate a buffer big enough to hold the field. As the column returns a length of zero, it's interpreted as the max 32-bit size (~2G) instead of the max 8-bit size (255 bytes). Needless to say, it can't allocate enough memory for the field.
Microsoft has acknowledged this as a problem, have a look at these for solutions:
http://support.microsoft.com/kb/q272951/
http://support.microsoft.com/kb/940895/
EDIT after question addenda:
So, given that your MySQL field is a LONGTEXT, it appears CRecordSet is trying to allocate the max possible size for it (2G). Do you really need 2 gig for a comments field? Typing at 80 wpm, 6cpw would take a typist a little over 7 years to fill that field, working 24 h/day with no rest :-).
It may be a useful exercise to have a look at all the columns in your database to see if they have appropriate data types. I'm not saying that you can't have a 2G column, just that you should be certain that it's necessary, especially in light of the fact that the current ODBC classes won't work with a field that big.
Read Pax's response. It gives a you a great understanding about why the problem happens.
Work Around:
This error will only happen if the field defined as (TEXT, LONGTEXT, etc) is NULL (and maybe empty). If there is data in the field then it will only allocate for the size the data in the field and not the max size (thereby causing the error).
So, if there is a case where you absolutely have to have these large fields. Here is a potential solution:
Give the field a default value in the database. (ie. '<blank>')
Then when displaying the value; you pass NULL/empty if you find default value.
Then when updating the value; you pass the default value if you find NULL/empty.
I second Pax's suggestion that this error is due to trying to allocate a buffer big enough to hold the biggest LONGTEXT possible. The client doesn't know how large the data is until it has fetched it.
LONGTEXT is indeed way larger than you would ever need in most applications. Consider using MEDIUMTEXT (max size 16MB) or just TEXT (max size 64KB) instead.
There are similar problems in PHP database interfaces. A PHP normally has a memory size limit and any fetch of a LONGBLOB or LONGTEXT is likely to exceed that limit.