We are working to upgrade our application to a more current version of Ruby & Rails. Our app integrates with a legacy database (SQL Server 2008 R2) that has a table with a column of image data type (we are unable to change this column to varbinary(max)). Previously we were able to save a binary into the image column. However now we are getting conversion errors.
We are working to upgrade to the following (among others):
Rails 4.2.1
ActiveRecord_SQLServer_Adapter (4.2.4)
tiny_tds (0.6.3.rc1)
freeTDS (v0.91.112)
When we now attempt to save into the image column, we get errors similar to:
TinyTds::Error: Unclosed quotation mark after the character string
Researching various issues within tiny_tds & activerecord_sqlserver_adapter, we decided to create a second table that matched the first but change the data type from image to varbinary(max). We can save a binary into the column.
The code causing the challenge is in a background job where we grab images from s3, store them locally and then push the image into the database. Again, we don't control the legacy database and thus can't change the data type (or confront the issue of why we are storing the image in the db in the first place).
...
#d = Doc.new
...
open("#{Rails.root}/cache/pictures/image.png", "wb") do |file|
file << open(r.image.url).read
end
#d.document = File.binread("#{Rails.root}/cache/pictures/image.png")
#d.save!
Given the upgrade has broken our saving images, we are trying to figure out how best to determine a fix. We could obviously roll back until we find a version that works. However we hope to find a fix. Anyone have any ideas?
Update:
We added the following configuration as we had triggers on the table being inserted: ActiveRecord::ConnectionAdapters::SQLServerAdapter.use_output_inserted = true
When we remove this configuration we get the following error:
TinyTds::Error: The target table 'doc' of the DML statement cannot have any enabled triggers if the statement contains an OUTPUT clause without INTO clause.
Note: We are unable to make any modifications to the triggers.
Per feedback on the ActiveRecord_SQLServer_Adapter site, we rolled back to 4.1.11 and we are now able to save into the image column.
We also had to add this snippet to overcome the issue with the triggers.
Related
I'v been asked to create an app in Oracle Apex that will allow me to drop a CSV file. The file contains a list of all active physicians and associated info in my area. I do not know where to begin! Requirements:
-after dropping CSV file to apex, remove unnecessary columns
-edit data in each field, ie if phone# > 7 characters and begins with 1, remove 1. Or remove all special characters from a column.
-The CSV contains physicians of every specialty, I only want to upload specific specialties to the database table.
I have a small amount of SQL experience from Uni, and I know some HTML and CSS, but beyond that I am lost. Please help!
Began tutorial on Oracle-Apex. Created upload wizard on a dev environment
User drops CSV file to apex
Apex edits columns to remove unneccesary characteres
Only uploads specific columns from CSV file
Only adds data when column "Specialties" = specific specialties
Does not add redundant data (physician is already located in table, do nothing)
Produces report showing all new physicians added to table
Huh, you're in deep trouble as you have to do some job using a tool you don't know at all, with limited knowledge of SQL language. Yes, it is said that Apex is simple to use, but nonetheless ... you have to know at least something. Otherwise, as you said, you're lost.
See if the following helps.
there's the CSV file
create a table in your database; its description should match the CSV file. Mention all columns it contains. Pay attention to datatypes, column lengths and such
this table will be "temporary" - you'll use it every day to load data from CSV files: first you'll delete all it contains, then load new rows
using Apex "Create page" Wizard, create the "Data loading" process. Follow the instructions (and/or read documentation about it). Once you're done, you'll have 4 new pages in your Apex application
when you run it, you should be able to load CSV file into that temporary table
That's the first stage - successfully load data into the database. Now, the second stage: fix what's wrong.
create another table in the database; it will be the "target" table and is supposed to contain only data you need (i.e. the subset of the temporary table). If such a table already exists, you don't have to create a new one.
create a stored procedure. It will read data from the temporary table and edit everything you've mentioned (remove special characters, remove leading "1", ...)
as you have to skip physicians that already exist in the target table, use NOT IN or NOT EXISTS
then insert "clean" data into the target table
That stored procedure will be executed after the Apex loading process is done; a simple way to do that is to create a button on the last page which will - when pressed - call the procedure.
The final stage is the report:
as you have to show new physicians, consider adding a column (into the target table) which will be a timestamp (perhaps DATE is enough, if you'll be doing it once a day) or process_id (all rows inserted in the same process will share the same value) so that you could distinguish newly added rows from the old ones
the report itself would be an Interactive report. Why? Because it is easy to create and lets you (or end users) to adjust it according to their needs (filter data, sort rows in a different manner, ...)
Good luck! You'll need it.
One of the tables in my DB has a BLOB column that stores images. So now I am setting up the page for this table. I have a bunch of IGs and such to process most of the data, but I set up a modal page to process the image.
The modal page gets the ID (which is the PK) into an item, and then it reads the image currently in the table into a 'Display Image' item. And I have a 'File browse...' item to upload new images.
Except I cannot get it to save.
I initially started with the display image item just having Setting Based on : BLOB column returned by SQL statement, as I couldn't get the source to work with the SQL query(Error Expected CHAR, source is BLOB), I managed to resolve this by putting automatic row processing on the page and then having the source be a column.
So now it displays well, with no errors.
But the save does nothing. I have tried saving by having the File browse reference the column and using automatic row processing, and there is just nothing. No errors pop up, but it just does nothing.
I have tried saving to APEX_APPLICATION_TEMP_FILES and then having a PLSQL DA or a PLSQL process to
SELECT blob_content
FROM APEX_APPLICATION_TEMP_FILES
WHERE name = :FILE_BROWSER_ITEM
And insert this into the table, but it just pops up a 'No data found' error.
I have gone through every bit of intel my google-fu has found, but I have failed to find a solution.
So I would appreciate any insight any of you might have.
Since noone answered, I stepped away from it for a bit and tried again at a later date. And now I made it work finaly.
I set up automatic row fetch and automatic row processing but disabled both of them, for some reason automatic row processing must be there so that you can have the source for the display image and file browse be the column.
Then I set the browse file to load into apex_application_temp_files.
and set up a process to be executed at page submit(but after the automatic row processing even though its disabled and shouldnt matter). The process executing the following code:
BEGIN
UPDATE MY_TABLE
SET MY_IMAGE = (SELECT blob_content
FROM apex_application_temp_files
WHERE name = :FILE_BROWSER_ITEM)
WHERE id = :ID;
END;
And I execute the page submit through a button with the action page submit and Database action being SQL UPDATE action.
I am guessing a fair bit of the things I did and have set up dont even matter, but I dont dare remove them for fear of breaking shit. What I have described here finaly works for me, and if you stumble upon this then you can try and I hope it works for you too, and you can try removing some of the disabled stuff and see if it still works.
I am getting below error while I save the transformation in pentaho spoon:
Error saving transformation to repository!
Error updating batch
Cannot insert duplicate key row in object 'dbo.R_STEP_ATTRIBUTE' with unique index 'IDX_RSAT'. The duplicate key value is (2314, PARTITIONING_SCHEMA, 0).
Everything was working fine before I ran a job that creates multiple excel files. While this job was running suddenly a memory issue occurred and the job was aborted. After that I tried to save my file but it is deleted for saving but not been saved. So I lost the job I created.
Please help me to know the reason.
The last save of the directory did not end gracefully.
There is a small chance that you can repair it by easing the db-caches file in the .kettle directory.
If it does not work, create a new repository and copy the current in the new. Try the global repository export/import. Then erase the old rep and do the same from the just rebuild repository.
The intermediary repository may be on files rather than on a database.
If it is the first time you do this, plan for a one-two hours.
There is a easy way to recover this.
As AlainD says, the problem occurs when you save or delete a transformations, and suddenly you lost the connection or had a problem with Kettle.
When that occurs, you will find a lot of step records into the table R_STEP_ATTRIBUTE. In the error shown is the [ID_TRANSFORMATION] = 2314.
So, if you check the table R_TRANSFORMATION with [ID_TRANSFORMATION] = 2314, maybe wont find any transformation with that id.
After check that, you can delete all the records related with that [ID_TRANSFORMATION], for example:
delete from R_STEP_ATTRIBUTE where ID_TRANSFORMATION=2314
We just solved this issue by executing the following SQL statement
DELETE
FROM R_STEP_ATTRIBUTE
WHERE ID_STEP NOT IN (SELECT ID_STEP FROM R_STEP)
I using Python Spark API while getting file name having an issue. For example
recordDataFrame=originalDataFrame \
.withColumn('file_name',input_file_name())
Added the file_name column to dataFrame.The column is added to the dataFrame
recordDataFrame.take(1)
Above shows the column with value
But when I am converting dataFrame to RDD or looping over RDD file_name column don't have any value.
For example,
rdd_data=recordDataFrame.rdd
print(rdd_data.take(1))
This will show file_name column with blank value
Or if I do looping over the dataFrame directly then also file name don't have any value
recordDataFrame.foreach(process_data)
But if I pass the static value to file_name instead of using input_file_name() while adding column then everything works fine
This is a bug which has been resolved in 2.0.0.
Disclaimer:
These are serious hacks and should be avoided unless you're desperate. Also non of these have been properly tested. If you can it is better to update.
Trigger a shuffle after loading the data for example with:
recordDataFrame.repartition("file_name")
or
recordDataFrame.orderBy("file_name")
Truncate lineage as shown in high-performance-spark/high-performance-spark-examples
(code is GPL licensed so it cannot be reproduced here but the main idea is to access internal Java RDD, cache it and recreate DataFrame):
cutLineage(recordDataFrame)
I'm trying to store file in SQL server using ODBC driver to a column defined as varbinary(max), when I use SQL server driver I get:
the text, ntext, or image pointer value conflicts with the column name specified, the insert/updtae of a text or image column did not succeed
When I use native client driver I get
string data right truncation
Both are symptoms of the same problem well documented in MSDN. Inserting BLOBs bigger than 400kb will trigger this error, any suggested fix!?
Migrating to OleDB is not an option.
The sqlsrv32.dll installed at my machine is, file version: 6.1.7601.17514
Finally i managed to find the right way,
All you have to do is in your 'DoFieldExchange' function is to:
m_rgODBCFieldInfos[6].m_nSQLType = -4;
BLOB's are always should be found at the end of your query, so m_rgODBCFieldInfos['x'] refers to your m_nFields-1 location in this array, if you have more than one BLOB you should interrogate which one is which.
In my case this solution solved both exceptions:
from Native Client 11:
"String data, right truncation."
from SQL Server:
"the text, ntext, or image pointer value conflicts with the column name specified, the insert/updtae of a text or image column did not succeed"
Cheers :)