I have two sql server tables which I would like to sync to a read sql database. I want to flatten the data from the source database into one table in the read database using the sync framework. Can I do this?
The schemas in the source and target need to match. You could add a view that joins the two source tables within the source database and presents the data in the same format that your 'read' database expects.
if you're using the older providers (same one used by VS Local Database Cache Project item), you can use a view on the server side, however, your client can only be SQL Ce. but even that is tricky, what constitues a changed row if a change can occur on two source tables? if table 1 is updated and table 2 is not or vice versa?
the newer provider for SqlSyncProvider dont support views as its change tracking is based on triggers and the entire provisioning works against tables.
#Scott, schema's or table structures dont need to match.
I'm trying to do the same, I suppose.
This is my question on stackoverflow:
Merging 2 tables in a single table with different schema
I worked on this problem for some times and I reached some results...
For now, I'm working on the case in which the changes are only tracked in the PERSON table (so if something change in ADDRESS the changes are not synchronized). But I suppose the code can be improved to track changes in ADDRESS too... And for now I'm not taking into consideration the changes in the destination db (in CUSTOMER table). This will be more difficult to code, I suppose...
Anyway, my solution add an handler to changesSelected, there I alter the DataTable adding the columns I need (Address and City). I get the Address and the City by a sql SELECT and updates the rows... This works for updated and inserted rows...
The problem raise with deleted rows. In my db CUSTOMER, the primary key must be Id-Address, and not only Id (or I can't have multiple ADDRESS for each PERSON). So, when SyncFX tries to perform a deletion, the keys don't match and the deletion doesn't affect any row... I don't know how to alter a DataRow with state deleted, and I also can't get the Address from the db... So I can't have an Id-Address information in the deleted DataRow...
For now, I can only perform a sql DELETE using the Id (the only available info for a deleted row)...
Please try to improve the code and post back, so we could help each other!
This is the code. First the addhandler, then the code into the handler.
AddHandler remoteProvider.ChangesSelected, AddressOf remoteProvider_ChangesSelected
...
Private Shared Sub remoteProvider_ChangesSelected(ByVal sender As Object, ByVal e As DbChangesSelectedEventArgs)
If (e.Context.DataSet.Tables.Contains("PersonGlobal")) Then
Dim person = e.Context.DataSet.Tables("PersonGlobal")
Dim AddressColumn As New DataColumn("Address")
AddressColumn.DataType = GetType(String)
AddressColumn.MaxLength = 10
AddressColumn.AllowDBNull = False
'NULL is not allowed, so set a defaultvalue
AddressColumn.DefaultValue = "Nessuna"
Dim CityColumn As New DataColumn("City")
CityColumn.DataType = GetType(String)
CityColumn.AllowDBNull = False
CityColumn.DefaultValue = 0
persona.Columns.Add(AddressColumn)
persona.Columns.Add(CityColumn)
Dim newPerson = person.Clone()
For i = 0 To person.Rows.Count - 1 Step 1
Dim row = person.Rows(i)
If (row.RowState <> DataRowState.Deleted) Then
Dim query As String = "SELECT * FROM dbo.address WHERE Id = " & row("AddressId")
Dim sqlCommand As New SqlCommand(query, serverConn)
serverConn.Open()
Dim reader As SqlDataReader = sqlCommand.ExecuteReader()
Try
While reader.Read()
row("Address") = CType(reader("Address"), String)
row("City") = CType(reader("City"), String)
' Solo importando mantengo i valori di RowState
newPerson.ImportRow(row)
End While
Finally
reader.Close()
End Try
serverConn.Close()
Else
' TODO - Non funziona la cancellazione!!!
' La cancellazione cerca la chiave primaria su cliente, che รจ ID-Via
' Noi abbiamo l'ID corretto, ma "nessuna" come via...
' Dobbiamo recuperare la via giusta...
Dim query As String = "DELETE FROM dbo.customer WHERE Id = " & row("Id", DataRowVersion.Original)
Dim sqlCommand As New SqlCommand(query, clientConn)
clientConn.Open()
sqlCommand.ExecuteNonQuery()
clientConn.Close()
End If
Next
newPerson.Columns.Remove(newPerson.Columns("AddressId"))
e.Context.DataSet.Tables.Remove(person)
e.Context.DataSet.Tables.Add(newPerson)
End If
End Sub
Related
The challenge is to update a table by scanning that same table for information. In this case, I want to find how many entries I received in an Upload dataset that have the same key (effectively duplicate instructions).
I tried the obvious code:
UPDATE Base AS TAR
INNER JOIN (select cdKey, count(*) as ct FROM Base GROUP BY cdKey) AS CHK
ON TAR.cdKey = CHK.cdKey
SET ctReferences = CHK.ct
This resulted in a non-updateable complaint. Some workarounds talked about adding DISTINCTROW, but that made no difference.
I tried creating a view (query in Ms/Access parlance); same failure.
Then I projected the set (SELECT cdKey, count(*) INTO TEMP FROM Base GROUP BY cdKey), and substituted TEMP for the INNER JOIN which worked.
Conclusion: reflexive updates are also non-updateable.
An initial thought was to embed a sub-select in the update, for example:
UPDATE Base TAR SET TAR.ctReferences = (select count(*) from Base CHK where CHK.cd = TAR.cd)
This also failed.
As this is part of a job I am calling, this SQL (like the other statements) are all strings executed by CurrentDb.Execute statements. I thought maybe I could make this a DLookup, I found that as cd is a string, I had a gaggle of double- and triple-quoted elements that was too messy to read (and maintain).
Best solution was to write a function so I could avoid having to do any sort of string manipulation. Hence, in a module there's a function:
Public Function PassHelperCtOccurs(ByRef cdX As String) As Long
PassHelperCtOccurs = DLookup("count(*)", "Base", "cd='" & cdX & "'")
End Function
And the call is:
CurrentDb().Execute ("UPDATE Base SET ctOccursCd =PassHelperCtOccurs(cd)")
I've a table "City" with more than 100k records.
The field "name" contains strings like "Roma", "La Valletta".
I receive a file with the city name, all in upper case as in "ROMA".
I need to get the id of the record that contains "Roma" when I search for "ROMA".
In SQL, I must do something like:
select id from city where upper(name) = upper(%name%)
How can I do this in kettle?
Note: if the city is not found, I use an Insert/update field to create it, so I must avoid duplicates generated by case-sensitive names.
You can make use of the String Operations steps in Pentaho Kettle. Set the Lower/Upper option to Y
Pass the city (name) from the City table to the String operations steps which will do the Upper case of your data stream i.e. city name. Join/lookup with the received file and get the required id.
More on String Operations step in pentaho wiki.
You can use a 'Database join' step. Here you can write the sql:
select id from city where upper(name) = upper(?)
and specify the city field name from the text file as parameter. With 'Number of rows to return' and 'Outer join?' you can control the join behaviour.
This solution doesn't work well with a large number of rows, as it will execute one query per row. In those cases Rishu's solution is better.
This is how I did:
First "Modified JavaScript value" step for create a query:
var queryDest="select coalesce( (select id as idcity from city where upper(name) = upper('"+replace(mycity,"'","\'\'")+"') and upper(cap) = upper('"+mycap+"') ), 0) as idcitydest";
Then I use this string as a query in a Dynamic SQL row.
After that,
IF idcitydest == 0 then
insert new city;
else
use the found record
This system make a query for file's row but it use few memory cache
I am using Java DB (Java DB is Oracle's supported version of Apache Derby and contains the same binaries as Apache Derby. source: http://www.oracle.com/technetwork/java/javadb/overview/faqs-jsp-156714.html#1q2).
I am trying to update a column in one table, however I need to join that table with 2 other tables within the same database to get accurate results (not my design, nor my choice).
Below are my three tables, ADSID is a key linking Vehicles and Customers and ADDRESS and ZIP in Salesresp are used to link it to Customers. (Other fields left out for the sake of brevity.)
Salesresp(address, zip, prevsale)
Customers(adsid, address, zipcode)
Vehicles(adsid, selldate)
The goal is to find customers in the SalesResp table that have previously purchased a vehicle before the given date. They are identified by address and adsid in Customers and Vechiles respectively.
I have seen updates to a column with a single join and in fact asked a question about one of my own update/joins here (UPDATE with INNER JOIN). But now I need to take it that one step further and use both tables to get all the information.
I can get a multi-JOIN SELECT statement to work:
SELECT * FROM salesresp
INNER JOIN customers ON (SALESRESP.ZIP = customers.ZIPCODE) AND
(SALESRESP.ADDRESS = customers.ADDRESS)
INNER JOIN vehicles ON (Vehicles.ADSId =Customers.ADSId )
WHERE (VEHICLES.SELLDATE<'2013-09-24');
However I cannot get a multi-JOIN UPDATE statement to work.
I have attempted to try the update like this:
UPDATE salesresp SET PREVSALE = (SELECT SALESRESP.address FROM SALESRESP
WHERE SALESRESP.address IN (SELECT customers.address FROM customers
WHERE customers.adsid IN (SELECT vehicles.adsid FROM vehicles
WHERE vehicles.SELLDATE < '2013-09-24')));
And I am given this error: "Error code 30000, SQL state 21000: Scalar subquery is only allowed to return a single row".
But if I change that first "=" to a "IN" it gives me a syntax error for having encountered "IN" (Error code 30000, SQL state 42X01).
I also attempted to do more blatant inner joins, but upon attempting to execute this code I got the the same error as above: "Error code 30000, SQL state 42X01" with it complaining about my use of the "FROM" keyword.
update salesresp set prevsale = vehicles.selldate
from salesresp sr
inner join vehicles v
on sr.prevsale = v.selldate
inner join customers c
on v.adsid = c.adsid
where v.selldate < '2013-09-24';
And in a different configuration:
update salesresp
inner join customer on salesresp.address = customer.address
inner join vehicles on customer.adsid = vehicles.ADSID
set salesresp.SELLDATE = vehicles.selldate where vehicles.selldate < '2013-09-24';
Where it finds the "INNER" distasteful: Error code 30000, SQL state 42X01: Syntax error: Encountered "inner" at line 3, column 1.
What do I need to do to get this multi-join update query to work? Or is it simply not possible with this database?
Any advice is appreciated.
If I were you I would:
1) Turn off autocommit (if you haven't already)
2) Craft a select/join which returns a set of columns that identifies the record you want to update E.g. select c1, c2, ... from A join B join C... WHERE ...
3) Issue the update. E.g. update salesrep SET CX = cx where C1 = c1 AND C2 = c2 AND...
(Having an index on C1, C2, ... will boost performance)
4) Commit.
That way you don't have worry about mixing the update and the join, and doing it within a txn ensures that nothing can change the result of the join before your update goes through.
I have made a query that ran successfully in the first version of my program, using ADO and C++ to query MS Access 2007 database.
However, the structure of my database had to be modified.
Fields that were once of type double are now varchar.
When I execute the same query on the modified database, it reports data type mismatch, as it should.
EDITED QUERY TO HELP THE POSTER OF THE FIRST SOLUTION:
Here is simplified version of my query:
wchar_t query = L" select ( ads(Field) + Field ) / 2 from MyTable where PrimaryKey = 1;";
Field was of type double, but now is varchar.
I have tried using CDbl like this:
wchar_t query = L" select ( abs( CDbl(Field) ) + CDbl(Field) ) / 2 from MyTable where PrimaryKey = 1;";
It works when I create query in MS Access, but in my program I still get data type mismatch error reported.
I have tried to find alternative on the Internet, and have thought that CAST or CONVERT can sole this, but it seems that they do not work in MS Access.
Is there any solution for this ?
Thank you.
Regards.
Have you tried to convert the value to Double not in the query but after the query has been run?
CAST and CONVERT are not Access SQL functions
I dont use c++ but even with a small subroutine in Access using the ADO object I cannot reproduce the error...
Sub test()
Dim rs As ADODB.Recordset
Set rs = New ADODB.Recordset
Dim db As ADODB.Connection
Set db = CurrentProject.Connection
rs.Open "SELECT (Abs(CDbl(Field))+CDbl(Field))/2 AS A FROM MyTable;", db
While rs.EOF = False
Debug.Print rs!A
rs.MoveNext
Wend
End Sub
I've a table [File] that has the following schema
CREATE TABLE [dbo].[File]
(
[FileID] [int] IDENTITY(1,1) NOT NULL,
[Name] [varchar](256) NOT NULL,
CONSTRAINT [PK_File] PRIMARY KEY CLUSTERED
(
[FileID] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
The idea is that the FileID is used as the key for the table and the Name is the fully qualified path that represents a file.
What I've been trying to do is create a Stored Procedure that will check to see if the Name is already in use if so then use that record else create a new record.
But when I stress test the code with many threads executing the stored procedure at once I get different errors.
This version of the code will create a deadlock and throw a deadlock exception on the client.
CREATE PROCEDURE [dbo].[File_Create]
#Name varchar(256)
AS
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRANSACTION xact_File_Create
SET XACT_ABORT ON
SET NOCOUNT ON
DECLARE #FileID int
SELECT #FileID = [FileID] FROM [dbo].[File] WHERE [Name] = #Name
IF ##ROWCOUNT=0
BEGIN
INSERT INTO [dbo].[File]([Name])
VALUES (#Name)
SELECT #FileID = [FileID] FROM [dbo].[File] WHERE [Name] = #Name
END
SELECT * FROM [dbo].[File]
WHERE [FileID] = #FileID
COMMIT TRANSACTION xact_File_Create
GO
This version of the code I end up getting rows with the same data in the Name column.
CREATE PROCEDURE [dbo].[File_Create]
#Name varchar(256)
AS
BEGIN TRANSACTION xact_File_Create
SET NOCOUNT ON
DECLARE #FileID int
SELECT #FileID = [FileID] FROM [dbo].[File] WHERE [Name] = #Name
IF ##ROWCOUNT=0
BEGIN
INSERT INTO [dbo].[File]([Name])
VALUES (#Name)
SELECT #FileID = [FileID] FROM [dbo].[File] WHERE [Name] = #Name
END
SELECT * FROM [dbo].[File]
WHERE [FileID] = #FileID
COMMIT TRANSACTION xact_File_Create
GO
I'm wondering what the right way to do this type of action is? In general this is a pattern I'd like to use where the column data is unique in either a single column or multiple columns and another column is used as the key.
Thanks
If you are searching heavily on the Name field, you will probably want it indexed (as unique, and maybe even clustered if this is the primary search field). As you don't use the #FileID from the first select, I would just select count(*) from file where Name = #Name and see if it is greater than zero (this will prevent SQL from retaining any locks on the table from the search phase, as no columns are selected).
You are on the right course with the SERIALIZABLE level, as your action will impact subsequent queries success or failure with the Name being present. The reason the version without that set causes duplicates is that two selects ran concurrently and found there was no record, so both went ahead with the inserts (which creates the duplicate).
The deadlock with the prior version is most likely due to the lack of an index making the search process take a long time. When you load the server down in a SERIALIZABLE transaction, everything else will have to wait for the operation to complete. The index should make the operation fast, but only testing will indicate if it is fast enough. Note that you can respond to the failed transaction by resubmitting: in real world situations hopefully the load will be transient.
EDIT: By making your table indexed, but not using SERIALIZABLE, you end up with three cases:
Name is found, ID is captured and used. Common
Name is not found, inserts as expected. Common
Name is not found, insert fails because another exact match was posted within milliseconds of the first. Very Rare
I would expect this last case to be truly exceptional, so using an exception to capture this very rare case would be preferable to engaging SERIALIZABLE, which has serious performance consequences.
If you do really have an expectation that it will be common to have posts within milliseconds of one another of the same new name, then use a SERIALIZABLE transaction in conjunction with the index. It will be slower in the general case, but faster when these posts are found.
First, create a unique index on the Name column. Then from your client code first check if the Name exists by selecting the FileID and putting the Name in the where clause - if it does, use the FileID. If not, insert a new one.
Using the Exists function might clean things up a little.
if (Exists(select * from table_name where column_name = #param)
begin
//use existing file name
end
else
//use new file name