As I have very little knowledge on how ESB's work in tandem with database I'm asking a question regarding how communication can take place between the two hoping I'll atleast be pointed in the right direction to search in!
SITUATION : We have two systems(one of them is the client's) on different networks which have their own databases. We are required to do a regular real-time data exchange of all points present in our database with the other. We are also required to have a provision to be abel to import data into our system. This exchange has to follow SOA functionality over customer provided Biztalk ESB.We are supposed to provide the exchange by the use of ODBC.
Question: My query is whether it is possible to integrate the databases to the ESB as some endpoints without making any use of WEBSERVICES or extra interfaces, and send the data over the ESB as a pull-push transfer mechanism?
I have tried searching the net for this situation but have not come up with a lot of straightforward answers. Could someone please point me in the right direction.
ESB Toolkit in BizTalk is not an ESB! It is just small additional tool for some special cases.
Let's stop talk about the ESB, we need to solve the technical problem, right?
As I can understand you have two SQL databases and want to integrate them.
To do so with BizTalk the easiest way is to use the WCF-SQL ports/adapters.
You start the Wizards for this adapter, choose the tables/sp-s which should provide data/consume data, the Wizard will generate all needed Xml schemas for you.
Then you will use BizTalk Mapper to create the Xslt maps, which will transfer one SQL data format to another.
They you will create a pair of ports. One will consume data from one SQL database, the second will insert data to another SQL database. One of this port will use the mentioned above Xslt map.
If you need more processing, you could create and orchestration to manage additional processing, sophisticated error handling, etc.
I would recommend using MSMQ. There's a fairly detailed description of it here
Related
I have to dump data from various sources to Informatica. Sources are some manual files which would be dumped via a SFTP server, some via APIs, some with direct DB connection. In that case, how do we connect the files from the server? via some kind of connection to the SFTP server, API endpoint connection, putting DB connection via DB endpoint? In these cases, how do we authenticate? i dont want to use the username/password, is there a way to use Active Directory connect?
How does informatica authenticate if the source of the files are genuine?
If you mean the source itself, then you need to decide if the source is genuine before you create a connection to it
If you mean how to secure the connection, then that is a property of the source and defined by the owner of the source. Informatica can use almost any industry-standard secure protocols and authentication methods
Any way to scan for malicious files?
Informatica can implement any business rules you want to define to determine if the data in a file is malicious
If you are asking is there a "magic button" you can press that will tell you if a file is malicious, then the answer is no
Answer to Question about PocketETL
Once you've identified all the functionality required to implement your overall architecture, you have 2 basic options for how you satisfy these requirements:
Identify a single tool that covers as much of the functionality as possible and then fill in the gaps with other tools
simplest to implement
should "just work"
unlikely to be "best of breed" in all areas
unlikely to the cheapest solution
Implement point solutions for each area of functionality
likely to be a better solution, for you, in each area
may be cheaper
but you have to get all the components working together, which is unlikely to be trivial
you need to know how to implement and configure multiple products, not just one
So you could use Informatica to do everything or you could use PocketETL to do the first piece of data movement and then other tools to implement the rest of data pipeline
I am fairly new to the subject and doing some research.
I have an ESB (using WSO2 ESB) and want to extract master data from the passing messages (like Customers, Orders, etc) and store them in DB to keep as a reference data. Source data is in XML coming from web services.
So there needs to be a component that will be able to maintain master data: insert new objects, delete old and update changed (would be also nice to have data events so ESB can route data accordingly).Basically, the logic will be similar for any entity type and it might be good idea to autogenerate it for all new entity types...
Options as I see them now:
Use Smooks with either SQLExecutor or Hibernate for persistence with all matching logic written either in smooks config or in DAO annotations
Use some open source ETL tool (like Talend, Kettle, Clover, etc). So the data will be passed to the ETL and all transformation logic is defined there. Also could accommodate future scenarios when they appear or can be an overkill..
.
Would appreciate if you share your thoughts and point me to the right direction.
You'd better to leave your database part to another tool.
If you have a fair amount of database interactions in your message flow, you can expect serious decreases in your performance.
However you do not need an ETL for the use case you explained. You can simply do it using WSO2 DSS by creating services to insert or update your data inside the database.
We have been using this for message logging purposes (inside DB) beside the ESB and are happy with that. It's better to use it as non-blocking fire-and-forget web services in your message flow within ESB. Hope this helps.
The title is probably not be explicit enough, so let me try to explain. I'm working on a new project, built on .NET, it consists of WPF clients that use WCF web services to access an Oracle database. The problem is not this basic architecture, but rather how it's supposed to work with what's already in place.
Currently, applications are written using PowerBuilder and connect to the database directly. Additionally, they use Oracle's SELECT .. FOR UPDATE statements extensively to manage concurrency by locking records. Since the new applications must exists side-by-side with the old ones, they are supposed to lock records in a similar manner too, but the new architecture that relies on web services does not make this easy.
For the moment, what we are thinking of doing is build a "data server" that would be called by the web services and would be responsible for accessing the database. The purpose of this server is to maintain the open connections/transaction needed to maintain record locks throughout several web service calls. This is needed because the "select" and the subsequent "update" parts of the operations that require SELECT .. FOR UPDATE are most likely going to happen in at least two separate web service calls ("get record" and "post update".)
I've searched the Internet for documentation regarding this kind of situation, but I can't seem to find much on the subject. Can this—i.e. keep record locks open throughout several web services calls—be done? Properly? Is my approach appropriate? Are there any published "best practices" on the matter?
Update: The original title of the question was How to maintain record locking with a service oriented architecture? I changed it following John's suggestion, hoping it may inspire some answers.
In general the SOA already have XA transaction turned on. So you can benefit from this and use the UPDATE / SELECT / UPDATE strategy. Meaning that say, you do UPDATE/SELECT as a single INVOKE example is:
UPDATE MY_TABLE SET state='Working' ... ;
SELECT * FROM MY_TABLE WHERE state='Working' ...
Then you can process the data with out fear of someone else will claim that data since it already in a different 'state', provided that the service you write only one in existing for each table.
Finally, you can complete it with the UPDATE ... SET ... state='Complete' where state='Working'. By the way, this is the same strategy the DatabaseAdapter use for polling.
I need to make a dashboard application using data from http://www.projects.zoho.com
It is a project management site.
ZOHO provides data about projects by APIs available at http://www.zoho.com/projects/developers/projects-api.html
So can I use XCelsius engage to make my dashboard?
Is it feasible & advisable?
Also tell me if any other tool like XCelsius is more suitable for me....
expecting satisfactory answers....
This shouldn't be a problem as long as the results returned via XML aren't too complex. Unfortunately Xcelsius has a hard time dealing with nested XML tags of more than a few levels so it is best to ensure that you try to conform data to a table structure. Taking this into consideration depending on complexity you may or may not have to massage data received from ZOHO prior to loading it into Xcelsius.
You also need to be mindful of flash domain security practices if you are not already aware of them.
I need to be able to access a mySQL database from my iPhone, for both read and write ops. Instead of using MCPKit (due to security and speed considerations), I'd like to access the db through a separate service. The app is iPhone SDK, so I need to get data back in XML form, not as a web page.
I am trying to decide whether to write a Java web service (SOAP) to provide this link, or to just throw together a PHP script on the server side. I can create either solution, but I don't know enough to figure out the advantages/disadvantages of the choice. Please help; thank you!
If you're writing both the client and the server, and performance isn't a significant issue, then the primary remaining consideration is development time.
So, what tools do you have at your disposal? Which platform will allow you to do this with the least amount of work? If it's a toss-up between the two, then pick the one you're most familiar with.