Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
What is the best practice to build an integration between Dynamics 365 and SAP BW (dynamics data should be imported to SAP BW on every update/change event).
Thanks.
As a first step, in order to work with Dynamics 365 data, we have dedicated components for connecting, reading and writing to your instance, which you can find as a part of our SSIS Integration Toolkit for Microsoft Dynamics 365.
Since we do not have dedicated components for SAP BW, if the API offers a Rest API, then you could use our HTTP Connection Manager to build a Restful connection with the API referring to the documentation, and then use our JSON components to read or write data to it. You can check out the Online Help Manuals for each of the above-mentioned components.
As for incremental loads, we have our Integration Gateway solution that offers real-time integration capabilities using webhooks. Webhook is a web callback URL that is generally triggered by an event or action in the system and delivers data to the target when it happens. To use this, you would need to obtain, register and configure webhooks for your specific instances. The configuration would depend on how SAP BW authentication works, and depending on that you may try and use the custom webhook option that we have to configure and set it up. We have a blog post that shows how this is done between HubSpot and Dynamics 365. A similar approach can be used for SAP BW.
Another way to perform an incremental load is by designing the package within SSIS in order to get the last run time and then fetch the records as per the timestamp and create a bidirectional integration. With Dynamics 365, if you meant CDS/CRM, then we have options to use the change tracking feature in CRM and get Entity Changes. We have a blog post on that as well with a sample package.
You would need to use the above-mentioned JSON components to create a bidirectional sync logic between SAP BW and Dynamics 365. Then you can deploy the packages to SSISDB catalog on your integration server, and schedule it using a SQL Server agent job. Schedule it to run at as close intervals as possible, so that all changes (inserts, updates, deletions) are captured and fully synced to both systems. You can also take a look at some other general incremental load blog posts we have.
Incremental Load in SSIS Made Easy
Incremental Load in SSIS with Diff Detector
Question is quite generic but let me add my 2 cents. Someone with architecture level knowledge could add even more relevant answer.
I would prefer such integration between 2 systems using Azuer service bus with azure functions.
Now why azure service bus? If in case at some data cases integration fails you have error messages/records which could be analyzed and retried for integration.
I would also propose webhook with azure functions. Here we move integration outside crm and avoid 2 minutes times out from crm.
I would also propose power automate. But integration logic could get complicated here and power automate might consume more API calls than needed.
There can be azure data factory but I have limited knowledge about it.
You could also look into 3rd party softwares like kingswaysoft and scribe which are specifically meant to perform integration.
My inclination is towards azure service bus with azure functions.
Related
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 days ago.
This post was edited and submitted for review 5 days ago and failed to reopen the post:
Original close reason(s) were not resolved
Improve this question
Question:
Can the below architecture be used in production?
Mobile app serviced by Firebase(FireStore + CloudStorage) and Cloud Functions for http calls to the external API) +
AchivementsApi deployed on Appengine standard conencted with CloudSQL.
Context:
I wrote two apps that have to be used as digital support for offline gaming events.
Expected usage pattern:
The traffic starts increasing at the beginning of the week when users have to do some online tasks, then a big spike will happen in the weekend when the offline gathering is taking place. We expect to have thousands of users. In the most optimistic case, let say we will reach 8000 users.
Flutter Mobile App
a. Authentication/Profile – for this I choose Firebase as is free and scalable (also this option provides monitoring, alarms, push notifications, etc.)
b. Event related data that in most cases will not change (event timetable, exhibitors, infos... nothing intensive here) – Using the Firebase backend with Cloud Firestore db looks like the obvious choice.
c. Images can be stored on Cloud Storage or even packaged with the app
d. Integration with Achievements Api – this implies sending REST requests to the below java service, using api key for auth. A scalable and safe(storing the api key) option for this seems to be Cloud Functions. Of course, if I opt for the dedicated back end deployed on Cloud Engine or App Engine or somewhere else that service can handle the Rest calls also
Springboot Achievements Api
Service with some complex queries but no process is extremely intensive or time consuming. This must be a stand alone service available for future integrations. Due to the choice made above of using Firestore, I was thinking that this can be deployed on stantard AppEngine environment with Cloud SQL connection.
I have exported Firestore collections to Google Big Query to make data analysis and aggregation.
What is the best practice (using Google Cloud Products) to serve Big Query outputs to a client web application?
Google provides seven client libraries for BigQuery. You can take any library and write a webserver that will serve requests from client web application. The webserver can use a GCP service account to access BigQuery on behalf of its clients.
One such sample is this project. It's written in TypeScript. Uses NodeJS library on the server and React for the client app. I'm the author.
You may try to have an express tour through Google Data Studio, looking for the main features what this Google analytics service can offer for the customers. If your aim stands for visualizing data from Bigquery, Data Studio is a good option, thus it provides a variety of informative dashboards and reports, allowing the user customize charts and graphs sharing them publicly or via user collaboration groups.
Data Studio spreads a lot of connectors to different data sources, hence you can find a separate Bigquery connector for further integration with data resources residing in Bigquery warehouse.
You can track for any future product enhancements here.
I am familiar with firebase platform, but I am relatively a new user of the google cloud platform as whole.
I am working on a project built using a microservices structure, and I do have so many question for which I cannot find an answer or better I cannot find any example.
Unfortunately all the example that I am able to find are way to simple to be able to extrapolate a viable answer for my issues.
I adopted the new cloud run offer, and I decided to play with the full managed version (not kubernetes). I built few microservices (each service is built using express for node or flask for python - depending on what the services does). Each microservices expose it's own endpoint and has it's own api to call the methods - and I use a service account to allow the application to perform the internal calls.
I now want to expose the application to the external (specifically to my client built using vuejs technology), and I was trying to leverage another google product to create and expose an api: the google endpoints.
My question (specifically referred to the cloud run structure) is related to how is possible and what I need to do to create an api endpoints to communicate with the client app, that internally calls multiple services and combine their response in one.
Just to be clear, let's make an example:
Cloud run service 1 -> crud user api
Cloud run service 2 -> crud product api
Cloud endpoint external visible api -> get user from service 1, and after get products from service 2 and return the combined response all green products for user Jane Doe.
How I can aggregate the response directly in the endpoint gateway, check for failure and if everything goes smooth send the aggregate response to the client?
I need to build the aggregate endpoint in something else, like a cloud function for example? or I can do it directly in the google endpoints gateway?
Note that for cloud run the google endpoints is another cloud run container.
Thanks guys for some help, running pretty much out of option here.
As per my understanding, API Gateway should just work as a proxy, presenting all micro services as a single endpoint. To this scenarios I think you can have following 2 approaches :
1: Implement a new micro service (or on any of the existing one) which will do invocations and aggregation of responses.
2: Client(like UI) can invoke the services and do the aggregation on their side as well.
I feel, it is not a good idea to do it at api-gateway.
In my opinion, from an architectural point of view, the best option for you is to create a new microservice which will take the responses from the other two and then, it will aggregate them.
I understand that you want to aggregate the responses in a api-geteway and you are not able to find code examples for it. Here I was able to find a guide on what are you wanting to implement. The full code implementation can be found in this repository.
Keep in mind though, this idea of implementation is not a best practice.
This is ok, only if those two services that are going to be combined are independent. Meaning there is no functional/business relation between them and the concurrency or inconsistency problem will not occur in the process of aggregating.
Our product requires support for a high level dashboard highlighting metrics pertaining to the business modeled by the product. Its a B2B product with a web application as a front-end.
We use Amazon RDS to store business data. What is the best approach to build a customizable dashboard in AWS?
The traditional approach would be to create metrics, process business data (analysis) and store the metrics/results in output tables. And then use a fancy charts library in the web application to build a support a panel/chart/graphic view per such metric. Support drill downs etc.
However, I believe this approach is outdated and too much work. Are there any readmade solutions available? Ideally, I should just be able to push results data to a third party solution, and the third party solution should allow for creation of custom dashboards (custom implies that the user can choose to select particular metrics/panels he would like to see in the dashboard, and their relative order& position in the dashboard) and embedding them in the products web application, to enable to end user to view as well as customize dashboards.
I'm aware of AWS quicksight, but it does not support embedding output views/dashboards into a web application.
I like Azure Power BI; At least you get to embed the dashboard in the web application, but I do not believe the end user can customize the embedded view. And ofcourse, Azure PowerBI is not on AWS ;).
Is there a mature 3rd party solution that we can evaluate? The scale of the data is very very small. So we are not looking for a high performance enterprise solution (might be too expensive anyway!). However, the need for customizability of dashboards view is high.
Embedded dashboards were added to QuickSight in November 2018, so you can now use that tool if you'd prefer.
https://docs.aws.amazon.com/quicksight/latest/user/embedded-dashboards-setup.html
Tableau is the first possibility, but it is expensive.
Infogram may also be an option: https://infogram.com/examples/dashboards
If you have small data volumes and you want cusomisation you may unfortunately have to look into writing or at least customising an existing solution.
I want to create a weather app for my android phone but now I got stuck on the backend part of the app.
I have found a weatherservice where I can, for free, get detailed information about a certain location through their webservice. But they have stated in their rules that I am not allowed to poll their service with a high frequency. So I thought that I could create a webservice on my own that retrieve weatherinformation from the weatherstation that I found and then make it available through my webservice so that my app only make calls to my service.
the communication will be like below
MyApp <--> MyWebService <--> commercial webservice
the android app talks to MyWebService. And my webservice talks to the commercial service.
So I want MyWebService to do to things.
retrieve information from the commercial webservice once every hour and update my database
handle requests from my androidApp
My problem is that I know to little about web application and web services. I don't really know what language to choose for the webservice.
PHP with soap or REST looks like a good candidate for the second task. But I can't find any sample on how to handle the first task. Is there any easy way to tell the server to run my script once every hour?
I have been looking a litle into C# as well which would suit me a litle bit more as I am more used to C#. But the same question arise here. How do I handle the second task?
This is something that I wanted to write for a long time, but I feel totaly lost here.
Doing things "once an hour" (or more generally, scheduling tasks) from a web-only application is tricky for a number of reasons. It is much better in general to use the built-in mechanism of the operating system to perform scheduled tasks (e.g. cron under Linux, or Scheduled Tasks under Windows), or to write a service/daemon process that handles the updates.
Having said that, there is a fairly straightforward way to meet your requirement. You can cache the result of the commercial web service in your web application tier, along with a timestamp of the last time you retrieved the information. When a web request comes into your web service from your app, first check the timestamp of the cache. If the timestamp is less than one hour old, just returned the cached weather data. If the timestamp is more than an hour, call the commercial web service directly from there, write the result and the current time into your cache, and return the data you just got to the app.
PHP is certainly well-suited to this kind of task. Detailed instructions on how to do that are beyond the scope of a Stack Overflow question. Google for PHP and caching, try out some examples, and ask detailed follow-up questions if you get stuck.