Will the data management API be updated to allow saving your own metadata to objects and files?
If I run a batch process on a set of models, I'd rather not have to run it again on un modified objects
All Forge APIs except the Design Automation one are read-only.
If you wish to save data in any way whatsoever except using design automation (currently to create or modify DWG files), you will have do so in your own repository somewhere.
If you wish to integrate your updates into the original seed CAD model, you can absolutely do so. However, at the current moment, Forge does not help you with this (except design automation for DWG).
How to achieve this yourself using the current state of things is the topic of my talk on connecting desktop and cloud, BIM and Forge:
http://thebuildingcoder.typepad.com/blog/2016/10/connecting-desktop-and-cloud-at-rtc-material.html
Related
My question is regarding the clean architecture. I am quite new to .NET Core, and my assignment is to integrate a third-party API into the console app so the data is stored in a SQL database. I have been researching for the past two days, and I found so many examples of how to create a repository service querying a local database, but I cannot see any decent example of how to integrate the data from external APIs to a local data storage solution, SQL database in my case.
Any information on how to structure services in a clean manner would help me a lot!
It basically boils down to the question whether you need to store locally data from a 3d party API. You are not the owner of that data.
In many cases it's better to query external API each time you need some data. You don't know if the data will change and when. E.g. you query each time weather data or currency convertion rate. External system knows how to produce such results, so ask the system each time. It's often ok to cache data from 3d party APIs, depending on the needs of your application (does your user needs weather updates with each refresh or once a day is enough, e.g. on a web site of a skiing area).
Now there can be cases when 3d party API is loosely coupled with your domain. E.g. you use a separate API to manage devices in a smart home solution. In this case we are talking about microservice architecture (Microservices by Fowler is a good start). It's OK in this case to build your local copy of the data from a 3d party API. Imaging your service is only interested in how often devices get replaced. So you may query device management API for broken devices and store this information locally in the form that is better for your application. You get data, reorganize it as it better fits you, store and use it.
So basically as always with such broad questions: it depends. On what are you building, on what external API provides, on what do you want to do with the data you query.
Going technical on your question: you query data e.g. via REST from a third party. This gets you TheirObjectDto. You process this data. Their object may change, you are not the owner of the contract. You create MyObject, which contains the data that you need. You save it to your DB (via repository, if you want). You build an Entity for it and DB table. When 3d party API changes and returns TheirObjectDto2 you still work with MyObject. Minimal change for you is to convert TheirObjectDto2 to MyObject. Your app continues working. In a second step if you want some new cool info of TheirObjectDto2 you modify your internal data structure.
we have a dataset of ~10 million entities or a certain Kind in Datastore. We want to change the products functionality, so we would like to change the fields on all Kind entities.
Is there a smart/quick way to do it, that does not involve iterating over all of the entities in series?
Probably you can use Dataflow to help you with your problem.
Dataflow is a stream and batch data processing service, fully managed by GCP.
It was open sourced in the Apache Beam project. It is fully compatible with this SDK. This allows you to test your developments locally before run them on GCP.
It exposes two main concepts, a PCollection, basically the data that is being handled by the tool, and pipelines, the different steps necessary to capture the data, the transformations that must be performed, and how and where the results obtained should be written.
It provides support for Java, Python and Go, and a rich feature set and variety of possible data sources and transformations.
In the specific case of Datastore, Dataflow provides support for read, write and delete data. See for instance the relevant documentation for Python.
You can see a good example of how to interact with datastore in the Apache Beam Github repository.
These two other articles could be also interesting: 1 2.
I would presume that you have to loop through each one and update it as it's a NoSQL data store like mongo from what I can see. We have a system that uses SQL and Mongo and the demoralised data is a pain, we had to write migrations that would loop through all and update.
I am working on a project that deals with execution of several models one after the other. For this, users need to upload a lot of files (mostly CSV) for each workflow, and each files has several columns.
Since understanding each file is difficult for users of our application, we want to provide friendly names, small descriptions, help texts, etc. for each file, and display them on our website.
These names and descriptions should be editable by people who are not developers (but will have access to the AWS account). So we would prefer a storage for this that provides some convenient user-interface for this.
In the world of AWS, what would you recommend as a storage for this use-case? Is dynamodb an overkill / inconvenient for this?
Should we have our a separate user-interface and service to implement this feature?
Your choice of User Interface and Storage are completely independent.
Storage should be selected based upon the type of data and how it will be accessed. It might be relational if you are querying and joining a lot of data, or it might be NoSQL (DynamoDB or even Amazon S3) if you need fast, predictable performance but no complex querying.
The User Interface should not be impacted by the choice of storage. It should present the data for viewing/editing in a way that is most convenient for users. There is no reason to have UI drive the storage choice (unless you simply want to use Google Sheets as your frontend).
I've just started messing around with AWS DynamoDB in my iOS app and I have a few questions.
Currently, I have my app communicating directly to my DynamoDB database. I've been reading around lately and people are saying this isn't the proper way to go about getting data from my database.
By this I mean is I just have a function in my code querying my Dynamo database and returning the result.
How I do it works but is there a better way I should be going about this?
Amazon DynamoDB itself is a highly-scalable service and standing up another server in front of it requires scaling the service also in line with the RCU/WCU configured for your tables, which we can and should avoid.
If your mobile application doesn't need a backend server and you can perform all the business functions from the mobile device, then you should probably think about
Using the AWS DynamoDB SDK for iOS devices to write your client application that runs on the mobile device
Use AWS Token Vending Machine to authenticate your mobile users to grant them credentials to be used to run operations on DynamoDB tables.
Control access (i.e what operations should be allowed on tables etc.,) using IAM policies.
HTH.
From what you say, I can guess that you are talking about a way you can distribute data to many clients (ios apps).
There are few integration patterns (a very good book on this: Enterprise Integration Patterns), one of which is called shared database. It is essentially about using a common database for multiple clients to share the data. Main drawback for that pattern (in your case) is that you are doing assumption about how the database schema looks like. It can potentially bring you some headache supporting the schema in the future, if your business logic changes.
The more advanced approach would be sending events on every change in your data instead of directly writing changes to the database from client apps. This way you can add additional processing to the events before the data they carry is written to the database. For example, you may want to change the event format in the new version of your app, but still want to support legacy users, so you add translation procedure which transforms both types of events to the format which fits the database schema. It's basically a question of whether to work with diffs vs snapshots.
You should be aware of added complexity of working with events, and it can be an overkill if your app is simple and changes in schema are unlikely.
Also consider that you can do data preprocessing using DynamoDB Streams, which gives you some advantages of using events still keeping it simple to implement.
Nowadays a lot of web applications are providing API for other applications to use.
I am new to the usage of API so I want to understand the use cases for it.
Lets take Basecamp as an example.
What are the use cases for using their API in my web application?
For inserting current data in my web application into a newly created Basecamp account instead of inserting everything manually which could take days or weeks if the data is huge?
For updating my application data when the user changes something in Basecamp. If so, how do I know for example when a user add/edit/remove a contact in Basecamp. Do I make a request and check every minute from the backend?
For making backup of the Basecamp data so I can move it to other applications if necessary?
Are all the above examples good use cases for the usage of API?
Are there more use cases?
I want to have a clear picture of why it's good to use another web service API and how I can leverage that on my application.
Thanks.
I've found the biggest reason to use and provide web services is to be able to programmatically drive the application with another process. This allows the coupling of different actions in different applications driven by one event/process/trigger.
For example I could create a use a webservice provided by Basecamp, my bug tracking database and the continuous integration server. I could tie all those things together and kick them off from a commit hook script.
I can have a monitor in production automatically open a ticket in our ticket tracker. This could trigger an autoremediation process from the ticket tracker which logs into the box remotely and restarts the service.
The other major reason I've seen to use and provide web service is to reduce double entry. If you do change management in your production environment that usually means you create Change tickets. The changes that occur may also need to be reflected in the Change Management Database which is usually a model of how production is suppose to look. Most of these systems don't automatically drive the update of your configuration item with the data from the change. Using web services you can stitch them together to eliminate the double (manual) entry that would normally occur.
APIs are used any time you want to get data to/from an application without using the default interface.
*I'd bet there's a mobile app would use the basecamp api.
*You could use the api to pull information from basecamp into another application (like project manager software or an individual's todo webpage)
*the geekiest of us may prefer to update basecamp from a script/command line rather than interrupting our work flow to open a web page and click around.