I have exported Firestore collections to Google Big Query to make data analysis and aggregation.
What is the best practice (using Google Cloud Products) to serve Big Query outputs to a client web application?
Google provides seven client libraries for BigQuery. You can take any library and write a webserver that will serve requests from client web application. The webserver can use a GCP service account to access BigQuery on behalf of its clients.
One such sample is this project. It's written in TypeScript. Uses NodeJS library on the server and React for the client app. I'm the author.
You may try to have an express tour through Google Data Studio, looking for the main features what this Google analytics service can offer for the customers. If your aim stands for visualizing data from Bigquery, Data Studio is a good option, thus it provides a variety of informative dashboards and reports, allowing the user customize charts and graphs sharing them publicly or via user collaboration groups.
Data Studio spreads a lot of connectors to different data sources, hence you can find a separate Bigquery connector for further integration with data resources residing in Bigquery warehouse.
You can track for any future product enhancements here.
Related
I'm trying to find a way to retrieve a specific Google Merchant Center report (BestSellers_TopProducts_) and upload it to BigQuery as part of a specific ETL process we're developing for a customer we have at my workplace.
So far, I know you can set up the BigQuery Data Transfer service so it automates the process of downloading this report but I was wondering if I could accomplish the same with Python and some API libraries from Google (like python-google-shopping) but I may be overdoing it and setting up the service is the way to go.
Is there a way to accomplish this rather than resorting to the aforementioned service?
On the other hand, and assuming the BigQuery Data Transfer service is the way to go, I see (in the examples) you need to create and provide the dataset you're going to extract the report data to so I guess the extraction is limited to the GCP project you're working with.
I mean... you can't extract the report data for a third-party even if you had the proper service account credentials, right?
This is about a Reporting Server solution.
I need some advice to choose a product, which will hold a SQL Database Server and a Web Service App (one that will make a call to a stored procedure and run an SSIS package - not much processing here -) and SSRS. I'm not familiar with this, it needs to be available 24/7, as I said there's no much processing just synchronizing data (few hundreds of thousands of records), what do you suggest me?
Requirements:
SQL Server Enterprise 2017: this will hold the database and execute
the SSIS package.
We have an SSIS package that will be executed from a .Net Web Service app which will execute a Stored Procedure on users demand.
The Server needs to run Reporting Services (SSRS).
Considerations:
Storage: Database will hold around 750K records (all text).
Bandwidth: There will be synchronization (data retrieval or updates
only) with an external system.
Use: the client has asked to consider a dedicated instance since they
will use it at their own discretion.
Now the only issue is, as far as I know, we can't call a Stored Procedure from the outside system (outside the server), or at least I have not found a way to do that, that's why I want to host both solutions in one place, so the Web Service App can call the Stored Procedure Locally.
So now I'm wondering, what should I do? should I leverage a full VM? how much will cost?
If you want to do PaaS and not have to manage infrastructure, take a look at the Azure App Service Environment is an Azure App Service feature that provides a fully isolated and dedicated environment for securely running App Service apps at high scale. This capability can host your:
Windows web apps
Linux web apps
Docker containers
Mobile apps
Functions
For SQL you can use Azure SQL Database Managed instance,a new deployment option of Azure SQL Database, providing near 100% compatibility with the latest SQL Server on-premises (Enterprise Edition) Database Engine, providing a native virtual network (VNet) implementation that addresses common security concerns, and a business model favorable for on-premises SQL Server customers. This is a fully isolated instance of SQL server.
I suggest you host a static site on blob, an Azure function on consumption model to make calls to SQL database and a SQL database. Of course, there are alternative architecture you can use, however all depends on detailed requirements.
I hope to upload data from Android app to Google Cloud Platform and do some basic machine learning/statistic operations. I have used firebase and upload the generated data on Android app to 'realtime database' on firebase of Google cloud platform. My next goal is to do some data processing, such as simple statistic and machine learning operations, I do not know how 'realtime database' could support these operations? If not, it seems Google Cloud Platform can do such operations in MySQL, how I transfer the data in 'realtime database' on firebase on My SQL? I am the fresh guy in GCP, hope get a clear direction. Thank you
You can use firebase_admin library to access Realtime Database data. Then you can either store it using one of many google cloud client libraries or use it directly in ML Job.
What I already have:
An asp.net core on .NET framework project which uses DocumentDB as its storage
An Azure WebJob which listens to a queue that my web project writes messages to for e-mail sending and other processing
Successfully deployed and running on Azure
This all works fine. In addition to the web project there is a Model and Data class library to separate the application into layers.
Currently, the web application invokes a web service and will save the result (a quite large xml document) in the cache and keep it there for 24 hours. This is not ideal as it takes a long time the first time. What I want instead is a nightly batch job which invokes this web service and then stores (overwrites) the response into persistent storage which the web application will then use instead.
I'm confused about which Azure "service" to use for this. What I have started on so far is another WebJob and the idea is to use the same DocumentDb storage to persist the web service response every night. However, I already have all the database repository etc. set up in the web application (Data class library), is it ok to just reference this project from the WebJob instead of having to rewrite some of the same code in the WebJob?
Is it better to use some of the other Azure storage options for this WebJob instead? Like Table Storage, Blob Storage etc? Basically the structure of the data received from the web service is very simple. For each item I just need to store a url, a title, description and unique product id. Obviously the web application needs to access this storage too by simply looking up the product id, and never writing to this storage.
Also, I'm not entirely sure if there is a better alternative than Azure WebJobs for this task, but it seems like the right approach.
Any feedback is appreciated. I'm generally just confused/overwhelmed by all the different services that Azure provides.
I'll answer some of your questions...
A webjob works fine for this task. If you have a webservice that is always on adding another webjob seems like a good idea. If your webservice isn't always on, you could have a look at Azure Functions. Azure Functions is sometimes called webjobs 2.0.
As for storage in Document DB there is a file size limit for 2MB (give or take). So, you'll have to find another solution there. I think that Azure Tables also have limitations on storage size, so you'll have to split the file in smaller chunks. So, recommended solution is to go with Azure Blobs.
You'll find some good reading in this answer regarding Blobs vs Tables vs SQL
- Getting started with Azure storage: Blobs vs Tables vs SQL Azure
i want a simple hosted data store for licensed for business applications. i want the following features:
REST-like access for CRUD operations (primarily adding records)
private and authenticated
makes for easy integration with a front end charting client like Google Visualization Apis
easy to use and set up
what about:
* Google Fusion Tables
* Google Cloud Services
* Google BigQuery
* Google Cloud SQL
or other non-google products. but i am imagining a cleaner integration between Google Charts and one of their backend data services.
Pros, Cons, Advice?
First, since this is Stack Overflow, I won't attempt to provide a judgement about how about "easy to use and setup" - that can be done by you reading the documentation for each product.
That being said, overall, the "right" answer really depends on what you are trying to do, and how much data you have. It also depends on what type of application you are building (this is Stack Overflow, so I am assuming you are a developer).
Relational Databases (like Google Cloud SQL) are great for maintaining transactional consistency but once your data grows massive it becomes difficult, expensive, or impossible to run analysis queries in a reasonable timeframe.
Google BigQuery is an analysis tool that allows developers to ask questions about really really big datasets using an SQL like language. It is 100% cloud based and is accessed via RESTful API - but it only allows for appending data, not changing individual records.