I'm currently writing logs to Cloud logging via google-fluentd which is great. Now I want to filter and retrieve those entries. The google-fluentd agent writes v1 format entries using the v1 API.
From what I gather the v1 API has no method to retrieve entries ( v1beta3.projects.logs.entries only exposes a 'write' method. Is this true?
The v2 API exposes a list method for entries (v2beta1.entries.list), however it doesn't seem to be able to return v1 entries.
I'm in disbelief that it isn't possible to somehow retrieve v1 entries (the web console does it after all). Can someone clear this up for me?
(I work for Google on Cloud Logging)
The v1 and v2 APIs store & retrieve the same underlying data, so you can read back entries using the v2 API that were written using the v1 API. Cloud Logging will automatically translate between the LogEntry representations.
You might be having trouble reading the entries back using the v2 API because some of the LogEntry fields are named differently, and the filter field in the entries.list request expects v2-style field names.
For example, to read back the most recent entry from the "syslog" log stream in the project ID "myproject" you could issue an API request like this:
POST https://logging.googleapis.com/v2beta1/entries:list
{
"filter": "logName=projects/myproject/logs/syslog",
"orderBy": "timestamp desc",
"pageSize": 1,
"projectIds": ["myproject"]
}
The gcloud beta logging read command in the Cloud SDK is also useful for experimenting with requests and seeing the structure of the entries being returned. You can pass the --log-http flag to have it show you the HTTP requests it's making, and --format=json to get the output in JSON instead of YAML.
Related
When using the Dev Tool to PUT data to a newly created index in OpenSearch the response is 404.
Request failed to get to the server
I am following the AWS tutorial using timeseries data instead.
// Works
PUT analytics-index
// Resolves to 404
PUT analytics-index/_doc/1
{
"foo": "bar",
}
Have you created a time-series collection? In that case,
For time series collections, you can't index by custom document ID. This operation is reserved for search use cases.
(docs)
This should be a very easy question but I can't wrap my head around what to use. I would like to create a data pipeline that fetches data from an outside/external API (for example, Spotify API) and perform some rather simple data cleaning on it, while either continue to create a JSON file in Cloud Storage or enter the data into BigQuery.
As far as I understand I can use Composer to do it, using DAGS etc but what I need here is something more simple/lightweight (mainly UI based) that doesn't cost as much as Composer does as well as being easier to use. What I am looking for is something like Data Factory in Azure.
So, in brief:
Login to a data source using username/password
Extract data from a well known format (CSV/Json)
Transform data, such as remove columns, perform simple filtering like date filtering.
Reformat the data into another format (JSON/CSV/BigQuery)
...without having to code everything from scratch.
Can I handle all of this with one GCP application or do I need to use combinations like Cloud Scheduler, Cloud Functions etc?
As always, you have several options...
Cloud Scheduler seems to be a requirement to trigger regularly the process (up to every minutes).
Then, you have 2 options:
Code the process: API Call, transform/clean the data, sink the data into the destination
Use Cloud Workflow: you can define the API calls that you want to do
Call the API
Store the raw data in BigQuery (API Call also, you have connectors to simplify the process)
Run a query in BigQuery to clean/format your data and store them into a final table (API Call also)
You can also perform a mix between Cloud Functions to get the data and clean/format the data with a query in BigQuery.
Doing something specific like that without starting from scratch... difficult...
EDIT 1
If you have a look to the documentation, you can see that sample
- getCurrentTime:
call: http.get
args:
url: https://us-central1-workflowsample.cloudfunctions.net/datetime
result: currentTime
- readWikipedia:
call: http.get
args:
url: https://en.wikipedia.org/w/api.php
query:
action: opensearch
search: ${currentTime.body.dayOfTheWeek}
result: wikiResult
- returnResult:
return: ${wikiResult.body[1]}
The first step getCurrentTime performs an external call and store the result in result: currentTime.
In the next step, you can reuse the result currentTime and get only the value that you want in another API call.
And you can plug steps like that.
If you need authentication, you can perform a call to secret manager to get the secret values and then to result the secret manager call result in subsequent steps.
For an easier connection to Google APIs, you can use connectors
I would like to have a monitoring on my Hasura API on Google Cloud Run. Actually I'm using the monitoring of Google Cloud but It is not really perfect. I have the count of 200 code request. But I want for example, the number of each query / mutation endpoint request.
I want :
count 123 : /graphql/user
count 234 :/graphql/profil
I have :
count 357 : /graphql
If you have an idea.
Thanks
You can't do this with GraphQL unfortunately. All queries are sent to the /v1/graphql endpoint on Hasura, and the only way to distinguish the operations is by parsing the query parameter of the HTTP request and grabbing the operation name.
If Google Cloud allows you to query properties in logs of HTTP requests, you can set up filters on the body, something like:
"Where [request params].query includes 'MyQueryName'"
Otherwise your two options are:
Use Hasura Cloud (https://hasura.io/cloud), which gives you a count of all operations and detailed metrics (response time, variables, etc) on your console dashboard
Write and deploy a custom middleware server or a script for a reverse proxy that handles this
Is there a simple way to retrieve all items from a DynamoDB table using a mapping template in an API Gateway endpoint? I usually use a lambda to process the data before returning it but this is such a simple task that a Lambda seems like an overkill.
I have a table that contains data with the following format:
roleAttributeName roleHierarchyLevel roleIsActive roleName
"admin" 99 true "Admin"
"director" 90 true "Director"
"areaManager" 80 false "Area Manager"
I'm happy with getting the data, doesn't matter the representation as I can later transform it further down in my code.
I've been looking around but all tutorials explain how to get specific bits of data through queries and params like roles/{roleAttributeName} but I just want to hit roles/ and get all items.
All you need to do is
create a resource (without curly braces since we dont need a particular item)
create a get method
use Scan instead of Query in Action while configuring the integration request.
Configurations as follows :
enter image description here
now try test...you should get the response.
to try it out on postman deploy the api first and then use the provided link into postman followed by your resource name.
API Gateway allows you to Proxy DynamoDB as a service. Here you have an interesting tutorial on how to do it (you can ignore the part related to index to make it work).
To retrieve all the items from a table, you can use Scan as the action in API Gateway. Keep in mind that DynamoDB limits the query sizes to 1MB either for Scan and Query actions.
You can also limit your own query before it is automatically done by using the Limit parameter.
AWS DynamoDB Scan Reference
I am trying to get a list of our streams from Akamai's Media Services API v1; however, I'm only getting a 504 Gateway Timeout. I believe this is due to the amount of streams being queried for, as I can get a single stream when I give the ID for the GET stream/{streamId} endpoint.
I am using the edgegrid node module and have tried adding query parameters such as limit, pageSize/page-size, and stream-name/streamName to limit my results to no avail.
Is there a way to limit the results in a query for this API?
edit:
We are using v1 of the API
If you're using v2 of the API, it should automatically return the results paged. The API endpoint /config-media-live/v2/msl-origin/streams includes the ability to specify query string parameters page and pageSize. page is an integer that tells the API which page number to return. pageSize is also an integer referring to the number of entries the page should contain.
Example: /config-media-live/v2/msl-origin/streams?page=2&pageSize=10&sortKey=createdDate&sortOrder=ASC