I have input stream InputStream taking input from Kafka. Then I use this stream to call HTTP API. In HTTP call response, I have to send results to another Kafka topic using OutputStream
app flow
InputStream (have userId) => RequestHttpStream => ResponseHTTPStream (dont have userId) => OutputStream (need userId here)
The problem is, i need userId to send event to OutputStream but the response HTTP stream ResponseHTTPStream don't have notion of userId. So I am looking for ways in siddhi to store and retrieve some key/values mapping. I read about tables in siddhi but seems like it's shared between all events. What I need is kind of like an app session data which only exists for that session only. Can I use partitions for this?
code
#source(type = 'kafka',
topic.list = "user-entered-code",
bootstrap.servers = "kafka:9092",
#map(type = 'json', validate.json = "true",
#payload("""{"userId":"{{userId}}","promoCode":{{promoCode}}}""")))
define stream InputStream (userId string, promoCode string);
#sink(type = 'kafka', topic = "user-code-attribution-updated", bootstrap.servers = "kafka:9092",
#map(type = 'json', validate.json = "true",
#payload("""{"userId":"{{userId}}","attributes": {{attributes}}""")))
define stream OutputStream (userId string, attributes object);
#sink(type = 'log')
#sink(type = 'http-call', sink.id = "attribution-request", publisher.url = "http://test-node-server:3031/search-attribution-id",
#map(type = 'json'))
define stream RequestHttpStream(attributionId string);
#sink(type = 'log')
#source(type = 'http-call-response', sink.id = "attribution-request", http.status.code = "200",
#map(type = 'json'))
define stream ResponseHTTPStream (attributes object);
from InputStream
select promoCode as attributionId
insert into RequestHttpStream;
from ResponseHTTPStream
select 'need userId here' as userId, attributes as attributes
insert into OutputStream;
Your use case is to keep an attribute between http-call sink and http-call-response source. This can be achieved by using transport properties. Since the attributes in the sink can be accessed in the source by referring to trp:<attribute name>
#sink(type = 'log')
#sink(type = 'http-call', sink.id = "attribution-request", publisher.url = "http://test-node-server:3031/search-attribution-id",
#map(type = 'json',
#payload("""{"attributionId":"{{attributionId}}"""")))
define stream RequestHttpStream(attributionId string, userId string);
#sink(type = 'log')
#source(type = 'http-call-response', sink.id = "attribution-request", http.status.code = "200",
#map(type = 'json',
#attributes(attributes='attributes', userId='userId')))
define stream ResponseHTTPStream (attributes object, userId string);
However please note, you have to use custom mapping in the sink since you do not want to send the userId to the HTTP endpoint
Related
This is a follow up to my issue at API Pagination in JSON Body (not Header) - How to Access with Power Query in Custom Connector?
I have built a custom connector for an API that sends its pagination data back in the body of the response, rather than the header. The way to use GetNextLink using header is documented in the GitHub custom connector example:
https://github.com/microsoft/DataConnectors/blob/master/samples/Github/github.pq
GetNextLink = (response, optional request) =>
let
// extract the "Link" header if it exists
link = Value.Metadata(response)[Headers][#"Link"]?,
links = Text.Split(link, ","),
splitLinks = List.Transform(links, each Text.Split(Text.Trim(_), ";")),
next = List.Select(splitLinks, each Text.Trim(_{1}) = "rel=""next"""),
first = List.First(next),
removedBrackets = Text.Range(first{0}, 1, Text.Length(first{0}) - 2)
in
try removedBrackets otherwise null;
Getting to the pagination data from the body of the response should be easier than the header, I've learned. But I am stumped. In the images, you can see the record I am trying to access that comes back through Web.Contents()
This is my code for the GetNextLink function. I am trying to get the "next" record above.
GetNextLink = (response) =>
// response is data already run through Web.Contents()
let
Source = Lines.FromBinary(response),
nextPage = Record.Field(Source[paging], "next")
in
try nextPage otherwise null;
I get the error back "We cannot convert a value of type Record to type Text". I feel like I am not getting the 'Source" correctly? I could certainly use help as I am a PowerQuery newbie!
Thanks!
GetNextLink = (response) =>
// response is data already run through Web.Contents()
let Source = Lines.FromBinary(response),
nextpage = try Source[paging][next] otherwise null
in nextpage
or
GetNextLink = (response) =>
// response is data already run through Web.Contents()
let Source = Lines.FromBinary(response)
nextpage = try Record.Field(Source[paging],"next") otherwise null
in nextpage
I am trying to use google-kms nodejs library.
What I was expecting is that the end result will be encrypted text but what I get is a buffer if I don't decide base64 or if I do then I get something like $�k+��l�k��:
Does someone know what's wrong r is my expectation wrong about the encoded text.
exports.encryptKMS = async (req, res) => {
let message = req.query.message || req.body.message || 'Hello World!';
const projectId = 'xxx';
const kms = require('#google-cloud/kms');
const client = new kms.KeyManagementServiceClient();
const locationId = 'global';
const keyRingId = 'Test-Ring-01';
const cryptoKeyId = 'Test-Crypto-01';
const cryptoKeyPath = client.cryptoKeyPath(
projectId,
locationId,
keyRingId,
cryptoKeyId
);
const [result] = await client.encrypt({name: cryptoKeyPath, plaintext: message});
console.log(result);
const cryptoText = Buffer.from(result.ciphertext, 'base64').toString('utf-8');
console.log(cryptoText);
res.status(200).send(cryptoText);
}
This looks correct to me, the encrypted object will be a base64 encoded binary string which you won't be able to see any structure in. If you take that result and feed it back into Decrypt you should recover the original message.
If you wanted a readable string, please provide more info -- there's such a thing as "Format-Preserving Encryption" which attempts to encrypt data in such a fashion that it will continue to be in a compatible format to the input, for when systems aren't ready to handle encrypted data. Cloud KMS doesn't support format-preserving encryption, but I may have some suggestions depending on what you need.
Thanks for using GCP and Cloud KMS!
I have some client that uses the Power BI App.
And with this API I want to get all the reports (.pbix) to my folder.
How can I get them?
You can simply open the report in your browser and save it as .pbix file:
If you want to do this using an API, you will need Export Report In Group REST API. To use it, you need to acquire an access token and add it to your request header. You can acquire it by calling some of the AcuireToken methods from ADAL.
You can use code like this (please note there is no error checking in the example):
string clientId = "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"; // Obtain at https://dev.powerbi.com/apps
string redirectUri = "https://login.live.com/oauth20_desktop.srf";
string resourceUri = "https://analysis.windows.net/powerbi/api";
string authorityUri = "https://login.windows.net/common/oauth2/authorize";
AuthenticationContext authContext = new AuthenticationContext(authorityUri, new TokenCache()); // PM> Install-Package Microsoft.IdentityModel.Clients.ActiveDirectory
var authenticationResult = await authContext.AcquireTokenAsync(resourceUri, clientId, new Uri(redirectUri), new PlatformParameters(PromptBehavior.Auto));
var accessToken = authenticationResult.AccessToken);
string powerBIApiUrl = "https://api.powerbi.com/v1.0/myorg/groups/{groupId}/reports/{reportKey}/Export"; // Replace groupId and reportKey with actual values
var request = WebRequest.Create(powerBIApiUrl) as HttpWebRequest;
request.KeepAlive = true;
request.Method = "GET";
request.ContentLength = 0;
request.ContentType = "application/json";
request.Headers.Add("Authorization", $"Bearer {accessToken}");
using (HttpWebResponse httpResponse = request.GetResponse() as System.Net.HttpWebResponse)
{
//Read httpResponse.GetResponseStream() to get the .pbix file
}
Hi I'm new with moodle and I'm getting an error when calling the webservice.
Currently I'm trying to retrieve a user from moodle with the following function core_user_get_users_by_field and I'm using rest service to do so. I already managed to create a user thus I am authenticated to use the service.
the error that I'm receiving is
Missing required key in single structure: field
The bellow is the code was used to create a User. the issue that I got from the error is that the parameter that I need to send for the post is not formatted well. Does anyone know how to search correctly with this method or any other method.
String token = "token";
String postData = "username=username";
string createRequest = string.Format("http://domain/webservice/rest/server.php?wstoken={0}&wsfunction={1}&moodlewsrestformat=json", token, "core_user_get_users_by_field");
// Call Moodle REST Service
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(createRequest);
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
// Encode the parameters as form data:
byte[] formData =
UTF8Encoding.UTF8.GetBytes(postData);
req.ContentLength = formData.Length;
// Write out the form Data to the request:
using (Stream post = req.GetRequestStream())
{
post.Write(formData, 0, formData.Length);
}
// Get the Response
HttpWebResponse resp = (HttpWebResponse)req.GetResponse();
Stream resStream = resp.GetResponseStream();
StreamReader reader = new StreamReader(resStream);
string contents = reader.ReadToEnd();
// Deserialize
JavaScriptSerializer serializer = new JavaScriptSerializer();
if (contents.Contains("exception"))
{
// Error
MoodleException moodleError = serializer.Deserialize<MoodleException>(contents);
}
else
{
// Good
}
The webservice core_user_get_users_by_field needs an associative array given as parameter with the following key:values
'field': 'id'
'values': array of integers (must be an array, possibly with just one value)
In PHP it would be, for example:
$parameters = array('field' => 'id', 'values' => array(13));
It means: the user whose 'id' has the value of 13. Of course, you can use other parameters as well: ('field'=>'lastname', 'values'=> array('Smith'))
The parameters you can choose are the fields of the Moodle 'user' table.
Try to build these parameters in your postData variable.
Here's URL that work with my put this url in postman and set http method to post method
hostname/webservice/rest/server.php?wstoken=any_token&wsfunction=core_user_get_users_by_field&field=email&values[0]=h#fci.com
&moodlewsrestformat=json
I'm having difficulty putting together a request by sending a json content, can not find anything in the 4.x documentation it's completely different from version 3.x
RestClient client = new RestClient();
client.Host = "localhost";
client.Port = 8080;
RestRequest request = new Grapevine.Client.RestRequest("/route1");
request.HttpMethod = HttpMethod.POST;
RestResponse response = client.Execute(request) as RestResponse;
Somewhere in your code - prior to sending your request - you need to set the body (or payload) of your request.
request.Payload = "send this data";
The payload is just a string, so it's up to you to serialize your objects to a JSON string before making the assignment (and set the ContentType property appropriately). The Json.NET library is widely used to accomplish this. You can also do this by hand:
request.ContentType = ContentType.JSON;
request.Payload = "{\"key\":\"value\"}";