Connecting to QuestDB from C# using Npgsql throws error - questdb

I'm trying to connect to questdb using Npgsql in C#:
var cs = "Host=localhost;Port=8812;Username=admin;Password=quest;Database=qdb";
var connection = new NpgsqlConnection(cs);
connection.Open();
I'm getting the following error:
Npgsql.PostgresException: '00000: ',', 'from' or 'over' expected

You have to use ServerCompatibilityMode=NoTypeLoading; in the connection string.
Full example from https://questdb.io/docs/develop/query-data
using Npgsql;
string username = "admin";
string password = "quest";
string database = "qdb";
int port = 8812;
var connectionString = $#"host=localhost;port={port};username={username};password={password};
database={database};ServerCompatibilityMode=NoTypeLoading;";
await using NpgsqlConnection connection = new NpgsqlConnection(connectionString);
await connection.OpenAsync();
var sql = "SELECT x FROM long_sequence(5);";
await using NpgsqlCommand command = new NpgsqlCommand(sql, connection);
await using (var reader = await command.ExecuteReaderAsync()) {
while (await reader.ReadAsync())
{
var station = reader.GetString(0);
var height = double.Parse(reader.GetString(1));
var timestamp = reader.GetString(2);
}
}

Related

Use Aws4RequestSigner to sign PAAPI 5 Request

I'm trying to use Aws4RequestSigner in a VS2015 form to sign a search request to Amazon PAAPI.
https://www.nuget.org/packages/Aws4RequestSigner/
I get this response from the API:
{"__type":"com.amazon.paapi5#IncompleteSignatureException","Errors":[{"Code":"IncompleteSignature","Message":"The request signature did not include all of the required components. If you are using an AWS SDK, requests are signed for you automatically; otherwise, go to https://webservices.amazon.com/paapi5/documentation/sending-request.html#signing."}]}
private async void Form1_Load(object sender, EventArgs e)
{
_accessKey = "x";
_secretKey = "x";
_service = "ProductAdvertisingAPIv1";
_region = "us-east-1";
_requestUri = new Uri("https://webservices.amazon.com/paapi5/searchitems");
var payload = new
{
Keywords = "Harry",
Marketplace = "www.amazon.com",
PartnerTag = "x0d-20",
PartnerType = "Associates",
Resources = new string[] { "Images.Primary.Small", "ItemInfo.Title", "Offers.Listings.Price" },
SearchIndex = "All"
};
string jsonString = JsonConvert.SerializeObject(payload);
var content = new StringContent(jsonString, Encoding.UTF8, "application/json");
var xAmzDate = GetTimeStamp();
content.Headers.Add("content-encoding", "amz-1.0");
content.Headers.Add("x-amz-date", xAmzDate);
content.Headers.Add("x-amz-target", "com.amazon.paapi5.v1.ProductAdvertisingAPIv1.SearchItems");
var request = new HttpRequestMessage
{
Method = HttpMethod.Post,
RequestUri = _requestUri,
Content = content
};
request.Headers.Host = "webservices.amazon.com";
var contentType = new MediaTypeHeaderValue("application/json");
contentType.CharSet = "utf-8";
request.Content.Headers.ContentType = contentType;
var signer = new AWS4RequestSigner(_accessKey, _secretKey);
request = await signer.Sign(request, _service, _region);
try
{
var client = new HttpClient();
var response = await client.SendAsync(request);
if (!response.IsSuccessStatusCode)
{
var error = await response.Content.ReadAsStringAsync();
}
// response.EnsureSuccessStatusCode();
txtDisplay.Text = await response.Content.ReadAsStringAsync();
}
catch (HttpRequestException ex)
{
string error = ex.Message;
txtDisplay.Text = error;
}
}
private string GetTimeStamp()
{
return DateTime.UtcNow.ToString("yyyyMMdd\\THHmmss\\Z");
}
It could be that the headers are being added incorrectly or Aws4RequestSigner is simply outdated.

PowerBI Embedded: DeleteUserAsAdminAsync returns unauthorized

Receiving an unauthorized response when trying to remove a user from a workspace (group). I'm using the .Net SDK method DeleteUserAsAdminAsync. I tried the DeleteUserInGroupAsync method with the same result. I've had no trouble retrieving groups, reports, even adding users to groups using the same authentication code shown below. I'm quite stumped.
public async Task<ResultObject> RemoveUsersFromWorkspaces(List<PowerBIWorkspace> powerBIWorkspaces)
{
// Authenticate using created credentials
AuthenticationResult authenticationResult = null;
authenticationResult = await DoAuthentication("CustomerAppRegistration");
var tokenCredentials =
new TokenCredentials(authenticationResult.AccessToken, "Bearer");
using (var client = new PowerBIClient(
new Uri("https://api.powerbi.com/"), tokenCredentials))
{
try
{
foreach (var wksp in powerBIWorkspaces)
{
// Remove the user to the workspace.
await client.Groups.DeleteUserAsAdminAsync(new Guid(wksp.WorkspaceId), wksp.UserEmail);
}
}
catch (Exception ex)
{
var errorObject = new ResultObject();
errorObject.Success = false;
errorObject.ErrorMessage = ex.Message;
return errorObject;
}
}
var resultObject = new ResultObject();
resultObject.Success = true;
resultObject.ErrorMessage = "";
return resultObject;
}
private const string AuthorityFormat = "https://login.microsoftonline.com/{0}/v2.0";
private const string MSGraphScope = "https://analysis.windows.net/powerbi/api/.default";
private async Task<AuthenticationResult> DoAuthentication(string appRegistrationSection)
{
var config = new ConfigurationBuilder()
.SetBasePath(AppDomain.CurrentDomain.BaseDirectory)
.AddJsonFile("appsettings.json").Build();
//var section = config.GetSection(nameof(AppRegistration));
var section = config.GetSection(appRegistrationSection);
var appRegistration = section.Get<AppRegistration>();
TenantID = appRegistration.TenantId;
ClientID = appRegistration.ClientId;
ClientSecret = appRegistration.ClientSecret;
IConfidentialClientApplication daemonClient;
daemonClient = ConfidentialClientApplicationBuilder.Create(ClientID)
.WithAuthority(string.Format(AuthorityFormat, TenantID))
.WithClientSecret(ClientSecret)
.Build();
AuthenticationResult authResult =
await daemonClient.AcquireTokenForClient(new[] { MSGraphScope }).ExecuteAsync();
return authResult;
}

Connect to Azure Text Analytics from Console App without await

I am trying to call an Azure API (Text Analytics API) from a C# console application with a HttpRequest and I do not want to use any DLLs or await
but using the below snippet I am receiving "Bad Request". Can someone help me where it is going wrong.
public static void ProcessText()
{
string apiKey = "KEY FROM AZURE";
var client = new HttpClient();
var queryString = HttpUtility.ParseQueryString(string.Empty);
// Request headers
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", apiKey);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var requestUri = "https://eastus.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment?" + queryString;
//HttpResponseMessage response;
// Request body
byte[] byteData = Encoding.UTF8.GetBytes("I really love Azure. It is the best cloud platform");
using (var content = new ByteArrayContent(byteData))
{
//content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
var response = client.PostAsync(requestUri, content).Result;
Console.WriteLine(response);
Console.ReadLine();
}
}
string apiKey = "<<Key from Azure>>";
var client = new HttpClient();
var queryString = HttpUtility.ParseQueryString(string.Empty);
// Request headers
client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", apiKey);
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
var requestUri = "https://**eastus**.api.cognitive.microsoft.com/text/analytics/v2.0/sentiment?" + queryString;
//HttpResponseMessage response;
var body = new
{
documents = new[]
{
new
{
ID="1", text="I really love Azure. It is the best cloud platform"
}
}
};
string json = JsonConvert.SerializeObject(body);
byte[] byteData = Encoding.UTF8.GetBytes(json);
dynamic item = null;
using (var con = new ByteArrayContent(byteData))
{
//content.Headers.ContentType = new MediaTypeHeaderValue("application/json");
var response = client.PostAsync(requestUri, con).Result;
if (response.StatusCode == HttpStatusCode.OK)
{
string res = string.Empty;
using (HttpContent content = response.Content)
{
Task<string> result = content.ReadAsStringAsync();
res = result.Result;
}
JavaScriptSerializer serializer = new JavaScriptSerializer();
item = serializer.Deserialize<object>(res);
}
}
Hi All, I could able to get the API output using the above approach

PowerBI Embedded V2 Direct Query - Azure Sql credentials not updated programatically

I'm programatically creating a workspace and importing a Direct-Query report into PowerBI Embedded V2. Everything works fine except the report data source credentials which are not updated.
This is the code for the import flow:
await _powerBIService.ImportPbixAsync(newWorkspaceId, reportNameWithoutExtension, fileStream);
Console.WriteLine("Imported report {0} for {1} ({2})", reportNameWithoutExtension, persona.Name, persona.Id);
string datasetId = null;
while (datasetId == null)
{
//get DataSource Id
Thread.Sleep(5000);
datasetId = await _powerBIService.GetDatasetIdFromWorkspace(newWorkspaceId, reportNameWithoutExtension);
}
//update the connection details
await _powerBIService.UpdateConnectionAsync(newWorkspaceId, datasetId, persona.SQLUser, persona.SQLPassword);
Console.WriteLine("Updated connection details for dataset {0}", reportNameWithoutExtension);
// get gateway ID
var gatewayId = await _powerBIService.GetGatewayIdFromWorkspaceAndDataset(newWorkspaceId, datasetId);
if (gatewayId != null)
{
//update credentials
await _powerBIService.UpdateGatewayDatasourcesCredentials(gatewayId, persona.SQLUser, persona.SQLPassword);
Console.WriteLine("Updated connection details for gateway {0}", reportNameWithoutExtension);
}
These are the individual methods:
public async Task UpdateConnectionAsync(string workspaceId, string datasetId, string sqlUser, string sqlPwd)
{
var bearerToken = await GetBearerTokenAsync();
var tokenCredentials = new TokenCredentials(bearerToken, "Bearer");
using (var client = new PowerBIClient(new Uri(_pbiApiUrl), tokenCredentials))
{
var dataSourcesResponse = await client.Datasets.GetDatasourcesInGroupAsync(workspaceId, datasetId);
var sqlDataSources = dataSourcesResponse.Value?.Where(s => s.DatasourceType == "Sql")
.Where(s => !s.ConnectionString.Contains(_reportsSQLServer) || !s.ConnectionString.Contains(_reportsSQLServer));
foreach (var sqlDataSource in sqlDataSources)
{
var updateDataSourceConnectionRequest = new UpdateDatasourceConnectionRequest();
updateDataSourceConnectionRequest.ConnectionDetails = new DatasourceConnectionDetails(_reportsSQLServer, _reportsSQLDatabase);
//updateDataSourceConnectionRequest.DatasourceSelector = new Datasource(datasourceId: sqlDataSource.DatasourceId);
var datasourcesRequest = new UpdateDatasourcesRequest();
datasourcesRequest.UpdateDetails = new List<UpdateDatasourceConnectionRequest>() { updateDataSourceConnectionRequest };
var result = await client.Datasets.UpdateDatasourcesInGroupAsync(workspaceId, datasetId, datasourcesRequest);
//await client.Gateways.UpdateDatasourceAsync()
}
}
}
public async Task UpdateGatewayDatasourcesCredentials(string gatewayId, string sqlUser, string sqlPassword)
{
var bearerToken = await GetBearerTokenAsync();
var tokenCredentials = new TokenCredentials(bearerToken, "Bearer");
using (var client = new PowerBIClient(new Uri(_pbiApiUrl), tokenCredentials))
{
var datasourcesResult = await client.Gateways.GetDatasourcesAsync(gatewayId);
var sqlGatewayDatasources = datasourcesResult.Value?
.Where(s => s.DatasourceType == "Sql")
.Where(s => s.ConnectionDetails.Contains(_reportsSQLServer) && s.ConnectionDetails.Contains(_reportsSQLDatabase));
foreach (var gatewayDatasource in sqlGatewayDatasources)
{
var updateDataSourceRequest = new UpdateDatasourceRequest();
updateDataSourceRequest.CredentialDetails = new CredentialDetails();
updateDataSourceRequest.CredentialDetails.CredentialType = "Basic";
updateDataSourceRequest.CredentialDetails.Credentials = "{\"credentialData\":[{\"name\":\"username\", \"value\":\"" + sqlUser + "\"},{\"name\":\"password\", \"value\":\"" + sqlPassword + "\"}]}";
updateDataSourceRequest.CredentialDetails.EncryptedConnection = "Encrypted";
updateDataSourceRequest.CredentialDetails.EncryptionAlgorithm = "None";
updateDataSourceRequest.CredentialDetails.PrivacyLevel = "None";
var result = await client.Gateways.UpdateDatasourceAsync(gatewayId, gatewayDatasource.Id, updateDataSourceRequest);
}
}
}
When I attempt to view the report there is no data being fetched from the database.
Interestingly I can login manually using the PowerBI portal using the same credentials. After the manual login the report is fetching data.
If after the manual login I delete the workspace and then recreate and re-import the report programatically then the report will show data. This makes me think there is some caching involved after the manual logic which is persisted outside of the workspace.
Any suggestions on what is missing here?

PowerBi - how to authenticate to app.powerbi.com silently

I have tried the method outlined in the Microsoft docs
which involves creating an app in Active Directory and then having code something very similar to:
var authContextUrl = "https://login.windows.net/common/oauth2/authorize";
var authenticationContext = new AuthenticationContext(authContextUrl);
var redirectUri = "https://dev.powerbi.com/Apps/SignInRedirect";
var pp = new PlatformParameters(PromptBehavior.Auto);
var result = authenticationContext.AcquireTokenAsync(PowerBiApiResource, clientId, new Uri(redirectUri), pp).GetAwaiter().GetResult();
if (result == null)
{
throw new InvalidOperationException("Failed to obtain the PowerBI API token");
}
var token = result.AccessToken;
return token;
I got this code working but it always insisted on prompting for a username and password, which is a problem for a function app.
I have also tried the approach in the silent function specified here
https://community.powerbi.com/t5/Developer/Data-Refresh-by-using-API-Need-Steps/m-p/209371#M6614
static string getAccessTokenSilently()
{
HttpWebRequest request = System.Net.HttpWebRequest.CreateHttp("https://login.windows.net/common/oauth2/token");
//POST web request to create a datasource.
request.KeepAlive = true;
request.Method = "POST";
request.ContentLength = 0;
request.ContentType = "application/x-www-form-urlencoded";
//Add token to the request header
request.Headers.Add("Authorization", String.Format("Bearer {0}", token));
NameValueCollection parsedQueryString = HttpUtility.ParseQueryString(String.Empty);
parsedQueryString.Add("client_id", clientID);
parsedQueryString.Add("grant_type", "password");
parsedQueryString.Add("resource", resourceUri);
parsedQueryString.Add("username", username);
parsedQueryString.Add("password", password);
string postdata = parsedQueryString.ToString();
//POST web request
byte[] dataByteArray = System.Text.Encoding.ASCII.GetBytes(postdata); ;
request.ContentLength = dataByteArray.Length;
//Write JSON byte[] into a Stream
using (Stream writer = request.GetRequestStream())
{
writer.Write(dataByteArray, 0, dataByteArray.Length);
var response = (HttpWebResponse)request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
dynamic responseJson = JsonConvert.DeserializeObject<dynamic>(responseString);
return responseJson["access_token"];
}
}
This code doesn't work.
Also this has issues, although I haven't tried it:
https://learn.microsoft.com/en-us/power-bi/developer/get-azuread-access-token
There doesn't appear to be anything up to date available that works that explains how to do this. Does anyone know how?
This is the best I've got so far. I have to create the application in AD using https://dev.powerbi.com/apps and then login using a powerbi pro userid and password, using the following code:
public static string GetPowerBiAccessToken(string tenantId, string clientId, string userId, string password)
{
var url = $"https://login.windows.net/{tenantId}/oauth2/token";
var request = (HttpWebRequest)WebRequest.Create(url);
request.KeepAlive = true;
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
var dataToPost = new Dictionary<string,string>
{
{"client_id", clientId},
{"grant_type", "password"},
{"resource", PowerBiApiResource},
{"username", userId},
{"password", password},
{"redirect_uri", "https://dev.powerbi.com/Apps/SignInRedirect" }
};
var postData = string.Empty;
foreach (var item in dataToPost)
{
if (!string.IsNullOrEmpty(postData))
postData += "&";
postData += $"{item.Key}={item.Value}";
}
var dataByteArray = System.Text.Encoding.ASCII.GetBytes(postData);
request.ContentLength = dataByteArray.Length;
using (var writer = request.GetRequestStream())
{
writer.Write(dataByteArray, 0, dataByteArray.Length);
}
var response = (HttpWebResponse)request.GetResponse();
var responseString = new StreamReader(response.GetResponseStream()).ReadToEnd();
var responseJson = JsonConvert.DeserializeObject<dynamic>(responseString);
return responseJson["access_token"];
}
(P.S. I appreciate the request part of this code could be much better but having spent over 2 days trying to work this out, I'm just glad to have something that is working right now. Also the redirectUrl matches the one in my app, I'm not sure if its essential or not).
In the event of errors, I used Fiddler 4 to tell me exactly what the error was rather than just getting 400 or 401 in the app.