i have a database on server A which also contains the default tables and sp that used by the aspnet_Regsql.exe process (mainly i use the authentication module of asp.net with those tables) and a backup Server B, which i use sync framework sqlOrchester and SqlProvider to sync between them together.
but for some reason, when i'm trying to do the provisioning process on the remote server A
i get an error on the line that creates the SqlSyncScopeProvisioning object with exception of DbSyncException exception, stating that i'm missing a primary key column on table aspnet_personalizationperuser, although that table has the primary key column [Id] which ig a uniqueidentifier column.
I added the lines of code i'm using for the provisioning process:
var TableNames = [Collection of all the table names in the database];
var dbScopeDescription =
BuildScopeFromTableList(this.RemoteProvider, TableNames);
protected DbSyncScopeDescription
BuildScopeFromTableList(RelationalSyncProvider provider, string[] SyncTablesNames)
{
DbSyncScopeDescription scopeDesc = new
DbSyncScopeDescription(SyncSettings.ScopeName);
SqlConnection serverConn =
(SqlConnection)provider.Connection;
foreach (String table in SyncTablesNames)
{
DbSyncTableDescription tableDesc =
SqlSyncDescriptionBuilder.GetDescriptionForTable(table, serverConn);
scopeDesc.Tables.Add(tableDesc);
}
return scopeDesc;
}
var serverConfig = new SqlSyncScopeProvisioning((SqlConnection)this.remoteProvider.Connection, dbScopeDescription);
Related
I'm new to DynamoDb and the intricacies of querying it - I understand (hopefully correctly) that I need to either have a partition Key or Global Secondary Index (GSI) in order to query against that value in the table.
I know I can use Appsync to query on a GSI by setting up a resolver - and this works. However, I have a setup using the Java AWS CDK (I'm writing in Kotlin) where I'm using Appsync and routing my queries into lambda Resolvers (so that once this works, I can do more complicated things later).
The Crux of the issue is that when I setup a Lambda to resolve my query, I end up with this Error msg: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Query condition missed key schema element: testName returned from the Lambda.
I think these should be the key snippets..
My DynamoDbBean..
#DynamoDbBean
data class Test(
#get:DynamoDbPartitionKey var id: String = "",
#get:DynamoDbSecondaryPartitionKey(indexNames = ["testNameIndex"])
var testName: String = "",
)
Using the CDK I Created GSI on
testTable.addGlobalSecondaryIndex(
GlobalSecondaryIndexProps.builder()
.indexName("testNameIndex")
.partitionKey(
Attribute.builder()
.name("testName")
.type(AttributeType.STRING)
.build()
)
.projectionType(ProjectionType.ALL)
.build())
Then, within my Lambda I am trying to query my DynamoDb table, using a fixed value here testName = A.
My Sample data in the Test table would be like so..
{
"id" : "SomeUUID",
"testName" : "A"
}
private var client: AmazonDynamoDB = AmazonDynamoDBClientBuilder.standard().build()
private var dynamoDB: DynamoDB = DynamoDB(client)
Lambda Resolver Snippets...
val table: Table = dynamoDB.getTable(TABLE_NAME)
val index: Index = table.getIndex("testNameIndex")
...
QuerySpec().withKeyConditionExpression("testNameIndex = :testName")
.withValueMap(ValueMap().withString(":testName", "A"))
val iterator: Iterator<Item> = index.query(querySpec).iterator()
while (iterator.hasNext()) {
logger.info(iterator.next().toJSONPretty())
}
This is what results in this Error msg: com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Query condition missed key schema element: testName
Am I on the wrong lines here? I know there is some mixing of libs between the 'enhanced' Dynamo sdk and the dynamodbv2 sdk - so if there is a better way of doing this query, I'd love to know!
Thanks!
Your QuerySpec's withKeyConditionExpression is initialized wrong. You need to init it like.
not testNameIndex it should be testName
QuerySpec().withKeyConditionExpression("testName = :testName")
.withValueMap(ValueMap().withString(":testName", "A"))
Trying to set up a client for my Amazon DynamoDB in Java 8 and am running into this error when I try to run my lambda function locally. I am trying to connect to Amazon DynamoDB and I already have set up in AWS Management Console.
Error trying to commit audit record:com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details. (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: InvalidSignatureException;
I am still new to AWS and trying to understand how it works. I am sure the credentials I provided matched the ones I have.
AmazonDynamoDB client = AmazonDynamoDBClient.builder()
.withRegion("us-east-2")
.withCredentials(new AWSStaticCredentialsProvider(new
BasicAWSCredentials("key","private key")))
.build();
DynamoDB dynamoDB = new DynamoDB(client);
Table table = dynamoDB.getTable("tableName")
Maybe you can try out changing according to example in AWS docs, without explicitly setting credential provider.
https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/CodeSamples.Java.html
This is my code for creating a table and this is working:
BasicAWSCredentials awsCreds = new BasicAWSCredentials("access_key_id", "secret_key_id");
AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
.withRegion(Regions.US_EAST_1)
.withCredentials(new AWSStaticCredentialsProvider(awsCreds))
.build();
// AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard()
// .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration("http://localhost:8000", "us-west-2"))
// .build();
DynamoDB dynamoDB = new DynamoDB(client);
String tableName = "Songs";
try {
System.out.println("Attempting to create table; please wait...");
Table table = dynamoDB.createTable(tableName,
Arrays.asList(
new KeySchemaElement("year", KeyType.HASH), // Partition
// key
new KeySchemaElement("title", KeyType.RANGE)), // Sort key
Arrays.asList(new AttributeDefinition("year", ScalarAttributeType.N),
new AttributeDefinition("title", ScalarAttributeType.S)),
new ProvisionedThroughput(10L, 10L));
table.waitForActive();
System.out.println("Success. Table status: " + table.getDescription().getTableStatus());
} catch (Exception e) {
System.err.println("Unable to create table: ");
System.err.println(e.getMessage());
}
I want to clear the tables stored in a specific dataset of bigquery.
In the console screen, you cannot delete multiple tables at once.
It was also not possible to delete using the * in the bq CLI.
Is there a way to clear multiple tables at once?
While in the documentation it is stated that you can delete only one table at a time, it is possible to make an API request with a Python script in order to delete all the tables inside a dataset.
I created and tested the following script:
from google.cloud import bigquery
#construct BigQuery client object
client = bigquery.Client()
#select your dataset and list the tables within it
dataset_id='project_id.dataset'
tables = client.list_tables(dataset_id)
#inititalizing the list of tables
list_tables=[]
for table in tables:
#Create a list with the able reference for deletion 'project.dataset_id.table_id'
id =".".join([table.project,table.dataset_id,table.table_id])
list_tables.append(id)
#List of tables
print(list_tables)
#Delete all the tables inside the list of tables
for table in list_tables:
#print(table)
client.delete_table(table)
print("{} {}".format("Number of deleted tables in the dataset:", len(list_tables)))
I executed the above code using Jupyter Notebook with Python 3. If you run it in your cloud shell environment, make sure you install all the dependencies pip install --upgrade google-cloud-bigquery.
I had similar situation and used the following Java code to delete the large no of tables.
Replace dataset and table_prefix values.
Set the service account JSON key file path as GOOGLE_APPLICATION_CREDENTIALS environment variable.
Code:
import java.util.ArrayList;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ForkJoinPool;
public class TableKiller {
String dataset = "MY_DATASET";
String table_prefix = "temp_table_";
public static void main(String[] args) throws ExecutionException, InterruptedException {
new TableKiller().deleteAll();
}
public void deleteAll() throws InterruptedException, ExecutionException {
ArrayList<String> tableNames = new ArrayList<>();
BigQuery bigQuery = BigQueryOptions.getDefaultInstance().getService();
int i = 1;
bigQuery.listTables(dataset, BigQuery.TableListOption.pageSize(1000))
.iterateAll()
.forEach(table -> {
String tableName = table.getTableId().getTable();
if (tableName.startsWith(table_prefix)) {
tableNames.add(tableName);
System.out.println("Added " + tableName + i);
}
});
ForkJoinPool forkJoinPool = new ForkJoinPool(200);
forkJoinPool.submit(() -> tableNames
.parallelStream()
.forEach(this::deleteTable)).get();
}
private void deleteTable(String tableName) {
BigQuery bigQuery = BigQueryOptions.getDefaultInstance().getService();
bigQuery.delete(TableId.of(dataset, tableName));
System.out.println("Deleted " + tableName);
}
}
I have embedded powerbi report which was working fine until I changed my database.
I observed datasets.IsEffectiveIdentityRequired (in below code) was false earlier, now as it is true, I'm getting an error - {"error":{"code":"InvalidRequest","message":"Creating embed token for accessing dataset 02c90e15-35dd-4036-a525-4f5d158bfade requires roles to be included in provided effective identity"}}
I'm using standard Embed service code.
// Create a Power BI Client object. It will be used to call Power BI APIs.
using (var client = new PowerBIClient(new Uri(ApiUrl), m_tokenCredentials))
{
// Get a list of reports.
var reports = await client.Reports.GetReportsInGroupAsync(WorkspaceId);
Report report = reports.Value.FirstOrDefault(r => r.Id.Equals(ReportId, StringComparison.InvariantCultureIgnoreCase));
var datasets = await client.Datasets.GetDatasetByIdInGroupAsync(WorkspaceId, report.DatasetId);
m_embedConfig.IsEffectiveIdentityRequired = datasets.IsEffectiveIdentityRequired;
m_embedConfig.IsEffectiveIdentityRolesRequired = datasets.IsEffectiveIdentityRolesRequired;
GenerateTokenRequest generateTokenRequestParameters;
// This is how you create embed token with effective identities
// HERE username IS NULL
if (!string.IsNullOrWhiteSpace(username))
{
var rls = new EffectiveIdentity(username, new List<string> { report.DatasetId });
if (!string.IsNullOrWhiteSpace(roles))
{
var rolesList = new List<string>();
rolesList.AddRange(roles.Split(','));
rls.Roles = rolesList;
}
// Generate Embed Token with effective identities.
generateTokenRequestParameters = new GenerateTokenRequest(accessLevel: "view", identities: new List<EffectiveIdentity> { rls });
}
else
{
// Generate Embed Token for reports without effective identities.
generateTokenRequestParameters = new GenerateTokenRequest(accessLevel: "view");
}
var tokenResponse = await client.Reports.GenerateTokenInGroupAsync(WorkspaceId, report.Id, generateTokenRequestParameters);
}
First, I completely understand that this error occurs as I'm not passing any identity. So, is there any option to disable IsEffectiveIdentityRequired?
Second, how to set users and roles in powerbi?
--I'm not a PowerBI expert--
IsEffectiveIdentityRequired is a read only property so you can't control it and there is no option to disable it.
Depending on the data source you are connecting to an effective identity may or may not be required.
If IsEffectiveIdentityRequired is true you need to pass an EffectiveIdentity when calling GenerateTokenRequest to generate an embed token. If the data source requires an effective identity and you do not pass one you will get an error when calling GenerateTokenRequest. You will also get an error if you pass an incomplete EffectiveIdentity, such as one that is missing roles when calling GenerateTokenRequest.
Here is an example of how you can use the IsEffectiveIdentityRequired property to generate an embed token with or without an effective identity depending on if the data source requires it or not.
List<EffectiveIdentity> eil = new List<EffectiveIdentity>();
EffectiveIdentity ef = new EffectiveIdentity();
// UserName
ef.Username = FullADUsername;
// Roles
List<string> Roles = new List<string>();
ef.Roles = Roles;
// Datasets
List<string> _Datasets = new List<string>();
_Datasets.Add(report.DatasetId);
ef.Datasets = _Datasets;
eil.Add(ef);
// Look up the data set of the report and look if we need to pass an Effective Identify
Dataset d = client.Datasets.GetDatasetByIdInGroup(WorkspaceId, report.DatasetId);
if (d.IsEffectiveIdentityRequired == true){
GenerateTokenRequest gtr = new GenerateTokenRequest("View", null, false, eil);
newEmbedToken = client.Reports.GenerateTokenInGroup(WorkspaceId, ReportId, gtr);
}
else
{
GenerateTokenRequest gtr = new GenerateTokenRequest();
newEmbedToken = client.Reports.GenerateTokenInGroup(WorkspaceId, ReportId, gtr);
}
I need to have a query on DynamoDB.
Currently I made so far this code:
AWSCredentials creds = new DefaultAWSCredentialsProviderChain().getCredentials();
AmazonDynamoDBClient client = new AmazonDynamoDBClient(creds);
client.withRegion(Regions.US_WEST_2);
DynamoDB dynamoDB = new DynamoDB(new AmazonDynamoDBClient(creds));
Table table = dynamoDB.getTable("dev");
QuerySpec spec = new QuerySpec().withKeyConditionExpression("tableKey = :none.json");
ItemCollection<QueryOutcome> items = table.query(spec);
System.out.println(table);
The returned value of table is: {dev: null}, which means the that teh description is null.
It's important to say that while i'm using AWS CLI with this command: aws dynamodb list-tables i'm getting a result of all the tables so if i'm also making the same operation over my code dynamoDB.listTables() is retrieving empty list.
Is there something that I'm doing wrong?
Do I need to define some more credentials before using DDB API ?
I was getting the same problem and landed here looking for a solution. As mentioned in javadoc of getDesciption
Returns the table description; or null if the table description has
not yet been described via {#link #describe()}. No network call.
Initially description is set to null. After the first call to describe(), which makes a network call, description gets set and getDescription can be used after that.