We started observing following error from yesterday while we are trying to access Consumer Group using SendListen Key. It is throwing Unauthorized error. If I use RootManagerSharedAccessKey, this works fine.
var connectionString = "Endpoint=sb://*****.servicebus.windows.net/;SharedAccessKeyName=SendListen;SharedAccessKey=****";
var ns = NamespaceManager.CreateFromConnectionString(connectionString);
var result= ns.GetConsumerGroupAsync("**", "**").GetAwaiter().GetResult();
Exception:
'The remote server returned an error: (401) Unauthorized. TrackingId:e2993246-9253-4cb7-ab54-2a9b32abfec7, SystemTracker:*.windows.net:dataoffload/ConsumerGroups/, Timestamp:8/11/2017 12:33:46 AM'
Any recent changes in EventHub?
According to your description, I checked this issue on my side. Here is my test, you could refer to it.
SETTING > Shared access policies
Code snippet:
var connString = "Endpoint=sb://{my-hubname}.servicebus.windows.net/;SharedAccessKeyName=SendListen;SharedAccessKey=aTu3sxJ1TMLXDWN6ErkHp28aVL/Fstfa49MkuibbQ7A=";
var ns = Microsoft.ServiceBus.NamespaceManager.CreateFromConnectionString(connString);
var result = ns.GetConsumerGroupAsync("myhub01", "$Default").GetAwaiter().GetResult();
Upon the settings, I could retrieve the description of my consumer group. Also, for the single Listen policy, it could work as well. Based on your error, I would recommend you checking your CONNECTION STRING. Also, you could leverage fiddler to collect the network traces for details.
Related
I'm getting
AccessDeniedException: User: arn:aws:iam::[ACCOUNTID]:user/esc-user-name is not authorized to perform: ssm:GetParametersByPath on resource: arn:aws:ssm:eu-central-1:[ACCOUNTID]:* because no identity-based policy allows the ssm:GetParametersByPath action
expection while reading the Second page, while Fist page of variables is fetched successfuly.
While deploying the Next.js application via ECS, I need to get relevant env variables during instance startup. The ESC-related role is permitted to read params on a particular path, like
/my-nextjs-app/stg/
I have no problems getting the expected result for the first page of params with
(code is not exactly the same for simplicity)
const initialCommand = new GetParametersByPathCommand({
Path: "/my-nextjs-app/stg/",
WithDecryption: true,
Recursive: true,
});
const response = await ssmClient.send(initialCommand)
As soon I receive a NextToken in the response, and trying to use it to fetch the next page in kind-of this way:
const nextCommand = new GetParametersByPathCommand({
Path: "/my-nextjs-app/stg/",
WithDecryption: true,
Recursive: true,
NextToken: response.NextToken,
});
await ssmClient.send(nextCommand)
And I got the permission denied error mentioned abowe.
It feels like when NextToken is defined in the command, SSMClient just ignores the Path param, and tries to use Token as source of all required data (I guess it somehow encoded into it, including pagination)
Giving permission to a whole arn:aws:ssm:eu-central-1:[ACCOUNTID]:* is not an option due to security reasons, and feels "dirty" anyway. My assumption is that if SSMClient was able to fetch the first page successfully - it should be able to proceed with the next ones as well with no additional permissions.
Meanwhile, using boto3 - all good with the same user\role.
Is it worth a bug report to #aws-sdk/client-ssm, or is there anything I've missed?
Intermittently getting the following error when connecting to an AWS keyspace using a lambda layer
All host(s) tried for query failed. First host tried, 3.248.244.53:9142: Host considered as DOWN. See innerErrors.
I am trying to query a table in a keyspace using a nodejs lambda function as follows:
import cassandra from 'cassandra-driver';
import fs from 'fs';
export default class AmazonKeyspace {
tpmsClient = null;
constructor () {
let auth = new cassandra.auth.PlainTextAuthProvider('cass-user-at-xxxxxxxxxx', 'zzzzzzzzz');
let sslOptions1 = {
ca: [ fs.readFileSync('/opt/utils/AmazonRootCA1.pem', 'utf-8')],
host: 'cassandra.eu-west-1.amazonaws.com',
rejectUnauthorized: true
};
this.tpmsClient = new cassandra.Client({
contactPoints: ['cassandra.eu-west-1.amazonaws.com'],
localDataCenter: 'eu-west-1',
authProvider: auth,
sslOptions: sslOptions1,
keyspace: 'tpms',
protocolOptions: { port: 9142 }
});
}
getOrganisation = async (orgKey) => {
const SQL = 'select * FROM organisation where organisation_id=?;';
return new Promise((resolve, reject) => {
this.tpmsClient.execute(SQL, [orgKey], {prepare: true}, (err, result) => {
if (!err?.message) resolve(result.rows);
else reject(err.message);
});
});
};
}
I am basically following this recommended AWS documentation.
https://docs.aws.amazon.com/keyspaces/latest/devguide/using_nodejs_driver.html
It seems that around 10-20% of the time the lambda function (cassandra driver) cannot connect to the endpoint.
I am pretty familiar with Cassandra (I already use a 6 node cluster that I manage) and don't have any issues with that.
Could this be a timeout or do I need more contact points?
Followed the recommended guides. Checked from the AWS console for any errors but none shown.
UPDATE:
Update to the above question....
I am occasionally (1 in 50 if I parallel call the function (5 concurrent calls)) getting the below error:
"All host(s) tried for query failed. First host tried,
3.248.244.5:9142: DriverError: Socket was closed at Connection.clearAndInvokePending
(/opt/node_modules/cassandra-driver/lib/connection.js:265:15) at
Connection.close
(/opt/node_modules/cassandra-driver/lib/connection.js:618:8) at
TLSSocket.
(/opt/node_modules/cassandra-driver/lib/connection.js:93:10) at
TLSSocket.emit (node:events:525:35)\n at node:net:313:12\n at
TCP.done (node:_tls_wrap:587:7) { info: 'Cassandra Driver Error',
isSocketError: true, coordinator: '3.248.244.5:9142'}
This exception may be caused by throttling in the keyspaces side, resulting the Driver Error that you are seeing sporadically.
I would suggest taking a look over this repo which should help you to put measures in place to either prevent the occurrence of this issue or at least reveal the true cause of the exception.
Some of the errors you see in the logs you will need to investigate Amazon CloudWatch metrics to see if you have throttling or system errors. I've built this AWS CloudFormation template to deploy a CloudWatch dashboard with all the appropriate metrics. This will provide better observability for your application.
A System Error indicates an event that must be resolved by AWS and often part of normal operations. Activities such as timeouts, server faults, or scaling activity could result in server errors. A User error indicates an event that can often be resolved by the user such as invalid query or exceeding a capacity quota. Amazon Keyspaces passes the System Error back as a Cassandra ServerError. In most cases this a transient error, in which case you can retry your request until it succeeds. Using the Cassandra driver’s default retry policy customers can also experience NoHostAvailableException or AllNodesFailedException or messages like yours "All host(s) tried for query failed". This is a client side exception that is thrown once all host in the load balancing policy’s query plan have attempted the request.
Take a look at this retry policy for NodeJs which should help resolve your "All hosts failed" exception or pass back the original exception.
The retry policies in the Cassandra drivers are pretty crude and will not be able to do more sophisticated things like circuit breaker patters. You may want to eventually use a "failfast" retry policy for the driver and handle the exceptions in your application code.
We switched to BigTable some time ago and since then there is a number of "404 requests" and also a high number of errors in the GCP Metrics console.
We see no errors in our logs and even data storage/retrieval seems to work as expected.
What is the cause for these errors and how is it possible to find out what is causing them?
As mentioned previously 404 means resource is not found. The relevant resource here is the Bigtable table (which could mean that either the instance id or table id are misconfigured in your application).
I'm guessing that you are looking at the metrics under APIs & Services > Cloud Bigtable API. These metrics show the response code from the Cloud Bigtable Service. You should be able to see this error rate under Monitoring > Metrics Explorer > metric:bigtable.googleapis.com/server/error_count and grouping by instance, method, error_code and app_profile. This will tell which instance and which RPC is causing the errors. Which let you grep your source code for incorrect usages.
A significantly more complex approach is that you can install an interceptor in Bigtable client that:
dumps the resource name of the RPC
once you identify the problematic table name, logs the stack trace of the caller
Something along these lines:
BigtableDataSettings.Builder builder = BigtableDataSettings.newBuilder()
.setProjectId("...")
.setInstanceId("...");
ConcurrentHashMap<String, Boolean> seenTables = new ConcurrentHashMap<>();
builder.stubSettings().setTransportChannelProvider(
EnhancedBigtableStubSettings.defaultGrpcTransportProviderBuilder()
.setInterceptorProvider(() -> ImmutableList.of(new ClientInterceptor() {
#Override
public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
MethodDescriptor<ReqT, RespT> methodDescriptor, CallOptions callOptions,
Channel channel) {
return new ForwardingClientCall.SimpleForwardingClientCall<ReqT, RespT>(channel.newCall(methodDescriptor, callOptions)) {
#Override
public void sendMessage(ReqT message) {
Message protoMessage = (Message) message;
FieldDescriptor desc = protoMessage.getDescriptorForType()
.findFieldByName("table_name");
if (desc != null) {
String tableName = (String) protoMessage.getField(desc);
if (seenTables.putIfAbsent(tableName, true) == null) {
System.out.println("Found new tableName: " + tableName);
}
if ("projects/my-project/instances/my-instance/tables/my-mispelled-table".equals(
tableName)) {
new RuntimeException(
"Fake error to get caller location of mispelled table id").printStackTrace();
}
}
delegate().sendMessage(message);
}
};
}
}))
.build()
);
Google Cloud Support here,
Without more insight I won’t be able to provide valid information about this 404 issue.
The issue must be either a typo or with the configuration, but cannot confirm with the shared data.
In order to provide more meaningful support, I would suggest you to open a Public Issue Tracker or a Google Cloud Support ticket.
I am running corda 4.5. My flows work perfectly when run using gradle task, deployNodes. But when I run the flow for the nodes created using the network bootstrapper, I run to the below exception.
Mon Jul 26 12:43:10 GMT 2021>>> start CreateAccount name: accountOne
▶︎ Starting
✘ Requesting signature by notary service
✘ Requesting signature by Notary service
✘ Validating response from Notary service
✘ Broadcasting transaction to participants
✘ Done
☂ java.lang.IllegalArgumentException: Flow sessions were not provided for the following transaction participants: [O=PartyA, L=New York, C=US]
From the logs:
inMemoryConfigSelectionLogger. - Did not detect a configuration for InMemory selection - enabling memory usage for token indexing. Please set stateSelection.inMemory.enabled to "false" to disable this
inMemoryConfigSelectionLogger. - No indexing method specified. Indexes will be created at run-time for each invocation of selectTokens - Cordapp configuration is incorrect due to exception - empty config: No configuration setting found for key 'stateSelection'
like Sneha said in the comments it's impossible to be confident about the source of the issues here without more context about the code here.
Remember that you want to be sure flow sessions are provided in a way similar to this where you specify the identities in a list and submit them to finality flow.
Party me = getOurIdentity();
Party notary = getServiceHub().getNetworkMapCache().getNotaryIdentities().get(0);
Command<YoContract.Commands.Send> command = new Command<YoContract.Commands.Send>(new YoContract.Commands.Send(), Arrays.asList(me.getOwningKey()));
YoState state = new YoState(me, this.target);
StateAndContract stateAndContract = new StateAndContract(state, YoContract.ID);
TransactionBuilder utx = new TransactionBuilder(notary).withItems(stateAndContract, command);
this.progressTracker.setCurrentStep(VERIFYING);
utx.verify(getServiceHub());
this.progressTracker.setCurrentStep(SIGNING);
SignedTransaction stx = getServiceHub().signInitialTransaction(utx);
this.progressTracker.setCurrentStep(FINALISING);
FlowSession targetSession = initiateFlow(this.target);
return subFlow(new FinalityFlow(stx, Collections.singletonList(targetSession), Objects.requireNonNull(FINALISING.childProgressTracker())));
}
source: https://github.com/corda/samples-java/blob/master/Features/dockerform-yocordapp/workflows/src/main/java/net/corda/samples/dockerform/flows/YoFlow.java
We have an app that uses AWS Cognito for authentication. The backend code (using AWS SDK for C# works fine mostly)
After the initial login, we obtain, ID, Access and Refresh TOKEN. Then every hour we try getting a new ID and ACCESS token by calling
public bool ExtendTokens(string userRefreshToken, out AdminInitiateAuthResponse output)
{
output = null;
AdminInitiateAuthRequest request = new AdminInitiateAuthRequest();
AdminInitiateAuthResponse response = new AdminInitiateAuthResponse();
try
{
request.UserPoolId = XXXXXXXXXXX;
request.ClientId = YYYYYYYYYY;
request.AuthFlow = AuthFlowType.REFRESH_TOKEN_AUTH;
request.AuthParameters.Add("REFRESH_TOKEN", userRefreshToken);
response = awsCognito_client.AdminInitiateAuth(request);
if (response != null)
{
output = response;
return true;
}
}
catch (Exception ex)
{
//log the exception and the inner exception!
}
return false;
}
on the backend side and passing them to the client. (The way this app works is that the client makes few calls every 3min to the server, and then server calls Cognito for authentication) then after 60min, renewing tokens the first time (60min after initial login) works fine! However, (after precisely 2hrs) the second time I get this error:
The remote server returned an error: (400) Bad Request.
Refresh Token has been revoked
For several internal/external users this happens right on the dot! 120min after they login using Username/Password. So it cannot be where a user signs out or we call GlobalSignOut accidentally. I have checked my code several places, I don't see where I might have goofed! I even use the same code module in another product and that one does not get kicked out at all! (That one does not make calls every now and then!)
Also, Tracking user devices is OFF. So it cannot be this answer.
Moreover, the Cognito Limitation document does not say anything about the total number of calls per account!
Other useful details: the default expiry of our refresh token is 15days. That's why I call this two hours expiry prematurely!
I am not able to reproduce this on my localhost, but it happens after deploying to IIS. I have checked the settings and the web.configs and I could not find any meaningful difference between the two that would invalidate my refresh tokens!
So, I was able to get around this (I still don't know the root cause) but the way to avoid this is to call the renew function every 45~50 minutes instead of waiting for that 1hr to pass! Doesn't make any sense but Its been 48hrs that my session is active and tokens are being renewed.