I am trying to utilize SSM's GetParametersByPath API and I am getting outdated results which looks like a ~3 seconds caching by Path parameter.
I am testing the following scenario:
Get parameters by path recursively
Put a parameter under the tested path
Get parameters by path again using same arguments as in (1)
I am getting the same response in step (3) as in step (1) regardless of the changes in step (2).
I am thinking that this is related to caching because the following 2 scenarios work as expected.
Correct behavior scenario 1:
Get parameters by path recursively
Put a parameter under the tested path
Sleep for 3 seconds
Get parameters by path again using same parameters as in (1)
Correct behavior scenario 2:
Put a parameter under the tested path
Get parameters by path recursively
This behavior is consistent across different SDKs which I tried: .Net, Python (boto3) and CLI, so this is not an SKD issue.
Here is a code snippet in Python with boto3 that replicates incorrect behavior:
import boto3
client = boto3.client('ssm')
first = client.get_parameters_by_path(
Path='/param-store-test/1',
Recursive=True,
WithDecryption=True,
MaxResults=10)
print(first['Parameters'][0]['Version'] if first['Parameters'] else None)
put_response = client.put_parameter(
Name='/param-store-test/1/2',
Value='test',
Type='SecureString',
KeyId='alias/aws/ssm',
Overwrite=True,
Tier='Standard')
print("v{}".format(put_response['Version']))
second = client.get_parameters_by_path(
Path='/param-store-test/1',
Recursive=True,
WithDecryption=True,
MaxResults=10)
print(second['Parameters'][0]['Version'] if second['Parameters'] else None)
This code gives me the following output when run for the first time:
None
v1
None
And when run for the second time:
1
v2
1
You can see the pattern - the first request is being cached.
According to API docs: Request results are returned on a best-effort basis.
So is this behavior considered to be correct?
Does this mean that I don't have any way to get all parameters by path in a reliable way?
I have noticed that get_parameters_by_path() does not work as I expected.
You might think that (simplified code):
put_parameter('/abc/def', "123")
get_parameters_by_path('/abc/def')
would return "123", whereas it seems to return [] nothing.
However,
get_parameter('/abc/def')
correctly returns "123".
Also,
get_parameters_by_path('/abc')
will return '/abc/def' => "123"
So it seems to work, but not quite as one might expect.
Related
I have taken a Flyway lambda from this repo and I am trying to implement a custom resource that invokes the lambda whenever the migration source folder changes. It works fine but occasionally the response object exceeds the 4096 byte limit and the stack fails to deploy. Below is my custom resource code:
flyway_resource = cr.AwsCustomResource(
self,
f"FlywayCustomResource-{construct_id}",
log_retention=RetentionDays.ONE_WEEK,
on_update=cr.AwsSdkCall(
service="Lambda",
action="invoke",
physical_resource_id=cr.PhysicalResourceId.of("FlywayTrigger"),
parameters={
"FunctionName": flyway_lambda_function.function_name,
"InvocationType": "RequestResponse",
"Payload": "{" +
"\"flywayRequest\":{\"flywayMethod\": \"migrate\"}," +
" \"assetHash\": \"" +
# The hash of this asset (i.e., the local migration folder) changes when the content
# of the folder changes and hence an UPDATED life cycle message is produced.
# (i.e., the params of the function have changed)
assets.Asset(self, "hashId", path=migrations_source).asset_hash +
"\"}"
},
# This would avoid lambda telling output size too big when erroring, we must use the right selector(s)
# output_paths=['info', 'errorMessage', 'errorType']
),
policy=cr.AwsCustomResourcePolicy.from_statements([
iam.PolicyStatement(
actions=["lambda:InvokeFunction"],
resources=[flyway_lambda_function.function_arn]
)
])
)
I know that I can use the output_paths parameter to limit the fields in the response that will be actually selected, but I am not quite sure I have the correct field names in the list. Here's a successful example response of the lambda:
For failure I have names such as 'errorMessage', 'errorType' in the root level (the same level that 'info' exists). I have two questions:
Is the commented output_paths parameter correct? I want to verify that the values of this parameter must match the keys in the json response of the lambda. I have a hard time understanding this in the docs. (If I uncomment it works but it might be because the the specified paths don't exist and the response becomes empty and hence does not exceed the limit anymore)
Assuming that 1) is correct how do I verify the response of the custom resource. In the case that the response in erroneous I would like for the deployment to fail. I know that I can do cr.get_response_field so with cr.get_response_field('info') or cr.get_response_field('errorMessage') perhaps I could check which of these keys exists in another lambda backed custom resource. Is there a better way?
If have multiple databases defined on a particular ArangoDB server, how do I specify the database I'd like an AQL query to run against?
Running the query through the REST endpoint that includes the db name (substituted into [DBNAME] below) ie:
/_db/[DBNAME]/_api/cursor
doesn't seem to work. The error message says 'unknown path /_db/[DBNAME]/_api/cursor'
Is this something I have to specify in the query itself?
Also: The query I'm trying to run is:
FOR col in COLLECTIONS() RETURN col.name
Fwiw, I haven't found a way to set the "current" database through the REST API. Also, I'm accessing the REST API from C++ using fuerte.
Tom Regner deserves primary credit here for prompting the enquiry that produced this answer. I am posting my findings here as an answer to help others who might run into this.
I don't know if this is a fuerte bug, shortcoming or just an api caveat that wasn't clear to me... BUT...
In order for the '/_db/[DBNAME/' prefix in an endpoint (eg full endpoint '/_db/[DBNAME/_api/cursor') to be registered and used in the header of a ::arangodb::fuerte::Request, it is NOT sufficient (as of arangodb 3.5.3 and the fuerte version available at the time of this answer) to simply call:
std::unique_ptr<fuerte::Request> request;
const char *endpoint = "/_db/[DBNAME/_api/cursor";
request = fuerte::createRequest(fuerte::RestVerb::Post,endpoint);
// and adding any arguments to the request using a VPackBuilder...
// in this case the query (omitted)
To have the database name included as part of such a request, you must additionally call the following:
request->header.parseArangoPath(endpoint);
Failure to do so seems to result in an error about an 'unknown path'.
Note 1: Simply setting the database member variable, ie
request->header.database = "[DBNAME]";
does not work.
Note 2: that operations without the leading '/_db/[DBNAME]/' prefix, seem to work fine using the 'current' database. (which at least for me, seems to be stuck at '_system' since as far as I can tell, there doesn't seem to be an endpoint to change this via the HTTP REST Api.)
The docs aren't very helpful right now, so just incase someone is looking for a more complete example, then please consider the following code.
EventLoopService eventLoopService;
// adjust the connection for your environment!
std::shared_ptr<Connection> conn = ConnectionBuilder().endpoint("http://localhost:8529")
.authenticationType(AuthenticationType::Basic)
.user(?) // enter a user with access
.password(?) // enter the password
.connect(eventLoopService);
// create the request
std::unique_ptr<Request> request = createRequest(RestVerb::Post, ContentType::VPack);
// enter the database name (ensure the user has access)
request->header.database = ?;
// API endpoint to submit AQL queries
request->header.path = "/_api/cursor";
// Create a payload to be submitted to the API endpoint
VPackBuilder builder;
builder.openObject();
// here is your query
builder.add("query", VPackValue("for col in collections() return col.name"));
builder.close();
// add the payload to the request
request->addVPack(builder.slice());
// send the request (blocking)
std::unique_ptr<Response> response = conn->sendRequest(std::move(request));
// check the response code - it should be 201
unsigned int statusCode = response->statusCode();
// slice has the response data
VPackSlice slice = response->slices().front();
std::cout << slice.get("result").toJson() << std::endl;
When I run this code (which will later be used to detect and extract text using Google Vision API in Python) I get the following errors:
Module 'google.cloud.vision_v1.types' has no 'Image' member pylint(no-member)
Instance of 'ImageAnnotatorClient' has no 'text_detection' member pylint(no-member)
from google.cloud import vision
from google.cloud.vision import types
import os, io
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = r'C:\Users\paul\VisionAPI\key.json'
client = vision.ImageAnnotatorClient()
FILE_NAME = 'im3.jpg'
FOLDER_PATH = r'C:\Users\paul\VisionAPI\images'
with io.open(os.path.join(FOLDER_PATH , FILE_NAME), 'rb') as image_file:
content = image_file.read()
image = vision.types.Image(content=content)
response = client.text_detection(image=image)
What does "Module/Instance of ___ has no members" mean?
I was able to reproduce the pylint error, though the script executes successfully when run (with minor modifications for my environment to change the filename being processed).
Therefore, I am assuming that by "run this code" you mean "run this code through pylint". If not, please update the question with how you are executing the code in a way that generates pylint errors.
This page describes the specific error you are seeing, and the case that causes a false positive for it. This is likely exactly the false positive you are hitting.
The Google Cloud Vision module appears to dynamically create these members, and pylint doesn't have a way to detect that they actually exist at runtime, so it raises the error.
Two options:
Tag the affected lines with a # pylint: disable=no-member annotation, as suggested in the page linked above.
Run pylint with the --ignore-modules=google.cloud.vision_v1 flag (or put the equivalent in your .pylintrc). You'll notice that even the actual module name is different than the one you imported :)
This is a similar question with more detail about workarounds for the E1101 error.
I'm trying to cache a set of strings per session by storing each one in their own variable and by using django.contrib.session.
I have the following code:
import copy
def get_result(request, operation):
previous_result = request.session.get(operation.name)
if previous_result:
result = copy.deepcopy(previous_result)
else:
result = get_json_response(operation)
request.session[operation.name] = copy.deepcopy(result)
return result
get_result() is
triggered via ajax requests
used for many different operations which may be called at the same time
may be called multiple times per operation in one session
This code works perfectly fine on my local environment. However, in production server where gevent and chausette is installed, it fails.
Most of the time, request.session.get(operation.name) would return None even when it is not the first time that get_result is called for that operation. In some cases, it returns a value but in some, it doesn't. There seems to be no pattern on when it does and doesn't work.
I suspect that the inconsistency is because different threads are referencing the session variable at different states. What would be the proper way to handle session variables in this case?
I did in fact have the same problems and also tried to save the session properly with the tweaks you posted.
In the end, what solved my problem was changing the default cache in settings.py to
'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
Using FileBasedCache instead helps as well, but it crashes in the local environment (development). Dummy works for local as well as production.
I have a very similar problem to the one described in this question. (The main difference being that i am using Grails 2.1).
Basically i want to write a unit test which does sth like:
// set params so i can create an obj.
def results = controller.save()
// validate results or model/view
results = controller.edit(id, version)
However, this doesn't work, because the first method action (save) has triggered a redirect and the second one would too, which leads to a CannotRedirectException.
With the message:
Cannot issue a redirect(..) here. A previous call to redirect(..) has already redirected the response.
The proposed solution for this (found in various places) is:
redirectArgs.clear()
However, this doesn't work, because since Grails 2, redirectArgs doesn't exist, there is only
response.redirectUr
which can only be read, not written to.
I have also already tried these things (in various combinations):
response.reset()
clearGrailsWebRequest()
bindGrailsWebRequest()
cleanupGrailsWeb()
controller.redirect([:])
controller.redirect(null)
controller = new OfferObjectController()
controller = mockController(OfferObjectController)
All without any luck.
I have totally run out of ideas what else could work/help.
Any suggestion is appreciated!
==== EDIT ====
i forgot to clarify that
response.reset()
kind of works, it lets me call another action, however it seems to undo/rollback the changes done by the previous action, which is exactly what i want to avoid (if i wanted that, i would put the call into its own test[method]).
The answer is
response.reset()
I don't know why you're receiving that exception. By the way, you should create one task one test.
I guess you can try this:
import org.codehaus.groovy.grails.web.servlet.mvc.GrailsWebRequest
.....
// set params so i can create an obj.
def results = controller.save()
def lastRequest = GrailsWebRequest.lookup().currentRequest
lastRequest.removeAttribute("org.codehaus.groovy.grails.REDIRECT_ISSUED")
// validate results or model/view
results = controller.edit(id, version)