Configuring Celery + AWS SQS to revoke tasks - django

I am running Celery+Kombu 4.4.6 on AWS SQS and want to revoke and terminate tasks.
Reading through documentation and SO posts, the transport needs to allow broadcast messages. SQS does not do broadcast messages and Celery+Kombu needs to use SimpleDB for those. That option was turned off by default long way back in version 1.x. To enables it, support_fanout = True needs to be added to the transport options.
But adding just that option is not working for me and I can't figure out what am I missing. Possible options are:
SimpleDB - it is not clear to me how do I even enable SimpleDB. I do see documentation in AWS, but I do not see it as a separate service.
Any additional config to be added?
Looking briefly at the SQS code, seems like SimpleDB is the only option for this. Is that correct?
Any other option to enable task revocation on SQS?
In my app.celery I have:
app = Celery('app',
broker=''sqs://<Access key>:<secret key>#')),
backend='cache+memcached://<host>:11211/')),
)
And in my app.settings I have:
CELERY_BROKER_URL='sqs://<access key>:<secret key>#'))
CELERY_BROKER_TRANSPORT_OPTIONS = {
'region': '<region>',
'supports_fanout': True,
}
CELERY_DEFAULT_QUEUE = 'app'
CELERY_DEFAULT_EXCHANGE = 'app'
CELERY_DEFAULT_ROUTING_KEY = 'app'


My final solution was to use Amazon MQ with a RabbitMQ instance. Amazon SimpleDB seems to be gone, making any support in Celery+Kombu obsolete and broken.

Related

How to use librdkafka to change retention time for the running kafka topic

We can use the following command in kafka machine to update the retention time for the running kafka topic:
bin/kafka-topics.sh --zookeeper <kafka_ip> --alter --topic <target_topic> --config retention.ms=86400000
But I don't want to login to the kafka machine and run the command.
I just want to use C or C++ to change kafka retention time for the running kafka topic in the remote producer machine.
Question is: Can we use api in librdkafka to update the retention time for the running kafka topic?
note: Now we can produce and consume kafka data with C/C++.

Use rd_kafka_AlterConfigs() and pass it a TOPIC resource with all current topic configuration as well as your updated configuration retention.ms.
https://docs.confluent.io/platform/current/clients/librdkafka/rdkafka_8h.html#ade8d161dfb86a94179d286f36ec5b28e

Is there also a way to map this using the c ++ API?
Until the topic configuration is set by RdKafka::Topic::create, why the conf object doesn't support 'retention.ms', as I found here https://docs.confluent.io/platform/current/installation/configuration/topic-configs.html#retention.ms 'retention.ms' is a topic configuration.

Mock SQS Config

I've created a mule application with the following configuration to SQS. This lives in my config.xml file.
<sqs:config name="amazonSQSConfiguration" accessKey="${aws.sqs.accessKey}" secretKey="${aws.sqs.secretKey}" url="${aws.sqs.baseUrl}" region="${aws.account.region}" protocol="${aws.sqs.protocol}" doc:name="Amazon SQS: Configuration">
<reconnect count="5" frequency="1000"/>
</sqs:config>
This is problematic for me because when this SQS configuration loads up, it tries to connect to Amazon SQS queue but can't because access to the queue is not accessible from my machine.
For munit, unit purposes, I'm looking for a way to stop this from trying to connect on load?
Is there a way I can mock this sqs:config?
Please note this is different from mocking the connector in my flow? In this case I need to mock the config.
Or happy for any other suggestions.
thanks

WSO2 API Manager 2.1 : Gateway not enforcing Throttling Limits

We have deployed API-M 2.1 in a distributed way (each component, GW, TM, KM are running in their own Docker image) on top on DC/OS 1.9 ( Mesos ).
We have issues to get the gateway to enforce throttling policies (should it be subscription tiers or app-level policies). Here is what we have managed to define so far:
The Traffic Manager itself does it job : it receives the event streams, analyzes them on the fly and pushes an event onto the JMS topic throttledata
The Gateway reads the message properly.
So basically we have discarded a communication issue.
However we found two potential issues:
In the event which is pushed to the TM component, the value of the appTenant is null (instead of carbon.super)- We have a single tenant defined.
When the gateway receives the throttling message, it decides to let the message go thinking the "stopOnQuotaReach" is set to false, when it is set to true (we checked the value in the database).
Digging into the source code, we related those two issues to a single source: the value for both values above are read from the authContext and apparently incorrectly set. We are stuck and running out of ideas of things to try and would need some pointers to what could be a potential source of the problem and things to check.
Can somebody help please ?
Thanks- Isabelle.

Is there two TM with HA enabled available in the system?
If the TM is HA enabled, how gateways publish data to TM. Is it load balanced data publishing or failover data publishing to the TMs?
Did you follow below articles to configure the environment with respect to your deployment?
http://wso2.com/library/articles/2016/10/article-scalable-traffic-manager-deployment-patterns-for-wso2-api-manager-part-1/
http://wso2.com/library/articles/2016/10/article-scalable-traffic-manager-deployment-patterns-for-wso2-api-manager-part-2/
Is throttling completely not working in your environment?
Have you noticed any JMS connection related logs in gateways nodes?

In these tests, we have disabled HA to avoid possible complications. Neither subscription nor app throttling policies are working, both because parameters that should have values have not the adequate value (appTenant, stopOnQuotaReach).
Our scenario is far more basic. If we go with one instance of each component, it fails as Isabelle described. And the only thing we know is that both parameters come from the Authentication Context.
Thank you!

Why doesn't CeleryCAM work with Amazon SQS?

I'm using Celery 2.4.6 and django-celery 2.4.2.
When I configure Celery to use Amazon SQS per the resolution on this question: Celery with Amazon SQS
I don't see anything in the celerycam table in the Django admin. If I switch back to RabbitMQ, the tasks start showing up again.
I have a lot (now 40+) queues in SQS named something like this: "celeryev-92e068c4-9390-4c97-bc1d-13fd6e309e19", which look like they might be related (some of the older ones even have an event in them), but nothing's showing up in the database and I see no errors in the celerycam log.
Any suggestions on what the issue might be or how to debug this further would be much appreciated.

SQS is a limited implementation of an AMQP bus. As I understand, it doesn't support PUB/SUB broadcasting like say rabbit-MQ does, which is necessary for events to work properly. SNS was put in place to support broadcasting, but its a separate system.
Some libraries/packages out there are using SimpleDB as a messaging model store as a hack on top of SQS to emulate proper AMQP behavior, but apparently celery does not have a full hack in place yet.

Celery with Amazon SQS

I want to use Amazon SQS as broker backed of Celery. There’s the SQS transport implementation for Kombu, which Celery depends on. However there is not enough documentation for using it, so I cannot find how to configure SQS on Celery. Is there somebody that had succeeded to configure SQS on Celery?

I ran into this question several times but still wasn't entirely sure how to setup Celery to work with SQS. It turns out that it is quite easy with the latest versions of Kombu and Celery. As an alternative to the BROKER_URL syntax mentioned in another answer, you can simply set the transport, options, user, and password like so:
BROKER_TRANSPORT = 'sqs'
BROKER_TRANSPORT_OPTIONS = {
'region': 'us-east-1',
}
BROKER_USER = AWS_ACCESS_KEY_ID
BROKER_PASSWORD = AWS_SECRET_ACCESS_KEY
This gets around a purported issue with the URL parser that doesn't allow forward slashes in your API secret, which seems to be a fairly common occurrence with AWS. Since there didn't seem to be a wealth of information out there about the topic yet, I also wrote a short blog post on the topic here:
http://www.caktusgroup.com/blog/2011/12/19/using-django-and-celery-amazon-sqs/

I'm using Celery 3.0 and was getting deprecation warnings when launching the worker with the BROKER_USER / BROKER_PASSWORD settings.
I took a look at the SQS URL parsing in kombo.utils.url._parse_url and it is calling urllib.unquote on the username and password elements of the URL.
So, to workaround the issue of secret keys with forward slashes, I was able to successfully use the following for the BROKER_URL:
import urllib
BROKER_URL = 'sqs://%s:%s#' % (urllib.quote(AWS_ACCESS_KEY_ID, safe=''),
urllib.quote(AWS_SECRET_ACCESS_KEY, safe=''))
I'm not sure if access keys can ever have forward slashes in them but it doesn't hurt to quote it as well.

For anybody stumbling upon this question, I was able to get Celery working out-of-the-box with SQS (no patching required), but I did need to update to the latest versions of Celery and Kombu for this to work (1.4.5 and 1.5.1 as of now). Use the config lines above and it should work (although you'll probably want to change the default region).
Gotcha: in order to use the URL format above, you need to make sure your AWS secret doesn't contain slashes, as this confuses the URL parser. Just keep generating new secrets until you get one without a slash.

Nobody answered about this. Anyway I tried to configure Celery with Amazon SQS, and it seems I achieved a small success.
Kombu should be patched for this, so I wrote some patches and there is my pull request as well. You can configure Amazon SQS by setting BROKER_URL of sqs:// scheme in Celery on the patched Kombu. For example:
BROKER_URL = 'sqs://AWS_ACCESS:AWS_SECRET#:80//'
BROKER_TRANSPORT_OPTIONS = {
'region': 'ap-northeast-1',
'sdb_persistence': False
}

I regenerated the credentials in the IAM consonle until I got a key without a slash (/). The parsing issues are only with that character, so if your secret doesn't have one you'll be fine.
Not the most terribly elegant solution, but definitely keeps the code clean of hacks.

Update for Python 3, removing backslashes from the AWS KEY.
from urllib.parse import quote_plus
BROKER_URL = 'sqs://{}:{}#'.format(
quote_plus(AWS_ACCESS_KEY_ID),
quote_plus(AWS_SECRET_ACCESS_KEY)
)

I was able to configure SQS on celery 4.3 (python 3.7) by using kombu.
from kombu.utils.url import quote
CELERY_BROKER_URL = 'sqs://{AWS_ACCESS_KEY_ID}:{AWS_SECRET_ACCESS_KEY}#'.format(
AWS_ACCESS_KEY_ID=quote(AWS_ACCESS_KEY_ID, safe=''),
AWS_SECRET_ACCESS_KEY=quote(AWS_SECRET_ACCESS_KEY, safe='')
)