Some of celery tasks in sqs are pending forever, I want to read those messages (tasks) before deleting.
On going to sqs console, I am able to see the encoded message I tried decoding it with
value = base64.b64decode(value.encode('utf-8')).decode('utf-8')
This gives me dict dump with keys
['body', 'headers', 'content-type', 'properties', 'content-encoding']
In this dict body lookes like encoded
I tried to decode it with same
value = base64.b64decode(value.encode('utf-8')).decode('utf-8')
but it gives error saying
UnicodeDecodeError: 'utf8' codec can't decode byte 0x87 in position 1: invalid start byte
Am I missing something?
How to decode this messages? Is there is any way to decode it?
It seems that "Celery" uses "pickle.dump" to turn the payload of the task into bytes, and then encode to base64. Doing the reverse operation we get the payload again.
import base64
import boto3
import pickle
queue_name = 'your-queue-name'
sqsr = boto3.resource('sqs')
queue = sqsr.get_queue_by_name(QueueName=queue_name)
for message in queue.receive_messages(MaxNumberOfMessages=10):
print(f'{message.message_id} >>> {message.receipt_handle}'
f' >>> {message.body} >>> {message.message_attributes}')
body_dict = json.loads(base64.b64decode(message.body))
celery_payload = pickle.loads(base64.b64decode(body_dict.get('body')))
print(celery_payload)
Related
I have a token file that is way too long to retrieve the contents from SecretsManager, so I am compressing the file and storing the bytes in SecretsManager. I then am trying to retrieve the secret value so I can use the token in my application. The secret value is the bytes from the file. I'm running into this error: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
Looking to see how I can fix this error and retrieve the secret value.
#declare tar file name
tar = "token.cache.tar.gz"
file = tarfile.open(tar,"w:gz")
file.add("token.cache")
file.close()
# store cache in aws secrets manager
client = boto3.client('secretsmanager')
with open("token.cache.tar.gz", "rb") as bts:
response = client.create_secret(
Name = 'ms-graph-binary',
SecretBinary = bts.read()
)
response = client.get_secret_value(
SecretId='ac-demo-ms-graph-binary'
) ['SecretBinary']
with tarfile.open(fileobj=response.decode('utf-8'), mode='r') as cf:
cf.extractall("token.cache.decompressed")
Edit: Ended up fixing the code and this is what I used to retrieve the secret.
import tarfile
import io
import boto3
client=boto3.client('secretsmanager')
with tarfile.open('token.cache.tar.gz', "w:gz") as tar:
tar.add('token.cache')
bts = open('token.cache.tar.gz','rb').read()
print("Length before",len(bts))
sec=client.update_secret(SecretId="ac-demo-ms-graph", SecretBinary=bts)
sec=client.get_secret_value(SecretId="ac-demo-ms-graph")['SecretBinary']
print("Length after",len(sec))
with tarfile.open(fileobj=io.BytesIO(sec), mode='r:gz') as t:
d=t.extractfile('token.cache')
#print file content
print("File content",str(d.read()))
I have a working python/boto script which posts a message to my AWS SQS queue. The message body however is hardcoded into the script.
I creates a file called ~/file which contains two values
$ cat ~/file
Username 'encrypted_password_string'
I would like my boto script (see below) to send a message to my AWS SQS queue that contains these two values.
Can anyone please advise how to modify my script below so the message body sent to SQS contains the contents of file ~/file. Please also take note of the special characters that exists within a encrypted password string
Example:
~/file
username d5MopV/EsfSKk8BExCyLHFwNfBrOTzQ1
#!/usr/bin/env python
conf = {
"sqs-access-key": "xxxx",
"sqs-secret-key": "xxxx",
"sqs-queue-name": "UserPassChange",
"sqs-region": "xxxx",
"sqs-path": "sqssend"
}
import boto.sqs
conn = boto.sqs.connect_to_region(
conf.get('sqs-region'),
aws_access_key_id = conf.get('sqs-access-key'),
aws_secret_access_key = conf.get('sqs-secret-key')
)
q = conn.create_queue(conf.get('sqs-queue-name'))
from boto.sqs.message import RawMessage
m = RawMessage()
m.set_body('hardcoded message')
retval = q.write(m)
print 'added message, got retval: %s' % retval
one way to get it working:
in the script I added
import commands
then added,
USERNAME = commands.getoutput("echo $(who am i | awk '{print $1}')")
PASS = commands.getoutput("cat /tmp/.s")
and then added these values to my message body :
MSG = RawMessage()
MSG.set_body(json.dumps({'pass': PASS, 'user': USERNAME}))
The following example shows how to use Boto3 to send a file to a receiver.
test_sqs.py
import boto3
from moto import mock_sqs
#mock_sqs
def test_sqs():
sqs = boto3.resource('sqs', 'us-east-1')
queue = sqs.create_queue(QueueName='votes')
queue.send_message(MessageBody=open('beer.txt').read())
messages = queue.receive_messages()
assert len(messages) == 1
assert messages[0].body == 'tasty\n'
My python code looks like :
import json
import boto.sqs
import boto
from boto.sqs.connection import SQSConnection
from boto.sqs.message import Message
from boto.sqs.message import RawMessage
sqs = boto.connect_sqs(aws_access_key_id='XXXXXXXXXXXXXXX',aws_secret_access_key='XXXXXXXXXXXXXXXXX')
q = sqs.create_queue("Nishantqueue") // Already present
q.set_message_class(RawMessage)
results = q.get_messages()
ret = "Got %s result(s) this time.\n\n" % len(results)
for result in results:
msg = json.loads(result.get_body())
ret += "Message: %s\n" % msg['message']
ret += "\n... done."
print ret
My SQS queue contains atleast 5 to 6 messages... when i execute this ... i get output as this and this is on every run, this code isn't able to pull the mssgs from the queue :
Got 0 result(s) this time.
...done.
I am sure i am missing something in the loop.... couldn't find though
Your code is retrieving messages from an Amazon SQS queue, but it doesn't seem to be deleting them. This means that messages will be invisible for a period of time (specified by the visibility_timeout parameter), after which they will reappear. The expectation is that if a message is not deleted within this time, then it has failed to be processed and should reappear on the queue to try again.
Here's some code that pulls a message from a queue, then deletes it after processing. Note the visibility_timeout specified when a message is retrieved. It is using read() to simply return one message:
#!/usr/bin/python27
import boto, boto.sqs
from boto.sqs.message import Message
# Connect to Queue
q_conn = boto.sqs.connect_to_region("ap-southeast-2")
q = q_conn.get_queue('queue-name')
# Get a message
m = q.read(visibility_timeout=15)
if m == None:
print "No message!"
else:
print m.get_body()
q.delete_message(m)
It's possible that your messages were invisible ("in-flight") when you tried to retrieve them.
I'm trying to send a packet that needs to arrive at its destination as hex in the form:
01 02 a1 a2 b1 b2
In Python2, the UDP method to send that would be:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.send(message, (UDP_IP, UDP_Port))
Message could be passed a hex string like:
\x01\x02\xa1\xa2\xb1\xb2 and that would arrive in the correct format.
In Python3, I cannot pass message the same value without getting the error str does not support the buffer interface. The problem then being is that if I pass message a bytestream b'\x01\x02\xa1\xa2\xb1\xb2' It will not arrive in the correct format.
Has anyone got any suggestions (other than port to 2.7)?
#!/usr/bin/python3
#Import block
import gzip, zlib
import time
import binascii
import struct
import socket
#Start of code
def compress():
"""The function to compress the packet"""
with open('results/file', 'r') as f:
'''Open the file with all the stuff in'''
for lines in f.readlines():
'''Each line represents one packet that will be sent'''
lines = (lines.strip("\n")) #Make sure their are no stray new lines
lines = lines.encode('UTF-8') #Encode for compression
compressed = gzip.compress(lines, 6)
pureHex = (binascii.hexlify(compressed))
return pureHex
def generator(pureHex):
"""The function to add the additional information to the packet"""
packet = [pureHex[i:i+2] for i in range(0, len(pureHex), 2)]
packet = b'\\x' + b'\\x'.join(packet) #Add the \\x that's needed
packet = packet.decode('UTF-8') #Requires decoding to remove the silly second \ that you have to add
return packet
def post(packet):
"""The function to post/send the message via UDP"""
sock.sendto(packet, (UDP_IP, UDP_PORT))
if __name__ == '__main__':
sock = socket.socket(socket.AF_INET,
socket.SOCK_DGRAM)
UDP_IP = "101.101.101.101" #Target IP Address
UDP_PORT = 5010 #Target Port Number
pureHex = compress()
packet = generator(pureHex)
post(packet)
You should send compressed directly and drop all other code that converts bytes to a hex string only to try to convert it back using an pseudo-emulation of Python representation for bytestrings.
Learn how bytes differ from their hex representation in Python source code:
>>> import binascii
>>> b = binascii.unhexlify('0102') # hex string to bytes
>>> b
b'\x01\x02' # representation: repr(b)
>>> len(_)
2
>>> b'\\x01\\x02' #XXX it is not what you want
b'\\x01\\x02'
>>> len(_)
8
>>> repr(b)
"b'\\x01\\x02'" # repr of a repr
>>> repr(repr(b))
'"b\'\\\\x01\\\\x02\'"' # repr of a repr of a repr
You need encode the packet. Change the line:
post(packet)
to:
post(bytes(packet,"utf-8"))
This worked for me.
I am recently finishing a spam classification application as my final project and now I meet a problem.
The problem came from a module to receive emails. I wrote the test code in a single .py file and it worked really well. Here is the code:
#!/usr/bin/env python
# coding=utf-8
import poplib
from email import parser
host = 'pop.qq.com'
username = 'xxxxx#qq.com'
password = 'xxxxxxxxxxxxx'
pop_conn = poplib.POP3_SSL(host)
pop_conn.user(username)
pop_conn.pass_(password)
messages = [pop_conn.retr(i) for i in range(1, len(pop_conn.list()[1]) + 1)]
# Concat message pieces:
messages = ["\n".join(mssg[1]) for mssg in messages]
#print messages
messages = [parser.Parser().parsestr(mssg) for mssg in messages]
i = 0
for message in messages:
i = i + 1
mailName = "mail"+str(i)
f = open(mailName + '.log', 'w');
print >> f, "Date: ", message["Date"]
print >> f, "From: ", message["From"]
print >> f, "To: ", message["To"]
print >> f, "Subject: ", message["Subject"]
print >> f, "Data: "
for part in message.walk():
contentType = part.get_content_type()
if contentType == 'text/plain' :
data = part.get_payload(decode=True)
print >> f, data
f.close()
pop_conn.quit()
But when I tried to transplant exactly the same code to my PyQt4 application, the problem came out in this line:
messages = ["\n".join(mssg[1]) for mssg in messages]
and this is the problem:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 4:ordinal not in range(128)
mssg[1] is a list that contains every line of the mail. I guess this is because the text from the mail was encoded by "utf-8" or "gbk" which can't be decoded by the default "ascii". So I tried to write the code like this:
messages = ["\n".join([m.decode("utf-8") for m in mssg[1]]) for mssg in messages]
The problem became like this:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xcc in position 7
I used Python chardet module to detect the encoding of the text of the email, and it turned out to be "ascii". Now I am really confused. Why the same code can't run on my small application? What is the real problem, and how I can fix it? I will be very appreciated for your help.
I finally solved this problem by receiving the email in a .py file and using my application to import that file. This may not be useful in other situations because I actually didn't solve the character encoding problem. When I was implementing my application, I met lots of encoding problems, and it's quite annoying. For this, I guess it is caused by some irregular text from my mail(maybe some pictures) which is shown in the following picture:
This was shown when I tried to print some of my email data on the screen. However, I still don't know why this cannot run in my application, though it worked well in a simple file. The character encoding problem is very annoying, and maybe I still have a long way to go.:-D