I met a bug, totally exploded my mind. I don't know how it is possible, I would like to share, and please, if you can offer knowledge and perhaps a solution with me.
here's a basic structure of my code:
user submit a request
when view function is called, create an instance.
when web socket disconnect, delete the instance.
the issue is: when I refresh the page on my location machine, sometimes last web socket is closed after view function is called
I don't know why this happens, let's call the page A and after refresh, new page B, shouldn't it be:
A is called, create instance 1
refresh
A's web socket closed, delete instance
B is called, create instance 2
And sometimes it works as expected, but other times the sequence turned to:
A is called, create instance 1
refresh
B is called, create instance 2
A's web socket closed, delete instance
and it breaks my code due to filter applied in deletion which the instance I needed after refresh is gone!
I don't know if I should do anything, because I reckon there is a big chance that this only happens on local machine.
some extracted code
view
#login_required
def chatFriendsView(request):
# ...
toread = Toread(
sender=sender,
receiver=request.user,
)
toread.save()
return render(request, 'chat/chat_friends.html')
model
class Toread(models.Model):
sender = models.ForeignKey(User, on_delete=models.CASCADE, related_name='sendToread')
receiver = models.ForeignKey(User, on_delete=models.CASCADE, related_name='receiveToread')
# ...
def __str__(self):
return str(self.receiver)
ws
class ChatConsumer(AsyncJsonWebsocketConsumer):
#staticmethod
#database_sync_to_async
def dismissToread(sender_pk, receiver_pk):
for tr in Toread.objects.filter(
sender=User.objects.get(pk=sender_pk),
receiver=User.objects.get(pk=receiver_pk),
):
tr.delete()
async def disconnect(self, close_code):
await ChatConsumer.dismissToread(
sender_pk=int(self.scope['user'].pk),
receiver_pk=int(self.room_name),
)
await self.channel_layer.group_discard(
self.room_group_name,
self.channel_name,
)
# ...
Related
I'm using Channels v2.
I want to integrate long-polling into my project.
The only consumer I see in the documentation for http long polling is the AsyncHttpConsumer.
The code I need to run in my handle function is not asynchronous. It connects to another device on the network using a library that is not asynchronous. From what I understand, this will cause the event loop to block, which is bad.
Can I run my handler synchronously, in a thread somehow? There's a SyncConsumer, but that seems to have something to do with Web Sockets. It doesn't seem applicable to Long Polling.
Using AsyncHttpConsumer as a reference, I was able to write an almost exact duplicate of the class, but subclassing SyncConsumer instead of AsyncConsumer as AsyncHttpConsumer does.
After a bit of testing, I soon realized that since my code was all running in a single thread, until the handle() method finished running, which presumably runs until done, the disconnect() method wouldn't be triggered, so there was no way to interrupt a long running handle() method, even if the client disconnects.
The following new version runs handle() in a thread, and gives the user 2 ways to check if the client disconnected:
from channels.consumer import SyncConsumer
from channels.exceptions import StopConsumer
from threading import Thread, Event
# We can't pass self.client_disconnected to handle() as a reference if it's
# a regular bool. That means if we use a regular bool, and the variable
# changes in this thread, it won't change in the handle() method. Using a
# class fixes this.
# Technically, we could just pass the Event() object
# (self.client_disconnected) to the handle() method, but then the client
# needs to know to use .is_set() instead of just checking if it's True or
# False. This is easier for the client.
class RefBool:
def __init__(self):
self.val = Event()
def set(self):
self.val.set()
def __bool__(self):
return self.val.is_set()
def __repr__(self):
current_value = bool(self)
return f"RefBool({current_value})"
class SyncHttpConsumer(SyncConsumer):
"""
Sync HTTP consumer. Provides basic primitives for building synchronous
HTTP endpoints.
"""
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.handle_thread = None
self.client_disconnected = RefBool()
self.body = []
def send_headers(self, *, status=200, headers=None):
"""
Sets the HTTP response status and headers. Headers may be provided as
a list of tuples or as a dictionary.
Note that the ASGI spec requires that the protocol server only starts
sending the response to the client after ``self.send_body`` has been
called the first time.
"""
if headers is None:
headers = []
elif isinstance(headers, dict):
headers = list(headers.items())
self.send(
{"type": "http.response.start", "status": status, "headers": headers}
)
def send_body(self, body, *, more_body=False):
"""
Sends a response body to the client. The method expects a bytestring.
Set ``more_body=True`` if you want to send more body content later.
The default behavior closes the response, and further messages on
the channel will be ignored.
"""
assert isinstance(body, bytes), "Body is not bytes"
self.send(
{"type": "http.response.body", "body": body, "more_body": more_body}
)
def send_response(self, status, body, **kwargs):
"""
Sends a response to the client. This is a thin wrapper over
``self.send_headers`` and ``self.send_body``, and everything said
above applies here as well. This method may only be called once.
"""
self.send_headers(status=status, **kwargs)
self.send_body(body)
def handle(self, body):
"""
Receives the request body as a bytestring. Response may be composed
using the ``self.send*`` methods; the return value of this method is
thrown away.
"""
raise NotImplementedError(
"Subclasses of SyncHttpConsumer must provide a handle() method."
)
def disconnect(self):
"""
Overrideable place to run disconnect handling. Do not send anything
from here.
"""
pass
def http_request(self, message):
"""
Sync entrypoint - concatenates body fragments and hands off control
to ``self.handle`` when the body has been completely received.
"""
if "body" in message:
self.body.append(message["body"])
if not message.get("more_body"):
full_body = b"".join(self.body)
self.handle_thread = Thread(target=self.handle, args=(full_body, self.client_disconnected), daemon=True)
self.handle_thread.start()
def http_disconnect(self, message):
"""
Let the user do their cleanup and close the consumer.
"""
self.client_disconnected.set()
self.disconnect()
self.handle_thread.join()
raise StopConsumer()
The SyncHttpConsumer class is used very similarly to how you would use the AsyncHttpConsumer class - you subclass it, and define a handle() method. The only difference is that the handle() method takes an extra arg:
class MyClass(SyncHttpConsumer):
def handle(self, body, client_disconnected):
while not client_disconnected:
...
Or you could, just like with the AsyncHttpConsumer class, override the disconnect() method instead if you prefer.
I'm still not sure if this is the best way to do this, or why Django Channels doesn't include something like this in addition to AsyncHttpConsumer. If anyone knows, please let us know.
I'm considering to use django-notifications and Web Sockets to send real-time notifications to iOS/Android and Web apps. So I'll probably use Django Channels.
Can I use Django Channels to track online status of an user real-time? If yes then how I can achieve this without polling constantly the server?
I'm looking for a best practice since I wasn't able to find any proper solution.
UPDATE:
What I have tried so far is the following approach:
Using Django Channels, I implemented a WebSocket consumer that on connect will set the user status to 'online', while when the socket get disconnected the user status will be set to 'offline'.
Originally I wanted to included the 'away' status, but my approach cannot provide that kind of information.
Also, my implementation won't work properly when the user uses the application from multiple device, because a connection can be closed on a device, but still open on another one; the status would be set to 'offline' even if the user has another open connection.
class MyConsumer(AsyncConsumer):
async def websocket_connect(self, event):
# Called when a new websocket connection is established
print("connected", event)
user = self.scope['user']
self.update_user_status(user, 'online')
async def websocket_receive(self, event):
# Called when a message is received from the websocket
# Method NOT used
print("received", event)
async def websocket_disconnect(self, event):
# Called when a websocket is disconnected
print("disconnected", event)
user = self.scope['user']
self.update_user_status(user, 'offline')
#database_sync_to_async
def update_user_status(self, user, status):
"""
Updates the user `status.
`status` can be one of the following status: 'online', 'offline' or 'away'
"""
return UserProfile.objects.filter(pk=user.pk).update(status=status)
NOTE:
My current working solution is using the Django REST Framework with an API endpoint to let client apps send HTTP POST request with current status.
For example, the web app tracks mouse events and constantly POST the online status every X seconds, when there are no more mouse events POST the away status, when the tab/window is about to be closed, the app sends a POST request with status offline.
THIS IS a working solution, depending on the browser I have issues when sending the offline status, but it works.
What I'm looking for is a better solution that doesn't need to constantly polling the server.
Using WebSockets is definitely the better approach.
Instead of having a binary "online"/"offline" status, you could count connections: When a new WebSocket connects, increase the "online" counter by one, when a WebSocket disconnects, decrease it. So that, when it is 0, then the user is offline on all devices.
Something like this
#database_sync_to_async
def update_user_incr(self, user):
UserProfile.objects.filter(pk=user.pk).update(online=F('online') + 1)
#database_sync_to_async
def update_user_decr(self, user):
UserProfile.objects.filter(pk=user.pk).update(online=F('online') - 1)
The best approach is using Websockets.
But I think you should store not just the status, but also a session key or a device identification. If you use just a counter, you are losing valuable information, for example, from what device is the user connected at a specific moment. That is key in some projects. Besides, if something wrong happens (disconnection, server crashes, etc), you are not going to be able to track what counter is related with each device and probably you'll need to reset the counter at the end.
I recommend you to store this information in another related table:
from django.db import models
from django.conf import settings
class ConnectionHistory(models.Model):
ONLINE = 'online'
OFFLINE = 'offline'
STATUS = (
(ONLINE, 'On-line'),
(OFFLINE, 'Off-line'),
)
user = models.ForeignKey(
settings.AUTH_USER_MODEL,
on_delete=models.CASCADE
)
device_id = models.CharField(max_lenght=100)
status = models.CharField(
max_lenght=10, choices=STATUS,
default=ONLINE
)
first_login = models.DatetimeField(auto_now_add=True)
last_echo = models.DatetimeField(auto_now=True)
class Meta:
unique_together = (("user", "device_id"),)
This way you have a record per device to track their status and maybe some other information like ip address, geoposition, etc. Then you can do something like (based on your code):
#database_sync_to_async
def update_user_status(self, user, device_id, status):
return ConnectionHistory.objects.get_or_create(
user=user, device_id=device_id,
).update(status=status)
How to get a device identification
There are plenty of libraries do it like https://www.npmjs.com/package/device-uuid. They simply use a bundle of browser parameters to generate a hash key. It is better than use session id alone, because it changes less frencuently.
Tracking away status
After each action, you can simply update last_echo. This way you can figured out who is connected or away and from what device.
Advantage: In case of crash, restart, etc, the status of the tracking could be re-establish at any time.
My answer is based on the answer of C14L. The idea of counting connections is very clever. I just make some improvement, at least in my case. It's quite messy and complicated, but I think it's necessary
Sometimes, WebSocket connects more than it disconnects, for example, when it has errors. That makes the connection keep increasing. My approach is instead of increasing the connection when WebSocket opens, I increase it before the user accesses the page. When the WebSocket disconnects, I decrease the connection
in views.py
def homePageView(request):
updateOnlineStatusi_goIn(request)
# continue normal code
...
def updateOnlineStatusi_goIn(request):
useri = request.user
if OnlineStatus.objects.filter(user=useri).exists() == False:
dct = {
'online': False,
'connections': 0,
'user': useri
}
onlineStatusi = OnlineStatus.objects.create(**dct)
else:
onlineStatusi = OnlineStatus.objects.get(user=useri)
onlineStatusi.connections += 1
onlineStatusi.online = True
onlineStatusi.save()
dct = {
'action': 'updateOnlineStatus',
'online': onlineStatusi.online,
'userId': useri.id,
}
async_to_sync(get_channel_layer().group_send)(
'commonRoom', {'type': 'sendd', 'dct': dct})
In models.py
class OnlineStatus(models.Model):
online = models.BooleanField(null=True, blank=True)
connections = models.BigIntegerField(null=True, blank=True)
user = models.OneToOneField(User, on_delete=models.CASCADE, null=True, blank=True)
in consummers.py
class Consumer (AsyncWebsocketConsumer):
async def sendd(self, e): await self.send(json.dumps(e["dct"]))
async def connect(self):
await self.accept()
await self.channel_layer.group_add('commonRoom', self.channel_name)
async def disconnect(self, _):
await self.channel_layer.group_discard('commonRoom', self.channel_name)
dct = await updateOnlineStatusi_goOut(self)
await self.channel_layer.group_send(channelRoom, {"type": "sendd", "dct": dct})
#database_sync_to_async
def updateOnlineStatusi_goOut(self):
useri = self.scope["user"]
onlineStatusi = OnlineStatus.objects.get(user=useri)
onlineStatusi.connections -= 1
if onlineStatusi.connections <= 0:
onlineStatusi.connections = 0
onlineStatusi.online = False
else:
onlineStatusi.online = True
onlineStatusi.save()
dct = {
'action': 'updateOnlineStatus',
'online': onlineStatusi.online,
'userId': useri.id,
}
return dct
I am currently working on developing a Twitch.tv chat and moderation bot(full code can be found on github here: https://github.com/DarkElement75/tecsbot; might not be fully updated to match the problem I describe), and in doing this I need to have many different Twisted TCP connections to various channels. I have (because of the way Twitch's Whisper system works) one connection for sending / receiving whispers, and need the ability for any connection to any channel can reference this whisper connection and send data on this connection, via the TwitchWhisperBot's write() function. However, I have yet to find a method that allows my current global function to reference this write() function. Here is what I have right now:
#global functions
def send_whisper(self, user, msg):
whisper_str = "/w %s %s" % (user, msg)
print dir(whisper_bot)
print dir(whisper_bot.transport)
whisper_bot.write("asdf")
def whisper(self, user, msg):
'''global whisper_user, whisper_msg
if "/mods" in msg:
thread.start_new_thread(get_whisper_mods_msg, (self, user, msg))
else:
whisper_user = user
whisper_msg = msg'''
if "/mods" in msg:
thread.start_new_thread(get_whisper_mods_msg, (self, user, msg))
else:
send_whisper(self, user, msg)
#Example usage of these (inside a channel connection):
send_str = "Usage: !permit add <user> message/time/<time> <message count/time duration/time unit>/permanent"
whisper(self, user, send_str)
#Whisper classes
class TwitchWhisperBot(irc.IRCClient, object):
def write(self, msg):
self.msg(self.channel, msg.encode("utf-8"))
logging.info("{}: {}".format(self.nickname, msg))
class WhisperBotFactory(protocol.ClientFactory, object):
wait_time = 1
def __init__(self, channel):
global whisper_bot
self.channel = channel
whisper_bot = TwitchWhisperBot(self.channel)
def buildProtocol(self, addr):
return TwitchWhisperBot(self.channel)
def clientConnectionLost(self, connector, reason):
# Reconnect when disconnected
logging.error("Lost connection, reconnecting")
self.protocol = TwitchWhisperBot
connector.connect()
def clientConnectionFailed(self, connector, reason):
# Keep retrying when connection fails
msg = "Could not connect, retrying in {}s"
logging.warning(msg.format(self.wait_time))
time.sleep(self.wait_time)
self.wait_time = min(512, self.wait_time * 2)
connector.connect()
#Execution begins here:
#main whisper bot where other threads with processes will be started
#sets variables for connection to twitch chat
whisper_channel = '#_tecsbot_1444071429976'
whisper_channel_parsed = whisper_channel.replace("#", "")
server_json = get_json_servers()
server_arr = (server_json["servers"][0]).split(":")
server = server_arr[0]
port = int(server_arr[1])
#try:
# we are using this to make more connections, better than threading
# Make logging format prettier
logging.basicConfig(format="[%(asctime)s] %(message)s",
datefmt="%H:%M:%S",
level=logging.INFO)
# Connect to Twitch IRC server, make more instances for more connections
#Whisper connection
whisper_bot = ''
reactor.connectTCP(server, port, WhisperBotFactory(whisper_channel))
#Channel connections
reactor.connectTCP('irc.twitch.tv', 6667, BotFactory("#darkelement75"))
The simplest solution here would be using the Singleton pattern, since you're guaranteed to only have a single connection of each type at any given time. Personally, with Twisted, I think the simplest solution is to use the reactor to store your instance (since reactor itself is a singleton).
So what you want to do is, inside TwitchWhisperBot, at sign in:
def signedOn(self):
reactor.whisper_instance = self
And then, anywhere else in the code, you can access that instance:
reactor.whisper_instance = self
Just for sanity's sake, you should also check if it's been set or not:
if getattr(reactor, 'whisper_instance'):
reactor.whisper_instance.write("test_user", "message")
else:
logging.warning("whisper instance not set")
I have a small web app with AngularJS front-end and Django ReST in the back. There's a strange hitch going on when I make POST request to the web service: the browser console clearly shows 3 parameters being sent, but the backend logging reports only 2 params received. The result is that the server throws a code 500 error due to a bad database lookup.
Here's the code:
Client
var b = newQuesForm.username.value;
$http.post('/myapp/questions/new', {username:b,title:q.title,description:q.description}).
success(function(data, status, headers, config) {
$http.get('/myapp/questions').success(function(data){
$scope.questions = data;
q = null;
$scope.newQuesForm.$setPristine();
}).error(function(data, status, headers, config) {
console.log(headers+data);
});
}).
error(function(data, status, headers, config) {
console.log(headers+data);
});
Both my manual logging and the dev console show a string like:
{"username":"admin","description":"What's your name?","title":"question 1"}
Server
class CreateQuestionSerializer(serializers.Serializer):
author = UserSerializer(required=False)
title = serializers.CharField(max_length=150)
description = serializers.CharField(max_length=350)
def create(self, data):
q= Question()
d = data
q.title = d.get('title')
q.description = d.get("description")
q.author = User.objects.get(username=d.get('username'))
q.save()
return q
Server-side logging shows the username parameter never succeeds in making the trip, and thus I end up with code 500 and error message:
User matching query does not exist. (No user with id=none)
What's causing some of the data to get lost?
So it turns out the problem was really with the serialization of fields, as #nikhiln began to point out. I followed his lead to refactor the code, moving the create() method to api.py, rather than serializers.py, and stopped relying altogether on the client-side data for the user's identity, something that was a bit silly in the first place (passing User to a hidden input in the view, and then harvesting the username from there and passing it back to the server in the AJAX params). Here's the new code, that works perfectly:
class QuestionCreate(generics.CreateAPIView):
model = Question
serializer_class = CreateQuestionSerializer
def create(self, request,*args,**kwargs):
q= Question()
d = request.data
q.title = d.get('title')
q.description = d.get("description")
q.author = request.user
q.save()
if q.pk:
return Response({'id':q.pk,'author':q.author.username}, status=status.HTTP_201_CREATED)
return Response({'error':'record not created'}, status=status.HTTP_400_BAD_REQUEST)
So here, I do it the right way: pull the User from the request param directly in the backend.
Scrapy application, but the question is really about the Python language - experts can probably answer this immediately without knowing the framework at all.
I've got a class called CrawlWorker that knows how to talk to so-called "spiders" - schedule their crawls, and manage their lifecycle.
There's a TwistedRabbitClient that has-one CrawlWorker. The client only knows how to talk to the queue and hand off messages to the worker - it gets completed work back from the worker asynchronously by using the worker method connect_to_scrape below to connect to a signal emitted by a running spider:
def connect_to_scrape(self, callback):
self._connect_to_signal(callback, signals.item_scraped)
def _connect_to_signal(self, callback, signal):
if signal is signals.item_scraped:
def _callback(item, response, sender, signal, spider):
scrape_config = response.meta['scrape_config']
delivery_tag = scrape_config.delivery_tag
callback(item.to_dict(), delivery_tag)
else:
_callback = callback
dispatcher.connect(_callback, signal=signal)
So the worker provides a layer of "work deserialization" for the Rabbit client, who doesn't know about spiders, responses, senders, signals, items (anything about the nature of the work itself) - only dicts that'll be published as JSON with their delivery tags.
So the callback below isn't registering properly (no errors either):
def publish(self, item, delivery_tag):
self.log('item_scraped={0} {1}'.format(item, delivery_tag))
publish_message = json.dumps(item)
self._channel.basic_publish(exchange=self.publish_exchange,
routing_key=self.publish_key,
body=publish_message)
self._channel.basic_ack(delivery_tag=delivery_tag)
But if I remove the if branch in _connect_to_signal and connect the callback directly (and modify publish to soak up all the unnecessary arguments), it works.
Anyone have any ideas why?
So, I figured out why this wasn't working, by re-stating it in a more general context:
import functools
from scrapy.signalmanager import SignalManager
SIGNAL = object()
class Sender(object):
def __init__(self):
self.signals = SignalManager(self)
def wrap_receive(self, receive):
#functools.wraps(receive)
def wrapped_receive(message, data):
message = message.replace('World', 'Victor')
value = data['key']
receive(message, value)
return wrapped_receive
def bind(self, receive):
_receive = self.wrap_receive(receive)
self.signals.connect(_receive, signal=SIGNAL,
sender=self, weak=False)
def send(self):
message = 'Hello, World!'
data = {'key': 'value'}
self.signals.send_catch_log(SIGNAL, message=message, data=data)
class Receiver(object):
def __init__(self, sender):
self.sender = sender
self.sender.bind(self.receive)
def receive(self, message, value):
"""Receive data from a Sender."""
print 'Receiver received: {0} {1}.'.format(message, value)
if __name__ == '__main__':
sender = Sender()
receiver = Receiver(sender)
sender.send()
This works if and only if weak=False.
The basic problem is that when connecting to the signal, weak=False needs to be specified. Hopefully someone smarter than me can expound on why that's needed.