asyncio task hangs midway - django

Question

So i am building a scraper which takes a bunch of urls, a success function that will run via celery if that url was fetched successfully and if any error occurs just return and collect the bunch of urls that were not successfull and send them to be scheduled again to a celery function.
Below is the code.
class AsyncRequest:
def __init__(self, urls_batch, callback, task_name, method, finish_callback=None, *args, **kwargs):
"""
:param urls_batch: List of urls to fetch in asycn
:param callback: Callback that process a successfull response
"""
self.tasks = []
self.headers = {
"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"
}
self.urls_batch = urls_batch
self.task_name = task_name
self.callback = callback
self.finish_callback = finish_callback
self.args = args
self.kwargs = kwargs
self.proxy = kwargs["proxy"] if "proxy" in kwargs.keys() else None
self.finish_callback = finish_callback
self.successfull_urls = []
self.verify_ssl = kwargs["verify_ssl"] if "verify_ssl" in kwargs.keys() else True
async def fetch(self, session, url, time_out=15, retry_limit=3, *args, **kwargs):
try:
for i in range(retry_limit):
try:
async with session.request(self.method, url, headers=self.headers,
timeout=ClientTimeout(total=None, sock_connect=time_out,
sock_read=time_out),
verify_ssl=self.verify_ssl, proxy=self.proxy) as response:
if response.status in [200, 203]:
result = await response.text()
self.successfull_urls.append(url)
# I dont think its a celery issue because even if i comment out the below line it still gets stuck
#self.callback.delay(result, url=url, *self.args, **self.kwargs)
return
else:
logger.error(
"{} ERROR===============================================>".format(self.task_name))
logger.error("status: {}".format(response.status))
except ClientHttpProxyError as e:
logger.error("{} ---> {} pkm: {}, timeout: {}".format(self.task_name, type(e), pkm,
proxy = UpdateProxy()
except Exception as e:
logger.error(
"{} ---> {} url: {}, timeout: {}!!!! returning".format(self.task_name, type(e), pkm, time_out))
logger.error("pkm: {} errored".format(self.kwargs["search_url_pkm_mapping"][url]))
async def main(self):
async with aiohttp.ClientSession(timeout=100) as session:
results = await asyncio.gather(*(self.fetch(session, url) for url in self.urls_batch),
return_exceptions=True)
logger.info("Gather operation done ----------> Results: {}".format(results))
logger.info("{} Successful Urls".format(len(self.successfull_urls)))
errored_urls = [url for url in self.urls_batch if url not in self.successfull_urls]
logger.error("{} urls errored".format(len(errored_urls)))
# Below code to send errored urls to a celery task that scheduled them again
# if self.finish_callback and len(errored_urls) > 0:
# self.finish_callback.delay(errored_urls, self.task_name, *self.args, **self.kwargs)
So what happens is if i send a batch of 50urls, almost 40 to 45 of them work perfectly and the remaining just hangs. Nothing happens, i expect that at least the tasks will throw sum error due to network issue or server or anything. They just should be finished and code after gather which shows the number of successfull urls and errored urls is executed. And this does not happen, the code just hangs. The log lines after gather are not executed and i dont know where the error is.
Any help will be appreciated highly.
EDIT: I have removed the celery code and just fetching and keeping track of successfull urls and errored one. It still gets stuck. If its necessary to note, i am sending the request to google(but even if google is blocking my request some error must be thrown right.)
EDIT2: One more thing i would like to list is if i just keep the url's batch size small such as 15 to 20. Then there is no hang and everything works as expected. But the moment i increase the urls batch to say 50 it stucks, on 3 to 5 urls.