I have a non-profitable website that I need to handle newsletter emails to probably thousand people (lets be realistic and give an upper bound of at most 2000 - 2500 registered users).
I have implemented email this way:
#login_required
def SendEmail(request):
receivers = []
users = Users.objects.all()
receivers.append(user.Email for user in users)
emailTypeSelected = request.POST.get('email_type', -1)
email_factory = EmailFactory()
emailManager = email_factory.create_email(emailTypeSelected)
emailManager.prepare("Some Title")
emailManager.send_email_to(receivers)
return render(request, 'new_user_email.html')
And here is the "abstract" class.
class Email(object):
title = ""
plain_message = ""
html_message = ""
def send_email_to(self, receivers):
send_mail(
self.title,
self.plain_message,
SENDER,
receivers,
html_message=self.html_message
)
I have tested this code and it takes a while to send 1 email to 1 user. My concern is that for thousand emails will put a big overhead to the server.
I was thinking to do the following:
Break the users into group of 100 and send email to those users every 30 minutes.
But I am not sure how this can be implemented. Seems that I will need to implement a sort of threads that will be triggered independently and handle the email for me.
Is there any design pattern that you are aware on how to solve this problem?
Now I know that the best way to do this is to use an external service that handle email newsletter and free up my server from doing this but as a non-profitable website I am trying to minimise the expenses as I already have to pay the server expenses. So at the moment I am trying to find a way to implement that in-house unless big problem arises which will force me to go into third-party services.
Related
I am able to get inbox emails and also able to get emails from specific folder but i am unable to get "starred" emails.
I tried below code. and i am expecting emails with "starred flag" in response.
from imap_tools import MailBox, A
# Create your views here.
def temp(request):
#Get date, subject and body len of all emails from INBOX folder
with MailBox('smtp.gmail.com').login('admin#gmail.com', 'password', 'INBOX') as mailbox:
temp="empty"
for msg in mailbox.fetch():
temp = (msg.date, msg.subject, msg.html)
return HttpResponse(temp)
https://github.com/ikvk/imap_tools/blob/master/examples/search.py
mailbox.fetch(AND(flagged=True))
(A long time ago in a galaxy far far away) star looked like a flag.
We used stripe for the payment system.
Imagine a user is doing different things in our system and for each part, he has to pay. We send these payments to Stripe by calling:
stripe.InvoiceItem.create()
Then we create and finalize the invoice for him by calling:
invoice=stripe.Invoice.create()
stripe.Invoice.finalize_invoice(invoice.id)
So if the user has to pay for 3 items:
item1 = 1
item2 = 2
item3 = 3
The finalize_invoice will have an id, total, ...., and:
total = 6
To test if all items are sending the correct amount to Stripe, I'd like to check the total.
In order to test our payment system, I had to mock Stripe, but the Stripe invoice total would always be zero.
I mocked stripe.InvoiceItem.create and stripe.Invoice.finalize_invoice and stripe.Invoice.create like this:
#patch("app_name.models.stripe.InvoiceItem.create")
#patch("app_name.models.stripe.Invoice.finalize_invoice")
#patch("app_name.models.stripe.Invoice.create")
def test_method(
self,
mock_create,
mock_finalize,
mock_invoice_item,
):
response = MagicMock()
# api_key and stripe_account from this link https://stripe.com/docs/api/connected_accounts
response.api_key = "sk_test_MSc53AbTknQXPy"
response.stripe_account = "acct_1032D82eZvKYlo2C" # Stripe account ID
# last version here https://stripe.com/docs/upgrades
response.stripe_version = "2022-08-01"
mock_invoice_item.return_value = response
response = MagicMock()
response.total = 20
response.invoice_pdf = "https://google.com"
response.id = "sk_test_MSc53AbTknQXPy"
mock_create.return_value = response
mock_finalize.return_value = response.id
Stripe might have a mocking feature.
Stripe-mock was not clear to me how to use it in unit tests.
I really don't know how you are mocking the different Stripe functions in order to pinpoint the issue with the the invoice total cost.
If you're thinking of using stripe-mock, I guess the best way to handle unit testing is to do so in an agnostic way (regardless of the stack), by running the stripe-mock Docker as described in the github Readme and create a proxy that will route any API call to the Docker URL instead of the actual API URL (https://api.stripe.com). This will allow you to do unit testing locally on your machine and even with your preferred CI/CD.
With that being said, please bear in mind that there are some known limitations described in the Readme doc.
REST API service has a limit of requests (say a maximum of 100 requests per minute). In Django, I am trying to allow USERs to access such API and retrieve data in real-time to update SQL tables. Therefore there is a problem that if multiple users are trying to access the API, the limit of requests is likely to be exceeded.
Here is a code snippet as an example of how I currently perform requests - each user will add a list of objects he wants to request and run request_engine().start(object_list) to access the API. I use multithreading to speed up requests. I also allow retrying failed API requests via setting a limit on the number of requests for each request object upper_limit.
As I understand there should be some queue for API requests. I anticipate there must be a more elegant solution for this, however, I could not find any similar examples. How can one implement/rewrite this for multiUSER usage with Django?
import requests
from multiprocessing.dummy import Pool as ThreadPool
N=50 # number of threads
upper_limit=1 # limit on the number of requests for a single object
class request_engine():
def __init__(self):
pass
def start(self,objs):
self.objs={obj:{'status':0,'data':None} for obj in objs}
done=False
while not done:
self.parallel_requests()
done=all(_['status']>upper_limit or _['status']==-1 for obj,_ in self.objs.items())
return dict(self.objs)
def single_request(self,request_obj):
URL = f"https://reqres.in/api/users?page={request_obj}"
r = requests.get(url = URL)
if r.ok:
res = r.json()
self.objs[request_obj]['status']=-1
self.objs[request_obj]['data']=res
else:
self.objs[request_obj]['status']+=1
def parallel_requests(self):
objs=[obj for obj,_ in self.objs.items() if _['status']!=-1 and _['status']<=upper_limit]
pool = ThreadPool(N)
pool.map(self.single_request, objs)
pool.close()
pool.join()
objs=[1,2,3,4,5,6,7,7,8,234,124,24,535,6,234,24,4,1,3,4,5,4,3,5,3,1,5,2,3,5,3]
result=request_engine().start(objs)
print([_['status'] for obj,_ in result.items()])
# status corresponds to the number of unsuccessful requests
# status=-1 implies success of the request
Thanks in advance.
I'm writing Django app and want to send out tokens using Web3 once Coinpayments sends me callback about successfull payment. The problem is that Coinpayments sends multiple callbacks at once and just in one case tokens are sending, other callbacks get replacement transaction underpriced error. I've already tried to use solutions like add +1 to nonce or remove this parameter, but that doesn't help me because transactions are still building with the same nonce. How can that be fixed or what am I doing wrong?
class CoinpaymentsIPNPaymentView(BaseCoinpaymentsIPNView):
def post(self, request, order_id, *args, **kwargs):
status = int(request.POST.get('status'))
order = Order.objects.get(id=order_id)
order.status = request.POST.get("status_text")
if not status >= 100:
order.save()
return JsonResponse({"status": status})
amount = Decimal(request.POST.get('amount1'))
record = Record.objects.create(
user=order.user,
type='i',
amount=amount,
)
order.record = record
order.save()
gold_record = GoldRecord.objects.get(from_record=record)
contract = w3.eth.contract(address=CONTRACT_ADDRESS, abi=ABI_JSON)
transaction = contract.functions.transfer(order.user.wallet.address, int(gold_record.amount * 10 ** 18)).buildTransaction({
'chainId': 1,
'gas': 70000,
'nonce': w3.eth.getTransactionCount(WALLET_ADDRESS) # address where all tokens are stored
})
signed_tx = w3.eth.account.signTransaction(transaction, WALLET_PRIVATE_KEY) # signing with wallet's above private key
tx_hash = w3.eth.sendRawTransaction(signed_tx.rawTransaction)
print(tx_hash.hex())
tx_receipt = w3.eth.waitForTransactionReceipt(tx_hash)
return JsonResponse({"status": status})
P.S. I've already asked it on Ethereum StackExchange, but nobody answered or commented it: https://ethereum.stackexchange.com/questions/80961/sending-tokens-out-on-coinpayments-success-payment-using-web3py
Ok, let the web know answer and solution that I found out by myself
Each transaction should have unique nonce, so I noticed that if I do a loop for sending transactions and set nonce as w3.eth.getTransactionCount(WALLET_ADDRESS) + index then it sends all transactions without any errors. So I removed instant coins sending (even removed waitForTransactionReceipt to speed up it), and made management command where I process all payouts and if it was sent successfully I assign its tx_hash and run it every 10 minutes with Heroku Scheduler
Objective: to provide end-user number of 'notifications' in (almost) real time.
To keep it simple, notifications should come when userX submits form XYZ and all other users should see that the number of notifications incremented by 1 (if userY see the number 50 that means there are 50 NEW XYZ forms).
Question #1: given my django channels web-socket, where should I iterate to get the result? At the moment I placed it under websocket_connect with an endless loop such that:
class EchoDiscussionNotificationConsumer(AsyncConsumer):
async def websocket_connect(self, event):
await self.send({
"type": "websocket.accept",
})
# NOT SURE THIS IS A GOOD DESIGN!
while True:
await asyncio.sleep(2)
rand = random.randint(1,100)
mesg = "#"+str(rand)
await self.send({
'type': 'websocket.send',
'text': mesg,
})
This works great but I don't think this is a good design.
Question #2: I don't want to query the db every 2 seconds what I had in mind is to query only when (1) the user logs in and (2) another user submits form XYZ. So once I have a 'table of notifications' from the db where should I store it (in-memory) to have faster access? (session?)
As you already suggested, you should have a Notification table. Notifications should be created each time a form is submitted then you can use the notification post_save signal to send notifications to the websocket.
In this way, you won't have to long-poll the DB as that defeats the purpose of websockets.
As for where to save the notifications, the DB is quite enough in this case unless you have a very high load.