I have a Django application with a health check endpoint that's using django-health-check.
In the url_patterns I have added the following line:
url(r'^ht/', include('health_check.urls')),
The issue is that the health check is filling all the Sentry transaction limits.
How can I exclude the health check endpoint in Sentry?
The way to handle this kind of cases is to use a sampling function to control the sample rate according to the URL or other parameters.
def traces_sampler(ctx):
if 'wsgi_environ' in ctx:
url = ctx['wsgi_environ'].get('PATH_INFO', '')
if url.startswith('/ht/'):
return 0 # Don't trace any
return 1 # Trace all
sentry_sdk.init(
# ...
traces_sampler=traces_sampler,
)
Here is a more complete example.
def traces_sampler(ctx):
if ctx['parent_sampled'] is not None:
# If this transaction has a parent, we usually want to sample it
# if and only if its parent was sampled.
return ctx['parent_sampled']
op = ctx['transaction_context']['op']
if 'wsgi_environ' in ctx:
# Get the URL for WSGI requests
url = ctx['wsgi_environ'].get('PATH_INFO', '')
elif 'asgi_scope' in ctx:
# Get the URL for ASGI requests
url = ctx['asgi_scope'].get('path', '')
else:
# Other kinds of transactions don't have a URL
url = ''
if op == 'http.server':
# Conditions only relevant to operation "http.server"
if url.startswith('/ht/'):
return 0 # Don't trace any of these transactions
return 0.1 # Trace 10% of other transactions
sentry_sdk.init(
# ...
traces_sampler=traces_sampler,
)
You can easily filter the events sent to the sentry by using the before_breadcrumb function in sentry. This function allows you to apply any filter you want on the sentry logs before sending them to the sentry. if you return None in the before_breadcrumb function, it will not send anything to the sentry.
Your code would be like this:
import sentry_sdk
def before_breadcrumb(crumb, hint):
if 'httplib_request' in hint:
# Check the request URL and return None if it is health check
return crumb
sentry_sdk.init(
# ...
before_breadcrumb=filter_events,
)
Related
I need to run some asynchronous tasks in a Django app, and I started to look into Google Cloud Tasks. I think I have followed all the instructions - and every possible variation I could think of, without success so far.
The problem is that all created tasks go to the queue, but fail to execute. The console and the logs report only a http code 301 (permanent redirection). For the sake of simplicity, I deployed the same code to two services of an App Engine (standard), and routed the tasks request to only one of them.
It looks like the code itself is working fine. When I go to "https://[proj].appspot.com/api/v1/tasks", the routine executes nicely and there's no redirection according to DevTools/Network. When Cloud Tasks try to call "/api/v1/tasks", it fails every time.
If anyone could take a look at the code below and point out what may be causing this failure, I'd appreciate very much.
Thank you.
#--------------------------------
# [proj]/.../urls.py
#--------------------------------
from [proj].api import tasks
urlpatterns += [
# tasks api
path('api/v1/tasks', tasks, name='tasks'),
]
#--------------------------------
# [proj]/api.py:
#--------------------------------
from django.views.decorators.csrf import csrf_exempt
#csrf_exempt
def tasks(request):
print('Start api')
payload = request.body.decode("utf-8")
print (payload)
print('End api')
return HttpResponse('OK')
#--------------------------------
# [proj]/views/manut.py
#--------------------------------
from django.views.generic import View
from django.shortcuts import redirect
from [proj].tasks import TasksCreate
class ManutView(View):
template_name = '[proj]/manut.html'
def post(self, request, *args, **kwargs):
relative_url = '/api/v1/tasks'
testa_task = TasksCreate()
resp = testa_task.send_task(
url=relative_url,
schedule_time=5,
payload={'task_type': 1, 'id': 21}
)
print(resp)
return redirect(request.META['HTTP_REFERER'])
#--------------------------------
# [proj]/tasks/tasks.py:
#--------------------------------
from django.conf import settings
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
from typing import Dict, Optional, Union
import json
import time
class TasksCreate:
def send_task(self,
url: str,
payload: Optional[Union[str, Dict]] = None,
schedule_time: Optional[int] = None, # in seconds
name: Optional[str] = None,
) -> None:
client = tasks_v2.CloudTasksClient()
parent = client.queue_path(
settings.GCP_PROJECT,
settings.GCP_LOCATION,
settings.GCP_QUEUE,
)
# App Engine task:
task = {
'app_engine_http_request': { # Specify the type of request.
'http_method': 'POST',
'relative_uri': url,
'app_engine_routing': {'service': 'tasks'}
}
}
if name:
task['name'] = name
if isinstance(payload, dict):
payload = json.dumps(payload)
if payload is not None:
converted_payload = payload.encode()
# task['http_request']['body'] = converted_payload
task['app_engine_http_request']['body'] = converted_payload
if schedule_time is not None:
now = time.time() + schedule_time
seconds = int(now)
nanos = int((now - seconds) * 10 ** 9)
# Create Timestamp protobuf.
timestamp = timestamp_pb2.Timestamp(seconds=seconds, nanos=nanos)
# Add the timestamp to the tasks.
task['schedule_time'] = timestamp
resp = client.create_task(parent, task)
return resp
# --------------------------------
# [proj]/dispatch.yaml:
# --------------------------------
dispatch:
- url: "*/api/v1/tasks"
service: tasks
- url: "*/api/v1/tasks/"
service: tasks
- url: "*appspot.com/*"
service: default
#--------------------------------
# [proj]/app.yaml & tasks.yaml:
#--------------------------------
runtime: python37
instance_class: F1
automatic_scaling:
max_instances: 2
service: default
#handlers:
#- url: .*
# secure: always
# redirect_http_response_code: 301
# script: auto
entrypoint: gunicorn -b :$PORT --chdir src server.wsgi
env_variables:
...
UPDATE:
Here are the logs for an execution:
{
insertId: "1lfs38fa9"
jsonPayload: {
#type: "type.googleapis.com/google.cloud.tasks.logging.v1.TaskActivityLog"
attemptResponseLog: {
attemptDuration: "0.008005s"
dispatchCount: "5"
maxAttempts: 0
responseCount: "5"
retryTime: "2020-03-09T21:50:33.557783Z"
scheduleTime: "2020-03-09T21:50:23.548409Z"
status: "UNAVAILABLE"
targetAddress: "POST /api/v1/tasks"
targetType: "APP_ENGINE_HTTP"
}
task: "projects/[proj]/locations/us-central1/queues/tectaq/tasks/09687434589619534431"
}
logName: "projects/[proj]/logs/cloudtasks.googleapis.com%2Ftask_operations_log"
receiveTimestamp: "2020-03-09T21:50:24.375681687Z"
resource: {
labels: {
project_id: "[proj]"
queue_id: "tectaq"
target_type: "APP_ENGINE_HTTP"
}
type: "cloud_tasks_queue"
}
severity: "ERROR"
timestamp: "2020-03-09T21:50:23.557842532Z"
}
At last I could make Cloud Tasks work, but only using http_request type (with absolute url). There was no way I could make the tasks run when they were defined as app_engine_http_request (relative url).
I had already tried the http_request type with POST, but that was before I exempted the api function from have the csrf token previously checked, and that was causing an error Forbidden (Referer checking failed - no Referer.): /api/v1/tasks, which I failed to connect to the csrf omission.
If someone stumble across this issue in the future, and find out a way to make app_engine_http_request work on Cloud Tasks with Django, I'd still like very much to know the solution.
The problem is that App Engine task handlers do not follow redirects, so you have to find out why the request is being redirected and make an exception for App Engine requests. In my case I was redirecting http to https and had to make an exception like so: (Node Express)
app.use((req, res, next) => {
const protocol = req.headers['x-forwarded-proto']
const userAgent = req.headers['user-agent']
if (userAgent && userAgent.includes('AppEngine-Google')) {
console.log('USER AGENT IS GAE, SKIPPING REDIRECT TO HTTPS.')
return next()
} else if (protocol === 'http') {
res.redirect(301, `https://${req.headers.host}${req.url}`)
} else {
next()
}
})
The problem is that all created tasks go to the queue, but fail to execute. The console and the logs report only a http code 301 (permanent redirection).
Maybe the request handler for your task endpoint wants a trailing slash.
Try changing this:
class ManutView(View):
template_name = '[proj]/manut.html'
def post(self, request, *args, **kwargs):
relative_url = '/api/v1/tasks'
...
to this:
class ManutView(View):
template_name = '[proj]/manut.html'
def post(self, request, *args, **kwargs):
relative_url = '/api/v1/tasks/'
...
Also just try hitting the task url yourself and see if you can get a task to run from curl
If someone stumble across this issue in the future, and find out a way
to make app_engine_http_request work on Cloud Tasks with Django, I'd
still like very much to know the solution.
#JCampos I manage to make it work on my Django app (I use in addition DRF but I do no think it causes a big difference).
from google.cloud import tasks_v2
from google.protobuf import timestamp_pb2
import datetime
class CloudTasksMixin:
#property
def _cloud_task_client(self):
return tasks_v2.CloudTasksClient()
def send_to_cloud_tasks(self, url, http_method='POST', payload=None,in_seconds=None, name=None):
""" Send task to be executed """
parent = self._cloud_task_client.queue_path(settings.TASKS['PROJECT_NAME'], settings.TASKS['QUEUE_REGION'], queue=settings.TASKS['QUEUE_NAME'])
task = {
'app_engine_http_request': {
'http_method': http_method,
'relative_uri': url
}
}
...
And then I use a view like this one:
class CloudTaskView(views.APIView):
authentication_classes = []
def post(self, request, *args, **kwargs):
# Do your stuff
return Response()
Finally I implement this url in the urls.py (from DRF) with csrf_exempt(CloudTaskView.as_view())
At first I had 403 error, but thanks to you and your comment with csrf_exempt, it is now working.
It seems that Cloud Tasks calls App Engine using a HTTP url (that's ok because probably they are in the same network), but if you are using HTTPs, Django should be redirecting (http -> https) any request that's being received, including your handler endpoint.
To solve this, you should tell Django to not redirect your handler.
You can use settings.SECURE_REDIRECT_EXEMPT for it.
For instance:
SECURE_REDIRECT_EXEMPT = [r"^api/v1/tasks/$"]
I am using Stripe in my Django application. I have the following test case: incorrect_cvc leads to an card_error. Now when correcting the CVC and using 4242 4242 4242 4242 what I except is a successful charge. However what I get pack is the following error message:
Request req_auTSTGSGoUVNUa: Keys for idempotent requests can only be
used with the same parameters they were first used with. Try using a
key other than 'k1qjchgqjw' if you meant to execute a different
request.
I am not aware of which parameters I changed. But I think it's not the idea that the checkout process basically doesn't work anymore after an card_error. Does anyone understand which parameters I "changed" that leads to this error message?
def checkout_page(request):
"""
* Check if session and ReservedItem exist.
* Generate order_item dict for every ReservedItem entry, that belongs
to order_reference.
If request.method is 'POST':
* Check if ticket reservation is still valid.
* Create entries in models OrderItem, Order & ReservedItem.
"""
session_order_reference = request.session.get('order_reference')
if request.session.get('order_reference'):
reserved_items = ReservedItem.objects.filter(
order_reference=session_order_reference
)
if not reserved_items:
return redirect('website:index')
else:
return redirect('website:index')
taxes_dict = {}
total_gross = total_tax_amount = 0
order_items_list = []
for item in reserved_items:
event = item.ticket.event
timestamp_of_reservation = item.created
total_gross += item.subtotal
order_item = {
'ticket': item.ticket,
'ticket_name': item.ticket.name,
'quantity': item.quantity,
'subtotal': item.subtotal,
'type': OrderType.ORDER,
}
total_tax_amount += add_tax(
item=item,
taxes_dict=taxes_dict,
order_item=order_item,
)
order_items_list.append(dict(order_item))
total_net = total_gross - total_tax_amount # TODO Marc: Calculate in add_vat func?
if request.method == 'POST':
# TODO Marc: Should live in forms.py or just models?
reservation_expired_redirect = check_if_reservation_expired(
request=request,
timestamp_of_reservation=timestamp_of_reservation,
organizer=event.organizer.slug,
event=event.slug,
)
if reservation_expired_redirect:
return reservation_expired_redirect
# TODO Marc: Should live in forms.py or just models?
ticket_is_on_sale = check_if_ticket_is_on_sale(
order_items_list=order_items_list,
request=request,
organizer=event.organizer.slug,
event=event.slug,
)
if ticket_is_on_sale:
return ticket_is_on_sale
billing = BillingForm(request.POST, prefix='billing')
order = OrderForm(request.POST, prefix='order')
if order.is_valid() and billing.is_valid():
# Charge via Stripe
stripe.api_key = "ABC" # TODO Marc: Change to env
token = request.POST.get('stripeToken')
# https://stripe.com/docs/api#error_handling
paid = False
try:
# Compare with transactions > models copy.py > class ChargeManager(models.Manager):
# Use Stripe's library to make requests...
total_gross_amount_in_smallest_unit = smallest_currency_unit(total_gross, 'eur') #TODO Marc: Replace eur
charge = stripe.Charge.create(
amount=total_gross_amount_in_smallest_unit, # TODO Marc > https://stripe.com/docs/currencies#zero-decimal
application_fee=100, # TODO Marc: Which currency?
currency='eur', # TODO Marc
source=token,
stripe_account="ABC", # TODO Marc: Replace with organizer stripe account
idempotency_key=session_order_reference,
)
new_charge_obj = Charge.objects.create(
amount=charge.amount,
charge_id=charge.id,
livemode=charge.livemode,
paid=charge.paid,
refunded=charge.refunded,
currency=charge.currency,
failure_code=charge.failure_code,
failure_message=charge.failure_message,
fraud_details=charge.fraud_details,
outcome=charge.outcome,
status=charge.status,
application_fee=charge.application_fee,
captured=charge.captured,
created=charge.created,
# TODO Marc: Add refunds:
# amount_refunded=charge.amount_refunded,
# etc.
)
application_fee = stripe.ApplicationFee.retrieve(charge.application_fee)
Fee.objects.create(
fee_id=application_fee.id,
livemode=application_fee.livemode,
currency=application_fee.currency,
amount=application_fee.amount,
charge=new_charge_obj,
# TODO Marc: Add refunds
)
paid = new_charge_obj.paid
except stripe.error.CardError as e:
# Since it's a decline, stripe.error.CardError will be caught
body = e.json_body
err = body.get('error', {})
messages.add_message(
request,
messages.ERROR,
err.get('message')
)
# return redirect(
# 'orders:order-list',
# order_reference=new_order.order_reference,
# access_key=new_order.access_key,
# )
# print("Type is: %s") % err.get('type')
# print("Code is: %s") % err.get('code')
# # param is '' in this case
# print("Param is: %s") % err.get('param')
# print("Message is: %s") % err.get('message')
except stripe.error.RateLimitError as e:
# Too many requests made to the API too quickly
pass
except stripe.error.InvalidRequestError as e:
# Invalid parameters were supplied to Stripe's API
pass
except stripe.error.AuthenticationError as e:
# Authentication with Stripe's API failed
# (maybe you changed API keys recently)
pass
except stripe.error.APIConnectionError as e:
# Network communication with Stripe failed
pass
except stripe.error.StripeError as e:
# Display a very generic error to the user, and maybe send
# yourself an email
pass
except Exception as e:
# Something else happened, completely unrelated to Stripe
pass
if paid:
# Create new attendee
i = 1
attendee_list = []
for item in reserved_items:
for _ in range(item.quantity): # noqa
new_attendee_dict = {
'event': item.ticket.event,
'ticket': item.ticket,
'ticket_name': item.ticket.name,
'ticket_reference': session_order_reference + "-" + str(i),
'ticket_code': get_random_string(length=10),
}
i += 1
attendee_list.append(dict(new_attendee_dict))
# Create new order
new_order_dict = {
'total_gross': total_gross,
'total_tax': total_tax_amount,
'total_net': total_net,
'total_gross_converted': total_gross, # TODO Marc
'event': event,
'order_reference': session_order_reference,
'status': OrderStatus.PENDING,
'access_key': get_random_string(length=10),
}
new_order = order.save(commit=False)
[setattr(new_order, k, v) for k, v in new_order_dict.items()]
new_order.save()
# Create order items
for item in order_items_list:
OrderItem.objects.create(order=new_order, **item)
# Create attendees
for item in attendee_list:
Attendee.objects.create(order=new_order, **item)
# Create billing profile
billing_profile = billing.save(commit=False)
billing_profile.order = new_order
billing_profile.save()
# Delete order_reference session
del request.session['order_reference']
return redirect(
'orders:order-list',
order_reference=new_order.order_reference,
access_key=new_order.access_key,
)
else:
billing = BillingForm(prefix='billing')
order = OrderForm(prefix='order')
context = {
'reserved_items': reserved_items,
'taxes': taxes_dict,
'total_net': total_net,
'total_gross': total_gross,
'currency': event.currency,
'order': order,
'billing': billing,
}
return render(request, 'checkout/checkout.html', context)
The problem is not with anything that you've changed, but rather what you haven't changed :)
On this line you are passing an idempotency_key:
charge = stripe.Charge.create(
...
idempotency_key=session_order_reference,
)
As described in the Stripe docs, you can pass an idempotency key with a request, which allows you to make the same request again in the future, using the same key, and you will get the same result as the first request. This is useful in case you didn't recieve the first response because of a network issue.
In this case, you have changed the CVC, which creates a new token variable. This means that your request is not identical to the previous request that used the same idempotency key. That doesn't make sense as you can only use the same idempotency key with identical requests, so you get this error from Stripe.
To resolve this, you should retry the charge creation using a freshly generated idempotency key. Generally, the key should be generated on each unique request that your application creates.
Had a similar issue where I was passing an indempotency_key and clients were not able to pay after their card was declined because the data that was sent was unique to the charge but not to the card. If for example their CVC was incorrect the subsequent charge will get created with the exact same idempotency key because data pertinent to the actually card was not taken into account.
The fix for this is to make sure your key is unique to the charge AND the card in this case including the card token can fix this.
Some other things to think about are things like partial payments, refunds, same/different ip, other metadata.
Stripe handles this case
when you are submitting wrong cvv
I tested with stripe test credit cards
https://stripe.com/docs/testing#cards
use that one which fails with cvv code than use valid card.
In my project I have requests library that sends POST request. Url for that request is hardcoded in function, which is accessed from views.py.
The problem is that when I dont have internet connection, or host, on which url is pointing, is down, I cant launch developer server, it gets stuck on Performing system check. However, if I comment the line with url, or change it to guarantee working host, check is going well.
What is good workaround here ?
views.py
def index(request):
s = Sync()
s.do()
return HttpResponse("Hello, world. You're at the polls index.")
sync.py
class Sync:
def do(self):
reservations = Reservation.objects.filter(is_synced=False)
for reservation in reservations:
serializer = ReservationPKSerializer(reservation)
dictionary = {'url': 'url', 'hash': 'hash', 'json': serializer.data}
encoded_data = json.dumps(dictionary)
r = requests.post('http://gservice.ca29983.tmweb.ru/gdocs/do.php', headers={'Content-Type': 'application/json'}, data=encoded_data)
if r.status_code == 200:
reservation.is_synced = True
reservation.save()
It might appear to be stuck because requests automatically retries the connection a few times. Try reducing the retry count to 0 or 1 with:
Can I set max_retries for requests.request?
am using python and google app engine majorly on jinja2 templates
i could like when a user registers a new account, they get a popup indicating that their registration is successful of even any alert on the very interface before moving to the next registration step.
def post(self):
user = (str(users.get_current_user().email()))
userquery = Users.query(Users.email == user)
count = userquery.count()
if count == 0:
#test if user is admin or employee
qry = Users.query()
count = qry.count()
if count == 0:
privilage = 'admin'
db_put = Users(
f_name=self.request.get("f_name"),
l_name = self.request.get("l_name"),
org = self.request.get("org"),
email=users.get_current_user().email(),
privilage = privilage
)
db_put.put()
How are you calling this POST method? Are you sending a form there directly (use method 1) or is this being done with an AJAX call (use method 2)?
Method 1
You can redirect to a GET page where you render a template with a success or error message for Jinja to use. This would however involve a page change.
import webapp2
class MyHandler(webapp2.RequestHandler):
def get(self): # Let's assume /someurl is mapped to this handler.
template_values = {}
notification = self.request.get('notification')
if notification:
template_values['notification'] = notification
self.response.set_status(200)
self.response.headers['Content-Type'] = 'text/html; charset=utf-8'
# Need to get the template from jinja and set it as template variable.
self.response.out.write(template.render(template_values))
def post(self):
# Do all your stuff here.
self.redirect('/someurl?notification=Success')
Alternatively you can set the parameters directly on the request instead of passing them as URI parameters:
def post(self):
# Do all your stuff here.
self.redirect('/someurl, params={'notification': 'Success'})
Method 2
In this method you can send back a JSON response with a success or error message. The caller (whatever function in your javascript that submitted the request to the backend) can use that to render a butterbar message or other popup notification of your choosing:
import json
import webapp2
class MyHandler(webapp2.RequestHandler):
def post(self):
# Do all your stuff here.
self.response.set_status(200)
self.response.headers['Content-Type'] = 'application/json; charset=utf-8'
self.response.headers['Content-Disposition'] = 'attachment'
self.response.out.write(json.JsonEncoder(sort_keys=True).encode('Success'))
For the latter, make sure you think about cross-site scripting (XSS) vulnerabilities and perhaps add a JSON prefix.
How can I get the request url in Scrapy's parse() function? I have a lot of urls in start_urls and some of them redirect my spider to homepage and as result I have an empty item. So I need something like item['start_url'] = request.url to store these urls. I'm using the BaseSpider.
The 'response' variable that's passed to parse() has the info you want. You shouldn't need to override anything.
eg. (EDITED)
def parse(self, response):
print "URL: " + response.request.url
The request object is accessible from the response object, therefore you can do the following:
def parse(self, response):
item['start_url'] = response.request.url
Instead of storing requested URL's somewhere and also scrapy processed URL's are not in same sequence as provided in start_urls.
By using below,
response.request.meta['redirect_urls']
will give you the list of redirect happened like ['http://requested_url','https://redirected_url','https://final_redirected_url']
To access first URL from above list, you can use
response.request.meta['redirect_urls'][0]
For more, see doc.scrapy.org mentioned as :
RedirectMiddleware
This middleware handles redirection of requests based on response status.
The urls which the request goes through (while being redirected) can be found in the redirect_urls Request.meta key.
Hope this helps you
You need to override BaseSpider's make_requests_from_url(url) function to assign the start_url to the item and then use the Request.meta special keys to pass that item to the parse function
from scrapy.http import Request
# override method
def make_requests_from_url(self, url):
item = MyItem()
# assign url
item['start_url'] = url
request = Request(url, dont_filter=True)
# set the meta['item'] to use the item in the next call back
request.meta['item'] = item
return request
def parse(self, response):
# access and do something with the item in parse
item = response.meta['item']
item['other_url'] = response.url
return item
Hope that helps.
Python 3.5
Scrapy 1.5.0
from scrapy.http import Request
# override method
def start_requests(self):
for url in self.start_urls:
item = {'start_url': url}
request = Request(url, dont_filter=True)
# set the meta['item'] to use the item in the next call back
request.meta['item'] = item
yield request
# use meta variable
def parse(self, response):
url = response.meta['item']['start_url']