I have a QuerySet consisting of a timestamp and a boolean value. When there are consecutive repeating values, I'd like to truncate them to only show the first (and maybe the last one too, but not mandatory).
e.g. the given queryet
<TimescaleQuerySet [
{'time': datetime.datetime(2023, 1, 23, 10, 57, 7, 971506, tzinfo=<UTC>), 'is_on': True},
{'time': datetime.datetime(2023, 1, 23, 10, 53, 11, 787303, tzinfo=<UTC>), 'is_on': True},
{'time': datetime.datetime(2023, 1, 23, 10, 53, 20, 646474, tzinfo=<UTC>), 'is_on': False},
{'time': datetime.datetime(2023, 1, 23, 10, 27, 7, 971506, tzinfo=<UTC>), 'is_on': False},
{'time': datetime.datetime(2023, 1, 23, 10, 23, 20, 646474, tzinfo=<UTC>), 'is_on': False},
{'time': datetime.datetime(2023, 1, 23, 10, 23, 11, 787303, tzinfo=<UTC>), 'is_on': True},
{'time': datetime.datetime(2023, 1, 23, 9, 57, 7, 971506, tzinfo=<UTC>), 'is_on': True}]
should truncate to
<TimescaleQuerySet [
{'time': datetime.datetime(2023, 1, 23, 10, 57, 7, 971506, tzinfo=<UTC>), 'is_on': True},
{'time': datetime.datetime(2023, 1, 23, 10, 53, 20, 646474, tzinfo=<UTC>), 'is_on': False},
{'time': datetime.datetime(2023, 1, 23, 10, 23, 11, 787303, tzinfo=<UTC>), 'is_on': True}]
I'm breaking my head over this. Is there an elegant way to achieve this? I want to avoid looping over the whole queryset, it's just too slow when theres >1000 results.
Related
I am using MongoDB and SimpleJWT in DjangoREST to authenticate and authorize users. I tried to implement user logout, whereby in SimpleJWT it's basically blacklisting a user token. When the first user logs in, everything seems okay and their refresh token is added to the Outstanding token table. But when I try to log in a second user, I get the below error:
raise BulkWriteError(full_result)
pymongo.errors.BulkWriteError: batch op errors occurred, full error: {'writeErrors': [{'index': 0, 'code': 11000, 'keyPattern': {'jti_hex': 1}, 'keyValue': {'jti_hex': None}, 'errmsg': 'E11000 duplicate key error collection: fsm_database.token_blacklist_outstandingtoken index: token_blacklist_outstandingtoken_jti_hex_d9bdf6f7_uniq dup key: { jti_hex: null
}', 'op': {'id': 19, 'user_id': 7, 'jti': '43bccc686fc648f5b60b22df3676b434', 'token': 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoicmVmcmVzaCIsImV4cCI6MTY1OTY1NDUzNCwiaWF0IjoxNjU5NTY4MTM0LCJqdGkiOiI0M2JjY2M2ODZmYzY0OGY1YjYwYjIyZGYzNjc2YjQzNCIsInVzZXJfaWQiOjd9.aQmt5xAyncfpv_kDD2pF7iS98Hld98LhG6ng-rCW23M', 'created_at': datetime.datetime(2022,
8, 3, 23, 8, 54, 125539), 'expires_at': datetime.datetime(2022, 8, 4, 23, 8, 54), '_id': ObjectId('62eb00064621b38109bbae16')}}], 'writeConcernErrors': [], 'nInserted': 0, 'nUpserted': 0, 'nMatched': 0, 'nModified': 0, 'nRemoved': 0, 'upserted': []}
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\MR.Robot\.virtualenvs\fsm-GjGxZg3c\lib\site-packages\djongo\cursor.py", line 51, in execute
self.result = Query(
File "C:\Users\MR.Robot\.virtualenvs\fsm-GjGxZg3c\lib\site-packages\djongo\sql2mongo\query.py", line 784, in __init__
self._query = self.parse()
File "C:\Users\MR.Robot\.virtualenvs\fsm-GjGxZg3c\lib\site-packages\djongo\sql2mongo\query.py", line 869, in parse
raise exe from e
djongo.exceptions.SQLDecodeError:
Keyword: None
Sub SQL: None
FAILED SQL: INSERT INTO "token_blacklist_outstandingtoken" ("user_id", "jti", "token", "created_at", "expires_at") VALUES (%(0)s, %(1)s, %(2)s, %(3)s, %(4)s)
Params: [7, '43bccc686fc648f5b60b22df3676b434', 'eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoicmVmcmVzaCIsImV4cCI6MTY1OTY1NDUzNCwiaWF0IjoxNjU5NTY4MTM0LCJqdGkiOiI0M2JjY2M2ODZmYzY0OGY1YjYwYjIyZGYzNjc2YjQzNCIsInVzZXJfaWQiOjd9.aQmt5xAyncfpv_kDD2pF7iS98Hld98LhG6ng-rCW23M', datetime.datetime(2022, 8, 3, 23, 8, 54, 125539), datetime.datetime(2022, 8, 4, 23, 8, 54)]
Version: 1.3.6
MongoDB seems to have a problem inserting the token for the second user in the outstanding table.
How can I fix this?
So I asked the library maintainers and they said that they don't support MongoDB. Check out this issue.
i am mounting multiple ebs volumes on an ec2 instance to perform some validations and check-sums but the problem is that before mounting volumes on the target mount point , i need to check whether that mount is not occupied, since i am creating mount points dynamically e.g
test_lambda.py (e.g)
device_prefixes = ['f','g','h','i','j','k','l','m','n','o','p']
attached_volumes = []
try:
for index,volume_id in enumerate(created_volumes):
MOUNT_POINT = "/dev/sd{}".format(device_prefixes[index])
I have tried commands using SSM like /proc/mounts, findmnt, but they are displaying partitions and other volumes but not target like (/dev/sde,/dev/sdf etc)
Thanks to the hint provided by jordan, i was able to clear my understanding about block device mapping , boto3 ec2 client has a method associated
`# where attribute can be one of the following` `"""'|'kernel'|'ramdisk'|'userData'|'disableApiTermination'|'instanceInitiatedShutdownBehavior'|'rootDeviceName'|'blockDeviceMapping'|'productCodes'|'sourceDestCheck'|'groupSet'|'ebsOptimized'|'sriovNetSupport'|'enaSupport'|'enclaveOptions'|'disableApiStop',z"""`
print(instance.describe_attribute(
instance.describe_attribute(
Attribute='blockDeviceMapping',
DryRun=False,
)
it does give us info about certain resources, apologies if my question was not clear in the first place,
{'BlockDeviceMappings': [{'DeviceName': '/dev/xvda', 'Ebs':
{'AttachTime': datetime.datetime(2022, 5, 27, 12, 4, 1,
tzinfo=tzlocal()), 'DeleteOnTermination': True, 'Status': 'attached',
'VolumeId': 'vol-05bb41df0e6bc4bf3'}}, {'DeviceName': '/dev/sdl',
'Ebs': {'AttachTime': datetime.datetime(2022, 5, 27, 15, 15, 51,
tzinfo=tzlocal()), 'DeleteOnTermination': False, 'Status': 'attached',
'VolumeId': 'vol-00be0e0f001fea1a9'}}, {'DeviceName': '/dev/sdm',
'Ebs': {'AttachTime': datetime.datetime(2022, 5, 27, 15, 15, 51,
tzinfo=tzlocal()), 'DeleteOnTermination': False, 'Status': 'attached',
'VolumeId': 'vol-09c552e77ba2a563f'}}, {'DeviceName': '/dev/sdn',
'Ebs': {'AttachTime': datetime.datetime(2022, 5, 27, 15, 15, 52,
tzinfo=tzlocal()), 'DeleteOnTermination': False, 'Status': 'attached',
'VolumeId': 'vol-08d28a4ffe5e71dff'}}, {'DeviceName': '/dev/sdo',
'Ebs': {'AttachTime': datetime.datetime(2022, 5, 27, 15, 15, 53,
tzinfo=tzlocal()), 'DeleteOnTermination': False, 'Status': 'attached',
'VolumeId': 'vol-0e63fa17bd6185ee3'}}, {'DeviceName': '/dev/sdf',
'Ebs': {'AttachTime': datetime.datetime(2022, 5, 27, 15, 36, 42,
tzinfo=tzlocal()), 'DeleteOnTermination': False, 'Status': 'attached',
'VolumeId': 'vol-058de095fcfb1ef05'}}, {'DeviceName': '/dev/sdg',
'Ebs': {'Atta
I am trying to merge coverage data from specific files from multiple runs into a single "report". In each project I generate test coverage for different files, each in a separate project folder and then at the end I pick and choose what files to include in my final project. If I use the following command line where I select each file with a --filter [complete path and filename], it works perfectly and I get a combined report with just these files. I also see it pick the files up in the log since I have -v option enabled shown below.
gcovr -g -k -s --html --html-details -o code_coverage_summary2.html -r . --filter firmware/crc32/crc_unit_tests.X/mcc_generated_files/boot/boot_verify_crc32.c --filter firmware/sha256/sha_unit_tests.X/mcc_generated_files/boot/boot_verify_sha256.c --filter firmware/checksum/checksum_unit_tests.X/mcc_generated_files/boot/boot_verify_checksum16.c --filter firmware/command_processor/bootloader_pic_tb.X/mcc_generated_files/boot/boot_process.c -v > log4.txt
Log File From above run
(many many lines before)
currdir C:\Users\murphy\Downloads\archive (5)\archive\src\test
gcov_fname C:\Users\murphy\Downloads\archive (5)\archive\src\test\firmware\sha256\sha_unit_tests.X\mcc_generated_files\boot\boot_verify_sha256.c.gcov
[' -', ' 0', 'Source', 'boot_verify_sha256.c\n']
source_fname None
root C:\Users\murphy\Downloads\archive (5)\archive\src\test
fname C:\Users\murphy\Downloads\archive (5)\archive\src\test\firmware\sha256\sha_unit_tests.X\mcc_generated_files\boot\boot_verify_sha256.c
Parsing coverage data for file C:\Users\murphy\Downloads\archive (5)\archive\src\test\firmware\sha256\sha_unit_tests.X\mcc_generated_files\boot\boot_verify_sha256.c
uncovered: {122}
covered: {49: 1, 54: 1, 55: 1, 57: 1, 59: 1, 60: 1, 61: 1, 64: 1, 65: 1, 70: 1, 75: 1, 80: 1, 81: 1, 83: 1, 85: 1, 88: 1, 90: 1, 92: 1, 95: 1, 97: 1, 100: 1, 102: 1, 105: 1, 107: 1, 110: 1, 112: 1, 115: 1, 117: 1, 120: 1, 125: 1, 127: 1, 132: 1}
branches: {}
noncode: {128, 129, 3, 131, 133, 6, 134, 9, 12, 23, 27, 30, 31, 33, 35, 37, 39, 44, 46, 48, 50, 51, 53, 56, 58, 62, 63, 66, 67, 69, 71, 72, 74, 76, 77, 79, 82, 84, 86, 87, 89, 91, 93, 94, 96, 98, 99, 101, 103, 104, 106, 108, 109, 111, 113, 114, 116, 118, 119, 121, 123, 124, 126}
Finding source file corresponding to a gcov data file
currdir C:\Users\murphy\Downloads\archive (5)\archive\src\test
gcov_fname C:\Users\murphy\Downloads\archive (5)\archive\src\test\firmware\crc32\crc_unit_tests.X\mcc_generated_files\boot\boot_verify_crc32.c.gcov
[' -', ' 0', 'Source', 'boot_verify_crc32.c\n']
source_fname None
root C:\Users\murphy\Downloads\archive (5)\archive\src\test
fname C:\Users\murphy\Downloads\archive (5)\archive\src\test\firmware\crc32\crc_unit_tests.X\mcc_generated_files\boot\boot_verify_crc32.c
Parsing coverage data for file C:\Users\murphy\Downloads\archive (5)\archive\src\test\firmware\crc32\crc_unit_tests.X\mcc_generated_files\boot\boot_verify_crc32.c
uncovered: set()
covered: {50: 1, 55: 1, 56: 1, 57: 1, 62: 1, 63: 1, 65: 1, 67: 1, 70: 1, 72: 1, 74: 1, 77: 1, 79: 1, 82: 1, 84: 1, 87: 1, 89: 1, 92: 1, 94: 1, 97: 1, 99: 1, 102: 1, 104: 1, 108: 1, 109: 1, 110: 1, 112: 1, 114: 1, 120: 1, 121: 1, 123: 1, 128: 1, 129: 1, 131: 1, 134: 1, 139: 1}
branches: {}
noncode: {3, 132, 133, 6, 135, 136, 9, 138, 12, 140, 141, 23, 28, 31, 33, 35, 38, 40, 45, 47, 49, 51, 52, 54, 58, 59, 61, 64, 66, 68, 69, 71, 73, 75, 76, 78, 80, 81, 83, 85, 86, 88, 90, 91, 93, 95, 96, 98, 100, 101, 103, 105, 106, 111, 113, 115, 122, 124}
Finding source file corresponding to a gcov data file
(many many lines after)
I have many more files to add and I really do not want to have a specific filter for each unique path & filename. What I want is a filter for all of the files that have the pattern boot_verify*.c in them. I have tried different regex expressions shown below but they all return no files selected.
--filter '(.+/)?boot_verify(.*)\.c$'
--filter '(.+/)?boot_verify_([a-z][A-Z][0-9]+)\.c$'
--filter '(.+/)?boot_verify_([a-z][A-Z][0-9]*)\.c$'
To force the issue I tried a regex with just a single file name below and even that did not work.
--filter '(.+/)?boot_verify_crc32\.c$'
I would really like to get the wild card pattern and the specific filename pattern working. What am I doing wrong?
Ken
When using regexes in command line arguments, they must be properly escaped.
When using Bash or another POSIX shell on Linux or other Unix-like operating systems including MacOS, surround the regex with single quotes. For example:
gcovr --filter '(.+/)?boot_verify_crc32\.c$'
See the Quoting section in the Bash Reference Manual or in the POSIX spec. In particular, escaping/quoting is necessary if the regex contains any of the following characters, or space characters:
| & ; < > ( ) $ ` \ " ' * ? [ # ˜ = %
When using cmd.exe on Windows, surround the regex with double quotes:
gcovr --filter "(.+/)?boot_verify_crc32\.c$"
I cannot find a good reference for this, though Parsing C Command-Line Arguments seems to contain the relevant rules.
When using PowerShell on Windows, surround the regex with single quotes:
gcovr --filter '(.+/)?boot_verify_crc32\.c$'
See the sections about Quoting Rules and about Parsing in the PowerShell documentation. In particular, escaping/quoting is necessary if the regex contains any of the following characters, or space characters:
' " ` , ; ( ) { } | & < > # #
Additionally, please note that gcovr 4.1 is from 2018. Since then, some fixes relating to Windows support and filter matching have been implemented. Assuming that you installed gcovr as pip3 install gcovr, you can upgrade to the latest stable release with:
pip3 install -U gcovr
I am trying to get request and response count in scrapyd,while running multiple spider means 8 spider dynamically.I am try to get those count using python.
following counts:
enter code here{'downloader/request_bytes': 130427,
'downloader/request_count': 273,
'downloader/request_method_count/GET': 273,
'downloader/response_bytes': 2169984,
'downloader/response_count': 273,
'downloader/response_status_count/200': 271,
'downloader/response_status_count/404': 2,
'dupefilter/filtered': 416,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2015, 7, 21, 14, 21, 38, 13000),
'item_scraped_count': 119,
'log_count/DEBUG': 406,
'log_count/INFO': 44,
'offsite/domains': 9,
'offsite/filtered': 5865,
'request_depth_max': 14,
'response_received_count': 273,
'scheduler/dequeued': 273,
'scheduler/dequeued/memory': 273,
'scheduler/enqueued': 273,
'scheduler/enqueued/memory': 273,
'start_time': datetime.datetime(2015, 7, 21, 14, 16, 41, 144000)}
enter code here
Thanks,
Use the Stats Collection from Scrapy.
With this you can access the statistics dumped at the end to the console and if you write your own middleware you can combine the results of your 8 spiders together -- like in the example of the documentation.
I have a dictionary list of items. For example
activities = [
{'start': datetime.datetime(2015, 9, 21, 8, 0), 'end': datetime.datetime(2015, 9, 21, 9, 0),
u'name': u'First Activity', 'description': 'desc of first activity...'},
{'start': datetime.datetime(2015, 9, 21, 9, 0), 'end': datetime.datetime(2015, 9, 21, 10, 0),
u'name': u'Second Activity', 'description': 'desc of next activity...'},
{'start': datetime.datetime(2015, 9, 22, 8, 0), 'end': datetime.datetime(2015, 9, 21, 9, 0),
u'name': u'Third Activity', 'description': 'desc of third activity...'},
{'start': datetime.datetime(2015, 9, 23, 8, 0), 'end': datetime.datetime(2015, 9, 21, 9, 0),
u'name': u'Next Activity', 'description': 'desc of next activity...'},
{'start': datetime.datetime(2015, 9, 23, 9, 0), 'end': datetime.datetime(2015, 9, 21, 10, 0),
u'name': u'Another Activity', 'description': 'desc of another activity...'},
{'start': datetime.datetime(2015, 9, 21, 10, 0), 'end': datetime.datetime(2015, 9, 21, 11, 0),
u'name': u'New Activity', 'description': 'desc of new activity...'}
]
I would like to do a regroup, first by the day, then perform a second regroup by the hour.
I'm having trouble enough just getting a regroup to work by either the day or by hour.
For example:
{% regroup activities by start|date:"l, n/d" as activity_list_by_day %}
{% for session in activity_list_by_day %}
{{ session.grouper }}
{% endfor %}
Isn't working out well. I don't see how to specify that I want the regroup to work by the day.
Is there some way to manipulate my 'start' item for these regroupings? Or might I need to create some extra items to work with?
Edited to add: I see now I can start out by sorting my activities list with:
activities.sort(key=lambda item:item['start'])
This helps tremendously. Actually might have solved my own problem...