How to mount a snapshot from boto? - amazon-web-services

I'm using boto 2.5.1, Python 2.7, Ubuntu Precise. I want to mount a snapshot on an EC2 instance. I've gotten as far as creating a volume from the snapshot, but then I can't figure out how to attach it. If I do:
[setup stuff elided]
c = EC2Connection()
print volume
print instance
c.attach_volume(volume, instance, "/dev/snap")
I get the amazingly unhelpful exception:
vol-2df00677
i-1509d364
Traceback (most recent call last):
File "./mongo_pulldown.py", line 48, in <module>
main()
File "./mongo_pulldown.py", line 28, in main
c.attach_volume(volume, instance, "/dev/snap")
File "/home/roy/deploy/current/python/local/lib/python2.7/site-packages/boto/ec2/connection.py", line 1530, in attach_volume
return self.get_status('AttachVolume', params, verb='POST')
File "/home/roy/deploy/current/python/local/lib/python2.7/site-packages/boto/connection.py", line 985, in get_status
raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
with no clue of what I've done wrong. I'm assuming the device name is arbitrary, and the attach call will create the device as part of the process? Or does the device have the exist already?
How can I get some more useful diagnostic than just "Bad Request"?

The attach_volume method takes an instance_id and a volume_id but you are passing objects. Try this:
c.attach_volume(volume.id, instance.id, "/dev/sdh")
The device_name should be a reasonable device name for the OS you are using. You can find more about what that value should be here.
boto uses standard Python logging so you can configure it to log as much or as little as you want. This gist shows a shortcut approach to get full debug logging. However, boto can only log what it has access to and it's possible the response from EC2 just doesn't provide much information.

Well, it turns out that I was passing something bad for device. I was passing '/dev/snap'. When I changed that to 'xvdg', things worked. It appears that '/dev/xvdg' also works (and has the same effect; the '/dev/' part appears to be ignored).
I wrote a little function to find the next available unused device name:
def get_device_name():
for c in 'fghijklmnop':
name = "xvd%s" % c
path = "/dev/%s" % name
try:
os.stat(path)
except OSError:
return path
I was hoping that by using a fixed name outside of the set that's normally used, I could avoid having to worry about this silliness.

Related

Kaggle API in colab datasets `!kaggle datasets list` error

I have a problem, i don't understand this error, when trying to list kaggles datasets in google colab.
Notebook config: Python 3.x, no hdw acc.
#to upload my kaggle token
from google.colab import files
files.upload()
#setting up the token
!pip install --upgrade kaggle
!mkdir -p ~/.kaggle
!cp kaggle.json ~/.kaggle/
!chmod 600 ~/.kaggle/kaggle.json
#and taking a look at datasets
!kaggle datasets list
Traceback (most recent call last):
File "/usr/local/bin/kaggle", line 8, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/kaggle/cli.py", line 51, in main
out = args.func(**command_args)
File "/usr/local/lib/python3.6/dist-packages/kaggle/api/kaggle_api_extended.py", line 940, in dataset_list_cli
max_size, min_size)
File "/usr/local/lib/python3.6/dist-packages/kaggle/api/kaggle_api_extended.py", line 905, in dataset_list
return [Dataset(d) for d in datasets_list_result]
File "/usr/local/lib/python3.6/dist-packages/kaggle/api/kaggle_api_extended.py", line 905, in <listcomp>
return [Dataset(d) for d in datasets_list_result]
File "/usr/local/lib/python3.6/dist-packages/kaggle/models/kaggle_models_extended.py", line 67, in __init__
self.size = File.get_size(self.totalBytes)
File "/usr/local/lib/python3.6/dist-packages/kaggle/models/kaggle_models_extended.py", line 107, in get_size
while size >= 1024 and suffix_index < 4:
TypeError: '>=' not supported between instances of 'NoneType' and 'int'
well, I would like to understand what happened, and how to fix it. Thank's in the advance.
jet.
I am encountering this problem as well. I noticed that if I set the use this call
kaggle datasets list --min-size 1
It will work. Note you will need version 1.5.6. I had 1.5.4 on a Colab instance and that version didn’t support that argument.
The problem seems to be bigquery/crypto-litecoin has no data. As a consequence of this, it looks like totalBytes is None in Dataset.
I've opened an issue on github and will created a PR. If you want a temporary work around, you can grab the file from my fork. You can use your traceback to determine where to put the file. Or alternatively, just use --min-size 1 so it will ignore the case when there are no data files.
I ran into the same problem.
Generate the Kaggle JSON API file. On the Widget/Icon in the top Right corner -> click "Account" -> Scroll down to "API" subsection, Click "Expire API Token" -> Click "Create New API Token"
In Google Colab. Upload your json file
Run the following code:
#first upload kaggle api file "kaggle.json" import os #this path contains the json file os.environ['KAGGLE_CONFIG_DIR'] = "/content"
#Find the competition or Dataset under Data. Like this: !kaggle competitions download -c jane-street-market-prediction
This worked for me after a lot of banging my head against the wall.
If you get errors still, you may need to link your Colab and Kaggle accounts. You can do this in the account settings portion of kaggle.

Win32Com Error Running with Task Manager

I have a python script that calls open an Excel file, calls a macro in the file, then closes. If I run the file from the CLI, it works. If i put it in Task Scheduler, I get an error from win32com.
Method opening the Excel file:
import win32com.client as WinCom
if os.path.exists(reportGeneratorFileName):
try:
xl = WinCom.Dispatch("Excel.Application")
xl.Workbooks.Open(Filename=os.path.abspath(reportGeneratorFileName))
xl.Application.Visible = False
xl.Application.Run("'{}'!Runner.Runner".format(reportGeneratorFileName))
l.info('Start Sleeping')
# Async mode of pythonw causes this to finish before the file is made
time.sleep(300)
l.info('Done Sleeping')
xl.Application.Quit()
except Exception as e:
l.error('Error updating file')
l.error(e, exc_info=True)
This is the Error i get:
06/04/2018 06:56:19 AM ERROR: (-2146959355, 'Server execution failed', None, None)
Traceback (most recent call last):
File "LAW Report.py", line 846, in createReport
xl = WinCom.Dispatch("Excel.Application")
File "c:\python27\lib\site-packages\win32com\client\__init__.py", line 95, in Dispatch
dispatch, userName = dynamic._GetGoodDispatchAndUserName(dispatch,userName,clsctx)
File "c:\python27\lib\site-packages\win32com\client\dynamic.py", line 114, in _GetGoodDispatchAndUserName
return (_GetGoodDispatch(IDispatch, clsctx), userName)
File "c:\python27\lib\site-packages\win32com\client\dynamic.py", line 91, in _GetGoodDispatch
IDispatch = pythoncom.CoCreateInstance(IDispatch, None, clsctx, pythoncom.IID_IDispatch)
com_error: (-2146959355, 'Server execution failed', None, None)
Now i am running 64 bit python 2.7 and 64 bit win32com while office is 32 bit, but as I said above, if I just run the script from the CLI it runs fine, just not from task manager. I am running this on a Windows Server 2012R2. I have tried configuring the task for 2008, 2008r2, and 2012r2. I also tried with highest privileges. I do need this to be able to run whether user is logged on or not. Everytime I have tested, the user has been logged on.
You need to
1) setup permissions in DCOMCnfg utility for user under which you're running -or-
2) change user to your desktop user account.
But first you need to get proper error code, not a generic message.
Pay attention that this should be considered as temporary solution since DCOM is badly supported by MS for automation as service. Consider switching to any library as soon as possible. There're numerous problems with Excel interop: you will have to reboot server from time to time, performance degradation with subsequent calls, you can't run Excels in parallel because of a good chance of errors, etc...
I encountered the same error of which my program can run from command prompt but not from task scheduler.
I solved this issue with these settings of task properties in task scheduler:
1. General->Security Options
a. make sure same user account you used to run python/microsoft program.
b. checked "Run only when user is logged on.
c. unchecked "Hidden"
For 1a, I'm using DOMAIN\userid that I used to login my laptop/PC.
This is the only way I found in order for the task scheduler to run successfully. Hope this helps.

AWS X-Ray ERROR:aws_xray_sdk.core.context:cannot find the current segment/subsegment

We recently added X-Ray to our code by having:
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
patch_all()
While this runs fine on AWS Lambda, but when trying to run locally during calling ElasticSearch we got the following exception:
ERROR:aws_xray_sdk.core.context:cannot find the current segment/subsegment, please make sure you have a segment open
queryCustomers - DEBUG - Caught exception for <function search_customer at 0x10bfcf0d0>
Traceback (most recent call last):
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/chalice/app.py", line 659, in _get_view_function_response
response = view_function(**function_args)
File "/Users/jameslin/projects/test-project/src/app.py", line 57, in search_customer
return query[0:size].execute().to_dict()['hits']['hits']
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/elasticsearch_dsl/search.py", line 639, in execute
**self._params
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 73, in _wrapped
return func(*args, params=params, **kwargs)
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/elasticsearch/client/__init__.py", line 632, in search
doc_type, '_search'), params=params, body=body)
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/elasticsearch/transport.py", line 312, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/elasticsearch/connection/http_requests.py", line 71, in perform_request
prepared_request = self.session.prepare_request(request)
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/aws_xray_sdk/ext/requests/patch.py", line 38, in _inject_header
inject_trace_header(headers, xray_recorder.current_subsegment())
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/aws_xray_sdk/core/recorder.py", line 251, in current_subsegment
entity = self.get_trace_entity()
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/aws_xray_sdk/core/recorder.py", line 316, in get_trace_entity
return self.context.get_trace_entity()
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/aws_xray_sdk/core/context.py", line 93, in get_trace_entity
return self.handle_context_missing()
File "/Users/jameslin/virtualenvs/test-project/lib/python3.6/site-packages/aws_xray_sdk/core/context.py", line 118, in handle_context_missing
raise SegmentNotFoundException(MISSING_SEGMENT_MSG)
aws_xray_sdk.core.exceptions.exceptions.SegmentNotFoundException: cannot find the current segment/subsegment, please make sure you have a segment open
I have no idea what his means and how to get rid of it, my google attempts gives not many relevant results and I also tried running the x-ray daemon locally but still having the same problem:
./xray_mac -o -n ap-southeast-2
When your code is running on AWS Lambda with tracing enabled, Lambda container will generate a segment representing the whole function invocation. It also sets the context as the environment variable so the SDK can link any subsegment created inside the function back to the parent segment.
If you run the same code locally the SDK still tries to create subsegments for the actual function code but it can't find any context, thus throwing the error you posted.
To solve this you will need to setup some environment variables to make sure the SDK has the same information as it were running on an actual Lambda container.
Make sure the SDK thinks it is running on a Lambda container by setting LAMBDA_TASK_ROOT with whatever value you'd like (only the presence of the key matters). You can see the source code here: https://github.com/aws/aws-xray-sdk-python/blob/master/aws_xray_sdk/core/lambda_launcher.py
Setting LAMBDA_TRACE_HEADER_KEY so the function has a tracing context. The value must be a trace header and you can see more details here: https://docs.aws.amazon.com/lambda/latest/dg/lambda-x-ray.html
This workaround is not ideal as it requires extra code changes from user side. We would like to provide better customer experience for testing X-Ray instrumented Lambda function locally. Do you mind sharing more details about how you are doing local testing and how you expect X-Ray tracing works in such testing environment, so we can have better improvement for your use case?

xlwings Workbook() call returns AttributeError: Open.Application

Hope you can help me out, first question I ask myself but I am always impressed by the professional answers I find here!
I am using xlwings to both read and write data from .xls files. I am not a seasoned programmer, and I make mistakes. From time to time, this means rebooting processes, Python and what-not.
Every now and again, I would get the following AttributeError upon calling Workbook(). The real problem is not that I don't have a clue what is happening (I tried reading the modules the error diagnostics reference to, but they are written on a level of Python that is beyond my skills), but that restarting and even rebooting(!) my laptop don't (always) solve the problem.
I wish I could give some clues as to what is happening, but it appears to be a randomly occurring problem and right now, three reboots and several restarts didn't fix it. Google can't help either. I must have run my program a hundred times by now and it wasn't always a problem! The error message, upon simply calling Workbook():
wb = Workbook()
Traceback (most recent call last):
File "<ipython-input-22-4a3c36eb9bf9>", line 1, in <module>
wb = Workbook()
File "C:\Program Files\Anaconda\lib\site-packages\xlwings\main.py", line 141, in __init__
self.xl_app, self.xl_workbook = xlplatform.new_workbook()
File "C:\Program Files\Anaconda\lib\site-packages\xlwings\_xlwindows.py", line 104, in new_workbook
xl_app = _get_latest_app()
File "C:\Program Files\Anaconda\lib\site-packages\xlwings\_xlwindows.py", line 88, in _get_latest_app
return xl_workbook_current.Application
File "C:\Program Files\Anaconda\lib\site-packages\win32com\client\dynamic.py", line 522, in __getattr__
raise AttributeError("%s.%s" % (self._username_, attr))
AttributeError: Open.Application
And as a follow up, the piece of code from 'dyamic.py' raising the error:
[...]
# If we are still here, and have a retEntry, get the OLE item
if not retEntry is None:
invoke_type = _GetDescInvokeType(retEntry, pythoncom.INVOKE_PROPERTYGET)
debug_attr_print("Getting property Id 0x%x from OLE object" % retEntry.dispid)
try:
ret = self._oleobj_.Invoke(retEntry.dispid,0,invoke_type,1)
except pythoncom.com_error, details:
if details.hresult in ERRORS_BAD_CONTEXT:
# May be a method.
self._olerepr_.mapFuncs[attr] = retEntry
return self._make_method_(attr)
raise
debug_attr_print("OLE returned ", ret)
return self._get_good_object_(ret)
# no where else to look.
raise AttributeError("%s.%s" % (self._username_, attr))
As I mentioned before, the last bit it too much for me :P
I guessed some lingering Excel.exe (or similar) process was interfering with the script after I closed some random window due to some random error, but I cant find one in the task manager. Also, this is the kind of thing a reboot should fix, right?! I also upgraded to xlwings .34 (latest version, as of now).
Windows 8.1 64 bit, Python 2.7 (Anaconda distribution, Spyder IDE), Excel 2013.
Any help is very much appreciated, of course!
Thanks a lot, kind regards,
Gordon.
EDIT:
Following advice given in the comments, I can now report that even at times when wb = Workbook() fails, from win32com.client import dynamic followed by dynamic.Dispatch('Excel.Application') returns <COMObject Excel.Application>.
Hope it helps!
Always check either any issues to open the excel sheet, such as password protected/Licensed issues..
If there is any pop up coming while opening excel, you may get the same error.

Simple reading/writing from/to a USB HID device in Python?

I've got a fairly simple USB HID device that I've been trying to figure out how to read from and write to using Python. I've been able to read from it using PyWinUSB, but the problem comes in when I try to write to it. Trying to write to it makes things explode.
For example:
device = hid.HidDeviceFilter(vendor_id = 0x0003, product_id = 0x1001).get_devices()[0]
This works fine. Then for reading raw data, which is all that I care about right now (I'll work with that once I can figure out how to write to the cursed thing):
def readData(data):
print(data)
return None
This works fine (in fact, it was quite exciting when I got to see it work). So I would assign the data handler like so:
device.set_raw_data_handler(readData)
And every time I hit a button, it's fine. The data comes through as you would expect. Which is great!
The problem comes when I want to write to the device.
Following the sample simple_send file as a template (which was probably not the best choice), I would do the following:
report = device.find_output_reports()[0]
Which would return a report object with a dictionary holding 4 entries. Is that correct? Do you write to a device using the output_reports object? Trying to do so by setting the report value to ANYTHING:
report[<key>] = "pneumonoultramicroscopicvolcanoconiosis"
report.send()
This would keep returning some obnoxious error that I can't interpret:
Traceback (most recent call last):
File "<pyshell#21>", line 1, in <module>
report.send()
File "C:\Python27\lib\site-packages\pywinusb-0.3.1-py2.7.egg\pywinusb\hid\core.py", line 1446, in send
self.__prepare_raw_data()
File "C:\Python27\lib\site-packages\pywinusb-0.3.1-py2.7.egg\pywinusb\hid\core.py", line 1401, in __prepare_raw_data
byref(self.__raw_data), self.__raw_report_size) )
File "C:\Python27\lib\site-packages\pywinusb-0.3.1-py2.7.egg\pywinusb\hid\winapi.py", line 382, in __init__
raise helpers.HIDError("hidP error: %s" % self.error_message_dict[error_code])
HIDError: hidP error: data index not found
I'm using Windows 7. I've managed to find (finally) a reference for the HID DLL exported functions, and I don't HAVE to (or, for that matter even really WANT to) use the PyWinUSB library. I just want to make this work, and it didn't seem like it would be that tough, but it has been.
Can someone tell me what it is I've been doing wrong here?
Thanks.
Also, I tried tracing the error call, and made it so far before the program just closed which was kind of disheartening.
i made it work with this
buffer= [0xFF]*33 # 33 = report size + 1 byte (report id)
buffer[0]=0x0 # report id
buffer[1]=0xFE
buffer[2]=0x00
buffer[3]=0xFF
out_report.set_raw_data(buffer)
out_report.send()
dev.close()
For me worked only this:
report.send([0x70, ..., 0x73 ])
The function call sequence with set_raw_data([0x70, ..., 0x73) and subsequent send() didn't work for me.