S3 Select with boto3 - internalerror - amazon-web-services

Has anyone got "S3 Select" (https://aws.amazon.com/blogs/aws/s3-glacier-select/ ,
https://aws.amazon.com/about-aws/whats-new/2018/04/amazon-s3-select-is-now-generally-available/) with boto3 (or even cli or another sdk) working? I am getting cryptic InternalError below:
Running this on EC2 that has an IAM role:
[ec2-user#ip-blah bin]$ ./python
Python 2.7.13 (default, Jan 31 2018, 00:17:36)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> s3 = boto3.client('s3')
>>> r = s3.select_object_content(
... Bucket='mybucketname',
... Key='mypath/file.txt',
... ExpressionType='SQL',
... Expression="select count(*) from s3object s",
... InputSerialization = {'CSV': {"FileHeaderInfo": "Use"}},
... OutputSerialization = {'CSV': {}},
... )
Traceback (most recent call last):
File "<stdin>", line 7, in <module>
File "/home/ec2-user/venv/local/lib/python2.7/site-packages/botocore/client.py", line 314, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ec2-user/venv/local/lib/python2.7/site-packages/botocore/client.py", line 612, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InternalError) when calling the SelectObjectContent operation (reached max retries: 4): We encountered an internal error. Please try again.

My guesses:
check permissions on s3
adapt 'RecordDelimiter','FieldDelimiter', 'QuoteCharacter' if neccessary on InputSerialization
check structure on csv file (number of headers matches data columns, escaping spec. chars, whitespaces, /n as new line def.,.)
try
...
Expression="SELECT * FROM S3Object s", InputSerialization={'CSV': {}}, OutputSerialization={'CSV': {}}, ...
hope that helps a little!

Related

CERTIFICATE_VERIFY_FAILED in urllib2 but not requests, on Python 2.7.18

My longstanding Python 2.7.17 environment got corrupted, and so I installed Python 2.7.18 using the macOS 64-bit installer downloadable from python.org. Now I get errors when loading HTTPS sites using urllib2 but not requests.
Here is the environment, running macOS Mojave:
Python 2.7.18 (v2.7.18:8d21aa21f2, Apr 19 2020, 20:48:48)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
Works fine with requests:
>>> import requests
>>> url='https://github.com'
>>> requests.get(url)
<Response [200]>
Fails with urllib2:
>>> import urllib2
>>> urllib2.urlopen(url)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 429, in open
response = self._open(req, data)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 447, in _open
'_open', req)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 407, in _call_chain
result = func(*args)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1241, in https_open
context=self._context)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1198, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)>
This occurs in both the plain 2.7.18 I installed and in virtual environments created from it. The problem is bigger than just urllib2, as certain database modules fail, too, that do not use urllib2. Moving to Python3 is not currently an option.
I tried pip install --upgrade certifi but that did not help.
Per [1] monkey-patching fixes the problem, but that is not a viable solution because this is destined to be production code.
Per [1] explicitly setting a non-verifying SSL context sometimes works, but again, it is not viable, plus it fails in certain third-party database interface code.
The answer may be in PEPP [2] or PEPP 493 [3], but the sun will swallow the earth before I understand those.
[1] urllib and "SSL: CERTIFICATE_VERIFY_FAILED" Error
[2] https://www.python.org/dev/peps/pep-0476/
[3] https://www.python.org/dev/peps/pep-0493/

Using Django in Docker unable to fetch rest api

I have built the webapp using Django and also configured for Rest api. I am able to fetch the data via curl and also by python script. Issue I'm running into, when using Docker container, when user fill in the data and submit, it goes thru and ran the python function, but it stop working at when fetching the data via api and thru errors
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.6/site-packages/requests/models.py", line 897, in json
return complexjson.loads(self.text, **kwargs)
File "/usr/lib64/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
I'm able to curl the URL inside the container. I get the same error when shell into the container in python3.6
sh-4.2# python3.6
Python 3.6.8 (default, May 2 2019, 20:40:44)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-36)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import requests, json
>>> url = 'http://127.0.0.1:8001/db/data/'
>>> response = requests.get(url)
>>> parsedjson = json.loads(response.text)
ERROR SAME AS ABOVE . . .
parsedjson = response.json()
ERROR SAME AS ABOVE . . .
>>> print(response.status_code)
503
>>> exit()
sh-4.2# curl http://127.0.0.1:8001/db/data/
[{"center":{"name":"ABC"},"main":{"code":{"name":"ABC"},"subnet":"10.10.12.0",...."}]
sh-4.2#
any idea? how to fix it?
thank you for the help in advance..

Invalid token error in Telegram Bot API

I'm new to telegram bot api.
I installed telegrom package 📦 and started to run my first code.
but I cannot run my first code !
Can anyone know why this is happening? what should I do ?
Thank you 🙏
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 12:39:47)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import telegram
>>> bot = telegram.Bot(token='TOKEN')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/telegram/bot.py", line 53, in __init__
self.token = self._validate_token(token)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/telegram/bot.py", line 79, in _validate_token
raise InvalidToken()
telegram.error.InvalidToken: Invalid token
>>>
For your line
bot = telegram.Bot(token='TOKEN')
You need to replace 'TOKEN' with your bot's token, as the string 'TOKEN' is just a placeholder.
You may generate a bot token by talking to BotFather and creating a new bot.

trying to connect to Windows winRM using kerberos winrm

so I have tested this under several different variations. I have a lab with Windows 2008 R2 servers and RHEL6 and Python2.6 installed. I am able to use ansible as means to remote into the windows server and perform a ping (I have followed all the instructions on how to do this).
I get the problem of not being able to do this with Rhel 7 and python2.7, however I am not sure that the difference in python versions is whats stopping me.
I constantly get this hostname_override error...
[alebede#linuxbox]$ klist -a
Ticket cache: FILE:/tmp/krb5cc_37575
Default principal: Admin_alebede#mydomain.bla
Valid starting Expires Service principal
02/10/2017 11:30:32 02/10/2017 21:30:32 krbtgt/mydmain.bla#mydomain.bla
renew until 02/10/2017 21:30:32
Addresses: (none)
[alebede#linuxbox]$ python
Python 2.7.5 (default, Oct 11 2015, 17:47:16)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import winrm
>>> s = winrm.Session('WINDOWSBOX', auth=('admin_alebede#mydomain.bla', 'mypassword'), transport='kerberos')
>>> r = s.run_cmd('ipconfig', ['/all'])
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/winrm/__init__.py", line 37, in run_cmd
shell_id = self.protocol.open_shell()
File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 132, in open_shell
res = self.send_message(xmltodict.unparse(req))
File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 207, in send_message
return self.transport.send_message(message)
File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 173, in send_message
self.session = self.build_session()
File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 140, in build_session
sanitize_mutual_error_response=False)
TypeError: __init__() got an unexpected keyword argument 'hostname_override'
With ansible it looks like this, the same hostname_override error:
mywinserver.mydomain.com> WINRM CONNECT: transport=ssl endpoint=https://mywinserver.mydomain.com:5986/wsman
<mywinserver.mydomain.com> WINRM CONNECTION ERROR: the specified credentials were rejected by the server
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/ansible/plugins/connection/winrm.py", line 154, in _winrm_connect
self.shell_id = protocol.open_shell(codepage=65001) # UTF-8
File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 132, in open_shell
res = self.send_message(xmltodict.unparse(req))
File "/usr/lib/python2.7/site-packages/winrm/protocol.py", line 207, in send_message
return self.transport.send_message(message)
File "/usr/lib/python2.7/site-packages/winrm/transport.py", line 190, in send_message
raise InvalidCredentialsError("the specified credentials were rejected by the server")
InvalidCredentialsError: the specified credentials were rejected by the server
mywinserver.mydomain.com | UNREACHABLE! => {
"changed": false,
"msg": "kerberos: __init__() got an unexpected keyword argument 'hostname_override', ssl: the specified credentials were rejected by the server",
"unreachable": true
}
Let me know what else I can try, also on the Windows 2008R2 server i can see that the linux box is trying to connect in the security event log. Not sure what else is going on. Again, this works in RHEL6 on 2 different windows 2008R2 servers.
Likely you have a mismatch in versions between winrm and requests-kerberos. I had the same issue when I had winrm installed via pip and requests-kerberos install via apt.
Verify that you are using a single package manager for all installations.

import theano results in ImportError

I'm trying to use theano but I get an error when I import it.
I've installed cuda_6.5.14_linux_64.run, and passed all the recommended test in Chapter 6 of this NVIDIA PDF.
Ultimately I want to be able to install pylearn2, but I get the exact same error as below when I try to compile it.
EDIT1: My theanorc looks like:
[cuda]
root = /usr/local/cuda-6.5
[global]
device = gpu
floatX=float32
If I replace gpu with cpu, the command import theano succeeds.
Python 2.7.8 |Anaconda 1.9.0 (64-bit)| (default, Aug 21 2014, 18:22:21)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
Imported NumPy 1.9.1, SciPy 0.14.0, Matplotlib 1.3.1
Type "scientific" for more details.
>>> import theano
Using gpu device 0: GeForce GTX 750 Ti
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/g/anaconda/lib/python2.7/site-packages/theano/__init__.py", line 92, in <module>
theano.sandbox.cuda.tests.test_driver.test_nvidia_driver1()
File "/home/g/anaconda/lib/python2.7/site-packages/theano/sandbox/cuda/tests/test_driver.py", line 28, in test_nvidia_driver1
profile=False)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/compile/function.py", line 223, in function
profile=profile)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/compile/pfunc.py", line 512, in pfunc
on_unused_input=on_unused_input)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 1312, in orig_function
defaults)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/compile/function_module.py", line 1181, in create
_fn, _i, _o = self.linker.make_thunk(input_storage=input_storage_lists)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/gof/link.py", line 434, in make_thunk
output_storage=output_storage)[:3]
File "/home/g/anaconda/lib/python2.7/site-packages/theano/gof/vm.py", line 847, in make_all
no_recycling))
File "/home/g/anaconda/lib/python2.7/site-packages/theano/sandbox/cuda/__init__.py", line 237, in make_thunk
compute_map, no_recycling)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/gof/op.py", line 606, in make_thunk
output_storage=node_output_storage)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line 948, in make_thunk
keep_lock=keep_lock)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line 891, in __compile__
keep_lock=keep_lock)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line 1322, in cthunk_factory
key=key, fn=self.compile_cmodule_by_step, keep_lock=keep_lock)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", line 996, in module_from_key
module = next(compile_steps)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/gof/cc.py", line 1237, in compile_cmodule_by_step
preargs=preargs)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/sandbox/cuda/nvcc_compiler.py", line 444, in compile_str
return dlimport(lib_filename)
File "/home/g/anaconda/lib/python2.7/site-packages/theano/gof/cmodule.py", line 284, in dlimport
rval = __import__(module_name, {}, {}, [module_name])
ImportError: ('The following error happened while compiling the node', GpuCAReduce{add}{1}(<CudaNdarrayType(float32, vector)>), '\n', '/home/g/.theano/compiledir_Linux-3.11.0-26-generic-x86_64-with-debian-wheezy-sid-x86_64-2.7.8-64/tmpWYqQw5/7173b40d34b57da0645a57198c96dbcc.so: undefined symbol: __fatbinwrap_66_tmpxft_00004bf1_00000000_12_cuda_device_runtime_compute_50_cpp1_ii_5f6993ef', '[GpuCAReduce{add}{1}(<CudaNdarrayType(float32, vector)>)]')
I encountered exactly the same question.
My solution is to replace cuda-6.5 with cuda-5.5, and everything works fine.
We also saw this error. We found that putting /usr/local/cuda-6.5/bin in $PATH seemed to fix it (even with the root = ... line in .theanorc).