i have setup superset with oauth in keycloak sso
in web is is woking ok using cookies
but for superset api we need to use a token
superset is a flask based app
using configs we setup:
JWT_ALGORITHM = 'RS256'
JWT_PUBLIC_KEY = """
-----BEGIN PUBLIC KEY-----
public_key_was_here
-----END PUBLIC KEY-----
"""
get token using command :
export TOKEN=`curl --request POST "$SSO_URL/auth/realms/$SSO_REALM/protocol/openid-connect/token?redirect_uri=$SUPERSET_URL/oauth-authorized/ozon_sso" \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode "username=$USERNAME" \
--data-urlencode "password=$PASSWORD" \
--data-urlencode 'scope=email profile roles' \
--data-urlencode 'grant_type=client_credentials' \
--data-urlencode 'client_id=k8s.superset' \
--data-urlencode "client_secret=$SSO_CLIENT_SECRET" | jq -r .access_token`
and then try to use it on superset:
curl -L --request GET $SUPERSET_URL/api/v1/dashboard/ --header "Authorization: Bearer $TOKEN" > ./dashboard_list.txt
and get a error in applications
seems like it unable to decode a token or key is incorrect
2022-12-15 07:24:00,487:ERROR:superset.views.base:invalid literal for int() with base 10: '674792eb-e01e-45b6-ad2e-75979db3d601'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1516, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1502, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/security/decorators.py", line 94, in wraps
verify_jwt_in_request()
File "/usr/local/lib/python3.8/site-packages/flask_jwt_extended/view_decorators.py", line 83, in verify_jwt_in_request
_request_ctx_stack.top.jwt_user = _load_user(jwt_header, jwt_data)
File "/usr/local/lib/python3.8/site-packages/flask_jwt_extended/view_decorators.py", line 141, in _load_user
user = user_lookup(jwt_header, jwt_data)
File "/usr/local/lib/python3.8/site-packages/flask_jwt_extended/internal_utils.py", line 25, in user_lookup
return jwt_manager._user_lookup_callback(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/security/manager.py", line 2042, in load_user_jwt
user = self.load_user(identity)
File "/usr/local/lib/python3.8/site-packages/flask_appbuilder/security/manager.py", line 2038, in load_user
return self.get_user_by_id(int(pk))
ValueError: invalid literal for int() with base 10: '674792eb-e01e-45b6-ad2e-75979db3d601'
10.220.74.237 - - [15/Dec/2022:07:24:00 +0000] "GET /api/v1/dashboard/ HTTP/1.1" 500 2337 "-" "curl/7.29.0"
and i don't know where to dig
Related
I'm trying to run the google cloud vision quickstart guide but when calling response = client.label_detection(image=image) I'm getting the following error:
ERROR:grpc._plugin_wrapping:AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x7fc1f3f66350>" raised exception!
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/credentials.py", line 111, in refresh
self._retrieve_info(request)
File "/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/credentials.py", line 88, in _retrieve_info
request, service_account=self._service_account_email
File "/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/_metadata.py", line 234, in get_service_account_info
return get(request, path, params={"recursive": "true"})
File "/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/_metadata.py", line 187, in get
response,
google.auth.exceptions.TransportError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7fc1fb046910>)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/grpc/_plugin_wrapping.py", line 90, in __call__
context, _AuthMetadataPluginCallback(callback_state, callback))
File "/usr/local/lib/python3.7/dist-packages/google/auth/transport/grpc.py", line 101, in __call__
callback(self._get_authorization_headers(context), None)
File "/usr/local/lib/python3.7/dist-packages/google/auth/transport/grpc.py", line 88, in _get_authorization_headers
self._request, context.method_name, context.service_url, headers
File "/usr/local/lib/python3.7/dist-packages/google/auth/credentials.py", line 133, in before_request
self.refresh(request)
File "/usr/local/lib/python3.7/dist-packages/google/auth/compute_engine/credentials.py", line 117, in refresh
six.raise_from(new_exc, caught_exc)
File "<string>", line 3, in raise_from
google.auth.exceptions.RefreshError: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7fc1fb046910>)
---------------------------------------------------------------------------
_InactiveRpcError Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)
71 try:
---> 72 return callable_(*args, **kwargs)
73 except grpc.RpcError as exc:
7 frames
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "Getting metadata from plugin failed with error: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7fc1fb046910>)"
debug_error_string = "UNKNOWN:Error received from peer vision.googleapis.com:443 {grpc_message:"Getting metadata from plugin failed with error: (\"Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\\nb\'\'\", <google.auth.transport.requests._Response object at 0x7fc1fb046910>)", grpc_status:14, created_time:"2022-09-13T14:48:30.187999422+00:00"}"
>
The above exception was the direct cause of the following exception:
ServiceUnavailable Traceback (most recent call last)
/usr/local/lib/python3.7/dist-packages/google/api_core/grpc_helpers.py in error_remapped_callable(*args, **kwargs)
72 return callable_(*args, **kwargs)
73 except grpc.RpcError as exc:
---> 74 raise exceptions.from_grpc_error(exc) from exc
75
76 return error_remapped_callable
ServiceUnavailable: 503 Getting metadata from plugin failed with error: ("Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Enginemetadata service. Status: 404 Response:\nb''", <google.auth.transport.requests._Response object at 0x7fc1fb046910>)
I'm executing the example code in a google colab, here's my code:
#connect to google drive
from google.colab import drive
drive.mount('/content/drive')
#download and extract google cloud client
!curl -O https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-cli-401.0.0-linux-x86_64.tar.gz
!tar -xf google-cloud-cli-401.0.0-linux-x86_64.tar.gz
#set credentials environment variable
!export GOOGLE_APPLICATION_CREDENTIALS=/content/drive/MyDrive/imagelabeling-1663076432940-26cfebd304cf.json
#install and init google cloud
!./google-cloud-sdk/install.sh --usage-reporting False --quiet
!./google-cloud-sdk/bin/gcloud init
#install google-cloud-vision libraries
!pip install --upgrade google-cloud-vision
#download cat image
!wget https://raw.githubusercontent.com/googleapis/python-vision/master/samples/snippets/quickstart/resources/wakeupcat.jpg
#run labeling code
import io
import os
# Imports the Google Cloud client library
from google.cloud import vision
# Instantiates a client
client = vision.ImageAnnotatorClient()
# The name of the image file to annotate
file_name = os.path.abspath('/content/wakeupcat.jpg')
# Loads the image into memory
with io.open(file_name, 'rb') as image_file:
content = image_file.read()
image = vision.Image(content=content)
# Performs label detection on the image file
response = client.label_detection(image=image)
labels = response.label_annotations
print('Labels:')
for label in labels:
print(label.description)
during !./google-cloud-sdk/bin/gcloud init I select Re-initialize this configuration [default] with new settings and then login with my google username by clicking on the weblink and copy-pasting the auth code. After that I'm selecting the testproject I've setup in the console.
I have confirmed that billing is enabled for this testproject. I have also enabled the vision API for this project and I have created a service account for this project with 'Owner' role and have created a json key file, which is located on the mounted google drive path `/content/drive/MyDrive/imagelabeling-1663076432940-26cfebd304cf.json``
Did I miss something?
I m trying to execute the folowing python code:
import logging
import sys
import docker, boto3
from base64 import b64decode
logging.basicConfig(stream=sys.stdout, level=logging.INFO)
LOCAL_REPOSITORY = '111111111111.dkr.ecr.us-east-1.amazonaws.com/my_repo:latest'
image = '111111111111.dkr.ecr.us-east-1.amazonaws.com/my_repo'
ecr_registry, _ = image.split('/', 1)
client = docker.from_env()
# Get login credentials from AWS for the ECR registry.
ecr = boto3.client('ecr')
response = ecr.get_authorization_token()
token = b64decode(response['authorizationData'][0]['authorizationToken'])
username, password = token.decode('utf-8').split(':', 1)
# Log in to the ECR registry with Docker.
client.login(username, password, registry=ecr_registry)
logging.info("loggined")
client.images.pull(image, auth_config={
username: username,
password: password
})
And got exception:
C:\myPath>python app/pull_example.py
INFO:botocore.credentials:Found credentials in environment variables.
INFO:root:loggined
Traceback (most recent call last):
File "C:\Python3\lib\site-packages\docker\api\client.py", line 261, in _raise_for_status
response.raise_for_status()
File "C:\Python3\lib\site-packages\requests\models.py", line 940, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+docker://localnpipe/v1.35/images/create?fromImage=111111111111.dkr.ecr.us-east-1.amazonaws.com%2Fmy_repo
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "app/pull_example.py", line 41, in <module>
password: password
File "C:\Python3\lib\site-packages\docker\models\images.py", line 445, in pull
repository, tag=tag, stream=True, **kwargs
File "C:\Python3\lib\site-packages\docker\api\image.py", line 415, in pull
self._raise_for_status(response)
File "C:\Python3\lib\site-packages\docker\api\client.py", line 263, in _raise_for_status
raise create_api_error_from_http_exception(e)
File "C:\Python3\lib\site-packages\docker\errors.py", line 31, in create_api_error_from_http_exception
raise cls(e, response=response, explanation=explanation)
docker.errors.APIError: 500 Server Error: Internal Server Error ("Get https://111111111111.dkr.ecr.us-east-1.amazonaws.com/v2/my_repo/tags/list: no basic auth credentials")
What is the problem? Why I can not pull image even after client.login call which happens wihtout any exeptions. What is the correct way to perform login and pull image from ECR repository and dockerpy?
This was happen due to - https://github.com/docker/docker-py/issues/2157
Deleting ~/.docker/config.json fixed the issue.
I recently updated my gcloud libraries from 118.0.0 to 132.0.0 and immediately remote_api_shell no longer worked. I went through a number of permutations of re-logging in, to set the application-default credentials through gcloud, and to use a service account and environment variable. All permutations failed with the same error message:
Traceback (most recent call last):
File "/Users/mbostwick/google-cloud-sdk/bin/remote_api_shell.py", line 133, in <module>
run_file(__file__, globals())
File "/Users/mbostwick/google-cloud-sdk/bin/remote_api_shell.py", line 129, in run_file
execfile(_PATHS.script_file(script_name), globals_)
File "/Users/mbostwick/google-cloud-sdk/platform/google_appengine/google/appengine/tools/remote_api_shell.py", line 160, in <module>
main(sys.argv)
File "/Users/mbostwick/google-cloud-sdk/platform/google_appengine/google/appengine/tools/remote_api_shell.py", line 156, in main
oauth2=True)
File "/Users/mbostwick/google-cloud-sdk/platform/google_appengine/google/appengine/tools/remote_api_shell.py", line 74, in remote_api_shell
secure=secure, app_id=appid)
File "/Users/mbostwick/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 769, in ConfigureRemoteApiForOAuth
rpc_server_factory=rpc_server_factory)
File "/Users/mbostwick/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 839, in ConfigureRemoteApi
app_id = GetRemoteAppIdFromServer(server, path, rtok)
File "/Users/mbostwick/google-cloud-sdk/platform/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 569, in GetRemoteAppIdFromServer
response = server.Send(path, payload=None, **urlargs)
File "/Users/mbostwick/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc_httplib2.py", line 259, in Send
NeedAuth()
File "/Users/mbostwick/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc_httplib2.py", line 235, in NeedAuth
RaiseHttpError(url, response_info, response, 'Too many auth attempts.')
File "/Users/mbostwick/google-cloud-sdk/platform/google_appengine/google/appengine/tools/appengine_rpc_httplib2.py", line 85, in RaiseHttpError
raise urllib2.HTTPError(url, response_info.status, msg, response_info, stream)
urllib2.HTTPError: HTTP Error 401: Unauthorized Too many auth attempts.
After back revving through 131.0.0 and 130.0.0, I just went back to 118.0.0, re-logged in and everything worked fine.
I did not update the running application after updating gcloud, as I'm in the middle of a release cycle at the moment, so that may have been the issue, but any help would be appreciated. Thanks!
TL;DR: This was fixed in gcloud version 134
Original answer: Run
gcloud auth application-default login --scopes=https://www.googleapis.com/auth/cloud-platform,https://www.googleapis.com/auth/userinfo.email
Now your remote shell should work again.
Details:
I think this was broken by the 128.0.0 update, along with the changes to the gcloud auth login command. The old tokens have the following scopes (according to Google's tokeninfo endpoint):
https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/appengine.admin https://www.googleapis.com/auth/compute https://www.googleapis.com/auth/plus.me
The new tokens from gcloud auth application-default login without any options only have:
https://www.googleapis.com/auth/cloud-platform
This is documented in gcloud auth application-default login --help
Version 134 details: The scopes requested are now:
https://www.googleapis.com/auth/userinfo.email https://www.googleapis.com/auth/cloud-platform
See discussion at https://groups.google.com/d/msg/google-appengine/ptc-76K6Kk4/9qr4601BBgAJ
I would like to ask if it is currently possible to use spark-ec2 script https://spark.apache.org/docs/latest/ec2-scripts.html together with credentials that are consisting not only from: aws_access_key_id and aws_secret_access_key, but it also contains aws_security_token.
When I try to run the script I am getting following error message:
ERROR:boto:Caught exception reading instance data
Traceback (most recent call last):
File "/Users/zikes/opensource/spark/ec2/lib/boto-2.34.0/boto/utils.py", line 210, in retry_url
r = opener.open(req, timeout=timeout)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
response = self._open(req, data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
'_open', req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
result = func(*args)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1214, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1184, in do_open
raise URLError(err)
URLError: <urlopen error [Errno 64] Host is down>
ERROR:boto:Unable to read instance data, giving up
No handler was ready to authenticate. 1 handlers were checked. ['QuerySignatureV2AuthHandler'] Check your credentials
Does anyone has some idea what can be possibly wrong? Is aws_security_token the problem?
It maybe seems to me more as boto than Spark problem.
I have tried both:
1) setting credentials in ~/.aws/credentials and ~/.aws/config
2) setting credential by commands:
export aws_access_key_id=<my_aws_access_key>
export aws_secret_access_key=<my_aws_seecret_key>
export aws_security_token=<my_aws_security_token>
My launch command is:
./spark-ec2 -k my_key -i my_key.pem --additional-tags "mytag:tag1,mytag2:tag2" --instance-profile-name "profile1" -s 1 launch test
you can setup your credentials & config using the command aws configure.
I had the same issue but in my case my AWS_SECRET_ACCESS_KEY had a slash, I regenerated the key until there was no slash and it worked
The problem was that I did not use profile called default after renaming everything worked well.
Here's the error:
Traceback (most recent call last):
File "/var/apps/tango/envs/tango-env/local/lib/python2.7/site-packages/django/core/handlers/wsgi.py", line 187, in __call__
self.load_middleware()
File "/var/apps/tango/envs/tango-env/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 45, in load_middleware
mw_class = import_by_path(middleware_path)
File "/var/apps/tango/envs/tango-env/local/lib/python2.7/site-packages/django/utils/module_loading.py", line 21, in import_by_path
module = import_module(module_path)
File "/var/apps/tango/envs/tango-env/local/lib/python2.7/site-packages/django/utils/importlib.py", line 40, in import_module
__import__(name)
File "/var/apps/tango/envs/tango-env/local/lib/python2.7/site-packages/django/middleware/doc.py", line 4, in <module>
warnings.warn(__doc__, PendingDeprecationWarning, stacklevel=2)
TypeError: expected string or buffer
TypeError: expected string or buffer
Tango is the user I created specifically for this project.
I'm using upstart so in /etc/init/tango-wsgi.conf looks like:
exec /var/apps/tango/envs/tango-env/bin/uwsgi \
--uid tango \
--home /var/apps/tango/envs/tango-env \
--pythonpath /var/apps/tango/tango/src \
--wsgi-file /var/apps/tango/tango/src/tango_cms/wsgi.py \
--socket /tmp/tango-uwsgi.sock \
--chmod-socket \
--logdate \
--optimize 2 \
--processes 2 \
--master \
--logto /var/apps/tango/logs/uwsgi.log
UPDATE:
My nginx.conf has following entry:
location / {
uwsgi_pass unix:/tmp/tango-uwsgi.sock;
include /etc/nginx/uwsgi_params;
}
Please help.
Same problem here. I fixed it deleting the following line on tango-wsgi.conf file:
--optimize 2 \
Optimize allows some kind of python optimization: More info
Not sure why it's an error, you could try removing "XViewMiddleware" if it's installed.