I am trying to establish a connection to Redshift using groovy and execute a query, I am not able to find any resources on how to do this. Please provide a solution.
I was trying to this in JAVA but couldn't find a similar solution in groovy
if you have code for java - it should work for groovy maybe with minor changes.
here is how jdbc connection in groovy could look like:
#GrabConfig(systemClassLoader=true)
#Grab(group='com.amazon.redshift', module='redshift-jdbc42', version='2.1.0.9')
import groovy.sql.Sql
def props = [
url: 'jdbc:redshift://adee-cluster.c9ld7lgff4o4.us-east-2.redshift.amazonaws.com:5555/dev',
user: 'awsuser',
password: 'changeme',
driver: 'com.amazon.redshift.jdbc42.Driver'
]
Sql.withInstance(props){sql->
println 'connected'
def query = "select id, first_name, last_name from public.employee where first_name like :firstName"
def params = [firstName:'Rob%']
sql.eachRow(query, params){row->
println "row: id=${row.id}, name=${row.first_name} ${row.last_name}"
}
}
Related
Concerning the following draft script I would like to know: How can I execute the named query I created?
I can access the query via the browser interface, but would like to execute it via the Session.
Here the answer is to use the client.start_query_execution(...) command. But whats the point when it is not the named query I created but instead a non-named query with the same query_string. Or am I missing something essential in how to use this?
import boto3
sess = boto3.session.Session(
region_name=region,
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY
)
athenaclient = sess.client('athena')
query_string = '''
SELECT *
FROM "ccindex"."ccindex"
WHERE crawl = 'CC-MAIN-2020-34'
AND subset = 'warc'
AND url_host_tld = 'de'
AND url_query IS NULL
AND url_path like '%impressum%'
LIMIT 20000
'''
resp = athenaclient.create_named_query(
Name='filter-ccindex-de',
Description='Filter *.de/impressum websites of Common Crawl index',
Database='ccindex',
QueryString=query_string
)
I don't think there is a direct option to pass named query to your start_query_execution method.But this can be achieved by using get_named_query which accepts Name of the named query and returns QueryString in response.
Then you can parse this response and pass QueryString to start_query_execution method.
My Django application uses elasticsearch to index several ressources.
Now I wanted to protect my elasticsearch instance with as password which is working fine if I use "curl -u" or so. Anyways from the elasticsearch_dsl documentation, found here: https://elasticsearch-dsl.readthedocs.io/en/latest/configuration.html, I do not understand what I have to do in order to setup elasticsearch that way that it uses a password for authentication and where do I have to place this code?! is smb. maybe able to pass show me some snippets of his configuration?
My current state looks like this:
settingy.py
ELASTICSEARCH_DSL = {
'default': {
'hosts': env.str('ELASTICSEARCH_HOST') + str(':') + env.str('ELASTICSEARCH_PORT'),
},
}
ELASTICSEARCH_DSL_SIGNAL_PROCESSOR = 'django_elasticsearch_dsl.signals.RealTimeSignalProcessor'
documents.py
from django_elasticsearch_dsl import Document, Index, fields
from elasticsearch_dsl import analyzer
from App.models import Post
# elasticsearch index
posts = Index('posts')
html_strip = analyzer(
'html_strip',
tokenizer="standard",
filter=["lowercase", "stop", "snowball"],
char_filter=["html_strip"]
)
#posts.document
class PostDocument(Document):
... more index stuff
According to the docs, I have to manually setup a default client connection where I can also pass the password and username for authentication which to me seems not to be possible at settings.py at moment.
Kind regards
You can pass the elasticsearch URL as
from urllib.parse import quote_plus as urlquote
elk_base_url = 'elasticsearch://{user_name}:{password}#{host_ip}:{host_port}'
elastic_search_url = elk_base_url.format(user_name='my_username',
password=urlquote('mysecret_password'),
# password may contain special characters
host_ip='my-elastic-host-ip',
host_port=9200)
ELASTICSEARCH_DSL = {
'default': {
'hosts': [elastic_search_url]
},
}
This solution has been tested under the following circumstances
Django==3.0.4
django-elasticsearch-dsl==7.1.1
logstash == kibana == elasticsearch == 7.6.0
If you are experiencing AuthenticationException(401, '') exception, it means you were provide the wrong credentials. Please do check the value of elastic_search_url and make sure the values are correct.
I am trying to implement the following:
import SoftLayer.API
username = 'set me!'
apiKey = 'set me too!'
serverId = 1234
client = SoftLayer.API.Client('SoftLayer_Hardware_Server', serverId, username, apiKey)
Here, I don't really know how to retrieve the serverId. How could I know my server ID for a specific server. Please help.
The SoftLayer_Account::getHardware retrieves information about your hardware objects, in which you can find the serverIds from your servers.
Try this python script:
"""
This script retrieves an account's associated hardware objects
Important manual pages:
http://sldn.softlayer.com/reference/services/SoftLayer_Account/getHardware
License: http://sldn.softlayer.com/article/License
Author: SoftLayer Technologies, Inc. <sldn#softlayer.com>
"""
# So we can talk to the SoftLayer API:
import SoftLayer.API
# For nice debug output:
from pprint import pprint as pp
# Your SoftLayer username and api key
API_USERNAME = 'set me'
API_KEY = 'set me'
# Creates a new connection to the API service.
client = SoftLayer.API.Client(username=API_USERNAME,api_key=API_KEY)
try:
hardwareObjects = client['SoftLayer_Account'].getHardware()
pp(hardwareObjects)
except SoftLayer.SoftLayerAPIError as e:
pp('Unable to get hardware objects faultCode=%s, faultString=%s'
% (e.faultCode, e.faultString))
This script will return the information from your servers, in which the "id" property refers to serverId from the server that you need.
However, if you wish to retrieve the information for an specific server, it can be done using Object Filters, here an example:
"""
This script retrieves a hardware information for an specific hardware object.
It is only necessary to specify the hostname from the server.
Important manual pages:
http://sldn.softlayer.com/reference/services/SoftLayer_Account/getHardware
http://sldn.softlayer.com/article/object-filters
License: http://sldn.softlayer.com/article/License
Author: SoftLayer Technologies, Inc. <sldn#softlayer.com>
"""
# So we can talk to the SoftLayer API:
import SoftLayer.API
# For nice debug output:
from pprint import pprint as pp
# Your SoftLayer username and api key
API_USERNAME = 'set me'
API_KEY = 'set me'
# Define the hostname from the hardware object
hostname = 'hostnametest'
# Declare an object filter to get an specific hardware object
filterHardware = {
'hardware': {
'hostname': {
'operation': hostname
}
}
}
# Creates a new connection to the API service.
client = SoftLayer.API.Client(username=API_USERNAME,api_key=API_KEY)
try:
hardwareObjects = client['SoftLayer_Account'].getHardware(filter=filterHardware)
pp(hardwareObjects)
except SoftLayer.SoftLayerAPIError as e:
pp('Unable to get the hardware object faultCode=%s, faultString=%s'
% (e.faultCode, e.faultString))
You need to specify the "hostname" from your server. The "id" in the response refers to serverId.
Some references:
Using Initialization Parameters in the SoftLayer
API
SoftLayer Python
Going Further with the SoftLayer API Python Client - Part 1
I have two servers running django. I'll call one server my "logging" server and another my "client" server. The client server wants to log a message with the logging server by passing over a username, password, and message over the internet. With my current implementation I'm hitting an error when trying to decrypt the encrypted message, username, and password that was sent over the wire. It looks like I should be trying to decrypt a "byte string" according to the pycrypto documentation, but I can't seem to create a byte string correctly since I haven't been able to get around this problem. Also, it feels like my implementation is taking me down a rabbit hole of security vulnerabilities and codec confusion. Is there a package that I should look at which already implements this type of functionality? If so what would that implementation look like?
client:
from Crypto.Hash import MD5
from Crypto.PublicKey import RSA
from base64 import b64decode
import urllib2
import urllib
#I realize recreating the hash everytime is slow. I just included it here for simplicity.
logger_public_signature_message = "I am a client :)"
logger_public_signature_hash = MD5.new(logger_public_signature_message).digest()
client_private_key = #private key
logger_public_key = #public key
client_private = RSA.importKey(client_private_key)
client_public = client_private.publickey()
logger_public = RSA.importKey(logger_public_key)
message = "my message"
username = "user"
password = "password"
encrypted_message = logger_public.encrypt(message, "ignored_param")
encrypted_username = logger_public.encrypt(username, "ignored_param")
encrypted_password = logger_public.encrypt(password, "ignored_param")
signature = client_private.sign(logger_public_signature_hash, '')
params = { "message": encrypted_message, "username": encrypted_username, "password": encrypted_password, "signature": signature }
url_encoded_params = urllib.urlencode(params)
url = 'http://localhost:8000/url/to/logger/'
req = urllib2.Request(url, url_encoded_params)
logger:
from Crypto.Hash import MD5
from Crypto.PublicKey import RSA
from base64 import b64decode
def log(request):
#I realize recreating the hash everytime is slow. I just included it here for simplicity.
logger_public_signature_message = "I am a client :)"
logger_public_signature_hash = MD5.new(logger_public_signature_message).digest()
client_public_key = #client public key
logger_private_key = #logger private key
client_public = RSA.importKey(client_public_key)
logger_private = RSA.importKey(logger_private_key)
p = request.POST
encrypted_message = urllib2.unquote(p["message"])
encrypted_username = urllib2.unquote(p["username"])
encrypted_password = urllib2.unquote(p["password"])
signature = urllib2.unquote(p["signature"])
#I'm hitting exceptions when trying to decrypt the encrypted messages.
#The exceptions are: "ValueError: Message too large" I think the issue is that
#I'm trying to decrypt a base64 string where I should be trying to decrypt a byte
#string from reading the documentation. But I haven't been able I guess to correctly
#create a byte string because I can't get it to work.
decrypted_message = logger_private.decrypt(encrypted_message.encode("base64"))
decrypted_username = logger_private.decrypt(encrypted_username.encode("base64"))
decrypted_password = logger_private.decrypt(encrypted_password.encode("base64"))
verified = client_public.verify(logger_public_signature_hash, signature)
I think you are putting a lot of effort into stuff, that doesn't need to be handled by Django.
Here is what I would usually do:
Use HTTPS, as transport encryption layer
Use HTTP Basic Auth. Basic auth is implemented in urllib2 as well as requests.
But there is an even better solution: Django REST framework
It provides you will a full blown REST API including different authentication solutions.
If you need any help, setting up one of these options, let me know and I'll add an example.
May it be that you would use sentry for logging? Of course if it isn't task for training.
I look at the sentry since it been django application, and now it surely is excellent production-ready solution.
We're using it in banking-sphere software development.
You are very close to decrypting the values on the server. The result of the encryption on the client is a tuple. When you urllib2.unquote the items on the server, you then recreate tuples from them.
For example:
>>> c = public.encrypt('Hello', "ignored")
>>> c
('3\xae0\x1f\xd7\xe4b\xd4\xf1\xf4\x88!Be\xff!\x1e\xda\x82\x10\x9bRy\x0c\xa0v\xed\x84\xf9\xe35\xc6QG\xcf\xb7\x1b\xea\x9fe\t\x9b\x8d\xd6\xf3\x8cw\xde\x17\xb5\xf7\x9a+\x84i%#\x8a\xdf\xf4\xdd\xc8wY',)
which in your code you pack into params like this:
>>> params = { "message" : c }
>>> params
{'message': ('3\xae0\x1f\xd7\xe4b\xd4\xf1\xf4\x88!Be\xff!\x1e\xda\x82\x10\x9bRy\x0c\xa0v\xed\x84\xf9\xe35\xc6QG\xcf\xb7\x1b\xea\x9fe\t\x9b\x8d\xd6\xf3\x8cw\xde\x17\xb5\xf7\x9a+\x84i%#\x8a\xdf\xf4\xdd\xc8wY',)}
>>> urllib.urlencode(params)
'message=%28%273%5Cxae0%5Cx1f%5Cxd7%5Cxe4b%5Cxd4%5Cxf1%5Cxf4%5Cx88%21Be%5Cxff%21%5Cx1e%5Cxda%5Cx82%5Cx10%5Cx9bRy%5Cx0c%5Cxa0v%5Cxed%5Cx84%5Cxf9%5Cxe35%5Cxc6QG%5Cxcf%5Cxb7%5Cx1b%5Cxea%5Cx9fe%5Ct%5Cx9b%5Cx8d%5Cxd6%5Cxf3%5Cx8cw%5Cxde%5Cx17%5Cxb5%5Cxf7%5Cx9a%2B%5Cx84i%25%40%5Cx8a%5Cxdf%5Cxf4%5Cxdd%5Cxc8wY%27%2C%29'
I would guess that urllib2.unquote(p["message"]) returns this (but I did not try this):
"('3\\xae0\\x1f\\xd7\\xe4b\\xd4\\xf1\\xf4\\x88!Be\\xff!\\x1e\\xda\\x82\\x10\\x9bRy\\x0c\\xa0v\\xed\\x84\\xf9\\xe35\\xc6QG\\xcf\\xb7\\x1b\\xea\\x9fe\\t\\x9b\\x8d\\xd6\\xf3\\x8cw\\xde\\x17\\xb5\\xf7\\x9a+\\x84i%#\\x8a\\xdf\\xf4\\xdd\\xc8wY',)"
then you can recreate the tuple at the server like this (m is the unquoted message):
>>> from ast import literal_eval
>>> literal_eval(m)
('3\xae0\x1f\xd7\xe4b\xd4\xf1\xf4\x88!Be\xff!\x1e\xda\x82\x10\x9bRy\x0c\xa0v\xed\x84\xf9\xe35\xc6QG\xcf\xb7\x1b\xea\x9fe\t\x9b\x8d\xd6\xf3\x8cw\xde\x17\xb5\xf7\x9a+\x84i%#\x8a\xdf\xf4\xdd\xc8wY',)
once you have the tuple back, you can decrypt it:
>>> private.decrypt(literal_eval(m))
'Hello'
It would be better to find a vetted and standard mechanism to do this rather than roll your own. For example, in your scheme, I could capture different messages between the client and server, and then mix and match messages and username/password pairs, making it appear that the messages came from different users.
However, with just this minor tweak (recreating the tuples from the "unquoted" strings) your code should decrypt just fine.
I want to get all the datastores from a specific cluster. I will be using this for my vapp deployment.
i tried the following snippet:
clusters = server.get_clusters()
for c_mor, c_name in clusters.items():
for ds_mor, name in server.get_datastores(from_mor=c_mor).items():
props = VIProperty(server, ds_mor)
print props.name
However, this does not give me the list. What am I doing wrong?
It's to late for answer this question. this my code.
from pysphere import VIServer, VIProperty
server = VIServer()
server.connect(server, username, password)
for ds_mor, name in server.get_hosts().items():
props = VIProperty(server, ds_mor)
for item in props.datastore :
print item.info.name
server.disconnect()
Old question but if anyone outside is looking for a solution (this is the first result in google), this will directly gather information from datastores:
from pysphere import VIServer, VIProperty
import ssl
default_context = ssl._create_default_https_context
server = VIServer()
#user = the-user-to-connect-to-vsphere
#password = the-password-from-the-user
#host = the-vsphere-hostname-or-ip
try:
ssl._create_default_https_context = ssl._create_unverified_context
server.connect(host, user, password)
for ds_mor, name in server.get_datastores().items():
props = VIProperty(server, ds_mor)
print "Datastore name: ", name
server.disconnect()
finally:
ssl._create_default_https_context = default_context
This will connect to self-signed VSphere, as is usual de case in an intranet.