I'm attempting to connect to my rds database inside of AWS via my code using pymysql using the following code (note that my endpoint and password variables will be replaced for the sake of my own privacy):
import pymysql
ENDPOINT = "*******"
PORT = "3306"
USER = "admin"
PASSWORD = "*******"
DBNAME = "appdb2"
print("INITIALIZING DATABASE")
conn = pymysql.connect(host=ENDPOINT, user=USER, password = PASSWORD, database = DBNAME)
cur = conn.cursor()
#Creates curser to interact with database
try:
cur.execute("DROP TABLE userdetails;")
print("table deleted")
#If no database, create table with inserts and commit and execute
except Exception as e:
print("cannot delete table")
cur.execute("CREATE TABLE userdetails(email VARCHAR(20), password VARCHAR(20), description VARCHAR(50), imagelocation VARCHAR(50));")
print("table created")
cur.execute("INSERT INTO userdetails(email, password, description, imagelocation) VALUES('alex.hamilton1996#gmail.com', 'password', 'this is a desc', 'Default.png');")
print("Insert Success")
cur.execute("INSERT INTO userdetails(email, password, description, imagelocation) VALUES('alex.hamilton19961#gmail.com', 'password', 'this is a desc', 'Default.png');")
print("Insert Success")
cur.execute("INSERT INTO userdetails(email, password, description, imagelocation) VALUES('alex.hamilton19962#gmail.com', 'password', 'this is a desc', 'Default.png');")
print("Insert Success")
cur.execute("INSERT INTO userdetails(email, password, description, imagelocation) VALUES('alex.hamilton19963#gmail.com', 'password', 'this is a desc', 'Default.png');")
print("Insert Success")
cur.execute("SELECT * FROM userdetails;")
I am left with a long error message as follows whenever I run this:
$ C:/Users/alexh/AppData/Local/Programs/Python/Python39/python.exe "c:/Users/alexh/Dropbox/PC/Desktop/Cloud Computing/testcode.py"
INITIALIZING DATABASE
Traceback (most recent call last):
File "C:\Users\alexh\AppData\Local\Programs\Python\Python39\lib\site-packages\pymysql\connections.py", line 613, in connect
sock = socket.create_connection(
File "C:\Users\alexh\AppData\Local\Programs\Python\Python39\lib\socket.py", line 843, in create_connection
raise err
File "C:\Users\alexh\AppData\Local\Programs\Python\Python39\lib\socket.py", line 831, in create_connection
sock.connect(sa)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "c:\Users\alexh\Dropbox\PC\Desktop\Cloud Computing\testcode.py", line 10, in <module>
conn = pymysql.connect(host=ENDPOINT, user=USER, password = PASSWORD, database = DBNAME)
File "C:\Users\alexh\AppData\Local\Programs\Python\Python39\lib\site-packages\pymysql\connections.py", line 353, in __init__
self.connect()
File "C:\Users\alexh\AppData\Local\Programs\Python\Python39\lib\site-packages\pymysql\connections.py", line 664, in connect
raise exc
pymysql.err.OperationalError: (2003, "Can't connect to MySQL server on 'appdb2.c7oob6smafaa.us-east-2.rds.amazonaws.com' (timed out)")
Whenever I go to the my RDS logs in AWS, I get the following error messages:
2022-08-03T16:13:01.357351Z 0 [Warning] [MY-011068] [Server] The syntax 'log_slave_updates' is deprecated and will be removed in a future release. Please use log_replica_updates instead.
2022-08-03T16:13:01.357380Z 0 [Warning] [MY-011069] [Server] The syntax '--master-info-repository' is deprecated and will be removed in a future release.
2022-08-03T16:13:01.357393Z 0 [Warning] [MY-011069] [Server] The syntax '--master-info-repository' is deprecated and will be removed in a future release.
2022-08-03T16:13:01.357455Z 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-file' is deprecated and will be removed in a future release.
2022-08-03T16:13:01.357462Z 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-repository' is deprecated and will be removed in a future release.
2022-08-03T16:13:01.357491Z 0 [Warning] [MY-011068] [Server] The syntax 'skip_slave_start' is deprecated and will be removed in a future release. Please use skip_replica_start instead.
2022-08-03T16:13:01.357498Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_exec_mode' is deprecated and will be removed in a future release. Please use replica_exec_mode instead.
2022-08-03T16:13:01.357506Z 0 [Warning] [MY-011068] [Server] The syntax 'slave_load_tmpdir' is deprecated and will be removed in a future release. Please use replica_load_tmpdir instead.
2022-08-03T16:13:01.358795Z 0 [Warning] [MY-010101] [Server] Insecure configuration for --secure-file-priv: Location is accessible to all OS users. Consider choosing a different directory.
2022-08-03T16:13:01.358834Z 0 [Warning] [MY-010918] [Server] 'default_authentication_plugin' is deprecated and will be removed in a future release. Please use authentication_policy instead.
2022-08-03T16:13:01.359306Z 0 [System] [MY-010116] [Server] /rdsdbbin/mysql/bin/mysqld (mysqld 8.0.28) starting as process 722
2022-08-03T16:13:01.445821Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --log-replica-updates work.
2022-08-03T16:13:01.513243Z 0 [Warning] [MY-010075] [Server] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: 25bc428b-1347-11ed-9128-02b016500430.
2022-08-03T16:13:01.553323Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2022-08-03T16:13:01.581857Z 1 [Warning] [MY-012191] [InnoDB] Scan path '/rdsdbdata/db/innodb' is ignored because it is a sub-directory of '/rdsdbdata/db/
2022-08-03T16:13:02.047699Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2022-08-03T16:13:02.330046Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --binlog-expire-logs-seconds work.
2022-08-03T16:13:02.337052Z 0 [Warning] [MY-013414] [Server] Server SSL certificate doesn't verify: unable to get issuer certificate
2022-08-03T16:13:02.337158Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2022-08-03T16:13:02.394864Z 0 [System] [MY-010931] [Server] /rdsdbbin/mysql/bin/mysqld: ready for connections. Version: '8.0.28' socket: '/tmp/mysql.sock' port: 3306 Source distribution.
Does anybody have any advice as to how I can fix this issue given the feedback that these logs gave me? Thank you.
Related
I tried the camunda community python client, from the repo (https://github.com/camunda-community-hub/camunda-8-code-studio/tree/main/src/PythonCloudWorker). I have set up caumnda 8 saas account to run my tasks from the repo.
I 'm getting error when i try to run the python file, posting the error. Any suggestions appriciated.
communda_connect.py:59: DeprecationWarning: There is no current event loop
loop = asyncio.get_event_loop()
E0118 00:29:19.302897000 6259650560 hpack_parser.cc:1218] Error parsing metadata: error=invalid value key=content-type value=text/plain; charset=utf-8
E0118 00:29:19.307140000 6259650560 hpack_parser.cc:1218] Error parsing metadata: error=invalid value key=content-type value=text/plain; charset=utf-8
E0118 00:29:19.310754000 6259650560 hpack_parser.cc:1218] Error parsing metadata: error=invalid value key=content-type value=text/plain; charset=utf-8
Traceback (most recent call last):
env/lib/python3.10/site-packages/grpc/aio/_call.py", line 236, in _raise_for_status
raise _create_rpc_error(await self.initial_metadata(), await
grpc.aio._call.AioRpcError: <AioRpcError of RPC that terminated with:
status = StatusCode.UNIMPLEMENTED
details = "Received http2 header with status: 404"
debug_error_string = "UNKNOWN:Error received from peer ipv4:32.12.17.224:443 {created_time:"2023-01-18T00:29:19.304994+05:30", grpc_status:12, grpc_message:"Received http2 header with status: 404"}"
>
During handling of the above exception, another exception occurred:
env/lib/python3.10/site-packages/pyzeebe/grpc_internals/zeebe_adapter_base.py", line 33, in _handle_grpc_error
raise pyzeebe_error
pyzeebe.errors.zeebe_errors.UnkownGrpcStatusCodeError
problem was i had not passed the region parameter which was defaulting to bru-2.
camunda_region = os.environ.get('CAMUNDA_CLUSTER_REGION')
channel = create_camunda_cloud_channel(client_id=zeebe_client_id, client_secret=zeebe_client_secret, cluster_id=camundacloud_cluster_id,region=camunda_region)
So i when to the airflow documentation for aws redshift there is 2 operator that can execute the sql query they are RedshiftSQLOperator and RedshiftDataOperator. I already implemented my job using RedshiftSQLOperator but i want to do it using RedshiftDataOperator instead, because i dont want to using postgres connection in RedshiftSQLOperator but AWS API.
RedshiftDataOperator Documentation
I had read this documentation there is aws_conn_id in the parameter. But when im trying to use the same connection id there is error.
[2023-01-11, 04:55:56 UTC] {base.py:68} INFO - Using connection ID 'redshift_default' for task execution.
[2023-01-11, 04:55:56 UTC] {base_aws.py:206} INFO - Credentials retrieved from login
[2023-01-11, 04:55:56 UTC] {taskinstance.py:1889} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/redshift_data.py", line 146, in execute
self.statement_id = self.execute_query()
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/redshift_data.py", line 124, in execute_query
resp = self.hook.conn.execute_statement(**filter_values)
File "/home/airflow/.local/lib/python3.7/site-packages/botocore/client.py", line 415, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/botocore/client.py", line 745, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (UnrecognizedClientException) when calling the ExecuteStatement operation: The security token included in the request is invalid.
From task id
redshift_data_task = RedshiftDataOperator(
task_id='redshift_data_task',
database='rds',
region='ap-southeast-1',
aws_conn_id='redshift_default',
sql="""
call some_procedure();
"""
)
What should i fill in the airflow connection ? Because in the documentation there is no example of value that i should fill to airflow. Thanks
Airflow RedshiftDataOperator Connection Required Value
Have you tried using the Amazon Redshift connection? There is both an option for authenticating using your Redshift credentials:
Connection ID: redshift_default
Connection Type: Amazon Redshift
Host: <your-redshift-endpoint> (for example, redshift-cluster-1.123456789.us-west-1.redshift.amazonaws.com)
Schema: <your-redshift-database> (for example, dev, test, prod, etc.)
Login: <your-redshift-username> (for example, awsuser)
Password: <your-redshift-password>
Port: <your-redshift-port> (for example, 5439)
(source)
and an option for using an IAM role (there is an example in the first link).
Disclaimer: I work at Astronomer :)
EDIT: Tested the following with Airflow 2.5.0 and Amazon provider 6.2.0:
Added the IP of my Airflow instance to the VPC security group with "All traffic" access.
Airflow Connection with the connection id aws_default, Connection type "Amazon Web Services", extra: { "aws_access_key_id": "<your-access-key-id>", "aws_secret_access_key": "<your-secret-access-key>", "region_name": "<your-region-name>" }. All other fields blank. I used a root key for my toy-aws. If you use other credentials you need to make sure that IAM role has access and the right permissions to the Redshift cluster (there is a list in the link above).
Operator code:
red = RedshiftDataOperator(
task_id="red",
database="dev",
sql="SELECT * FROM dev.public.users LIMIT 5;",
cluster_identifier="redshift-cluster-1",
db_user="awsuser",
aws_conn_id="aws_default"
)
I am working with AWS MWAA (Apache Airflow). I want to send an email in MWAA upon completion of my pipeline.I have set the following configuration
Now when I run my dag using an Email Operator, it gives me an error.
File "/usr/lib64/python3.7/socket.py", line 707, in create_connection
for res in getaddrinfo(host, port, 0, SOCK_STREAM):
File "/usr/lib64/python3.7/socket.py", line 752, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
[2022-05-19, 11:11:21 UTC] {{local_task_job.py:154}} INFO - Task exited with return code 1
[2022-05-19, 11:11:21 UTC] {{local_task_job.py:264}} INFO - 0 downstream tasks scheduled from follow-on schedule check
Then I changed my configuration to
It now gives the following error
File "/usr/lib64/python3.7/smtplib.py", line 642, in auth
raise SMTPAuthenticationError(code, resp)
smtplib.SMTPAuthenticationError: (530, b'Must issue a STARTTLS command first')
[2022-05-19, 12:22:39 UTC] {{local_task_job.py:154}} INFO - Task exited with return code 1
[2022-05-19, 12:22:39 UTC] {{local_task_job.py:264}} INFO - 0 downstream tasks scheduled from follow-on schedule check
Can you please tell me where I am doing wrong or how should I configure this to send an email to a particular email address from any domain?
Your smtp host variable is an email address and not a host.
It should be smtp.gmail.com not smtp#gmail.com
You've hopefully also changed your password as you have shared it publicly in that screenshot and anyone could use it now.
I am running the following setup at AWS:
Elastic Loadbalancer in front of two EC2 machines (Amazon Linux) with a docker container that the salt-master runs in
Two EC2 instances with salt-minions installed
The 'master' value in the minion config is set to the dns of the loadbalancer (SaltMaster-env-vpc-test.szfegmankg.us-east-1.elasticbeanstalk.com)
The ELB accepts all traffic from the minions
The Salt-masters accept all traffic from the ELB as well as from the minions
The Salt-masters PKI Folder is shared between the two masters
The Salt-masters have the same private+public keys
The Salt-masters run on 2017.7.1
The Salt-minions run on 2016.11.5 (I tried it with 2017.7.1, but got the same results)
The Salt-minions accept all traffic from the ELB as well as from the masters
The master config looks as follows:
open_mode: True
worker_threads: 20
auto_accept: True
log_level: error
log_level_logfile: debug
extension_modules: srv/salt/ext
rest_cherrypy:
port: 8000
disable_ssl: True
debug: True
external_auth:
pam:
saltdev:
- .*
- '#runner'
# Setting the job_cache to redis.
# The redis config settings are generated at the start of the docker container and
# will be written into /etc/salt/master.d/redis.conf
master_job_cache: redis
cache: redis
pki_dir: /etc/salt/pki/master/efs
The minion config looks as follows:
id: WIN-AB3GO7BJ72I
log_file: C:\salt.log
multiprocessing: False
log_level_logfile: debug
pki_dir: /conf/pki/minion
master: SaltMaster-env-vpc-test.szfegmankg.us-east-1.elasticbeanstalk.com
master_type: str
master_alive_interval: 30
open_mode: True
root_dir: c:\salt
ipc_mode: tcp
recon_default: 1000
recon_max: 199000
recon_randomize: True
In the master log files, I can see on both masters:
2017-09-05 10:06:18,118 [salt.utils.verify][DEBUG ][35] This salt-master instance has accepted 2 minion keys.
A salt-key -L on both masters yield the same result:
Accepted Keys:
WIN-AB3GO7BJ72I
WIN-EDMP9VB716B
Denied Keys:
Unaccepted Keys:
Rejected Keys:
So it looks like all is fine and everything should work. However, a test.ping is extremely flaky. Sometimes it works, but most of the time it doesnt.
Most of the time neither master gets any return from the minion and on the minion side I can see in the log that the minion never receives the message to execute 'test.ping' from the master.
Example 1:
test.ping from Master1:
root#d7383ff8f8bf:/# salt 'WIN-EDMP9VB716B' test.ping
[ERROR ] Exception raised when processing __virtual__ function for salt.loaded.int.cache.consul. Module will not be loaded: 'module' object has no attribute 'Consul'
[ERROR ] An un-handled exception was caught by salt's global exception handler:
KeyError: 'redis.ls'
Traceback (most recent call last):
File "/usr/bin/salt", line 10, in <module>
salt_main()
File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 476, in salt_main
client.run()
File "/usr/lib/python2.7/dist-packages/salt/cli/salt.py", line 173, in run
for full_ret in cmd_func(**kwargs):
File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 805, in cmd_cli
**kwargs):
File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 1597, in get_cli_event_returns
connected_minions = salt.utils.minions.CkMinions(self.opts).connected_ids()
File "/usr/lib/python2.7/dist-packages/salt/utils/minions.py", line 577, in connected_ids
search = self.cache.ls('minions')
File "/usr/lib/python2.7/dist-packages/salt/cache/__init__.py", line 244, in ls
return self.modules[fun](bank, **self._kwargs)
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1113, in __getitem__
func = super(LazyLoader, self).__getitem__(item)
File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 101, in __getitem__
raise KeyError(key)
KeyError: 'redis.ls'
Traceback (most recent call last):
File "/usr/bin/salt", line 10, in <module>
salt_main()
File "/usr/lib/python2.7/dist-packages/salt/scripts.py", line 476, in salt_main
client.run()
File "/usr/lib/python2.7/dist-packages/salt/cli/salt.py", line 173, in run
for full_ret in cmd_func(**kwargs):
File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 805, in cmd_cli
**kwargs):
File "/usr/lib/python2.7/dist-packages/salt/client/__init__.py", line 1597, in get_cli_event_returns
connected_minions = salt.utils.minions.CkMinions(self.opts).connected_ids()
File "/usr/lib/python2.7/dist-packages/salt/utils/minions.py", line 577, in connected_ids
search = self.cache.ls('minions')
File "/usr/lib/python2.7/dist-packages/salt/cache/__init__.py", line 244, in ls
return self.modules[fun](bank, **self._kwargs)
File "/usr/lib/python2.7/dist-packages/salt/loader.py", line 1113, in __getitem__
func = super(LazyLoader, self).__getitem__(item)
File "/usr/lib/python2.7/dist-packages/salt/utils/lazy.py", line 101, in __getitem__
raise KeyError(key)
KeyError: 'redis.ls'
I am aware that the redis error will be fixed soon https://github.com/saltstack/salt/issues/43295
Example 2:
test.ping from Master1, ~ 1 Minute after Example 1:
root#d7383ff8f8bf:/# salt 'WIN-EDMP9VB716B' test.ping
WIN-EDMP9VB716B:
True
Also during my tests, a test.ping from Master2 never succeeded.
I would like to know if there is some flaw in my setup that I am not seeing, or if Salt only works with an HA Proxy as an ELB?
Or maybe Salt doesn't work at all behind an ELB?
See https://github.com/saltstack/salt/issues/43368 for more answers.
TL;DR
Because there is no session stickyness for TCP connections, it is currently not possible to work with a saltmaster that is behind an ELB, if you use the ELB's ip/name as an entrypoint.
I will try to setup outgoing mail server in odoo 9 .so i fill all the field and test connection and the connection also success , but at the time of send mail it will generate an error .
Field fill like that:-
Name : sendmail
Priority: 10
SMTP Server : smtp.office365.com
SMTP Port:25
Debugging: enable
Connection Security:TLS (STARTTLS)
Username:my yser name
Password:password
But, when we send any mail then it will generate the below error
16-12-06 10:04:28,440 426 INFO test openerp.addons.base.ir.ir_mail_server: Mail delivery failed via SMTP server 'smtp.office365.com'.
SMTPDataError: 550
5.7.60 SMTP; Client does not have permissions to send as this sender
2016-12-06 10:04:28,443 426 ERROR test openerp.addons.mail.models.mail_mail: failed sending mail (id: 136) due to Mail Delivery Failed
Mail delivery failed via SMTP server 'smtp.office365.com'.
SMTPDataError: 550
5.7.60 SMTP; Client does not have permissions to send as this sender
Traceback (most recent call last):
File "/usr/lib/python2.7/dist- packages/openerp/addons/mail/models/mail_mail.py", line 262, in send
res = IrMailServer.send_email(msg, mail_server_id=mail.mail_server_id.id)
File "/usr/lib/python2.7/dist-packages/openerp/api.py", line 248, in wrapper
return new_api(self, *args, **kwargs)
File "/usr/lib/python2.7/dist-packages/openerp/api.py", line 490, in new_api
result = method(self._model, cr, uid, *args, **old_kwargs)
File "/usr/lib/python2.7/dist-packages/openerp/addons/base/ir/ir_mail_server.py", line 483, in send_email
raise MailDeliveryException(_("Mail Delivery Failed"), msg)
MailDeliveryException: (u'Mail Delivery Failed', u"Mail delivery failed via SMTP server 'smtp.office365.com'.\nSMTPDataError: 550\n5.7.60 SMTP; Client does not have permissions to send as this sender")
So i tried too much for this , but i am not getting any solution , if you have any solutin , please share with me .
Use port 587.
The error message tells you that the sender is invalid - you can only send as the mailbox owner (primary SMTP address) or as one of the proxy addresses associated with the mailbox.
Removing all catchall parameters (mail.catchall.domain and mail.catchall.alias) under "Settings" -> "Technical" -> "Parameters" -> "System Parameters" and it work like charm .
WORKS LIKE CHARM :
Removing all catchall parameters (mail.catchall.domain and mail.catchall.alias) under "Settings" -> "Technical" -> "Parameters" -> "System Parameters" and it work like charm . TY Debasish
I'm on Odoo V12
It was not suffisient for my problem, I had to delete the alias domain but there another thing to check :
I initialy created my odoo installation with a GMAIL address, worked a bit but had to switch for a pro e-mail because all my invitations e-mail was blocked by Google Bot beacause it look liked suspicious. It did this only in Odoo v12 because there is more links in the mail.
So I configured my real smtp server in Odoo but get the error 550. Odoo kept in the COMPANY settings the primary gmail address and tried to send on my other smtp server with the gmail name. The other server didn't accepted it so sent me back error 550.
Once i putted my new e-mail address in the company description, and deleted alias domain it worked !!
PS : Don't try to edit ir_mail_server.py to put in bruteforce your e-mail ... Doesn't work ..