Flask Application is not getting deployed at AWS elastic beanstalk - amazon-web-services

Question

I have to deploy flask app on amazon elastic beanstalk
I was following these steps to deploy on elastic beanstalk
http://www.alcortech.com/steps-to-deploy-python-flask-mysql-application-on-aws-elastic-beanstalk/
error code I'm getting
----------------------------------------
/var/log/eb-engine.log
----------------------------------------
2020/08/04 17:54:08.190038 [INFO] Copying file /opt/elasticbeanstalk/config/private/healthd/healthd.conf to /var/proxy/staging/nginx/conf.d/elasticbeanstalk/healthd.conf
2020/08/04 17:54:08.191770 [INFO] Executing instruction: configure log streaming
2020/08/04 17:54:08.191779 [INFO] log streaming is not enabled
2020/08/04 17:54:08.191783 [INFO] disable log stream
2020/08/04 17:54:08.192853 [INFO] Running command /bin/sh -c systemctl show -p PartOf amazon-cloudwatch-agent.service
2020/08/04 17:54:08.298022 [INFO] Running command /bin/sh -c systemctl stop amazon-cloudwatch-agent.service
2020/08/04 17:54:08.303818 [INFO] Executing instruction: GetToggleForceRotate
2020/08/04 17:54:08.303831 [INFO] Checking if logs need forced rotation
2020/08/04 17:54:08.303852 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:859160877773:stack/awseb-e-dxcnd8btg7-stack/fd1c0e90-d67a-11ea-895d-0ee443750bc7 -r AWSEBAutoScalingGroup --region us-east-1
2020/08/04 17:54:09.170590 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:859160877773:stack/awseb-e-dxcnd8btg7-stack/fd1c0e90-d67a-11ea-895d-0ee443750bc7 -r AWSEBBeanstalkMetadata --region us-east-1
2020/08/04 17:54:09.501785 [INFO] Copying file /opt/elasticbeanstalk/config/private/rsyslog.conf to /etc/rsyslog.d/web.conf
2020/08/04 17:54:09.503412 [INFO] Running command /bin/sh -c systemctl restart rsyslog.service
2020/08/04 17:54:10.455082 [INFO] Executing instruction: PostBuildEbExtension
2020/08/04 17:54:10.455106 [INFO] No plugin in cfn metadata.
2020/08/04 17:54:10.455116 [INFO] Starting executing the config set Infra-EmbeddedPostBuild.
2020/08/04 17:54:10.455138 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-init -s arn:aws:cloudformation:us-east-1:859160877773:stack/awseb-e-dxcnd8btg7-stack/fd1c0e90-d67a-11ea-895d-0ee443750bc7 -r AWSEBAutoScalingGroup --region us-east-1 --configsets Infra-EmbeddedPostBuild
2020/08/04 17:54:10.827402 [INFO] Finished executing the config set Infra-EmbeddedPostBuild.
2020/08/04 17:54:10.827431 [INFO] Executing instruction: CleanEbExtensions
2020/08/04 17:54:10.827453 [INFO] Cleaned ebextensions subdirectories from app staging directory.
2020/08/04 17:54:10.827457 [INFO] Executing instruction: RunPreDeployHooks
2020/08/04 17:54:10.827478 [INFO] The dir .platform/hooks/predeploy/ does not exist in the application. Skipping this step...
2020/08/04 17:54:10.827482 [INFO] Executing instruction: stop X-Ray
2020/08/04 17:54:10.827486 [INFO] stop X-Ray ...
2020/08/04 17:54:10.827504 [INFO] Running command /bin/sh -c systemctl show -p PartOf xray.service
2020/08/04 17:54:10.834251 [WARN] stopProcess Warning: process xray is not registered
2020/08/04 17:54:10.834271 [INFO] Running command /bin/sh -c systemctl stop xray.service
2020/08/04 17:54:10.844029 [INFO] Executing instruction: stop proxy
2020/08/04 17:54:10.844061 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2020/08/04 17:54:10.929856 [WARN] stopProcess Warning: process nginx is not registered
2020/08/04 17:54:10.929893 [INFO] Running command /bin/sh -c systemctl stop nginx.service
2020/08/04 17:54:10.935107 [INFO] Executing instruction: FlipApplication
2020/08/04 17:54:10.935119 [INFO] Fetching environment variables...
2020/08/04 17:54:10.935125 [INFO] No plugin in cfn metadata.
2020/08/04 17:54:10.936360 [INFO] Purge old process...
2020/08/04 17:54:10.936404 [INFO] Register application processes...
2020/08/04 17:54:10.936409 [INFO] Registering the proc: web
2020/08/04 17:54:10.936423 [INFO] Running command /bin/sh -c systemctl show -p PartOf web.service
2020/08/04 17:54:10.942911 [INFO] Running command /bin/sh -c systemctl daemon-reload
2020/08/04 17:54:11.190918 [INFO] Running command /bin/sh -c systemctl reset-failed
2020/08/04 17:54:11.195011 [INFO] Running command /bin/sh -c systemctl is-enabled eb-app.target
2020/08/04 17:54:11.198465 [INFO] Copying file /opt/elasticbeanstalk/config/private/aws-eb.target to /etc/systemd/system/eb-app.target
2020/08/04 17:54:11.200382 [INFO] Running command /bin/sh -c systemctl enable eb-app.target
2020/08/04 17:54:11.275179 [ERROR] Created symlink from /etc/systemd/system/multi-user.target.wants/eb-app.target to /etc/systemd/system/eb-app.target.
2020/08/04 17:54:11.275218 [INFO] Running command /bin/sh -c systemctl start eb-app.target
2020/08/04 17:54:11.280436 [INFO] Running command /bin/sh -c systemctl enable web.service
2020/08/04 17:54:11.355233 [ERROR] Created symlink from /etc/systemd/system/multi-user.target.wants/web.service to /etc/systemd/system/web.service.
2020/08/04 17:54:11.355273 [INFO] Running command /bin/sh -c systemctl show -p PartOf web.service
2020/08/04 17:54:11.360364 [INFO] Running command /bin/sh -c systemctl is-active web.service
2020/08/04 17:54:11.363811 [INFO] Running command /bin/sh -c systemctl start web.service
2020/08/04 17:54:11.389333 [INFO] Executing instruction: start X-Ray
2020/08/04 17:54:11.389349 [INFO] X-Ray is not enabled.
2020/08/04 17:54:11.389354 [INFO] Executing instruction: start proxy with new configuration
2020/08/04 17:54:11.389382 [INFO] Running command /bin/sh -c /usr/sbin/nginx -t -c /var/proxy/staging/nginx/nginx.conf
2020/08/04 17:54:11.594594 [ERROR] nginx: the configuration file /var/proxy/staging/nginx/nginx.conf syntax is ok
nginx: configuration file /var/proxy/staging/nginx/nginx.conf test is successful
2020/08/04 17:54:11.595275 [INFO] Running command /bin/sh -c cp -rp /var/proxy/staging/nginx/. /etc/nginx
2020/08/04 17:54:11.603198 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2020/08/04 17:54:11.618752 [INFO] Running command /bin/sh -c systemctl daemon-reload
2020/08/04 17:54:11.716763 [INFO] Running command /bin/sh -c systemctl reset-failed
2020/08/04 17:54:11.724234 [INFO] Running command /bin/sh -c systemctl show -p PartOf nginx.service
2020/08/04 17:54:11.735835 [INFO] Running command /bin/sh -c systemctl is-active nginx.service
2020/08/04 17:54:11.743306 [INFO] Running command /bin/sh -c systemctl start nginx.service
2020/08/04 17:54:11.810080 [INFO] Executing instruction: configureSqsd
2020/08/04 17:54:11.810096 [INFO] This is a web server environment instance, skip configure sqsd daemon ...
2020/08/04 17:54:11.810102 [INFO] Executing instruction: startSqsd
2020/08/04 17:54:11.810105 [INFO] This is a web server environment instance, skip start sqsd daemon ...
2020/08/04 17:54:11.810110 [INFO] Executing instruction: Track pids in healthd
2020/08/04 17:54:11.810114 [INFO] This is an enhanced health env...
2020/08/04 17:54:11.810138 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf aws-eb.target | cut -d= -f2
2020/08/04 17:54:11.819320 [INFO] healthd.service nginx.service cfn-hup.service
2020/08/04 17:54:11.819352 [INFO] Running command /bin/sh -c systemctl show -p ConsistsOf eb-app.target | cut -d= -f2
2020/08/04 17:54:11.826094 [INFO] web.service
2020/08/04 17:54:11.826211 [INFO] Executing instruction: RunPostDeployHooks
2020/08/04 17:54:11.826223 [INFO] The dir .platform/hooks/postdeploy/ does not exist in the application. Skipping this step...
2020/08/04 17:54:11.826228 [INFO] Executing cleanup logic
2020/08/04 17:54:11.826308 [INFO] CommandService Response: {"status":"SUCCESS","api_version":"1.0","results":[{"status":"SUCCESS","msg":"Engine execution has succeeded.","returncode":0,"events":[]}]}
2020/08/04 17:54:11.826448 [INFO] Platform Engine finished execution on command: app-deploy
2020/08/04 17:55:26.814753 [INFO] Starting...
2020/08/04 17:55:26.814816 [INFO] Starting EBPlatform-PlatformEngine
2020/08/04 17:55:26.817259 [INFO] no eb envtier info file found, skip loading env tier info.
2020/08/04 17:55:26.817348 [INFO] Engine received EB command cfn-hup-exec
2020/08/04 17:55:26.939483 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:859160877773:stack/awseb-e-dxcnd8btg7-stack/fd1c0e90-d67a-11ea-895d-0ee443750bc7 -r AWSEBAutoScalingGroup --region us-east-1
2020/08/04 17:55:27.277717 [INFO] Running command /bin/sh -c /opt/aws/bin/cfn-get-metadata -s arn:aws:cloudformation:us-east-1:859160877773:stack/awseb-e-dxcnd8btg7-stack/fd1c0e90-d67a-11ea-895d-0ee443750bc7 -r AWSEBBeanstalkMetadata --region us-east-1
2020/08/04 17:55:27.829610 [INFO] checking whether command tail-log is applicable to this instance...
2020/08/04 17:55:27.829630 [INFO] this command is applicable to the instance, thus instance should execute command
2020/08/04 17:55:27.829635 [INFO] Engine command: (tail-log)
2020/08/04 17:55:27.830551 [INFO] Executing instruction: GetTailLogs
2020/08/04 17:55:27.830557 [INFO] Tail Logs...
2020/08/04 17:55:27.834471 [INFO] Running command /bin/sh -c tail -n 100 /var/log/eb-engine.log
----------------------------------------
/var/log/web.stdout.log
----------------------------------------
Aug 4 17:54:11 ip-172-31-20-145 web: [2020-08-04 17:54:11 +0000] [3881] [INFO] Starting gunicorn 20.0.4
Aug 4 17:54:11 ip-172-31-20-145 web: [2020-08-04 17:54:11 +0000] [3881] [INFO] Listening at: http://127.0.0.1:8000 (3881)
Aug 4 17:54:11 ip-172-31-20-145 web: [2020-08-04 17:54:11 +0000] [3881] [INFO] Using worker: threads
Aug 4 17:54:11 ip-172-31-20-145 web: [2020-08-04 17:54:11 +0000] [3918] [INFO] Booting worker with pid: 3918
----------------------------------------
/var/log/nginx/access.log
----------------------------------------
----------------------------------------
/var/log/nginx/error.log
----------------------------------------
My application.py file is on the root and its source code
from pprint import pprint
import re
import smtplib
import ssl
import docxpy
import glob
import time
import spacy
import requests
import json
import pickle
import numpy as np
import pandas as pd
import tensorflow as tf
from flask import Flask
from flask_restful import Api, Resource, reqparse
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.preprocessing.text import one_hot
from tensorflow.keras.models import model_from_json
import en_core_web_sm
NLP = en_core_web_sm.load()
df = pd.read_csv('skill_train.csv')
df=df.dropna()
df['skill']=pd.to_numeric(df['skill'])
negitive=df[df['skill']==0]
positive=df[df['skill']==1]
application = Flask(__name__)
api = Api(application)
class Candidate:
def __init__(self,file_link):
__text = docxpy.process(file_link).strip()
self.__resume={
'Name':self.__extract_name(__text),
'Phone Number':self.__extract_phone(__text),
'Email':self.__extract_email(__text),
'Experience':self.__extract_experience(__text),
'Skills':list(),
'Title':'',
'match':0,
'file_path':file_link,
}
def get_resume(self):
return self.__resume
def __extract_name(self,text):
try:
return text[:text.index('\n')]
except:
return None
def __extract_email(self,text):
email_pattern = re.compile(r'\S+#\S+\.\S+')
try:
return email_pattern.findall(text)[0].upper()
except:
try:
__hyperlinks = text.data['links'][0][0].decode('UTF-8')
return email_pattern.findall(__hyperlinks)[0].upper()
except:
return None
def __extract_phone(self,text):
phone_pattern = re.compile(r'(\d{3}[-\.\s]??\d{3}[-\.\s]??\d{4}|\(\d{3}\)[-\.\s]*\d{3}[-\.\s]??\d{4}|\d{3}[-\.\s]??\d{4})')
try:
return ''.join(phone_pattern.findall(text)[0]) if len(''.join(phone_pattern.findall(text)[0]))>=10 else None
except:
return None
def __extract_experience(self,text):
try:
__exp_pattern = re.compile(r'\d\+ years|\d years|\d\d\+ years|\d\d years|\d\d \+ Years|\d \+ Years')
__exp = __exp_pattern.findall(text)
return str(max([int(re.findall(re.compile(r'\d+'),i)[0]) for i in __exp])) + '+ years'
except:
try:
__date_patt = re.compile(r"\d{2}[/-]\d+")
__dates_list = __date_patt.findall(text)
try:
__year_list=[int(date[-4:]) for date in __dates_list]
except:
__year_list=[int(date[-2:]) for date in __dates_list]
return str(max(__year_list)-min(__year_list))+'+ years'
except:
return None
class JobDescription:
def __init__(self,args):
self.description=args['job_description'].upper()
self.__title=self.__get_title(self.description) if 'job_title' not in args else args['job_title'].upper()
__doc=NLP(self.description)
__noun_chunks=set([chunk.text.upper() for chunk in __doc.noun_chunks])
self.__skills=list(self.__get_skills(list(__noun_chunks)))
def title(self):
return self.__title
def skills(self):
return self.__skills
def __clean_data(self,noun_chunks):
subs=[r'^[\d|\W]*','EXPERIENCE','EXPERT','DEVELOPER','SERVICES','STACK','TECHNOLOGIES',
'JOBS','JOB',r'\n',' ',r'\t','AND','DEV','SCRIPTS','DBS','DATABASE','DATABASES','SERVER',
'SERVERS',r'^\d+']
__clean_chunks=[]
for chunk in noun_chunks:
for sub in subs:
chunk=(re.sub(sub,' ',chunk).strip())
filtered_chunk=[]
chunk=chunk.split(' ')
for word in chunk:
for sub in subs:
word=(re.sub(sub,' ',word).strip())
if word != '':
if not NLP.vocab[word.strip()].is_stop:
filtered_chunk.append(word.strip())
filtered_chunk=' '.join(filtered_chunk)
if filtered_chunk != '' and filtered_chunk != ' ':
if ',' in filtered_chunk:
__clean_chunks+=filtered_chunk.split(',')
elif '/' in filtered_chunk:
__clean_chunks+=filtered_chunk.split('/')
else:
__clean_chunks.append(filtered_chunk)
return set([chunk.strip() for chunk in __clean_chunks])
def __get_skills(self,nounChunks):
with open('skill_model.json','r') as f:
model=f.read()
sq_model = model_from_json(model)
sq_model.load_weights('skillweights.h5')
__clean_chunks=list(self.__clean_data(nounChunks))
__onehot_repr=[one_hot(words,25000)for words in __clean_chunks]
__test_data=pad_sequences(__onehot_repr,padding='pre',maxlen=6)
__results = [(x,y[0])for x,y in zip(__clean_chunks,sq_model.predict_classes(np.array(__test_data)))]
ones=set(positive['chunk'])
zeros=set(negitive['chunk'])
for i,result in enumerate(__results):
if result[0] in ones and result[1] !=1:
__results[i]=(result[0],1)
if result[0] in zeros and result[1] !=0:
__results[i]=(result[0],0)
return set([x[0] for x in __results if x[1]==1])
def __get_title(self,text):
try:
__role=re.findall(re.compile(r'POSITION[ ]*:[\w .\(\)]+|ROLE[ ]*:[\w .\(\)]+|TITLE[ ]*:[\w .\(\)]+'),text)[0].split(':')[1].strip()
if '(' in __role:
__role=re.findall(re.compile(r'\([\w ]+\)'),__role)[0][1:-1].strip()
return __role.upper()
except:
return None
def __matcher(self,resume):
__text = docxpy.process(resume['file_path']).upper()
if self.__title in __text:
resume['Title']=self.__title
for skill in self.__skills:
if skill in __text:
resume['Skills'].append(skill)
resume['Skills'] = list(set(resume['Skills']))
resume['match'] = 0.0 if len(self.__skills)==0 else (len(resume['Skills'])/len(self.__skills))*100
return resume
def filter_matches(self,candidates):
if self.__title != None:
__matches = []
for user in candidates:
resume = user.get_resume()
result = self.__matcher(resume)
if (result['Title']!='' and result['match']>60) or result['match']>60:
__matches.append(result)
return sorted(__matches, key=lambda match:match['match'], reverse=True)
else:
print('Unable to extract Role try writing Role:...... or Position:....')
def send_mail(self,matches):
__port = 465
__smtp_server = "smtp.gmail.com"
__sender_email = 'sonai20202015#gmail.com'
__password = 'Sonai#123'
context = ssl.create_default_context()
with smtplib.SMTP_SSL(__smtp_server, __port, context=context) as server:
server.login(__sender_email, __password)
for Candidate in matches:
__reciver_email = Candidate['Email']
__message=f'''Subject: Job offer
Hi {Candidate['Name']},
This is an autogenrated email from an ATS SONAI we found your resume to be a
good match for {self.__title} job
'''
server.sendmail(__sender_email,__reciver_email, __message)
def get_acess(self):
auth_url = 'https://secure.dice.com/oauth/token'
auth_header = {'Authorization': 'Basic dHM0LWhheWRlbnRlY2hub2xvZ3k6Yzk0NWI4YmItMmRmNi00Yjk4LThmNDUtMTg4ZWU5Mjk3ZGEz', 'Content-Type': 'application/x-www-form-urlencoded'}
auth_data = {'grant_type': 'password', 'username': 'haydentechnology#dice.com', 'password': '635n3E7s'}
try:
auth_response = requests.request('POST',auth_url,headers=auth_header,data=auth_data)
auth_code = auth_response.status_code
auth_response = json.loads(auth_response.content.decode())
return (auth_code,auth_response)
except:
return(0,'')
def boolean_skills(self):
with open('output.pkl','rb') as f:
data = pickle.load(f)
if self.__title in data:
output = []
for skill in self.__skills:
if skill in data[self.__title][0] and data[self.__title][0][skill]>(3/4)*data[self.__title][1]:
continue
output.append(skill)
return output
return self.__skills
def search_with_api(self):
auth_response = self.get_acess()
if auth_response[0] == 200:
token = auth_response[1]['access_token']
headers = {'Authorization':f'bearer {token}'}
url = 'https://talent-api.dice.com/v2/profiles/search?q='
boolean_skills = self.boolean_skills()
for skill in boolean_skills:
url += f'{skill}&'
url = url + self.__title
print('\n',url,'\n')
try:
output = requests.request('GET',url,headers=headers)
output = json.loads(output.content.decode())
return output
except:
return ('error while finding users')
else:
return ('Authentication error with dice')
class Search_Candidates(Resource):
def post(self):
parser = reqparse.RequestParser()
parser.add_argument("application_type",required=False)#String
parser.add_argument("application_name",required=False)#String
parser.add_argument("application_internal_only",required=False)#Boolean
parser.add_argument("application_applicant_history",required=False)#Boolean
parser.add_argument("application_years_of_employement_needed",required=False)#Float
parser.add_argument("application_number_of_refrences",required=False)#Float
parser.add_argument("application_flag_voluntarily_resign",required=False)#Boolean
parser.add_argument("application_flag_past_employer_contracted",required=False)#Boolean
parser.add_argument("email_template_default_address",required=False)#String
parser.add_argument("task",required=False)#List
parser.add_argument("job_title",required=True)#String
parser.add_argument("employement_status",required=False)#String
parser.add_argument('job_description', required=True)#String
parser.add_argument("joinig_date",required=False)#String as ISO STANDARDS
parser.add_argument("salary",required=False)#Float
parser.add_argument("average_hours_weekly",required=False)#Float
parser.add_argument("post_title",required=False)#String
parser.add_argument("post_details_category",required=False)#String
parser.add_argument("number_of_open_position",required=False)#Float
parser.add_argument("general_application",required=False)#Boolean
args = parser.parse_args()
response = self.find_matches(args)
response = json.dumps(response)
return response
def find_matches(self,args):
file_paths=glob.glob(r'demo_word_file\*.docx')
candidates=[Candidate(file_path) for file_path in file_paths]
job = JobDescription(args)
start_time=time.time()
results = job.filter_matches(candidates)
pprint(f'Found and Sorted {len(results)} results in {time.time()-start_time} secs from {len(candidates)} files')
matches = [matches for matches in job.filter_matches(candidates)]
if not len(matches) == 0:
matches_with_email=[match for match in matches if match['Email'] != None]
job.send_mail(matches_with_email)
else:
results = job.search_with_api()
return results
def run():
file_paths=glob.glob(r'demo_word_file\*.docx')
candidates=[Candidate(file_path) for file_path in file_paths]
text = docxpy.process('jobtest.docx')
args= {'job_description': text}
job = JobDescription(args)
results = job.filter_matches(candidates)
return results
if __name__ == "__main__":
api.add_resource(Search_Candidates,'/findmatches/')
application.run('localhost',8080,debug=True)
My requirement.txt file is here
# Automatically generated by https://github.com/damnever/pigar.
# application.py: 15
Flask == 1.0.4
# application.py: 16
Flask_RESTful == 0.3.8
# application.py: 5
docxpy == 0.8.5
# application.py: 12
numpy == 1.19.1
# application.py: 13
pandas == 1.1.0
# application.py: 9
requests == 2.18.4
spacy>=2.2.0,<3.0.0
https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-2.2.0/en_core_web_sm-2.2.0.tar.gz#egg=en_core_web_sm
# application.py: 14,17,18,19
tensorflow == 1.14.0
Flask-SQLAlchemy==2.4.3
itsdangerous==1.1.0
Jinja2==2.11.2
MarkupSafe==1.1.1
pytz==2020.1
six==1.15.0
SQLAlchemy==1.3.18
Werkzeug==1.0.1
Environment health status is ok but in environment url I am constantly getting 404 not found
My code is working on development server but its not working here on production server

Answer 1

One reason is probably incorrect port.
You are using port 8080:
application.run('localhost',8080,debug=True)
but default port on EB for your application is 8000. If you want to use non-default port, you can define EB environment variable PORT with the value of 8080. You can do this using .ebextenations or in EB console.
Also, there could be many other issues, which are not apparent yet. For example, the tutorial linked is using old version of EB environment, based on Amazon Linux 1, but you are using Amazon Linux 2. There are many differences between AL1 and AL2 which make them incompatible.
Tensorflow is a resource hungry package. Although instance type is not specified in your question, t2.micro can be too small for it, in case you are using it for testing or development.