Hello I am trying to create my Google Cloud Function in Python by using Terraform
resource "google_cloudfunctions_function" "function" {
name = "scheduled-cloud-function-python-api-request"
description = "A Python Cloud Function that is triggered by a Cloud Schedule."
runtime = "python37"
available_memory_mb = 128
source_archive_bucket = google_storage_bucket.bucket.name
source_archive_object = google_storage_bucket_object.archive.name
trigger_http = true
entry_point = "http_handler" # This is the name of the function that will be executed in your Python code
}
And it's giving me an error at this code block after I do terraform apply:
Error code 3, message: Function failed on loading user code.
What am I doing wrong here?
This is my python code :
import requests
import csv
request = requests.get(f'https://api.sportsdata.io/v3/nba/scores/json/TeamSeasonStats/2022?key=')
nba_team_stats = request.json()
data_file = open('nba_team_stats.csv', 'w')
csv_writer = csv.writer(data_file)
count = 0
for stats in nba_team_stats:
if count == 0:
header = stats.keys()
csv_writer.writerow(header)
count += 1
csv_writer.writerow(stats.values())
data_file.close()
Am I writing my Python Code in an undesired way for the GCP function to execute?
Related
I'm using SageMaker to create and deploy a SKLearn estimator (endpoint). I created a my_file.py custom script, but when I fit the estimator, in the logs I see:
TypeError: 'NoneType' object is not subscriptable
where the object is a pandas dataframe obtained by reading a parquet dataset.
If I run the my_file.py file locally, the dataset is not set to None.
Here's my SageMaker code:
import boto3
import sagemaker
from sagemaker.sklearn import SKLearn
my_estimator = SKLearn(entry_point = 'my_file.py',
source_dir = 'my_dir',
py_version = 'py3',
role = sagemaker.get_execution_role(),
instance_type = '<instance_type>',
instance_count = <number_of_instances>,
framework_version = '<version>',
output_path = 'my_path',
base_job_name = 'my_job_name')
# These directories are S3 directories
model_dir = 'my_model_dir'
dataset = 'my_dataset_dir'
file1 = 'my_file1_dir'
empty_dir1 = 'my_empty_dir1'
empty_dir2 = 'my_empty_dir2'
empty_dir3 = 'my_empty_dir3'
my_estimator.fit(inputs = {'dataset': dataset,
'file1': file1,
'emptydir1': empty_dir1,
'emptydir2': empty_dir2,
'emptydir3': empty_dir3}, logs = True)
And here's my my_file.py code:
# importing libraries
# defining some functions
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('--model-dir', type=str, default=os.environ['SM_MODEL_DIR'])
parser.add_argument('--dataset', type=str, default=os.environ['SM_CHANNEL_DATASET'])
parser.add_argument('--file1', type=str, default=os.environ['SM_CHANNEL_FILE1'])
parser.add_argument('--emptydir1', type=str, default=os.environ['SM_CHANNEL_EMPTYDIR1'])
parser.add_argument('--emptydir2', type=str, default=os.environ['SM_CHANNEL_EMPTYDIR2'])
parser.add_argument('--emptydir3', type=str, default=os.environ['SM_CHANNEL_EMPTYDIR3'])
args, _ = parser.parse_known_args()
filename = args.dataset
dataset = pd.read_parquet(filename)
print(len(dataset))
# The following line results in the error
dataset['column1'] = dataset['column1'].astype(str)
# other lines of code
Printing the length of the dataset gives me the correct length, so the dataframe is not empty.
Also, the values in the dataframe are neither None nor NaN.
Don't know why I get this error. Any advice?
EDIT
After re-running the fit code some times the error disappeared
Can you try running the following code locally with the same dataset.
dataset['column1'] = dataset['column1'].astype(str)
It seems as if CW is emitting an error specific to this API call/line, it is best to perhaps test this locally first before ingesting into a SageMaker script as this is dependent on the libraries that you are utilizing. You can also use SageMaker local mode to capture these errors early.
I am trying to train a PyTorch model through SageMaker. I am running a script main.py (which I have posted a minimum working example of below) which calls a PyTorch Estimator. I have the code for training my model saved as a separate script, train.py, which is called by the entry_point parameter of the Estimator. These scripts are hosted on a EC2 instance in the same AWS region as my SageMaker domain.
When I try running this with instance_type = "ml.m5.4xlarge", it works ok. However, I am unable to debug any problems in train.py. Any bugs in that file simply give me the error: 'AlgorithmError: ExecuteUserScriptError', and will not allow me to set breakpoint() lines in train.py (encountering a breakpoint throws the above error).
Instead I am trying to run in local mode, which I believe does allow for breakpoints. However, when I reach estimator.fit(inputs), it hangs on that line indefinitely, giving no output. Any print statements that I put at the start of the main function in train.py are not reached. This is true no matter what code I put in train.py. It also did not throw an error when I had an illegal underscore in the base_job_name parameter of the estimator, which suggests that it does not even create the estimator instance.
Below is a minimum example which replicates the issue on my instance. Any help would be appreciated.
### File structure
main.py
customcode/
|
|_ train.py
### main.py
import sagemaker
from sagemaker.pytorch import PyTorch
import boto3
try:
# When running on Studio.
sess = sagemaker.session.Session()
bucket = sess.default_bucket()
role = sagemaker.get_execution_role()
except ValueError:
# When running from EC2 or local machine.
print('Performing manual setup.')
bucket = 'MY-BUCKET-NAME'
region = 'us-east-2'
role = 'arn:aws:iam::MY-ACCOUNT-NUMBER:role/service-role/AmazonSageMaker-ExecutionRole-XXXXXXXXXX'
iam = boto3.client("iam")
sagemaker_client = boto3.client("sagemaker")
boto3.setup_default_session(region_name=region, profile_name="default")
sess = sagemaker.Session(sagemaker_client=sagemaker_client, default_bucket=bucket)
hyperparameters = {'epochs': 10}
inputs = {'data': f's3://{bucket}/features'}
train_instance_type = 'local'
hosted_estimator = PyTorch(
source_dir='customcode',
entry_point='train.py',
instance_type=train_instance_type,
instance_count=1,
hyperparameters=hyperparameters,
role=role,
base_job_name='mwe-train',
framework_version='1.12',
py_version='py38',
input_mode='FastFile',
)
hosted_estimator.fit(inputs) # This is the line that freezes
### train.py
def main():
breakpoint() # Throws an error in non-local mode.
return
if __name__ == '__main__':
print('Reached') # Never reached in local mode.
main()
i am trying to create efs filesystem using cloudformation template inside lambda using boto3. And interested to return output as Filesystemid from stack using describe_stack. however i am getting null values in return. please suggest where i am making mistake.
error is:
Response
null
Code is:
import boto3
import time
import json
import botocore
datetime = time.strftime("%Y%m%d%H%M%S")
stackname = 'My-EFS'
region = "ap-south-1"
client = boto3.client('cloudformation')
s = boto3.Session(region_name=region)
def lambda_handler(event, context):
response = client.create_stack(
StackName= stackname,
TemplateURL='https://cloudnaeem.s3.amazonaws.com/efs.yaml',
)
waiter = client.get_waiter('stack_create_complete')
res = waiter.wait(
StackName=stackname,
)
stack = client.describe_stacks(StackName=stackname)
FileSystem_id=None
for v in stack["Stacks"][0]["Outputs"]:
if v["OutputKey"] == "FileSystemId":
FileSystem_id = v["OutputValue"]
return FileSystem_id
template output is :
Outputs:
EFS:
Description: The created EFS
Value: !Ref EFSFileSystem
Your output is called EFS but you are looking for FileSystemId. Your code should be thus:
for v in stack["Stacks"][0]["Outputs"]:
if v["OutputKey"] == "EFS":
FileSystem_id = v["OutputValue"]
Trying to run a Lambda function to invoke SSM and define an EC2 tag to push the same on multiple instances with the below script. Getting the below error when trying to execute. I am just started learning to write a script and using aws lambda first time. Please help me to fix.
import boto3
ssm = boto3.client('ssm')
ec2 = boto3.resource('ec2')
def lambda_handler(event, context):
filters = (
Name = 'tag:Product',
Values = ['Essay']
)
instances = ('filters')
response = ssm.send_command(
InstanceIds=instances,
DocumentName='xxxxxxxxxxxxx',
DocumentVersion='$DEFAULT',
DocumentHash='916fdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxdcdbe7940',
DocumentHashType='Sha256'
)
print(response)
Error :
Response:
{
"errorMessage": "Syntax error in module 'lambda_function': invalid syntax (lambda_function.py, line 7)",
"errorType": "Runtime.UserCodeSyntaxError",
"stackTrace": [
" File \"/var/task/lambda_function.py\" Line 7\n Name = 'tag:Product',\n"
]
}
Request ID:
"8cb4cd39-b744-41da-befb-5f60b6e49fa4"
Function logs:
START RequestId: 8cb4cd39-b744-41da-befb-5f60b6e49fa4 Version: $LATEST
[ERROR] Runtime.UserCodeSyntaxError: Syntax error in module 'lambda_function': invalid syntax (lambda_function.py, line 7)
Traceback (most recent call last):
File "/var/task/lambda_function.py" Line 7
Name = 'tag:Product',END RequestId: 8cb4cd39-b744-41da-befb-5f60b6e49fa4
REPORT RequestId: 8cb4cd39-b744-41da-befb-5f60b6e49fa4
There are several issues:
Wrong indentation.
There is no such thing in python as
filters = (
Name = 'tag:Product',
Values = ['Essay']
)
maybe you meant dictionary?:
filters = {
'Name':'tag:Product',
'Values': ['Essay']
}
InstanceIds=instances should be a list of strings, not a literal string of 'filters'.
The closes to fixing the code is the following:
import boto3
ssm = boto3.client('ssm')
ec2 = boto3.resource('ec2')
def lambda_handler(event, context):
filters = [{
'Name':'tag:Product',
'Values': ['Essay']
}]
instances = [instance.id for instance in ec2.instances.filter(Filters = filters)]
response = ssm.send_command(
InstanceIds=instances,
DocumentName='xxxxxxxxxxxxx',
DocumentVersion='$DEFAULT',
DocumentHash='916fdxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxdcdbe7940',
DocumentHashType='Sha256'
)
print(response)
I'm trying to loop through every region and check if a stack has drifted or not, and then print a list of drifted stacks.
# !/usr/bin/env python
import boto3
import time
## Create a AWS Session
session = boto3.Session(profile_name='default', region_name='us-east-1')
if __name__ == '__main__':
## Connect to the EC2 Service
client = session.client('ec2')
## Make a list of all the regions
response = client.describe_regions()
for region in response['Regions']:
name = region['RegionName']
print("Drifted CFn in region: " + name)
## Connect to the CFn service in the region
cloudformationClient = boto3.client("cloudformation")
stacks = cloudformationClient.describe_stacks()
detection_id = cloudformationClient.detect_stack_drift(StackName=stacks)
for stack in stacks['Stacks']:
while True:
time.sleep(3)
# sleep between api calls to prevent lockout
response = cloudformationClient.describe_stack_drift_detection_status(
StackDriftDetectionId=detection_id
)
if response['DetectionStatus'] == 'DETECTION_IN_PROGRESS':
continue
else:
print("Stack" + stack + " has a drift status:" + response)
I am still new to Python and am unsure why it's failing on the StackName on line 22 when I know that the that's the name of the variable in "detect_stack_drift" that I'm trying to parse. Some help would be appreciated!
See these lines:
stacks = cloudformationClient.describe_stacks()
detection_id = cloudformationClient.detect_stack_drift(StackName=stacks)
The describe_stacks() call returns:
{
'Stacks': [
{
'StackId': 'string',
'StackName': 'string',
...
},
],
'NextToken': 'string'
}
However, the detect_stack_drift() function is expecting a single string in StackName.