Im very new to Api and our use case is to stream the live data from webcam to kinesis video stream (kvs).
Steps Taken : Created the ubuntu server on aws and installed the CPP SDK.
created the kinesis video stream in aws
downloaded and installed the Gstreamer on my local.
I was trying to put the rtsp sample data to gstream which is on Ec2ubuntu server server i ran below query
$ gst-launch-1.0 rtspsrc location="rtsp://YourCameraRtspUrl" short-header=TRUE ! rtph264depay ! video/x-h264, format=avc,alignment=au ! kvssink stream-name="YourStreamName" storage-size=512 access-key="YourAccessKey" secret-key="YourSecretKey" aws-region="YourAWSRegion"
im getting attached error may be we need to open some port on ec2 ?
Suggestion required :how can i put my local webcam video to kinesis ?
thanks
If you use macOS, you can stream your webcam into Kinesis Video Stream like the command below setting fps 1;
AWS_ACCESS_KEY_ID=<YOUR_AWS_ACCESS_KEY_ID>
AWS_SECRET_ACCESS_KEY=<YOUR_AWS_SECRET_ACCESS_KEY>
AWS_REGION=<YOUR_AWS_REGION>
STREAM_NAME=<YOUR_STREAM_NAME>
gst-launch-1.0 -v avfvideosrc \
! clockoverlay font-desc="Sans bold 60px" \
! videorate \
! video/x-raw,framerate=1/1 \
! vtenc_h264_hw allow-frame-reordering=FALSE realtime=TRUE max-keyframe-interval=2 bitrate=512 \
! h264parse \
! video/x-h264,stream-format=avc,alignment=au \
! kvssink stream-name="${STREAM_NAME}" storage-size=512 \
access-key="${AWS_ACCESS_KEY_ID}" \
secret-key="${AWS_SECRET_ACCESS_KEY}" \
aws-region="${AWS_REGION}" \
frame-timecodes=true \
framerate=1
It's not immediately clear from your question what your scenario is. I assume if you refer to a webcam then it's attached to your computer. Depending of your use case, you could check out the producer libraries that will help integrating.
Java: https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-java
C++: https://github.com/awslabs/amazon-kinesis-video-streams-producer-sdk-cpp
C: https://github.com/awslabs/amazon-kinesis-video-streams-producer-c
You could ask relevant questions and get support in GitHub by cutting an Issue in the appropriate repository
Related
I'm trying to automate creation of connector in airflow by github action, but since it is an external provider, the payload that need to be sent to airflow REST API doesn't work and i didn't find any documentation on how to do it.
So here is the PAYLOAD i'm trying to send :
PAYLOAD = {
"connection_id": CONNECTOR,
"conn_type": "google_cloud_platform",
"extra": json.dumps({
"google_cloud_platform": {
"keyfile_dict" : open(CONNECTOR_SERVICE_ACCOUNT_FILE, "r").read(),
"num_retries" : 2,
}
})
}
According to the airflow documentation here
And the information i found on the "create connector" page of airflow UI :
Airflow UI create connector page
But i received no error (code 200) and the connector is created but doesn't have the settings i tried to configure.
I confirm the creation works on the UI.
Does anyone have a solution or document that refer to the exact right payload i need to sent to airflow rest api ? Or maybe i miss something.
Airflow version : 2.2.3+composer
Cloud Composer version (GCP) : 2.0.3
Github runner version : 2.288.1
Language : Python
Thanks guys and feel free to contact me for further questions.
Bye
#vdolez was write, it's kind of a pain to format the payload to have the exact same format airflow REST API want. it's something like this :
"{\"extra__google_cloud_platform__key_path\": \"\",
\"extra__google_cloud_platform__key_secret_name\": \"\",
\"extra__google_cloud_platform__keyfile_dict\": \"{}\",
\"extra__google_cloud_platform__num_retries\": 5,
\"extra__google_cloud_platform__project\": \"\",
\"extra__google_cloud_platform__scope\": \"\"}"
And when you need to nest dictionnary inside some of these field, not worth the time and effort. But in case someone want to know, you have to escape every special character.
I change my workflow to notify competent users to create connector manually after my pipeline succeed.
I will try to contact airflow/cloud composer support to see if we can have a feature for better formatting.
You might be running into encoding/decoding issues while sending data over the web.
Since you're using Composer, it might be a good idea to use Composer CLI to create a connection.
Here's how to run airflow commands in Composer:
gcloud composer environments run ENVIRONMENT_NAME \
--location LOCATION \
SUBCOMMAND \
-- SUBCOMMAND_ARGUMENTS
Here's how to create a connection with the native Airflow commands:
airflow connections add 'my_prod_db' \
--conn-type 'my-conn-type' \
--conn-login 'login' \
--conn-password 'password' \
--conn-host 'host' \
--conn-port 'port' \
--conn-schema 'schema' \
...
Combining the two, you'll get something like:
gcloud composer environments run ENVIRONMENT_NAME \
--location LOCATION \
connections \
-- add 'my_prod_db' \
--conn-type 'my-conn-type' \
--conn-login 'login' \
--conn-password 'password' \
--conn-host 'host' \
--conn-port 'port' \
--conn-schema 'schema' \
...
You could run this in a Docker image where gcloud is already installed.
I ran the command. I have an ubuntu 18 running on arm64 with an external monitor connected to nvidia jetson xavier.
gst-launch-1.0 videotestsrc ! videoconvert ! autovideosink
However it doesn't do anything . neither does it play any video. Terminal return following line
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
I am trying to run multiple Spark Structured Streaming jobs (on EMR) that read from Kafka topics and write to different paths in S3 (each performed within their respective jobs). I have configured my cluster to use the CapacityScheduler. Here is a snippet of the code that I am trying to run:
df = spark \
.readStream \
.format("kafka") \
.option("kafka.bootstrap.servers", <BOOTSTRAP_SERVERS>) \
.option("subscribePattern", "<MY_TOPIC>") \
.load() \
.selectExpr("CAST(key AS STRING)", "CAST(value AS STRING)")
output = df \
.writeStream \
.format("json") \
.outputMode("update") \
.option("checkpointLocation", "s3://<CHECKPOINT_LOCATION>") \
.option("path", "s3://<SINK>") \
.start() \
.awaitTermination()
I tried running two jobs in parallel:
spark-submit --queue <QUEUE_1> --deploy-mode cluster --master yarn <STREAM_1_SCRIPT>.py
spark-submit --queue <QUEUE_2> --deploy-mode cluster --master yarn <STREAM_2_SCRIPT>.py
During execution, I noticed that the second job was not writing to S3 (even though the first job was). I also noticed a huge spike in the utilization via the Spark UI for the second job.
After stopping the first job, the data showed up for the second job in S3. Is it not possible to run two separate Spark Structured Streaming jobs that write to sinks (specifically on S3) in parallel? Does the write operation cause a some kind of blocking?
Yes, you can !
That's it's not something that had multiple sources docummented but, the only thing that you need its share the spark context between your threads of multiple jobs.
I make a multiple spark structtured streaming pipeline following this article https://cm.engineering/multiple-spark-streaming-jobs-in-a-single-emr-cluster-ca86c28d1411 any questions you can send me an email or talk inbox to me.
Thank you !
I am running a training job on GCP for object detection using my own dataset. My training job script is like this:
JOB_NAME=object_detection"_$(date +%m_%d_%Y_%H_%M_%S)"
echo $JOB_NAME
gcloud ml-engine jobs submit training $JOB_NAME \
--job-dir=gs://$1 \
--scale-tier BASIC_GPU \
--runtime-version 1.12 \
--packages $PWD/models/research/dist/object_detection-0.1.tar.gz,$PWD/models/research/slim/dist/slim-0.1.tar.gz,/tmp/pycocotools/pycocotools-2.0.tar.gz \
--module-name $PWD/models/research/object_detection.model_main \
--region europe-west1 \
-- \
--model_dir=gs://$1 \
--pipeline_config_path=gs://$1/data/fast_rcnn_resnet101_coco.config
It fails at the following line :
python -m $PWD/models/research/object_detection.model_main --model_dir=gs://my-hand-detector --pipeline_config_path=gs://my-hand-detector/data/fast_rcnn_resnet101_coco.config --job-dir gs://my-hand-detector/
/usr/bin/python: Import by filename is not supported.
Based on logs, this is the source of error which I have understood. Any help in this regard would be helpful. Thank you.
I assume that you are using model_main.py file from Tensorflow GitHub repository. Using it, I have been able to replicate your error message. After troubleshooting, I successfully submitted the training job and could train the model properly.
In order to address your issue I suggest you to follow this tutorial, taking special consideration to the following steps:
Make sure to have an updated version of tensorflow (1.14 doesn’t include all necessary capabilities)
Properly generate TFRecords from input data and upload them to GCS bucket
Configure object detection pipeline (set the proper paths to data and label map)
In my case, I have reproduced the workflow using PASCAL VOC input data (See this).
I have this gstreamer command that plays stream from one Ubuntu 16.04 box but two others.
As far as I know I have same packages installed regarding gstreamer on all boxes because I ran sudo apt-get install gstreamer1-0* so that all gstreamer 1.0 packages are installed. I find it strange that it does not work on other
Following commands where used:
gst-launch-1.0 rtspsrc location=rtsp://<user>:<password>#<IP>/axis-media/media.amp user-id=root user-pw=xxxxxxxxxxc latency=150 ! decodebin max-size-time=30000000000 ! videoconvert ! autovideosink
or
gst-launch-1.0 playbin uri=rtsp://<user>:<password>#<IP>/axis-media/media.amp
This will open a stream of a Axis camera with h264. I don't understand why it does not work on two Ubuntu 16.04 boxes but works on one. All of these are the same Ubuntu 16.04 with same gstreamer packages installed.
Could there maybe be another package except gstreamer that is necessary in order for gstreamer to stream rtsp?
The error a get when it is not playing
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Got context from element 'autovideosink0': gst.gl.GLDisplay=context, gst.gl.GLDisplay=(GstGLDisplay)"\(GstGLDisplayX11\)\ gldisplayx11-0";
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://root:pass#172.26.134.166/axis-media/media.amp
(gst-launch-1.0:4036): GLib-GIO-WARNING **: Ignoring invalid ignore_hosts value '*]'
ERROR: from element /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0: Could not open resource for reading and writing.
Additional debug info:
gstrtspsrc.c(6795): gst_rtspsrc_retrieve_sdp (): /GstPipeline:pipeline0/GstRTSPSrc:rtspsrc0:
Failed to connect. (Generic error)
ERROR: pipeline doesn't want to preroll.
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...
Found the command from this webpage:
http://gstreamer-devel.966125.n4.nabble.com/gstreamer-client-pipeline-to-view-video-from-AXIS-M1054-Network-Camera-td4667092.html
Well it seems that it was the proxy that caused the issue. When I disabled proxy I was able to stream with gstreamer on all boxes.
Best regards