Kinesis Video Streaming is Not Working for Specific Cameras

Kinesis Video Streaming is Not Working for Specific Cameras - amazon-web-services

I am using Kinesis Video Streaming for a solution and I am facing some problems with few RTSP sources. To all rtsp sources I am also validating on VLC and it is working well, except for a huge delay to start to stream the video.
I am using gst-launch-1.0 to stream to Kinesis
gst-launch-1.0 -q rtspsrc location=MYRTSP_URL short-header=TRUE ! rtph264depay ! h264parse ! kvssink stream-name=MYSTREAM_NAME storage-size=128 access-key=MYACCESS_KEY secret-key=MYSECRET_KEY aws-region=us-east-1 retention-period=168
Below there is result after command execution:
log4cplus:ERROR could not open file ../kvs_log_configuration
INFO - createKinesisVideoClient(): Creating Kinesis Video Client
2022-07-04 19:37:21 [140474457073472] INFO - heapInitialize(): Initializing native heap with limit size 134217728, spill ratio 0% and flags 0x00000001
2022-07-04 19:37:21 [140474457073472] INFO - heapInitialize(): Creating AIV heap.
2022-07-04 19:37:21 [140474457073472] INFO - heapInitialize(): Heap is initialized OK
2022-07-04 19:37:21 [140474457073472] DEBUG - getSecurityTokenHandler invoked
2022-07-04 19:37:21 [140474457073472] DEBUG - Refreshing credentials. Force refreshing: 0 Now time is: 1656963441548249256 Expiration: 0
2022-07-04 19:37:21 [140474457073472] INFO - createDeviceResultEvent(): Create device result event.
2022-07-04 19:37:21 [140474457073472] DEBUG - clientReadyHandler invoked
2022-07-04 19:37:21 [140474457073472] INFO - try creating stream
2022-07-04 19:37:21 [140474457073472] INFO - Creating Kinesis Video Stream MYSTREAM_NAME
2022-07-04 19:37:21 [140474457073472] INFO - createKinesisVideoStream(): Creating Kinesis Video Stream.
2022-07-04 19:37:21 [140474457073472] INFO - logStreamInfo(): SDK version: 70f74f14cf27b09f71dc1889f36eb6e04cdd90a8
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Kinesis Video Stream Info
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Stream name: MYSTREAM_NAME
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Streaming type: STREAMING_TYPE_REALTIME
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Content type: video/h264
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Max latency (100ns): 600000000
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Fragment duration (100ns): 20000000
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Key frame fragmentation: Yes
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Use frame timecode: Yes
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Absolute frame timecode: Yes
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Nal adaptation flags: 0
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Average bandwith (bps): 4194304
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Framerate: 25
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Buffer duration (100ns): 1200000000
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Replay duration (100ns): 400000000
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Connection Staleness duration (100ns): 600000000
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Store Pressure Policy: 1
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): View Overflow Policy: 1
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Segment UUID: NULL
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Frame ordering mode: 0
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Track list
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Track id: 1
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Track name: kinesis_video
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Codec id: V_MPEG4/ISO/AVC
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Track type: TRACK_INFO_TYPE_VIDEO
2022-07-04 19:37:21 [140474457073472] DEBUG - logStreamInfo(): Track cpd: NULL
2022-07-04 19:37:21 [140474457073472] INFO - writeHeaderCallback(): RequestId: 9165f977-7fe3-413b-9b22-6ac81a5f7e8a
2022-07-04 19:37:22 [140474293741120] DEBUG - describeStreamCurlHandler(): DescribeStream API response: {"StreamInfo":{"CreationTime":1.656961857643E9,"DataRetentionInHours":168,"DeviceName":"Kinesis_Video_Device","IngestionConfiguration":null,"KmsKeyId":"arn:aws:kms:us-east-1:MYACCOUNT_NUMBER:alias/aws/kinesisvideo","MediaType":"video/h264","Status":"ACTIVE","StreamARN":"arn:aws:kinesisvideo:us-east-1:MYACCOUNT_NUMBER:stream/MYSTREAM_NAME/1656961857643","StreamName":"MYSTREAM_NAME","Version":"MYVERSION"}}
2022-07-04 19:37:22 [140474293741120] INFO - describeStreamResultEvent(): Describe stream result event.
2022-07-04 19:37:22 [140474293741120] INFO - writeHeaderCallback(): RequestId: 65213fea-917f-4d14-88d6-fb854d3a08cd
2022-07-04 19:37:22 [140474285348416] DEBUG - getStreamingEndpointCurlHandler(): GetStreamingEndpoint API response: {"DataEndpoint":"https://AAAAAAA.kinesisvideo.us-east-1.amazonaws.com"}
2022-07-04 19:37:22 [140474285348416] INFO - getStreamingEndpointResultEvent(): Get streaming endpoint result event.
2022-07-04 19:37:22 [140474285348416] DEBUG - getStreamingTokenHandler invoked
2022-07-04 19:37:22 [140474285348416] DEBUG - Refreshing credentials. Force refreshing: 1 Now time is: 1656963442867748518 Expiration: 18446744073709551615
2022-07-04 19:37:22 [140474285348416] INFO - getStreamingTokenResultEvent(): Get streaming token result event.
2022-07-04 19:37:22 [140474285348416] DEBUG - streamReadyHandler invoked
2022-07-04 19:37:22 [140474285348416] Stream is ready
INFO - kinesisVideoStreamFormatChanged(): Stream format changed.
DEBUG - Dropping frame with flag: 97920:02:08.5 / 99:99:99.
2022-07-04 19:39:31 [140473742124608] INFO - putStreamResultEvent(): Put stream result event. New upload handle 0
INFO - writeHeaderCallback(): RequestId: dc9a7e24-b124-5e42-87a7-3a109b2af291
2022-07-04 19:39:32 [140473750517312] DEBUG - Dropping frame with flag: 1536
DEBUG - postReadCallback(): Pausing CURL read for upload handle: 0
DEBUG - Dropping frame with flag: 15360:02:10.0 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:10.6 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:11.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:12.6 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:13.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:14.6 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:15.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:16.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:17.6 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:18.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:19.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:20.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:21.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:22.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:23.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:24.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:25.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:26.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:27.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:28.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:29.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:30.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:31.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:32.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:33.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:34.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:35.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:36.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:37.5 / 99:99:99.
WARN - curlCompleteSync(): curl perform failed for url https://AAAAAAA.kinesisvideo.us-east-1.amazonaws.com/putMedia with result Timeout was reached: Operation too slow. Less than 30 bytes/sec transferred the last 30 seconds
2022-07-04 19:40:02 [140473750517312] WARN - curlCompleteSync(): HTTP Error 0 : Response: (null)
Request URL: https://AAAAAAA.kinesisvideo.us-east-1.amazonaws.com/putMedia
Request Headers:
Authorization: AWS4-HMAC-SHA256 Credential=MYACCESS_KEY/20220704/us-east-1/kinesisvideo/aws4_request, SignedHeaders=connection;host;transfer-encoding;user-agent;x-amz-date;x-amzn-fragment-acknowledgment-required;x-amzn-fragment-timecode-type;x-amzn-producer-start-timestamp;x-amzn-stream-name, Signature=4c548d1966bfc9d90980b92b71409750a85cd756ed23271
2022-07-04 19:40:02 [140473750517312] DEBUG - putStreamCurlHandler(): Network thread for Kinesis Video stream: MYSTREAM_NAME with upload handle: 0 exited. http status: 0
2022-07-04 19:40:02 [140473750517312] WARN - putStreamCurlHandler(): Stream with streamHandle 94086962655504 uploadHandle 0 has exited without triggering end-of-stream. Service call result: 599
2022-07-04 19:40:02 [140473750517312] INFO - kinesisVideoStreamTerminated(): Stream 0x559253fcb110 terminated upload handle 0 with service call result 599.
2022-07-04 19:40:02 [140473750517312] DEBUG - defaultStreamStateTransitionHook(): Stream state machine retry count: 0
2022-07-04 19:40:02 [140473750517312] DEBUG - defaultStreamStateTransitionHook():
KinesisVideoStream base result is [599]. Executing KVS retry handler of retry strategy type [1]
DEBUG - Dropping frame with flag: 15360:02:39.4 / 99:99:99.
2022-07-04 19:40:02 [140473742124608] DEBUG - defaultStreamStateTransitionHook(): Stream state machine retry count: 1
2022-07-04 19:40:02 [140473742124608] DEBUG - defaultStreamStateTransitionHook():
KinesisVideoStream base result is [599]. Executing KVS retry handler of retry strategy type [1]
2022-07-04 19:40:02 [140473742124608] DEBUG - streamReadyHandler invoked
2022-07-04 19:40:02 [140473742124608] DEBUG - defaultStreamStateTransitionHook(): Stream state machine retry count: 2
2022-07-04 19:40:02 [140473742124608] DEBUG - defaultStreamStateTransitionHook():
KinesisVideoStream base result is [599]. Executing KVS retry handler of retry strategy type [1]
2022-07-04 19:40:02 [140473742124608] INFO - putStreamResultEvent(): Put stream result event. New upload handle 1
DEBUG - Dropping frame with flag: 15360:02:39.6 / 99:99:99.
INFO - writeHeaderCallback(): RequestId: dc2f1583-c74c-3c15-8712-51d03e572d8f
DEBUG - postReadCallback(): Pausing CURL read for upload handle: 1
DEBUG - Dropping frame with flag: 15360:02:41.4 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:41.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:42.6 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:43.6 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:44.6 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:45.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:46.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:47.6 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:48.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:49.6 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:50.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:51.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:52.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:53.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:54.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:55.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:56.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:57.5 / 99:99:99.
DEBUG - Dropping frame with flag: 15360:02:58.5 / 99:99:99.
After receive this output I closed it using CTRL+C.
Anyone knows what can be done to solve this problem?
Thanks in advance.

Solved by replacing /cam/realmonitor?channel=1&subtype=0 by /cam/realmonitor?channel=1&subtype=0 on RTSP, on Linux & is a reserved character.

Related

Spark on Kubernetes: How to improve my performance?

I am running a Spark App over a Kubernetes cluster (at the moment, I do not have permission to resize or rescale this cluster). I think I have mostly 3 important issues that are impacting my performance and I would like to ask to this community if there is something that I could do to make it better. I am going to order them according to the priorities that I set myself.
1- I do not have permissions on K8s cluster, and I think the configuration that I set for my Spark App is not taking effect. It is just a guess because I do not have much experience on K8s. So, I have configured my Spark App like this:
return SparkSession.builder \
.config("spark.executor.instances", "5") \
.config("spark.executor.cores", "4") \
.config("spark.driver.cores", "2") \
.config("spark.kubernetes.executor.request.cores", 3.5) \
.config("spark.executor.memory", "10g") \
.config("spark.driver.memory", "6g") \
.config("spark.sql.broadcastTimeout", "600") \
.config("spark.memory.fraction", 0.2) \
.config("spark.kubernetes.memoryOverheadFactor", 0.2) \
But when the pod is created, I get this from LOG:
'containers': [{'args': ['/bin/bash',
'-c',
'spark-submit --master '
'"k8s://https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_PORT_443_TCP_PORT}" '
'--deploy-mode client --name "${POD_NAME}" '
'--conf "spark.driver.host=${POD_IP}" '
'--conf spark.driver.memory=40g --conf '
'spark.driver.memoryOverhead=0.4 --conf '
'spark.eventLog.dir=s3a://*****-spark/history/ireland '
'--conf spark.eventLog.enabled=true --conf '
Driver's memory is 40gb instead of 6gb, Driver's memoryOverhead is 0.4 instead pf 0.2. So, when I see this I get confused and I am not sure if the number of cores/executor and executor's memory is applied like I configured.
2- The last task of this Spark App writes a Dataframe into a Hive Table (format parquet) on a S3 Bucket.
results.withColumn("make", F.col("name_make_dict")).filter(F.col("name_mod1_dict").isNotNull()) \
.select(*json.loads(parser.get("config", "formatMatched"))) \
.withColumnRenamed("name_mod1_dict", "name_model_dict") \
.repartition(30, "inserted_at_date", "brand", "make") \
.write.insertInto(f"{schema}.cars_features")
And I have checked the LOG:
[2022-04-16 10:32:23,040] {pod_launcher.py:156} INFO - b'22/04/16 10:32:23 INFO CodeGenerator: Code generated in 35.851449 ms\n'
[2022-04-16 10:32:23,097] {pod_launcher.py:156} INFO - b'22/04/16 10:32:23 INFO SparkContext: Starting job: insertInto at NativeMethodAccessorImpl.java:0\n'
[2022-04-16 10:32:23,100] {pod_launcher.py:156} INFO - b'22/04/16 10:32:23 INFO DAGScheduler: Registering RDD 146 (insertInto at NativeMethodAccessorImpl.java:0) as input to shuffle 9\n'
[2022-04-16 10:32:23,100] {pod_launcher.py:156} INFO - b'22/04/16 10:32:23 INFO DAGScheduler: Registering RDD 149 (insertInto at NativeMethodAccessorImpl.java:0) as input to shuffle 10\n'
[2022-04-16 10:32:23,100] {pod_launcher.py:156} INFO - b'22/04/16 10:32:23 INFO DAGScheduler: Got job 18 (insertInto at NativeMethodAccessorImpl.java:0) with 30 output partitions\n'
and it finishes very fast:
[2022-04-16 10:33:02,044] {pod_launcher.py:156} INFO - b'22/04/16 10:33:02 INFO TaskSchedulerImpl: Removed TaskSet 34.0, whose tasks have all completed, from pool \n'
[2022-04-16 10:33:02,048] {pod_launcher.py:156} INFO - b'22/04/16 10:33:02 INFO DAGScheduler: ResultStage 34 (insertInto at NativeMethodAccessorImpl.java:0) finished in 26.634 s\n'
[2022-04-16 10:33:02,048] {pod_launcher.py:156} INFO - b'22/04/16 10:33:02 INFO DAGScheduler: Job 18 finished: insertInto at NativeMethodAccessorImpl.java:0, took 38.948620 s\n'
but then it starts like this and takes like 10 minutes to write files into S3:
[2022-04-16 10:33:05,305] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:33:10,321] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:33:15,337] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:33:20,354] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:33:25,370] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:33:30,383] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:33:35,399] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:33:40,416] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:33:45,432] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:33:50,450] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:33:55,466] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:00,482] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:05,498] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:10,513] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:15,527] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:20,541] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:25,559] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:30,578] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:35,593] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:40,614] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:45,628] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:50,645] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:34:55,662] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:00,679] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:05,690] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:10,705] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:15,722] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:20,740] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:25,757] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:30,775] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:35,788] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:40,799] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:45,817] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:50,834] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:35:55,852] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:00,867] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:05,889] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:10,904] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:15,913] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:20,926] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:25,941] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:30,957] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:33,949] {pod_launcher.py:156} INFO - b'22/04/16 10:36:33 INFO FileFormatWriter: Write Job eefcf8a0-9ba9-4752-a51e-f4bc27e5ffcc committed.\n'
[2022-04-16 10:36:33,954] {pod_launcher.py:156} INFO - b'22/04/16 10:36:33 INFO FileFormatWriter: Finished processing stats for write job eefcf8a0-9ba9-4752-a51e-f4bc27e5ffcc.\n'
[2022-04-16 10:36:35,972] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:40,991] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:46,006] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:51,022] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:36:56,038] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:37:01,057] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:37:06,072] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:37:10,856] {pod_launcher.py:156} INFO - b"22/04/16 10:37:10 INFO FileUtils: Creating directory if it doesn't exist: s3a://*********/local/odyn/cars_pricing_cvt/cars_features/inserted_at_date=2022-04-16/brand=autovit/make=JAGUAR\n"
[2022-04-16 10:37:11,089] {base_job.py:197} DEBUG - [heartbeat]
[2022-04-16 10:37:12,166] {pod_launcher.py:156} INFO - b'22/04/16 10:37:12 INFO SessionState: Could not get hdfsEncryptionShim, it is only applicable to hdfs filesystem.\n'
[2022-04-16 10:37:12,655] {pod_launcher.py:156} INFO - b'22/04/16 10:37:12 INFO Hive: Renaming src: s3a://*********/local/odyn/cars_pricing_cvt/cars_features/.hive-staging_hive_2022-04-16_10-26-50_235_365683970089316725-1/-ext-10000/inserted_at_date=2022-04-16/brand=autovit/make=JAGUAR/part-00013-87f53444-2a15-4a4c-b7bc-0b3304ba9bb7.c000, dest: s3a://*********/local/odyn/cars_pricing_cvt/cars_features/inserted_at_date=2022-04-16/brand=autovit/make=JAGUAR/part-00013-87f53444-2a15-4a4c-b7bc-0b3304ba9bb7.c000, Status:true\n'
[2022-04-16 10:37:12,967] {pod_launcher.py:156} INFO - b'22/04/16 10:37:12 INFO Hive: New loading path = s3a://*********/local/odyn/cars_pricing_cvt/cars_features/.hive-staging_hive_2022-04-16_10-26-50_235_365683970089316725-1/-ext-10000/inserted_at_date=2022-04-16/brand=autovit/make=JAGUAR with partSpec {inserted_at_date=2022-04-16, brand=autovit, make=JAGUAR}\n'
[2022-04-16 10:37:13,045] {pod_launcher.py:156} INFO - b"22/04/16 10:37:13 INFO FileUtils: Creating directory if it doesn't exist: s3a://*********/local/odyn/cars_pricing_cvt/cars_features/inserted_at_date=2022-04-16/brand=gratka/make=LANCIA\n"
[2022-04-16 10:37:14,065] {pod_launcher.py:156} INFO - b'22/04/16 10:37:14 INFO SessionState: Could not get hdfsEncryptionShim, it is only applicable to hdfs filesystem.\n'
[2022-04-16 10:37:14,576] {pod_launcher.py:156} INFO - b'22/04/16 10:37:14 INFO Hive: Renaming src: s3a://*********/local/odyn/cars_pricing_cvt/cars_features/.hive-staging_hive_2022-04-16_10-26-50_235_365683970089316725-1/-ext-10000/inserted_at_date=2022-04-16/brand=gratka/make=LANCIA/part-00009-87f53444-2a15-4a4c-b7bc-0b3304ba9bb7.c000, dest: s3a://*********/local/odyn/cars_pricing_cvt/cars_features/inserted_at_date=2022-04-16/brand=gratka/make=LANCIA/part-00009-87f53444-2a15-4a4c-b7bc-0b3304ba9bb7.c000, Status:true\n'
[2022-04-16 10:37:14,803] {pod_launcher.py:156} INFO - b'22/04/16 10:37:14 INFO Hive: New loading path = s3a://*********/local/odyn/cars_pricing_cvt/cars_features/.hive-staging_hive_2022-04-16_10-26-50_235_365683970089316725-1/-ext-10000/inserted_at_date=2022-04-16/brand=gratka/make=LANCIA with partSpec {inserted_at_date=2022-04-16, brand=gratka, make=LANCIA}\n'
[2022-04-16 10:37:14,870] {pod_launcher.py:156} INFO - b"22/04/16 10:37:14 INFO FileUtils: Creating directory if it doesn't exist: s3a://*********/local/odyn/cars_pricing_cvt/cars_features/inserted_at_date=2022-04-16/brand=otomoto/make=MG\n"
[2022-04-16 10:37:15,808] {pod_launcher.py:156} INFO - b'22/04/16 10:37:15 INFO SessionState: Could not get hdfsEncryptionShim, it is only applicable to hdfs filesystem.\n'
[2022-04-16 10:37:16,106] {base_job.py:197} DEBUG - [heartbeat]
I have tried with repartition(30, ...) as you can see above and with coalesce(1) but the write speed is the same. I do not know why this is happening and if it is an expected behaviour. I have checked this:
Extremely slow S3 write times from EMR/ Spark
Spark s3 write (s3 vs s3a connectors)
Spark Write to S3 Storage Option
3- When I am reading some tables from Hive on S3 like this (for one day is 40.4 MB->2467 objects):
sparkSession \
.sql(
f"""select * from {schema}.f_car_info where inserted_at_date = to_date('{date}', "yyyy-MM-dd")""") \
.select(F.col("id_ad"), F.col("name_make_ad"), F.col("name_make_dict"), F.col("name_model_ad"),
F.col("name_version_ad"), F.col("name_generation_ad"), F.col("prod_year_ad"), F.col("brand"))
I get this from LOG:
[2022-04-16 10:25:38,271] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO InMemoryFileIndex: Listing leaf files and directories in parallel under 352 paths. The first several paths are: s3a://*********/local/odyn/cars_pricing_cvt/f_car_info/inserted_at_date=2022-04-16/brand=autovit/name_make_dict=ABARTH, s3a://*********/local/odyn/cars_pricing_cvt/f_car_info/inserted_at_date=2022-04-16/brand=autovit/name_make_dict=ACURA, s3a://*********/local/odyn/cars_pricing_cvt/f_car_info/inserted_at_date=2022-04-16/brand=autovit/name_make_dict=ALFA ROMEO, s3a://*********/local/odyn/cars_pricing_cvt/f_car_info/inserted_at_date=2022-04-16/brand=autovit/name_make_dict=ASTON MARTIN, s3a://*********/local/odyn/cars_pricing_cvt/f_car_info/inserted_at_date=2022-04-16/brand=autovit/name_make_dict=AUDI, s3a://*********/local/odyn/cars_pricing_cvt/f_car_info/inserted_at_date=2022-04-16/brand=autovit/name_make_dict=AUSTIN, s3a://*********/local/odyn/cars_pricing_cvt/f_car_info/inserted_at_date=2022-04-16/brand=autovit/name_make_dict=BENTLEY, s3a://*********/local/odyn/cars_pricing_cvt/f_car_info/inserted_at_date=2022-04-16/brand=autovit/name_make_dict=BMW, s3a://*********/local/odyn/cars_pricing_cvt/f_car_info/inserted_at_date=2022-04-16/brand=autovit/name_make_dict=BUICK, s3a://*********/local/odyn/cars_pricing_cvt/f_car_info/inserted_at_date=2022-04-16/brand=autovit/name_make_dict=CADILLAC.\n'
[2022-04-16 10:25:38,564] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO SparkContext: Starting job: persist at NativeMethodAccessorImpl.java:0\n'
[2022-04-16 10:25:38,584] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO DAGScheduler: Got job 0 (persist at NativeMethodAccessorImpl.java:0) with 352 output partitions\n'
[2022-04-16 10:25:38,587] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO DAGScheduler: Final stage: ResultStage 0 (persist at NativeMethodAccessorImpl.java:0)\n'
[2022-04-16 10:25:38,587] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO DAGScheduler: Parents of final stage: List()\n'
[2022-04-16 10:25:38,587] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO DAGScheduler: Missing parents: List()\n'
[2022-04-16 10:25:38,593] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at persist at NativeMethodAccessorImpl.java:0), which has no missing parents\n'
[2022-04-16 10:25:38,689] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 88.3 KB, free 7.7 GB)\n'
[2022-04-16 10:25:38,721] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 31.9 KB, free 7.7 GB)\n'
[2022-04-16 10:25:38,724] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 100.126.80.1:34339 (size: 31.9 KB, free: 7.7 GB)\n'
[2022-04-16 10:25:38,727] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1184\n'
[2022-04-16 10:25:38,749] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO DAGScheduler: Submitting 352 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at persist at NativeMethodAccessorImpl.java:0) (first 15 tasks are for partitions Vector(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14))\n'
[2022-04-16 10:25:38,753] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO TaskSchedulerImpl: Adding task set 0.0 with 352 tasks\n'
[2022-04-16 10:25:38,794] {pod_launcher.py:156} INFO - b'22/04/16 10:25:38 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, 100.126.159.37, executor 1, partition 0, PROCESS_LOCAL, 7494 bytes)\n'
[2022-04-16 10:25:45,006] {pod_launcher.py:156} INFO - b'22/04/16 10:25:45 INFO TaskSetManager: Finished task 350.0 in stage 0.0 (TID 350) in 107 ms on 100.126.111.179 (executor 5) (352/352)\n'
[2022-04-16 10:25:45,009] {pod_launcher.py:156} INFO - b'22/04/16 10:25:45 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool \n'
[2022-04-16 10:25:45,009] {pod_launcher.py:156} INFO - b'22/04/16 10:25:45 INFO DAGScheduler: ResultStage 0 (persist at NativeMethodAccessorImpl.java:0) finished in 6.371 s\n'
[2022-04-16 10:25:45,015] {pod_launcher.py:156} INFO - b'22/04/16 10:25:45 INFO DAGScheduler: Job 0 finished: persist at NativeMethodAccessorImpl.java:0, took 6.451078 s\n'
[2022-04-16 10:25:45,121] {pod_launcher.py:156} INFO - b'22/04/16 10:25:45 INFO PrunedInMemoryFileIndex: It took 6916 ms to list leaf files for 352 paths.\n'
Yes, they are 352! tasks. I imagine when I have much more files in a table in future. it will impact the read time. I have checked this:
Why so many tasks in my spark job? Getting 200 Tasks By Default
and used sqlContext.setConf("spark.sql.shuffle.partitions", "15”) but no changes.
Could you please give me some ideas on these 3 issues?
Many thanks!!!

Hey looks like you are working on some cool stuff! I can't comment on #2 and #3 but for #1 I can probably shed some light. I haven't really used Spark.
My guess is for Spark specifying fields at runtime override whatever you are trying to do with SparkSession.builder in your code.
Those overriding runtime args can be at either the container image level OR the kubernetes pod configuration level. Since you did not share that info it is hard for me to try and figure out which one is your problem, but my guess is it at the pod definition level, these can override the container image settings.
Kubernetes Pod Definition Level
For example in a pod definition (or in a deployment) check this out:
https://kubernetes.io/docs/tasks/inject-data-application/define-command-argument-container/
Example Kubernetes Pod Definition
apiVersion: v1
kind: Pod
metadata:
name: Example
spec:
containers:
- name: <container name>
image: <your image>
command:
- '/bin/bash'
args:
- '-c'
- 'spark-submit --master '
- '"k8s://https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_PORT_443_TCP_PORT}" '
- '--deploy-mode client --name "${POD_NAME}" '
- '--conf "spark.driver.host=${POD_IP}" '
- '--conf spark.driver.memory=40g --conf '
- 'spark.driver.memoryOverhead=0.4 --conf '
- 'spark.eventLog.dir=s3a://*****-spark/history/ireland '
- '--conf spark.eventLog.enabled=true --conf '
You could then change the args section of this yaml pod definition to be what you want. I hope this helps you out! or at least points you in the right direction.

Disable DEBUG logs in gstreamer?

I'm running gstreamer in docker and GST_DEBUG can be used to turn on additional debugging logs, but even if I set GST_DEBUG=2 or GST_DEBUG=3, then several DEBUG logs still happen. Here's an example:
2021-10-08 14:04:55 [139759980640000] DEBUG - postReadCallback(): Wrote 65524 bytes to Kinesis Video. Upload stream handle: 0
2021-10-08 14:04:55 [139759980640000] DEBUG - postReadCallback(): Wrote 65524 bytes to Kinesis Video. Upload stream handle: 0
2021-10-08 14:04:55 [139759980640000] DEBUG - postReadCallback(): Wrote 65524 bytes to Kinesis Video. Upload stream handle: 0
2021-10-08 14:04:55 [139759980640000] DEBUG - postReadCallback(): Wrote 31003 bytes to Kinesis Video. Upload stream handle: 0
2021-10-08 14:04:55 [139759980640000] DEBUG - postReadCallback(): Pausing CURL read for upload handle: 0
2021-10-08 14:04:55 [139759980640000] DEBUG - postWriteCallback(): Curl post body write function for stream with handle: DaveTest and upload handle: 0 returned: {"EventType":"RECEIVED","FragmentTimecode":1633701893567,"FragmentNumber":"91343852333181580945486776886085710683522911738"}
2021-10-08 14:04:55 [139759980640000] DEBUG - fragmentAckReceivedHandler invoked
2021-10-08 14:04:55 [139759980640000] DEBUG - postReadCallback(): Wrote 20153 bytes to Kinesis Video. Upload stream handle: 0
2021-10-08 14:04:55 [139759980640000] DEBUG - postReadCallback(): Pausing CURL read for upload handle: 0
2021-10-08 14:04:55 [139759980640000] DEBUG - postWriteCallback(): Curl post body write function for stream with handle: DaveTest and upload handle: 0 returned: {"EventType":"BUFFERING","FragmentTimecode":1633701895543,"FragmentNumber":"91343852333181580950438537043227232143278319293"}
2021-10-08 14:04:55 [139759980640000] DEBUG - fragmentAckReceivedHandler invoked
2021-10-08 14:04:55 [139759980640000] DEBUG - postWriteCallback(): Curl post body write function for stream with handle: DaveTest and upload handle: 0 returned: {"EventType":"PERSISTED","FragmentTimecode":1633701893567,"FragmentNumber":"91343852333181580945486776886085710683522911738"}
2021-10-08 14:04:55 [139759980640000] DEBUG - fragmentAckReceivedHandler invoked
2021-10-08 14:04:55 [139759980640000] DEBUG - postReadCallback(): Wrote 9598 bytes to Kinesis Video. Upload stream handle: 0
2021-10-08 14:04:55 [139759980640000] DEBUG - postReadCallback(): Pausing CURL read for upload handle: 0
2021-10-08 14:04:55 [139759980640000] DEBUG - Kinesis Video client and stream metrics
>> Overall storage byte size: 536870912
>> Available storage byte size: 536261448
>> Allocated storage byte size: 609464
>> Total view allocation byte size: 144080
>> Total streams frame rate (fps): 1175
>> Total streams transfer rate (bps): 29187312 (28503 Kbps)
>> Current view duration (ms): 433
>> Overall view duration (ms): 1999
>> Current view byte size: 283686
>> Overall view byte size: 606536
>> Current frame rate (fps): 1175.58
>> Current transfer rate (bps): 29187312 (28503 Kbps)
How can I turn these off?

The issue was the kvssink plugin and that it has it's own logging setup. It can be set using a config file and log-config with the path to that file (see here)

How exactly do you set this variable? I was using it multiple times by exporting it: export GST_DEBUG=3 and it worked as it should. See this link: https://gstreamer.freedesktop.org/documentation/gstreamer/gstinfo.html?gi-language=c for info how to use it programatically.

while using openapi Braid for generating api for my corda project its showing some error in the last steep

Downloads\bootcamp-openapi-master\bootcamp-openapi-master\build\nodes>openapi-generator generate -i http://localhost:10200/swagger.json -g javascript -o ./code-gen --
api-package io.generated.api --model-package io.generated.model
[main] ERROR io.swagger.v3.parser.util.RemoteUrl - unable to read
java.net.SocketException: Unexpected end of file from server
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection$10.run(Unknown Source)
I am trying to work on the corda openapi generation project but its showing me above error in the last step. The project I am trying to work is
https://blog.b9lab.com/cordacon-2019-highlights-braid-server-and-openapi-generator-for-corda-flows-api-s-d24179ccb27c
and similar
https://github.com/corda/openapi-sample
But the problem is that I have done same as show in the link but my last step that is generating api is not executing and throwing the above mentioned errors. And one more thing is that my one step is showing different output compared to the one shown in the Git.
MY OUTPUT
10:03:39.328 [main] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - Starting Braid on port: 10200
10:03:39.718 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupRouter starting...
10:03:39.781 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.DocsHandlerFactory - activating OpenAPI V3
10:03:39.874 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - swagger json bound to https://localhost:10200/swagger.json
10:03:39.890 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - Swagger UI bound to https://localhost:10200/
10:03:42.999 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - registering: /cordapps/bootcamp-openapi-master/flows/bootcamp.GetAllTokensFlow
10:03:43.062 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - registering: /cordapps/bootcamp-openapi-master/flows/bootcamp.TokenIssueFlowInitiator
10:03:43.108 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - REST end point bound to https://localhost:10200/api/rest
10:03:43.108 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupRouter complete -- 3406 msec
10:03:43.889 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupWebserver complete -- 781 msec
10:03:43.889 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - Braid server started on
10:03:43.905 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - Braid service mounted on https://localhost:10200/api/
10:03:43.905 [vert.x-eventloop-thread-1] INFO io.bluebank.braid.corda.BraidServer - Braid server started successfully on 10200
Their OUTPUT
10:03:39.328 [main] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - Starting Braid on port: 10200
10:03:39.718 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupRouter starting...
10:03:39.781 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.DocsHandlerFactory - activating OpenAPI V2
10:03:39.874 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - swagger json bound to http://localhost:10200/swagger.json
10:03:39.890 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - Swagger UI bound to http://localhost:10200/
10:03:42.999 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - registering: /cordapps/bootcamp-openapi-master/flows/bootcamp.GetAllTokensFlow
10:03:43.062 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.server.BraidCordaStandaloneServer - registering: /cordapps/bootcamp-openapi-master/flows/bootcamp.TokenIssueFlowInitiator
10:03:43.108 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.rest.RestMounter - REST end point bound to http://localhost:10200/api/rest
10:03:43.108 [braid-startup-threadpool-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupRouter complete -- 3406 msec
10:03:43.889 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - BraidVerticle.setupWebserver complete -- 781 msec
10:03:43.889 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - Braid server started on
10:03:43.905 [vert.x-eventloop-thread-0] INFO io.bluebank.braid.corda.BraidVerticle - Braid service mounted on https://localhost:10200/api/
10:03:43.905 [vert.x-eventloop-thread-1] INFO io.bluebank.braid.corda.BraidServer - Braid server started successfully on 10200
The difference is that their don't contains https but I don't think this could raise any error.

In your Braid server output it says activating OpenAPI V2 meaning your server is generating the API's with OpenAPI Specification version 2, while your openapi-generator is throwing error ERROR io.swagger.v3.parser.util.RemoteUrl meaning it's looking for v3 (not v2) that's why it's unable to parse the file.
In my Medium article (and R3 tutorial) we both use v3.
From my article (notice the 3 after 10200:
localhost:10004 user1 test 10200 3 “/home/your-user/path-to-project/bootcamp-openapi/build/nodes/PartyA/cordapps”
From R3's article (notice how we both use openapi version 3):
Now, find the down triangle to open up the run configuration. At the Program arguments, paste in:
localhost:10004 user1 test 10200 3
"/YOUR-PATH-TO-THIS-FOLDER/bootcamp-cordapp/build/nodes/PartyA/cordapps"
RPC connection address: localhost:10004
node login username: user1
node login password: test
Your desired expose port: 10200
openapi version: 3
Cordapp folder to pick up the jar: "YOUR-PATH-TO-THIS-FOLDER/bootcamp-cordapp/build/nodes/PartyA/cordapps"
So I'm not sure why you decided to use version 2.

Slow upload speed in aws deploy push command

I am trying to use AWS CodeDeploy. I use aws deploy push --debug command. The file to be uploaded is around 250 KB. But upload doesn't finish. Following is the logs displayed.
2017-10-27 11:11:40,601 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
PUT
/frontend-deployer/business-services-0.0.1-SNAPSHOT-classes.jar
partNumber=39&uploadId=.olvaJkxreDZf1ObaHCMtHmkQ5DFE.uZ9Om0sxZB08YG3tqRWBxmGLTFWSYQaj9mHl26LPJk..Stv_vPB5NMaV.zAqsYX6fZz_S3.uN5J4FlxHZFXoeTkMiBSYQB2C.g
content-md5:EDXgvJ8Tt5tHYZ6Nkh7epg==
host:s3.us-east-2.amazonaws.com
x-amz-content-sha256:UNSIGNED-PAYLOAD
x-amz-date:20171027T081140Z
content-md5;host;x-amz-content-sha256;x-amz-date
UNSIGNED-PAYLOAD
...
2017-10-27 11:12:12,035 - MainThread - botocore.endpoint - DEBUG - Sending http request: <PreparedRequest [PUT]>
2017-10-27 11:12:12,035 - MainThread - botocore.awsrequest - DEBUG - Waiting for 100 Continue response.
2017-10-27 11:12:12,189 - MainThread - botocore.awsrequest - DEBUG - 100 Continue response seen, now sending request body.
Even though the file is fairly small (250 KB), upload doesn't finish.
On the other hand, upload via aws s3 cp command lasts 1 second.
How can I increase the upload speed in aws deploy push command?

Dynamic-DynamoDB not scaling down

I've been testing Dynamic-DynamoDB on a single table, and it does not want to scale down the provisioning. Can anybody see what I've done wrong?
Here is the log from one cycle. I let it run overnight... This message sequence goes on and on.
2014-03-31 12:58:51,617 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned read units: 25
2014-03-31 12:58:51,683 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned read units: 25
2014-03-31 12:58:51,683 - dynamic-dynamodb - INFO - myTestTable - Consumed read units: 0%
2014-03-31 12:58:51,702 - dynamic-dynamodb - INFO - myTestTable - Read throttle count: 0
2014-03-31 12:58:51,719 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned write units: 100
2014-03-31 12:58:51,779 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned write units: 100
2014-03-31 12:58:51,779 - dynamic-dynamodb - INFO - myTestTable - Consumed write units: 0%
2014-03-31 12:58:51,806 - dynamic-dynamodb - INFO - myTestTable - Write throttle count: 0
2014-03-31 12:58:51,806 - dynamic-dynamodb - INFO - myTestTable - No need to change provisioning
And, here is the configuration for the table:
[table: myTestTable]
reads-upper-threshold: 90
reads-lower-threshold: 30
increase-reads-with: 50
decrease-reads-with: 50
increase-reads-unit: percent
decrease-reads-unit: percent
min-provisioned-reads: 5
max-provisioned-reads: 25
writes-upper-threshold: 90
writes-lower-threshold: 30
increase-writes-with: 50
decrease-writes-with: 50
increase-writes-unit: percent
decrease-writes-unit: percent
min-provisioned-writes: 5
max-provisioned-writes: 100
#maintenance-windows: 22:00-23:59,00:00-06:00
sns-message-types: scale-up, scale-down
allow-scaling-down-reads-on-0-percent: true
allow-scaling-down-writes-on-0-percent: true
#always-decrease-rw-together: true
The only thing I have not tried yet is setting the maintenance window times. I assume that when they are not set, it will do updates at any time.
Update. I found that attempt at a scale up event that happened overnight during a test. Obviously, I'm at the max already so it didn't do anything, as I would expect. I just don't understand why the scale down is not working.
2014-03-30 23:27:16,789 - dynamic-dynamodb - INFO - myTestTable - Consumed read units: 0%
2014-03-30 23:27:16,808 - dynamic-dynamodb - INFO - myTestTable - Read throttle count: 0
2014-03-30 23:27:16,827 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned write units: 100
2014-03-30 23:27:16,880 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned write units: 100
2014-03-30 23:27:16,880 - dynamic-dynamodb - INFO - myTestTable - Consumed write units: 117%
2014-03-30 23:27:16,901 - dynamic-dynamodb - INFO - myTestTable - Write throttle count: 0
2014-03-30 23:27:16,902 - dynamic-dynamodb - INFO - myTestTable - Reached provisioned writes max limit: 100
2014-03-30 23:27:16,902 - dynamic-dynamodb - INFO - myTestTable - No need to change provisioning
2014-03-30 23:27:17,104 - dynamic-dynamodb - DEBUG - Sleeping 300 seconds until next check
2014-03-30 23:32:17,277 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned read units: 25
2014-03-30 23:32:17,353 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned read units: 25
2014-03-30 23:32:17,354 - dynamic-dynamodb - INFO - myTestTable - Consumed read units: 0%
2014-03-30 23:32:17,375 - dynamic-dynamodb - INFO - myTestTable - Read throttle count: 0
2014-03-30 23:32:17,433 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned write units: 100
2014-03-30 23:32:17,481 - dynamic-dynamodb - DEBUG - myTestTable - Currently provisioned write units: 100
2014-03-30 23:32:17,481 - dynamic-dynamodb - INFO - myTestTable - Consumed write units: 151%
2014-03-30 23:32:17,501 - dynamic-dynamodb - INFO - myTestTable - Write throttle count: 0
2014-03-30 23:32:17,501 - dynamic-dynamodb - INFO - myTestTable - Reached provisioned writes max limit: 100
2014-03-30 23:32:17,502 - dynamic-dynamodb - INFO - myTestTable - No need to change provisioning
2014-03-30 23:32:17,695 - dynamic-dynamodb - DEBUG - Sleeping 300 seconds until next check

Seems to be a bug in the version.
https://github.com/sebdah/dynamic-dynamodb/issues/142
I will stop posting these here and use GitHub if I have any more issues. Sebdah answers very quickly.

In this case it may have been a bug, but if this still happens to some people, there is a limit of how many times per 24 hour period you can scale down a DynamoDB table. Once you reach that limit, DynamoDB will not let you scale down more times and will throw an error if you try to request to do it again.

We Keep Coding

c++ django amazon-web-services regex python-2.7 google-cloud-platform list unit-testing opengl ember.js