Ansible parsing lists inside dictionary - list

- name: Use Collections
hosts: all
connection: local
gather_facts: no
vars:
ansible_python_interpreter: /usr/local/bin/python3.9
tasks:
- name: Collect BIG-IP information
bigip_device_info:
gather_subset:
- virtual-servers
provider:
server: y.y.y.y
user: abc
password: abc1
validate_certs: False
delegate_to: localhost
register: output
- name: debug
debug: msg={{output.ansible_facts| json_query(jmesquery)}}
vars:
jmesquery: "[*].{Satus: item.availability_status, Destinationaddress: item.destination_address, Destination: item.destination}"
i am getting below output, empty message
ansible-playbook -i inv bigipfacts.yml
PLAY [Use Collections] ***************************************************************************************************************************************************************************
TASK [Collect BIG-IP information] ****************************************************************************************************************************************************************
ok: [y.y.y.y]
TASK [debug] *************************************************************************************************************************************************************************************
ok: [y.y.y.y] => {
"msg": ""
}
PLAY RECAP ***************************************************************************************************************************************************************************************
y.y.y.y : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Required/wanted output- in future i want parse more from bottom output. but for now these three values.
"availability_status": "available",
"destination": "/1.1.1..1:8080",
"destination_address": "2.2.2.2",
"availability_status": "available",
"destination": "/3.3.3.3.3%22:8080",
"destination_address": "4.4.4.4",
output.ansible_facts has below info, i want to parse below output and extract only few values in a list like shown above
TASK [debug] *************************************************************************************************************************************************************************************
ok: [x.x.x.x] => {
"msg": {
"ansible_net_queried": true,
"ansible_net_virtual_servers": [
{
"auto_lasthop": "default",
"availability_status": "available",
"client_side_bits_in": 0,
"client_side_bits_out": 0,
"client_side_current_connections": 0,
"client_side_evicted_connections": 0,
"client_side_max_connections": 0,
"client_side_pkts_in": 0,
"client_side_pkts_out": 0,
"client_side_slow_killed": 0,
"client_side_total_connections": 0,
"cmp_enabled": "yes",
"cmp_mode": "all-cpus",
"connection_limit": 0,
"connection_mirror_enabled": "no",
"cpu_usage_ratio_last_1_min": 0,
"cpu_usage_ratio_last_5_min": 0,
"cpu_usage_ratio_last_5_sec": 0,
"current_syn_cache": 0,
"default_pool": "/POOL01",
"destination": "/1.1.1..1:8080",
"destination_address": "2.2.2.2",
"destination_port": 8080,
"enabled": "no",
"ephemeral_bits_in": 0,
"ephemeral_bits_out": 0,
"ephemeral_current_connections": 0,
"ephemeral_evicted_connections": 0,
"ephemeral_max_connections": 0,
"ephemeral_pkts_in": 0,
"ephemeral_pkts_out": 0,
"ephemeral_slow_killed": 0,
"ephemeral_total_connections": 0,
"full_path": "/p1/VS01",
"gtm_score": 0,
"hardware_syn_cookie_instances": 0,
"max_conn_duration": 0,
"mean_conn_duration": 0,
"min_conn_duration": 0,
"name": "vs01",
"nat64_enabled": "no",
"profiles": [
{
"context": "all",
"full_path": "/Common/http",
"name": "http"
},
{
"context": "all",
"full_path": "/Common/oneconnect",
"name": "oneconnect"
},
{
"context": "all",
"full_path": "/Common/tcp",
"name": "tcp"
}
],
"protocol": "tcp",
"rate_limit": -1,
"rate_limit_destination_mask": 0,
"rate_limit_mode": "object",
"rate_limit_source_mask": 0,
"snat_type": "automap",
"software_syn_cookie_instances": 0,
"source_address": "0.0.0.0%22/0",
"source_port_behavior": "preserve",
"status_reason": "The virtual server is available",
"syn_cache_overflow": 0,
"syn_cookies_status": "not-activated",
"total_hardware_accepted_syn_cookies": 0,
"total_hardware_syn_cookies": 0,
"total_requests": 0,
"total_software_accepted_syn_cookies": 0,
"total_software_rejected_syn_cookies": 0,
"total_software_syn_cookies": 0,
"translate_address": "yes",
"translate_port": "yes",
"type": "standard",
"vlans": [
"/Common/vlans"
]
},
{
"auto_lasthop": "default",
"availability_status": "available",
"client_side_bits_in": 0,
"client_side_bits_out": 0,
"client_side_current_connections": 0,
"client_side_evicted_connections": 0,
"client_side_max_connections": 0,
"client_side_pkts_in": 0,
"client_side_pkts_out": 0,
"client_side_slow_killed": 0,
"client_side_total_connections": 0,
"cmp_enabled": "yes",
"cmp_mode": "all-cpus",
"connection_limit": 0,
"connection_mirror_enabled": "no",
"cpu_usage_ratio_last_1_min": 0,
"cpu_usage_ratio_last_5_min": 0,
"cpu_usage_ratio_last_5_sec": 0,
"current_syn_cache": 0,
"default_pool": "/POOL01",
"destination": "/3.3.3.3.3%22:8080",
"destination_address": "4.4.4.4",
"destination_port": 8080,
"enabled": "no",
"ephemeral_bits_in": 0,
"ephemeral_bits_out": 0,
"ephemeral_current_connections": 0,
"ephemeral_evicted_connections": 0,
"ephemeral_max_connections": 0,
"ephemeral_pkts_in": 0,
"ephemeral_pkts_out": 0,
"ephemeral_slow_killed": 0,
"ephemeral_total_connections": 0,
"full_path": "/p1/VS01",
"gtm_score": 0,
"hardware_syn_cookie_instances": 0,
"max_conn_duration": 0,
"mean_conn_duration": 0,
"min_conn_duration": 0,
"name": "vs02",
"nat64_enabled": "no",
"profiles": [
{
"context": "all",
"full_path": "/Common/http",
"name": "http"
},
{
"context": "all",
"full_path": "/Common/oneconnect",
"name": "oneconnect"
},
{
"context": "all",
"full_path": "/Common/tcp",
"name": "tcp"
}
],
"protocol": "tcp",
"rate_limit": -1,
"rate_limit_destination_mask": 0,
"rate_limit_mode": "object",
"rate_limit_source_mask": 0,
"snat_type": "automap",
"software_syn_cookie_instances": 0,
"source_address": "0.0.0.0%22/0",
"source_port_behavior": "preserve",
"status_reason": "The virtual server is available",
"syn_cache_overflow": 0,
"syn_cookies_status": "not-activated",
"total_hardware_accepted_syn_cookies": 0,
"total_hardware_syn_cookies": 0,
"total_requests": 0,
"total_software_accepted_syn_cookies": 0,
"total_software_rejected_syn_cookies": 0,
"total_software_syn_cookies": 0,
"translate_address": "yes",
"translate_port": "yes",
"type": "standard",
"vlans": [
"/Common/vlan"
]
}
]
}
}

You omitted ansible_net_virtual_servers entirely, and appear to have conflated with_items: (which sets an item name in jinja2) with JMESPath which has no such implicit name creation
- debug:
msg: >-
{{ output.ansible_facts.ansible_net_virtual_servers
| json_query(jmesquery) }}
vars:
jmesquery: "[*].{Satus: availability_status, Destinationaddress: destination_address, Destination: destination}"
produces
ok: [localhost] => {
"msg": [
{
"Destination": "/1.1.1..1:8080",
"Destinationaddress": "2.2.2.2",
"Satus": "available"
},
{
"Destination": "/3.3.3.3.3%22:8080",
"Destinationaddress": "4.4.4.4",
"Satus": "available"
}
]
}

Related

Private Ethereum Node is unable to add peers

My genesis file (used same for both nodes)
"config": {
"chainId": 42,
"homesteadBlock": 0,
"eip150Block": 0,
"eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"eip155Block": 0,
"eip158Block": 0,
"byzantiumBlock": 0,
"constantinopleBlock": 0,
"petersburgBlock": 0,
"istanbulBlock": 0,
"ethash": {}
},
"nonce": "0x0",
"timestamp": "0x62c17562",
"extraData": "0x0000000000000000000000000000000000000000000000000000000000000000",
"gasLimit": "0x47b760",
"difficulty": "0x80000",
"mixHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
"coinbase": "0x0000000000000000000000000000000000000000",
"alloc": {
"0000000000000000000000000000000000000000": {
"balance": "0x1"
},
"0000000000000000000000000000000000000001": {
"balance": "0x1"
},
"0000000000000000000000000000000000000002": {
"balance": "0x1"
},
"0000000000000000000000000000000000000003": {
"balance": "0x1"
},
"0000000000000000000000000000000000000004": {
"balance": "0x1"
},
My .sh file for node 1
geth --identity "miner1" --networkid 42 --mine --datadir "~/RPI/miner1" --nodiscover --http --http.port "8042" --port "30303" --http.corsdomain "*" --nat "any" --http.api db,eth,web,web3,private,miner,admin,txpool,debug,personal,net --allow-insecure-unlock --unlock 0 --password ~/RPI/password.sec --ipcpath "~/.ethereum/geth.ipc"
for node 2:
geth --identity "miner2" --networkid 42 --datadir "~/RPI/miner2" --mine --nodiscover --http --http.port "8043" --port "30304" --http.corsdomain "*" --nat "any" --http.api db,eth,web,web3,private,miner,admin,txpool,debug,personal,net --allow-insecure-unlock --unlock 0 --password ~/RPI/password.sec --ipcpath "~/.ethereum/geth.ipc"
I have tried using admin.addPeer but always returns [] tried another method by adding static-nodes.json file but didn't work either.
content of my static-nodes.json:
[
“enode://f9d48547c373fd61124d110791168572382298979d18ab5d6314cd399538ef77b8ff826b870a3474ad351f175b6d4d6f03c09ab4382b4d075e6a3c7fdeee58aa#35.247.47.68:30303?discport=0”,
“enode://991f8279ae27e665151621a9940b42e208bb7cda0108c3cda219059bed28fbb7ec869b04db84d173fbb1cecdce5fef222d010f02a90238e7d8a53b163f6b0a2a#34.105.65.4:30304?discport=0”
]
One thing that I noticed which was very peculiar was mode info:
id: "1fdd33dee80b18a109b51e083ab05fe70d98a2c80d55da9c8679b412a438e180",
ip: "127.0.0.1",
listenAddr: "[::]:30303",
name: "Geth/miner1/v1.10.20-stable-8f2416a8/linux-amd64/go1.18.1",
ports: {
discovery: 0,
listener: 30303
},
protocols: {
eth: {
config: {
byzantiumBlock: 0,
chainId: 42,
constantinopleBlock: 0,
eip150Block: 0,
eip150Hash: "0x0000000000000000000000000000000000000000000000000000000000000000",
eip155Block: 0,
eip158Block: 0,
ethash: {},
homesteadBlock: 0,
istanbulBlock: 0,
petersburgBlock: 0
},
difficulty: 147322604,
genesis: "0xd3ad0f15c44cf0543e7cccce24ce94024026b14a547428f1df6d9d89ab1d6fe7",
head: "0x82714efde94e31b659b232a62a88beb9be1d1cc9a8846f7bde6781276b35ec89",
network: 42
},
snap: {}
}
For some reason in both nodes my discovery ports is set to 0 I feel like that could be the issue but have not been able to find the way to change it. I was under the impression when you set ports both discovery and listener are set to that value.
If that is not the issue then I have no idea what it could be.

Set segment duration with media convert

With AWS media convert everytime I convert an mp4 (6 second video) to a dash, the segment duration is about 30 seconds however I'd much appreciate if it were 1 second or less:
<SegmentTemplate timescale="90000" duration="324000" startNumber="1"/>
<Representation id="1" width="1280" height="720" bandwidth="72000000" codecs="avc1.4d4032">
<SegmentTemplate media="5f8283b60a3ac3640191892_$Number%09d$.mp4" initialization="5f8283b60a3ac3640191892init.mp4" duration="324000" startNumber="1"/>
</Representation>
<Representation id="2" width="1920" height="1080" bandwidth="16200000" codecs="avc1.4d4029">
<SegmentTemplate media="5f8283b60a3ac3640191891_$Number%09d$.mp4" initialization="5f8283b60a3ac3640191891init.mp4" duration="324000" startNumber="1"/>
</Representation>
</AdaptationSet>
I've tried messing with the job settings, but nothing I did seemed to work. Sometimes I got it down to 10 seconds, but that still isn't great, anyway here are the settings I used:
$jobSetting = [
"OutputGroups"=> [
[
"CustomName"=> "nicenice",
"Name"=> "DASH ISO",
"Outputs"=> [
[
"ContainerSettings"=> [
"Container"=> "MPD"
],
"VideoDescription"=> [
"Width"=> 1920,
"ScalingBehavior"=> "DEFAULT",
"Height"=> 1080,
"TimecodeInsertion"=> "DISABLED",
"AntiAlias"=> "ENABLED",
"Sharpness"=> 50,
"CodecSettings"=> [
"Codec"=> "H_264",
"H264Settings"=> [
"InterlaceMode"=> "PROGRESSIVE",
"NumberReferenceFrames"=> 3,
"Syntax"=> "DEFAULT",
"Softness"=> 0,
"GopClosedCadence"=> 1,
"GopSize"=> 60,
"Slices"=> 1,
"GopBReference"=> "DISABLED",
"SlowPal"=> "DISABLED",
"SpatialAdaptiveQuantization"=> "ENABLED",
"TemporalAdaptiveQuantization"=> "ENABLED",
"FlickerAdaptiveQuantization"=> "DISABLED",
"EntropyEncoding"=> "CABAC",
"Bitrate"=> 16200000,
"FramerateControl"=> "INITIALIZE_FROM_SOURCE",
"RateControlMode"=> "CBR",
"CodecProfile"=> "MAIN",
"Telecine"=> "NONE",
"MinIInterval"=> 0,
"AdaptiveQuantization"=> "HIGH",
"CodecLevel"=> "AUTO",
"FieldEncoding"=> "PAFF",
"SceneChangeDetect"=> "ENABLED",
"QualityTuningLevel"=> "SINGLE_PASS",
"FramerateConversionAlgorithm"=> "DUPLICATE_DROP",
"UnregisteredSeiTimecode"=> "DISABLED",
"GopSizeUnits"=> "FRAMES",
"ParControl"=> "INITIALIZE_FROM_SOURCE",
"NumberBFramesBetweenReferenceFrames"=> 2,
"RepeatPps"=> "DISABLED",
"DynamicSubGop"=> "STATIC"
]
],
"AfdSignaling"=> "NONE",
"DropFrameTimecode"=> "ENABLED",
"RespondToAfd"=> "NONE",
"ColorMetadata"=> "INSERT"
],
"NameModifier"=> "1"
],
[
"ContainerSettings"=> [
"Container"=> "MPD"
],
"VideoDescription"=> [
"Width"=> 1280,
"ScalingBehavior"=> "DEFAULT",
"Height"=> 720,
"TimecodeInsertion"=> "DISABLED",
"AntiAlias"=> "ENABLED",
"Sharpness"=> 50,
"CodecSettings"=> [
"Codec"=> "H_264",
"H264Settings"=> [
"InterlaceMode"=> "PROGRESSIVE",
"NumberReferenceFrames"=> 3,
"Syntax"=> "DEFAULT",
"Softness"=> 0,
"GopClosedCadence"=> 1,
"GopSize"=> 60,
"Slices"=> 1,
"GopBReference"=> "DISABLED",
"SlowPal"=> "DISABLED",
"SpatialAdaptiveQuantization"=> "ENABLED",
"TemporalAdaptiveQuantization"=> "ENABLED",
"FlickerAdaptiveQuantization"=> "DISABLED",
"EntropyEncoding"=> "CABAC",
"Bitrate"=> 7200000,
"FramerateControl"=> "INITIALIZE_FROM_SOURCE",
"RateControlMode"=> "CBR",
"CodecProfile"=> "MAIN",
"Telecine"=> "NONE",
"MinIInterval"=> 0,
"AdaptiveQuantization"=> "HIGH",
"CodecLevel"=> "AUTO",
"FieldEncoding"=> "PAFF",
"SceneChangeDetect"=> "ENABLED",
"QualityTuningLevel"=> "SINGLE_PASS",
"FramerateConversionAlgorithm"=> "DUPLICATE_DROP",
"UnregisteredSeiTimecode"=> "DISABLED",
"GopSizeUnits"=> "FRAMES",
"ParControl"=> "INITIALIZE_FROM_SOURCE",
"NumberBFramesBetweenReferenceFrames"=> 2,
"RepeatPps"=> "DISABLED",
"DynamicSubGop"=> "STATIC"
]
],
"AfdSignaling"=> "NONE",
"DropFrameTimecode"=> "ENABLED",
"RespondToAfd"=> "NONE",
"ColorMetadata"=> "INSERT"
],
"NameModifier"=> "2"
],
[
"ContainerSettings"=> [
"Container"=> "MPD"
],
"AudioDescriptions"=> [
[
"AudioTypeControl"=> "FOLLOW_INPUT",
"AudioSourceName"=> "Audio Selector 1",
"CodecSettings"=> [
"Codec"=> "AAC",
"AacSettings"=> [
"AudioDescriptionBroadcasterMix"=> "NORMAL",
"Bitrate"=> 96000,
"RateControlMode"=> "CBR",
"CodecProfile"=> "LC",
"CodingMode"=> "CODING_MODE_2_0",
"RawFormat"=> "NONE",
"SampleRate"=> 48000,
"Specification"=> "MPEG4"
]
],
"LanguageCodeControl"=> "FOLLOW_INPUT"
]
],
"NameModifier"=> "3"
]
],
"OutputGroupSettings"=> [
"Type"=> "DASH_ISO_GROUP_SETTINGS",
"DashIsoGroupSettings"=> [
"SegmentLength"=> 1,
"Destination"=> "s3://cactustestphp/videouploads/".$link . "/".$link,
"FragmentLength"=> 2,
"SegmentControl"=> "SEGMENTED_FILES",
"MpdProfile"=> "MAIN_PROFILE",
"HbbtvCompliance"=> "NONE"
]
]
]
],
"AdAvailOffset"=> 0,
"Inputs"=> [
[
"AudioSelectors"=> [
"Audio Selector 1"=> [
"Offset"=> 0,
"DefaultSelection"=> "DEFAULT",
"ProgramSelection"=> 1
]
],
"VideoSelector"=> [
"ColorSpace"=> "FOLLOW",
"Rotate"=> "DEGREE_0",
"AlphaBehavior"=> "DISCARD"
],
"FilterEnable"=> "AUTO",
"PsiControl"=> "USE_PSI",
"FilterStrength"=> 0,
"DeblockFilter"=> "DISABLED",
"DenoiseFilter"=> "DISABLED",
"InputScanType"=> "AUTO",
"TimecodeSource"=> "ZEROBASED",
"FileInput"=> "s3://cactustestphp/videouploads/test/". $fileid
]
]
];
Json:
{
"Queue": "!!",
"UserMetadata": {
"Customer": "Amazon"
},
"Role": "!!",
"Settings": {
"OutputGroups": [
{
"CustomName": "nicenice",
"Name": "DASH ISO",
"Outputs": [
{
"ContainerSettings": {
"Container": "MPD"
},
"VideoDescription": {
"Width": 3840,
"ScalingBehavior": "DEFAULT",
"Height": 2160,
"TimecodeInsertion": "DISABLED",
"AntiAlias": "ENABLED",
"Sharpness": 50,
"CodecSettings": {
"Codec": "H_264",
"H264Settings": {
"InterlaceMode": "PROGRESSIVE",
"NumberReferenceFrames": 3,
"Syntax": "DEFAULT",
"Softness": 0,
"FramerateDenominator": 1,
"GopClosedCadence": 1,
"GopSize": 30,
"Slices": 1,
"GopBReference": "DISABLED",
"SlowPal": "DISABLED",
"SpatialAdaptiveQuantization": "ENABLED",
"TemporalAdaptiveQuantization": "ENABLED",
"FlickerAdaptiveQuantization": "DISABLED",
"EntropyEncoding": "CABAC",
"Bitrate": 66200000,
"FramerateControl": "SPECIFIED",
"RateControlMode": "CBR",
"CodecProfile": "MAIN",
"Telecine": "NONE",
"FramerateNumerator": 30,
"MinIInterval": 0,
"AdaptiveQuantization": "HIGH",
"CodecLevel": "AUTO",
"FieldEncoding": "PAFF",
"SceneChangeDetect": "ENABLED",
"QualityTuningLevel": "SINGLE_PASS",
"FramerateConversionAlgorithm": "DUPLICATE_DROP",
"UnregisteredSeiTimecode": "DISABLED",
"GopSizeUnits": "FRAMES",
"ParControl": "INITIALIZE_FROM_SOURCE",
"NumberBFramesBetweenReferenceFrames": 2,
"RepeatPps": "DISABLED",
"DynamicSubGop": "STATIC"
}
},
"AfdSignaling": "NONE",
"DropFrameTimecode": "ENABLED",
"RespondToAfd": "NONE",
"ColorMetadata": "INSERT"
},
"NameModifier": "1"
},
{
"ContainerSettings": {
"Container": "MPD"
},
"VideoDescription": {
"Width": 1920,
"ScalingBehavior": "DEFAULT",
"Height": 1080,
"TimecodeInsertion": "DISABLED",
"AntiAlias": "ENABLED",
"Sharpness": 50,
"CodecSettings": {
"Codec": "H_264",
"H264Settings": {
"InterlaceMode": "PROGRESSIVE",
"NumberReferenceFrames": 3,
"Syntax": "DEFAULT",
"Softness": 0,
"FramerateDenominator": 1,
"GopClosedCadence": 1,
"GopSize": 30,
"Slices": 1,
"GopBReference": "DISABLED",
"SlowPal": "DISABLED",
"SpatialAdaptiveQuantization": "ENABLED",
"TemporalAdaptiveQuantization": "ENABLED",
"FlickerAdaptiveQuantization": "DISABLED",
"EntropyEncoding": "CABAC",
"Bitrate": 16200000,
"FramerateControl": "SPECIFIED",
"RateControlMode": "CBR",
"CodecProfile": "MAIN",
"Telecine": "NONE",
"FramerateNumerator": 30,
"MinIInterval": 0,
"AdaptiveQuantization": "HIGH",
"CodecLevel": "AUTO",
"FieldEncoding": "PAFF",
"SceneChangeDetect": "ENABLED",
"QualityTuningLevel": "SINGLE_PASS",
"FramerateConversionAlgorithm": "DUPLICATE_DROP",
"UnregisteredSeiTimecode": "DISABLED",
"GopSizeUnits": "FRAMES",
"ParControl": "INITIALIZE_FROM_SOURCE",
"NumberBFramesBetweenReferenceFrames": 2,
"RepeatPps": "DISABLED",
"DynamicSubGop": "STATIC"
}
},
"AfdSignaling": "NONE",
"DropFrameTimecode": "ENABLED",
"RespondToAfd": "NONE",
"ColorMetadata": "INSERT"
},
"NameModifier": "2"
},
{
"ContainerSettings": {
"Container": "MPD"
},
"VideoDescription": {
"Width": 1280,
"ScalingBehavior": "DEFAULT",
"Height": 720,
"TimecodeInsertion": "DISABLED",
"AntiAlias": "ENABLED",
"Sharpness": 50,
"CodecSettings": {
"Codec": "H_264",
"H264Settings": {
"InterlaceMode": "PROGRESSIVE",
"NumberReferenceFrames": 3,
"Syntax": "DEFAULT",
"Softness": 0,
"FramerateDenominator": 1,
"GopClosedCadence": 1,
"GopSize": 30,
"Slices": 1,
"GopBReference": "DISABLED",
"SlowPal": "DISABLED",
"SpatialAdaptiveQuantization": "ENABLED",
"TemporalAdaptiveQuantization": "ENABLED",
"FlickerAdaptiveQuantization": "DISABLED",
"EntropyEncoding": "CABAC",
"Bitrate": 5200000,
"FramerateControl": "SPECIFIED",
"RateControlMode": "CBR",
"CodecProfile": "MAIN",
"Telecine": "NONE",
"FramerateNumerator": 30,
"MinIInterval": 0,
"AdaptiveQuantization": "HIGH",
"CodecLevel": "AUTO",
"FieldEncoding": "PAFF",
"SceneChangeDetect": "ENABLED",
"QualityTuningLevel": "SINGLE_PASS",
"FramerateConversionAlgorithm": "DUPLICATE_DROP",
"UnregisteredSeiTimecode": "DISABLED",
"GopSizeUnits": "FRAMES",
"ParControl": "INITIALIZE_FROM_SOURCE",
"NumberBFramesBetweenReferenceFrames": 2,
"RepeatPps": "DISABLED",
"DynamicSubGop": "STATIC"
}
},
"AfdSignaling": "NONE",
"DropFrameTimecode": "ENABLED",
"RespondToAfd": "NONE",
"ColorMetadata": "INSERT"
},
"NameModifier": "3"
},
{
"ContainerSettings": {
"Container": "MPD"
},
"VideoDescription": {
"Width": 640,
"ScalingBehavior": "DEFAULT",
"Height": 360,
"TimecodeInsertion": "DISABLED",
"AntiAlias": "ENABLED",
"Sharpness": 50,
"CodecSettings": {
"Codec": "H_264",
"H264Settings": {
"InterlaceMode": "PROGRESSIVE",
"NumberReferenceFrames": 3,
"Syntax": "DEFAULT",
"Softness": 0,
"FramerateDenominator": 1,
"GopClosedCadence": 1,
"GopSize": 30,
"Slices": 1,
"GopBReference": "DISABLED",
"SlowPal": "DISABLED",
"SpatialAdaptiveQuantization": "ENABLED",
"TemporalAdaptiveQuantization": "ENABLED",
"FlickerAdaptiveQuantization": "DISABLED",
"EntropyEncoding": "CABAC",
"Bitrate": 1200000,
"FramerateControl": "SPECIFIED",
"RateControlMode": "CBR",
"CodecProfile": "MAIN",
"Telecine": "NONE",
"FramerateNumerator": 30,
"MinIInterval": 0,
"AdaptiveQuantization": "HIGH",
"CodecLevel": "AUTO",
"FieldEncoding": "PAFF",
"SceneChangeDetect": "ENABLED",
"QualityTuningLevel": "SINGLE_PASS",
"FramerateConversionAlgorithm": "DUPLICATE_DROP",
"UnregisteredSeiTimecode": "DISABLED",
"GopSizeUnits": "FRAMES",
"ParControl": "INITIALIZE_FROM_SOURCE",
"NumberBFramesBetweenReferenceFrames": 2,
"RepeatPps": "DISABLED",
"DynamicSubGop": "STATIC"
}
},
"AfdSignaling": "NONE",
"DropFrameTimecode": "ENABLED",
"RespondToAfd": "NONE",
"ColorMetadata": "INSERT"
},
"NameModifier": "4"
},
{
"ContainerSettings": {
"Container": "MPD"
},
"VideoDescription": {
"Width": 256,
"ScalingBehavior": "DEFAULT",
"Height": 144,
"TimecodeInsertion": "DISABLED",
"AntiAlias": "ENABLED",
"Sharpness": 50,
"CodecSettings": {
"Codec": "H_264",
"H264Settings": {
"InterlaceMode": "PROGRESSIVE",
"NumberReferenceFrames": 3,
"Syntax": "DEFAULT",
"Softness": 0,
"FramerateDenominator": 1,
"GopClosedCadence": 1,
"GopSize": 30,
"Slices": 1,
"GopBReference": "DISABLED",
"SlowPal": "DISABLED",
"SpatialAdaptiveQuantization": "ENABLED",
"TemporalAdaptiveQuantization": "ENABLED",
"FlickerAdaptiveQuantization": "DISABLED",
"EntropyEncoding": "CABAC",
"Bitrate": 200000,
"FramerateControl": "SPECIFIED",
"RateControlMode": "CBR",
"CodecProfile": "MAIN",
"Telecine": "NONE",
"FramerateNumerator": 30,
"MinIInterval": 0,
"AdaptiveQuantization": "HIGH",
"CodecLevel": "AUTO",
"FieldEncoding": "PAFF",
"SceneChangeDetect": "ENABLED",
"QualityTuningLevel": "SINGLE_PASS",
"FramerateConversionAlgorithm": "DUPLICATE_DROP",
"UnregisteredSeiTimecode": "DISABLED",
"GopSizeUnits": "FRAMES",
"ParControl": "INITIALIZE_FROM_SOURCE",
"NumberBFramesBetweenReferenceFrames": 2,
"RepeatPps": "DISABLED",
"DynamicSubGop": "STATIC"
}
},
"AfdSignaling": "NONE",
"DropFrameTimecode": "ENABLED",
"RespondToAfd": "NONE",
"ColorMetadata": "INSERT"
},
"NameModifier": "5"
},
{
"ContainerSettings": {
"Container": "MPD"
},
"AudioDescriptions": [
{
"AudioTypeControl": "FOLLOW_INPUT",
"AudioSourceName": "Audio Selector 1",
"CodecSettings": {
"Codec": "AAC",
"AacSettings": {
"AudioDescriptionBroadcasterMix": "NORMAL",
"Bitrate": 96000,
"RateControlMode": "CBR",
"CodecProfile": "LC",
"CodingMode": "CODING_MODE_2_0",
"RawFormat": "NONE",
"SampleRate": 48000,
"Specification": "MPEG4"
}
},
"LanguageCodeControl": "FOLLOW_INPUT"
}
],
"NameModifier": "6"
}
],
"OutputGroupSettings": {
"Type": "DASH_ISO_GROUP_SETTINGS",
"DashIsoGroupSettings": {
"SegmentLength": 1,
"Destination": "!!/videouploads/5fa1ababa7cea975176544/5fa1ababa7cea975176544",
"FragmentLength": 1,
"SegmentControl": "SEGMENTED_FILES",
"MpdProfile": "MAIN_PROFILE",
"HbbtvCompliance": "NONE"
}
}
}
],
"AdAvailOffset": 0,
"Inputs": [
{
"AudioSelectors": {
"Audio Selector 1": {
"Offset": 0,
"DefaultSelection": "DEFAULT",
"ProgramSelection": 1
}
},
"VideoSelector": {
"ColorSpace": "FOLLOW",
"Rotate": "DEGREE_0",
"AlphaBehavior": "DISCARD"
},
"FilterEnable": "AUTO",
"PsiControl": "USE_PSI",
"FilterStrength": 0,
"DeblockFilter": "DISABLED",
"DenoiseFilter": "DISABLED",
"TimecodeSource": "ZEROBASED",
"FileInput": "!!/videouploads/test/5fa1ababa7cea975176544.mp4"
}
]
},
"AccelerationSettings": {
"Mode": "DISABLED"
},
"StatusUpdateInterval": "SECONDS_60",
"Priority": 0
}
So, in other words, what would I have to change to make my segment duration set to 1 second or less?
Thank you for providing the job's JSON settings for further review. Looking at the OutputGroupSettings I can see that you are specifying a SegmentLength value of 1 second so the resulting fragmented MP4 files should be about 1 second in length give or take a few frames.
How are you confirming that the resulting fMP4 files are 30 and 10 seconds respectively? The most accurate way of determining this information is to combine a variant's init MP4 with one of the fragments into a separate file and observe it using a media inspector like Mediainfo or ffprobe. I used your job settings on an MP4 file I had available to me and confirmed that the resulting MP4 segments produces are 1 second in length:
Concatenate a segment with its init file for the full MP4 asset
$ cat 5fa1ababa7cea9751765441init.mp4 >> 5fa1ababa7cea9751765441_concat.mp4 && cat 5fa1ababa7cea9751765441_000000001.mp4 >> 5fa1ababa7cea9751765441_concat.mp4
Probe the newly concatenated file to review the details
$ ffprobe -hide_banner -i 5fa1ababa7cea9751765441_concat.mp4
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '5fa1ababa7cea9751765441_concat.mp4':
Metadata:
major_brand : isom
minor_version : 1
compatible_brands: isomavc1dash
creation_time : 2020-11-20T20:30:59.000000Z
Duration: 00:00:01.03, start: 0.066667, bitrate: 7658 kb/s
Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 3840x2160 [SAR 1:1 DAR 16:9], 7648 kb/s, 30 fps, 30 tbr, 90k tbn, 60 tbc (default)
Metadata:
creation_time : 2020-11-20T20:30:59.000000Z
handler_name : ETI ISO Video Media Handler
encoder : Elemental H.264
I'm curious how you are confirming the resulting duration of each fMP4 asset and if you can try the above steps as well if you have not already done so.

using mongodb case insentive regex with case insentive index

is mongo regex ignoring my index? I have a case insentive index, but by the look of things my regex search recognize it and ignores it.
db.getCollection("myCol").find({ value: /^mysearchVal/i }}).explain(...)
I have 95, 708 docs total.
output:
{
"queryPlanner": {
"plannerVersion": 1,
"namespace": "myDb.myCol",
"indexFilterSet": false,
"parsedQuery": {
"Value": {
"$regex": "^mysearchVal",
"$options": "i"
}
},
"winningPlan": {
"stage": "FETCH",
"filter": {
"Value": {
"$regex": "^mysearchVal",
"$options": "i"
}
},
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {
"Value": 1
},
"indexName": "value_case_insensitive_and_unique",
"collation": {
"locale": "en",
"caseLevel": false,
"caseFirst": "off",
"strength": 2,
"numericOrdering": false,
"alternate": "non-ignorable",
"maxVariable": "punct",
"normalization": false,
"backwards": false,
"version": "57.1"
},
"isMultiKey": false,
"multiKeyPaths": {
"Value": []
},
"isUnique": true,
"isSparse": false,
"isPartial": false,
"indexVersion": 2,
"direction": "forward",
"indexBounds": {
"Value": [
"[\"\", {})",
"[/^mysearchVal/i, /^mysearchVal/i]"
]
}
}
},
"rejectedPlans": []
},
"executionStats": {
"executionSuccess": true,
"nReturned": 1,
"executionTimeMillis": 1447,
"totalKeysExamined": 95708,
"totalDocsExamined": 95708,
"executionStages": {
"stage": "FETCH",
"filter": {
"Value": {
"$regex": "^mysearchVal",
"$options": "i"
}
},
"nReturned": 1,
"executionTimeMillisEstimate": 1270,
"works": 95709,
"advanced": 1,
"needTime": 95707,
"needYield": 0,
"saveState": 785,
"restoreState": 785,
"isEOF": 1,
"invalidates": 0,
"docsExamined": 95708,
"alreadyHasObj": 0,
"inputStage": {
"stage": "IXSCAN",
"nReturned": 95708,
"executionTimeMillisEstimate": 596,
"works": 95709,
"advanced": 95708,
"needTime": 0,
"needYield": 0,
"saveState": 785,
"restoreState": 785,
"isEOF": 1,
"invalidates": 0,
"keyPattern": {
"Value": 1
},
"indexName": "value_case_insensitive_and_unique",
"collation": {
"locale": "en",
"caseLevel": false,
"caseFirst": "off",
"strength": 2,
"numericOrdering": false,
"alternate": "non-ignorable",
"maxVariable": "punct",
"normalization": false,
"backwards": false,
"version": "57.1"
},
"isMultiKey": false,
"multiKeyPaths": {
"Value": []
},
"isUnique": true,
"isSparse": false,
"isPartial": false,
"indexVersion": 2,
"direction": "forward",
"indexBounds": {
"Value": [
"[\"\", {})",
"[/^mysearchVal/i, /^mysearchVal/i]"
]
},
"keysExamined": 95708,
"seeks": 1,
"dupsTested": 0,
"dupsDropped": 0,
"seenInvalidated": 0
}
},
"allPlansExecution": []
},
"ok": 1.0
}
the output shows 95,708 keys and docs examined, 1 doc returned. really? did the index apply in this case or am I missing a point or two?
Case insensitive regular expression queries generally cannot use
indexes effectively. The $regex implementation is not collation-aware
and is unable to utilize case-insensitive indexes.
https://docs.mongodb.com/manual/reference/operator/query/regex/#index-use

Spark streaming to Elasticsearch from Cassandra and MySQL correct configuration

I have inherited a cluster of mess and trying to figure out if this is the correct way to setup spark.
Goal: we have approx 13M records in Mysql across several tables that needs to be index in ElasticSearch. We have similarly have several million records in Cassandra that needs to be index in Elasticsearch also.
We do a total reindexation because a lot data gets changed in the DB and currently the application doesn't crud to ES.
All setup are on AWS Ec2 clusters with some RDS for MYSQL.
I have 4 EC2 (m4.xlarge) machines running ElasticSearch Nodes.
I have 3 EC2 machines running Cassandra Nodes
I have 1 Master + 1 Slave RDS running MySQL.
I have 1 C3.XLarge EC2 with Spark setup (everything is actually default configuration).
The scala application that was developed takes the mysql data, runs a mapping and transform them into RDD and push to the ES. Same goes for Cassandra.
Issue:
CPU + I/O taps out the ElasticSearch machines and I am at 100%. The entire process takes several hours.
Before I optimize code - should i be running the Spark system on standalone or should I be using it differently and why (so i can tell the IT I need to deploy or change the servers)
Thanks
UPDATE
{
"timestamp": 1470397729366,
"name": "es-server-1",
"transport_address": "inet[/XXX.xx.xx.XX:9300]",
"host": "es-server-1",
"ip": [
"inet[/xxx.xx.xx.XX:9300]",
"NONE"
],
"indices": {
"docs": {
"count": 13297950,
"deleted": 4132597
},
"store": {
"size": "19gb",
"size_in_bytes": 20405153604,
"throttle_time": "8.5h",
"throttle_time_in_millis": 30887045
},
"indexing": {
"index_total": 1306040463,
"index_time": "39.4d",
"index_time_in_millis": 3409319879,
"index_current": 0,
"delete_total": 3176,
"delete_time": "468ms",
"delete_time_in_millis": 468,
"delete_current": 0,
"noop_update_total": 0,
"is_throttled": false,
"throttle_time": "3.7h",
"throttle_time_in_millis": 13387134
},
"get": {
"total": 0,
"get_time": "0s",
"time_in_millis": 0,
"exists_total": 0,
"exists_time": "0s",
"exists_time_in_millis": 0,
"missing_total": 0,
"missing_time": "0s",
"missing_time_in_millis": 0,
"current": 0
},
"search": {
"open_contexts": 2,
"query_total": 91817250,
"query_time": "7.7d",
"query_time_in_millis": 669964087,
"query_current": 0,
"fetch_total": 6111935,
"fetch_time": "26.8m",
"fetch_time_in_millis": 1609543,
"fetch_current": 0
},
"merges": {
"current": 1,
"current_docs": 140211,
"current_size": "358.5mb",
"current_size_in_bytes": 375921439,
"total": 1308682,
"total_time": "52.2d",
"total_time_in_millis": 4515401497,
"total_docs": 22683582490,
"total_size": "18.3tb",
"total_size_in_bytes": 20196592980131
},
"refresh": {
"total": 6513707,
"total_time": "5.9d",
"total_time_in_millis": 515651065
},
"flush": {
"total": 49408,
"total_time": "1.7d",
"total_time_in_millis": 147150501
},
"warmer": {
"current": 0,
"total": 1696226,
"total_time": "26.4m",
"total_time_in_millis": 1589979
},
"filter_cache": {
"memory_size": "429.2mb",
"memory_size_in_bytes": 450131652,
"evictions": 1767369
},
"id_cache": {
"memory_size": "0b",
"memory_size_in_bytes": 0
},
"fielddata": {
"memory_size": "411mb",
"memory_size_in_bytes": 431001032,
"evictions": 0
},
"percolate": {
"total": 0,
"get_time": "0s",
"time_in_millis": 0,
"current": 0,
"memory_size_in_bytes": -1,
"memory_size": "-1b",
"queries": 0
},
"completion": {
"size": "0b",
"size_in_bytes": 0
},
"segments": {
"count": 272,
"memory": "113.4mb",
"memory_in_bytes": 118989720,
"index_writer_memory": "0b",
"index_writer_memory_in_bytes": 0,
"index_writer_max_memory": "813.7mb",
"index_writer_max_memory_in_bytes": 853250866,
"version_map_memory": "0b",
"version_map_memory_in_bytes": 0,
"fixed_bit_set": "32.1mb",
"fixed_bit_set_memory_in_bytes": 33692696
},
"translog": {
"operations": 0,
"size": "17b",
"size_in_bytes": 17
},
"suggest": {
"total": 4,
"time": "0s",
"time_in_millis": 0,
"current": 0
},
"query_cache": {
"memory_size": "0b",
"memory_size_in_bytes": 0,
"evictions": 0,
"hit_count": 0,
"miss_count": 0
}
},
"os": {
"timestamp": 1470397728691,
"uptime": "4.2h",
"uptime_in_millis": 15430550,
"load_average": [
0.01,
0.13,
0.17
],
"cpu": {
"sys": 0,
"user": 0,
"idle": 98,
"usage": 0,
"stolen": 0
},
"mem": {
"free": "1.5gb",
"free_in_bytes": 1627320320,
"used": "14.1gb",
"used_in_bytes": 15200206848,
"free_percent": 42,
"used_percent": 57,
"actual_free": "6.6gb",
"actual_free_in_bytes": 7102046208,
"actual_used": "9gb",
"actual_used_in_bytes": 9725480960
},
"swap": {
"used": "0b",
"used_in_bytes": 0,
"free": "0b",
"free_in_bytes": 0
}
},
"process": {
"timestamp": 1470397729391,
"open_file_descriptors": 1072,
"cpu": {
"percent": 8,
"sys": "2.8d",
"sys_in_millis": 247968490,
"user": "55.5d",
"user_in_millis": 4803132120,
"total": "58.4d",
"total_in_millis": 5051100610
},
"mem": {
"resident": "8.8gb",
"resident_in_bytes": 9499480064,
"share": "490.4mb",
"share_in_bytes": 514248704,
"total_virtual": "13gb",
"total_virtual_in_bytes": 14003888128
}
},
"jvm": {
"timestamp": 1470397728691,
"uptime": "168.6d",
"uptime_in_millis": 14575120452,
"mem": {
"heap_used": "4gb",
"heap_used_in_bytes": 4337667264,
"heap_used_percent": 51,
"heap_committed": "7.8gb",
"heap_committed_in_bytes": 8378908672,
"heap_max": "7.8gb",
"heap_max_in_bytes": 8378908672,
"non_heap_used": "90.5mb",
"non_heap_used_in_bytes": 94943920,
"non_heap_committed": "112mb",
"non_heap_committed_in_bytes": 117506048,
"pools": {
"young": {
"used": "263mb",
"used_in_bytes": 275871008,
"max": "266.2mb",
"max_in_bytes": 279183360,
"peak_used": "266.2mb",
"peak_used_in_bytes": 279183360,
"peak_max": "266.2mb",
"peak_max_in_bytes": 279183360
},
"survivor": {
"used": "513.3kb",
"used_in_bytes": 525696,
"max": "33.2mb",
"max_in_bytes": 34865152,
"peak_used": "33.2mb",
"peak_used_in_bytes": 34865152,
"peak_max": "33.2mb",
"peak_max_in_bytes": 34865152
},
"old": {
"used": "3.7gb",
"used_in_bytes": 4061270560,
"max": "7.5gb",
"max_in_bytes": 8064860160,
"peak_used": "5.8gb",
"peak_used_in_bytes": 6271548128,
"peak_max": "7.5gb",
"peak_max_in_bytes": 8064860160
}
}
},
"threads": {
"count": 75,
"peak_count": 90
},
"gc": {
"collectors": {
"young": {
"collection_count": 2827443,
"collection_time": "22.1h",
"collection_time_in_millis": 79772838
},
"old": {
"collection_count": 2420,
"collection_time": "4.2m",
"collection_time_in_millis": 253792
}
}
},
"buffer_pools": {
"direct": {
"count": 3519,
"used": "82.1mb",
"used_in_bytes": 86127850,
"total_capacity": "82.1mb",
"total_capacity_in_bytes": 86127850
},
"mapped": {
"count": 250,
"used": "2.8gb",
"used_in_bytes": 3045093527,
"total_capacity": "2.8gb",
"total_capacity_in_bytes": 3045093527
}
}
},
"thread_pool": {
"generic": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 4,
"completed": 16062931
},
"index": {
"threads": 4,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 4,
"completed": 35253
},
"bench": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"get": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"snapshot": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"merge": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 3112788
},
"suggest": {
"threads": 4,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 4,
"completed": 4
},
"bulk": {
"threads": 4,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 4,
"completed": 46447866
},
"optimize": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"warmer": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 159210301
},
"flush": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 756948
},
"search": {
"threads": 12,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 12,
"completed": 133987209
},
"listener": {
"threads": 2,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 26868988
},
"percolate": {
"threads": 0,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 0,
"completed": 0
},
"management": {
"threads": 5,
"queue": 0,
"active": 1,
"rejected": 0,
"largest": 5,
"completed": 227031495
},
"refresh": {
"threads": 1,
"queue": 0,
"active": 0,
"rejected": 0,
"largest": 2,
"completed": 6532756
}
},
"network": {
"tcp": {
"active_opens": 1782589,
"passive_opens": 20362132,
"curr_estab": 226,
"in_segs": 2535219205,
"out_segs": 2254643744,
"retrans_segs": 227538,
"estab_resets": 273558,
"attempt_fails": 14303,
"in_errs": 100,
"out_rsts": 260243
}
},
"fs": {
"timestamp": 1470397728691,
"total": {
"total": "196.6gb",
"total_in_bytes": 211110395904,
"free": "177.4gb",
"free_in_bytes": 190571290624,
"available": "167.4gb",
"available_in_bytes": 179800317952,
"disk_reads": 567609070,
"disk_writes": 216561071,
"disk_io_op": 784170141,
"disk_read_size": "15.6tb",
"disk_read_size_in_bytes": 17215484946432,
"disk_write_size": "23.2tb",
"disk_write_size_in_bytes": 25537856344064,
"disk_io_size": "38.8tb",
"disk_io_size_in_bytes": 42753341290496,
"disk_queue": "0",
"disk_service_time": "0"
},
"data": [
{
"path": "/data/elasticsearch/data0/es-server-1/nodes/0",
"mount": "/data/elasticsearch/data0",
"dev": "/dev/xvdf",
"total": "98.3gb",
"total_in_bytes": 105555197952,
"free": "88.8gb",
"free_in_bytes": 95364030464,
"available": "83.7gb",
"available_in_bytes": 89978544128,
"disk_reads": 280044788,
"disk_writes": 107670720,
"disk_io_op": 387715508,
"disk_read_size": "7.7tb",
"disk_read_size_in_bytes": 8567588643840,
"disk_write_size": "11.5tb",
"disk_write_size_in_bytes": 12689373003776,
"disk_io_size": "19.3tb",
"disk_io_size_in_bytes": 21256961647616,
"disk_queue": "0",
"disk_service_time": "0"
},
{
"path": "/data/elasticsearch/data1/es-server-1/nodes/0",
"mount": "/data/elasticsearch/data1",
"dev": "/dev/xvdg",
"total": "98.3gb",
"total_in_bytes": 105555197952,
"free": "88.6gb",
"free_in_bytes": 95207260160,
"available": "83.6gb",
"available_in_bytes": 89821773824,
"disk_reads": 287564282,
"disk_writes": 108890351,
"disk_io_op": 396454633,
"disk_read_size": "7.8tb",
"disk_read_size_in_bytes": 8647896302592,
"disk_write_size": "11.6tb",
"disk_write_size_in_bytes": 12848483340288,
"disk_io_size": "19.5tb",
"disk_io_size_in_bytes": 21496379642880,
"disk_queue": "0",
"disk_service_time": "0"
}
]
},
"transport": {
"server_open": 104,
"rx_count": 883258049,
"rx_size": "1.8tb",
"rx_size_in_bytes": 2051353458225,
"tx_count": 682553258,
"tx_size": "1.6tb",
"tx_size_in_bytes": 1804963466280
},
"http": {
"current_open": 12,
"total_opened": 19973834
},
"breakers": {
"request": {
"limit_size_in_bytes": 3351563468,
"limit_size": "3.1gb",
"estimated_size_in_bytes": 0,
"estimated_size": "0b",
"overhead": 1,
"tripped": 0
},
"fielddata": {
"limit_size_in_bytes": 5027345203,
"limit_size": "4.6gb",
"estimated_size_in_bytes": 431001032,
"estimated_size": "411mb",
"overhead": 1.03,
"tripped": 0
},
"parent": {
"limit_size_in_bytes": 5865236070,
"limit_size": "5.4gb",
"estimated_size_in_bytes": 431001032,
"estimated_size": "411mb",
"overhead": 1,
"tripped": 0
}
}
}

how to apply conditions on JSON response in Jmeter?

Response is as follows,
I have to extract all 'deviceResponseStatus.id' where status="PENDING"
how to implement it using JSON Path or REGEX in Jmeter?
{
"apiResponseStatus": "SUCCESS",
"deviceResponseList": [
{
"apiResponseStatus": "SUCCESS",
"id": 23,
"ownership": "CORPORATE",
"deviceName": "Demimbu",
"deviceType": "MOBILE",
"osType": "ANDROID",
"inTotalStorage": 0,
"inAvailableStorage": 0,
"exTotalStorage": 0,
"exAvailableStorage": 0,
"osVersion": "4.2",
"buildVersion": "1.01.08",
"status": "PENDING",
"isDeleted": false,
"policyIdNames": {
"id": 3,
"name": "Test Policy 3"
},
"deviceGroupIdNames": {
"id": 1,
"name": "Default Group"
},
"userIdName": {
"id": 1,
"name": "Randy Lewis"
},
"lastConnected": 1423810405000,
"createdByIdName": {
"id": 1,
"name": "Randy Lewis"
},
"locked": false,
"traceOn": false,
"userEmail": "mdm.user1#gmail.com",
"enrollmentDate": 1423790255000,
"simNumbers": [
"1236547896"
],
"imeiCodes": [],
"mobileNumbers": [
"4558585858858"
],
"createDate": 1421927250000,
"lastModified": 1423810406000,
"rooted": false,
"roaming": false,
"compliant": false
},
{
"apiResponseStatus": "SUCCESS",
"id": 24,
"ownership": "CORPORATE",
"deviceName": "Photobean",
"deviceType": "MOBILE",
"osType": "ANDROID",
"inTotalStorage": 20941258752,
"inAvailableStorage": 8024924160,
"exTotalStorage": 0,
"exAvailableStorage": 0,
"bluetoothMacAddress": "8C:3A:E3:BE:C2:04",
"wirelessMacAddress": "8c:3a:e3:51:bb:e3",
"model": "LG-D686",
"osVersion": "4.4.2",
"buildVersion": "1.01.09",
"status": "PENDING",
"isDeleted": false,
"policyIdNames": {
"id": 3,
"name": "Test Policy 3"
},
"deviceGroupIdNames": {
"id": 1,
"name": "Default Group"
},
"userIdName": {
"id": 1,
"name": "Randy Lewis"
},
"make": "LGE",
"lastConnected": 1423648831000,
"createdByIdName": {
"id": 1,
"name": "Randy Lewis"
},
"locked": false,
"traceOn": false,
"userEmail": "mdm.user1#gmail.com",
"enrollmentDate": 1423628675000,
"simNumbers": [
"404909008623970"
],
"imeiCodes": [
"359004053152889",
"359004053152871"
],
"mobileNumbers": [
"638847339387"
],
"createDate": 1422435029000,
"lastModified": 1423648831000,
"rooted": true,
"roaming": false,
"compliant": false
},
{
"apiResponseStatus": "SUCCESS",
"id": 25,
"ownership": "CORPORATE",
"deviceName": "Livetube",
"deviceType": "MOBILE",
"osType": "ANDROID",
"inTotalStorage": 0,
"inAvailableStorage": 0,
"exTotalStorage": 0,
"exAvailableStorage": 0,
"osVersion": "4.2",
"buildVersion": "1.01.08",
"status": "APPROVAL_PENDING",
"isDeleted": false,
"policyIdNames": {
"id": 3,
"name": "Test Policy 3"
},
"deviceGroupIdNames": {
"id": 1,
"name": "Default Group"
},
"userIdName": {
"id": 1,
"name": "Randy Lewis"
},
"lastConnected": 1423810522000,
"createdByIdName": {
"id": 1,
"name": "Randy Lewis"
},
"locked": false,
"traceOn": false,
"userEmail": "mdm.user1#gmail.com",
"enrollmentDate": 1423790371000,
"simNumbers": [
"1236547896"
],
"imeiCodes": [],
"mobileNumbers": [
"4558585858858"
],
"createDate": 1422435030000,
"lastModified": 1423810522000,
"rooted": false,
"roaming": false,
"compliant": false
},
{
"apiResponseStatus": "SUCCESS",
"id": 26,
"ownership": "CORPORATE",
"deviceName": "Riffwire",
"deviceType": "MOBILE",
"osType": "IOS",
"inTotalStorage": 0,
"inAvailableStorage": 0,
"exTotalStorage": 0,
"exAvailableStorage": 0,
"status": "PENDING",
"isDeleted": false,
"policyIdNames": {
"id": 3,
"name": "Test Policy 3"
},
"deviceGroupIdNames": {
"id": 1,
"name": "Default Group"
},
"userIdName": {
"id": 1,
"name": "Randy Lewis"
},
"createdByIdName": {
"id": 1,
"name": "Randy Lewis"
},
"locked": false,
"traceOn": false,
"userEmail": "mdm.user1#gmail.com",
"simNumbers": [],
"imeiCodes": [],
"mobileNumbers": [],
"createDate": 1422435030000,
"lastModified": 1423216312000,
"rooted": false,
"roaming": false,
"compliant": false
},
Also, Please suggest any good option if any.
You question is JMeter specific, but i believe it's actually regular Json Path expression. For example, in SoapUI it could be done with
$.deviceResponseList[?(#.status=='PENDING')].id
Probably this could help in some way
JMeter provides JSON Path Extractor via JMeter Plugins Extras With Libs Set
I believe that this is the best way to accomplish your goal.
See Using the XPath Extractor in JMeter guide (scroll down to "Parsing JSON" chapter) for the plugin installation instructions and some sample JSON Path queries.