I am trying out various options for setting spark driver memory in yarn.
Use Case:
I have a spark cluster with 1 master and 2 slaves
master : r5d xlarge - 8 vcore, 32GB
slave : r5d xlarge - 8 vcore, 32GB
I am using Apache Zeppelin to run the queries on spark cluster. Spark interpreter is configured with properties provided by Zeppelin. I am using spark 2.3.1 running on YARN. I want to create 4 interpreters so that 4 users can parallelly use this cluster.
Config 1:
spark.submit.deployMode client
spark.driver.cores 7
spark.driver.memory 24G
spark.driver.memoryOverhead 3072M
spark.executor.cores 1
spark.executor.memory 3G
spark.executor.memoryOverhead 512M
spark.yarn.am.cores 1
spark.yarn.am.memory 3G
spark.yarn.am.memoryOverhead 512M
Below is the spark executor UI:
Config 2:
spark.submit.deployMode client
spark.driver.cores 7
spark.driver.memory 12G
spark.driver.memoryOverhead 3072M
spark.executor.cores 1
spark.executor.memory 3G
spark.executor.memoryOverhead 512M
spark.yarn.am.cores 1
spark.yarn.am.memory 3G
spark.yarn.am.memoryOverhead 512M
Below is the spark executor UI:
Questions:
Why is the container size of driver 0?
Is the spark.memory.fraction calculated as (spark.driver.memory-300)*0.6 ? If so, why is it not exact ? (14.22, 7.02 resp)
Why is the container size of executor 3.8 GB ? According to my configuration, it should be 3G + 512M = 3.5 GB. This issue was not there with spark 2.1
No of VCores available to YARN is 8 per node. How is this possible since AWS gives vCPU with their instances? Hence I should only be getting 4 VCores according to AWS.
https://aws.amazon.com/ec2/instance-types/r5/
If I want to use 4 interpreters, should I distribute 32 GB of master equally to all the interpreters?
Driver:
spark.driver.cores 2
spark.driver.memory 7G
spark.driver.memoryOverhead 1024M
Related
When DataFusion runs a data pipeline, it persists in the provisioning state and then stops.
As a result, Dataproc cannot be created.
Dataproc's settings are as follows:
- Master
- Number of masters : 1
- Master Cores : 2
- Master Memory(GB) : 4
- Master Disk Size(GB) : 1000
- Worker
- Number of Workers : 2
- Worker Cores : 4
- Worker Memory(GB) : 16
- Worker Disk Size(GB) : 1500
In the data pipeline, the driver and executor are as follows:
- Executor
- CPU : 2
- Memory : 4
- Driver
- CPU : 2
- Memory : 4
If I actually look at dataproc in Google Cloud Console window, it will be provisioned and then disappear. Please share your opinion on how to solve this problem.
I'm using Spark 2.4.5 running on AWS EMR 5.30.0 with r5.4xlarge instances (16 vCore, 128 GiB memory, EBS only storage, EBS Storage:256 GiB) : 1 master, 1 core and 30 task.
I launched Spark Thrift Server on the master node and it's the only job that is running on the cluster
sudo /usr/lib/spark/sbin/start-thriftserver.sh --conf spark.blacklist.enabled=true --conf spark.blacklist.stage.maxFailedExecutorsPerNode=4 --conf spark.blacklist.task.maxTaskAttemptsPerNode=3 --conf spark.driver.cores=12 --conf spark.driver.maxResultSize=10g --conf spark.driver.memory=86000M --conf spark.driver.memoryOverhead=10240 --conf spark.kryoserializer.buffer.max=768m --conf spark.rpc.askTimeout=700 --conf spark.sql.broadcastTimeout=800 --conf spark.sql.sources.partitionOverwriteMode=dynamic --conf spark.task.maxFailures=20
Then I launch SQL queries on it with JDBC but when heavy queries are running, the UI gets really slow. I thought it would be fine if I put spark.driver.cores=12 (there are 16 in the master node) and spark.driver.memory=86000M (there are 128GB of memory) to leave some margin for the master node to be able to run other processes like the history server but it is still slow.
So I guess there are other settings that I can edit to make the UI works fine but I'm not sure what.
Those are the settings from spark-defaults.conf in the cluster FYI:
spark.driver.extraClassPath /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar
spark.driver.extraLibraryPath /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
spark.executor.extraClassPath /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar
spark.executor.extraLibraryPath /usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native
spark.eventLog.enabled true
spark.eventLog.dir hdfs:///var/log/spark/apps
spark.history.fs.logDirectory hdfs:///var/log/spark/apps
spark.sql.warehouse.dir hdfs:///user/spark/warehouse
spark.sql.hive.metastore.sharedPrefixes com.amazonaws.services.dynamodbv2
spark.yarn.historyServer.address <xxxxx>:18080
spark.history.ui.port 18080
spark.shuffle.service.enabled true
spark.yarn.dist.files /etc/spark/conf/hive-site.xml
spark.driver.extraJavaOptions -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
spark.dynamicAllocation.enabled true
spark.blacklist.decommissioning.enabled true
spark.blacklist.decommissioning.timeout 1h
spark.resourceManager.cleanupExpiredHost true
spark.stage.attempt.ignoreOnDecommissionFetchFailure true
spark.decommissioning.timeout.threshold 20
spark.executor.extraJavaOptions -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=70 -XX:MaxHeapFreeRatio=70 -XX:+CMSClassUnloadingEnabled -XX:OnOutOfMemoryError='kill -9 %p'
spark.hadoop.yarn.timeline-service.enabled false
spark.yarn.appMasterEnv.SPARK_PUBLIC_DNS $(hostname -f)
spark.files.fetchFailure.unRegisterOutputOnHost true
spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version.emr_internal_use_only.EmrFileSystem 2
spark.hadoop.mapreduce.fileoutputcommitter.cleanup-failures.ignored.emr_internal_use_only.EmrFileSystem true
spark.hadoop.fs.s3.getObject.initialSocketTimeoutMilliseconds 2000
spark.sql.parquet.output.committer.class com.amazon.emr.committer.EmrOptimizedSparkSqlParquetOutputCommitter
spark.sql.parquet.fs.optimized.committer.optimization-enabled true
spark.sql.emr.internal.extensions com.amazonaws.emr.spark.EmrSparkSessionExtensions
spark.sql.sources.partitionOverwriteMode dynamic
spark.executor.instances 1
spark.executor.cores 16
spark.driver.memory 2048M
spark.executor.memory 109498M
spark.default.parallelism 32
spark.emr.maximizeResourceAllocation true```
The problem was having only 1 core instance as the logs were saved in HDFS so this instance became a bottleneck.
I added another core instance and it's going much better now.
Another solution could be to save the logs to S3/S3A instead of HDFS, changing those parameters in spark-defaults.conf (make sure they are changed in the UI config too) but it might require adding some JAR files to work.
spark.eventLog.dir hdfs:///var/log/spark/apps
spark.history.fs.logDirectory hdfs:///var/log/spark/apps
I performed a cluster node installation using this guide [OpenStack Charms Deployment Guide].(https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/install-maas.html), where the type of network is a Flat network and the components used are:
Maas
Juju
Openstack
My lab is composed by following devices:
1 IBM System 3540 M4 Maas (500GB HDD - 8GB RAM - 1 Nic)
1 IBM System 3540 M4 Juju (500GB HDD - 8GB RAM -1 Nic)
4 IBM System 3540 M4 Openstack (500GBx2 HDD - 16GB RAM - 2 Nic)
1 Palo Alto Network Firewall
Public Network 10.20.81.0/24 - Private Network 10.0.0.0/24
Maas: 10.20.81.1
Juju: 10.20.81.2
Openstack 10.20.81.21-24
Gateway 10.20.81.254
Instance: 10.0.0.9 - 10.20.81.215 (floating)
network plan
10.20.81.0/24
+-------------+
Firewall
10.20.81.254
+-------------+
|
+-------------------------------------------------------------+
Switch
vlan81 vlan81 vlan81
+-------------------------------------------------------------+
| | || | | |
+--------------+ +------------+ +------------------+
|Maas+Juju |Juju Gui| |Openstack
|10.20.81.1 |10.20.81.2 |10.20.81.21-24
+--------------+ +-------------+ +------------------+
|
+--------------------------------------------+
Private Subnet-1 Public Subnet-2
10.0.0.0/24 10.20.81.0/24
+---+----+--+ +----+------+
| | +----+ |
| | | | |
| +--------+ VR +-------------+
| | |
+--+-+ +----+
| |
| VM |
| .9 |
| |
+----+
On my lab, the nodes for Openstack present two eth interface, the first one (eno2) the single external network used as floating IP, then the other one (eno3) for the private network.
On Juju gui I've that:
neutron-gateway:
bridge-mappings: physnet1:br-ex
data-port: br-ex:eno2
neutron-api:
flat-network-providers: physnet1
I've opened this post https://ask.openstack.org/en/question/119783/no-route-to-instance-ssh-and-ping-no-route-to-host/ to resolve the problem about the Ping and Ssh connection to my instance, but during same check I've seen this issue on neutron-gateway:
error: "could not add network device eno2 to ofproto (Device or resource busy)"
Maybe that is the cause of my first issue, but I don't understand how I can fix it.
$:juju ssh neutron-gateway/0
Welcome to Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-46-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
System information as of Tue Mar 19 16:07:19 UTC 2019
System load: 0.64 Processes: 409
Usage of /: 5.7% of 273.00GB Users logged in: 0
Memory usage: 13% IP address for lxdbr0: 10.122.135.1
Swap usage: 0% IP address for br-eno2: 10.20.81.21
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
3 packages can be updated.
0 updates are security updates.
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.
ovs-vsctl show output
ubuntu#os-compute01:~$ sudo ovs-vsctl show
6f8542aa-45d7-409d-8787-8983f3c643eb
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-ex
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "eno2"
Interface "eno2"
error: "could not add network device eno2 to ofproto (Device or resource busy)"
Port br-ex
Interface br-ex
type: internal
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
Port "gre-0a145118"
Interface "gre-0a145118"
type: gre
options: {df_default="true", in_key=flow, local_ip="10.20.81.21", out_key=flow, remote_ip="10.20.81.24"}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-int
Interface br-int
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "tapb0b04b07-8f"
tag: 2
Interface "tapb0b04b07-8f"
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port "tap2354468c-88"
tag: 4
Interface "tap2354468c-88"
Port "tap6d2b2fe0-47"
tag: 4
Interface "tap6d2b2fe0-47"
ovs_version: "2.10.0"
juju status
$:juju status
Model Controller Cloud/Region Version SLA Timestamp
openstack maas-cloud-controller maas-cloud 2.5.1 unsupported 22:10:17Z
App Version Status Scale Charm Store Rev OS Notes
ceph-mon 13.2.4+dfsg1 active 3 ceph-mon jujucharms 31 ubuntu
ceph-osd 13.2.4+dfsg1 active 3 ceph-osd jujucharms 273 ubuntu
ceph-radosgw 13.2.4+dfsg1 active 1 ceph-radosgw jujucharms 262 ubuntu
cinder 13.0.2 active 1 cinder jujucharms 276 ubuntu
cinder-ceph 13.0.2 active 1 cinder-ceph jujucharms 238 ubuntu
glance 17.0.0 active 1 glance jujucharms 271 ubuntu
keystone 14.0.1 active 1 keystone jujucharms 288 ubuntu
mysql 5.7.20-29.24 active 1 percona-cluster jujucharms 272 ubuntu
neutron-api 13.0.2 active 1 neutron-api jujucharms 266 ubuntu
neutron-gateway 13.0.2 active 1 neutron-gateway jujucharms 256 ubuntu
neutron-openvswitch 13.0.2 active 3 neutron-openvswitch jujucharms 255 ubuntu
nova-cloud-controller 18.0.3 active 1 nova-cloud-controller jujucharms 316 ubuntu
nova-compute 18.0.3 active 3 nova-compute jujucharms 290 ubuntu
ntp 3.2 active 4 ntp jujucharms 31 ubuntu
openstack-dashboard 14.0.1 active 1 openstack-dashboard jujucharms 271 ubuntu
rabbitmq-server 3.6.10 active 1 rabbitmq-server jujucharms 82 ubuntu
Unit Workload Agent Machine Public address Ports Message
ceph-mon/0 active idle 1/lxd/0 10.20.81.4 Unit is ready and clustered
ceph-mon/1 active idle 2/lxd/0 10.20.81.8 Unit is ready and clustered
ceph-mon/2* active idle 3/lxd/0 10.20.81.5 Unit is ready and clustered
ceph-osd/0 active idle 1 10.20.81.23 Unit is ready (1 OSD)
ceph-osd/1 active idle 2 10.20.81.22 Unit is ready (1 OSD)
ceph-osd/2* active idle 3 10.20.81.24 Unit is ready (1 OSD)
ceph-radosgw/0* active idle 0/lxd/0 10.20.81.15 80/tcp Unit is ready
cinder/0* active idle 1/lxd/1 10.20.81.18 8776/tcp Unit is ready
cinder-ceph/0* active idle 10.20.81.18 Unit is ready
glance/0* active idle 2/lxd/1 10.20.81.6 9292/tcp Unit is ready
keystone/0* active idle 3/lxd/1 10.20.81.20 5000/tcp Unit is ready
mysql/0* active idle 0/lxd/1 10.20.81.17 3306/tcp Unit is ready
neutron-api/0* active idle 1/lxd/2 10.20.81.7 9696/tcp Unit is ready
neutron-gateway/0* active idle 0 10.20.81.21 Unit is ready
ntp/0* active idle 10.20.81.21 123/udp chrony: Ready
nova-cloud-controller/0* active idle 2/lxd/2 10.20.81.3 8774/tcp,8775/tcp,8778/tcp Unit is ready
nova-compute/0 active idle 1 10.20.81.23 Unit is ready
neutron-openvswitch/1 active idle 10.20.81.23 Unit is ready
ntp/2 active idle 10.20.81.23 123/udp chrony: Ready
nova-compute/1 active idle 2 10.20.81.22 Unit is ready
neutron-openvswitch/2 active idle 10.20.81.22 Unit is ready
ntp/3 active idle 10.20.81.22 123/udp chrony: Ready
nova-compute/2* active idle 3 10.20.81.24 Unit is ready
neutron-openvswitch/0* active idle 10.20.81.24 Unit is ready
ntp/1 active idle 10.20.81.24 123/udp chrony: Ready
openstack-dashboard/0* active idle 3/lxd/2 10.20.81.19 80/tcp,443/tcp Unit is ready
rabbitmq-server/0* active idle 0/lxd/2 10.20.81.16 5672/tcp Unit is ready
Machine State DNS Inst id Series AZ Message
0 started 10.20.81.21 nbe8q3 bionic Openstack Deployed
0/lxd/0 started 10.20.81.15 juju-26461e-0-lxd-0 bionic Openstack Container started
0/lxd/1 started 10.20.81.17 juju-26461e-0-lxd-1 bionic Openstack Container started
0/lxd/2 started 10.20.81.16 juju-26461e-0-lxd-2 bionic Openstack Container started
1 started 10.20.81.23 pdnc7c bionic Openstack Deployed
1/lxd/0 started 10.20.81.4 juju-26461e-1-lxd-0 bionic Openstack Container started
1/lxd/1 started 10.20.81.18 juju-26461e-1-lxd-1 bionic Openstack Container started
1/lxd/2 started 10.20.81.7 juju-26461e-1-lxd-2 bionic Openstack Container started
2 started 10.20.81.22 yxkyet bionic Openstack Deployed
2/lxd/0 started 10.20.81.8 juju-26461e-2-lxd-0 bionic Openstack Container started
2/lxd/1 started 10.20.81.6 juju-26461e-2-lxd-1 bionic Openstack Container started
2/lxd/2 started 10.20.81.3 juju-26461e-2-lxd-2 bionic Openstack Container started
3 started 10.20.81.24 bgqsdy bionic Openstack Deployed
3/lxd/0 started 10.20.81.5 juju-26461e-3-lxd-0 bionic Openstack Container started
3/lxd/1 started 10.20.81.20 juju-26461e-3-lxd-1 bionic Openstack Container started
3/lxd/2 started 10.20.81.19 juju-26461e-3-lxd-2 bionic Openstack Container started
iptables
Any suggestions please. I am still unable to solve the problem. Thanks.
update 26/03/19:
On Juju gui I've that:
neutron-gateway:
bridge-mappings: physnet1:br-ex
data-port: br-ex:eno2
neutron-api:
flat-network-providers: physnet1
Before to make the deploy of Openstack I've changed data-port from br-ex:eno2 to br-ex:eno3
neutron-gateway:
bridge-mappings: physnet1:br-ex
data-port: br-ex:eno3
The issue on eno2 is been resolved but the ping to instance is still present.
I found the following post which explained using the ovftool command line utility to set an IP address: https://www.mylesgray.com/virtualisation/deploying-ovaovf-remote-vcenter-using-ovftool/
My problem is that when I run the ovftool against the ova (and the ovf inside), I do not get the output I need to determine the Instance ID for the network adapater. The only Instance ID that comes up is for the disk, which doesn't work.
[root#delphi OVA_Build]# ovftool SSMS-2.1.3-Stage-root-unlocked.ova
OVF version: 1.0
VirtualApp: false
Name: SSMS2.1.3-Prod-OVF
Download Size: 2.68 GB
Deployment Sizes:
Flat disks: 200.00 GB
Sparse disks: 6.79 GB
Networks:
Name: VM Network
Description: The VM Network network
Virtual Machines:
Name: SSMS2.1.3-Prod-OVF
Operating System: centos64guest
Virtual Hardware:
Families: vmx-07
Number of CPUs: 4
Cores per socket: 1
Memory: 4.00 GB
Disks:
Index: 0
Instance ID: 8
Capacity: 200.00 GB
Disk Types: SCSI-lsilogic
NICs:
Adapter Type: PCNet32
Connection: VM Network
I'm trying to run the following command (using env variables $VERSION, $BUILD_NUMBER, and $BUILD_IP_ADDRESS) but I do not know what goes in the InstanceID slot. Alternately, I would be ok with a command that keeps the existing IP address, but right now the networking info (IP, Gateway, Netmask) are all getting cleared out.
ovftool \
-ds=Skywalker \
-n=SSMS-$VERSION-Stage-$BUILD_NUMBER \
--ipAllocationPolicy=fixedPolicy \
--prop:vami.netmask0.<InstanceID>=255.255.255.0\
--prop:vami.gateway0.<InstanceID>=172.18.77.1\
--prop:vami.ip0.<InstanceID>=$BUILD_IP_ADDRESS\
--prop:vami.DNS.<InstanceID>=64.102.6.247\
--powerOn \
SSMS-2.1.3-Stage-root-unlocked.ova \
vi://root:rootpass#dlo-vcenter-rtp.cisco.com/Corona/host/RTP
I need you pro help to figure what is my issue with the ES & Logstash.
I'm using Elasticsearch 1.1.0 and logstash 1.4.0 to push logs to ES & Kibana .
My servers are located in AWS ( Master -> V.Core (4) , ECU (8) , RAM (15 GB) )
Node is the same .
My logstash configuration :
I'm taking the log files from S3 and put the locally in the Server
after that the logstash take them and push them to ES cluster .
Logstash + Kibana + ES Master (Located in 1 server)
those files are something like 12M size and I have more than 20,000 files.
MY ES configuration (Master)
cluster.name: MY-CLUSTER-NAME
node.name: MY-NODE-NAME
node.master: true
node.data: true
path.data: /PATH_TO_DATA/data
path.logs: /PATH_TO_LOGS/logs
ES Configuration (Node)
cluster.name: MY-CLUSTER-NAME (Same name)
node.name: MY-NODE-NAME (Another name)
node.master: false (This is node not master)
node.data: true
path.data: /PATH_TO_DATA/data
path.logs: /PATH_TO_LOGS/logs
For checking the cluster status :
http://MASTER_IP:9200/_cluster/health
this is the result :
{
"cluster_name": "es-cluster-onetagv2",
"status": "green",
"timed_out": false,
"number_of_nodes": 2,
"number_of_data_nodes": 2,
"active_primary_shards": 5,
"active_shards": 10,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}
my Java version ( Don't know if this is important ) :
java version "1.7.0_51" OpenJDK Runtime Environment
(amzn-2.4.4.1.36.amzn1-x86_64 u51-b02) OpenJDK 64-Bit Server VM (build
24.45-b08, mixed mode)
MY issue is that I'm trying to push more than 400 million hits for 1 day ,
and I can push in 24 hours something like 60 million. so I'm always behind.
I can see also that the ES taking 100% CPU USAGE.
but I don't know if this is the problem.
Maybe you can guide me what I'm doing wrong and how I can push big logs to ES fast.
Split the input file and give each file different name so that multiple inputs work on it, which helps logstash to read multiple files parallelly by different nodes.
Please increase the number of ES DataNodes to 4, and 2 ES Masters. This can be achieved by running two logstash instances( provides 2 Masters and 2 DataNodes) and 2 Elasticsearch Nodes(provides 2 Data Nodes).