The two NICs bridged together ingress are redirected to ifb, but it don't work ,why? - qos

I have a host that have two NICs, so it can be seen as a gateway, and the two network ports bridge. Then I make the two network interfaces' ingress traffic to redirect to an IFB, and then do the flow control of the IFB, but it do not work, why? The two NICs mentioned above one is WAN port, and another is LAN port.They are bridge. My script below:
#!/bin/sh
WAN=eth6
LAN=eth7
ifbdev=ifb0
#enable ifb interface
modprobe ifb numifbs=1
ip link set dev $ifbdev down
#add tc rules for WAN port
tc qdisc del dev $WAN root
tc qdisc del dev $WAN ingress
tc qdisc add dev $WAN ingress handle ffff:
tc filter add dev $WAN parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev $ifbdev
#add tc rules for LAN port
tc qdisc del dev $LAN root
tc qdisc del dev $LAN ingress
tc qdisc add dev $LAN ingress handle ffff:
tc filter add dev $LAN parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev $ifbdev
#add tc rules for IFB virtual interface
ip link set dev $ifbdev up
tc qdisc del dev $ifbdev root
tc qdisc del dev $ifbdev ingress
tc qdisc add dev $ifbdev root handle 3: htb default 30
tc class add dev $ifbdev parent 3: classid 3:3 htb rate 8Mbit
tc class add dev $ifbdev parent 3:3 classid 3:30 htb rate 8Mbit ceil 8Mbit
tc qdisc add dev $ifbdev parent 3:30 handle 330: sfq perturb 10
tc filter add dev $ifbdev parent 3:0 protocol ip u32 match u32 0 0 flowid 3:30

A quick google search gave me this.
https://serverfault.com/questions/350023/tc-ingress-policing-and-ifb-mirroring
See if it helps. Please post specifics of the problem if this doesn't solve yours.

Related

Condor master node and workers only see the master node

I am trying to set a HTCondor batch system, but when I do condor_status it only shows the master in both the master and worker nodes. They both show this:
Name OpSys Arch State Activity LoadAv Mem
[master ip] LINUX X86_64 Unclaimed Idle 0.000 973
Total Owner Claimed Unclaimed Matched Preempting Backfill Drain
X86_64/LINUX 1 0 0 1 0 0 0 0
Total 1 0 0 1 0 0 0 0
Condor_restart on the master node works fine, but on the worker nodes yields this error:
ERROR
SECMAN:2010:Received "DENIED" from server for user unauthenticated#unmapped using no authentication method, which may imply host-based security. Our address was '[ip address of master]', and server's address was '[ip address of worker]'. Check your ALLOW settings and IP protocols.
Here are the config files:
of the master node:
CONDOR_HOST = [private ip of master]
DAEMON_LIST = COLLECTOR, MASTER, NEGOTIATOR, SCHEDD, STARTD
# to avoid user authentication
HOSTALLOW_READ = *
HOSTALLOW_WRITE = *
HOSTALLOW_ADMINISTRATOR = *
of the worker node:
CONDOR_HOST = [private ip of master]
DAEMON_LIST = MASTER, STARTD
# to avoid user authentication
HOSTALLOW_READ = *
HOSTALLOW_WRITE = *
HOSTALLOW_ADMINISTRATOR = *
I am allowing on the same security group:
All TCP TCP 0 - 65535
All ICMP-IPv4 All
SSH on port 22
This is how it looks like (security group ending in '6')
Apparently the issue was running condor_reconfig -full. I just reinstalled it without doing that and using systemctl restart condor instead and it worked. If someone wants to bring some insight on why it was so please do so :)

AWS CDK setting a second listener + target ignores the target port

I have an ECS container which runs two endpoints on two different ports.
I configure a network load balancer infront of it to have two listeners, each with their own target group.
AWS CDK code for my stack is here (Note: I changed the construct in my example)
class MyStack(Stack):
def __init__(self, scope: Construct, construct_id: str, certificate: Certificate, vpc: Vpc, repository: Repository, subnets: SubnetSelection, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
cluster: Cluster = Cluster(self, "MyCluster", vpc=vpc, container_insights=True)
image: ContainerImage = ContainerImage.from_ecr_repository(repository=repository, tag="latest")
task_definition: FargateTaskDefinition = FargateTaskDefinition(
self, "MyTaskDefinition", cpu=512, memory_limit_mib=1024,
)
container: ContainerDefinition = task_definition.add_container(
"MyContainer", image=image, environment={}
)
# As you can see, here I add two port mappings on my container
container.add_port_mappings(PortMapping(container_port=9876, host_port=9876))
container.add_port_mappings(PortMapping(container_port=8000, host_port=8000))
load_balancer: NetworkLoadBalancer = NetworkLoadBalancer(
self, "MyNetworkLoadBalancer",
load_balancer_name="my-nlb",
vpc=vpc,
vpc_subnets=subnets,
internet_facing=False
)
security_group: SecurityGroup = SecurityGroup(
self, "MyFargateServiceSecurityGroup",
vpc=vpc,
allow_all_outbound=True,
description="My security group"
)
security_group.add_ingress_rule(
Peer.any_ipv4(), Port.tcp(9876), 'Allow a connection on port 9876 from anywhere'
)
security_group.add_ingress_rule(
Peer.any_ipv4(), Port.tcp(8000), "Allow a connection on port 8000 from anywhere"
)
service: FargateService = FargateService(
self, "MyFargateService",
cluster=cluster,
task_definition=task_definition,
desired_count=1,
health_check_grace_period=Duration.seconds(30),
vpc_subnets=subnets,
security_groups=[security_group]
)
# Listener 1 is open to incoming connections on port 9876
listener_9876: NetworkListener = load_balancer.add_listener(
"My9876Listener",
port=9876,
protocol=Protocol.TLS,
certificates=[ListenerCertificate(certificate.certificate_arn)],
ssl_policy=SslPolicy.TLS12_EXT
)
# Incoming connections on 9876 are redirected to the container on 9876
# A health check is done on 8000/health
listener_9876.add_targets(
"My9876TargetGroup", targets=[service], port=9876, protocol=Protocol.TCP,
health_check=HealthCheck(port="8000", protocol=Protocol.HTTP, enabled=True, path="/health")
)
# Listener 2 is open to incoming connections on port 443
listener_443: NetworkListener = load_balancer.add_listener(
"My443Listener",
port=443,
protocol=Protocol.TLS,
certificates=[ListenerCertificate(certificates.quickfix_certificate.certificate_arn)],
ssl_policy=SslPolicy.TLS12_EXT
)
# Incoming connections on 443 are redirected to the container on 8000
# A health check is done on 8000/health
listener_443.add_targets(
"My443TargetGroup", targets=[service], port=8000, protocol=Protocol.TCP,
health_check=HealthCheck(port="8000", protocol=Protocol.HTTP, enabled=True, path="/health")
)
Now I deploy this stack successfully, but the result is not what I expected
Two target groups directing traffic to my container, but both on port 9876.
I read in the documentation that it is possible to have a load balancer direct traffic to different ports via different target groups.
Am I doing something wrong? Or does AWS CDK not support this?
I double checked the synthesized cloudformation template. It properly generates two target groups, one with port 9876 and one with port 8000.
Hi you need create a target from service then add as a target to listener.
const target = service.loadBalancerTarget({
containerName: 'MyContainer',
containerPort: 8000
}));

broadcast_rpc_address in 3 seed and 3 non-seed node cassandra cluster deployed on AWS

What should i set broadcast_rpc_address in 3 seed and 3 non-seed node cassandra cluster deployed on AWS?
rpc address is set to wildcard 0.0.0.0,
Seed nodes launched using static ENI,
Non seed nodes launched using ASG,
ALL the nodes are launched in private subnet which are able to connect to internet using NAT gateway.
Added cassandra.yaml file which i am using
cluster_name: 'Cassandra Cluster'
num_tokens: 256
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
permissions_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
disk_failure_policy: stop
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
saved_caches_directory: /var/lib/cassandra/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "seednode-A-IP,seednode-B-IP,seednode-C-IP"
concurrent_reads: 32
concurrent_writes: 32
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 10.8.9.83
start_native_transport: true
native_transport_port: 9042
start_rpc: true
rpc_address: 0.0.0.0
broadcast_rpc_address: NAT-GATEWAY-IP
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
column_index_size_in_kb: 64
compaction_throughput_mb_per_sec: 16
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
cross_node_timeout: false
endpoint_snitch: Ec2Snitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
keystore: conf/.keystore
keystore_password: cassandra
truststore: conf/.truststore
truststore_password: cassandra
client_encryption_options:
enabled: false
keystore: conf/.keystore
keystore_password: cassandra
internode_compression: all
inter_dc_tcp_nodelay: false
Okie, found the issue,
In cassandra versions 3.o and above cql uses 9042 for connection , after changing the LB listener port from 9160 to 9042, it worked

what address should i use for broadcast_rpc_address in cassandra.yaml

cluster_name: 'Cassandra Cluster'
num_tokens: 256
hinted_handoff_enabled: true
max_hint_window_in_ms: 10800000
hinted_handoff_throttle_in_kb: 1024
max_hints_delivery_threads: 2
authenticator: AllowAllAuthenticator
authorizer: AllowAllAuthorizer
permissions_validity_in_ms: 2000
partitioner: org.apache.cassandra.dht.Murmur3Partitioner
data_file_directories:
- /var/lib/cassandra/data
commitlog_directory: /var/lib/cassandra/commitlog
disk_failure_policy: stop
key_cache_size_in_mb:
key_cache_save_period: 14400
row_cache_size_in_mb: 0
row_cache_save_period: 0
saved_caches_directory: /var/lib/cassandra/saved_caches
commitlog_sync: periodic
commitlog_sync_period_in_ms: 10000
commitlog_segment_size_in_mb: 32
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
- seeds: "seednode-A-IP,seednode-B-IP,seednode-C-IP"
concurrent_reads: 32
concurrent_writes: 32
trickle_fsync: false
trickle_fsync_interval_in_kb: 10240
storage_port: 7000
ssl_storage_port: 7001
listen_address: 10.8.9.83
start_native_transport: true
native_transport_port: 9042
start_rpc: true
rpc_address: 0.0.0.0
broadcast_rpc_address: NAT-GATEWAY-IP
rpc_port: 9160
rpc_keepalive: true
rpc_server_type: sync
thrift_framed_transport_size_in_mb: 15
incremental_backups: false
snapshot_before_compaction: false
auto_snapshot: true
tombstone_warn_threshold: 1000
tombstone_failure_threshold: 100000
column_index_size_in_kb: 64
compaction_throughput_mb_per_sec: 16
read_request_timeout_in_ms: 5000
range_request_timeout_in_ms: 10000
write_request_timeout_in_ms: 2000
cas_contention_timeout_in_ms: 1000
truncate_request_timeout_in_ms: 60000
request_timeout_in_ms: 10000
cross_node_timeout: false
endpoint_snitch: Ec2Snitch
dynamic_snitch_update_interval_in_ms: 100
dynamic_snitch_reset_interval_in_ms: 600000
dynamic_snitch_badness_threshold: 0.1
request_scheduler: org.apache.cassandra.scheduler.NoScheduler
server_encryption_options:
internode_encryption: none
keystore: conf/.keystore
keystore_password: cassandra
truststore: conf/.truststore
truststore_password: cassandra
client_encryption_options:
enabled: false
keystore: conf/.keystore
keystore_password: cassandra
internode_compression: all
inter_dc_tcp_nodelay: false
we have Cassandra cluster deployed on AWS with 3 seed nodes (static ENI attached) and 3 non seed nodes in Auto scaling group.
I have rpc_address set to 0.0.0.0, can someone tell me what should be the broadcast_rpc_address in cassandra.yaml file?
I was using cassandra 2.0.7 version before and was able to connect to cluster fine with just rpc_address as 0.0.0.0 and without broadcast_rpc_address this unset, but when i upgraded to 3.11.1 it is giving me error
CassandraDaemon.java:708 - Exception encountered during startup: If rpc_address is set to a wildcard address (0.0.0.0), then you must set broadcast_rpc_address to a value other than 0.0.0.0
If you are using Cassandra 2.1 or greater, you can configure broadcast_rpc_address. you can configure broadcast_rpc_address to be public IP.
Cassandra uses the "broadcast_address/listen_address" for internode connectivity and "broadcast_rpc_address/rpc_address" for the rpc interface (client -> coordinator (Cassandra node) requests).

Erlang - Ejabberd join_cluster Error: {no_ping ...}

Hi I have been trying to set up an ejabberd cluster.
However on trying to join from node2 to node1 , i get an error saying
On node 2:
# ejabberdctl join_cluster ejabberd#<internal ip of node1>
Error: {no_ping,'ejabberd#<internal ip of node1>'}
I can clearly ping node1 from node2.
Both the node are hosted in the same region on AWS.
I have tried allowing all traffic on node 1.
Both have the same .erlang.cookie.
Not sure why i continue to get that error.
# ejabberdctl status
The node 'ejabberd#<internal ip of node1>' is started with status: started
ejabberd 16.03.107 is running in that node
# netstat -lnptu
tcp 0 0 0.0.0.0:4369 0.0.0.0:* LISTEN 2190/epmd
tcp 0 0 0.0.0.0:5269 0.0.0.0:* LISTEN 2233/beam.smp
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 975/sshd
tcp 0 0 0.0.0.0:52189 0.0.0.0:* LISTEN 2233/beam.smp
tcp 0 0 0.0.0.0:5280 0.0.0.0:* LISTEN 2233/beam.smp
tcp 0 0 0.0.0.0:5222 0.0.0.0:* LISTEN 2233/beam.smp
tcp6 0 0 :::4369 :::* LISTEN 2190/epmd
tcp6 0 0 :::22 :::* LISTEN 975/sshd
ejabberdctl.cfg on node1 :
ERLANG_NODE=ejabberd#<internal IP of node1>
ejabberd.yml on node1:
loglevel: 4
log_rotate_size: 10485760
log_rotate_date: ""
log_rotate_count: 1
log_rate_limit: 100
hosts:
- "<external ip of node1>"
listen:
-
port: 5222
module: ejabberd_c2s
max_stanza_size: 65536
shaper: c2s_shaper
access: c2s
-
port: 5269
module: ejabberd_s2s_in
-
port: 5280
module: ejabberd_http
request_handlers:
"/websocket": ejabberd_http_ws
web_admin: true
http_bind: true
captcha: true
auth_method: internal
shaper:
normal: 1000
fast: 50000
max_fsm_queue: 1000
acl:
local:
user_regexp: ""
loopback:
ip:
- "127.0.0.0/8"
access:
max_user_sessions:
all: 10
max_user_offline_messages:
admin: 5000
all: 100
local:
local: allow
c2s:
blocked: deny
all: allow
c2s_shaper:
admin: none
all: normal
s2s_shaper:
all: fast
announce:
admin: allow
configure:
admin: allow
muc_admin:
admin: allow
muc_create:
local: allow
muc:
all: allow
pubsub_createnode:
local: allow
register:
all: allow
trusted_network:
loopback: allow
language: "en"
modules:
mod_adhoc: {}
mod_announce: # recommends mod_adhoc
access: announce
mod_blocking: {} # requires mod_privacy
mod_caps: {}
mod_carboncopy: {}
mod_client_state: {}
mod_configure: {} # requires mod_adhoc
mod_disco: {}
mod_irc: {}
mod_http_bind: {}
mod_last: {}
mod_muc:
host: "conference.#HOST#"
access: muc
access_create: muc_create
access_persistent: muc_create
access_admin: muc_admin
mod_muc_admin: {}
mod_offline:
access_max_user_messages: max_user_offline_messages
mod_ping: {}
mod_privacy: {}
mod_private: {}
mod_pubsub:
access_createnode: pubsub_createnode
ignore_pep_from_offline: true
last_item_cache: false
plugins:
- "flat"
- "hometree"
- "pep" # pep requires mod_caps
mod_roster: {}
mod_shared_roster: {}
mod_stats: {}
mod_time: {}
mod_vcard:
search: false
mod_version: {}
allow_contrib_modules: true
I faced the same issue while setting up Ejabberd cluster on EC2. Here solution for reference.
Make sure following ports are open on internal/private network
5222 - xmpp client connection
5280 - web portal
4369 - EPMD
5269 - S2S
4200 - 4210 node communication
Also allow internal ping (icmp packets) just in case.
Next set FIREWALL_WINDOW option in ejabberdctl.cfg file as follows. This will set Erlang use a defined range of port instead of dynamic
ports for node communication. (refer ejabberdctl.cfg)
FIREWALL_WINDOW=4200-4210
And use full node names for you Ejabberd nodes eg: ejabberd#srv1.example.com
it seems you are missing configration in ejabberdctl.cfg change the following line in your ejabberdctl.cfg file-
#INET_DIST_INTERFACE=127.0.0.1 to
INET_DIST_INTERFACE=104.10.120.122 (whatever your host public ip)
and open erlang console and run the following command-
net_adm:ping('ejabberd#ejabberd1'). # your node
if it will return pong now you can do cluster between ejabberd nodes.
run the following command to make cluster-
ejabberdctl join_cluster 'ejabberd#ejabberd1'
Frist, #Uday Sawant's method is mandatory.
and also you should add each node info into /etc/hosts
for example, if your nodes are
ejabberd#node1
ejabberd#node2
set these to host file for two systems.
for os,
add your ejabbered node hostname
vi /etc/hosts
...
node1 10.0.100.1
node2 10.0.100.2
for erlang
vi $HOME/.hosts.erlang
'node1'.
'node2'.
host file for ejabberd