The mysterious bug was causing two things to happen
None of my VirtualServices were working despite being correctly formatted and having checked the fields several times.
On istioctl proxy-status the entire RDS column was STALE.
Upon looking at the istio-proxy logs -c discovery (greping for RDS), I saw the following error.
2019-02-27T19:09:58.644652Z warn ads ADS:RDS: ACK ERROR ... ... ... "Only unique values for domains are permitted. Duplicate entry of domain 172.16.x.y"
How do I fix this?
Info
Istio version 1.0.6
Kubernetes version 1.10.x-gke
The key to solving this was that IP address in the log. After searching for where in my configuration I had that IP address, it turns out it was in my ServiceEntries.
One of my ServiceEntries looked like this:
spec:
addresses:
- 172.16.x.y
hosts:
- 172.16.x.y
location: MESH_EXTERNAL
ports:
- name: http
number: 80
protocol: HTTP
- name: https
number: 443
protocol: HTTPS
resolution: DNS
It turns out you cannot have multiple ports in there. I deleted the HTTPS block and, like magic, everything worked. The istioctl proxy-status command displayed everything in the RDS as SYNCED and all of my VirtualServices started working again.
Related
new feature of "WSO IS 5.9.0" is the deployment.toml , but i have not found the configuration options nor the way how one can set the xml config file from this file.
For example, if I want to enable in carbon.xml option EnableHTTPAdminConsole, what should one do?
[server]
hostname = "my.server.com"
node_ip = "127.0.0.1"
base_path = "https://$ref{server.hostname}:${carbon.management.port}"
enable_h_t_t_p_admin_console = true
enable_http_admin_console = true
EnableHTTPAdminConsole = true
does not work
Also, i have tried to modify in my docker image:
wso2is-5.9.0/repository/resources/conf/templates/repository/conf/carbon.xml.j2
or
wso2is-5.9.0/conf/carbon.xml
But all these files gets overwritten.
My UseCase is to use WSO2IS in K8S without the port.
https://wso2is.my.domain/ > k8s nginx ingress : 443 (manages certificate) > wso2is-service > wso2is-pod : 9763 (plain http)
However the question still resides, what configuration options are available in deployment.toml
This seems like not possible through the deployment.toml. As a workaround, you can uncomment the property in
wso2is-5.9.0/repository/resources/conf/templates/repository/conf/carbon.xml.j2
Report this as an issue: https://github.com/wso2/product-is/issues
If the above fix is not getting applied, properly your docker image is getting overridden with the default configs. Can you try to build a new docker image with requested changes? This link https://github.com/wso2/docker-is/tree/5.9.0/dockerfiles/ubuntu/is can help you to build the image.
But I am not sure why you cannot access ssl(9443) from Nginx ingress. Maybe you can try this sample Nginx ingress https://github.com/wso2/kubernetes-is/blob/master/advanced/is-pattern-1/templates/identity-server-ingress.yaml
The answer to question what can be configured using deployment.toml has answered Buddhima, so i will mark his answer as answer.
One can look trhough the templates f.e.
wso2is-5.9.0/repository/resources/conf/templates/repository/conf/carbon.xml.j2
And can see all the options.
Answer to EnableHTTPAdminConsole answered pulasthi7 that it was intented left out.
I found workaround for the ingress to connect to ssl
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/rewrite-target: /$1
nginx.ingress.kubernetes.io/service-upstream: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
name: wso2is-ingress
namespace: wso2is
spec:
tls:
- hosts:
- wso2is.k8s.mydomain.com
secretName: tls-wso2is
rules:
- host: wso2is.k8s.mydomain.com
http:
paths:
- backend:
serviceName: wso2is-is-service
servicePort: 9443
path: /(.*)
The most important line:
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
This way the connection to is encrypted to ingress nginx using its own certificate, and from nginx to the pod using certificate in the pod.
I am new to using proxy, so I didnt quite follow why this isn't working. Not sure if this a Trafeik question, or a simple "I dont know how routing works" question
I followed the Traefik tutorial as show on their website here : https://docs.traefik.io/
Their docker-compose.yml looks like this:
version: '3'
services:
reverse-proxy:
image: traefik # The official Traefik docker image
command: --api --docker # Enables the web UI and tells Traefik to listen to docker
ports:
- "80:80" # The HTTP port
- "8080:8080" # The Web UI (enabled by --api)
volumes:
- /var/run/docker.sock:/var/run/docker.sock # So that Traefik can listen to the Docker events
whoami:
image: containous/whoami # A container that exposes an API to show its IP address
labels:
- "traefik.frontend.rule=Host:whoami.docker.localhost"
So now I wanted to run this same yml file on my ec2 instance. I make a change to the last line so that it looks like this instead:
- "traefik.frontend.rule=Host:whoami.docker.<ec2-XXX>.<region>.compute.amazonaws.com"
So I assumed that if I visited http://whoami.docker.<ec2-XXX>.<region>.compute.amazonaws.com, I would see my whoami app's response. However, I get a response from my ISP that that wesbite does not exist. If I access http://<ec2-XXX>.<region>.compute.amazonaws.com:8080 I can see my Traefik console fine.
I think its got to do with web addresses, and you can only have two items before the website, like x.y.website.com, and the url I am using to access my ec2 is already using those two slots. I am unsure what to search for.
Do I need to register a site/ buy a domain first?
How would I connect this site to my ec2 instance?
Was I correct as to why http://whoami.docker.<ec2-XXX>.<region>.compute.amazonaws.com was not working?
Has anyone facing this issue with docker pull. we recently upgraded docker to 18.03.1-ce from then we are seeing the issue. Although we are not exactly sure if this is related to docker, but just want to know if anyone faced this problem.
We have done some troubleshooting using tcp dump the DNS queries being made were under the permissible limit of 1024 packet. which is a limit on EC2, We also tried working around the issue by modifying the /etc/resolv.conf file to use a higher retry \ timeout value, but that didn't seem to help.
we did a packet capture line by line and found something. we found some responses to be negative. If you use Wireshark, you can use 'udp.stream eq 12' as a filter to view one of the negative answers. we can see the resolver sending an answer "No such name". All these requests that get a negative response use the following name in the request:
354XXXXX.dkr.ecr.us-east-1.amazonaws.com.ec2.internal
Would anyone of you happen to know why ec2.internal is being adding to the end of the DNS? If run a dig against this name it fails. So it appears that a wrong name is being sent to the server which responds with 'no such host'. Is docker is sending a wrong dns name for resolution.
We see this issue happening intermittently. looking forward for help. Thanks in advance.
Expected behaviour
5.0.25_61: Pulling from rrg
Digest: sha256:50bbce4af6749e9a976f0533c3b50a0badb54855b73d8a3743473f1487fd223e
Status: Downloaded newer image forXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/rrg:5.0.25_61
Actual behaviour
docker-compose up -d rrg-node-1
Creating rrg-node-1
ERROR: for rrg-node-1 Cannot create container for service rrg-node-1: Error response from daemon: Get https:/XXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/v2/: dial tcp: lookup XXXXXXXX.dkr.ecr.us-east-1.amazonaws.com on 10.5.0.2:53: no such host
Steps to reproduce the issue
docker pull XXXXXXXX.dkr.ecr.us-east-1.amazonaws.com/rrg:5.0.25_61
Output of docker version:
(Docker version 18.03.1-ce, build 3dfb8343b139d6342acfd9975d7f1068b5b1c3d3)
Output of docker info:
([ec2-user#ip-10-5-3-45 ~]$ docker info
Containers: 37
Running: 36
Paused: 0
Stopped: 1
Images: 60
Server Version: swarm/1.2.5
Role: replica
Primary: 10.5.4.172:3375
Strategy: spread
Filters: health, port, containerslots, dependency, affinity, constraint
Nodes: 12
Plugins:
Volume:
Network:
Log:
Swarm:
NodeID:
Is Manager: false
Node Address:
Kernel Version: 4.14.51-60.38.amzn1.x86_64
Operating System: linux
Architecture: amd64
CPUs: 22
Total Memory: 80.85GiB
Name: mgr1
Docker Root Dir:
Debug Mode (client): false
Debug Mode (server): false
Experimental: false
Live Restore Enabled: false
WARNING: No kernel memory limit support)
I'm trying to get Ejabberd to work with letsencrypt certificates on centos7.
I keep getting errors about the certificates not being signed by a known CA.
I have created the certificates by certbot, and I joined the privkey and fullchain files to single file.
All c2s connections work fine, but s2s connections don't.
When starting Ejabberd I see the following relevant log entries:
[warning] <0.606.0>#ejabberd_pkix:check_ca_dir:386 CA directory /etc/ssl/certs doesn't contain hashed certificate files; configuring 'ca_path' option might help
[warning] <0.606.0>#ejabberd_pkix:mk_cert_state:240 certificate from /opt/ejabberd/conf/xxxx.pem is invalid: certificate is signed by unknown CA
Connections to for example draugr.de generate the following entries:
[info] <0.793.0>#ejabberd_s2s_in:handle_auth_failure:206 (tls|<0.792.0>) Failed inbound s2s EXTERNAL authentication draugr.de -> XXXXX.net (::FFFF:89.163.212.45): unable to get local issuer certificate
I hope someone can help me out, thanks!
[EDIT 2020 may]
It looks like ejabberd now has automatic acme support (meaning it can request the certificate on its own from letsencrypt). So what you read below is obsolete.
As of 2018 november,
Merely installing letsencrypt using certbot is enough click here to see how. Ejabberd uses the provided certificates.
Note that you may need to register multiple subdomains for some strict jabber clients to work properly.
conference.yourjabberdomain.com
pubsub.yourjabberdomain.com
upload.yourjabberdomain.com
yourjabberdomain.com
or install a wildcard certificate from letsencrypt
sudo certbot certonly --server https://acme-v02.api.letsencrypt.org/directory --manual --preferred-challenges dns -d *.yourjabberdomain.com
I think there is a rule in the ejabberd config file ejabberd.yml that allows this to happen
certfiles:
- "/etc/letsencrypt/live/*/*.pem"
I was able to solve it myself finally but i am out of office the next few days and can't get you the exact configuration to solve it.
But if I recall correctly I downloaded the CA bundle here https://curl.haxx.se/docs/caextract.html and there was some configuration parameter for ejabberd to use this CA bundle in stead of the default one.
Hope it helps you.
If it is working for c2s and not working for s2s then it looks like the s2s block in configuration file is not updated with certfile. I believe you have something like this for c2s:
port: 5222
ip: "::"
module: ejabberd_c2s
starttls: true
certfile: 'CERTFILE'
protocol_options: 'TLSOPTS'
Similarly your s2s block should have:
port: 5269
ip: "::"
module: ejabberd_s2s_in
starttls: true
certfile: 'CERTFILE'
protocol_options: 'TLSOPTS'
max_stanza_size: 131072
shaper: s2s_shaper
I'm deploying to production as follow:
HOSTS = [
'api1.app.io',
'api2.app.io',
'api3.app.io',
'api4.app.io',
'api5.app.io',
'api6.app.io',
'api7.app.io',
'api8.app.io',
'api9.app.io',
'api10.app.io'
].freeze
HOSTS.each do |host|
server host, roles: %i[app web], user: 'deploy', port: 2323
end
although it fails when capistrano tries to ssh into the servers because it still tries to connect using port 22...
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing on host api2.app.io: Net::SSH::ConnectionTimeout
Net::SSH::ConnectionTimeout: Net::SSH::ConnectionTimeout
Errno::ETIMEDOUT: Connection timed out - connect(2) for xx.xx.xx.xx:22
Is there any other way to configure the ports?
The :port option should work. If it does not, that it is a bug. Please open a bug report at https://github.com/capistrano/capistrano/issues .
As a workaround, you can also specify the port via the hostname using the <host>:<port> syntax, like this:
HOSTS.each do |host|
server "#{host}:#{port}", roles: %i[app web], user: 'deploy'
end