I'm trying to transfer logs from Filebeat to Logstash
Both are running on different EC2 instances in the same network.
Apparently the socks5 protocol is used instead of http.
This is my filebeat.yml config file
filebeat.prospectors:
- type: log
paths:
- /camel-logs/app.log
output.logstash:
hosts: ["remote-host:5044"]
proxy_url: socks5://10.0.0.10:5044
filebeat.inputs:
- type: log
paths:
- /camel-logs/app.log
Honestly, I don't really know whether I should use prospectors, inputs, or both here. Neither works for now.
I'm positive that TCP port 5044 between the two hosts is open and accessible, but I don't know if socks5 is even possible over TCP? My knowledge about this stuff is quite limited.
I'm getting this error:
pipeline/output.go:74 Failed to connect: dial tcp 10.0.0.10:5044: getsockopt: connection refused
This could also be interesting:
log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":24},"total":{"ticks":40,"time":52,"value":40},"user":{"ticks":20,"time":28}},"info":{"ephemeral_id":"192acef7-0adb-4fbb-adfe-90cade7a5498","uptime":{"ms":30011}},"memstats":{"gc_next":4194304,"memory_alloc":2166616,"memory_total":4100568,"rss":21409792}},"filebeat":{"events":{"active":334,"added":335,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":318,"filtered":17,"published":318,"retry":852,"total":335}}},"registrar":{"states":{"current":1,"update":1},"writes":2},"system":{"cpu":{"cores":2},"load":{"1":3.22,"15":0.47,"5":1.23,"norm":{"1":1.61,"15":0.235,"5":0.615}}}}}}
Related
I have MongooseIM server configured with docker-compose in an EC2 instance in AWS.
I intend to give access to some mobile clients with SSL through an ELB (AWS) on port 5222 (module ejabberd_c2s of mongooseim) in the following way:
SSL (Secure TCP) -> 5222 -> TCP -> 5222 (EC2 Instance Port)
In the ejabberd_c2s module configuration I have the following:
{ 5222, ejabberd_c2s, [
%%
%% If TLS is compiled in and you installed a SSL
%% certificate, specify the full path to the
%% file and uncomment this line:
%%
{certfile, "priv/ssl/fake_server.pem"}, starttls,
%%{zlib, 10000},
%% https://www.openssl.org/docs/apps/ciphers.html#CIPHER_STRINGS
%% {ciphers, "DEFAULT:!EXPORT:!LOW:!SSLv2"},
{access, c2s},
{shaper, c2s_shaper},
{max_stanza_size, 65536},
{protocol_options, ["no_sslv3"]}
]},
But customers can not connect, the only message I get on the server is this:
mongooseim_server_dev | 10:58:25.885 [info] (#Port<0.27608>) Accepted connection {{10,0,17,246},42571} -> {{172,18,0,2},5222}
mongooseim_server_dev | 10:58:25.885 [debug] Received XML on stream = "���yw�\��.ndEt�;�����fn�A>� n:�=5��</A
"ngooseim_server_dev | ��kj98����g#32ED�(#
mongooseim_server_dev | 10:58:25.885 [debug] Send XML on stream = <<"<?xml version='1.0'?><stream:stream xmlns='jabber:client' xmlns:stream='http://etherx.jabber.org/streams' id='2B421BCD2D077161' from='localhost' version='1.0'>">>
mongooseim_server_dev | 10:58:25.886 [debug] Send XML on stream = <<"<stream:error><xml-not-well-formed xmlns='urn:ietf:params:xml:ns:xmpp-streams'/></stream:error>">>
mongooseim_server_dev | 10:58:25.886 [debug] Send XML on stream = <<"</stream:stream>">>
The Mongoose documentation does not offer me any solution and I do not see anyone with this error.
Any help or clue?
From your description and MongooseIM log snippet I reckon that the client is starting an encrypted connection from the beginning - that's why the "Received XML" seems to be garbage.
In XMPP an initially plaintext connection is upgraded to a secure connection using STARTTLS. This should work fine with ELB with TCP forwarding and no TLS termination, you just have to make sure the client is not trying to use SSL/TLS from the get go, but uses STARTTLS. All popular XMPP libraries should have this option, it's part of core XMPP.
[...] it is easier to put an ELB TCP to TCP and encrypt by TLS once the connection is open?
Exactly.
I mainly use an ELB to avoid having to handle SSL by myself and if I can not get it, would it be better to directly expose the mongoose server to the Internet?
ELB can't be used for SSL termination for plain XMPP. The available options are:
ELB forwards plain TCP, MongooseIM plain XMPP listener is used - Client opens a TCP connection but upgrades it via STARTTLS, all EC2 instances require cert provisioning.
ELB is set up for HTTPS termination, MongooseIM uses BOSH listener - BOSH is XMPP over HTTP, so has some overhead, but the benefit of SSL/TLS offloading might be worth it, no headache with certs on EC2 instances.
I have installed Kafka software on EC2. My problem is connecting to broker from outside the AWS. It all work for me from inside.
So I can start the broker, and both kafka-console-producer and consumer works (from the same server). I have ports 2181 and 9092 open to the remote location, towards from where I would like to use producer. So from my development (local) machine .. If I do telnet 9092 - it connects me. If i try to use kafka-console-producer i get this error.
[2017-03-09 15:04:44,971] ERROR Error when sending message to topic topic2 with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for topic2-0: 1521 ms has passed since batch creation plus linger time
I tried all sorts of combination with and on server.properties file - with keys listeners and advertised.listeners.
I would really appreciate some help...
It might be caused by the fact that the public hostname/ip of AWS machines cannot be used inside AWS. If so you need to fudge a bit. 2 things are needed:
make sure you set advertised.listeners to your private address
in your local /etc/hosts, bind the local hostname of aws (eg. ip-10-0-0-1.eu-west-1.compute.internal) to the public IP
Then make sure you always only use the private hostname. This has been the root cause for me of many weird issues not giving any logs.
I've got 3 servers on aws. each with open jdk 7 and zookeeper 3.4.6 all have unique elastic ip's.
each conf/zoo.cfg has
clientPort=2181
server.1=server1:2888:3888
server.2=server2:2888:3888
server.3=server3:2888:3888
then i start it with ./zkServer.sh start (says STARTED)
and the zookeeper.out says
2015-01-14 09:27:55,919 [myid:1] - INFO [Thread-1:QuorumCnxManager$Listener#504] - My election bind port: /server1ipaddress:3888
2015-01-14 09:27:55,920 [myid:1] - ERROR [/server1ipaddress:3888:QuorumCnxManager$Listener#517] - Exception while listening
java.net.BindException: Cannot assign requested address
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:330)
at org.apache.zookeeper.server.quorum.QuorumCnxManager$Listener.run(QuorumCnxManager.java:507)
So it cant open the port.
i've eventually opened all ports on aws security to rule that out.
telnet into 2181 with ruok gets imok.
telnet into 2888 cannot connect. connection refused.
telnet into 3888 cannot connect. connection refused.
netstat shows that nothing is blocking 2888 and 3888
i've even tried this with all 3 servers having zookeeper started.
whats going on? how do i get those ports open for use.
Your problem is answered here.
In a few words: on each ZooKeeper machine, at your conf/zoo.cfg, you have to set the current server's IP to 0.0.0.0.
For example: if you are currently on server1, the config should contain the following lines:
server.1=0.0.0.0:2888:3888
server.2=server2:2888:3888
server.3=server3:2888:3888
This step solved the problem in my case.
Cross verify myid's on all the nodes based on the zoo.cfg. The same issue happened to me, upon looking myid pattern got changed on 2 of the nodes.
I have an application which connects a remote database server.
If mysql server stops for a reason and stars succesfully after that, my application cannot detect server status quickly. It takes nearly 20 seconds to reconnect to the database server. So my gui freezes. I do not want a gui freeze for 20 seconds
So far I tried
mysql_ping
mysql_real_connect
functions
MYSQL_OPT_RECONNECT
MYSQL_OPT_CONNECT_TIMEOUT
options
My enviroment is not multi-threaded. So
how to do a faster detection?
If you do networking synchronously, be prepared for freezes. For this very reason it makes sense to do data-manipulation in a separate thread.
You could try telnet to the mysql port (usually 3306). If you get a connection refused, mysql isn't listening.
Working.
root#XXXXXX:~# telnet localhost 3306
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
L
5.6.4-m7)#m#_8:W�hP5YBzaXs[MOmysql_native_password
Down.
root#XXXXXX:~# telnet localhost 3306
Trying 127.0.0.1...
telnet: Unable to connect to remote host: Connection refused
The refused message is almost instant.
As already discussed by others, i won't talk about using multiple threads or processes. Can you connect to your mysql server on tcp? That way in most scenario's you would receive a tcp fin immediately to indicate a closed connection, though at times this might not be the case even. But most robust applications do a proper close.
shell> mysql --protocol=TCP
MYSQL how to specify protocol
If server doesn't accept it, i believe it can be enabled from config settings.
However, this does not address scenarios such as server suddenly gets off the network, or you client's connection is down etc.
during the last week I spent all my time trying to access the MCF in offline mode. I'm working behind a company network (proxy) and the MCF try to do things that conflict with the local network.
I've followed several different tutorials such as 1. Working offline with MCF and 2. Working offline with MCF. But the result keeps the same, even if I change all sort of configuration on my ubuntu.
Trying to set up the target.
vm target http: //api.mycloud.me
HTTP exception: Errno::ECONNREFUSED:Connection refused - connect(2)
The MCF console show the following information:
Identity: mycloud.me (ok)
Admin: admin#mycloud.me
IP address 10.0.x.x (network up / offline)
When I ping to the IP address, I got positive return.
PING 10.0.x.x (10.0.x.x) 56(84) bytes of data.
64 bytes from 10.0.x.x: icmp_req=1 ttl=62 time=1.06 ms
64 bytes from 10.0.x.x: icmp_req=2 ttl=62 time=0.896 ms
64 bytes from 10.0.x.x: icmp_req=3 ttl=62 time=0.880 ms
--- 10.0.2.15 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 0.880/0.948/1.069/0.089 ms
But if I try to do a telnet to the port 80 or a ssh I got connection refused error.
ssh: connect to host mycloud.me port 22: Connection refused
I don't know what I need to do to fix this, if anyone have a tip that help me to figure out a solution, I'll be very thankful.
Cheers!
OK dudes! That I fixed it!
So, after some problems to understand what was happening, I could finally connect to the Micro Cloud. I'm still validating the information from the two tutorials above, because could have some conflicted data.
I didn't test if it is necessary to set a nameserver to the dhclient, but the second tutorial seems to be more reliable. Just one tip, run the ssh -L tunnel on a separate terminal, and leave it open. This wasn't so clear for people like me, that was not used to working with network administration.
Thanks for the help.
given the assigned IP address, it looks like you are using bridged networking, have you tried changing the VM configuration to use NAT instead?
This will use an interface exclusive to your local machine and the VM and shouldn't be affected by your corporate network.