Character encoding issue between logstash and logstash-forwarder - amazon-web-services

Question

I have the following setup -
[logstash-forwarder nodes] -> [Amazon's elastic load balancer] -> [logstash nodes]
I start logstash-forwarder with the following config file -
{
"network": {
"servers": ["<Load_balancer_DNS_name>:443"],
"ssl key": "/etc/pki/private/logstash-forwarder.key",
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt",
"timeout": 15
},
"files": [
{
"paths": [ "-" ],
"fields": { "type": "stdin" }
}
]
}
And I start logstash with following settings -
input {
tcp {
port => "7286"
codec => plain {
charset => "UTF-8"
}
}
}
output {
stdout { }
elasticsearch {
host => "<cluster_node_ip>"
protocol => "http"
}
}
Now I feed some input at the command line from logstash-forwarder just to see if it is reachable perfectly at logstash. So when I type "Hello World" or just any other plain text on the logstash-forwarder side, I receive the following on logstash node, instead of the original text -
Received an event that has a different character encoding than you configured. {:text=>"1W\\u0000\\u0000\\u0000\\u00011C\\u0000\\u0000\\u0000ox^2ta```\\u0004bV fI\\xCB\\xCCI\\u0005\\xF1uA\\x9C\\x8C\\xFC\\xE2\\u0012 -\\x90Y\\xA0kh\\xA0kha\\xAAkdh\\xACkb\\u0006\\u0014c\\xCBOK+N\\u0005\\xC92\\u001A\\x80\\x94\\xE6d\\xE6\\x81\\xF4\\t\\xBBy\\xFA9\\xFAć\\xB8\\u0006\\x87\\xC4{{:9\\xFA9\\xDAۃ\\xA4K*\\v#Ҭ\\xC5%)\\x99y\\u0000\\u0000\\u0000\\u0000\\xFF\\xFF\\u0001\\u0000\\u0000\\xFF\\xFF\\u001A\\x93\\u0015\\xA2", :expected_charset=>"UTF-8", :level=>:warn}

Answer 1

logstash-forwarder uses a unique protocol to communicate with logstash, named 'lumberjack'.
You need to have the key & crt on the logstash server as well, and use a lumberjack input to handle it:
input {
lumberjack {
# The port to listen on
port => 7286
# The paths to your ssl cert and key
ssl_certificate => "path/to/logstash-forwarder.crt"
ssl_key => "path/to/logstash-forwarder.key"
# Set this to whatever you want.
type => "somelogs"
}
}
What you're seeing is the encrypted lumberjack messages.
https://github.com/elasticsearch/logstash-forwarder#use-with-logstash