I'm using logstash 2.3.4 and Amazon Elasticsearch Service (2.3) .
My config
input {
jdbc {
# Postgres jdbc connection string to our database, mydb
jdbc_connection_string => "jdbc:mysql://awsmigration.XXXXXXXX.ap-southeast-1.rds.amazonaws.com:3306/table_receipt?zeroDateTimeBehavior=convertToNull&autoReconnect=true&useSSL=false"
# The user we wish to execute our statement as
jdbc_user => "XXXXXXXX"
jdbc_password => "XXXXXXXX"
# The path to our downloaded jdbc driver
jdbc_driver_library => "/opt/logstash/drivers/mysql-connector-java-5.1.39/mysql-connector-java-5.1.39-bin.jar"
# The name of the driver class for Postgresql
jdbc_driver_class => "com.mysql.jdbc.Driver"
# our query
statement => "SELECT * from Receipt"
jdbc_paging_enabled => true
jdbc_page_size => 200
}
}
output {
#stdout { codec => json_lines }
amazon_es {
hosts => ["search-XXXXXXXX.ap-southeast-1.es.amazonaws.com"]
region => "ap-southeast-1"
index => "slurp_receipt"
document_type => "Receipt"
document_id => "%{uid}"
}
}
After running a command
bin/logstash agent -f db.conf
I got this error :
Attempted to send a bulk request to Elasticsearch configured at '["https://search-XXXXXXXX.ap-southeast-1.es.amazonaws.com:443"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided? {:client_config=>{:hosts=>["https://search-slurp-wjgudsrlz66esh6hyrijaagamu.ap-southeast-1.es.amazonaws.com:443"], :region=>"ap-southeast-1", :aws_access_key_id=>nil, :aws_secret_access_key=>nil, :transport_options=>{:request=>{:open_timeout=>0, :timeout=>60}, :proxy=>nil}, :transport_class=>Elasticsearch::Transport::Transport::HTTP::AWS, :logger=>nil, :tracer=>nil, :reload_connections=>false, :retry_on_failure=>false, :reload_on_failure=>false, :randomize_hosts=>false, :http=>{:scheme=>"https", :user=>nil, :password=>nil, :port=>443}}, :error_message=>"undefined method `credentials' for nil:NilClass", :error_class=>"NoMethodError", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.1.36/lib/aws-sdk-core/signers/v4.rb:24:in `initialize'", "/opt/logstash/vendor/local_gems/b0f0ff24/logstash-output-amazon_es-1.0-java/lib/logstash/outputs/amazon_es/aws_v4_signer_impl.rb:36:in `signer'", "/opt/logstash/vendor/local_gems/b0f0ff24/logstash-output-amazon_es-1.0-java/lib/logstash/outputs/amazon_es/aws_v4_signer_impl.rb:48:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.2/lib/faraday/rack_builder.rb:139:in `build_response'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/faraday-0.9.2/lib/faraday/connection.rb:377:in `run_request'", "/opt/logstash/vendor/local_gems/b0f0ff24/logstash-output-amazon_es-1.0-java/lib/logstash/outputs/amazon_es/aws_transport.rb:49:in `perform_request'", "org/jruby/RubyProc.java:281:in `call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'", "/opt/logstash/vendor/local_gems/b0f0ff24/logstash-output-amazon_es-1.0-java/lib/logstash/outputs/amazon_es/aws_transport.rb:45:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform_request'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:90:in `bulk'", "/opt/logstash/vendor/local_gems/b0f0ff24/logstash-output-amazon_es-1.0-java/lib/logstash/outputs/amazon_es/http_client.rb:53:in `bulk'", "/opt/logstash/vendor/local_gems/b0f0ff24/logstash-output-amazon_es-1.0-java/lib/logstash/outputs/amazon_es.rb:321:in `submit'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/local_gems/b0f0ff24/logstash-output-amazon_es-1.0-java/lib/logstash/outputs/amazon_es.rb:318:in `submit'", "/opt/logstash/vendor/local_gems/b0f0ff24/logstash-output-amazon_es-1.0-java/lib/logstash/outputs/amazon_es.rb:351:in `flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:219:in `buffer_flush'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:216:in `buffer_flush'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/buffer.rb:159:in `buffer_receive'", "/opt/logstash/vendor/local_gems/b0f0ff24/logstash-output-amazon_es-1.0-java/lib/logstash/outputs/amazon_es.rb:311:in `receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/outputs/base.rb:83:in `multi_receive'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/outputs/base.rb:83:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:error}
May i know how to solve this problems?
thank you
Related
I'm trying to push documents from local to elastic server in AWS, and when trying to do so I get 403 error and Logstash keeps on trying to establish connection with the server like so:
[2021-05-09T11:09:52,707][TRACE][logstash.inputs.file ][main] Registering file input {:path=>["~/home/ubuntu/json_try/json_try.json"]}
[2021-05-09T11:09:52,737][DEBUG][logstash.javapipeline ][main] Shutdown waiting for worker thread {:pipeline_id=>"main", :thread=>"#<Thread:0x5033269f run>"}
[2021-05-09T11:09:53,441][DEBUG][logstash.outputs.amazonelasticsearch][main] Waiting for connectivity to Elasticsearch cluster. Retrying in 4s
[2021-05-09T11:09:56,403][INFO ][logstash.outputs.amazonelasticsearch][main] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>https://my-dom.co:8001/scans, :path=>"/"}
[2021-05-09T11:09:56,461][WARN ][logstash.outputs.amazonelasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"https://my-dom.co:8001/scans", :error_type=>LogStash::Outputs::AmazonElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '403' contacting Elasticsearch at URL 'https://my-dom.co:8001/scans/'"}
[2021-05-09T11:09:56,849][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2021-05-09T11:09:56,853][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2021-05-09T11:09:57,444][DEBUG][logstash.outputs.amazonelasticsearch][main] Waiting for connectivity to Elasticsearch cluster. Retrying in 8s
.
.
.
I'm using the following logstash conf file:
input {
file{
type => "json"
path => "~/home/ubuntu/json_try/json_try.json"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output{
amazon_es {
hosts => ["https://my-dom.co/scans"]
port => 8001
ssl => true
region => "us-east-1b"
index => "snapshot-%{+YYYY.MM.dd}"
}
}
Also I've exported AWS keys for the SSL to work. Is there anything I'm missing in order for the connection to succeed?
I've been able to solve this by using elasticsearch as my output plugin instead of amazon_es.
This usage will require cloud_id of the target AWS node, cloud_auth for it and also the target index in elastic for the data to be sent to. So the conf file will look something like this:
input {
file{
type => "json"
path => "~/home/ubuntu/json_try/json_try.json"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
output{
elasticsearch {
cloud_id: "node_name:node_hash"
cloud_auth: "auth_hash"
index: "snapshot-%{+YYYY.MM.dd}"
}
}
After upgrading to a php 7.1 server I get a "connection refused error"
Upgrading my localhost to php 7.2 results in a similar error.
Below I have included the details of the localhost.
I am trying to use doctrine stand alone without Symfony or Lavarel. I use as MAC and currently Bitnami Mamp stack 7.2.0-0
I am able to access the database via phpyadmin (the credentials username/password are fine)
My PHP.ini file contains:
pdo_mysql.default_socket = "/Applications/mampstack-7.2.0-0/mysql/tmp/mysql.sock"
I have composer installed.
{
"require": {
"doctrine/orm": "2.5.*",
"symfony/yaml": "2.*"
},
"autoload": {
"psr-0": {"": "src/"}
}
}
My bootstrap.php looks like: (I have tried both localhost and 127.0.0.1)
<?php
// bootstrap.php
use Doctrine\ORM\Tools\Setup;
use Doctrine\ORM\EntityManager;
require_once "vendor/autoload.php";
$paths = array(__DIR__ . "/entities");
$dbParams = array(
'driver' => 'pdo_mysql',
'host' => '127.0.0.1', // localhost
'user' => 'root',
'password' => '123456',
'dbname' => 'doctrine_test'
);
$isDevMode = TRUE;
$config = Setup::createAnnotationMetadataConfiguration($paths, $isDevMode);
$entityManager = EntityManager::create($dbParams, $config);
I have a basic config/cli-config.php file
require_once "bootstrap.php";
return \Doctrine\ORM\Tools\Console\ConsoleRunner::createHelperSet($entityManager);
However I am still stuck with a connection error:
An exception occurred in driver: SQLSTATE[HY000] [2002] Connection refused
After hours and hours adding another mysql user turned out to be the solution. For some reason the root user isn't allowed? It solved the issue both on the localhost and production environment.
I'm trying to give Blackfire.io (by Sensiolabs) a try to profile an existing PHP application running on a Vagrant machine (with PHP 5.3) on Mac.
I'm using Chef to provision my machine with Blackfire, but when running "vagrant provision" I get the following error:
default: STDERR: The server ID parameter is not set. Please run
blackfire-agent -register to configure it.
..which I already did
This is my Vagrant file:
is_windows = (RbConfig::CONFIG['host_os'] =~ /mswin|mingw|cygwin/)
Vagrant.configure("2") do |config|
..
config.vm.box = "covex/ubuntu1204-x64"
config.omnibus.chef_version = :latest
config.vm.provision "chef_solo" do |chef|
chef.json = {
:blackfire => {
:'server-id' => "d4860b49-be67-404b-9fa1-b..",
:'server-token' => "c412751f30d6c724033d8408e.."
}
}
chef.add_recipe "blackfire"
end
end
I followed the installation steps on https://blackfire.io/getting-started, except for the Probe paragraph.
Is my Vagrant file wrongly configured, so it can't read the server ID and token? Is the "brew install blackfire-php53" needed for this, if so, is there a way to configure this through my Vagrant file?
Guessing you are using https://supermarket.chef.io/cookbooks/blackfire
You missed the agent node in the config tree
{
"blackfire" => {
"agent" => {
"server-id" => "your server-id",
"server-token" => "your server-token",
}
}
}
I am working through deploying a Zf2 app with Doctrine on Aws Beanstalk. Composer is running and setting up everything and afterwards I run a symfony console command to generate the databases and schemas using the parameters specified in my config. I followed the examples out of here to get the rds params for the configuration.
http://www.michaelgallego.fr/blog/2013/05/24/how-to-deploy-safe-zf-2-applications-on-amazon-elastic-beanstalk/
All this works up to accessing the website. At which point it gives me a Pdo error accessing the database.
Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000] [2002]
No such file or directory' in
/var/app/current/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:43 Stack
trace: #0
/var/app/current/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php(43):
PDO->__construct('mysql:host=loca...', 'username', 'password', Array) #
/var/app/current/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOMySql/Driver.php(45):
Doctrine\DBAL\Driver\PDOConnection->__construct('mysql:host=loca...', 'username',
'password', Array) #2
/var/app/current/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(360):
Doctrine\DBAL\Driver\PDOMySql\Driver->connect(Array, 'username', 'password', Array) #3
/var/app/current/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(429):
Doctrine\DBAL\Connection->connect() #4
/var/app/current/vendor/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(389):
Doctrine\DBAL\Connection->getDatabasePlatformVersion() #5
/var/app/current/vendor/doctrine/dbal/lib/Doctrine/DBAL/C in
The zf2 app is apigility. After deployment if I log into the server and run the php command
php public/index.php development enable
It will produce the same error as the frontend website.
PHP Fatal error: Uncaught exception 'PDOException' with message 'SQLSTATE[HY000] [2002]
No such file or directory' in
/var/app/current/vendor/doctrine/dbal/lib/Doctrine/DBAL/Driver/PDOConnection.php:43
I am setting up the database schemas through console commands that are in the aws config. This does setup the correct database and runs the schema and data import.
.ebextensions/composer.config
container_commands:
01installDev:
command: "/usr/bin/composer.phar install --dev"
02makeDatabase:
command: "/usr/bin/php /var/app/ondeck/cwg.php database:create"
03createDatabase:
command: "/usr/bin/php /var/app/ondeck/cwg.php database:fresh"
config/autoload/database.global.php
return array(
'doctrine' => array(
'connection' => array(
'orm_default' => array(
'driverClass' => 'Doctrine\\DBAL\\Driver\\PDOMySql\\Driver',
'params' => array(
'host' => $_SERVER['RDS_HOSTNAME'],
'port' => $_SERVER['RDS_PORT'],
'user' => $_SERVER['RDS_USERNAME'],
'password' => $_SERVER['RDS_PASSWORD'],
'dbname' => 'api_default',
),
),
),
),
);
When I run the console commands I am just creating a doctrine instance with the parameters in these files, that is the only difference from the zf2 app, but I do get the correct params.
Trying to debug this I dumped the $options and $pdo variables inside a doctrine orm module factory.
vendor/doctrine/doctrine-orm-module/src/DoctrineORMModule/Service/DBALConnectionFactory.php
var_dump($options);die();
object(DoctrineORMModule\Options\DBALConnection)#369 (8) { ["configuration":protected]=>
string(11) "orm_default" ["eventmanager":protected]=> string(11) "orm_default
["pdo":protected]=> NULL ["driverClass":protected]=> string(36)
"Doctrine\DBAL\Driver\PDOMySql\Driver" ["wrapperClass":protected]=> NULL
["params":protected]=> array(5) {
["host"]=> string(9) "localhost"
["port"]=> string(4) "3306"
["user"]=> string(8) "username"
["password"]=> string(8) "password"
["dbname"]=>string(8) "database"
} ["doctrineTypeMappings":protected]=> array(0) {
["__strictMode__":protected]=> bool(true) }
var_dump($pdo);die();
NULL
The parameters that are being used are the default doctrine parameters. I dont know if this is to early in the process if zf2 hasn't loaded the config files but I haven't found any other places related to the connection that will var_dump before the error. What am I doing wrong either in my aws configuration setup, or firewall rules between the rds servers and the api (which I assumed beanstalk configured for you since it set everything up), or not doing my zf2 configuration correctly? Any help is appreciated.
edit
I moved the var_dump further down to line 60
var_dump($connection->getParams());die();
array(8) { ["driverClass"]=> string(36) "Doctrine\DBAL\Driver\PDOMySql\Driver
["wrapperClass"]=> NULL ["pdo"]=> NULL ["host"]=> string(9) "localhost" ["port"]=>
string(4) "3306" ["user"]=> string(8) "username" ["password"]=> string(8) "password
["dbname"]=> string(8) "database" }
I think this pretty well shows that Doctrine does not see my configuration at all and somehow I am doing something or not doing something.
Somehow inside the application.config.php file a static location was set for configuration.
'config_glob_paths' => array(
'/home/myhome/api/config/autoload/{,*.}{global,local}.php'
),
Once I changed that to
'config_glob_paths' => array(
'config/autoload/{,*.}{global,local}.php'
),
Doctrine started getting the correct params.
I have two elasticsearch nodes setup in EC2 and am trying to use logstash with it. I get this error when I run logstash:
log4j, [2014-02-24T10:45:32.722] WARN: org.elasticsearch.discovery.zen.ping.unicast: [Ishihara, Shirow] failed to send ping to [[#zen_unicast_1#][inet[/10.110.65.91:9300]]]
org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream
at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:169)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:123)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
That's a snippet of it.
Here is the conf file I am using with logstash:
input {
redis {
host => "10.110.65.91"
# these settings should match the output of the agent
data_type => "list"
key => "logstash"
# We use the 'json' codec here because we expect to read
# json events from redis.
codec => json
}
}
output {
stdout { debug => true debug_format => "json"}
elasticsearch {
host => "10.110.65.91"
cluster => searchbuild
}
}
~
I'm running Logstash on .91 (have a second terminal window open) Am I missing something?
I had to change "elasticsearch" to "elasticsearch_http".
Fixed.