Graylog web page is running as below:
Following the documentation for spring boot: graylog-springboot
However, nothing shows in the result. Could you Please advise me if you know what im doing wrong.
I have created log4j.xml file as below:
<appender name="graylog" class="org.graylog2.log.GelfAppender">
<param name="graylogHost" value="ec2-x-x-x-x.ap-west-1.compute.amazonaws.com"/>
<param name="originHost" value="ec2-x-x-x-x.eu-west-1.compute.amazonaws.com"/>
<param name="graylogPort" value="12201"/>
<param name="extractStacktrace" value="true"/>
<param name="addExtendedInformation" value="true"/>
<param name="facility" value="log4j"/>
<param name="Threshold" value="INFO"/>
<param name="additionalFields" value="{'environment': 'DEV', 'application': 'GraylogDemoApplication'}"/>
Accordingly opened a port for 9000 and 12201 in the security group.
in the build.gradle:
dependencies {
compile 'org.springframework.boot:spring-boot-starter-web'
compile 'org.springframework.boot:spring-boot-starter-log4j2'
implementation group: 'org.graylog2', name: 'gelfj', version: '1.1.16'
}
configurations {
all {
exclude group: 'org.springframework.boot', module: 'spring-boot-starter-logging'
}}
In the application.properties file:
Ensure that the version of Elastic you are running is compatible, currently the highest supported version is 7.10.2. If this is a fresh install it would be worth considering running Opensearch, this would also mean Graylog version installed should be above 4.3. Elastic and Graylog are going through a divorce.
Try the below command, replacing the hostname to test if anything is being ingested.
echo '{ "version": "1.1", "host": "example.org", "short_message": "A short message that helps you identify what is going on", "level": 5, "_some_info": "foo" }\0' | nc -w 1 HOSTNAME 12201
Is there anything in the logs suggesting the message is being dropped?
Where are you sending logs from, does networking/firewalls need to be considered.
Related
I have got an issue with my application, it logs request along with its query param which may contain sensitive data in access log. application is configured with logback.xml & embedded jetty.
jetty server is customized with below accessLogCustomer
public JettyServerCustomizer accessLogCustomizer() {
return server -> {
Slf4jRequestLog requestLog = new Slf4jRequestLog();
requestLog.setExtended(true);
requestLog.setLogLatency(true);
requestLog.setPreferProxiedForAddress(true);
requestLog.setLogTimeZone(userTimezone == null ? ZoneId.systemDefault().getId() : userTimezone);
requestLog.setLogDateFormat("Y-MM-dd HH:mm:ss, SSS Z");
RequestLogHandler requestLogHandler = new RequestLogHandler();
requestLogHandler.setRequestLog(requestLog);
requestLogHandler.setHandler(server.getHandler());
server.setHandler(requestLogHandler);
};
}
logback.xml
<appender name="access" class="ch.qos.logback.core.rolling.RollingFileAppender">
<File>${logs.dir}/abc-access.log</File>
<encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
<layout class="ch.qos.logback.classic.PatternLayout">
<Pattern>%m %n</Pattern>
</layout>
<charset>UTF-8</charset>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<FileNamePattern>${logs.dir}/abc-access.%d.log.gz</FileNamePattern>
</rollingPolicy>
</appender>
<logger name="org.eclipse.jetty.server.RequestLog" additivity="false">
<appender-ref ref="access"/>
</logger>
request logged in access log
192.168.0.100 - - [2021-05-20 15:48:15,093 +0530] "POST /myAPI/v2/customer/message?myID=123&messageText=hello HTTP/1.0" 200 0 "-" "PostmanRuntime/7.26.8" 475
I am trying to avoid messageText from access log, but not getting any solution.
Use the CustomRequestLog and Slf4jRequestLogWriter instead.
You'll want the special format option %U which emits the URL path, without the query string (which is available as %q btw)
Your resulting configuration would look like this ...
Slf4jRequestLogWriter slfjRequestLogWriter = new Slf4jRequestLogWriter();
String format = "%{client}a - %u %t %m \"%U\" %s %O \"%{Referer}i\" \"%{User-Agent}i\"";
CustomRequestLog customRequestLog = new CustomRequestLog(slfjRequestLogWriter, format);
server.setRequestLog(customRequestLog);
Play with the format line, read the Javadoc on CustomRequestLog to know what you can do.
Some notes:
The example format is not strictly following the Extended NCSA format (as it's missing the HTTP version portion, and the HTTP method is outside of the quoted section, but that is usually not a problem for many users)
Slf4jRequestLogWriter is only concerned with taking the formatted log line and sending it to the slf4j-api, it does nothing else.
RequestLogHandler is deprecated and not a recommended usage anymore (as it does not log bad requests and context-less requests), use the Server.setRequestLog(RequestLog) instead.
Jetty will use the CustomRequestLog's Pattern to produce a String, this String is forwarded to the Slf4jRequestLogWriter as a slf4j logging event message, which is then logged per your existing slf4j + logback configuration.
I'm trying to reproduce this tutorial : YouTube API, Version 3 on Rails
in order to apply it on my own project. But I'm having a hard with it since few days.
At first, I had this error :
A request to YouTube API caused an unexpected server error: To display
more verbose errors, change the configuration of Yt with: Yt.configure
do |config| config.log_level = :debug end
I updated RVM and Ruby and I'm getting this error now :
Yt::Errors::Forbidden in VideosController#create A request to YouTube
API was considered forbidden by the server: To display more verbose
errors, change the configuration of Yt with: Yt.configure do |config|
config.log_level = :debug end
I already :
get ruby and rvm updated
tried different version of the yt gem
tried that : OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE
tried that : config.force_ssl = false
this
curl -X GET -H "content-length: 0" -H "user-agent: Yt::Request (gzip)" -H "host: www.googleapis.com" "https://www.googleapis.com/youtube/v3/videos?id=wuZfOIWwM_Y&part=snippet"
return that :
Using Rails 4.2.4, Ruby 2.3.0;
Source code at : https://github.com/NeimadTL/YT_Sample_App
Any help, suggestions would be strongly and sincerely appreciated.
forbidden (403) forbidden Access forbidden. The request may not be properly authorized.
Answer: The request you are making is not authorized. update: Change key= to access_token=
Possible cause:
https://www.youtube.com/annotations_invideo?key=
You are trying to run a request annotations_invideo (which I cant actually find any were in the documentation) and you are applying an API key to it. API keys only work with public data. Either annotations_invideo is not a valid request to the API or its something that you need to be authenticated for. If you need to be authenticated then you will need an access token and then apply access_token= instead of key=
where exactly did you find annotations_invideo ?
Update:
Lucky for me it has been under an hour since you posted your question I was able to take
https://www.youtube.com/annotations_invideo?access_token=AIzaSyBSvIOM0EGX1tcrf5IAlYJuH_ttqVgTO4Q&video_id=BPNYv0vd78A
and dump it in a web browser it returned data.
<document>
<annotations>
<annotation author="" id="annotation_1585555999" log_data="ei=B2k9WIOCB8X0dNKokKAG&a-id=annotation_1585555999&xble=1&a-type=4&a-v=BPNYv0vd78A" style="title" type="text">
<TEXT>Hello, world!</TEXT>
<segment>
<movingRegion type="rect">
<rectRegion d="0" h="25.2779998779" t="0:00.000" w="75.0" x="13.1540002823" y="67.3239974976"/>
<rectRegion d="0" h="25.2779998779" t="0:02.089" w="75.0" x="13.1540002823" y="67.3239974976"/>
</movingRegion>
</segment>
<appearance bgAlpha="0.25" bgColor="0" borderAlpha="0.10000000149" effects="" fgColor="16777215" fontWeight="bold" highlightFontColor="16777215" textSize="21.6642"/>
</annotation>
<annotation id="channel:563d3ce4-0000-20cc-8fd5-001a11463304" style="playlist" type="promotion" log_data="ei=B2k9WIOCB8X0dNKokKAG&a-type=12&a-ch=UCwCnUcLcb9-eSrHa_RQGkQQ&xble=1&a-id=563d3ce4-0000-20cc-8fd5-001a11463304&l-class=2&link-id=PLuW4g7xujBWfU26JUTW1DGs3hk4LD5KaL&a-v=BPNYv0vd78A">
<data>
{"playlist_length":"200","session_data":{"itct":"CAIQwTcY____________ASITCMOh497wzdACFUU6HQodUhQEZCj4HTICaXZIwN_33vSX1vkE","annotation_id":"563d3ce4-0000-20cc-8fd5-001a11463304","feature":"iv","ei":"B2k9WIOCB8X0dNKokKAG","src_vid":"BPNYv0vd78A"},"is_mobile":false,"text_line_2":"Adorable Kids","text_line_1":"Check this playlist","image_url":"https:\/\/i.ytimg.com\/vi\/yDrLVqRHAsw\/mqdefault.jpg","start_ms":1000,"collapse_delay_ms":86400000,"end_ms":3000}
</data>
<segment/>
<action trigger="click" type="openUrl">
<url type="hyperlink" target="new" value="https://www.youtube.com/watch?v=yDrLVqRHAsw&list=PLuW4g7xujBWfU26JUTW1DGs3hk4LD5KaL"/>
</action>
</annotation>
</annotations>
</document>
Note: I wonder why this is returning XML and not Json it has me thinking this is an older api. Found it you are using the YouTube API v2 which is deprecated It should have been shut down .
https://youtube-eng.googleblog.com/2014/09/have-you-migrated-to-data-api-v3-yet.html
you should drop this and move to the YouTube API v3
So I saw there were a few other questions of this type, but none seemed to solve my issue.
I am attempting to take Springboot logs from files, parse out useful information, and send the result to Elasticsearch, and ultimately read from Kibana. My fluentd.conf looks like the following:
<source>
type tail
read_from_head true
path /path/to/log/
pos_file /path/to/pos_file
format /^(?<date>[0-9]+-[0-9]+-[0-9]+\s+[0-9]+:[0-9]+:[0-9]+.[0-9]+)\s+(?<log_level>[Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?)\s+(?<pid>[0-9]+)\s+---\s+(?<message>.*)$/
tag my.app
</source>
<match my.app>
type stdout
</match>
<match my.app>
type elasticsearch
logstash_format true
host myhosthere
port 9200
index_name fluentd-app
type_name fluentd
</match>
Given a typical Springboot log line:
2015-07-16 19:20:04.074 INFO 16649 --- [ main] {springboot message}
By also writing to stdout as a test, I see my parser is resulting in:
{
"date":"2015-07-16 19:20:04.074",
"log_level":"INFO",
"pid":"16649",
"message":"[ main] {springboot message}"
}
However, when this gets written to Elasticsearch, all that results is:
{
_index: "fluentd-app-2015.07.16",
_type: "fluentd",
_id: "AU6YT5sjvkxiJXWCxeM8",
_score: 1,
_source: {
message: "2015-07-16 19:20:04.074 INFO 16649 --- [ main] {springboot message}",
#timestamp: "2015-07-16T19:20:04+00:00"
}
},
From what I had read about fluentd-plugin-elasticsearch I expected _source to contain all of the parsed fields that I see in stdout. I have also tried the grok parser - though it seems apparent the issue lies with understanding of the fluentd elasticsearch plugin. How do I get the fields I parsed to persist to elasticsearch?
When calling phpunit on a very simple standard test, it produces the following output and error:
Command (in symfony2 root):
phpunit -c build/ src
Output:
PHPUnit 4.1.6-6-g43914fa by Sebastian Bergmann.
Configuration read from /srv/xxxxxxxxx/build/phpunit.xml
PHP Fatal error: Declaration of Symfony\Bridge\Doctrine\ContainerAwareEventManager::removeEventListener() must be compatible with Doctrine\Common\EventManager::removeEventListener($events, $listener = NULL) in /srv/xxxxxxxxx/vendor/symfony/symfony/src/Symfony/Bridge/Doctrine/ContainerAwareEventManager.php on line 141
PHP Stack trace:
PHP 1. {main}() /usr/local/composer/vendor/phpunit/phpunit/phpunit:0
PHP 2. PHPUnit_TextUI_Command::main() /usr/local/composer/vendor/phpunit/phpunit/phpunit:57
PHP 3. PHPUnit_TextUI_Command->run() /usr/local/composer/vendor/phpunit/phpunit/src/TextUI/Command.php:132
PHP 4. PHPUnit_TextUI_TestRunner->doRun() /usr/local/composer/vendor/phpunit/phpunit/src/TextUI/Command.php:179
PHP 5. PHPUnit_Framework_TestSuite->run() /usr/local/composer/vendor/phpunit/phpunit/src/TextUI/TestRunner.php:426
PHP 6. PHPUnit_Framework_TestSuite->run() /usr/local/composer/vendor/phpunit/phpunit/src/Framework/TestSuite.php:699
PHP 7. PHPUnit_Framework_TestCase->run() /usr/local/composer/vendor/phpunit/phpunit/src/Framework/TestSuite.php:699
PHP 8. PHPUnit_Framework_TestResult->run() /usr/local/composer/vendor/phpunit/phpunit/src/Framework/TestCase.php:760
PHP 9. PHPUnit_Framework_TestCase->runBare() /usr/local/composer/vendor/phpunit/phpunit/src/Framework/TestResult.php:686
PHP 10. PHPUnit_Framework_TestCase->runTest() /usr/local/composer/vendor/phpunit/phpunit/src/Framework/TestCase.php:826
PHP 11. ReflectionMethod->invokeArgs() /usr/local/composer/vendor/phpunit/phpunit/src/Framework/TestCase.php:962
PHP 12. Syw\Admin\CoreBundle\Tests\Controller\DebugControllerTest->testIndex() /usr/local/composer/vendor/phpunit/phpunit/src/Framework/TestCase.php:962
PHP 13. Symfony\Bundle\FrameworkBundle\Test\WebTestCase::createClient() /srv/xxxxxxxxx/src/Syw/Admin/CoreBundle/Tests/Controller/DebugControllerTest.php:13
PHP 14. Symfony\Component\HttpKernel\Kernel->boot() /srv/xxxxxxxxx/vendor/symfony/symfony/src/Symfony/Bundle/FrameworkBundle/Test/WebTestCase.php:47
PHP 15. Symfony\Component\HttpKernel\Kernel->initializeContainer() /srv/xxxxxxxxx/app/bootstrap.php.cache:2300
PHP 16. Symfony\Component\DependencyInjection\ContainerBuilder->compile() /srv/xxxxxxxxx/app/bootstrap.php.cache:2521
PHP 17. Symfony\Component\DependencyInjection\Compiler\Compiler->compile() /srv/xxxxxxxxx/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/ContainerBuilder.php:619
PHP 18. JMS\AopBundle\DependencyInjection\Compiler\PointcutMatchingPass->process() /srv/xxxxxxxxx/vendor/symfony/symfony/src/Symfony/Component/DependencyInjection/Compiler/Compiler.php:117
PHP 19. JMS\AopBundle\DependencyInjection\Compiler\PointcutMatchingPass->processInlineDefinitions() /srv/xxxxxxxxx/vendor/jms/aop-bundle/JMS/AopBundle/DependencyInjection/Compiler/PointcutMatchingPass.php:66
PHP 20. JMS\AopBundle\DependencyInjection\Compiler\PointcutMatchingPass->processDefinition() /srv/xxxxxxxxx/vendor/jms/aop-bundle/JMS/AopBundle/DependencyInjection/Compiler/PointcutMatchingPass.php:85
PHP 21. class_exists() /srv/xxxxxxxxx/vendor/jms/aop-bundle/JMS/AopBundle/DependencyInjection/Compiler/PointcutMatchingPass.php:110
PHP 22. spl_autoload_call() /srv/xxxxxxxxx/vendor/jms/aop-bundle/JMS/AopBundle/DependencyInjection/Compiler/PointcutMatchingPass.php:110
PHP 23. Composer\Autoload\ClassLoader->loadClass() /srv/xxxxxxxxx/vendor/jms/aop-bundle/JMS/AopBundle/DependencyInjection/Compiler/PointcutMatchingPass.php:0
PHP 24. Composer\Autoload\includeFile() /usr/local/composer/vendor/composer/ClassLoader.php:301
This is the phpunit.xml in [symfony-root]/build:
<?xml version="1.0" encoding="UTF-8"?>
<!-- http://www.phpunit.de/manual/current/en/appendixes.configuration.html -->
<phpunit
backupGlobals = "false"
backupStaticAttributes = "false"
colors = "true"
convertErrorsToExceptions = "true"
convertNoticesToExceptions = "true"
convertWarningsToExceptions = "true"
processIsolation = "false"
stopOnFailure = "false"
syntaxCheck = "false"
bootstrap = "../app/bootstrap.php.cache" >
<php>
<server name="KERNEL_DIR" value="app/" />
</php>
<testsuites>
<testsuite name="Onlinexxxxxxxxx Test Suite">
<directory>src/*/*Bundle/Tests</directory>
<directory>src/*/*/*Bundle/Tests</directory>
</testsuite>
</testsuites>
<filter>
<whitelist>
<directory>src</directory>
<exclude>
<directory>src/*/*Bundle/Resources</directory>
<directory>src/*/*Bundle/Tests</directory>
<directory>src/*/*/*Bundle/Resources</directory>
<directory>src/*/*/*Bundle/Tests</directory>
<directory>src/*/Bundle/*Bundle/Resources</directory>
<directory>src/*/Bundle/*Bundle/Tests</directory>
</exclude>
</whitelist>
<blacklist>
<directory>src/*/*Bundle/Resources</directory>
<directory>src/*/*Bundle/Tests</directory>
<directory>src/*/*/*Bundle/Resources</directory>
<directory>src/*/*/*Bundle/Tests</directory>
<directory>src/*/Bundle/*Bundle/Resources</directory>
<directory>src/*/Bundle/*Bundle/Tests</directory>
</blacklist>
</filter>
<logging>
<log type="coverage-html" target="../build/coverage" title="GMS" charset="UTF-8" yui="true" highlight="true"
lowUpperBound="35" highLowerBound="70"/>
<log type="coverage-clover" target="../build/logs/clover.xml"/>
<log type="junit" target="../build/logs/junit.xml" logIncompleteSkipped="false"/>
</logging>
</phpunit>
And this is the very simple Test class:
<?php
namespace Syw\Admin\CoreBundle\Tests\Controller;
use Symfony\Bundle\FrameworkBundle\Test\WebTestCase;
class DebugControllerTest extends WebTestCase
{
public function testIndex()
{
$client = static::createClient();
$crawler = $client->request('GET', '/debug');
}
public function testDetail()
{
$client = static::createClient();
$crawler = $client->request('GET', '/detail');
}
}
Any help with this would be highly aprecciated.
By the way, the
Symfony\Bridge\Doctrine\ContainerAwareEventManager::removeEventListener()
and the
Doctrine\Common\EventManager::removeEventListener($events, $listener = NULL)
are of course still the original ones from the bundles/Symfony2.
Generally nothing in symfony2 or the corebundles were changed, just only our own bundle.
By the way (#2): We are using Symfony 2.4 (but I've just also tried to upgrade to Symfony 2.6 which didn't solve this problem)
Edit:
Like mentioned in the comments below, I've also tried the following things:
Upgrade Symfony from 2.4 to 2.6 -> no success, same error
Downgrade some doctrine bundles -> no success, same error
The composer.json actually looks this way (for the requirements):
"require": {
"ali/datatable": "dev-master",
"doctrine/data-fixtures": "dev-master",
"doctrine/doctrine-bundle": "~1.2",
"doctrine/doctrine-fixtures-bundle": "dev-master",
"doctrine/doctrine-migrations-bundle": "2.1.*#dev",
"doctrine/migrations": "1.0.*#dev",
"doctrine/orm": "~2.2,>=2.2.3",
"dropbox/dropbox-sdk": "1.1.*",
"friendsofsymfony/rest-bundle": "#dev",
"gedmo/doctrine-extensions": "dev-master",
"google-api-php-client": "0.6.7",
"hybridauth/hybridauth": "3.0.0.*#dev",
"incenteev/composer-parameter-handler": "~2.0",
"jms/i18n-routing-bundle": "dev-master",
"jms/security-extra-bundle": "dev-master",
"jms/serializer-bundle": "#dev",
"knplabs/knp-paginator-bundle": "dev-master",
"languagedetect": "1.0",
"languagetool": "2.8",
"lexik/maintenance-bundle": "dev-master",
"misd/guzzle-bundle": "~1.0",
"nelmio/api-doc-bundle": "#dev",
"nelmio/cors-bundle": "~1.0",
"php": ">=5.3.3",
"sensio/distribution-bundle": "2.3.*",
"sensio/framework-extra-bundle": "~3.0",
"sensio/generator-bundle": "~2.3",
"stfalcon/tinymce-bundle": "dev-master",
"stof/doctrine-extensions-bundle": "~1.1#dev",
"symfony/assetic-bundle": "2.3.*",
"symfony/monolog-bundle": "2.3.*",
"symfony/swiftmailer-bundle": "2.3.*",
"symfony/symfony": "2.4.*",
"twig/extensions": "1.0.*"
},
Okay... after many, many hours of testing and trying with downgrades, upgrades and all these things, I've now found out, that the problem simply is phpunit itself.
I had "PHPUnit 4.1.6-6-g43914fa" installed, as you can see in my output above.
I've now downgraded to 3.7.13 ("phpunit/phpunit": "~3.6") and this is working without any problems.
I have a site that runs using moderngov.co.uk (you send them a template, which they then upload). I'm trying to crawl this site so it can indexed by Solr and searched through a drupal site. I can crawl the vast majority of websites out there, but for some reason I am unable to crawl this one: http://scambs.moderngov.co.uk/uuCoverPage.aspx?bcr=1
The specific error I get is this:
Injector: starting at 2013-10-17 13:32:47
Injector: crawlDb: X-X/crawldb
Injector: urlDir: urls/seed.txt
Injector: Converting injected urls to crawl db entries.
Injector: total number of urls rejected by filters: 1
Injector: total number of urls injected after normalization and filtering: 0
Injector: Merging injected urls into crawl db.
Injector: finished at 2013-10-17 13:32:50, elapsed: 00:00:02
Thu, Oct 17, 2013 1:32:50 PM : Iteration 1 of 2
Generating a new segment
Generator: starting at 2013-10-17 13:32:51
Generator: Selecting best-scoring urls due for fetch.
Generator: filtering: false
Generator: normalizing: true
Generator: topN: 50000
Generator: 0 records selected for fetching, exiting ...
I'm not sure if it's got something to do with the regex patterns Nutch uses to parse html, or if there's a redirect that's causing issues, or something else entirely. Below are a few of the nutch config files:
Here are the urlfilters: http://pastebin.com/ZqeZUJa1
sysinfo:
Windows 7 (64-bit)
Solr 3.6.2
Apache Nutch 1.7
If anyone has come across this problem before, or might know why this is happening, any help would be greatly appreciated.
Thanks
I tried that seed url and I got this error:
Denied by robots.txt: http://scambs.moderngov.co.uk/uuCoverPage.aspx?bcr=1
Looking at the robots.txt file of that site:
# Disallow all webbot searching
User-agent: *
Disallow: /
You have to set a specific user agent in Nutch and modify the website to accept crawling form your user agent.
The property to change in Nutch is in conf/nutch-site.xml:
<property>
<name>http.agent.name</name>
<value>nutch</value>
</property>
try this
<property>
<name>db.fetch.schedule.class</name>
<value>org.apache.nutch.crawl.AdaptiveFetchSchedule</value>
</property>
<property>
<name>db.fetch.interval.default</name>
<value>10</value>
<description>The default number of seconds between re-fetches of a page (30 days).
</description>
</property>
<property>
<name>db.fetch.interval.max</name>
<!-- for now always re-fetch everything -->
<value>100</value>
<description>The maximum number of seconds between re-fetches of a page
(less than one day). After this period every page in the db will be re-tried, no
matter what is its status.
</description>
</property>