I have a pretty simple application which send tcp log to promtail. (So promtail doesn't gaver data from file)
The problem is that the last log sent to protail isn't stored in loki until another log comes.
Let's say that I logged "foo", I'll see nothing in grafana. Then if I log "bar" to promtail, I'll only see foo in grafana, then if I log another "foo", I'll see only "foo" and "bar"...
I have this architecture :
Dockerised C++ app which use spdlog lib to log through tcp;
Dockerised promtail which only gaver from tcp;
Dockerised loki and grafana both are vanilla versions
My log format is the following :
<165>4 %Y-%m-%dT%H:%M:%S.%eZ MY_APPLICATION MY_PROJECT - 1 - %v,
The %v is composed of various str that I build in one before loging.
Promtail doesn't show any errors.
If there are some network nerds, when I ctrl+c (end) my app, the log bufferised in promtail is finaly stored in Loki and visible in grafana. I might think that my logs lack a kind of "end of log" sign to have this to happen...
Thanks for the help, any suggestion or even question will be appreciated !
(Sorry for my not so good english)
Here is some code to reproduce the problem :
C++ app
static std::shared_ptr<spdlog::logger> tcp_logger_ptr;
void init_logger() {
spdlog::init_thread_pool(4096, 1);
auto tcp_sink_config = spdlog::sinks::tcp_sink_config(172.17.0.1, 1514);
tcp_logger_ptr = spdlog::create_async<spdlog::sinks::tcp_sink_mt>("TCP_logger", tcp_sink_config);
tcp_logger_ptr->set_pattern(PROMTAIL_LOG_PATTERN);
}
Then to log
void tcp_log(spdlog::level::level_enum log_type, const char* log_message) {
tcp_logger_ptr->log(log_type, log_message);
}
Promtail docker config
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: syslog
syslog:
listen_address: 0.0.0.0:1514
idle_timeout: 60s
label_structured_data: yes
labels:
job: "syslog"
relabel_configs:
- source_labels: ['__syslog_message_hostname']
target_label: 'host'
Ok thanks everyone for the help ! (lol)
I fugured it out, and it is pretty disappointing to be honnest...
The answer was the idle_timeout: 60s
It just says to promtail that if it has nothing to do for 60s, it flush what it got stored (my log in this case). So you see it comming, with an idle timeout of 0.1 the fonctionning is pretty good.
Recently I add a broadcast upload extension to my host app to implement system wide screen cast.I found the broadcast upload extension sometimes stopped for unknown reason.If I debug the broadcast upload extension process in Xcode, it stopped without stopping at a breakpoint（If the extension is killed for 50M bytes memory limit, it will stopped at a breakpoint, and Xcode will point out that it's killed for 50M bytes memory limit）.For more imformation, I read the console log line by line.Finally, I found a significant line:
osanalyticshelper Process replayd [26715] killed by jetsam reason highwater
It looks like the ReplayKit serving process 'replayd' is killed by jetsam, and the reason is 'highwater'.So I searched the internet for more imformation.And I found a post:
https://www.jianshu.com/p/30f24bb91222
After reading that,I checked the JetsamEvent report in device, and found that when the 'replayd' process was killed it occupied 100M bytes memory.Is there a 100M bytes memory limit for 'replayd' process?How can I avoid it to occupy more than 100M bytes memory?
Further more, I found that this problem offen occured if the previous extension process is stopped via RPBroadcastSampleHandler's finishBroadcastWithError method.If I stop the extension via control center button, this rarely occured.
As comparison, when the 'Wemeet' app stop it's broadcast upload extension, it raraly cause this problem.I compared the console log when 'Wemeet' stop it's broadcast extension and the log my app stop it's broadcast extension.I found this line is different:
Wemeet:
mediaserverd MEDeviceStreamClient.cpp:429 AQME Default-InputOutput: client stopping: <ZenAQIONodeClient#0x1080f7a40, sid:0x3456e, replayd(30213), 'prim'>; running count now 0
My app:
mediaserverd MEDeviceStreamClient.cpp:429 AQME Default-InputOutput: client stopping: <ZenAQIONodeClient#0x107e869a0, sid:0x3464b, replayd(30232), 'prim'>; running count now 3
As we can see, the 'running count' is different.
To debug a locked file problem, we're calling SysInternal's Handle64.exe 4.11 from a .NET process (via Process.Start with asynchronous output redirection). The calling process hangs on Process.WaitForExit because the Handle64 process doesn't exit (for more than two hours).
We took a dump of the corresponding Handle64 process and checked it in the Visual Studio 2017 debugger. It shows two threads ("Main Thread" and "ntdll.dll!TppWorkerThread").
Main thread's call stack:
ntdll.dll!NtWaitForSingleObject () Unknown
ntdll.dll!LdrpDrainWorkQueue() Unknown
ntdll.dll!RtlExitUserProcess() Unknown
kernel32.dll!ExitProcessImplementation () Unknown
handle64.exe!000000014000664c() Unknown
handle64.exe!00000001400082a5() Unknown
kernel32.dll!BaseThreadInitThunk () Unknown
ntdll.dll!RtlUserThreadStart () Unknown
Worker thread's call stack:
ntdll.dll!NtWaitForSingleObject() Unknown
ntdll.dll!LdrpDrainWorkQueue() Unknown
ntdll.dll!LdrpInitializeThread() Unknown
ntdll.dll!_LdrpInitialize() Unknown
ntdll.dll!LdrInitializeThunk() Unknown
My question is: Why would a process hang in LdrpDrainWorkQueue? From https://stackoverflow.com/a/42789684/62838, I gather that this is the Windows 10 parallel loader at work, but why would it get stuck while exiting the process? Can this be caused by how we invoke Handle64 from another process? I.e., are we doing something wrong or is this rather a bug in Handle64?
How long did you wait?
According to this analysis,
The worker thread idle timeout is set to 30 seconds. Programs which
execute in less than 30 seconds will appear to hang due to
ntdll!TppWorkerThread waiting for the idle timeout before the process
terminates.
I would recommend trying to set the registry key specified in that article to disable the parallel loader and see if this resolved the issue.
Parent Key: HKLM\Software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\handle64.exe
Value Name: MaxLoaderThreads
Type: DWORD
Value: 1 to disable
I'm facing a problem due to a BAD table design in HBase. The problem is millions of records end up under the same row key (1 cf). Till 2.5M I was able to run mapReduces with Spark by scanning a single row, but now some of the rows are reaching 5 or 6 millions and whenever I perform a scan or get, all my regionservers are down within a couple of minutes. I'm working with HDP 2.2 and HBase 0.98.4.2.2
So far I've tried:
confPoints.setInt("hbase.rpc.timeout",6000000)
...
scanPoints.setBatch(1000)
Before creating a new table with new rowkey design I really need to process this data. I'm new to HBase so maybe some of this suggestions sound stupid but:
Would increasing java heap size help in any way?
Is there a possibility of splitting a row into 2 or more rows?
Can I run a MapReduce over the raw stored data in HDFS without passing through HBase?
Any other idea?
Thanks!
EDITED:
Actually, the second option I think it is not feasible cause hbase doesn't let to update records, just delete + create again.
EDITED 2:
Each record in a row is about tens of bytes. The problem when having millions of records per row is when trying to scan this kind of row is that after a couple of minutes region servers start to go down one by one. Maybe trying to get a row of 512MB aprox. is too big for my cluster configuration: 6 nodes of 8GB each.
Searching in the HBase logs the only exception I can find is this on:
2015-08-25 15:07:19,722 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-0] handler.OpenRegionHandler: Opened my-hbase-table,20150807.33,1439222912086.e731d603bb5d1f0d593736eab922069c. on ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
2015-08-25 15:07:19,724 INFO [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] regionserver.HRegion: Replaying edits from hdfs://ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:8020/apps/hbase/data/data/default/my-hbase-table/3bc481ff534f0907e6b99d5eff1793f5/recovered.edits/0000000000011099011
2015-08-25 15:07:19,725 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-2] zookeeper.ZKAssign: regionserver:60020-0x24f65d7e5df025c, quorum=ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XXX.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XX3.eu-west-1.compute.internal:2181, baseZNode=/hbase-unsecure Transitioned node 4945982779c1cba7b1726e77a45d405a from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
2015-08-25 15:07:19,725 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-2] handler.OpenRegionHandler: Transitioned 4945982779c1cba7b1726e77a45d405a to OPENED in zk on ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
2015-08-25 15:07:19,726 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-2] handler.OpenRegionHandler: Opened my-hbase-table,20150727.33,1438203991635.4945982779c1cba7b1726e77a45d405a. on ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
2015-08-25 15:07:19,733 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] zookeeper.ZKAssign: regionserver:60020-0x24f65d7e5df025c, quorum=ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XXX.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XX3.eu-west-1.compute.internal:2181, baseZNode=/hbase-unsecure Attempting to retransition opening state of node 3bc481ff534f0907e6b99d5eff1793f5
2015-08-25 15:07:19,734 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] regionserver.HRegion: Applied 0, skipped 1, firstSequenceidInLog=11099011, maxSequenceidInLog=11099011, path=hdfs://ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:8020/apps/hbase/data/data/default/my-hbase-table/3bc481ff534f0907e6b99d5eff1793f5/recovered.edits/0000000000011099011
2015-08-25 15:07:19,734 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] regionserver.HRegion: Empty memstore size for the current region my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5.
2015-08-25 15:07:19,737 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] regionserver.HRegion: Deleted recovered.edits file=hdfs://ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:8020/apps/hbase/data/data/default/my-hbase-table/3bc481ff534f0907e6b99d5eff1793f5/recovered.edits/0000000000011099011
2015-08-25 15:07:19,759 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] wal.HLogUtil: Written region seqId to file:hdfs://ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:8020/apps/hbase/data/data/default/my-hbase-table/3bc481ff534f0907e6b99d5eff1793f5/recovered.edits/11099013_seqid ,newSeqId=11099013 ,maxSeqId=11099010
2015-08-25 15:07:19,761 INFO [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] regionserver.HRegion: Onlined 3bc481ff534f0907e6b99d5eff1793f5; next sequenceid=11099013
2015-08-25 15:07:19,764 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] zookeeper.ZKAssign: regionserver:60020-0x24f65d7e5df025c, quorum=ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XXX.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XX3.eu-west-1.compute.internal:2181, baseZNode=/hbase-unsecure Attempting to retransition opening state of node 3bc481ff534f0907e6b99d5eff1793f5
2015-08-25 15:07:19,773 INFO [PostOpenDeployTasks:3bc481ff534f0907e6b99d5eff1793f5] regionserver.HRegionServer: Post open deploy tasks for region=my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5.
2015-08-25 15:07:19,773 DEBUG [PostOpenDeployTasks:3bc481ff534f0907e6b99d5eff1793f5] regionserver.CompactSplitThread: Small Compaction requested: system; Because: Opening Region; compaction_queue=(0:1), split_queue=0, merge_queue=0
2015-08-25 15:07:19,774 DEBUG [regionserver60020-smallCompactions-1440529300855] compactions.RatioBasedCompactionPolicy: Selecting compaction from 4 store files, 0 compacting, 4 eligible, 10 blocking
2015-08-25 15:07:19,774 DEBUG [regionserver60020-smallCompactions-1440529300855] compactions.ExploringCompactionPolicy: Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 3 permutations with 0 in ratio
2015-08-25 15:07:19,774 DEBUG [regionserver60020-smallCompactions-1440529300855] compactions.RatioBasedCompactionPolicy: Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate.
2015-08-25 15:07:19,775 DEBUG [regionserver60020-smallCompactions-1440529300855] regionserver.CompactSplitThread: Not compacting my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5. because compaction request was cancelled
2015-08-25 15:07:19,787 INFO [PostOpenDeployTasks:3bc481ff534f0907e6b99d5eff1793f5] catalog.MetaEditor: Updated row my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5. with server=ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
2015-08-25 15:07:19,787 INFO [PostOpenDeployTasks:3bc481ff534f0907e6b99d5eff1793f5] regionserver.HRegionServer: Finished post open deploy task for my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5.
2015-08-25 15:07:19,788 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] zookeeper.ZKAssign: regionserver:60020-0x24f65d7e5df025c, quorum=ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XXX.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XX3.eu-west-1.compute.internal:2181, baseZNode=/hbase-unsecure Transitioning 3bc481ff534f0907e6b99d5eff1793f5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
2015-08-25 15:07:19,791 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] zookeeper.ZKAssign: regionserver:60020-0x24f65d7e5df025c, quorum=ip-XXX-XX-XX-XX2.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XXX.eu-west-1.compute.internal:2181,ip-XXX-XX-XX-XX3.eu-west-1.compute.internal:2181, baseZNode=/hbase-unsecure Transitioned node 3bc481ff534f0907e6b99d5eff1793f5 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED
2015-08-25 15:07:19,791 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] handler.OpenRegionHandler: Transitioned 3bc481ff534f0907e6b99d5eff1793f5 to OPENED in zk on ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
2015-08-25 15:07:19,791 DEBUG [RS_OPEN_REGION-ip-XXX-XX-XX-XXX:60020-1] handler.OpenRegionHandler: Opened my-hbase-table,20150824.33,1440473855617.3bc481ff534f0907e6b99d5eff1793f5. on ip-XXX-XX-XX-XXX.eu-west-1.compute.internal,60020,1440528949321
2015-08-25 15:07:20,344 INFO [B.DefaultRpcServer.handler=3,queue=3,port=60020] regionserver.HRegionServer: Client tried to access missing scanner 1
2015-08-25 15:07:20,346 DEBUG [B.DefaultRpcServer.handler=3,queue=3,port=60020] ipc.RpcServer: B.DefaultRpcServer.handler=3,queue=3,port=60020: callId: 36 service: ClientService methodName: Scan size: 25 connection: 172.31.40.100:42285
org.apache.hadoop.hbase.UnknownScannerException: Name: 1, already closed?
at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3150)
at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94)
at java.lang.Thread.run(Thread.java:745)
EDITED 3:
I've tried to do range scans within a row with a ColumnRangeFilter and it is working without putting down any region servers:
scan 'my-table', {STARTROW=>'row-key',ENDROW=>'row-key', FILTER=> ColumnRangeFilter.new(Bytes.toBytes('first_possible_column_prefix'),true,Bytes.toBytes('another_possible_column_prefix’),false)}
This code in Spark though is putting the region servers down, same behavior as before:
val scanPoints = new Scan()
scanPoints.setStartRow((queryDate+"."+venueId).getBytes())
scanPoints.setStopRow((queryDate+"."+venueId+"1").getBytes())
scanPoints.setFilter(new ColumnRangeFilter(Bytes.toBytes("first_possible_column_prefix"),true,Bytes.toBytes("another_possible_column_prefix"),false))
...
val confPoints = HBaseConfiguration.create()
confPoints.set(TableInputFormat.INPUT_TABLE, Utils.settings.HBaseWifiVisitorsTableName)
confPoints.set("hbase.zookeeper.quorum", Utils.settings.zQuorum);
confPoints.setInt("zookeeper.session.timeout", 6000000)
confPoints.set("hbase.zookeeper.property.clientPort", Utils.settings.zPort);
confPoints.set("zookeeper.znode.parent",Utils.settings.HBaseZNode)
confPoints.set("hbase.master", Utils.settings.HBaseMaster)
confPoints.set("hbase.mapreduce.scan.column.family","positions")
confPoints.setLong("hbase.client.scanner.max.result.size",2147483648L)
confPoints.setLong("hbase.server.scanner.max.result.size",2147483648L)
confPoints.setInt("hbase.rpc.timeout",6000000)
confPoints.setInt("hbase.client.operation.timeout",6000000)
confPoints.set(TableInputFormat.SCAN, convertScanToString(scanPoints))
...
val rdd = sc.newAPIHadoopRDD(confPoints, classOf[TableInputFormat], classOf[org.apache.hadoop.hbase.io.ImmutableBytesWritable], classOf[org.apache.hadoop.hbase.client.Result]).cache()
If I would be able to make this Spark Job to work, I could iterate through the whole row scanning intervals to process it entirely.
I've started using Stackshot utility in order to debug my executable and got the log below. The command i used is : sudo /usr/libexec/stackshot -i -p [pid]
However, I don't see an easy way to interpret the function backtrace into a readable format of function_name / line_number (similar to the equivalent command in lldb/gdb).
Thread ID: 0x130
Thread state: 0x9 == TH_WAIT|TH_UNINT
Thread priority: 0x52/0x52
Thread wait_event: 0xffffff8026cd77a8
Kernel Stack:
Return Frame
0xffffff80002d8452 0xffffff80005e3e45
0xffffff800023695b 0xffffff80005e4072
0xffffff8000235d5c 0xffffff800042f6e4
0xffffff800022d18e 0xffffff80002d7607
0xffffff80005e3e45 0x00000008feedface
0xffffff80005e4072 0x00000000ffffff80
0xffffff800042f6e4 0x0000000000000000
0xffffff80002d7607 0x0000000900000000