Flume takes time to upload a file to HDFS - hdfs

Question

I need to get your assistance with regard to checking why does flume take time to upload flatfiles to HDFS. I tried uploading just 1 file (10MB size) however, 17 hours has past it's still uploading with ".tmp". When I checked the Log Details, it seems like it's stuck in the Channel:
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-1
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-2
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-3
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-4
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-5
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-6
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-7
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-8
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.EventQueueBackingStoreFile CheckpointBackupCompleted
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-9
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-10
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-11
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-12
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-13
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-14
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-15
Nov 10, X:XX:XX.XXX PM INFO org.apache.flume.channel.file.LogFile Closing RandomReader /data5/flumedata/log-16
Here is the configuration:
agent.sources = source1
agent.channels = channel1
agent.sinks = sinks1
agent.sources.source1.type = spooldir
agent.sources.source1.spoolDir = /data1/forupload
agent.sources.source1.channels = channel1
agent.sources.source1.basenameHeader = true
agent.channels.channel1.type = file
agent.channels.channel1.capacity = 1000000
agent.channels.channel1.transactionCapacity = 10000
agent.channels.channel1.checkpointDir = /data5/checkpoint
agent.channels.channel1.dataDirs = /data5/flumedata
agent.channels.channel1.useDualCheckpoints = true
agent.channels.channel1.backupCheckpointDir = /data5/backupcheckpoint
agent.channels.channel1.maxFileSize = 900000000
agent.sinks.sinks1.type = hdfs
agent.sinks.sinks1.hdfs.path = /user/flume
agent.sinks.sinks1.hdfs.filetype = DataStream
agent.sinks.sinks1.channel = channel1
agent.sinks.sinks1.hdfs.filePrefix = %{basename}
agent.sinks.sinks1.hdfs.fileSuffix = .csv
agent.sinks.sinks1.hdfs.rollInterval = 0
agent.sinks.sinks1.hdfs.rollSize = 0
agent.sinks.sinks1.hdfs.rollCount = 0
Appreciate your help with this

Answer 1

I think all data has been sent. You can check if the file you want to send has been removed to file.name.COMPLETED. If it has been removed, the file should be already sent.
However, there might be some data that's still in the file channel, since data is transmitted in batch. If the size of data left is less than the batch size, it will be left in the channel.
In order to finish sending, you can use kill -SIGTERM flume_process_id to kill the process. When flume receives this signal, it flushes all data left to HDFS. And the file on HDFS will be renamed, i.e. remove the .tmp suffix.