We have a hourly bucketed StreamingFileSink (to HDFS) where records are relatively infrequent.
Is there a way to configure Flink to flush records to in-progress file as soon as they arrive (less than 1 minute), instead of Flink keeping them in buffer?
The requirement is for successor data analysis process to read in-progress file near-real time.
I know how to shorten InactivityInterval but it results in too many small files in the end, which is undesirable.
May be you can look at the implementation of the write function. There are several implementations which will write the data to the hdfs file in real time.StreamingFileSink itself will not keep them in buffer, but there will be some buffers inside the FileOutputStream
public interface InProgressFileWriter<IN, BucketID> extends PartFileInfo<BucketID> {
/**
* Write a element to the part file.
*
* #param element the element to be written.
* #param currentTime the writing time.
* #throws IOException Thrown if writing the element fails.
*/
void write(final IN element, final long currentTime) throws IOException;
/**
* #return The state of the current part file.
* #throws IOException Thrown if persisting the part file fails.
*/
InProgressFileRecoverable persist() throws IOException;
/**
* #return The state of the pending part file. {#link Bucket} uses this to commit the pending
* file.
* #throws IOException Thrown if an I/O error occurs.
*/
PendingFileRecoverable closeForCommit() throws IOException;
/** Dispose the part file. */
void dispose();
// ------------------------------------------------------------------------
/** A handle can be used to recover in-progress file.. */
interface InProgressFileRecoverable extends PendingFileRecoverable {}
/** The handle can be used to recover pending file. */
interface PendingFileRecoverable {}
}
StreamingFileSink time data is written to HDFS in real time and will not keep them in buffer
In order to support exactly once semantics, the in-progress file will only be renamed into a formal file during checkpoint and then you can do successor data analysis
StreamingFileSink does not support real-time data query, you can reduce the interval of cp to improve the real-time performance of data visibility， but if the cp interval is too small, it will easily lead to small file problems
Have you considered the enable the file compaction option? https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#file-compaction
Related
I am trying to immediately jump to the DFU bootloader via software on the STM32F405RG, but instead it is doing a software reset and the RCC_FLAG_SFTRST flag is thrown. The program continues to repeatedly execute the code for jumping to bootloader and finally stays in bootloader mode after multiple seconds. I experienced success with entering bootloader mode on the first try if interrupts are disabled. However, I cannot upload code while in DFU mode since it requires interrupts to be enabled. I am unsure what is causing the bootloader to jump back to the application and was hoping to get assistance on this. Below is the code for jumping to bootloader, which is called immediately after Init_HAL() under main.
void JumpToBootloader(void) {
void (*SysMemBootJump)(void);
volatile uint32_t addr = 0x1FFF0000;
HAL_RCC_DeInit();
HAL_DeInit();
/**
* Step: Disable systick timer and reset it to default values
*/
SysTick->CTRL = 0;
SysTick->LOAD = 0;
SysTick->VAL = 0;
/**
* Step: Interrupts are not disabled since they are needed for DFU mode
*/
// __disable_irq();
/**
* Step: Remap system memory to address 0x0000 0000 in address space
* For each family registers may be different.
* Check reference manual for each family.
*
* For STM32F4xx, MEMRMP register in SYSCFG is used (bits[1:0])
* For STM32F0xx, CFGR1 register in SYSCFG is used (bits[1:0])
* For others, check family reference manual
*/
__HAL_SYSCFG_REMAPMEMORY_SYSTEMFLASH(); //Call HAL macro to do this for you
/**
* Step: Set jump memory location for system memory
* Use address with 4 bytes offset which specifies jump location where program starts
*/
SysMemBootJump = (void (*)(void)) (*((uint32_t *)(addr + 4)));
/**
* Step: Set main stack pointer.
* This step must be done last otherwise local variables in this function
* don't have proper value since stack pointer is located on different position
*
* Set direct address location which specifies stack pointer in SRAM location
*/
__set_MSP(*(uint32_t *)addr);
/**
* Step: Actually call our function to jump to set location
* This will start system memory execution
*/
SysMemBootJump();
}
Taking a look at the Application Note AN2606, Rev.55, chapter 28 STM32F40xxx/41xxx devices bootloader, page 127, one can see that the first step is called "System Reset".
The DFU is not designed to be executed from your application, but to be called according to the bootloader selection after a system reset. Before you are even entering the main function of your application, a few steps are already executed to prepare the device. These steps may be different for the DFU. Calling the DFU from an unexpected system state can lead to undefined behaviour.
The best way to enter the DFU is to latch the BOOT0 pin high and then perform a system reset either by software reset, or by IWDG.
Another potential problem can arise from the HSE, which has to be provided externally, however since you stated that the DFU does eventually work, this seems unlikely.
I'm working on a S32K148 with VS Code and J-Link. This is part of NXP's S32Kxxx series of 32-bit ARM Cortex M4-based MCUs targeted for high-reliability automobile and industrial applications.
I want to flash the chip using JFlash (with J-Link), but it seems that flashing has been disabled.
My research suggests that I need to have a LinkScript file for the S32Kxxx device, but I cannot find such a file anywhere.
Is my assumption correct that a LinkScript file is needed? If so, where can I find this file?
First, you need to ensure that the MCU's reset pin is connected to the reset pin of the J-Link. Without this done properly, you will not be able to connect via J-Link.
Next, it must be noted that the design of the S32Kxxx series makes it impossible to attach to a debugging session via J-Link out of the box. The Segger wiki contains a more detailed description:
ECC protected internal RAM
The device series provide ECC protected internal RAM. By default, J-Link resets the MCU on connect and initializes the RAM contents to 0x00. This is done for the following reasons:
If a memory window in the debugger is open during the debug session and points to a non-initialized RAM region, on the next single step etc. a non-maskable ECC error interrupt would be thrown
J-Link temporarily uses some portions of the RAM during flash programming and accesses to non-initialized RAM areas would throw non maskable ECC error interrupts
Attach to debug session
For the reasons mentioned [above, in the section "ECC protected internal RAM"], no out of the box attach is possible. This is because there is no way to distinguish between an attach and a connect.
To attach to the device the following J-Link script file can be used:
File:NXP_Kinetis_S32_Attach.JLinkScript
Note:
This script file is only supposed to be used in case of an attach, as it skips the device specific connect.
So, your research is correct: in order to attach to the device, you must use a J-Link script, which can be downloaded here from Segger. The contents are reprinted below:
* (c) SEGGER Microcontroller GmbH & Co. KG *
* The Embedded Experts *
* www.segger.com *
**********************************************************************
-------------------------- END-OF-HEADER -----------------------------
File : NXP_Kinetis_S32_Attach.JLinkScript
Purpose : Script file to skip the device specific connect for the
NXP Kinetis S32 devices, to make an attach possible.
Literature:
[1] J-Link User Guide
*/
/*********************************************************************
*
* Constants (similar to defines)
*
**********************************************************************
*/
/*********************************************************************
*
* Global variables
*
**********************************************************************
*/
/*********************************************************************
*
* Local functions
*
**********************************************************************
*/
/*********************************************************************
*
* Global functions
*
**********************************************************************
*/
/*********************************************************************
*
* InitTarget()
*
* Function description
* If present, called right before performing generic connect sequence.
* Usually used for targets which need a special connect sequence.
* E.g.: TI devices with ICEPick TAP on them where core TAP needs to be enabled via specific ICEPick sequences first
*
* Return value
* >= 0: O.K.
* < 0: Error
*
* Notes
* (1) Must not use high-level API functions like JLINK_MEM_ etc.
* (2) For target interface JTAG, this device has to setup the JTAG chain + JTAG TAP Ids.
*/
int InitTarget(void) {
//
// As this script file is only used for attach purposes, no special handling is required.
// Therefore, we override the default connect handled by the J-Link DLL for this device.
// as it would trigger a reset.
//
return 0;
}
/*************************** end of file ****************************/
I am developing a gstreamer application (plugin) that sinks from a video stream, analyzes each buffer for a certain condition and then if that condition is present passes the buffer to the plugin source. If the condition is not present for a given buffer, the buffer should be dropped and the plugin source should not receive this buffer.
Upon looking through the gstreamer documentation and tutorials as a newcomer to gstreamer, I cannot find a way for my plugin to "drop" a buffer.
Try using GstProbe for data buffers and return GST_PAD_PROBE_DROP or GST_PAD_PROBE_HANDLED when yours conditions are met.
If your plugin is based on GstBaseTransform, you should implement your own transform_frame_ip or transform_frame. If so, you can return GST_BASE_TRANSFORM_FLOW_DROPPED:
/**
* GST_BASE_TRANSFORM_FLOW_DROPPED:
*
* A #GstFlowReturn that can be returned from transform and transform_ip to
* indicate that no output buffer was generated.
*/
#define GST_BASE_TRANSFORM_FLOW_DROPPED GST_FLOW_CUSTOM_SUCCESS
In my audio processing code (audio processor) I have something like below code. I am trying to avoid this below repetitive code. Wondering if there is any better way to write this code?
Code is just an illustration to understand the problem: I have several audio processing blocks where the audio of one block is passed as input to second block and so on until the final processed output. There are 4 major components of this 'blocks': Initialization, processing, reset, and de-initilzation.
/* some code here to initialize audio processing blocks */
//snip
/*
* order of audio processing is important here.
* Is there any way to avoid writing below code
* for every processing block that gets added in
* future in c++11 perhaps??
*/
if (t1) {
t1->process(iFrame, oFrame);
iFrame = oFrame;
}
if (t2) {
t2->reset();
t2->process(iFrame, oFrame);
iFrame = oFrame;
}
if (t3) {
t3->process(iFrame, oFrame);
iFrame = oFrame;
}
/*
* if in future we want this way t1 - > t3 - > t2 then code change is required,
* is there any way to avoid this code change
*/
In essence I am looking for a generic way to handle this:
P1 -> P2 -> P3 -> P4
At compile time we should be able to define the order of processing blocks. Initialize the blocks with values read from a configuration file.
After that the process function is called for each of them and final output is produced.
During the processing some blocks can have reset function and other callbacks based on the processing blocks.
After the processing is done, we need to de-initialize the blocks.
How do we pass output of first block to input of second block and so on.
P stands for processing block.
https://ideone.com/GEvaKm
I've written a tool which uses the QSerialPort to write to a serial device. It runs for a certain time until I get the following error message:
ASSERT: "bytesTransferred == writeChunkBuffer.size()" in file
qserialport_win.cpp, line 511
My sending function looks like this:
/**
* #brief Send text to device
* #param text
* #return Success/Fail
*/
bool Serial::send(QString text)
{
if (connectionStatus && qsp.isWritable()) {
QByteArray buffer = text.toLatin1();
if (buffer.size() != qsp.write(buffer))
qDebug() << "Send does not work";
qsp.flush();
msleep(15);
return true;
} else {
return false;
}
}
If I understand it correctly I write the text (which is around 20 chars) with flush to the device, wait 15ms and repeat it afterwards. I don't really understand why I get this message?
// EDIT:
After some time I figured out what the problem was. I forgot to mention, why I wait 15 ms. That was part of the documentation, to wait after I send the data. The biggest problem, related to sending data was, that I run QSerialPort in a separate thread. By using that I run in trouble. I moved it to the MainThread and use signal slot design by Qt.
Without knowing the API you're using or how it works, there is no inherent reason why a write() method should transfer all the data supplied, especially when it returns a write count: a clear signal that it may not transfer everything in one go.
The only problem is the assertion itself. You should loop until the data is written, not assume it is all written in a single write. Nor should you sleep between writes, in a vain attempt to outguess the device you're sending to. This is literally just a waste of time.