I am just starting out with Flume. Trying to figure out how to extract fields from source log files using the interceptor regex_extract. However, between the source and sink the log file does not seem to change regardless of what I set in the conf file. Anyone have any ideas? Just trying to go from 1,a,b,c in the source to 1 in the sink.
Source file is simply in a test structure of 1,a,b,c,d yet always outputs in spooldir2 as 1,a,b,c,d
a1.sources = fs1
a1.sinks = hdfs-sink
a1.channels = parse
a1.sources.fs1.type = spooldir
a1.sources.fs1.spoolDir = /tmp/spooldir
a1.sources.fs1.fileHeader = true
a1.sources.fs1.interceptors = i1
a1.sources.fs1.interceptors.i1.type = regex_extractor
a1.sources.fs1.interceptors.i1.regex = ^(\\d)
a1.sources.fs1.interceptors.i1.serializers = s1
a1.sources.fs1.interceptors.i1.serializers.s1.name = extracted
a1.sinks.hdfs-sink.type = file_roll
a1.sinks.hdfs-sink.sink.directory = /tmp/spooldir2
a1.channels.parse.type = memory
a1.channels.parse.capacity = 1000
a1.channels.parse.transactionCapacity = 100
a1.sources.fs1.channels = parse
a1.sinks.hdfs-sink.channel = parse
Related
I have a file which has multiple file in it. Multiple files have excel files. I could put them on a list
excel_files = []
for path, dirs, files in os.walk(bugra_path):
for filename in files:
if filename == 'desktop.ini': continue
excel_files.append(filename)
The output is like that:
['2019-11-kudup.xlsx', '2019-12-kudup.xlsx', '2019-11-kur.xlsx', '2019-12-kur.xlsx', '2019-11-bilateral_buys.xlsx', '2019-12-bilateral_buys.xlsx', '2019-11-bilateral_sells.xlsx', '2019-12-bilateral_sells.xlsx', '2019-11-dam_buys.xlsx', '2019-12-dam_buys.xlsx', '2019-11-dam_sells.xlsx', '2019-12-dam_sells.xlsx', '2019-11-productions.xlsx', '2019-12-productions.xlsx', '2019-11-ptf_smf.xlsx', '2019-12-ptf_smf.xlsx']
However, this not enough for me, I want to split this list into multiple lists based on its element names. I mean want to achieve this:
kudup_list = ['2019-11-kudup.xlsx', '2019-12-kudup.xlsx']
kur_list = ['2019-11-kur.xlsx', '2019-12-kur.xlsx']
bilateral_buys_list = ['2019-11-bilateral_buys.xlsx', '2019-12-bilateral_buys.xlsx']
bilateral_sells_list = ['2019-11-bilateral_sells.xlsx', '2019-12-bilateral_sells.xlsx']
dam_buys_list = ['2019-11-dam_buys.xlsx', '2019-12-dam_buys.xlsx']
dam_sells_list = ['2019-11-dam_sells.xlsx', '2019-12-dam_sells.xlsx']
productions_list = ['2019-11-productions.xlsx', '2019-12-productions.xlsx']
ptf_smf_list = ['2019-11-ptf_smf.xlsx', '2019-12-ptf_smf.xlsx']
How can I get this output?
My script is working so far to open a remote FTP connection, change directory, and download a file. My last two steps would be to delete the remove file once it's fully downloaded and then close the connection. ACF documentation (and cfdocs) seems to have very little information on this. Here's what I have so far:
ftpConnection = ftpService.open(
action = 'open',
connection = variables.ftpConnectionName,
server = variables.ftpServerName,
username = '***************',
password = '***************',
secure='true');
if( ftpConnection.getPrefix().succeeded ){
fileList = ftpService.listdir(directory = variables.ftpPath, connection= variables.ftpConnectionName, name='pendingFiles', stopOnError='true').getResult();
if( fileList.recordCount ){
changeFtpConnectionDir = ftpService.changeDir(
connection = variables.ftpConnectionName,
directory = variables.ftpPath);
getFtpConnection = ftpService.getFile(
connection = variables.ftpConnectionName,
remoteFile = fileList.name,
localFile = local.localPath & fileList.name,
failIfExists = false,
timeout = 3000
);
deleteRemoteFile = ftpService.remove(
connection = variables.ftpConnectionName,
remoteFile = fileList.name
};
closeFtp = ftpService.close(
connection = variables.ftpConnectionName
);
};
};
Error is thrown on the remoteFile = fileList.name. Since I already changed directory I don't think I need to put the full path here.
I put the entire script up since there doesn't seem to be many resources out there about using the newer ftpServer() functions.
D'oh - my issue was a typo:
deleteRemoteFile = ftpService.remove(
connection = variables.ftpConnectionName,
remoteFile = fileList.name
);// had } instead of )
I'll still leave this up as a resource for ftpService()
To be able to use different templates in a TYPO3 CMS 7.6.18 setup. I include a pageTSConfig.ts file with a backend layout looking like this. (see also this pastebin: https://pastebin.com/BcYKrYKh and this how it looks like in the TYPO3 object browser: https://pastebin.com/LVXqNUZC
### Backend Layouts #####
mod.web_layout.BackendLayouts{
1 {
title = Standaard Layout
config {
backend_layout {
colCount = 2
rowCount = 1
rows {
1 {
columns {
1 {
name = linker_inhoud
colPos = 1
}
2 {
name = midden_inhoud
colPos = 0
}
}
}
}
}
}
}
}
Since the BE-Layout is done with a file, in my TSconfig.ts I have added pagets__0 to it like the manual mentioned. Still this is not working. What Am I missing here? This is a part of the TSconfig I have in place here is the complete config:
page.10.file.stdWrap.cObject = CASE
page.10.file.stdWrap.cObject {
key.data = levelfield:-1, backend_layout_next_level, pagelayout, slide
key.override.field = pagelayout
default = TEXT
default.value = fileadmin/templates/index.html
pagets__0 = TEXT
pagets__0.value = fileadmin/templates/index.html
pagets__1 = TEXT
pagets__1.value = fileadmin/templates/layouts/small_header_page.html
pagets__2 = TEXT
pagets__2.value = fileadmin/templates/layouts/alternatieve_pagina.html
}
All to be included with:
There is a difference between "backend_layout", which is a the name of a real database field that can be fetched by "levelfield" and "pagelayout", which is a kind of virtual field to get rid of the "levelfield" approach.
https://docs.typo3.org/typo3cms/TyposcriptReference/DataTypes/Gettext/Index.html#pagelayout
key.data = pagelayout
should do the whole job for you and only in this case you can use stuff like
pagets__x
to access the actual layout.
So your code should either be:
page.10 = FLUIDTEMPLATE
page.10.file.cObject = CASE
page.10.file.cObject {
key.data = pagelayout
default = TEXT
default.value = fileadmin/templates/index.html
pagets__0 = TEXT
pagets__0.value = fileadmin/templates/index.html
pagets__1 = TEXT
pagets__1.value = fileadmin/templates/layouts/small_header_page.html
pagets__2 = TEXT
pagets__2.value = fileadmin/templates/layouts/alternatieve_pagina.html
}
or it should be
page.10 = FLUIDTEMPLATE
page.10.file.cObject = CASE
page.10.file.cObject {
key.data = levelfield:-1, backend_layout_next_level, slide
key.override.field = backend_layout
default = TEXT
default.value = fileadmin/templates/index.html
1 = TEXT
1.value = fileadmin/templates/layouts/small_header_page.html
2 = TEXT
2.value = fileadmin/templates/layouts/alternatieve_pagina.html
}
But not a mix of both approaches.
Before I tell you what might be wrong with your code, let me explain you a few things.
You have placed your templates in the fileadmin directory. This is not the place where to put these files any longer, because the fileadmin is a public place for resources like images, videos or documents. It might be available for every backend user in the filelist and the editor should not be able to edit the template in any case. The suggested way to handle your templates is to put them into an own extension that can be installed via the extension manager.
In your pastebin snippets, there is a line with userFunc = tx_templavoila_pi1->main_page, you may mixing up stuff in your installation and don't want to use FLUIDTEMPLATE alongside templavoila, because it could be confusing what rendering method is used for what stuff on your page. Better stick to templavoila or Fluid for the entirety of the TYPO3 installation.
Now, you have these lines in your TypoScript:
key.data = levelfield:-1, backend_layout_next_level, pagelayout, slide
key.override.field = pagelayout
There is no field pagelayout in the pages records. The field you rather want to address is backend_layout.
My goal is to restore the lost .proto files written by someone else from existing c++ protobuf messages. By using the Descriptor and EnumDescriptor I was able to do the following:
const google::protobuf::EnumDescriptor* logOptionDesc =
bgs::protocol::LogOption_descriptor();
std::string logOptionStr = logOptionDesc->DebugString();
bgs::protocol::EntityId entityId;
const google::protobuf::Descriptor* entityIdDesc = entityId.GetDescriptor();
std::string entityIdStr = entityIdDesc->DebugString();
The logOptionStr string I got looked something like this:
enum LogOption {
HIDDEN = 1;
HEX = 2;
}
and entityIdStr:
message EntityId {
required fixed64 high = 1 [(.bgs.protocol.log) = HEX];
required fixed64 low = 2 [(.bgs.protocol.log) = HEX];
}
Notice the EntityId message contains some field options. Without resolving this dependency I cannot generate a FileDescriptor that can help me restore the .proto files. I suspect the EntityId string should look something like the following:
import "LogOption.proto";
package bgs.protocol;
extend google.protobuf.FieldOptions {
optional LogOptions log = HEX;
}
message EntityId {
required fixed64 high = 1 [(.bgs.protocol.log) = HEX];
required fixed64 low = 2 [(.bgs.protocol.log) = HEX];
}
Is it possible to restore the .proto files that require additional information such as package, field options and imports? What else do I need to do to restore the .proto files?
I have a need to create an action that will:
1. copy a selected part (selected by hand) of an image in an already opened file
2. paste selection into new file
3. save new file as jpg file, but not with default file name of "untitled.jpg" - instead use a unique name or use a auto-increment suffix
Because the action will be run multiple times on different selections from the same image, saving each selection with a unique name or auto-incremented suffix would save the step of manually supplying the filename each time a different selection is saved.
I can create an action that gets to the save-as step, but don't know if it is possible to modify the default save as name as described above. Is it possible?
No. Tried it before with no success. You have to save manually.
Don't think this is possible with an action but you can write a script do to it.
I have created a script for similar work. It uses a technique to generate unique filenames and save the file.
/************************************************************************
* Author: Nishant Kumar
* Description: This script iterates through a template to create
* jpg images with id card numbers.
* Date: 08-03-2015
***********************************************************************/
//current id count
var id_count = 0;
//total no of id cards to produce
var total_id_cards = 42;
//no. of cards per sheet
var card_per_sheet = 21;
//Save path related to current file
var save_path = app.activeDocument.path;
//do an iteration, number the cards and save file
do{
//iterate 24 nos in each document
for(var i = 0; i<card_per_sheet; i++){
id_count++;
app.activeDocument.layers[i].textItem.contents = id_count;
}
//Create a jpg document with standard options
jpgSaveOptions = new JPEGSaveOptions();
jpgSaveOptions.embedColorProfile = true;
jpgSaveOptions.formatOptions = FormatOptions.STANDARDBASELINE;
jpgSaveOptions.matte = MatteType.NONE;
jpgSaveOptions.quality = 12;
//Save jpg with incremental file names (1.jpg, 2.jpg), make sure the path exists
jpgFile = new File( save_path + "/output/" + id_count/card_per_sheet + ".jpeg" );
app.activeDocument.saveAs(jpgFile, jpgSaveOptions, true, Extension.LOWERCASE);
}while(id_count < total_id_cards);
I know this is old, but still. You can use the following script.
How to use a script:
Copy the following script in notepad, and save it in directory similar to "C:\Program Files (x86)\Adobe\Adobe Photoshop CS2\Presets\Scripts" with the extension JSX.
To run the scrip in photoshop, go to File > Scripts > "Your Script".
#target photoshop
main();
function main(){
if(!documents.length) return;
var Name = app.activeDocument.name.replace(/.[^.]+$/, '');
Name = Name.replace(/\d+$/,'');
try{
var savePath = activeDocument.path;
}catch(e){
alert("You must save this document first!");
}
var fileList= savePath.getFiles(Name +"*.jpg").sort().reverse();
var Suffix = 0;
if(fileList.length){
Suffix = Number(fileList[0].name.replace(/\.[^\.]+$/, '').match(/\d+$/));
}
Suffix= zeroPad(Suffix + 1, 4);
var saveFile = File(savePath + "/" + Name + "_" + Suffix + ".jpg");
SaveJPG(saveFile);
}
function SaveJPG(saveFile){
//Create a jpg document with standard options
jpgSaveOptions = new JPEGSaveOptions();
jpgSaveOptions.embedColorProfile = true;
jpgSaveOptions.formatOptions = FormatOptions.STANDARDBASELINE;
jpgSaveOptions.matte = MatteType.NONE;
jpgSaveOptions.quality = 12;
//Save jpg with incremental file names (1.jpg, 2.jpg), make sure the path exists
activeDocument.saveAs(saveFile, jpgSaveOptions, true, Extension.LOWERCASE);
};
function zeroPad(n, s) {
n = n.toString();
while (n.length < s) n = '0' + n;
return n;
};