PowerBI: Check if the folder/directory we are loading to exists or not using Power Query - powerbi

I am a newbie in PowerBi and currently working on a POC where I need to load data from a folder or directory. Before this load, I need to check if
1) the respective folder exists
2) the file under the folder is with.csv extension.
Ex. Let suppose we have a file '/MyDoc2004/myAction.csv'.
Here first we need to check if MyDoc2004 exists and then if myAction file is with.csv extension.
Is there any way we can do this using Power Query?

1. Check if the folder exists
You can apply Folder.Contents function with the absolute path of the folder, and handle the error returned when the folder does not exist with try ... otherwise ... syntax.
let
absoluteFolderPath = "C:/folder/that/may/not/exist",
folderContentsOrError = Folder.Contents(absoluteFolderPath),
alternativeResult = """" & absoluteFolderPath & """ is not a valid folder path",
result = try folderContentsOrError otherwise alternativeResult
in
result
2. Check if the file is with .csv extension
I'm not sure what output you are expecting.
Here is a way to get the content of the file by full path including ".csv", or return an alternative result if not found.
let
absoluteFilePath = "C:/the/path/myAction.csv",
fileContentsOrError = File.Contents(absoluteFilePath),
alternativeResult = """" & absoluteFilePath & """ is not a valid file path",
result = try fileContentsOrError otherwise alternativeResult
in
result
If this is not what you are looking for, please update the question with the expected output.
Hope it helps.

Related

How to check if a joint path exists before creating it using a for statement

I'm trying to use makedirs to create subfolders in an existing folder since I will have to create the same folders at least 40 more times based on projects but I want to check if the folders exist first in case I accidentally run the script twice on the same project.
I thought that maybe os.path.join.exists would work but I don't think this is real.
import os
mypath = r'path'
subfolder_names = [
'01_subfolder',
'02_subfolder',
'03_subfolder',
'04_subfolder',
'06_subfolder',
'07_subfolder',
'08_subfolder',
'09_subfolder',
'10_subfolder'
]
for subfolder_name in subfolder_names:
# doesn't work but I need something to check first
if not os.path.join.exists(mypath,subfolder_name)
os.makedirs(os.path.join(mypath,subfolder_name))
output it should put the folders or say "these ones already exist" it would be nice if it would spit out which folders already exist and which ones don't.
You are pretty close. You need to join first and then check for existence:
subfolder_names = ['01_subfolder', '02_subfolder', '03_subfolder', '04_subfolder', '06_subfolder', '07_subfolder', '08_subfolder', '09_subfolder', '10_subfolder']
for subfolder_name in subfolder_names:
p = os.path.join(mypath, subfolder_name
if not os.path.join.exists(p):
os.makedirs(p)
join and exists are separate functions.
You can also choose to do this functionally:
from itertools import filterfalse, repeat
subfolder_names = ['01_subfolder', '02_subfolder', '03_subfolder', '04_subfolder', '06_subfolder', '07_subfolder', '08_subfolder', '09_subfolder', '10_subfolder']
paths = map(os.path.join, repeat(mypath), subfolder_names)
missing_paths = filterfalse(os.path.exists, paths)
for missing_path in missing_paths:
os.makedirs(missing_path)

Give warning if folder contains more than one file type using RegExp

Using RegExp to search a folder for file types held in an Array.
How can I show a fail if the folder contains more than one different file type but it is perfectly OK to have multiple of the same file type in the folder.
Here's my code:
var AllowedFileTypes = ["orf", "tif", "tiff", "jpg", "jpeg"];
var regex = new RegExp('.+\.(?:' + AllowedFileTypes.join('|')+ ')$','i');
fileList = inputFolder.getFiles(regex);
Thanks

Use metadata in filename when saving harbor input Liquidsoap

So I have an instance of Liquidsoap, which I am using to stream to an Icecast server.
I'd like to record any live broadcasts that take place automatically, which I am now doing and is working well.
What I'd like to do is use the metadata (specifically the songname) of the live show when creating the mp3 archive.
#!/usr/local/bin/liquidsoap
set("log.file",true)
set("log.file.path","/var/log/liquidsoap/radiostation.log")
set("log.stdout",true)
set("log.level",3)
#-------------------------------------
set("harbor.bind_addr","0.0.0.0")
#-------------------------------------
backup_playlist = playlist("/home/radio/playlists/playlist.pls",conservative=true,reload_mode="watch")
output.dummy(fallible=true,backup_playlist)
#-------------------------------------
live_dj = input.harbor(id="live",port=9000,password="XXX", "live")
date = '%m-%d-%Y'
time = '%H:%M:%S'
output.file(%mp3, "/var/www/recorded-shows/#{Title} - Recorded On #{date} At #{time}.mp3", live_dj, fallible=true)
#time_stamp = '%m-%d-%Y, %H:%M:%S'
#output.file(%mp3, "/var/www/recorded-shows/live_dj_#{time_stamp}.mp3", live_dj, fallible=true)
#-------------------------------------
on_fail = single("/home/radio/fallback/Enei -The Moment Feat DRS.mp3")
#-------------------------------------
source = fallback(track_sensitive=false,
[live_dj, backup_playlist, on_fail])
# We output the stream to icecast
output.icecast(%mp3,id="icecast",
mount="myradio.mp3",
host="localhost", password="XXX",
icy_metadata="true",description="cool radio",
url="http://myradio.fm",
source)
I have added #{title} where I would like my song title to appear, sadly though I am unable to get this populate.
My Dj's use BUTT and the show title is connected as part of their connection, so the data should be available pre recording.
Any advice is much appreciated!
This is far from being as easy as it seems.
The title metadata is dynamic, thus not available as a variable on script initialization
The filename argument of output.file is compiled when the script is initialized
A solution would consist in:
Defining a variable reference title to populate with live metadata
Output to a temporary file
Rename the file on close using on_close argument with output.file (in this case, we can just prepend the title)
This would give the following code (on a Linux box, change mv with ren on Windows):
date = '%m-%d-%Y'
time = '%H:%M:%S'
# Title is a reference
title = ref ""
# Populate with metadata
def get_title(m)
title := m['title']
end
live_dj = on_metadata(get_title,live_dj)
# Rename file on close
def on_close(filename)
# Generate new file name
new_filename = "#{path.dirname(filename)}/#{!title} - #{basename(filename)}"
# Rename file
system("mv '#{filename}' '#{new_filename}'")
end
output.file(on_close=on_close, %mp3, "/var/www/recorded-shows/Recorded On #{date} At #{time}.mp3", live_dj, fallible=true)
I tested a similar scenario and it works just well. Just beware that this will create a new file every time a DJ disconnects or updates the title. Also keep in mind that time stamps will be resolved by output.file.
This is based on the following example from a Liquidsoap dev: https://github.com/savonet/liquidsoap/issues/661#issuecomment-439935854)

Search Informatica for text in SQL override

Is there a way to search all the mappings, sessions, etc. in Informatica for a text string contained within a SQL override?
For example, suppose I know a certain stored procedure (SP_FOO) is being called somewhere in an INFA process, but I don't know where exactly. Somewhere I think there is a Post SQL on a source or target calling it. Could I search all the sessions for Post SQL containing SP_FOO ? (Similar to what I could do with grep with source code.)
You can use Repository queries for querying REPO tables(if you have enough access) to get data related with all the mappings,transformations,sessions etc.
Please use the below link to get almost all kind of repo queries.Ur answers can be find in the below link.
https://uisapp2.iu.edu/confluence-prd/display/EDW/Querying+PowerCenter+data
select *--distinct sbj.SUBJECT_AREA,m.PARENT_MAPPING_NAME
from REP_SUBJECT sbj,REP_ALL_MAPPINGS m,REP_WIDGET_INST w,REP_WIDGET_ATTR wa
where sbj.SUBJECT_ID = m.SUBJECT_ID AND
m.MAPPING_ID = w.MAPPING_ID AND
w.WIDGET_ID = wa.WIDGET_ID
and sbj.SUBJECT_AREA in ('TLR','PPM_PNLST_WEB','PPM_CURRENCY','OLA','ODS','MMS','IT_METRIC','E_CONSENT','EDW','EDD','EDC','ABS')
and (UPPER(ATTR_VALUE) like '%PSA_CONTACT_EVENT%'
-- or UPPER(ATTR_VALUE) like '%PSA_MEMBER_CHARACTERISTIC%'
-- or UPPER(ATTR_VALUE) like '%PSA_REPORTING_HH_CHRSTC%'
-- or UPPER(ATTR_VALUE) like '%PSA_REPORTING_MEMBER_CHRSTC%'
)
--and m.PARENT_MAPPING_NAME like '%ARM%'
order by 1
Please let me know if you have any issues.
Another less scientific way to do this is to export the workflow(s) as XML and use a text editor to search through them for the stored procedure name.
If you have read access to the schema where the informatica repository resides, try this.
SELECT DISTINCT f.subj_name folder, e.mapping_name, object_type_name,
b.instance_name, a.attr_value
FROM opb_widget_attr a,
opb_widget_inst b,
opb_object_type c,
opb_attr d,
opb_mapping e,
opb_subject f
WHERE a.widget_id = b.widget_id
AND b.widget_type = c.object_type_id
AND ( object_type_name = 'Source Qualifier'
OR object_type_name LIKE '%Lookup%'
)
AND a.widget_id = b.widget_id
AND a.attr_id = d.attr_id
AND c.object_type_id = d.object_type_id
AND attr_name IN ('Sql Query')--, 'Lookup Sql Override')
AND b.mapping_id = e.mapping_id
AND e.subject_id = f.subj_id
AND a.attr_value is not null
--AND UPPER (a.attr_value) LIKE UPPER ('%currency%')
Yes. There is a small java based tool called Informatica Meta Query.
Using that tool, you can search for any information that is present in the Informatica meta data tables.
If you cannot find that tool, you can write queries directly in the Informatica Meta data tables to get the required information.
Adding few more lines to solution provided by Data Origin and Sandeep.
It is highly advised not to query repository tables directly. Rather, you can create synonyms or views and then query those objects to avoid any damage to rep tables.
In our dev/ prod environment application programmers are not granted any direct access to repo. tables.
As querying the Informatica database isn't the best idea, I would suggest you to export all the workflows in your folder into xml using Repository Manager. From Rep Mgr you can select all of them once and export them at once. Then write a java program to search the pattern from the xml's you have.
I have written a sample prog here, please modify it as per your requirement:
make a spec file with workflow names(specFileName).
main()
{
try {
File inFile = new File(specFileName);
BufferedReader reader = new BufferedReader(newFileReader(infile));
String tectToSearch = '<YourString>';
String currentLine;
while((currentLine = reader.readLine()) != null)
{
//trim newline when comparing with String
String trimmedLine = currentLine.trim();
if(currentline has the string pattern)
{
SOP(specFileName); //specfile name
}
}
reader.close();
}
catch(IOException ex)
{
System.out.println("Error reading to file '" + specFileName +"'");
}
}

Does Qt Linguist offer the ability to add new entries to the editable .ts file?

I didn't find a way to do this - only to edit the translations to the existing fields.
If there is no way to achieve this - how should this be done (somehow automatically, because right now I was manually adding
<message>
<source>x</source>
<translation>xx</translation>
</message>
blocks to my .ts file and I assume that's not the correct way.
No, that's not the correct way :) Use tr() in the code to mark strings for translation.
For example
label->setText( tr("Error") );
The you run lupdate for your project to extract them to a .ts. See here for more details.
Or do you need to translate strings that are not in the source code?
I just wrote a python script to insert new entries
into the .ts file for a homegrown parser using ElementTree. It doesnt make the code pretty
when it adds it, but I believe it works just fine (so far):
from xml.etree import ElementTree as ET
tree = ET.parse(infile)
doc = tree.getroot()
for e in tree.getiterator()
if e.tag == "context":
for child in e.getchildren():
if child.tag == "name" and child.text == target:
elem = ET.SubElement(e, "message")
src = ET.SubElement(elem, "source")
src.text = newtext
trans = ET.SubElement(elem, "translation")
trans.text = "THE_TRANSLATION"
tree.write(outfile)
Where infile is the .ts file, outfile may be the same as infile or different.
target is the context you are looking for to add a new message into,
and newtext is of course the new source text.