Tracking failed jobs in SAS - sas

Question

I'm looking to improve the way to track a server that uses SAS to collect information from several databases, the challenge here is that it's one of those servers that a company used for a while without any governance, folders are everywhere, no actual folder structure, anyone was doing whatever they pleased with it. You know the drill.
The option that I found for now is to put a macro on the jobs that are failing that will track the logs, and if it finds an error, then it should send out an email to a distribution list.
But this solution isn't a very good software architecture, every time a new job is scheduled in the crontab, I need to slap the macro there, and if the things move to another directory it stops working and all of that, a lot of manual work cascading down.
A solution that I'm looking for would be finding something that can read all logs from a list of directories (and this would be the biggest manual update it needs then, but only this, and only because I don't have authorization to move the folders), fetch the ones that have errors, and output this to a web page, so then we can only check that web page. to know centralized information (when it's scheduled, when it last ran, etc, etc).
By any chance does anyone have any suggestions here on how to do this? I thought of running a SAS script to just output a txt file that I can fetch somewhere and then reading that with Javascript on a web page, but I ran into a few difficulties on how to loop through all of the folders, and because this scripting language doesn't fell really feel all that great for this purpose. But I'm out of ideas for now. Recommendations?

Answer 1

If you are using Node server side to deliver web content, you can write a function that scans the folders listed in a control file for log files containing ERROR: messages and reports them.
For example:
log folders.txt (control file)
d:\temp\logs\job-1
d:\temp\logs\tuesday jobs
d:\jobs\wednesday\job set 1
d:\jobs\wednesday\job set 2\prod
node app (test version)
const express = require('express');
const app = express();
const port = 8081;
app
.get('/sas/logs/errors', (req, res) => {
res.setHeader('Content-Type', 'text/html');
processFolders(res);
res.end();
})
;
app.listen(port, () => console.log(`Listening on port ${port}`));
const fs = require('fs');
const readline = require('readline');
const path = require('path');
const listOfFoldersFile = 'log folders.txt';
function processFolders(res) {
var level1Count = 0;
var level2Count = 0;
var level3Count = 0;
var content;
try {
content = fs.readFileSync(listOfFoldersFile, 'UTF-8');
}
catch (err) {
console.log(err.toString());
res.write('<p>Configuration error in processFolders.</p>');
return;
}
const folders = content.split(/\r?\n/);
for (let folder of folders) {
scanForLogs(folder);
}
if (level1Count == 0) {
res.write('<p>No error messages found in SAS logs.</p>');
return;
}
return;
function scanForLogs(folder) {
//console.log(`scan folder ${folder}`);
var logfiles;
try {
logfiles = fs.readdirSync (folder).filter(filename => filename.match(/\.log$/));
}
catch (err) {
console.log(err.toString());
res.write('<p>Problem scanning a log folder.</p>');
return;
}
level2Count = 0;
for (let logfile of logfiles) {
parseSASLog(folder, logfile);
}
}
function parseSASLog(folder, filename) {
//console.log(`parse log ${path.join(folder,filename)}`);
var content;
try {
content = fs.readFileSync(path.join(folder,filename), 'UTF-8');
}
catch (err) {
console.log(err.toString());
res.write(`<p>Problem reading file ${filename}</p>`);
return;
}
const lines = content.split(/\r?\n/);
level3Count = 0;
var linenum = 0;
for (const line of lines) {
linenum++;
if (line.match (/^ERROR:/)) {
reportErrorMessage(folder, filename, line, linenum);
}
}
if (level3Count) {
res.write('</pre>');
}
}
function reportErrorMessage(folder, filename, line, linenum) {
//console.log(`reportErrorMessage`);
level1Count++;
level2Count++;
level3Count++;
if (level1Count == 1) {
res.write('<h1>SAS errors</h1>');
}
if (level2Count == 1) {
res.write(`<h2>folder: ${folder}<h2>`);
}
if (level3Count ==1) {
res.write(`<h3>file: ${filename}</h3><pre>`);
}
res.write(`${linenum}: ${line}\n`);
}
}
Errors reported (example)
Other approaches could include
asynchronous server side scans delivering output to client side piecewise through signalR
delivering error report as json data which the client side uses to render the report