I am trying to query a rocket UNIVERSE database. For the most part it works, until I hit certain types of fields that are of type I (not all, but some). In the vendor documentation (EPICOR ECLIPSE) it mentions as a note the following, "Any dictionary that contains a reference to a common file handle will not work without a call to 'OPEN.STANDARD.FILES' so you may need to wrap standard dictionaries."
So my question is how to do that?
When I query the database directly from TCL (cd c:/u2/eclipse and type "uv" to get to the TCL environment) I get the following.
"LIST PSUB TSN.COMMENT 07:37:39am 22 Mar 2014 PAGE 1
#ID..................................... TSN..........
**Program "DICT.GET.LEDGER.DET.VALUE": Line 9, Improper data type.**
When I run the same query within the vendor's application environment it works. Their environment is a DOS like menu system that does allow one to drop down to a TCL environment as well. But, obviously something in their managed environment satisfies dependencies that are needed to make a successful query.
The first few lines of the subroutine are as follows:
>ED OC OPEN.STANDARD.FILES
429 lines long.
----: P
0001: SUBROUTINE
0002: $INCLUDE AD.DIR CC~COMMON
0003: *
0004: *
0005: *
0006: *
0007: *
0008: *
0009: *
0010: *
0011: *
0012: *
0013: *
0014: *
0015: *
0016: *
0017: *
0018: *
0019: *
0020: IF FILES.ARE.OPEN$ THEN RETURN
0021: *
0022: OPEN 'ABC.CODES' TO ABCCFILE ELSE
----:
0023: FLNM = 'ABC.CODES'
----:
0024: GOSUB EXIT.OPN
.
.
.
To create a wrapper for this routine, do the following:
>ED BP OPEN.STANDARD.FILES.TCL
001 * OPEN.STANDARD.FILES.TCL
002 CALL OPEN.STNADARD.FILES
003 STOP
004 END
>BASIC BP OPEN.STANDARD.FILES.TCL
>CATALOG BP OPEN.STANDARD.FILES.TCL
Then you can execute OPEN.STANDARD.FILES.TCL before your list statement. I just noticed that you are have this tagged for "u2netsdk". Are you accessing Epicor using .NET api, or the "cd c:/u2/eclipse" and "uv"?
If you are using the .NET api, then you can call OPEN.STANDARD.FILES directly from the .NET api before your EXECUTE the list statement.
-Nathan
Related
My attribute table has a field named "FEATURE", inside of this contains two feature classes, Airport and Seaplane Base. I am trying to create two buffers, a 15000 meter around Airplane and 7500 meter around Seaplane Bases.
import arcpy
from arcpy import env
env.workspace = "E:\Python Programming\Lab5a"
fcs = arcpy.ListFeatureClasses()
for fc in fcs:
arcpy.Buffer_analysis(fc, ["FEATURE"], '"FEATURE" LIKE \'%Airport\'', "15000 METERS")
arcpy.Buffer_analysis(fc, ["FEATURE"], '"FEATURE" LIKE \'%Seaplane Base\'', "7500 METERS")
This does not work at all and fails to execute tool.
First off your file path is wrong.
env.workspace = r"E:\Python Programming\Lab5a" the r stands for raw string
Basically you have to escape the backslash. Your current string doesn't make the path you think it does.
I need to do build a simple regression model around the productivity of various land units in SAS but I am fairly new to it. I have the following parameters:
Prodbanana / ELEVATION / SLOPE / SOILTYPE
The productivity vaules of banana are in kg/ha; The elevation and slope parameters are already classified (6 classes) as well as SOILTYPE (6 classes)
The model should be: Productivity = x1* ELEVATION + x2*SLOPE + x3*SOILTYPE
I tried so far:
proc glm data=Work.Banana2;
class ELEVATION SLOPE SOILTYPE;
model Prodbanana = ELEVATION + SLOPE + SOILTYPE;
run;
But it returns the following error:`
45 proc glm data=Work.Banana2;
46 class ELEVATION SLOPE SOILTYPE;
47 model Prodbanana = ELEVATION + SLOPE + SOILTYPE;
-
22
-----
202
NOTE: The previous statement has been deleted.
ERROR 22-322: Syntax error, expecting one of the following: a name, ;, (, *, -, /, #,
CHARACTER, CHAR, NUMERIC, |.
ERROR 202-322: The option or parameter is not recognized and will be ignored.
48 run;`
Any suggestions?
cheers
The addition operator is unnecessary. You can find a quick guide to model specification syntax
here. Works across SAS regression procedures.
I'm trying to write a perl regex to match the 5th column of files that contain 11 columns. There's also a preamble and footer which are not data. Any good thoughts on how to do this? Here's what I have so far:
if($line =~ m/\A.*\s(\b\w{9}\b)\s+(\b[\d,.]+\b)\s+(\b[\d,.sh]+\b)\s+.*/i) {
And this is what the forms look like:
No. Form 13F File Number Name
____ 28-________________ None
[Repeat as necessary.]
FORM 13F INFORMATION TABLE
TITLE OF VALUE SHRS OR SH /PUT/ INVESTMENT OTHER VOTING AUTHORITY
NAME OF INSURER CLASS CUSSIP (X$1000) PRN AMT PRNCALL DISCRETION MANAGERS SOLE SHARED NONE
Abbott Laboratories com 2824100 4,570 97,705 SH sole 97,705 0 0
Allstate Corp com 20002101 12,882 448,398 SH sole 448,398 0 0
American Express Co com 25816109 11,669 293,909 SH sole 293,909 0 0
Apollo Group Inc com 37604105 8,286 195,106 SH sole 195,106 0 0
Bank of America com 60505104 174 12,100 SH sole 12,100 0 0
Baxter Internat'l Inc com 71813109 2,122 52,210 SH sole 52,210 0 0
Becton Dickinson & Co com 75887109 8,216 121,506 SH sole 121,506 0 0
Citigroup Inc com 172967101 13,514 3,594,141 SH sole 3,594,141 0 0
Coca-Cola Co. com 191216100 318 6,345 SH sole 6,345 0 0
Colgate Palmolive Co com 194162103 523 6,644 SH sole 6,644 0 0
If you ever do write a regex this long, you should at least use the x flag to ignore whitespace, and importantly allow whitespace and comments:
m/
whatever
something else # actually trying to do this
blah # for fringe case X
/xi
If you find it hard to read your own regex, others will find it Impossible.
I think a regular expression is overkill for this.
What I'd do is clean up the input and use Text::CSV_XS on the file, specifying the record separator (sep_char).
Like Ether said, another tool would be appropriate for this job.
#fields = split /\t/, $line;
if (#fields == 11) { # less than 11 fields is probably header/footer
$the_5th_column = $fields[4];
...
}
My first thought is that the sample data is horribly mangled in your example. It'd be great to see it embedded inside some <pre>...</pre> tags so columns will be preserved.
If you are dealing with columnar data, you can go after it using substr() or unpack() easier than you can using regex. You can use regex to parse out the data, but most of us who've been programming Perl a while also learned that regex is not the first tool to grab a lot of times. That's why you got the other comments. Regex is a powerful weapon, but it's also easy to shoot yourself in the foot.
http://perldoc.perl.org/functions/substr.html
http://perldoc.perl.org/functions/unpack.html
Update:
After a bit of nosing around on the SEC edgar site, I've found that the 13F files are nicely formatted. And, you should have no problem figuring out how to process them using substr and/or unpack.
FORM 13F INFORMATION TABLE
VALUE SHARES/ SH/ PUT/ INVSTMT OTHER VOTING AUTHORITY
NAME OF ISSUER TITLE OF CLASS CUSIP (x$1000) PRN AMT PRN CALL DSCRETN MANAGERS SOLE SHARED NONE
- ------------------------------ ---------------- --------- -------- -------- --- ---- ------- ------------ -------- -------- --------
3M CO COM 88579Y101 478 6051 SH SOLE 6051 0 0
ABBOTT LABS COM 002824100 402 8596 SH SOLE 8596 0 0
AFLAC INC COM 001055102 291 6815 SH SOLE 6815 0 0
ALCATEL-LUCENT SPONSORED ADR 013904305 172 67524 SH SOLE 67524 0 0
If you are seeing the 13F files unformatted, as in your example, then you are not viewing correctly because there are tabs between columns in some of the files.
I looked through 68 files to get an idea of what's out there, then wrote a quick unpack-based routine and got this:
3M CO, COM, 88579Y101, 478, 6051, SH, , SOLE, , 6051, 0, 0
ABBOTT LABS, COM, 002824100, 402, 8596, SH, , SOLE, , 8596, 0, 0
AFLAC INC, COM, 001055102, 291, 6815, SH, , SOLE, , 6815, 0, 0
ALCATEL-LUCENT, SPONSORED ADR, 013904305, 172, 67524, SH, , SOLE, , 67524, 0, 0
Based on some of the other files here's some thoughts on how to process them:
Some of the files use tabs to separate the columns. Those are trivial to parse and you do not need regex to split the columns. 0001031972-10-000004.txt appears to be that way and looks very similar to your example.
Some of the files use tabs to align the columns, not separate them. You'll need to figure out how to compress multiple tab runs into a single tab, then probably split on tabs to get your columns.
Others use a blank line to separate the rows vertically so you'll need to skip blank lines.
Others allow wrap columns to the next line (like a spreadsheet would in a column that is not wide enough. It's not too hard to figure out how to deal with that, but how to do it is being left as an exercise for you.
Some use centered column alignment, resulting in leading and trailing whitespace in your data. s/^\s+//; and s/\s+$//; will become your friends.
The most interesting one I saw appeared to have been created correctly, then word-wrapped at column 78, leading me to think some moron loaded their spreadsheet or report into their word processor then saved it. Reading that is a two step process of getting rid of the wrapping carriage-returns, then re-processing the data to parse out the columns. As an added task they also have column headings in the data for page breaks.
You should be able to get 100% of the files parsed, however you'll probably want to do it with a couple different parsing methods because of the use of tabs and blank lines and embedded column headers.
Ah, the fun of processing data from the wilderness.
I need to replace a single text phrase across my entire website domain with another one. What is the best way to do a mass search/ replace?
If you can do it file-by-file, then you could use a simple Perl one-liner:
perl -pi -e 's/search/replace/gi' filename.txt
If you are on a UNIX system with a shell, you can combine this with find to search and replace text on files in subdirectoies:
find /dir/to/files -iname 'foo.*' -exec perl ... {}\;
where ... is the above perl command.
I use KEDIT to do this every day. I have a script I wrote called TooMany.kex which allows me to edit a list of files across computers and networks. The only other way I know how to do it is using a shell script - you already have that.
* TooMany.kex - perform commands on files using a directory
* (this is designed to issue edit commands to too many files for the ring buffer)
* Multiple commands are separated by a semicolon ";"
*
* eg. TooMany c\one\two\**;c\two\three\**;file
* commands:
* 1. kedit "dirfileid.1()" (nodefext noprof'
* 2. c\one\two\**
* 3. c\two\three\**
* 4. file
*
parse arg CmdStr
if ftype.1() \= 'DIR' then do
'alert /The File Type Must Be "DIR"./ title /TooMany/'
exit
end
'nomsg less /</|/>/'
if rc = 0 then do
if nbscope.1() = 0 then do
'alert /No files found/ title /TooMany/'
exit
end
end
'top'
* give user something to look at while macro is running
'extract /nbfile/fileid'
* the number of files can change depending on the setting SCOPE/DISPLAY or ALL
size = nbscope.1()
if scope.1() = "ALL" then size = size.1()
nfiles = size
'msg Processing' size 'files.'
'refresh'
* save the directory file name
dir_fileid = fileid.1()
do nfiles - 1
* if less than 3K ISA free, leave early so user has some to work with
if memory.3() < 3 then do
'alert $TooMany aborting. ISA nearly full. You Forgot To File.$ title $TooMany$'
'qquit'
exit
end
'down'
'refresh'
'kedit "'dirfileid.1()'" (nodefext noprof'
if rc \= 0 then do
'alert $TooMany aborting. KEDIT rc='rc'$ title $TooMany$'
exit
end
Call ExecuteCommands
* edit file # 1 in the ring
'kedit "'fileid.1'" (noprof'
*'refresh'
end
* quit out of dir.dir and edit the last file
'next'
fid = dirfileid.1()
** 'qquit'
'kedit "'fid'" (nodefext noprof'
Call ExecuteCommands
'msg TooMany:' nfiles 'file(s) processed'
exit
ExecuteCommands:
* special skip files - don't edit the directory file
if dir_fileid = fileid.1() then return
* Execute commands separated by ";"
istart = 1
do forever
if pos(";",CmdStr,istart) = 0 then do
command substr(CmdStr,istart,length(CmdStr))
return
end
else do
iend = pos(";",CmdStr,istart)
command substr(CmdStr,istart,iend - istart)
istart = iend + 1
if istart > length(CmdStr) then return
end
end
return
I have a file of about 30000 lines of data that I want to load into a sqlite3 database. Is there a faster way than generating insert statements for each line of data?
The data is space-delimited and maps directly to an sqlite3 table. Is there any sort of bulk insert method for adding volume data to a database?
Has anyone devised some deviously wonderful way of doing this if it's not built in?
I should preface this by asking, is there a C++ way to do it from the API?
wrap all INSERTs in a transaction, even if there's a single user, it's far faster.
use prepared statements.
You want to use the .import command. For example:
$ cat demotab.txt
44 92
35 94
43 94
195 49
66 28
135 93
135 91
67 84
135 94
$ echo "create table mytable (col1 int, col2 int);" | sqlite3 foo.sqlite
$ echo ".import demotab.txt mytable" | sqlite3 foo.sqlite
$ sqlite3 foo.sqlite
-- Loading resources from /Users/ramanujan/.sqliterc
SQLite version 3.6.6.2
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> select * from mytable;
col1 col2
44 92
35 94
43 94
195 49
66 28
135 93
135 91
67 84
135 94
Note that this bulk loading command is not SQL but rather a custom feature of SQLite. As such it has a weird syntax because we're passing it via echo to the interactive command line interpreter, sqlite3.
In PostgreSQL the equivalent is COPY FROM:
http://www.postgresql.org/docs/8.1/static/sql-copy.html
In MySQL it is LOAD DATA LOCAL INFILE:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
One last thing: remember to be careful with the value of .separator. That is a very common gotcha when doing bulk inserts.
sqlite> .show .separator
echo: off
explain: off
headers: on
mode: list
nullvalue: ""
output: stdout
separator: "\t"
width:
You should explicitly set the separator to be a space, tab, or comma before doing .import.
I've tested some pragmas proposed in the answers here:
synchronous = OFF
journal_mode = WAL
journal_mode = OFF
locking_mode = EXCLUSIVE
synchronous = OFF + locking_mode = EXCLUSIVE + journal_mode = OFF
Here's my numbers for different number of inserts in a transaction:
Increasing the batch size can give you a real performance boost, while turning off journal, synchronization, acquiring exclusive lock will give an insignificant gain. Points around ~110k show how random background load can affect your database performance.
Also, it worth to mention, that journal_mode=WAL is a good alternative to defaults. It gives some gain, but do not reduce reliability.
C# Code.
You can also try tweaking a few parameters to get extra speed out of it. Specifically you probably want PRAGMA synchronous = OFF;.
Increase PRAGMA cache_size
to a much larger number. This will
increase the number of pages cached
in memory. NOTE: cache_size is a per-connection setting.
Wrap all inserts into a single transaction rather than one transaction per row.
Use compiled SQL statements to do the inserts.
Finally, as already mentioned, if you are willing forgo full ACID compliance, set PRAGMA synchronous = OFF;.
RE: "Is there a faster way that generating insert statements for each line of data?"
First: Cut it down to 2 SQL statements by making use of Sqlite3's Virtual table API e.g.
create virtual table vtYourDataset using yourModule;
-- Bulk insert
insert into yourTargetTable (x, y, z)
select x, y, z from vtYourDataset;
The idea here is that you implement a C interface that reads your source data set and present it to SQlite as a virtual table and then you do a SQL copy from the source to the target table in one go. It sounds harder than it really is and I've measured huge speed improvements this way.
Second: Make use of the other advise provided here i.e. the pragma settings and making use of a transaction.
Third: Perhaps see if you can do away with some of the indexes on the target table. That way sqlite will have less indexes to update for each row inserted
There is no way to bulk insert, but
there is a way to write large chunks
to memory, then commit them to the
database. For the C/C++ API, just do:
sqlite3_exec(db, "BEGIN TRANSACTION",
NULL, NULL, NULL);
...(INSERT statements)
sqlite3_exec(db, "COMMIT TRANSACTION", NULL, NULL, NULL);
Assuming db is your database pointer.
A good compromise is to wrap your INSERTS between BEGIN; and END; keyword i.e:
BEGIN;
INSERT INTO table VALUES ();
INSERT INTO table VALUES ();
...
END;
Depending on the size of the data and the amount of RAM available, one of the best performance gains will occur by setting sqlite to use an all-in-memory database rather than writing to disk.
For in-memory databases, pass NULL as the filename argument to sqlite3_open and make sure that TEMP_STORE is defined appropriately
(All of the above text is excerpted from my own answer to a separate sqlite-related question)
I found this to be a good mix for an one shot long import.
.echo ON
.read create_table_without_pk.sql
PRAGMA cache_size = 400000; PRAGMA synchronous = OFF; PRAGMA journal_mode = OFF; PRAGMA locking_mode = EXCLUSIVE; PRAGMA count_changes = OFF; PRAGMA temp_store = MEMORY; PRAGMA auto_vacuum = NONE;
.separator "\t" .import a_tab_seprated_table.txt mytable
BEGIN; .read add_indexes.sql COMMIT;
.exit
source: http://erictheturtle.blogspot.be/2009/05/fastest-bulk-import-into-sqlite.html
some additional info: http://blog.quibb.org/2010/08/fast-bulk-inserts-into-sqlite/
If you are just inserting once, I may have a dirty trick for you.
The idea is simple, first inserting into a memory database, then backup and finally restore to your original database file.
I wrote the detailed steps at my blog. :)
I do a bulk insert with this method:
colnames = ['col1', 'col2', 'col3']
nrcols = len(colnames)
qmarks = ",".join(["?" for i in range(nrcols)])
stmt = "INSERT INTO tablename VALUES(" + qmarks + ")"
vals = [[val11, val12, val13], [val21, val22, val23], ..., [valn1, valn2, valn3]]
conn.executemany(stmt, vals)
colnames must be in the order of the column names in the table
vals is a list of db rows
each row must have the same length, and
contain the values in the correct order
Note that we use executemany, not execute