I have a reasonable sized ( around 40k lines) machine learning system written in C++. This is still in active development and I need to run experiments regularly even as I make changes to my code.
The output of my experiments is captured in simple text files. What I would like to do when looking at these results is have some way of figuring out the exact version of the code that produced it. I usually have around 5 to 6 experiments running simultaneously, each on a slightly different version of the code.
I would like to know for instance that a set of results was obtained by compiling version 1 of file A, version 2 of file B etc (I just need some identifier and the output of "git describe" will do fine here ).
My idea is to somehow include this info when compiling the binary. This way, this can be printed out along with the results.
Any suggestions how this can be done in a nice way. In particular, any nice way of doing this with git?
I generate a single source file as part of my build process that looks like this:
static const char version_cstr[] = "93f794f674 (" __DATE__ ")";
const char * version()
{
return version_cstr;
}
Then its easy to log the version out on startup.
I originally used a DEFINE on the command line, but that meant every version change everything got recompiled by the build system - not nice for a big project.
Here's the fragment of scons I use for generating it, maybe you can adapt it to your needs.
# Lets get the version from git
# first get the base version
git_sha = subprocess.Popen(["git","rev-parse","--short=10","HEAD"], stdout=subprocess.PIPE ).communicate()[0].strip()
p1 = subprocess.Popen(["git", "status"], stdout=subprocess.PIPE )
p2 = subprocess.Popen(["grep", "Changed but not updated\\|Changes to be committed"], stdin=p1.stdout,stdout=subprocess.PIPE)
result = p2.communicate()[0].strip()
if result!="":
git_sha += "[MOD]"
print "Building version %s"%git_sha
def version_action(target,source,env):
"""
Generate file with current version info
"""
fd=open(target[0].path,'w')
fd.write( "static const char version_cstr[] = \"%s (\" __DATE__ \")\";\nconst char * version()\n{\n return version_cstr;\n}\n" % git_sha )
fd.close()
return 0
build_version = env.Command( 'src/autogen/version.cpp', [], Action(version_action) )
env.AlwaysBuild(build_version)
You can use $Id:$ in your source file, and Git will substitute that with the sha1 hash, if you add the file containing this phrase in .gitattributes with the option "ident" (see gitattributes).
Related
I installed the Castro library for astronomical modelling, and I followed the build setup described here. I successfully changed the parameters in GNUmakefile (as described in the tutorial), but when I ran the command make in ....Castro/Exec/hydro_tests/Sedov directory, I got several errors, the majority of them being 'M_PI' was not declared in this scope. I googled about the error, and the majority of the answers were referring to building of own code (one file, or so). The main reason was clear: I am using MS stuff (I am running Windows, with installed g++ and VS compiler) and M_PI is not supported by default. Instead, I should define _USE_MATH_DEFINES.
The problem is that I want to have a convenient solution: I don't want to search for every instance of M_PI and replace it, or add to every file #define _USE_MATH_DEFINES. Instead, I want to alter the make command so that _USE_MATH_DEFINES is defined in every file. Unfortunately, I haven't yet used make much, so I don't know how to do this.
The GNUmakefile (which is the only make-related file) doesn't help much:
PRECISION = DOUBLE
PROFILE = FALSE
DEBUG = FALSE
DIM = 2
COMP = gnu
USE_MPI = FALSE
USE_OMP = FALSE
USE_MHD = FALSE
USE_FORT_MICROPHYSICS := FALSE
BL_NO_FORT := TRUE
# define the location of the CASTRO top directory
CASTRO_HOME := ../../..
# This sets the EOS directory in $(MICROPHYSICS_HOME)/EOS
EOS_DIR := gamma_law
# This sets the network directory in $(MICROPHYSICS_HOME)/Networks
NETWORK_DIR := general_null
NETWORK_INPUTS = gammalaw.net
Bpack := ./Make.package
Blocs := .
include $(CASTRO_HOME)/Exec/Make.Castro
I, therefore, ask you to give the correct make command, or give any other option of bypassing the error.
EDIT: as #MadScientist points out, the position of DEFINES= in the commandline matters:
make DEPENDS=... is an overriding variable which ignores extensions
DEPENDS=... make sets a value from the environment which is fine.
I originally had the first form, which is wrong.
It looks like that system extends preprocessor defines set in the DEFINES variable, so the following should work:
DEFINES=-D__USE_MATH_DEFINES make
If you don't want to type that everytime, you can do
export DEFINES=-D__USE_MATH_DEFINES
once and then call
make several times.
I've decided to create a 'question' which is a partial answer and relates to another question (but to which there may well be better answers.) The rationale here is that I'm currently unable to comment or post to this post below due to my terrible reputation ... (I'm new to this site I found the answers to another post on stack overflow linked here useful, but they didnt in the end answer the question I had.)
Basically I was finding that projects would completely rebuild when switching from using MSBuild on the command line (wrapped up in a set of scripts) on a set of automatically generated sln/vcproj files to a build in developer studio VS2013. Turning on diagnostics in developer studio did not give any indication to why.
After a fair bit of poking around I discovered that the reason for this is that when MSBuild was doing the build, deep down files with the extension .lastbuildstate that get created by the build in the tlog directories contained the entry
VCToolArchitecture=Native64Bit
When I then loaded the project into developer studio, a complete rebuild would occur of the same configuration. The build would change these files: now they'd contain
VCToolArchitecture=Native32Bit
Even though I was definitely doing a 64 bit build ... both MSBuild and developer studio I believe are actually 32 bit tools that are doing a 64 bit build, so why MSBuild writes Native64Bit is a mystery here!
I was able to work around the problem then by scripting a replacement of Native64Bit with Native32Bit after a command line build before a build in developer studio. This then seems to not rebuild everything and the resultant execs seem to execute just fine ... but I cant help feeling that this sort of monkeying around should be unnecessary.
Intrigued to know if others have seen similar issues and what a better workaround would be (e.g. is there some masterful MSBuild switch that controls this?)
I've decided to mark this post answered as the question is effectively an answer, but if useful, here is a script, written in python, that may aid folk who have a similar problem to the one I was having.
The script will walk through the directory tree (provided as the first argument) and edit any files with the extension ".lastbuildstate" so that the string Native64Bit is replaced with Native32Bit.
If you provide a second argument the script will do the reverse so Native32Bit will be replaced with Native64Bit.
import os
import string
def modify_file_without_changing_modification_time(file, to_32 = True):
if to_32 == True:
from_string = "Native64Bit"
to_string = "Native32Bit"
else:
from_string = "Native32Bit"
to_string = "Native64Bit"
mtime = os.path.getmtime(file)
atime = os.path.getatime(file)
newlines = None
with open(file,"r") as f:
lines = f.readlines()
newlines = [ string.join(string.split(line,from_string),to_string) for line in lines ]
with open(file,"w") as f:
for line in newlines:
f.write(line)
os.utime(file,(atime,mtime))
def set_buildstate(directory, to32):
for root, dirs, files in os.walk("."):
for name in files:
fullpath = os.path.join(root, name)
if os.path.splitext(fullpath)[1] == ".lastbuildstate":
modify_file_without_changing_modification_time(fullpath,to32)
if __name__ == "__main__":
import sys
# By default sets build state to 32 bit (which prevents developer studio rebuilds)
to_32 = True
if len(sys.argv) > 2:
to_32 = False
set_buildstate(sys.argv[1],to_32)
I'm interested in implementing a program written in C++ inside a Ruby on Rails 4 App.
The C++ app runs in console and works by giving it a file, it processes it and then it outputs another file.
I found 'Ruby Inline', a gem that allegedly allows the use of foreign code:
https://rubygems.org/gems/RubyInline
gem 'RubyInline', '~> 3.12', '>= 3.12.4'
Inline allows you to write foreign code within your ruby code. It automatically determines if the code in question has changed and builds it only when necessary. The extensions are then automatically loaded into the class/module that defines it. You can even write extra builders that will allow you to write inlined code in any language. Use Inline::C as a template and look at Module#inline for the required API.
According to the documentation, it allows the use of C and C++ programs like this:
require 'inline'
class MyTest
inline(:C) do |builder|
builder.include '<iostream>'
builder.add_compile_flags '-x c++', '-lstdc++'
builder.c '
void hello(int i) {
while (i-- > 0) {
std::cout << "hello" << std::endl;
}
}'
end
end
t = MyTest.new()
t.hello(3)
The problem is that I'm not entirely sure how to implement it with a larger C++ program made out of multiple files, I'm not proficient in C++.
This seems like a gem to add the C++ code inside a ruby script and the C++ program is made out of multiple files and folders.
Will Ruby Inline work for what I want to do or is there a better option out there?
Is there a way to call the C++ app directly without putting it inside a ruby file?
The file to be given to the C++ app will be uploaded via the web interface, how can I feed it to the C++ program so it outputs the result?
Ruby supports the functionality to run native commands using the ` character. For example, we have a C++ program below, that takes two arguments, and then prints to standard output both of them, concatenated together. The course code for this program can be seen below:
#include <iostream>
int main(int argc, char ** argv){
std::cout << argv[1] << argv[2] << std::endl;
return 0;
}
While the C++ code, really only needs to be compiled a single time, and this step wouldn't likely be necessary, we can compile it from within ruby by doing the following: (You can replace g++ with the compiler of your choice.) We will set the variable success to true if the command successfully compiles, false otherwise.
pathToSource = "your/path/to/file.cpp"
outputBinaryFile = "your/desired/output/file.bin"
success = system("g++ #{pathToSource} -o #{outputBinaryFile}")
Now that we have compiled our code, we can execute it by running the following in ruby:
word1 = "Hello"
word2 = "World"
commandOutput = `./#{outputBinaryFile} '#{word1}' '#{word2}'`
Which will set commandOutput to: "HelloWorld\n"
You may need to change small things such as the compiler depending on the system that you are running, but this should function for you on *NIX systems.
I have created a game with Python 2.7 and am having some issues with the running of it after converting it to an executable. If I change to the directory through cmd it allows me to run it with all the formatting that has been applied, as such:
If I choose to access it through simply executing the game, it seems to be reading the different entities in the class containing the colours from the backslash as a string.
The class that is providing the information is as follows,
class bcolors:
HEADER = '\033[95m'
OKBLUE = '\033[94m'
OKGREEN = '\033[92m'
WARNING = '\033[93m'
FAIL = '\033[91m'
ENDC = '\033[0m'
The general code looks like this:
print bcolors.FAIL + ("Try combining the cd command and room you want to enter") + bcolors.END
If you are running Windows, you may need to install Ansicon. This link will probably be of use to you: Colorama for Python, Not returning colored print lines on Windows
So I have a software program that for reasons that are beyond this post, I will not include but to put it simply, I'd like to "MOD" the original software. The program is launched from a Windows Application named ViaNet.exe with accompanying DLL files such as ViaNetDll.dll. The Application is given an argument such as ./Statup.cat. There is also a WatchDog process that uses the argument ./App.cat instead of the former.
I was able to locate a log file buried in my Windows/Temp folder for the ViaNet.exe Application. Looking at the log it identifies files such as:
./Utility/base32.atc:_Encode32 line 67
./Utilities.atc:MemFun_:Invoke line 347
./Utilities.atc:_ForEachProperty line 380
./Cluster/ClusterManager.atc:ClusterManager:GetClusterUpdates line 1286
./Cluster/ClusterManager.atc:ClusterManager:StopSync line 505
./Cluster/ClusterManager.atc:ConfigSynchronizer:Update line 1824
Going to those file locations reveal files by those names, but not ending with .atc but instead .cat. The log also indicates some sort of Class, Method and Line # but .cat files are in binary form.
Searching the program folder for any files with the extension .atc reveals three -- What I can assume are uncompiled .cat files -- files. Low and behold, once opened it's obviously some sort of source code -- with copyright headers, lol.
global ConfigFolder, WriteConfigFile, App, ReadConfigFile, CreateAssocArray;
local mgrs = null;
local email = CreateAssocArray( null);
local publicConfig = ReadConfigFile( App.configPath + "\\publicConfig.dat" );
if ( publicConfig != null )
{
mgrs = publicConfig.cluster.shared.clusterGroup[1].managers[1];
local emailInfo = publicConfig.cluster.shared.emailServer;
if (emailInfo != null)
{
if (emailInfo.serverName != "")
{
email.serverName = emailInfo.serverName;
}
if (emailInfo.serverEmailAddress != "")
{
email.serverEmailAddress = emailInfo.serverEmailAddress;
}
if (emailInfo.adminEmailAddress != null)
{
email.adminEmailAddress = emailInfo.adminEmailAddress;
}
}
}
if (mgrs != null)
{
WriteConfigFile( ConfigFolder + "ZoneInfo.dat", mgrs);
}
WriteConfigFile( ConfigFolder + "EmailInfo.dat", email);
So to end this as simply as possible, I'm trying to find out two things. #1 What Programming Language is this? and #2 Can the .cat be decompiled back to .atc. files? -- and vice versa. Looking at the log it would appear that the Application is decoding/decompiling the .cat files already to interpret them verses running them as bytecode/natively. Searching for .atc on Google results in AutoCAD. But looking at the results, shows it to be some sort of palette files, nothing source code related.
It would seem to me that if I can program in this unknown language, let alone, decompile the existing stuff, I might get lucky with modding the software. Thanks in advance for any help and I really really hope someone has an answer for me.
EDIT
So huge news people, I've made quite an interesting discovery. I downloaded a patch from the vendor, it contained a batch file that was executing ViaNet.exe Execute [Patch Script].atc. I quickly discovered that you can use Execute to run both .atc and .cat files equally, same as with no argument. Once knowing this I assumed that there must be various arguments you can try, well after a random stroke of luck, there is one. That being Compile [Script].atc. This argument will compile also any .atc file to .cat. I've compiled the above script for comparison: http://pastebin.com/rg2YM8Q9
So I guess the goal now is to determine if it's possible to decompile said script. So I took a step further and was successful at obtaining C++ pseudo code from the ViaNet.exe and ViaNetDll.dll binaries, this has shed tons of understanding on the proprietary language and it's API they use. From what I can tell each execution is decompiled first then ran thru the interpreter. They also have nicknamed their language ATCL, still no idea what it stands for. While searching the API, I found several debug methods with names like ExecuteFile, ExecuteString, CompileFile, CompileString, InspectFunction and finally DumpObjCode. With the DumpObjCode method I'm able to perform some sort of dump of script files. Dump file for above script: http://pastebin.com/PuCCVMPf
I hope someone can help me find a pattern with the progress I made. I'm trying my best to go over the pseudo code but I don't know C++, so I'm having a really hard time understanding the code. I've tried to seperate what I can identify as being the compile script subroutines but I'm not certain: http://pastebin.com/pwfFCDQa
If someone can give me an idea of what this code snippet is doing and if it looks like I'm on the right path, I'd appreciate it. Thank you in advanced.