Compiling PythonService.cpp from win32 python extension build - c++

Question

I need some help with building pythonservice.exe (PythonService.cpp) from https://github.com/kovidgoyal/pywin32/tree/master/win32/src.
It is said (https://github.com/kovidgoyal/pywin32) that VS2015 is required, so I downloaded the community version. I opened a new win32 console project and imported to source files only the PythonService.cpp.
In project settings I have:
Include directories: C:\Python27_32bit\include;C:\pywin32-master\win32\src;$(VC_IncludePath);$(WindowsSDK_IncludePath);
Library directories: C:\Python27\Lib\site-packages\win32\libs;C:\Python27_32bit\libs;C:\pywin32-master\win32\src;$(LibraryPath);C:\pywin32-master\lib\x32\win32\libs;
Now, when building I get the unresolved external symbol error:
1>PythonService.obj : error LNK2001: unresolved external symbol "__declspec(dllimport) char * __cdecl GetPythonTraceback(struct _object *,struct _object *,struct _object *)" (__imp_?GetPythonTraceback##YAPADPAU_object##00#Z)
GetPythonTraceback is defined in PyWinTypesmodule.cpp but when I add it to my project it gives other errors requiring other unresolved functions. I don't want to compile the whole win32 project. All I need is pythonservices.exe. Is this possible to accomplish without compiling the whole project?
Would be thankful for any help!
Thanks!
Alexei

Answer 1

I eventually found a solution to my question (not without help from #CristiFati 's answer!). I downloaded https://github.com/mhammond/pywin32/tree/b222 and used the setup.py script to build the whole package. Also, I installed the required SDK package (for python 2.6+ Microsoft Windows SDK for Windows 7 and .NET Framework 4 (version 7.1)). After digging a little bit I found that the compilation and linkage are done with Visual C++ for Python 9.0 and below are the compilation and linkage recipes for pythonservice.pyd and pythonservice.exe (python lists for cmd parameter to calls of os.spawnv(os.P_WAIT, executable, cmd):
For pythonservice.pyd:
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\WinSDK\\Bin\\x64\\mc.exe"','-h', 'win32/src', '-r', 'build\\temp.win-amd64-2.7\\Release\\win32/src', 'win32/src/PythonServiceMessages.mc']
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\WinSDK\\Bin\\x64\\rc.exe"','/fobuild\\temp.win-amd64-2.7\\Release\\win32/src/PythonServiceMessages.res','build\\temp.win-amd64-2.7\\Release\\win32/src\\PythonServiceMessages.rc']
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe"','/c', '/nologo', '/Ox','/W3','/GS', '/DNDEBUG', '/MD', '-DDISTUTILS_BUILD', '-D_CRT_SECURE_NO_WARNINGS','-Icom/win32com/src/include', '-Iwin32/src', '-IC:\\Python27\\include', '-IC:\\Python27\\PC','"-IC:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.1A\\include"', '/Tpwin32/src/PythonService.cpp','/Fobuild\\temp.win-amd64-2.7\\Release\\win32/src/PythonService.obj', '-DPYSERVICE_BUILD_DLL', '/Zi','/Fdbuild\\temp.win-amd64-2.7\\Release\\servicemanager_vc.pdb', '/EHsc', '/DUNICODE', '/D_UNICODE', '/DWINNT']
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\link.exe"', '/DLL', '/nologo', '/INCREMENTAL:NO', '/LIBPATH:C:\\Python27\\libs', '/LIBPATH:C:\\Python27\\PCbuild\\amd64', '/LIBPATH:C:\\Python27\\PC\\VS9.0\\amd64', '/LIBPATH:build\\temp.win-amd64-2.7\\Release', '"/LIBPATH:C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.1A\\lib\\x64"', 'user32.lib', 'ole32.lib', 'advapi32.lib', 'shell32.lib', '/EXPORT:initservicemanager', 'build\\temp.win-amd64-2.7\\Release\\win32/src/PythonServiceMessages.res', 'build\\temp.win-amd64-2.7\\Release\\win32/src/PythonService.obj', '/OUT:build\\lib.win-amd64-2.7\\win32\\servicemanager.pyd', '/IMPLIB:build\\temp.win-amd64-2.7\\Release\\win32/src\\servicemanager.lib', '/MANIFEST:NO', '/MACHINE:amd64', '/BASE:0x1e7d0000', '/DEBUG', '/PDB:build\\temp.win-amd64-2.7\\Release\\servicemanager.pdb']
For pythonservice.exe:
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\cl.exe"','/c', '/nologo', '/Ox', '/W3', '/GS-', '/DNDEBUG', '/MD', '-DDISTUTILS_BUILD', '-D_CRT_SECURE_NO_WARNINGS','-Icom/win32com/src/include', '-Iwin32/src', '-IC:\\Python27\\include', '-IC:\\Python27\\PC','"-IC:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.1A\\include"','/Tpwin32\\src\\PythonService.cpp', '/Fobuild\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\PythonService.obj','/Zi', '/Fdbuild\\temp.win-amd64-2.7\\Release\\pythonservice_vc.pdb', '/EHsc', '/DUNICODE', '/D_UNICODE', '/DWINNT']
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\WinSDK\\Bin\\x64\\rc.exe"','-DDISTUTILS_BUILD', '-D_CRT_SECURE_NO_WARNINGS', '-Icom/win32com/src/include', '-Iwin32/src', '-IC:\\Python27\\include','-IC:\\Python27\\PC', '"-IC:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.1A\\include"','/fobuild\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\PythonService.res', 'win32\\src\\PythonService.rc']
['"C:\\Users\\username\\AppData\\Local\\Programs\\Common\\Microsoft\\Visual C++ for Python\\9.0\\VC\\Bin\\amd64\\link.exe"','/nologo', '/INCREMENTAL:NO', '/LIBPATH:C:\\Python27\\libs', '/LIBPATH:C:\\Python27\\PCbuild\\amd64','/LIBPATH:C:\\Python27\\PC\\VS9.0\\amd64', '/LIBPATH:build\\temp.win-amd64-2.7\\Release','"/LIBPATH:C:\\Program Files (x86)\\Microsoft SDKs\\Windows\\v7.1A\\lib\\x64"', 'user32.lib', 'advapi32.lib', 'ole32.lib','shell32.lib', 'build\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\PythonService.obj','build\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\PythonService.res','/OUT:build\\lib.win-amd64-2.7\\win32\\pythonservice.exe','/MANIFESTFILE:build\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\pythonservice.exe.manifest','/SUBSYSTEM:CONSOLE', '/MACHINE:amd64', '/DEBUG', '/PDB:build\\temp.win-amd64-2.7\\Release\\pythonservice.pdb']
['mt.exe', '-nologo', '-manifest','build\\temp.win-amd64-2.7\\Release\\pythonservice\\win32\\src\\pythonservice.exe.manifest.orig','-outputresource:build\\lib.win-amd64-2.7\\win32\\pythonservice.exe;1']
['C:\\Python27\\python.exe', 'C:\\Python27\\Lib\\site-packages\\win32\\lib\\win32verstamp.py', '--version=2.7.222.1', '--comments=https://github.com/mhammond/pywin32', '--original-filename=pythonservice.exe','--product=PyWin32', '--quiet', 'build\\lib.win-amd64-2.7\\win32\\pythonservice.exe']

Answer 2

The 1st thing that I see here is combining Python 2.7 and VStudio 2015. According to [Python.Wiki]: WindowsCompilers, for VStudion 2015 (or 14.0), you should use Python 3.5 or Python 3.6 (I have the former). Check [SO]: Simstring (python) installation in windows (#CristiFati's answer) - (the 1st part), about compatibilities and restrictions in terms of:
Platform
Tools versions
Architecture
Also, some (basic) information about building C code on Win: [SO]: LNK2005 Error in CLR Windows Form (#CristiFati's answer)
Downloaded original PyWin32 from: [GitHub]: mhammond/pywin32 - (b222) (to pywin32-b222.zip). The "${PYWIN32_SRC_DIR}\win32\src\PythonService.cpp" is identical to its correspondent from the URL you mentioned
Create a dir and cd to it (should be empty). This will be the %ROOT_DIR%, and all the paths that I'm going to use will be relative to it (except of course for absolute ones), and this will be the default dir (when unspecified)
Unpack the downloaded .zip somewhere. We (directly) need from it:
"${PYWIN32_SRC_DIR}\win32\src\PythonService.cpp" - as you mentioned it, the source code
"${PYWIN32_SRC_DIR}\win32\src\PythonServiceMessages.mc" - message text file
I copied them in a dir called src:
E:\Work\Dev\StackOverflow\q048931801>dir /b
src
E:\Work\Dev\StackOverflow\q048931801>dir /b "src"
PythonService.cpp
PythonServiceMessages.mc
Since the build involves resources and Custom Build Steps (that I'm not very good at), and also it doesn't involve lots of files, I'm not going to use the IDE
The next best approach would be to use Makefiles, but since VStudio has a bug [MS.MSDN]: ntwin32.mak not found:
It affects VStudio 2015 as well
It requires a one time fix
I did the fix on my machine and it works, but you most likely didn't on yours, and I don't want to unnecessarily burden you with this step
I am going to do everything manually
Setting up VStudio environment:
[MS.Docs]: Use the MSVC toolset from the command line
You could also check [SO]: How to build a DLL version of libjpeg 9b? (#CristiFati's answer) (1st part) for more details about manually building
I'm using vcvarsall.bat ([MS.MSDN]: Where is vcvarsall.bat file?) from the command line
E:\Work\Dev\StackOverflow\q048931801>"c:\Install\x86\Microsoft\Visual Studio Community\2015\VC\vcvarsall.bat" amd64
You'll (obviously) need to supply your VStudio installation path
amd64 is because I only have the 64bit (x64 or AMD64) version of Python 3.5 installed
Compile the message file into a resource file - using mc.exe ([MS.Docs]: Message Compiler (MC.exe))
E:\Work\Dev\StackOverflow\q048931801>mc "src\PythonServiceMessages.mc"
MC: Compiling src\PythonServiceMessages.mc
E:\Work\Dev\StackOverflow\q048931801>dir /b
MSG00001.bin
PythonServiceMessages.h
PythonServiceMessages.rc
src
As seen, it generated 3 new files
Compile the resource file - using rc.exe ([MS.Docs]: Resource Compiler)
E:\Work\Dev\StackOverflow\q048931801>rc /NOLOGO /r "PythonServiceMessages.rc"
E:\Work\Dev\StackOverflow\q048931801>dir /b
MSG00001.bin
PythonServiceMessages.h
PythonServiceMessages.rc
PythonServiceMessages.res
src
Done with the resources, going to the code (PythonService.cpp). As you already mentioned, it won't compile OOTB. I'm not going to list all the failed attempts, instead I'm going to list the steps I took, and only show the final (successful) compile. So, the file needs code from other files. The simplest way is to copy it from the other files and place it in this one. I chose to paste it around line #1530 (just before the "* Entry points" section):
PyWinTypesModule.cpp - GetPythonTraceback function (lines 1142 - 1235)
PyUnicode.cpp - PyWin_AutoFreeBstr::PyWin_AutoFreeBstr, PyWin_AutoFreeBstr::~PyWin_AutoFreeBstr, PyWin_AutoFreeBstr::SetBstr (lines 204 - 219)
Either apply the changes manually, or save:
--- PythonService.cpp.orig 2018-01-20 21:43:10.000000000 +0200
+++ PythonService.cpp 2018-02-23 20:02:20.186966800 +0200
## -1529,6 +1529,120 ##
}
}
+// #TODO: cfati - copied from PyUnicode.cpp (lines 204 - 219)
+PyWin_AutoFreeBstr::PyWin_AutoFreeBstr( BSTR bstr /*= NULL*/ )
+ : m_bstr(bstr)
+{
+ return;
+}
+
+PyWin_AutoFreeBstr::~PyWin_AutoFreeBstr()
+{
+ SysFreeString(m_bstr);
+}
+
+void PyWin_AutoFreeBstr::SetBstr( BSTR bstr )
+{
+ SysFreeString(m_bstr);
+ m_bstr = bstr;
+}
+
+
+// #TODO: cfati - copied from PyWinTypesModule.cpp (lines 1142 - 1235)
+// Function to format a python traceback into a character string.
+#define GPEM_ERROR(what) {errorMsg = "<Error getting traceback - " ## what ## ">";goto done;}
+char *GetPythonTraceback(PyObject *exc_type, PyObject *exc_value, PyObject *exc_tb)
+{
+ // Sleep (30000); // Time enough to attach the debugger (barely)
+ char *result = NULL;
+ char *errorMsg = NULL;
+ PyObject *modStringIO = NULL;
+ PyObject *modTB = NULL;
+ PyObject *obFuncStringIO = NULL;
+ PyObject *obStringIO = NULL;
+ PyObject *obFuncTB = NULL;
+ PyObject *argsTB = NULL;
+ PyObject *obResult = NULL;
+
+ /* Import the modules we need - cStringIO and traceback */
+#if (PY_VERSION_HEX < 0x03000000)
+ modStringIO = PyImport_ImportModule("cStringIO");
+#else
+ // In py3k, cStringIO is in "io"
+ modStringIO = PyImport_ImportModule("io");
+#endif
+
+ if (modStringIO==NULL) GPEM_ERROR("cant import cStringIO");
+ modTB = PyImport_ImportModule("traceback");
+ if (modTB==NULL) GPEM_ERROR("cant import traceback");
+
+ /* Construct a cStringIO object */
+ obFuncStringIO = PyObject_GetAttrString(modStringIO, "StringIO");
+ if (obFuncStringIO==NULL) GPEM_ERROR("cant find cStringIO.StringIO");
+ obStringIO = PyObject_CallObject(obFuncStringIO, NULL);
+ if (obStringIO==NULL) GPEM_ERROR("cStringIO.StringIO() failed");
+
+ /* Get the traceback.print_exception function, and call it. */
+ obFuncTB = PyObject_GetAttrString(modTB, "print_exception");
+ if (obFuncTB==NULL) GPEM_ERROR("cant find traceback.print_exception");
+ argsTB = Py_BuildValue("OOOOO"
+#if (PY_VERSION_HEX >= 0x03000000)
+ "i"
+ // Py3k has added an undocumented 'chain' argument which defaults to True
+ // and causes all kinds of exceptions while trying to print a goddam exception
+#endif
+ ,
+ exc_type ? exc_type : Py_None,
+ exc_value ? exc_value : Py_None,
+ exc_tb ? exc_tb : Py_None,
+ Py_None, // limit
+ obStringIO
+#if (PY_VERSION_HEX >= 0x03000000)
+ ,0 // Goddam undocumented 'chain' param, which defaults to True
+#endif
+ );
+ if (argsTB==NULL) GPEM_ERROR("cant make print_exception arguments");
+
+ obResult = PyObject_CallObject(obFuncTB, argsTB);
+ if (obResult==NULL){
+ // Chain parameter when True causes traceback.print_exception to fail, leaving no
+ // way to see what the original problem is, or even what error print_exc raises
+ // PyObject *t, *v, *tb;
+ // PyErr_Fetch(&t, &v, &tb);
+ // PyUnicodeObject *uo=(PyUnicodeObject *)v;
+ // DebugBreak();
+ GPEM_ERROR("traceback.print_exception() failed");
+ }
+ /* Now call the getvalue() method in the StringIO instance */
+ Py_DECREF(obFuncStringIO);
+ obFuncStringIO = PyObject_GetAttrString(obStringIO, "getvalue");
+ if (obFuncStringIO==NULL) GPEM_ERROR("cant find getvalue function");
+ Py_DECREF(obResult);
+ obResult = PyObject_CallObject(obFuncStringIO, NULL);
+ if (obResult==NULL) GPEM_ERROR("getvalue() failed.");
+
+ /* And it should be a string all ready to go - duplicate it. */
+ if (PyString_Check(obResult))
+ result = strdup(PyString_AsString(obResult));
+#if (PY_VERSION_HEX >= 0x03000000)
+ else if (PyUnicode_Check(obResult))
+ result = strdup(_PyUnicode_AsString(obResult));
+#endif
+ else
+ GPEM_ERROR("getvalue() did not return a string");
+
+done:
+ if (result==NULL && errorMsg != NULL)
+ result = strdup(errorMsg);
+ Py_XDECREF(modStringIO);
+ Py_XDECREF(modTB);
+ Py_XDECREF(obFuncStringIO);
+ Py_XDECREF(obStringIO);
+ Py_XDECREF(obFuncTB);
+ Py_XDECREF(argsTB);
+ Py_XDECREF(obResult);
+ return result;
+}
/*************************************************************************
*
as "src\PythonService.diff" That is a diff. See [SO]: Run/Debug a Django application's UnitTests from the mouse right click context menu in PyCharm Community Edition? (#CristiFati's answer) (Patching utrunner section) for how to apply patches on Win (basically, every line that starts with one "+" sign goes in, and every line that starts with one "-" sign goes out). I am using Cygwin:
E:\Work\Dev\StackOverflow\q048931801>"c:\Install\x64\Cygwin\Cygwin\AllVers\bin\patch.exe" "src\PythonService.cpp" "src\PythonService.diff"
patching file 'src\PythonService.cpp'
Compile the code - using cl.exe ([MS.Docs]: Compiling a C/C++ Program)
E:\Work\Dev\StackOverflow\q048931801>cl /GS /W1 /Zc:wchar_t /I"." /I"c:\Install\x64\Python\Python\3.5\include" /I"e:\Work\Dev\Fati\WinBuild\OPSWpython27\src\pywin32-b222\win32\src" /Gm- /O2 /Zc:inline /fp:precise /D "UNICODE" /D "_UNICODE" /D "WIN32" /D "WIN64" /D "NDEBUG" /D "BUILD_PYWINTYPES" /errorReport:prompt /WX- /Zc:forScope /Gd /MD /c "src\PythonService.cpp"
Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24215.1 for x64
Copyright (C) Microsoft Corporation. All rights reserved.
PythonService.cpp
E:\Work\Dev\StackOverflow\q048931801>dir /b "*.obj"
PythonService.obj
Arguments are standard ones ([MS.Docs]: Compiler Options Listed Alphabetically), specific for this case:
/D "BUILD_PYWINTYPES" (macro definition)
/D "WIN64" - 64bit
Include paths (/I) - you'll have to adapt them to match your paths:
"c:\Install\x64\Python\Python\3.5\include" (Python path)
"e:\Work\Dev\Fati\WinBuild\OPSWpython27\src\pywin32-b222\win32\src" (pywin32 path)
Link everything together - using link.exe ([MS.Docs]: Linking)
E:\Work\Dev\StackOverflow\q048931801>link /NOLOGO /MACHINE:X64 /SUBSYSTEM:CONSOLE /ERRORREPORT:PROMPT /LIBPATH:"c:\Install\x64\Python\Python\3.5\libs" /NXCOMPAT /DYNAMICBASE "kernel32.lib" "user32.lib" "advapi32.lib" "shell32.lib" "ole32.lib" "oleaut32.lib" "PythonService.obj" "PythonServiceMessages.res" /OUT:"PythonService.exe"
Creating library PythonService.lib and object PythonService.exp
E:\Work\Dev\StackOverflow\q048931801>dir /b "*.exe"
PythonService.exe
Again, arguments are standard ones ([MS.Docs]: Linker Options), specific for this case:
/MACHINE:X64 - 64bit
Library paths (/LIBPATH) - adapt it to match yours:
"c:\Install\x64\Python\Python\3.5\libs (Python path)
One last step before running the service. Since it depends on python35.dll, the OS must know where to look for it. Simplest way is to add its path to %PATH% ([MS.Docs]: Dynamic-Link Library Search Order)
E:\Work\Dev\StackOverflow\q048931801>set PATH=%PATH%;c:\Install\x64\Python\Python\3.5
E:\Work\Dev\StackOverflow\q048931801>PythonService.exe
P - Python Service Manager
Options:
-register - register the EXE - this should generally not be necessary.
-debug servicename [parms] - debug the Python service.
NOTE: You do not start the service using this program - start the
service using Control Panel, or 'net start service_name'