OpenGL glutInit() : XOpenDisplay() causing segmentation fault - c++

Question

I'm carrying out a project for virtualization of CUDA API. The project is based on QEMU hyper-visor. I'm using the latest version 2.6.0rc3. I have completed the core module and this question is regarding demoing it.QEMU 2.6.0rc3 has OpenGL support.
I ran the following program on the VM to test OpenGL support & it executed without any issue.
#include <GL/freeglut.h>
#include <GL/gl.h>
void renderFunction()
{
glClearColor(0.0, 0.0, 0.0, 0.0);
glClear(GL_COLOR_BUFFER_BIT);
glColor3f(1.0, 1.0, 1.0);
glOrtho(-1.0, 1.0, -1.0, 1.0, -1.0, 1.0);
glBegin(GL_POLYGON);
glVertex2f(-0.5, -0.5);
glVertex2f(-0.5, 0.5);
glVertex2f(0.5, 0.5);
glVertex2f(0.5, -0.5);
glEnd();
glFlush();
}
int main(int argc, char** argv)
{
glutInit(&argc, argv);
glutInitDisplayMode(GLUT_SINGLE);
glutInitWindowSize(500,500);
glutInitWindowPosition(100,100);
glutCreateWindow("OpenGL - First window demo");
glutDisplayFunc(renderFunction);
glewInit();
glutMainLoop();
return 0;
}
I also used NVIDIA samples graphics demo named "simpleGL" available with CUDA 6.5 toolkit at https://developer.nvidia.com/cuda-toolkit-65. The demo uses OpenGL to depict waveforms and CUDA for underlying calculations to simulate it. When I run this demo program, a segmentation fault occurs at the glutInit() call. Here's the related code segment from the demo.
bool initGL(int argc, char **argv)
{
glutInit(&argc, argv);
glutInitDisplayMode(GLUT_SINGLE);
glutInitWindowSize(window_width, window_height);
glutCreateWindow("Cuda GL Interop (VBO)");
glutDisplayFunc(display);
glutKeyboardFunc(keyboard);
glutMotionFunc(motion);
glutTimerFunc(REFRESH_DELAY, timerEvent,0);
// initialize necessary OpenGL extensions
glewInit();
if (! glewIsSupported("GL_VERSION_2_0 "))
{
fprintf(stderr, "ERROR: Support for necessary OpenGL extensions missing.");
fflush(stderr);
return false;
}
// default initialization
glClearColor(0.0, 0.0, 0.0, 1.0);
glDisable(GL_DEPTH_TEST);
// viewport
glViewport(0, 0, window_width, window_height);
// projection
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluPerspective(60.0, (GLfloat)window_width / (GLfloat) window_height, 0.1, 10.0);
SDK_CHECK_ERROR_GL();
return true;
}
Here's the gdb call stack.
#0 0x00007ffff57d2872 in XOpenDisplay ()
from /usr/lib/x86_64-linux-gnu/libX11.so.6
#1 0x00007ffff76af2a3 in glutInit ()
from /usr/lib/x86_64-linux-gnu/libglut.so.3
#2 0x000000000040394d in initGL(int, char**) ()
#3 0x0000000000403b6a in runTest(int, char**, char*) ()
#4 0x00000000004037dc in main ()
According to my research the segmentation fault occurs when an attempt to open a window is made. My knowledge of internal working of OpenGL is very limited, some help in this regard is much appreciated. Thanks.

Answer 1

I'm carrying out a project for virtualization of CUDA API
Without support from NVidia I doubt you can do this on your own.
You're doing a few things that clash in a crass way:
First off you're running everything in a QEmu environment, which means, that, if you don't pass through the GPU via IOMMU virtualization into the VM there's nothing a CUDA runtime in there could work with. CUDA is designed to talk directly to the GPU.
Next you're using the Mesa OpenGL implementation inside the VM. Mesa has a dedicated backend to pass OpenGL commands through QEmu to a OpenGL implementation "outside" of it. This is more or less a remote procedure call and it piggybacks over the very same code paths that also implement indirect GLX via X11 transport.
CUDA internally links against libGL.so, but the libGL.so it expects to see is the one of the NVidia drivers, not some arbitrary libGL.so. Since libcuda.so and libGL.so come as a part of the same driver package, namely the NVidia drivers. There's certain internal "knowledge" about the particular libGL.so that the corresponding libcuda.so has and tries to use. Without the right libGL.so it won't work.
If you want to use CUDA in a VM (perfectly possible) you have to pass through the whole GPU into the VM. You can do this by loading the pci_stub kernel module, configuring the NVidia GPU as device to be attached to the stub, then launch the QEmu VM with pass through of the GPU device (actually it should also be possible to hot-plug passthrough it, but I never tried that). For this to work the nvidia kernel module must not have taken ownership of the GPU. So in case you have multiple NVidia GPUs and want to pass through only a subset of them, you have to attach those to the pci_stub before loading the nvidia kernel module. Then inside the VM you can use the NVidia drivers as usual.