Vulkan RT BLAS build sometimes fail. No Validation Layer message - c++

Question

I have a problem when creating BLAS in Vulkan with Ray Tracing. Basically, not always, but often when I send the command "vkCmdBuildAccelerationStructuresKHR" via a commandBuffer in the Compute queue the VkDevice becomes VK_ERROR_DEVICE_LOST. The vkQueueSubmit returns VK_SUCCESS, but when I try to wait for the sent command to finish vkDeviceWaitIdle returns VK_ERROR_DEVICE_LOST. All the buffers used are allocated without errors and it is possible to obtain the address on the device. I also use the VMA (Vulkan Memory Management) library to manage the allocations. The buffers were created with the property VK_SHARING_MODE_EXCLUSIVE but are only used in the commandBuffer of the Compute queue. The real problem is that the validation layer does not give any error messages.
The code for creating vertex buffer:
VkBufferCreateInfo vkVertexBufferCreateInfo{};
vkVertexBufferCreateInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO;
vkVertexBufferCreateInfo.size = vertexSize;
vkVertexBufferCreateInfo.usage = VK_BUFFER_USAGE_ACCELERATION_STRUCTURE_BUILD_INPUT_READ_ONLY_BIT_KHR
| VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT
| VK_BUFFER_USAGE_VERTEX_BUFFER_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT;
vkVertexBufferCreateInfo.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
vkVertexBufferCreateInfo.pQueueFamilyIndices = VK_NULL_HANDLE;
vkVertexBufferCreateInfo.queueFamilyIndexCount = 0;
VmaAllocationCreateInfo vmaVertexBufferAllocationCreateInfo{};
vmaVertexBufferAllocationCreateInfo.flags = VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT;
vmaVertexBufferAllocationCreateInfo.requiredFlags = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
The code for creating index buffer:
VkBufferCreateInfo vkIndexBufferCreateInfo{};
vkIndexBufferCreateInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO;
vkIndexBufferCreateInfo.size = faceSize;
vkIndexBufferCreateInfo.usage = VK_BUFFER_USAGE_ACCELERATION_STRUCTURE_BUILD_INPUT_READ_ONLY_BIT_KHR
| VK_BUFFER_USAGE_TRANSFER_DST_BIT | VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT
| VK_BUFFER_USAGE_INDEX_BUFFER_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT;
vkIndexBufferCreateInfo.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
vkIndexBufferCreateInfo.pQueueFamilyIndices = VK_NULL_HANDLE;
vkIndexBufferCreateInfo.queueFamilyIndexCount = 0;
VmaAllocationCreateInfo vmaIndexBufferAllocationCreateInfo = {};
vmaIndexBufferAllocationCreateInfo.flags = VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT;
vmaIndexBufferAllocationCreateInfo.requiredFlags = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
The code to create struct for geometry info:
// Query the 64-bit vertex/index buffer device address value through which buffer memory
// can be accessed in a shader
std::optional<VkDeviceAddress> vertexBufferAddress = geometry.getVertexBuffer().getBufferDeviceAddress();
if (vertexBufferAddress.has_value() == false)
{
OV_LOG_ERROR("Fail to retrive the device address of the vertex buffer. Probably geometry not uploaded.");
return false;
}
std::optional<VkDeviceAddress> faceBufferAddress = geometry.getFaceBuffer().getBufferDeviceAddress();
if (faceBufferAddress.has_value() == false)
{
OV_LOG_ERROR("Fail to retrive the device address of the face buffer. Probably geometry not uploaded.");
return false;
}
VkDeviceOrHostAddressConstKHR vertexDeviceOrHostAddressConst = {};
vertexDeviceOrHostAddressConst.deviceAddress = vertexBufferAddress.value();
VkDeviceOrHostAddressConstKHR faceDeviceOrHostAddressConst = {};
faceDeviceOrHostAddressConst.deviceAddress = faceBufferAddress.value();
// Structure specifying a triangle geometry in a bottom-level acceleration structure
VkAccelerationStructureGeometryTrianglesDataKHR accelerationStructureGeometryTrianglesData = {};
accelerationStructureGeometryTrianglesData.sType =
VK_STRUCTURE_TYPE_ACCELERATION_STRUCTURE_GEOMETRY_TRIANGLES_DATA_KHR;
accelerationStructureGeometryTrianglesData.pNext = NULL;
// Vertex glm::vec3
accelerationStructureGeometryTrianglesData.vertexFormat = VK_FORMAT_R32G32B32_SFLOAT;
accelerationStructureGeometryTrianglesData.vertexData = vertexDeviceOrHostAddressConst;
// sizeof(float) * 3 => vertex
// sizeof(uint32_t) * 3 => normal / uv / tangent
accelerationStructureGeometryTrianglesData.vertexStride = sizeof(Ov::Geometry::VertexData);
// # vertices = vertex buffer size bytes / vertex stride
accelerationStructureGeometryTrianglesData.maxVertex = geometry.getNrOfVertices();
accelerationStructureGeometryTrianglesData.indexType = VK_INDEX_TYPE_UINT32;
accelerationStructureGeometryTrianglesData.indexData = faceDeviceOrHostAddressConst;
// transformData is a device or host address to memory containing an optional reference to
// a VkTransformMatrixKHR structure
accelerationStructureGeometryTrianglesData.transformData = transformData;
// Union specifying acceleration structure geometry data
VkAccelerationStructureGeometryDataKHR accelerationStructureGeometryData = {};
accelerationStructureGeometryData.triangles = accelerationStructureGeometryTrianglesData;
// Structure specifying geometries to be built into an acceleration structure
VkAccelerationStructureGeometryKHR& accelerationStructureGeometry = reserved->geometriesAS.emplace_back();
accelerationStructureGeometry.sType = VK_STRUCTURE_TYPE_ACCELERATION_STRUCTURE_GEOMETRY_KHR;
accelerationStructureGeometry.pNext = NULL;
accelerationStructureGeometry.geometryType = VK_GEOMETRY_TYPE_TRIANGLES_KHR;
accelerationStructureGeometry.geometry = accelerationStructureGeometryData;
accelerationStructureGeometry.flags = geometryFlags;
// Structure specifying build offsets and counts for acceleration structure builds
VkAccelerationStructureBuildRangeInfoKHR& accelerationStructureBuildRangeInfoKHR = reserved->geometriesBuildRangeAS.emplace_back();
// primitiveCount defines the number of primitives for a corresponding acceleration structure geometry.
accelerationStructureBuildRangeInfoKHR.primitiveCount = geometry.getNrOfFaces();
accelerationStructureBuildRangeInfoKHR.primitiveOffset = 0;
accelerationStructureBuildRangeInfoKHR.firstVertex = 0;
accelerationStructureBuildRangeInfoKHR.transformOffset = 0;
Here is the code for building BLAS:
// Structure specifying the geometry data used to build an acceleration structure.
reserved->accelerationStructureBuildGeometryInfo.sType =
VK_STRUCTURE_TYPE_ACCELERATION_STRUCTURE_BUILD_GEOMETRY_INFO_KHR;
reserved->accelerationStructureBuildGeometryInfo.pNext = NULL;
reserved->accelerationStructureBuildGeometryInfo.type = type;
reserved->accelerationStructureBuildGeometryInfo.flags = flags;
// VK_BUILD_ACCELERATION_STRUCTURE_MODE_BUILD_KHR => specifies that the destination acceleration
// structure will be built using the specified geometries.
reserved->accelerationStructureBuildGeometryInfo.mode = VK_BUILD_ACCELERATION_STRUCTURE_MODE_BUILD_KHR;
reserved->accelerationStructureBuildGeometryInfo.srcAccelerationStructure = VK_NULL_HANDLE;
reserved->accelerationStructureBuildGeometryInfo.dstAccelerationStructure = VK_NULL_HANDLE;
reserved->accelerationStructureBuildGeometryInfo.geometryCount = nrOfgeometriesStructuresAS;
// The index of each element of the pGeometries or ppGeometries members of VkAccelerationStructureBuildGeometryInfoKHR
// is used as the geometry index during ray traversal.The geometry index is available in ray shaders via the
// RayGeometryIndexKHR built - in, and is used to determine hitand intersection shaders executed
// during traversal.The geometry index is available to ray queries via the OpRayQueryGetIntersectionGeometryIndexKHR instruction.
reserved->accelerationStructureBuildGeometryInfo.pGeometries = geometriesStructuresAS.data();
reserved->accelerationStructureBuildGeometryInfo.ppGeometries = NULL;
reserved->accelerationStructureBuildGeometryInfo.scratchData = {};
// Structure specifying build sizes for an acceleration structure
VkAccelerationStructureBuildSizesInfoKHR accelerationStructureBuildSizesInfo = {};
accelerationStructureBuildSizesInfo.sType = VK_STRUCTURE_TYPE_ACCELERATION_STRUCTURE_BUILD_SIZES_INFO_KHR;
accelerationStructureBuildSizesInfo.pNext = NULL;
accelerationStructureBuildSizesInfo.accelerationStructureSize = 0;
accelerationStructureBuildSizesInfo.updateScratchSize = 0;
accelerationStructureBuildSizesInfo.buildScratchSize = 0;
// Retrieve the required size for an acceleration structure
// VK_ACCELERATION_STRUCTURE_BUILD_TYPE_DEVICE_KHR => requests the memory requirement for operations
// performed by the device.
PFN_vkGetAccelerationStructureBuildSizesKHR pvkGetAccelerationStructureBuildSizesKHR =
(PFN_vkGetAccelerationStructureBuildSizesKHR)vkGetDeviceProcAddr(logicalDevice.get().getVkDevice(), "vkGetAccelerationStructureBuildSizesKHR");
pvkGetAccelerationStructureBuildSizesKHR(logicalDevice.get().getVkDevice(),
VK_ACCELERATION_STRUCTURE_BUILD_TYPE_HOST_KHR,
&reserved->accelerationStructureBuildGeometryInfo,
&reserved->accelerationStructureBuildGeometryInfo.geometryCount,
&accelerationStructureBuildSizesInfo);
////////////////////
// Scratch buffer //
////////////////////
#pragma region ScratchBuffer
///////////////////////////
// Create scratch buffer //
///////////////////////////
// Create info buffer
VkBufferCreateInfo vkScratchBufferCreateInfo{};
vkScratchBufferCreateInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO;
// The second field of the struct is size, which specifies the size of the buffer in bytes
vkScratchBufferCreateInfo.size = accelerationStructureBuildSizesInfo.buildScratchSize;
// The third field is usage, which indicates for which purposes the data in the buffer
// is going to be used. It is possible to specify multiple purposes using a bitwise or.
// VK_BUFFER_USAGE_ACCELERATION_STRUCTURE_BUILD_INPUT_READ_ONLY_BIT_KHR => specifies that the buffer is suitable for
// use as a read-only input to an acceleration structure build.
// VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT => specifies that the buffer can be used to retrieve a buffer device address
// via vkGetBufferDeviceAddress and use that address to access the buffer’s memory from a shader.
vkScratchBufferCreateInfo.usage = VK_BUFFER_USAGE_ACCELERATION_STRUCTURE_BUILD_INPUT_READ_ONLY_BIT_KHR
| VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT | VK_BUFFER_USAGE_STORAGE_BUFFER_BIT;
// buffers can also be owned by a specific queue family or be shared between multiple
// at the same time.
// VK_SHARING_MODE_CONCURRENT specifies that concurrent access to any range or image subresource of the object
// from multiple queue families is supported.
vkScratchBufferCreateInfo.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
// From which queue family the buffer will be accessed.
vkScratchBufferCreateInfo.pQueueFamilyIndices = NULL;
vkScratchBufferCreateInfo.queueFamilyIndexCount = 0;
// Create allocation info
VmaAllocationCreateInfo vmaScratchBufferAllocationCreateInfo = {};
// VMA_ALLOCATION_CREATE_MAPPED_BIT => Set this flag to use a memory that will be persistently
// mappedand retrieve pointer to it. It is valid to use this flag for allocation made from memory
// type that is not HOST_VISIBLE. This flag is then ignored and memory is not mapped. This is useful
// if you need an allocation that is efficient to use on GPU (DEVICE_LOCAL) and still want to map it
// directly if possible on platforms that support it (e.g. Intel GPU).
vmaScratchBufferAllocationCreateInfo.flags = VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT;
// Flags that must be set in a Memory Type chosen for an allocation.
vmaScratchBufferAllocationCreateInfo.requiredFlags = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
if (!reserved->scratchBuffer.create(vkScratchBufferCreateInfo, vmaScratchBufferAllocationCreateInfo))
{
OV_LOG_ERROR("Fail creation scrath buffer for BLAS.");
this->free();
return false;
}
////////////////////////
// Set scratch buffer //
////////////////////////
std::optional<VkDeviceAddress> deviceAddress = reserved->scratchBuffer.getBufferDeviceAddress();
if (deviceAddress.has_value() == false)
{
OV_LOG_ERROR("Fail to retrieve the scratch buffer device address.");
this->free();
return false;
}
VkDeviceOrHostAddressKHR scratchDeviceOrHostAddress = {};
scratchDeviceOrHostAddress.deviceAddress = deviceAddress.value();
// ScratchData is the device or host address to memory that will be used as scratch memory for the build.
reserved->accelerationStructureBuildGeometryInfo.scratchData = scratchDeviceOrHostAddress;
#pragma endregion
/////////////////
// BLAS buffer //
/////////////////
#pragma region BLASBuffer
// Create BLASBuffer
// Create info buffer
VkBufferCreateInfo vkBLASBufferCreateInfo{};
vkBLASBufferCreateInfo.sType = VK_STRUCTURE_TYPE_BUFFER_CREATE_INFO;
// The second field of the struct is size, which specifies the size of the buffer in bytes
vkBLASBufferCreateInfo.size = accelerationStructureBuildSizesInfo.accelerationStructureSize;
// The third field is usage, which indicates for which purposes the data in the buffer
// is going to be used. It is possible to specify multiple purposes using a bitwise or.
// VK_BUFFER_USAGE_TRANSFER_SRC_BIT specifies that the buffer can be used as the source of a transfer command.
vkBLASBufferCreateInfo.usage = VK_BUFFER_USAGE_ACCELERATION_STRUCTURE_STORAGE_BIT_KHR |
VK_BUFFER_USAGE_SHADER_DEVICE_ADDRESS_BIT;
// buffers can also be owned by a specific queue family or be shared between multiple
// at the same time.
// VK_SHARING_MODE_CONCURRENT specifies that concurrent access to any range or image subresource of the object
// from multiple queue families is supported.
vkBLASBufferCreateInfo.sharingMode = VK_SHARING_MODE_EXCLUSIVE;
// From which queue family the buffer will be accessed.
vkBLASBufferCreateInfo.pQueueFamilyIndices = NULL;
vkBLASBufferCreateInfo.queueFamilyIndexCount = 0;
// Create allocation info
VmaAllocationCreateInfo vmaBLASBufferAllocationCreateInfo = {};
// VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT => Allocation strategy that chooses smallest possible free range for the allocation
// to minimize memory usage and fragmentation, possibly at the expense of allocation time.
vmaBLASBufferAllocationCreateInfo.flags = VMA_ALLOCATION_CREATE_STRATEGY_MIN_MEMORY_BIT;
// Flags that must be set in a Memory Type chosen for an allocation.
vmaBLASBufferAllocationCreateInfo.requiredFlags = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
if (!reserved->accelerationStructureBuffer.create(vkBLASBufferCreateInfo, vmaBLASBufferAllocationCreateInfo))
{
OV_LOG_ERROR("Fail creation scrath buffer for BLAS.");
this->free();
return false;
}
#pragma endregion
//////////////
// Build AS //
//////////////
#pragma region BuildAS
// Structure specifying the parameters of a newly created acceleration structure object
VkAccelerationStructureCreateInfoKHR accelerationStructureCreateInfo = {};
accelerationStructureCreateInfo.sType = VK_STRUCTURE_TYPE_ACCELERATION_STRUCTURE_CREATE_INFO_KHR;
accelerationStructureCreateInfo.pNext = NULL;
accelerationStructureCreateInfo.createFlags = 0;
accelerationStructureCreateInfo.buffer = reserved->accelerationStructureBuffer.getVkBuffer();
accelerationStructureCreateInfo.offset = 0;
accelerationStructureCreateInfo.size = accelerationStructureBuildSizesInfo.accelerationStructureSize;
accelerationStructureCreateInfo.type = type;
accelerationStructureCreateInfo.deviceAddress = 0;
// Create a new acceleration structure object
PFN_vkCreateAccelerationStructureKHR pvkCreateAccelerationStructureKHR =
(PFN_vkCreateAccelerationStructureKHR)vkGetDeviceProcAddr(logicalDevice.get().getVkDevice(), "vkCreateAccelerationStructureKHR");
if (pvkCreateAccelerationStructureKHR(logicalDevice.get().getVkDevice(), &accelerationStructureCreateInfo, NULL,
&reserved->accelerationStructure) != VK_SUCCESS)
{
OV_LOG_ERROR("Fail to create AS, id: %d.", this->Ov::Object::getId());
this->free();
return false;
}
// dstAccelerationStructure is a pointer to the target acceleration structure for the build.
reserved->accelerationStructureBuildGeometryInfo.dstAccelerationStructure = reserved->accelerationStructure;

Answer 1

After a good night's sleep I found the answer to the problem. Basically, the error is due to an incorrect parameter passed to the function pvkGetAccelerationStructureBuildSizesKHR(). As parameter pMaxPrimitiveCounts I was passing the number of geometries but it is necessary to pass the number of primitives for each geometry.
Vulkan spec:
If pBuildInfo->geometryCount is not 0, pMaxPrimitiveCounts must be a valid pointer to an array of pBuildInfo->geometryCount uint32_t values