Program was killed after function returning - c++

Question

My program was killed when the function "dbmSetIndex" was returned.
You can see the killed position below. (=> part)
Could you explain what happened?
(x86 intel)
0x0000000000420723 <+83>: mov rdi,r14
0x0000000000420726 <+86>: call 0x405260 <dbmSetIndex#plt>
0x000000000042072b <+91>: test eax,eax
0x000000000042072d <+93>: mov ebp,eax
=> 0x000000000042072f <+95>: mov DWORD PTR [r12],eax
0x0000000000420733 <+99>: jne 0x4207d0 <FnDBBase::SelectSecurity(s_oms_security*, char*)+256>
0x0000000000420739 <+105>: lea rsi,[rip+0x4197d] # 0x4620bd
Here is the dbmSetIndex code.
I can't find what part of this code caused this problem.
int dbmSetIndex ( dbmHandle* aHandle, const char* aTable, const char* aIndexName )
{
dbmInternalHandle* pHandle = NULL;
_TRY
{
pHandle = (dbmInternalHandle*)aHandle->mHandle;
// Water Mark가 다르면 걍 리턴해라.
if ( unlikely ( aHandle->mMark != DBM_HANDLE_MARK ) )
{
DBM_ERR ( "invalide magic number (%ld)", aHandle->mMark );
_THROW( ERR_DBM_INVALID_HANDLE );
}
if( pHandle->mRemote != NULL )
{
if( pHandle->mRemote->mSockFd > 0 )
{
_CALL( dbmRemoteSetIndex( aHandle, aTable, aIndexName ) );
_RETURN;
}
}
/****************************************
* DataObject 맵핑.
****************************************/
memset_s( &pHandle->mData, 0x00, sizeof( pHandle->mData ) );
memcpy_s ( pHandle->mData.mTableName, aTable, strlen_s ( aTable ) + 1 );
pHandle->mData.mTransType = DBM_SET_INDEX;
pHandle->mData.mUserData = (char*)aIndexName;
/****************************************
* mAct 호출.
****************************************/
_CALL( pHandle->mTrans->mAct ( pHandle ) );
}
_CATCH
{
_CATCH_ERR;
}
_FINALLY
_END
}

Answer 1

You provided a little bit of dissassembly which shows that the call to the function you provided returned already. The crash did not occur within the function, but afterwards:
Call of your function:
0x0000000000420726 <+86>: call 0x405260 <dbmSetIndex#plt>
Here you returned already:
0x000000000042072b <+91>: test eax,eax
The critical line is a memory access to the address stored in register r12 (write):
0x000000000042072f <+95>: mov DWORD PTR [r12],eax
Let your debugger show the registers and have a look at the content of r12. It is very likely that it is 0x0000000000000000 or a small value, thus a null pointer (or null reference), but it could contain, too, an invalid address (uninitialised pointer!).
There is little more left to tell from the information you provide, though. You will have to look at the code location where the function is called -- it must be within function FnDBBase::SelectSecurity(s_oms_security*, char*) because you jump (jne) to an offset within this function ([...] + 256). There should be an if involved (test + jne instructions) and probably some pointer assignment. Possibly something like this:
SomeClass* s = [...];
s->someMember = dbmSetIndex([...]); // (*)
if(*s->someMember)
(*): Failure location, error happened during assignment after the function completed already. Don't count 100% on finding such an assignment, it could be, too, a call to an inlined setter function.
And here we see, too, why r12 is not necessarily 0: it would contain the precalculated offset of someMember within SomeClass, i. e. &(s->someMember) which could well be, if s is 0, e. g. 16, 28, ... Precisely, the value contained then is equal to offsetof(SomeClass, somemember).