c++ md5 hash not equal to vb.net code socket - c++

Question

server -> AUTH GateKeeper S :GKSSP\0\0\0\0\0\0\0\0\0r!W\tvM
Client -> AUTH GateKeeper S :GKSSP\0\0\0\0\0\0\0\0\0"g#çójDî3(ƒP¡Á"
VB.NET tcp code:
Do While Stream.CanRead
responseData = Trim(System.Text.Encoding.UTF8.GetString(data2, 0, bytes))
Feed = responseData.Split(Chr(13), Chr(10))
Dim datax As [Byte]()
For Each line As String In Feed
If line <> "" Then
Select Case UCase(Split(line.Trim, " ")(0))
Case "AUTH"
If (Split(line, " ")(2) = "S") And (Split(line, " ")(3) <> ":OK") Then
ChOK = ":GKSSP\0\0\0" & Chr(2) & "\0\0\0" & Chr(3) & "\0\0\0" & Challenge_1(Mid(line, InStr(line, "\0\0\0\0\0\0") + Len("\0\0\0\0\0\0"))) & "Sm(" &
HexToAsc("e4") & "HS" & HexToAsc("c1") & "M" & HexToAsc("847f8293f98d") & "UC"
message = "AUTH GateKeeper S " & ChOK & vbCrLf
VB.NET challenge functions
Public Function Challenge_1(Challenge As String) As String
Dim c1 As String, a1 As String, c2 As String, a2 As String
c1 = "edp{}e|wxrdse}}u666666666666666666666666666666666666666666666666" + Challenge
a1 = HexToAsc(GetIRC7PWD(c1))
Dim Aaah As New String("\"c, 48)
c2 = HexToAsc("0f0e1a11170f161d12180e190f17171f") & Aaah & a1
a2 = HexToAsc(GetIRC7PWD(c2))
Challenge_1 = a2
End Function
Public Function GetIRC7PWD(source As String)
Dim md5Hash As MD5 = MD5.Create()
' Dim source As String = "edp{}e|wxrdse}}u666666666666666666666666666666666666666666666666"
Dim data As Byte() = md5Hash.ComputeHash(Encoding.Default.GetBytes(source))
Dim sBuilder As New StringBuilder()
Dim i As Integer
For i = 0 To data.Length - 1
sBuilder.Append(data(i).ToString("x2"))
Next i
Return sBuilder.ToString()
End Function
The above code authenticates successfully.
My C++ code works fine, until it reads data sent from the AUTH and MD5's it. However, it creates the wrong MD5 hash:
while (true)
{
ZeroMemory(buff, 1024);
int bytesReceived = NetDll_recv(XNCALLER_SYSAPP,Sock, buff, 1024, 0);
if (bytesReceived == SOCKET_ERROR)
{
}
if (bytesReceived > 0) {
std::string Recieved(buff, bytesReceived);
std::stringstream iss(Recieved);
while(iss.good())
{
std::string SingleLine;
getline(iss,SingleLine);
if (!SingleLine.empty()) {
stringstream ss(SingleLine);
string s;
int xUp = 1;
vector <string> tokens;
while (getline(ss, s, ' ')) {
tokens.push_back(s);
xUp++;
}
if (tokens[0] == "AUTH" && tokens[2] == "S") {
MD5 md5;
string ChOK3 = SingleLine.substr(46);
string ChallengeCode = "edp{}e|wxrdse}}u666666666666666666666666666666666666666666666666" + ChOK3;
char* cz = const_cast<char*>(ChallengeCode.c_str());
string g1 = md5.digestString( cz );
string a1 = hexToASCII(g1);
int n = 48;
char cx = '\\';
string c2 = hexToASCII("0f0e1a11170f161d12180e190f17171f") + std::string(n, cx) + a1;
char* c = const_cast<char*>(c2.c_str());
string a2 = md5.digestString( c );
string a3 = hexToASCII(a2);
//string ChOK4 = ChOK + a2 + "Sm(" + hexToASCII("e4") + "HS" + hexToASCII("c1") + "M" + hexToASCII("847f8293f98d") + "UC\r\n";
string ChOK4 = "AUTH GateKeeper S :GKSSP\\0\\0\\0\x02\\0\\0\\0\x03\\0\\0\\0" + a3 + "Sm(" + hexToASCII("e4") + "HS" + hexToASCII("c1") + "M" + hexToASCII("847f8293f98d") + "UC\r\n";
const char *cstr = ChOK4.c_str();
NetDll_send(XNCALLER_SYSAPP, Sock,ChOK4.c_str(),ChOK4.length(), 0);
}
It looks like my code is the same.
Is it because the characters are UTF-16? How can I read the buffer in UTF-16 so that I can properly MD5 the string?

Answer 1

The two codes are NOT doing the same thing, which is why they are producing different results. Differences are highlighted below:
While reading from the socket:
the VB code is:
reading arbitrary bytes (without regard to message boundaries)
converting the bytes from UTF-8 to a UTF-16 string (without regard to codeunit boundaries)
trimming the string
splitting the string on line breaks
parsing each line
the C++ code is:
reading arbitrary bytes (without regard to message boundaries)
converting the bytes as-is to a std::string (without regard to encoding)
splitting the string on line breaks (without trimming it first)
parsing each line
Up to this point, given the data shown, the results are logically the same for both codes (though technically different layouts in memory), even though they are being processed differently. But even up to this point, both codes are NOT handling the socket data adequately, and there IS potential for data corruption and data loss here.
Now, assuming no corruption/loss has actually occurred, then for the actual AUTH command:
the VB code is:
extracting the challenge from position 37, after the first "\0\0\0\0\0\0" substring (thus, the extracted challenge is "\0\0\0r!W\tvM")
AUTH GateKeeper S :GKSSP\0\0\0\0\0\0\0\0\0r!W\tvM
^ ^ ^
1 25 37
converting the challenge from UTF-16 to ANSI bytes
calculating an MD5 for those bytes
calculating a 2nd MD5 based on the 1st MD5
creating a response for the 2nd MD5
the C++ code is:
extracting the challenge from fixed index 46 (position 47) (thus, the extracted challenge is "tvM")
AUTH GateKeeper S :GKSSP\0\0\0\0\0\0\0\0\0r!W\tvM
^ ^ ^ ^
0 24 36 46
getting the challenge's raw UTF-8 bytes
calculating an MD5 for those bytes
calculating a 2nd MD5 based on the 1st MD5
creating a response for the 2nd MD5
So, the biggest issues I see that you need to fix are the following:
in both languages, fix the socket reading logic to avoid data loss. Right now, you are extracting all bytes from the socket and throwing away any incomplete lines that are waiting for more bytes to arrive. You need to save all raw bytes from the socket into an intermediate buffer first, and then extract only complete lines from that buffer, leaving behind incomplete lines in the buffer so they can be finished on subsequent reads.
on the VB side, for ASCII-only data (as you have shown), the conversion from UTF-8 to UTF-16 to ANSI is loss-less, but for non-ASCII data this will be lossy. Consider converting the UTF-16 data back to UTF-8 instead of ANSI, to match the C++ code.
on the C++ side, fix the code to extract the server's challenge from the correct string position to match the VB code.