11 replies to this topic

[OfficerFlake]

Hi there all. I've been using GML for a while now and I love the ease and power behind it. Lately, I'm progressing onto TCP/IP based coding, trying to make a simple proxy server that can detect packet types for a flight simulator program i use. The aim is to extend the functions that the standard server doesn't have.

I've got everything working the way it should except for one problem:

Whenever i try to use "ReadChars" it cuts short for the null characters. This has been trialled and tested and confirmed.

Because of the complexity of the issue, a GML source won't be extremely helpful, but anyway...

if (peekmessage(client,0,clientbuffer) > 4) {
        complete = readchars(buffsize(clientbuffer),clientbuffer);
        //show_message("CLIENT SENDS")
        clearbuffer(clientbuffer);
        receivemessage(client,4,clientbuffer);
        lengthpacket = real(readuint(clientbuffer));
        //show_message("LENGTH= :" + string(lengthpacket) + ":")
        receivemessage(client,4,clientbuffer);
        type = real(readuint(clientbuffer));
        //show_message("TYPE = :" + string(type) + ":")
        receivemessage(client,lengthpacket-4,clientbuffer);
        data = readchars(lengthpacket-4,clientbuffer);
        //show_message("DATA = :" + string(string_length(data)) + ":")
        clearbuffer(clientbuffer);
        //writeuint(real(lengthpacket),clientbuffer);
        //show_message("size length: " + string(real(buffsize(clientbuffer))))
        //writeuint(real(type),clientbuffer);
        //show_message("size type: " + string(real(buffsize(clientbuffer))-4))
        //writechars(string(data),clientbuffer);
        //show_message("size data: " + string(real(buffsize(clientbuffer))-8))
        //show_message("size final: " + string(real(buffsize(clientbuffer))))
        //input = readchars(24,clientbuffer)
        //clearbuffer(clientbuffer);
        //show_message("INPUT = %" + input + "%")
        show_message(str2hex(complete)) //A simple debug script to show the raw packet data.
        //show_message("length: " + string(length) + "#type: " + string(type) + "#data: " + string(data))
        switch (real(type)) {
            case (1): show_message("LoginPacketData: " + string(data))break; //login packet
            }
        sendmessage(server, "", 0, clientbuffer) //socket, ip, port, buffer
        clearbuffer(clientbuffer);
        }

EXPECTED PACKET:

[UnsignedInt-SizeOfPacket(MinusThisInteger)][UnsignedInt-ThePacketTypeID][Characters-ThePacketData]

So that's:

4bytes:4bytes:restofthedata

example:

18:00:00:00:01:00:00:00:64:6f:69:6e:67:5f:74:65:73:74:73:00:00:00:00:00:7f:db:32:01

Packet I am working with:
"[int-24]" + "[int-01]"+ "[YSRAAF][ACM]Fl" + "[NULL]" + "[int-20110207]"
This particular code is a login packet, which contains 1 int for size, 1 int for type, 16 bytes (last byte is always "NULL") username, 1 int for version.

What my code ends up reading:
Size: 24 (Correct)
Type: 1 (Correct)
Length Data: 15 (Incorrect, is terminated by the "NULL")
Data: "[YSRAAF][ACM]Fl" (Should also have "NULL" plus int-20110207)

LONG STORY SHORT:
How can I get readchars to work around the "null" character termination so that the variable "data" becomes length 20 instead of 15?
Thanks for any help!

Edited by OfficerFlake, 03 February 2012 - 04:13 AM.

[OfficerFlake]

39.dll has a break function is null is encountered:

for(i = readpos; i < count; i++)
	{
		if(data[i] == '\0')
			break;
	}

So i need to work out how to recompile 39.dll without that, or i should be able to make do with a read character by character in a repeat loop.

[Gamer3D]

No. Strings passed to DLLs are NULL-terminated. In other words, the string MUST stop when a NULL character is reached or it will be accessing invalid memory.

You should work around this in GM by using escape characters. For example:

Before sending to DLL:
str = string_replace_all(str, '\', "\\");
str = string_replace_all(str, chr(0), "\0");

After receiving from DLL:
str = string_replace_all(str, "\0", chr(0));
str = string_replace_all(str, "\\", '\');

The general idea: Reserve a character, and use it to indicate that a sequence of characters indicates some unusable character.

[OfficerFlake]

No. Strings passed to DLLs are NULL-terminated. In other words, the string MUST stop when a NULL character is reached or it will be accessing invalid memory.

You should work around this in GM by using escape characters. For example:

Before sending to DLL:
str = string_replace_all(str, '\', "\\");
str = string_replace_all(str, chr(0), "\0");

After receiving from DLL:
str = string_replace_all(str, "\0", chr(0));
str = string_replace_all(str, "\\", '\');

The general idea: Reserve a character, and use it to indicate that a sequence of characters indicates some unusable character.

That's a problem. The program that sends the message sends the null character, and I can't avoid that. so 39DLL has to be able to work around that.

You're saying that the sending program needs to change the null to something else, and the reciving program needs to change it back, right? But i can't change what is sent to start with.

I'm coding for a program that is already written and compiled, closed source.

I'll try my idea, and if that doesn't work, then I will need to write my own DLL most likely... if that is the case, then I might just move to a lower level language instead sadly.

[Gamer3D]

That's a problem. The program that sends the message sends the null character, and I can't avoid that. so 39DLL has to be able to work around that.

You're saying that the sending program needs to change the null to something else, and the reciving program needs to change it back, right? But i can't change what is sent to start with.

I'm coding for a program that is already written and compiled, closed source.

I'll try my idea, and if that doesn't work, then I will need to write my own DLL most likely... if that is the case, then I might just move to a lower level language instead sadly.

The problem (probably) isn't the DLL. The problem is how strings are passed. If you want to pass a NULL character, you CAN'T use a NULL-terminated string.

I haven't look at 39DLL's source, but if 39DLL can internally accept a NULL character, then you can change the source to use escape sequences (Things like "\0") to communicate to GM. It's the same "Replace character A with sequence B" thing. This would be done immediately before sending a string to GM and immediately after receiving a string from GM.

Be warned though. Most programs just use NULL-terminated strings. It's easy, it's well-known, and it usually isn't a problem.

[icuurd12b42]

Not to negate anything said...

you can have string that can have multiple nuls in them. it's not really absolutely needed to have null termination either. it depends on the functions that were used in the dll and how the string is passed back and forth.

For example, In FMOD, you can fill up a 256 size string buffer with spectrum data and character 1 can have a nul representing a 0 value while character 2 is 200 and char 256 is 5 as value....

This re-asserts post#2.

Does the string hold multiple nuls, the only what to know is to loop through the buffer manually, because draw text or message box or debug message wont help you because they do stop at the first nul they hit even if there are other printable character beyond the first nul.

[OfficerFlake]

There can be multiple nulls. If the username, for example, is less then 15 characters, the following characters are null too.

ALSO: My debug shows both the variable (which will never pass the null anyway)

and it shows the length of the variable (15)

Edited by OfficerFlake, 05 February 2012 - 02:16 PM.

[Gamer3D]

Not to negate anything said...

you can have string that can have multiple nuls in them. it's not really absolutely needed to have null termination either. it depends on the functions that were used in the dll and how the string is passed back and forth.

For example, In FMOD, you can fill up a 256 size string buffer with spectrum data and character 1 can have a nul representing a 0 value while character 2 is 200 and char 256 is 5 as value....

You have a point, sir. Passing the string's byte length will work, but GM doesn't do this automatically. (To my knowledge anyway) Either way, you're going to be re-writing a small portion of the DLL.

Whether you have to do GM-side coding depends on how GM passes strings. If you pass the string byte length, then you can theoretically just read that many bytes. If GM passes the string's length somehow, then you can just use that.

Anyway, look up how the FMOD DLL does it. It should help you out in that regard.

[2DLuis]

39.dll has a break function is null is encountered:

for(i = readpos; i < count; i++)
	{
		if(data[i] == '\0')
			break;
	}

So i need to work out how to recompile 39.dll without that, or i should be able to make do with a read character by character in a repeat loop.

Recompile the dll then. You'll need to rearrange some of the cpp file names for it to compile though, and if you recompile it, you should also address the maximum buffer size of 2000 (you'll get memory leaks if you exceed that buffer size in 39dll). WSA can handle up to 8192 bytes if I'm not mistaken.

[icuurd12b42]

Whether you have to do GM-side coding depends on how GM passes strings. If you pass the string byte length, then you can theoretically just read that many bytes. If GM passes the string's length somehow, then you can just use that.

I pass a pre-allocated string to the dll

s = string_repeat(0,256);
external_call(global.func, s,256);
return s;

in the dll I do a mem copy to the passed string pointer the size passed

Also, GM's string have an integer prefix before the string data telling you the size of the string. so you can cast that memory to a int and dont need to pass the size;
double Func(LPSTR gmbuffer)
{
char t = "Hello";
UINT len = *((UINT*)(gmbuffer-sizeof(UINT)));
memcpy((void*)gmbuffer,(void*)t,min(strlen(t),len));
return 0.0f;
}


now as for returning a string instead of copying bytes to a passed argument, I think the nul thing has effect in that case. When GM converts the char* to a GM string, it needs a way to know the size (to set that prefix I mentioned) and allocate and copy the char* to it's own memory. It probably does a strlen on the string to figure things out.
char* Func(LPSTR gmbuffer)
{
char t[] = "Hello"; //there is a nul added by the compiler "Hello\0"
return t;
}

a gm string is something like

struct GMs
{
UINT size;
char *string; //this is passed back a forth to the dll
};

[OfficerFlake]

Peekmessage is getting the right size. (28)

I think receivemessage is the issue. I'm a complete noob at C++ though, perhaps someone could explain how to get rid of the whole null character issue. I'm reading the source, but only understanding 1/5th of it.

example, if my string was "hello\0", and the compiler adds a null onto that ("hello\0\0"), i should get "hello\0" not "hello"?

I'm completely lost.

Edited by OfficerFlake, 16 February 2012 - 04:46 AM.

[icuurd12b42]

I looked at the code and yes, a few \0 in there when you ask for a string

use readbyte in a loop according to peak message
buff = "";
size = peakmessage(socket);
repeat(size)
{
buf += chr(readbyte(socket))
}
debug(string_length(buf));

or try to simply avoid having data that has chr(0) in there... or format the buffer so you can make sense on the other end to avoid "blind" reading...

The API seems to support all kinds of data types but if you read the buffer in a "blind" read, who knows was data your gonna try to stuff in a string...


if I do
writeint(0)
writeint(20)

I cant just readchars() on the other end, readchars assume you are reading (and expecting to read) a string with \0 at the end of it.... first int in the buffer is 0, so that is 8 bytes with 0 in it or "\0\0\0\0\0\0\0\0". readchars will give you an empty string.

It's akin to call file_text_real_string on a binary file

if you wrote 5 ints, 10 double and 3 string, on the other end, read 5 ints, 10 doubles and 3 strings

It's a little difficult from what I understand from other asynchronous communications... You are dealing with a buffer that is of uncertain length each read. you are not sure where you are in the buffer, if the buffer will continue to fill in the next read, or if the data you are reading is complete (like the buffer may have been full so the rest will transmit in the next iteration, you only got "Hell" where "o World" is still pending on the other side). The data is split and cannot be read fully in one read.

I'm not sure if 39dll handles data splitting for you.


When dealing with a file you can write the whole thing and read the whole thing where you can format your writing and your reading accordingly. Making every read match with every write is easy in a file

writing:
i = 0
writeint(numints)
repeat(numints)
{
writeint(array[i])
i+=1
}
writestring(AString)


reading:
i = 0
numints = readint()
repeat(numints)
{
array[i] = readint()
i+=1
}
AString = readstring()


I think the same can be done with 39dll but perhaps you need to be more careful. Like making sure you never write more than the transmit buffer size