85 replies to this topic

[Medo42]

Programmers sometimes need to move data between different, unrelated extensions. An example would be an extension for file reading/writing, and one for networking: Both extensions would likely have some sort of buffer data structure, but if you want to read a file and send it over a connection, how do you get the data from the file extension into the networking extension? You will probably need to copy it over GML somehow. 39dll solves this particular problem by having both file access and networking functions, but in my opinion this is not a good solution, unless your ideal goal is to create the "everything extension". I prefer modularity though, which is why I don't plan to include file access functions in my own Faucet Networking extension. In my opinion, the better solution is to solve the problem of data exchange between extensions.

One option would be to create a "Shared Buffers" dll that is linked by all extensions which want to exchange data in this way. Windows will only load the dll into memory once for the game process, so it can be used as a central point to manage and exchange shared data buffers. If both the file and the networking extension in the example above used this dll, the user could simple read a file into a shared buffer and pass that to the networking extension for sending. A more complex implementation could offer interfaces to exchange data from "internal" buffers of extensions which use different implementations, or other types of data exchange, like data streams.

So far, this is only an idea that still needs to be fleshed out, but I want to put it out there to find out what you think. One problem with the shared dll approach is that this dll would have to be in the game directory when running, and some people probably don't like that. Additionally, the dll would have to be adopted by several extensions to be really useful. A major problem is to ensure stability of the Buffer dll's interface, so that you don't get different extensions using different, incompatible versions of the Buffers dll.

But before any work goes into creating this, please tell me your opinion. Do you think this would be useful to you as an extension/dll user? Would you, as an extension/dll creator, support this interface? Discuss!

[Medusar]

I can't really think of many occasions when you really need to transfer data between separate DLLs... Your networking example is one but as you mentioned that's already been accounted for. Apart from that, most DLLs for GM allow complex calculations to be done in compiled code or they provide an interface to an external API. You'd hardly ever even want to share data between unrelated DLLs as they tend to use their own structures. So lots of data would not be compatible.
The developer could decide to expose an API for his DLL but I'd prefer any contributors to PM me so that the extra functionality would end up in the same code base. Either way this would not involve a buffer DLL.

[Maarten Baert]

I had exactly the same problem while writing my own networking dll (Http Dll 2), which is why I added buffers. All my other DLL's simply convert the data to a very long hexadecimal string, which can be safely passed through GML. This is clearly not very efficient.

I would prefer letting GM load the Shared Buffers dll, as a normal extension. Then the GEX could have a function 'shared_buffer_handle()' that returns a pointer to a controller class inside the shared buffers DLL. Other DLLs could then store this pointer during initialization, and use the pointer to access the buffers later. Since the DLL is only loaded once (by GM), there can only be one version of the DLL at any time. All other DLLs can use any version of the shared buffers DLL as long as the interface of the DLL never changes. This is possible with function pointers. Virtual functions should work too, but since different compilers use different vtable formats this won't work if one DLL was compiled in VC++ and another was compiled in GCC.

class Buffer; // opaque pointer
class SharedBufferInterface {
    
    private:
    Buffer* (*createbuffer)();
    void (*destroybuffer)(Buffer*);
    // ...
    
    public:
    inline Buffer* CreateBuffer() { return (*createbuffer)(); }
    inline void DestroyBuffer(Buffer* buffer) { (*destroybuffer)(buffer); }
    // ...
    
};

I would probably use this DLL in my own DLLs if we would create one. I have code for buffers that are very similar to 39dll's buffers, I will create a simple proof of concept to test the idea.

@Medusar: It would be really useful for serialization: converting a data structure to a string or loading a data structure from a string (like ds_map_write/ds_map_read). This is useful to save/load the game. Many DLLs use data structures, it would be great if we could save them all to a single file without having to use temp files. This would also make it a lot easier to send the contents of a data structure to another client in a multiplayer game.

[Medo42]

You make an interesting point about the vtable - the way it looks, you need a C interface instead of a C++ one in order to make GCC and MSVC work together. And using two seperate versions for the two compilers would defeat the entire point of course.

Since the DLL is only loaded once (by GM), there can only be one version of the DLL at any time.

I didn't test this, but from my understanding Windows only loads a dll once per process even if it is requested multiple times.

[Tha_14]

OOOHHHHH,Big Replies!!!
I need a lot of time to read that.

[Medo42]

Another thought, it's possible to implement this without an extra dll and offer a statically linked library instead. The idea is to use handles which contain a pointer to the buffer descriptor instead of a generated integer as is usually done. As long as everything runs in 32-bit mode you should be able to store the pointer's integer value in a GM real without a problem, but it feels slightly less naughty to encode it to a GM string instead.

[icuurd12b42]

You can pass along the shared pointer via a function call in GM

1) define external call to sharedmem.dll called CreateMem
2) define external call in dll1 called Dll1SetSharedMem
2) define external call in dll2 called Dll2SetSharedMem


var mem; mem = CreateMem();
Dll1SetSharedMem(mem);
Dll2SetSharedMem(mem);


in sharemem CreateMem
return (double)(DWORD) GlobalAllocPtr(GMEM_FIXED!GMEM_ZEROINIT, 1024);

in dll 1 and dll2
void* sharedmem = NULL;

void Dll1SetSharedMem(double mem) //same for Dll2SetSharedMem in dll2
{
sharedmem = (void*)(DWORD) mem;
}

Edited by icuurd12b42, 18 April 2011 - 06:07 PM.

[paul23]

uhm wait, don't cast pointers like that. - There's no way this is guaranteed to work: with 64-bit computers pointers can be larger than the mantissa of doubles. - And I'm unsure how this works out (I think this depends on the compiler wether they're mem-copied or "copied-by-value"), and as someone raised in this post there are a lot of other "potential" problems when using pointers like this.


Now I am not in favour of the whole idea, instead I would try to redesign and make a "tree-structure" of dependency: GM (root) handles 1 or 2 dlls, and then those dlls handle sub dlls (and the communication between sub-dlls). Dlls don't know about their "roots" or other dlls except for their "children".
If you let memory be shared between dlls and generally do stuff like that, in bigger projects you'll always run into the problem: "who owns what", which can slow development down a lot!

[Medo42]

uhm wait, don't cast pointers like that. - There's no way this is guaranteed to work: with 64-bit computers pointers can be larger than the mantissa of doubles.

That's why I said "As long as everything runs in 32-bit mode", which is currently the case for Game Maker related things.

- And I'm unsure how this works out (I think this depends on the compiler wether they're mem-copied or "copied-by-value"), and as someone raised in this post there are a lot of other "potential" problems when using pointers like this.

Again, for 32-bit pointers this shouldn't be a problem if done this way, assuming Game Maker's real type always corresponds to double. It doesn't matter if they are stored in FPU registers now and then, because those have an even larger mantissa than the normal double representation. Copying the pointer to a string instead (in some encoding that avoids the 0-byte of course) is a bit more straightforward in terms of correctness though.

Now I am not in favour of the whole idea, instead I would try to redesign and make a "tree-structure" of dependency: GM (root) handles 1 or 2 dlls, and then those dlls handle sub dlls (and the communication between sub-dlls). Dlls don't know about their "roots" or other dlls except for their "children".
If you let memory be shared between dlls and generally do stuff like that, in bigger projects you'll always run into the problem: "who owns what", which can slow development down a lot!

The "Who owns what" question has to be solved once - when designing the dll. After that everyone just has to follow the rules.
I don't see where you are going with your idea. Arranging dlls in a tree structure implies that they are dependend on each other, but what I would like to get is a standard way to exchange information between independend extensions. For example, imagine an SHA256 dll that creates a digest over a buffer. It would be good if that function could be developed to just work on any buffer, without all extension programmers having to add this new dll as a "child" dll.

[paul23]

uhm wait, don't cast pointers like that. - There's no way this is guaranteed to work: with 64-bit computers pointers can be larger than the mantissa of doubles.

That's why I said "As long as everything runs in 32-bit mode", which is currently the case for Game Maker related things.

- And I'm unsure how this works out (I think this depends on the compiler wether they're mem-copied or "copied-by-value"), and as someone raised in this post there are a lot of other "potential" problems when using pointers like this.

Again, for 32-bit pointers this shouldn't be a problem if done this way, assuming Game Maker's real type always corresponds to double. It doesn't matter if they are stored in FPU registers now and then, because those have an even larger mantissa than the normal double representation. Copying the pointer to a string instead (in some encoding that avoids the 0-byte of course) is a bit more straightforward in terms of correctness though.

Now I am not in favour of the whole idea, instead I would try to redesign and make a "tree-structure" of dependency: GM (root) handles 1 or 2 dlls, and then those dlls handle sub dlls (and the communication between sub-dlls). Dlls don't know about their "roots" or other dlls except for their "children".
If you let memory be shared between dlls and generally do stuff like that, in bigger projects you'll always run into the problem: "who owns what", which can slow development down a lot!

The "Who owns what" question has to be solved once - when designing the dll. After that everyone just has to follow the rules.
I don't see where you are going with your idea. Arranging dlls in a tree structure implies that they are dependend on each other, but what I would like to get is a standard way to exchange information between independend extensions. For example, imagine an SHA256 dll that creates a digest over a buffer. It would be good if that function could be developed to just work on any buffer, without all extension programmers having to add this new dll as a "child" dll.

I'm saying that there is no need for a "standard" way: GM acts as root and should handle these translations.

Also consider the loading:
if A is a dll and B needs to get memory allocated by A: how can B be sure that A is not freed, and the memory cleared?

[icuurd12b42]

@paul... about the casts... Problem occurs when you don't do a progressive cast down Like char t= (char)MyDouble; You have to progressively cast to the next smaller type char t = (char)(int)(DWORD)MyDouble;

As for who owns what, in my example, GM owns the memory but SharedMem.dll manages it. You have to decide on what is located in the address and who can to what to it.

Best scenario is to have a garbage system... Simplest is to simply global allocate a large chunk that will exist from start to end of the application and this chunk can be used for data passing. AKA the mem_fixed flag used here.

Remember data sharing is the goal of this concept. Each dll could copy the passed data in it's own space. You just read and write to the buffer as a means to pass along data.


Here I have an in head elaborate design for shared resources for a GMTOOLS (sharing resources and functions)

http://cid-fba0b7e57.../Public/GMTools

see tools overview

Edited by icuurd12b42, 19 April 2011 - 02:04 AM.

[Maarten Baert]

I just finished my proof of concept:
http://gm.maartenbaert.be/shared_buffers.zip
I've tested it in GCC and VC++, both versions are compatible.

The main DLL (shared_buffers.dll) exports buffer functions to GM, so games can access the buffers too:

shared_buffers_get_handle()buffer_create()buffer_destroy(id)buffer_exists(id)buffer_to_string(id)buffer_get_pos(id)buffer_get_length(id)buffer_at_end(id)buffer_get_error(id)buffer_clear_error(id)buffer_clear(id)buffer_set_pos(id,pos)buffer_read_from_file(id,filename)buffer_write_to_file(id,filename)buffer_read_int8(id)buffer_read_uint8(id)buffer_read_int16(id)buffer_read_uint16(id)buffer_read_int32(id)buffer_read_uint32(id)buffer_read_int64(id)buffer_read_uint64(id)buffer_read_float32(id)buffer_read_float64(id)buffer_write_int8(id,value)buffer_write_uint8(id,value)buffer_write_int16(id,value)buffer_write_uint16(id,value)buffer_write_int32(id,value)buffer_write_uint32(id,value)buffer_write_int64(id,value)buffer_write_uint64(id,value)buffer_write_float32(id,value)buffer_write_float64(id,value)buffer_read_string(id)buffer_write_string(id,string)buffer_read_data(id,len)buffer_write_data(id,string)buffer_read_hex(id,len)buffer_write_hex(id,string)buffer_write_buffer(id,id2)buffer_write_buffer_part(id,id2,pos,len)

Please tell me if I forgot something important. The functions that can be called by the other DLLs are almost the same, but they use pointers instead of ids:

Buffer* Create();void Destroy(Buffer* buffer);unsigned int GetID(Buffer* buffer);Buffer* Find(unsigned int id);const char* ToString(Buffer* buffer);char* GetData(Buffer* buffer);unsigned int GetPos(Buffer* buffer);unsigned int GetLength(Buffer* buffer);bool IsAtEnd(Buffer* buffer);bool GetError(Buffer* buffer);void ClearError(Buffer* buffer);void Clear(Buffer* buffer);void SetPos(Buffer* buffer, unsigned int newpos);void SetLength(Buffer* buffer, unsigned int newlength);bool ReadFromFile(Buffer* buffer, const char* filename);bool WriteToFile(Buffer* buffer, const char* filename);int8_t ReadInt8(Buffer* buffer);uint8_t ReadUint8(Buffer* buffer);int16_t ReadInt16(Buffer* buffer);uint16_t ReadUint16(Buffer* buffer);int32_t ReadInt32(Buffer* buffer);uint32_t ReadUint32(Buffer* buffer);int64_t ReadInt64(Buffer* buffer);uint64_t ReadUint64(Buffer* buffer);float ReadFloat32(Buffer* buffer);double ReadFloat64(Buffer* buffer);void WriteInt8(Buffer* buffer, int8_t value);void WriteUint8(Buffer* buffer, uint8_t value);void WriteInt16(Buffer* buffer, int16_t value);void WriteUint16(Buffer* buffer, uint16_t value);void WriteInt32(Buffer* buffer, int32_t value);void WriteUint32(Buffer* buffer, uint32_t value);void WriteInt64(Buffer* buffer, int64_t value);void WriteUint64(Buffer* buffer, uint64_t value);void WriteFloat32(Buffer* buffer, float value);void WriteFloat64(Buffer* buffer, double value);const char* ReadString(Buffer* buffer);void WriteString(Buffer* buffer, const char* string);void ReadData(Buffer* buffer, char* ptr, unsigned int bytes);void WriteData(Buffer* buffer, const char* ptr, unsigned int bytes);void ReadHex(Buffer* buffer, char* ptr, unsigned int bytes);void WriteHex(Buffer* buffer, const char* ptr, unsigned int bytes);void WriteBuffer(Buffer* buffer, Buffer* source);void WriteBufferPart(Buffer* buffer, Buffer* source, unsigned int pos, unsigned int bytes);

The functions GetID and Find can be used to convert pointers to ids and vice versa.

The function shared_buffers_get_handle() is used to pass the pointer to the other DLL:

shared_buffers_init();
test_init();

// initialize shared buffers interface for test.dll
test_init_shared_buffers(shared_buffers_get_handle());

Now the other DLL can use the pointer to manipulate buffers:

#include "gm.h"
#include "SharedBuffers.h"

SharedBufferInterface *sbi = NULL;

gmexport double test_init_shared_buffers(double handle) {
	sbi = (SharedBufferInterface*)(gm_double_to_uint(handle));
	return 1;
}

gmexport double test_writetobuffer(double id) {
	if(sbi == NULL) return 0;
	Buffer *b = sbi->Find(gm_double_to_uint(id));
	if(b == NULL) return 0;
	sbi->WriteInt32(b, 12345);
	sbi->WriteString(b, "Hello world");
	sbi->WriteFloat32(b, 42.42f);
	return 1;
}

Memory management is done by shared_buffers.dll, other DLLs can't allocate or free memory. They should call the Resize function.

Using it in other DLLs is very simple, just include SharedBuffers.h. That's it.

This system has the added benefit of modularity: If some user doesn't need the buffer functionality, that user can choose not to add shared_buffers.dll to his/her game. The part of the DLL that doesn't need buffers will still work.

We could also use shared memory to pass the pointer of the struct, or even to store the entire struct, but I think it's easier to simply pass the pointer through GM. If we use shared memory we would still need functions to tell DLLs to load the shared memory at the right time, because it's possible some DLLs are loaded before shared_buffers.dll is loaded.

What do you think? Would you use something like this in your DLLs? What should be changed to make it better?

Edited by Maarten Baert, 19 April 2011 - 10:53 AM.

[Medo42]

@paul... about the casts... Problem occurs when you don't do a progressive cast down Like char t= (char)MyDouble; You have to progressively cast to the next smaller type char t = (char)(int)(DWORD)MyDouble;

That cast has undefined behaviour if the double value is out of range for the DWORD type. And with "undefined behaviour" I don't mean that you could get a wrong result, but that it can cause your program to crash. And that's not theoretic, it happened to me when developing Faucet Networking. Always make sure that the source value fits in the target type when you cast floating point values to anything else - then you don't need this strange cascade of casts either.

To avoid this problem you can use boost::numeric_cast, which throws an exception if the source is out of range, or you can use my own clipped_cast, which results in the highest/lowest legal value for the target if the source is out of range. In your example, you could use it like "char t = clipped_cast<char>(MyDouble)".

I found this very informative in this matter: http://www.boost.org...efinitions.html

Here I have an in head elaborate design for shared resources for a GMTOOLS (sharing resources and functions)

That looks very comprehensive, but it's far beyond the scope of what I want. In my opinion, finding a good way to share buffers is difficult enough for a single project

Here are two different ways to define the ownership/responsibility and rules. There are more different possibilities of course, but something to think about:

Model 1:
Buffers are owned by the extensions that created them, and can be shared for read-only access by other extensions. The owning extension defines for how long a buffer is valid. If extensions are allowed to keep references to the buffer, they must assume that the buffer can change and become invalid at any time between function calls (it is possible to provide a safe way for checking this). Otherwise the extension has to create a copy to work on.

This is probably the simplest model, and it allows efficient access to buffers of other extensions without (usually) requiring to copy. In order to allow sharing buffers with more sophisticated data structures without copying, a buffer can be modeled as a list of memory blocks instead of a single one. There could be a convenience function for people who don't want to deal with this, which will always give you a buffer as a single block of memory by making a copy if it consists of multiple regions.

This model can be implemented without an extra dll.

Model 2:
Buffers are owned by a Shared Buffers dll, and can be read and written by all extensions which have a reference. That means buffers would have to follow a generic implementation provided in the dll, or at least a generic interface which can be implemented by the extensions if they really need something more sophisticated. Also, all extensions would have to assume that buffers can be changed between function calls, unless e.g. a read-only wrapper for buffers in provided. On the upside, you could provide a single set of GM functions for reading/writing bytes, floats etc., which is more difficult if buffers are managed by the extensions.

This is more complex, but allows to have a unified Buffer implementation for most purposes.

Actually, thinking about this some more, this can also be achieved without an extra dll.

Edit: I was still writing this when Maarten posted

Edited by Medo42, 19 April 2011 - 10:34 AM.

[Medo42]

Great stuff Maarten. Here are a few thoughts:
The finished thing should have a very permissive license so that it can be used with any extension. Something like MIT or (even simpler but equivalent) ISC would be preferable for me.

Now for some technical comments

shared_buffers_get_handle()
I don't really like relying on the size of int as you do here... I know we do it anyway in the end, but I'd still change that to uintptr_t
And I'd prefer an automatic way to share this information, so that the user doesn't have to care about it. I guess an extension could do it as part of an automatic initialization script though and hide it this way.

buffer_get_error(id)
buffer_clear_error(id)

I don't think it's very useful to handle buffer overruns like this. If someone writes code that reads past the end of a buffer, it is not likely that he will explicitly check for errors either. The only real improvement would be with exceptions, but GM doesn't support anything like that of course. As it is, I'd prefer the behaviour to be undefined but safe. In my own Buffers implementation I just read until the end and then stop there, the rest of the returned data is undefined (unless you try to read a string - in that case a reasonable behaviour is still possible and I just return the first part of the string)

buffer_read_from_file(id,filename)
buffer_write_to_file(id,filename)

IMO, that should go into its own dll/extension - the very purpose of this project is to make things more modular

More criticism later
Can you upload this to github or some other collaboration platform?

Edited by Medo42, 19 April 2011 - 12:42 PM.

[Maarten Baert]

Great stuff Maarten. Here are a few thoughts:
The finished thing should have a very permissive license so that it can be used with any extension. Something like MIT or (even simpler but equivalent) ISC would be preferable for me.

Okay, I'm using ISC now.

shared_buffers_get_handle()
I don't really like relying on the size of int as you do here... I know we do it anyway in the end, but I'd still change that to uintptr_t

Changed unsigned int to uintptr_t .

And I'd prefer an automatic way to share this information, so that the user doesn't have to care about it. I guess an extension could do it as part of an automatic initialization script though and hide it this way.

I still have to try this, I'm not 100% sure extensions can call other extensions during initialization. It might not work correctly if the extensions are loaded in the wrong order.

buffer_get_error(id)
buffer_clear_error(id)

I don't really like the idea that a buffer can have an error. If someone writes code that reads past the end of a buffer, it is not likely that he will explicitly check for errors either. The only real improvement would be with exceptions, but GM doesn't support anything like that of course. As it is, I'd prefer the behaviour to be undefined but safe. In my own Buffers implementation I just read until the end and then stop there, the rest of the returned data is undefined (unless you try to read a string - in that case a reasonable behaviour is still possible and I just return the first part of the string)

They are optional, you can simply ignore the errors if you want. If you read past the end of the buffer, you will simply get the default value for that type (which is either 0, 0.0, an empty string, or a block of null bytes depending on the function you're using). I added them because some users might want to make sure the data they've just read is actual data. This could be important in some situations.

buffer_read_from_file(id,filename)
buffer_write_to_file(id,filename)

IMO, that should go into its own dll/extension - the very purpose of this project is to make things more modular

I know, but they're so simply I thought it would be silly not to add them. If I comment them out the DLL becomes 0.5KB smaller, that's hardly noticeable. We can still write a separate extension with more complex file functions.

Can you upload this to github or some other collaboration platform?

I have very little experience with revision control software, could you upload it somewhere?

I just made some changes to the code. The previous version would write everything to the end of the buffer, the new version writes it at the current position and resizes the buffer if needed. This is similar to writing to files. Now it's also possible to write to the middle of the buffer. I also added buffer_set_length (for GM).

The link is still the same:
http://gm.maartenbaert.be/shared_buffers.zip
I'm writing the documentation now.

[Medo42]

They are optional, you can simply ignore the errors if you want. If you read past the end of the buffer, you will simply get the default value for that type (which is either 0, 0.0, an empty string, or a block of null bytes depending on the function you're using). I added them because some users might want to make sure the data they've just read is actual data. This could be important in some situations.

In this situation you should check *first* whether there is actually enough data left in the buffer.

I just made some changes to the code. The previous version would write everything to the end of the buffer, the new version writes it at the current position and resizes the buffer if needed. This is similar to writing to files. Now it's also possible to write to the middle of the buffer. I also added buffer_set_length (for GM).

These are both things I don't see much practical use for (unless you can come up with a good example). You could always add these functions later if it turns out they are needed, but you can't remove them again once they are included in a release.

[Maarten Baert]

In this situation you should check *first* whether there is actually enough data left in the buffer.

Yes, but that's not always possible. If you're reading null-terminated strings from the buffer, you don't know the size in advance - unless you read the string byte by byte, which is cumbersome.

These are both things I don't see much practical use for (unless you can come up with a good example). You could always add these functions later if it turns out they are needed, but you can't remove them again once they are included in a release.

You can also use buffers as large arrays, which is useful to save memory. A 1024x1024 ds_grid uses 24*1024*1024 bytes = 24MB, but if you're only storing bytes this could be done with a 1MB buffer. A 3D particle system DLL could also use buffers to store the position of the particles, so it can be passed to a 3D graphics DLL directly without the overhead of rewriting the entire buffer every time something changes.

I think it can also be useful for some file formats. Some file formats store 'pointers' to other parts of the file in the file itself, which is cumbersome if you can't 'go back' to set the pointers to parts of the file that have been written later.

I think it would be great to have a simple 'memory block' data structure in GM, which gives you the same freedom as C++. I don't see why you'd want to limit it to sequential writing. Random access writing doesn't make the DLL significantly more complicated, slower or harder to use. You can still do sequential reading or writing, you just have to set the position to 0 if you want to start reading from the start again.

I've finished the help file, it's included in the ZIP:
http://gm.maartenbaert.be/shared_buffers.zip

Edited by Maarten Baert, 19 April 2011 - 03:45 PM.

[Medo42]

You are moving very fast with the implementation, which is nice, but I hope you don't mind if I try out some ideas and change what you have done so far around a bit. Then you'll finally be able to criticise my stuff instead of me just nagging about yours . I hope to be constructive with this though, so don't take it as anything against you - I just want to make sure this in the best possible shape before people start using it.

The array/file-like usage could be useful in some cases, it is just not what I had in mind originally.

What I want to do is try to find a minimal interface for buffers and seperate the existing functions into an implementation of that interface (e.g. the functions for reading/writing blocks of data) and "helper" functions which only use that interface (e.g. reading/writing specific data types).

The second point would be to make this all work without a dll, using a very small statically linked library instead. I have a plan for how that would work already, and it would get rid of the initialization problem entirely. Maybe it can even be done with just a .hpp file and no library at all.

And when that works, the GM buffer manipulation functions can be put into their own dll/extension that just uses the buffer sharing code like any other extension. This might seems a little bit pointless, but I'd like to try it because splitting things up into independend components is very often a good idea. If nothing else, it helps avoid the argument whether there should be file access functions included in the dll, because it's now in an exchangable component, even if no alternative is ever written.

Edited by Medo42, 20 April 2011 - 01:47 PM.

[sabriath]

Call me stupid...but why not just simply do this:

//in gm

var temp;

temp = "012345678";

external_call(get_pointer, someid, temp);
external_call(give_pointer, someotherid, temp);

I did a proof-of-concept that GM will actually pass strings byref to their data (not the variable itself, but the data it contains is retained). This allowed me to create hooks into classes by creating enough space for a string to hold an entire class and then calling a 'create' function to move it there. I then made it so that the class could hold pointer data to DLL allocated space, and since it's a pointer, GM wouldn't mess up the actual data for it.

You could use this same technique to pass a pointer reference between DLL extensions by creating a "get_pointer" function on one and a "give_pointer" on another. Since the DLL would allocate the space (and the OS would link it to the running applications handle), it doesn't change it's location between extensions

An example:

gmexportd get_pointer(double id, buffer* pt)
{
  pt = mybuffers[static_cast<int>(id)];
}


gmexportd give_pointer(double id, buffer* pt)
{
  otherbuffers[static_cast<int>(id)] = pt;
}

However, I am not sure how this will help programmers by "modularizing" a standard way of communicating between extensions. It might be great for 2 extensions to combine minds and come up with something together, but trying to come up with a general standard?

[Maarten Baert]

You are moving very fast with the implementation, which is nice, but I hope you don't mind if I try out some ideas and change what you have done so far around a bit. Then you'll finally be able to criticise my stuff instead of me just nagging about yours . I hope to be constructive with this though, so don't take it as anything against you - I just want to make sure this in the best possible shape before people start using it.

Sure, no problem .

The array/file-like usage could be useful in some cases, it is just not what I had in mind originally.

What I want to do is try to find a minimal interface for buffers and seperate the existing functions into an implementation of that interface (e.g. the functions for reading/writing blocks of data) and "helper" functions which only use that interface (e.g. reading/writing specific data types).

I think that's the main difference between my idea and yours: You're trying to solving one problem: "DLLs need a way to transfer data to each other", but I'm trying to solve a second problem at the same time: "Many DLLs need buffers, it would be much simpler if all DLLs could use the same buffers instead of their own implementation".

The reason I think we have to solve both problems at once is that most DLLs won't be compatible, unless they were specifically designed to communicate with each other. If you create a platform to transfer raw data without any formatting, it will be used by only a few DLLs. But if you create a shared buffers system that also allows GML users to read or write data, all DLLs that need buffers can use it. And there are lots of DLLs that could use buffers:
- data structure DLLs can use it to serialize/unserialize data structures
- file IO Dlls can use it to save/load files
- socket DLLs can use it to send/receive data
- HTTP request DLLs can use it to store the data that was downloaded from the server
- cryptography DLLs can use it to encrypt/decrypt binary data or calculate hashes (MD5/SHA1/...)
- compression DLLs can use it to compress/decompress data
- ...
The possibilities are endless, and all those DLLs would instantly be compatible with each other - simply because they're using the same buffers. You could serialize ANY data structure, use ANY compression DLL to compress that data, then use ANY cryptography DLL to encrypt it, and finally use ANY network DLL to send the data to another computer. This is currently not possible without using temp files or converting the data to a hex string every time.

I agree with what you said about a minimal interface. Most DLLs won't need functions like buffer_write_int32, you can easily do that yourself in C++ if you have buffer_write_data, which takes any number of bytes. I will try to rewrite it to make the interface a bit simpler.

The second point would be to make this all work without a dll, using a very small statically linked library instead. I have a plan for how that would work already, and it would get rid of the initialization problem entirely. Maybe it can even be done with just a .hpp file and no library at all.

I could be wrong, but I don't think that will work that easily. If DLLs have their own runtime libraries (e.g. because they use different compilers), they will also have their own heap. So you can't just allocate memory in DLL_A and resize the same buffer in DLL_B - unless you use functions like GlobalAlloc, I think (but I'm not sure). MSDN says GlobalAlloc is slower than the default malloc, I will have to test this to see if the difference is relevant (if it works in the first place).

By the way, statically linked libraries are not compatible with other compilers, so you would have to create a separate library for every compiler. It would be easier to simply add one .cpp file and one .h file IMHO.

I think we will need the helper functions anyway, so why not keep it simple and just add the functions to the helper functions DLL?

And when that works, the GM buffer manipulation functions can be put into their own dll/extension that just uses the buffer sharing code like any other extension. This might seems a little bit pointless, but I'd like to try it because splitting things up into independend components is very often a good idea. If nothing else, it helps avoid the argument whether there should be file access functions included in the dll, because it's now in an exchangable component, even if no alternative is ever written.

Good point, but I think the buffers would be almost useless without the helper functions.

Call me stupid...but why not just simply do this:
[...]
I did a proof-of-concept that GM will actually pass strings byref to their data (not the variable itself, but the data it contains is retained). This allowed me to create hooks into classes by creating enough space for a string to hold an entire class and then calling a 'create' function to move it there. I then made it so that the class could hold pointer data to DLL allocated space, and since it's a pointer, GM wouldn't mess up the actual data for it.

I'd prefer not to rely on this, because it could change in the future (if YYG decides to create a new runner for GM9, for example).

However, I am not sure how this will help programmers by "modularizing" a standard way of communicating between extensions. It might be great for 2 extensions to combine minds and come up with something together, but trying to come up with a general standard?

See above .

Edited by Maarten Baert, 20 April 2011 - 04:02 PM.