85 replies to this topic

[PsichiX]

Title: Ether - Network Library For Game Maker

Genre: Network Engine

Download: http://psichix.gmcla.../Ether_1100.zip
File size: 116 kb.

Authors:

Programming:

• PsichiX - XenoN Core
• Snake - GMapi

Description: Ether is a network library (based on XenoN Core Ether module and GMapi; also is a part of SixthSense ) for GameMaker, that contain features of most other libraries (like all-in-one dll). Supports sockets to network communication, buffers to store data, sessions for automated client-server communication (for multiplayer games primarily), HTTP requests to communicate with Web servers (asynchronous download/upload files and data from/to the server), and XML languages reader (HTML websites). Project file contain examples of each engine feature, and I'm working on other advanced examples.

TODO examples list:

• files uploader using communication with sendspace
• search engine using communication with google
• data streaming to clients (movies, audio, etc.)

Edited by PsichiX, 27 December 2010 - 04:27 AM.

[YellowAfterlife]

Looks great. Everything in examples works.
Question 1: {Requests} use POST\GET methods?
Question 2: What is the licence for using this?

[PsichiX]

1.
for GET method use: EtSessionDownload(); or EtSessionDownloadAsync() and EtSessionDownloadProgress() [asynchronous mode];
for POST method use: EtSessionSendForm(); or EtSessionSendFormAsync() and EtSessionSendFormProgress() [asynchronous mode];
for other methods use: EtSessionRequest(); or EtSessionRequestAsync() and EtSessionRequestProgress() [asynchronous mode];
EtSessionSendForm*() and EtSessionRequest*() works like download example (some arguments are different - more info in function scripts)

2. totally free for commercial use

Edited by PsichiX, 24 May 2010 - 10:49 AM.

[TheMagicNumber]

Looks great, but are there any more details available? Some descriptions of the words you used would be nice.

Can we expect better documentation?
Are sessions and requests mainly used for HTTP?
How are the packets structured?
Is GMAPI only used for callbacks (it seems)?

I noticed that 39DLL does have better buffer <-> file functions. You only have load and save. Also, there seems to be a problem with downloading the Google homepage.

Edited by TheMagicNumber, 24 May 2010 - 01:43 PM.

[PsichiX]

Can we expect better documentation?

yes, but i don't know when it will be done.

Are sessions and requests mainly used for HTTP?

Session types: Download, SendForm, Request use HTTP protocol, other functions are strictly userdefined protocols. Now i'm working on Mail (SMTP, POP3) protocol

How are the packets structured?

Socket send whole buffer at once, with size-header (if we don't know how much data trying to send and receive) or without size-header.
Game session send data like socket.
HTTP request session send data like socket but receive parts and flush them to buffer.
Now I'm working on sending splited parts of buffer, and receiving like in HTTP requests session, for sockets and game sessions.

Is GMAPI only used for callbacks (it seems)?

only for callbacks

I noticed that 39DLL does have better buffer <-> file functions. You only have load and save.

what functions I missed?

Also, there seems to be a problem with downloading the Google homepage.

i will check this, thanks for information

[TheMagicNumber]

39DLL can read from files somewhat like C can, it can read chunks of a file into the buffer. You can only read an entire file into the buffer with Ether.

I just saw EtBufferSetResize and EtBufferGetResize, what do these do?

By the way, if you want to keep your capitalizations, EtSocketGetUDPbuffSize and EtSocketSetUDPbuffSize should capitalize "buff".

[PsichiX]

39DLL can read from files somewhat like C can, it can read chunks of a file into the buffer. You can only read an entire file into the buffer with Ether.

Ether can read (and save) parts of files too. EtBufferLoad() (and EtBufferSave()) second argument is bytes offset and third argument is part size (0 mean read whole file)

I just saw EtBufferSetResize and EtBufferGetResize, what do these do?

default resizing buffer mode is adjust buffer space to needed size (if resize is 0), but if resize is more than 0 then buffer resize operation will reserve bonus memory space and when buffer space will be all filled then it will resize to bonus size again. example:
SetResize to 0:
step 1: [1][1][1][1]
step 2: [1][1][1][1][1][1][1][1]
step 3: [1][1][1][1][1][1][1][1][1][1][1][1]
SetResize to 8:
step 1: [1][1][1][1][0][0][0][0]
step 2: [1][1][1][1][1][1][1][1]
step 3: [1][1][1][1][1][1][1][1][1][1][1][1][0][0][0][0]
[1] - filled memory space
[0] - empty memory space
this feature allow to fast data writing. (memory block is reallocated not in each writing but only if whole memory space is filled)

By the way, if you want to keep your capitalizations, EtSocketGetUDPbuffSize and EtSocketSetUDPbuffSize should capitalize "buff". tongue.gif

good point ;P

Edited by PsichiX, 25 May 2010 - 11:39 AM.

[TheMagicNumber]

You didn't have those parameters in the scripts I have, did you update it or something?

Same thing I did for my buffer in CanNet.

[PsichiX]

You didn't have those parameters in the scripts I have, did you update it or something?

oups :0 i forgot add them, sorry, today i will do this

[PsichiX]

Update to version 1.0.2.0!
Added missed arguments in EtSessionSend(), EtSocketSend(), EtBufferSave() and EtBufferLoad() scripts. Now You can send part of buffer, not only whole buffer data.

[TheMagicNumber]

Do you think you can add HTML5 WebSocket support?

[PsichiX]

I was read something about this, and i think it is possible to add websocket support to new Ether version

[TheMagicNumber]

Awesome! Looking forward to it.

[TheMagicNumber]

I was just going to start making a websocket example, but I need to make the packets in a specific format. I need no formatting.

Question: How are packets formatted?

Found that, having problems detecting if a client lost connection, though.

Edited by TheMagicNumber, 22 June 2010 - 10:47 PM.

[LoopStan]

Wow. This DLL seems to be feature rich, and the people that have used it, really like it. I will have to use this in my current project. Once I get the hang of it, and start to use it, i'll let you know how I feel about it from a person who isn't so good in this area.

EDIT : I've looked over it and ran the example, and it works really well! I do have a request though, is there a possibility of being able to remove the GMApi portion, so we can use it in other languages as well? Or would you rather keep it to gm.

Edited by LoopStan, 22 June 2010 - 11:09 PM.

[PsichiX]

There is special version of this lib for C++, based on Ether module of my XenoN Core engine. I will compile it without GMapi (for non-GM apps) today and you will get it tomorrow

Edited by PsichiX, 23 June 2010 - 06:24 AM.

[LoopStan]

Thank you very much! This will help me a ton, since the current project I'm working on is split between Gm and C++. Can't wait!

[PsichiX]

Sorry, but yesterday I was too busy to make that, but today I will do this for sure

[LoopStan]

Hey, can you release this without gmapi so i can use it in a c++ server for my game? Thanks

[kalzme]

can the xml parser read CDATA properly?
if so, how?

for example this:

<item>
<title><![CDATA[bla bla bla]]></title>            
</item>

i have already tried to read it like this:

EtXmlTagGet(xml, 'item#id:0/title/');

but that doesn't work.

[TheMagicNumber]

I am certain EtXmlTagGet does not support "paths" to elements, you must do that yourself. Look at an example.

[PsichiX]

loopstan: i will try do this in this week
kaizme: i will add CDATA and comments tags parser.

[kalzme]

loopstan: i will try do this in this week
kaizme: i will add CDATA and comments tags parser.

thnx, that would be great!

I am certain EtXmlTagGet does not support "paths" to elements, you must do that yourself. Look at an example.

this is taken from the example & it works for me on normal tags with no CDATA in it..:

// set title tag value as window caption
room_caption = EtXmlTagGet(
                xml,                // XML element
                'html/head/title/'  // tag path (last '/' character is very important for setting/getting tags values! it's like trying to get unnamed tag - unnamed tags are values inside tags, for example: '<tag>value</tag>')
                )

[TheMagicNumber]

I never saw that, thanks.

[kalzme]

btw, is there any way to read the response header fields?
e.g., Content-Length, Content-Type, Date etc.?

[IceMetalPunk]

@kalzme: Yeah. Send the request yourself with the sockets, then the response will include the headers.

@PsichiX: This is great, but I'm having some issues with the XML functions.

Firstly, does including the XML declaration (<?xml version="1.0" encoding="UTF-8"?>) in the beginning of the text cause problems with EtXmlParse? I'm getting the XML from a web server and I want to know if I should remove that before parsing.

Second, I'm wondering why my attempt to parse an XML document (received successfully from a web server via sockets) isn't working.

I have this format XML file (received from Twitter API with user info removed for privacy):

<?xml version="1.0" encoding="UTF-8"?>
<statuses type="array">
<status>
  <created_at>Sat Aug 21 04:06:53 +0000 2010</created_at>
  <id>21720949158</id>
  <text>[TWEET HERE]</text>
  <source>web</source>
  <truncated>false</truncated>
  <in_reply_to_status_id></in_reply_to_status_id>
  <in_reply_to_user_id></in_reply_to_user_id>
  <favorited>false</favorited>
  <in_reply_to_screen_name></in_reply_to_screen_name>
  <retweet_count></retweet_count>
  <retweeted>false</retweeted>
  <user>
    <id>6972182</id>
    <name>[USERNAME HERE]</name>
    <screen_name>[USER'S SCREEN NAME HERE]</screen_name>
    <location></location>
    <description></description>
    <profile_image_url>[USER'S PROFILE PIC URL HERE]</profile_image_url>
    <url>[USER'S PROFILE URL HERE]</url>
    <protected>false</protected>
    <followers_count>58719</followers_count>
    <profile_background_color>000000</profile_background_color>
    <profile_text_color>000000</profile_text_color>
    <profile_link_color>FF0099</profile_link_color>
    <profile_sidebar_fill_color>CCCCCC</profile_sidebar_fill_color>
    <profile_sidebar_border_color>000000</profile_sidebar_border_color>
    <friends_count>137</friends_count>
    <created_at>Wed Jun 20 17:02:52 +0000 2007</created_at>
    <favourites_count>0</favourites_count>
    <utc_offset>-18000</utc_offset>
    <time_zone>Eastern Time (US &amp; Canada)</time_zone>
    <profile_background_image_url>[USER'S PROFILE BACKGROUND IMAGE URL HERE]</profile_background_image_url>
    <profile_background_tile>false</profile_background_tile>
    <profile_use_background_image>true</profile_use_background_image>
    <notifications>false</notifications>
    <geo_enabled>false</geo_enabled>
    <verified>false</verified>
    <following>true</following>
    <statuses_count>2427</statuses_count>
    <lang>en</lang>
    <contributors_enabled>false</contributors_enabled>
    <follow_request_sent>false</follow_request_sent>
    <listed_count>1786</listed_count>
    <show_all_inline_media>false</show_all_inline_media>
  </user>
  <geo/>
  <coordinates/>
  <place/>
  <contributors/>
</status>
.
.
.
</statuses>

That said, when I try to get EtXmlTagExists(xml,"statuses/") (I've also tried without the ending slash), it returns 0, even though the first tag is clearly <statuses>. Am I doing something wrong?

-IMP

[PsichiX]

<statuses type="array">

this tag should looks like that:

<statuses type="array" />

because XML reader does not support old standard, but i see that some popular sites like twitter still use this (ehh ;0).
I will add this to next update, and reading <?xml?> too, because now reader parse only pure tags, even CDATA are not parsed, and CDATA will be readable soon

btw, is there any way to read the response header fields?

engine already does support this thing, i will add it to Ether.dll too.

Edited by PsichiX, 22 August 2010 - 10:29 AM.

[kalzme]

@PsichiX: thank you, that would also help me a lot. One more question though: does the EtSessionDownload() function take the the 'Transfer-Encoding' header field into account? (Transfer-encoding)

@IMP: I knew that, but it would be more easy to just use

EtSessionDownload(url, buffer);

and then be able to access the headers.
and for your question, i use it like this:

var title, descr, link;
title = EtXmlTagGet(xml, '?xml/rss/channel/title/');
descr = EtXmlTagGet(xml, '?xml/rss/channel/description/');
link = EtXmlTagGet(xml, '?xml/rss/channel/link/');

This is how I use it, so maybe you should also include the '?xml/' at the beginning?
I'll try test it out soon.

EDIT:
I got it to work:

str = '
<?xml version="1.0" encoding="UTF-8"?>
<statuses type="array">
<status>
  <created_at>Sat Aug 21 04:06:53 +0000 2010</created_at>
  <source>web</source>
  <truncated>false</truncated>
</status>
</statuses>
'

EtLibraryLoad('Ether.dll');

buffer = EtBufferCreate()
EtBufferWriteString(buffer, str);

xml = EtXmlCreate()
EtXmlParse(xml, buffer, chr(13) + chr(10) + chr(9) + ' ');

EtBufferFree(buffer);
EtBufferDestroy(buffer);

show_message(EtXmlTagGet(xml, '?xml/statuses/status/created_at/'));

EtXmlDestroy(xml);
EtLibraryFree();

Edited by kalzme, 22 August 2010 - 12:53 PM.

[IceMetalPunk]

@PsichiX: Correct me if I'm wrong (I don't know much about XML, to be honest), but aren't you supposed to only end a tag with /> if it's self-closing, i.e. has nothing inside it? In this case, the <statuses> tag has everything inside it .

@kalzme: Thank you! That worked! I never thought the parse was considering the <?xml?> tag as a tree tag . Thanks!

That said, now I'm a little confused. The <statuses> tag is an array of <status> tags. So how would I, for example, get the value of the created_at tag in the third (or fifth, or thirtieth) <status> tag?

*EDIT* Nevermind. I should look at the examples before asking questions, shouldn't I ? I've got it now, and it's working great!

-IMP

Edited by IceMetalPunk, 22 August 2010 - 04:09 PM.

[kalzme]

@PsichiX: Correct me if I'm wrong (I don't know much about XML, to be honest), but aren't you supposed to only end a tag with /> if it's self-closing, i.e. has nothing inside it? In this case, the <statuses> tag has everything inside it .

@kalzme: Thank you! That worked! I never thought the parse was considering the <?xml?> tag as a tree tag . Thanks!

-IMP

You are correct, I guess he didn't see the closing tag at the end.
I'm glad it worked.

Oh and by the way if the <statuses> tag has multiple <status> tags in it you can easily read them this way:

count = 0;
while(true) {
    var conf;
    conf = '?xml/statuses/status#id:' + string(count);
    if (EtXmlTagExists(xml, conf)) {
        EtXmlTagGet(xml, conf + '/created_at/');
        count += 1;
    }
    else {
        break;
    }
}

Edited by kalzme, 22 August 2010 - 04:05 PM.

[IceMetalPunk]

@kalzme: I'm officially ready to declare you my savior ! Is there a list of which special hashtags I can use here? I'm sure it would prevent many more questions on my end later on .

-IMP

[kalzme]

@kalzme: I'm officially ready to declare you my savior ! Is there a list of which special hashtags I can use here? I'm sure it would prevent many more questions on my end later on .

-IMP

Sorry, I think have to disappoint you on this one.

With hashtags, you mean like: 'status#id:' + count
or do you mean the hashtags used by twitter?

I've only found those in the example:

• #next - This one is used to get the next tag elemen
• #new - This one is used to create a new tag
• #id:0 - This is used to get a specific tag (if you have multiple tags with the same name)

there's also characters like:
'.' - This means you use the parent tag
'~' - This can be used to get an attribute (<statuses type="array"> so you would use ~ to get the type)

That's all I know
They all appear in the example, so you can see for yourself how they work.

[IceMetalPunk]

Sorry, I think have to disappoint you on this one.

With hashtags, you mean like: 'status#id:' + count
or do you mean the hashtags used by twitter?

I've only found those in the example:

• #next - This one is used to get the next tag elemen
• #new - This one is used to create a new tag
• #id:0 - This is used to get a specific tag (if you have multiple tags with the same name)

there's also characters like:
'.' - This means you use the parent tag
'~' - This can be used to get an attribute (<statuses type="array"> so you would use ~ to get the type)

That's all I know
They all appear in the example, so you can see for yourself how they work.

Yeah, I meant the Ether hashtags. I saw #next and #new, but must have missed the #id:? tag before.

@PsichiX: Are those all the tags/operators we can use? If not, any chance you could post a quick reference for all the possible tags? It doesn't need to be too detailed; just something like "#next - Gets the next tag" for each...

-IMP

[PsichiX]

there are only a few hashtags for now like:
#next - get next tag
#prev - get previous tag
#id:3 - get 3th tag with that name
. - get current tag (if you are use parent tag parametr in function)
.. - get parent tag (one level up in tag-tree)
~ - get an attribute of tag

obviously you can mix them, example:
"html/body/table/tr#id:3/td/span#next#next" - this will go to 3th tr tag in html/body/table/ then 3th (because will find first span, then next + next = +2 so that = 3) span tag in html/body/table/tr#id:3/td/

"html/head/title/../.." - this will go to html tag

[IceMetalPunk]

OK, just wanted to be sure there weren't any "secret tags" .

But I've hit an odd problem with Ether now. I have an XML tree parsed from this:

<?xml version="1.0" encoding="UTF-8"?>
<status>
  <created_at>Wed Aug 25 03:04:47 +0000 2010</created_at>
  <id>22055906984</id>
  <text>Testing TweetyMotion...please ignore this tweet. Thanks :)</text>
  <source>web</source>
  <truncated>false</truncated>
  <in_reply_to_status_id></in_reply_to_status_id>
  <in_reply_to_user_id></in_reply_to_user_id>
  <favorited>false</favorited>
  <in_reply_to_screen_name></in_reply_to_screen_name>
  <retweet_count></retweet_count>
  <retweeted>false</retweeted>
  <user>
    <id>15929425</id>
    <name>IceMetalPunk</name>
    <screen_name>IceMetalPunk</screen_name>
    <location>USA</location>
    <description></description>
    <profile_image_url>http://a0.twimg.com/profile_images/923214624/bc7095da-66e8-4dba-8c36-b1b4d4cc5a8b_normal.png</profile_image_url>
    <url></url>
    <protected>false</protected>
    <followers_count>3</followers_count>
    <profile_background_color>000000</profile_background_color>
    <profile_text_color>000000</profile_text_color>
    <profile_link_color>0000FF</profile_link_color>
    <profile_sidebar_fill_color>e0ff92</profile_sidebar_fill_color>
    <profile_sidebar_border_color>87bc44</profile_sidebar_border_color>
    <friends_count>8</friends_count>
    <created_at>Thu Aug 21 08:57:28 +0000 2008</created_at>
    <favourites_count>0</favourites_count>
    <utc_offset>-18000</utc_offset>
    <time_zone>Quito</time_zone>
    <profile_background_image_url>http://s.twimg.com/a/1282688694/images/themes/theme1/bg.png</profile_background_image_url>
    <profile_background_tile>false</profile_background_tile>
    <profile_use_background_image>true</profile_use_background_image>
    <notifications>false</notifications>
    <geo_enabled>false</geo_enabled>
    <verified>false</verified>
    <following>false</following>
    <statuses_count>7</statuses_count>
    <lang>en</lang>
    <contributors_enabled>false</contributors_enabled>
    <follow_request_sent>false</follow_request_sent>
    <listed_count>0</listed_count>
    <show_all_inline_media>false</show_all_inline_media>
  </user>
  <geo/>
  <coordinates/>
  <place/>
  <contributors/>
</status>

Now, according to Ether, the tag ?xml/status/text exists. Great. But when I try to get its value using EtXmlTagGet(xml,"?xml/status/text/"), I get a blank string. Any idea why?

-IMP

[kalzme]

...
Now, according to Ether, the tag ?xml/status/text exists. Great. But when I try to get its value using EtXmlTagGet(xml,"?xml/status/text/"), I get a blank string. Any idea why?

-IMP

Is that the only xml?
because otherwise it could be because you have multiple <status> tags so to read from it you would have to specify the id with the #id: hashtag like this: "?xml/status#id:0/text/".

That's all I can come up with.

(EDIT: typo)

Edited by kalzme, 25 August 2010 - 08:10 AM.

[IceMetalPunk]

Yes, that's the only XML. That's the result of updating your status (posting a Tweet) via Twitter API. It sends back your status as confirmation, so there's only one status tag. I'll try using status#id:0 anyway, just to see if it works, though.

*EDIT* Nope, still showing up as an existing tag, but a blank string for the text value...

-IMP

Edited by IceMetalPunk, 26 August 2010 - 06:02 PM.

[Doogie_Forever]

Your download script doesn't support chunked transfer encoding for website data, if you could possibly implement this in a future build it would make a lot of work with PHP easier.
I had to use some hacky PHP script to get it to return data that works properly with your dll.

http://en.wikipedia....ansfer_encoding

[IceMetalPunk]

Your download script doesn't support chunked transfer encoding for website data, if you could possibly implement this in a future build it would make a lot of work with PHP easier.
I had to use some hacky PHP script to get it to return data that works properly with your dll.

http://en.wikipedia....ansfer_encoding

Chunked encoding is a fairly simple format. So while it would, of course, be useful to have Ether's download functions support it, it would be easier to use the socket functions to parse it yourself rather than trying to make it work on the PHP end.

-IMP

*EDIT* I've worked around my previous problem by implementing my own simple parsing, since I'll always need only text and date information in this one context.

Now I'm having a more distressing issue, one I can't work around.

I have a (third-party, non-GM) application that is both binding and sending information on the same port to 127.0.0.1 (basically using TCP/IP as a communications method for any program that wants information from it). I want to listen in on this information, but it seems that because the program binds the port, I cannot do so with Ether.

So when I start the program first, then my GM program, Ether cannot listen on the port. And when I start my GM program first, then the third-party one, the third-party one says it can't bind/connect. There must be a way to fix this, right? How?

Edited by IceMetalPunk, 06 September 2010 - 08:44 AM.

[majix560]

@IceMetalPunk
You have to use a different port.
A port is bound to the program using it to help simplify networking programming.
Scenario of why this is good

Spoiler

Thea reason why only one program can send and receive on a port is because if more then one is running on the same port then something along the following lines would/could happen
Program a and program b are running on the same port
program b receives data on that port
program a thinks the data is for it and crashes

@PsichiX
3th is not gramaticaly correct
that's like saying "thirth" instead of saying "third"
the two letters are supposed to be the sound at the end, not the beginning
(feel free to call me a grammar nazi)

Edited by majix560, 07 September 2010 - 03:53 AM.

[majix560]

Could you please make Ether into an extension?

[PsichiX]

i can promise that i will try do this

[TheMagicNumber]

What's wrong with scripts?

[IceMetalPunk]

@IceMetalPunk
You have to use a different port.
A port is bound to the program using it to help simplify networking programming.
Scenario of why this is good

Spoiler

Thea reason why only one program can send and receive on a port is because if more then one is running on the same port then something along the following lines would/could happen
Program a and program b are running on the same port
program b receives data on that port
program a thinks the data is for it and crashes

Right, that's what I thought. Unfortunately, this third-part program binds and sends on the same port. It's designed to send data to Flash--can Flash somehow bypass the restriction? And if I stick with GM, is there any way around this?

-IMP

[majix560]

@IceMetalPunk
You have to use a different port.
A port is bound to the program using it to help simplify networking programming.
Scenario of why this is good

Spoiler

Thea reason why only one program can send and receive on a port is because if more then one is running on the same port then something along the following lines would/could happen
Program a and program b are running on the same port
program b receives data on that port
program a thinks the data is for it and crashes

Right, that's what I thought. Unfortunately, this third-part program binds and sends on the same port. It's designed to send data to Flash--can Flash somehow bypass the restriction? And if I stick with GM, is there any way around this?

-IMP

I don't know.
I was just theorizing.

[LoopStan]

I am requesting that Ether have the GMAPi taken out for other uses in other languages. Sorry if I have asked a lot, but I'm depending on it for my new project.

[PsichiX]

currently i'm busy because i have to finish some commercial work, and for now i can't do anything with this. Just be patient, it's just a couple days

[TheMagicNumber]

I am requesting that Ether have the GMAPi taken out for other uses in other languages. Sorry if I have asked a lot, but I'm depending on it for my new project.

It's a GM DLL. The use of GMAPI is great, don't ruin it.

[Consoft]

I am requesting that Ether have the GMAPi taken out for other uses in other languages. Sorry if I have asked a lot, but I'm depending on it for my new project.

It's a GM DLL. The use of GMAPI is great, don't ruin it.

Didn't realise he was strictly and utterly contained to releasing just one version of Ether?

[LoopStan]

I am looking to use the DLL and its capabilities in C++. This is why I am requesting the GMAPI to be removed, but still have one with it in. GMAPI is great, just not when you need to do cross language work. I am using the DLL with GMAPI in my new program right now. Thanks for giving me a time estimate on the time, now I can plan my project around that.