Jump to content


Photo

String Tokenizer


  • Please log in to reply
50 replies to this topic

#1 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 12 March 2009 - 11:22 PM

I need my string tokenizer ( http://en.wikipedia....exical_analysis ) to be more heavily tested and this seems like a pretty good place to do so. So, I have made it into a DLL.

If anybody would use this DLL seriously, it can be used to make your own interpreted language with speed improvements over a GML interpreter (made in GML, not interpreting GML). Some people have made their own mini scripting language to load things into a game (like external objects) and this can help you with that. I've used this to make simple interpreters and parsing text data from a PHP script into a list of servers.

I suggest nobody uses this. Make your own, this version is probably filled with issues.

Download: UpUrLoad

Important Info:
- It will not give you negative numbers, maybe you can figure out how to do that yourself (or, help me).
- This is not an interpreter, it simply splits a string into tokens.
- Keywords aren't case sensitive.

Please notify me of any bugs (post them here) so I can try to fix them.

Edited by TheMagicNumber, 04 September 2011 - 11:13 PM.

  • 3

#2 cswenson1

cswenson1

    CS Games

  • New Member
  • 725 posts

Posted 13 March 2009 - 06:44 PM

I think you should add a tutorial on how to understand, or even use this dll... From what i saw, it was lots of codes and functions that made no sence whatsoever
  • 0


My DLL's:
MWin DLL - Message Box DLL - Battery Life DLL

How to: Make a DLL for Game Maker - Part1 | Part2

#3 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 13 March 2009 - 07:27 PM

A lot of it is for listing the actual tokens. It truly isn't hard to use, you just need to get a good understanding of what each function does.

Update:
Re-uploaded with some more help, doesn't really explain what it does but what the functions do.

Edited by GMREC1111, 13 March 2009 - 08:18 PM.

  • 2

#4 uuf6429

uuf6429

    Covac Software

  • New Member
  • 2522 posts
  • Version:Unknown

Posted 14 March 2009 - 06:11 PM

Impressive, but can't find a real use for this - yet :lol:.

Hmm, maybe an xml tokenizer or something? But be usefull in developing a small language in GM.
  • 0

#5 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 18 March 2009 - 06:46 PM

Update:
Added some options for toggling the use of C-style comments, and the inclusion of quote characters in TkSTRING tokens.

I am also working on something with this, I mentioned it in the first post. I am attempting the creation of an external (editable) file that can create things in-game dynamically, easily, and universally. I need a few suggestions on what to add into it.
  • 2

#6 PickleMan

PickleMan

    Programmer

  • New Member
  • 995 posts
  • Version:Unknown

Posted 18 March 2009 - 07:15 PM

Like what? Brainfsck?
  • 0

#7 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 18 March 2009 - 07:59 PM

No, all Brainf*ck (come on, it's a real name) is made of is single character commands, meaning it can't really be split up in a way that it is useful. So, all you do there is loop through each character. If you want to, try replacing the stuff inside of Tok_SetSource() to any code you would like, it should tokenize it with no problem.

There is only one programming language I know of that will not work with, that would be Whitespace.

Edited by GMREC1111, 18 March 2009 - 08:00 PM.

  • 2

#8 Tahnok

Tahnok

    Friendly Madman

  • GMC Member
  • 1854 posts
  • Version:GM:Studio

Posted 18 March 2009 - 10:19 PM

It's a nice little DLL, and does what it promises, but it seems kinda over complicated. I mean, this can be done in GML. I'm not quite sure someone would want to include a DLL just to tokenize.
  • 0

gmc_signature.png


#9 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 19 March 2009 - 01:53 AM

can be done in GML

That is merely doing something like PHP's explode() function, this is much more then that. This does not split at any character. Tokens are identified to what category they should fall into, as in, tell the difference between a real and string in code without you needing to check.
  • 2

#10 LoopStan

LoopStan

    North-See Developer

  • GMC Member
  • 1398 posts

Posted 19 March 2009 - 02:15 AM

Wow, sweet! Great job, this can be so useful! 9/10!
  • 0

Posted Image


#11 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 19 March 2009 - 02:19 AM

Wow, sweet! Great job, this can be so useful! 9/10!

Thanks. :lol:
  • 2

#12 Tahnok

Tahnok

    Friendly Madman

  • GMC Member
  • 1854 posts
  • Version:GM:Studio

Posted 19 March 2009 - 04:21 AM

can be done in GML

That is merely doing something like PHP's explode() function, this is much more then that. This does not split at any character. Tokens are identified to what category they should fall into, as in, tell the difference between a real and string in code without you needing to check.

Ah OK, my mistake. I was getting confused because what Java calls the string tokenizer does what that script I linked to does, splits a string based on some token.
  • 0

gmc_signature.png


#13 Revel

Revel

    ɹǝqɯǝɯ ɔɯƃ

  • GMC Member
  • 4935 posts
  • Version:GM8

Posted 19 March 2009 - 04:47 AM

Any limits on the size of the string?
  • 0

#14 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 19 March 2009 - 10:14 AM

Any limits on the size of the string?

There is always a limit when it comes to a computer's memory. I just Google'd it and saw one page that said it has a near-2GB maximum. I never really took the length into consideration.
  • 2

#15 BrokenGlass

BrokenGlass

    GMC Member

  • New Member
  • 524 posts

Posted 21 March 2009 - 10:08 AM

Any limits on the size of the string?

There is always a limit when it comes to a computer's memory. I just Google'd it and saw one page that said it has a near-2GB maximum. I never really took the length into consideration.


*facepalm*
  • 0

#16 008008

008008

    GMC Member

  • New Member
  • 45 posts

Posted 30 April 2009 - 07:56 PM

Brilliant!!!!!!!!!
I was looking for a tokenizer in C++ but I couldn't find one, anyway this is super. :mellow:
I suppose you couldn't post the source as well? :whistle:

008008
  • 0

I am 5,6,7 registered!
GMC Member Since '06


#17 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 30 April 2009 - 10:39 PM

It's not C++. It's Delphi (Pascal). As for the source (I have it in C++), I'd rather not.

Edited by GMREC1111, 24 June 2009 - 01:14 PM.

  • 2

#18 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 24 June 2009 - 01:14 PM

Meh, here's the source: <old>

Have fun.

Edited by TheMagicNumber, 11 August 2010 - 12:43 AM.

  • 1

#19 Troy Martin

Troy Martin

    Surreal Earth Studios

  • New Member
  • 91 posts
  • Version:Unknown

Posted 25 June 2009 - 10:12 PM

:skull: All your source is in a header file? That's a little odd, don'tcha think?

But anyways, looks awesome! 9/10

Edited by Troy Martin, 25 June 2009 - 10:15 PM.

  • 0
Project Mindcrime Development milliBlog

Now I know you won't refuse
Because we've got so much to do
And you've got nothing more to lose
So take this number
And welcome
To Operation: Mindcrime

#20 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 25 June 2009 - 10:35 PM

I don't think its too odd. I didn't feel like doing a lot with the C++ version.
  • 2

#21 ze1

ze1

    GMC Member

  • GMC Member
  • 153 posts
  • Version:GM:Studio

Posted 12 February 2010 - 01:03 AM

Download link is down.
  • 0

#22 Razon

Razon

    GMC Member

  • GMC Member
  • 268 posts
  • Version:GM:Studio

Posted 12 February 2010 - 02:12 AM

http://host-a.net/rot740/Tokenizer.zip

Edited by Razon, 12 February 2010 - 06:53 AM.

  • 0

#23 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 12 February 2010 - 05:28 AM

Couldn't read my signature? :)

Anyways, I'll reupload it.

Done.

Edited by TheMagicNumber, 13 February 2010 - 04:39 AM.

  • 2

#24 iWizard

iWizard

    GMC Member

  • New Member
  • 47 posts

Posted 15 February 2010 - 05:53 PM

I suggest that you include in your first post that the command "Tok_SetSource()" does not reset the position of the string. In my case, I am processing commands that the user enters with this, but every time I did that, it would be missing all or part of the string depending on if one of my commands is longer.

For example:
1st Command: Tok_SetSource("command 1 variable 1") This Returns: "command" "1" "variable" "1"
2nd Command: Tok_SetSource("command 2 variable 2 asdf") This Returns: "asdf"

But if you have:
1st Command: Tok_SetSource("command 1 variable 1 asdf") This Returns: "command" "1" "variable" "1" "asdf"
2nd Command: Tok_SetSource("command 2 variable 2") This Returns: nothing
I was extremely confused when I reached this point. This was because the position was set to 25 when the second command was executed. So, at position 25 of the second command, there is nothing, therefore, it would return nothing.

This is because the position that the tokenizer starts at is not reset when running Tok_SetSource(). You must, between each command, execute Tok_SetPos(0). I am not sure if this was intentional, but it certainly was confusing for me. ;)

So I would need to do this instead:
1st Command: Tok_SetSource("command 1 variable 1") This Returns: "command" "1" "variable" "1"
2nd Command Tok_SetPos(0)
3rd Command: Tok_SetSource("command 2 variable 2 set 1") This Returns: "command" "2" "variable" "2" "set" "1"
Then after setting Tok_SetPos(0), everything works again. :)

--EDIT--
Do you think you can also include an escape character. Like using the \ to cancel out a quote? That would be a great feature to have.

Edited by iWizard, 15 February 2010 - 06:04 PM.

  • 0
~ iWizardPro
Simplicity. Creativity. Knowledge.

#25 YellowAfterlife

YellowAfterlife

    GMC Member

  • Global Moderators
  • 4157 posts
  • Version:GM:Studio

Posted 15 February 2010 - 06:09 PM

I remember this since gm lolcode sample used this ;)
Creating programming language in gm isn't the best plan ever (perfomance issues, you know), but this is just great.
  • 0
If my posts contain broken links, try looking around my website. I gradually make blog posts for any examples I make.
_.png_.gif

#26 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 16 February 2010 - 10:15 AM

In the set source script, add Tok_Reset(). I'll make a new version that has C-style escape sequences.
  • 1

#27 MrJackSparrow2

MrJackSparrow2

    GMC Member

  • GMC Member
  • 632 posts
  • Version:GM:Studio

Posted 18 February 2010 - 05:51 AM

A little thing that bothers me, 3.14 turns up as:
3 TKNumber
. TKDelimiter
14 TKNumber

I suppose you could just check if there is a number after the delimiter, however I think you should check this beforehand in the DLL. IDK, maybe I'm just lazy and picky. :)
Kick ass DLL though, good job. And yea, I just realized you said that in the file after I posted this... >_> Sorry.

Edited by MrJackSparrow2, 18 February 2010 - 05:53 AM.

  • 0

#28 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 19 February 2010 - 11:22 AM

I removed decimal numbers because it would catch "3........................14" as a number, I'll add it back and fix that in the new version.
  • 1

#29 MrJackSparrow2

MrJackSparrow2

    GMC Member

  • GMC Member
  • 632 posts
  • Version:GM:Studio

Posted 19 February 2010 - 10:36 PM

I removed decimal numbers because it would catch "3........................14" as a number, I'll add it back and fix that in the new version.

Oh, I never even thought of that. It's not a big deal but cool.
  • 0

#30 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 17 March 2010 - 10:29 PM

Working on new version right now. Should be done soon.

Edit: Should be done tomorrow (March 18th). Finished, but needs testing.

Edited by TheMagicNumber, 18 March 2010 - 02:55 AM.

  • 1

#31 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 18 March 2010 - 09:48 PM

Done the new version. :)

- Redone in C++.
- Multiple tokenizers (untested, please share results).
- Fixed decimal number tokens.
- Strings have escape sequences. (\n, \r, \t, \", \')
- You can get the line the tokenizer is on.
- Fixed setsource bug where it doesn't reset pos.
- Can detect basic "errors."

Some downsides:
- No options (yet!).
- Static delimiter list (currently).

I can't think of anything else. It would be nice if I could get some dynamic tokenizer created, with rules for tokens and stuff. Anybody interested?

Edited by TheMagicNumber, 21 March 2010 - 02:31 PM.

  • 2

#32 MrJackSparrow2

MrJackSparrow2

    GMC Member

  • GMC Member
  • 632 posts
  • Version:GM:Studio

Posted 23 March 2010 - 03:30 AM

What happened to comments?
  • 0

#33 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 23 March 2010 - 10:34 AM

What about them? :chikin
  • 2

#34 MrJackSparrow2

MrJackSparrow2

    GMC Member

  • GMC Member
  • 632 posts
  • Version:GM:Studio

Posted 28 March 2010 - 12:58 AM

The tokenizer doesn't support single quotations as a string.

Original String:"
MyName = 'Slim Shady';
MyAge = 16;
{show_message('Hi');}";


~Debug Syntax Checking Errors (0)~
~End of Errors~

Lines: 3


String: MyName
Token: TkWord

String: =
Token: TkDelimiter

String: '
Token: TkUnknown

String: Slim
Token: TkWord

String: Shady'
Token: TkWord

String: ;
Token: TkDelimiter

String:

Token: TkUnknown

String:

Token: TkEOL

String: MyAge
Token: TkWord

String: =
Token: TkDelimiter

String: 16
Token: TkNumber

String: ;
Token: TkDelimiter

String:

Token: TkUnknown

String:

Token: TkEOL

String: {
Token: TkDelimiter

String: show_message
Token: TkWord

String: (
Token: TkDelimiter

String: '
Token: TkUnknown

String: Hi'
Token: TkWord

String: )
Token: TkDelimiter

String: ;
Token: TkDelimiter

String: }
Token: TkDelimiter



That is my parsed data. I apologize for the long post.
"It would be nice if I could get some dynamic tokenizer created, with rules for tokens and stuff. Anybody interested?" What exactly do you mean by that?

Edited by MrJackSparrow2, 28 March 2010 - 01:00 AM.

  • 0

#35 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 28 March 2010 - 12:14 PM

Oh, I didn't port that over yet. I'll start working on it later.

And I meant one that you could define the rules for yourself, meaning you could add the single-quotations into it alone.
  • 2

#36 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 30 March 2010 - 09:56 PM

Update.

Added singe-quotation strings. They do not accept escape sequences.
Added custom delimiter list, new function to set it. Invalid delimiters would be whitespace (space, tab, etc.) and will not work properly.

Suggest more options to be added.
  • 2

#37 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 05 June 2010 - 03:43 AM

Updated the DLL, no functions changed. It will now not change case of (key)words.
  • 2

#38 fire_soul453

fire_soul453

    aarons video games

  • GMC Member
  • 542 posts

Posted 05 June 2010 - 03:29 PM

I think I might be able to use this. Thanks!
  • 0
Game maker Top sites
I'm sick of paypal. From now on I'm not buying indie games unless I can use something else...

#39 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 05 June 2010 - 06:52 PM

No problem!

Just report any problems you encounter. I use this and would appreciate feedback.
  • 2

#40 Brother156

Brother156

    GMC Member

  • Validating
  • 498 posts

Posted 15 June 2010 - 11:27 PM

The old Tokenizer worked p[retty well, although I never posted anything about it. But this version gives me an error:

___________________________________________
ERROR in
action number 1
of Create Event
for object obj_control:

Error defining an external function.

I extracted it, but it dosn't work.




#41 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 16 June 2010 - 12:29 AM

I would suggest you check the paths GM8 is looking in, and if not that, it might have been built improperly and need the MSVC++ 2008 runtime (should be statically linked, though).

It worked for others, it should for you.
  • 2

#42 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 28 July 2010 - 06:58 PM

Update!
- Error reporting (unterminated strings, stuff like that)
- User set newline character sequence
- User set whitespace characters
- User set comment containers
- User set string containers

Planned features include:
- Escape sequences (user defined, removed in this update, suggestions are welcome)
- Setting for case sensitivity
- Multi-character delimiters
- Loading keywords from a character separated list

Download is in first post, please contact me if any problems are found.

Edited by TheMagicNumber, 01 August 2010 - 03:26 AM.

  • 2

#43 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 01 August 2010 - 03:51 AM

Update again.

Escape sequences, case sensitivity, multi-character delimiters.
  • 2

#44 Aragon1029

Aragon1029

    GMC Member

  • New Member
  • 940 posts

Posted 01 August 2010 - 05:01 AM

Nice update, makes it even more useful.
Average GMCer probably doesn't even know how to use this though.

Edited by Aragon1029, 01 August 2010 - 05:01 AM.

  • 0


Rave Breakout! A new game for iOS devices!
Ever wondered how many lines of code your game has?
Guys, I have a great idea, let's protect the public through censorship.


#45 Burr

Burr

    GMC Member

  • New Member
  • 1 posts

Posted 10 August 2010 - 04:04 AM

Thanks for the DLL! I don't plan on using it for anything as prestigious as developing a scripting language (though I might eventually, now that you've given me the idea) but I think it'll end my search for the most efficient way of storing variable-length two-dimensional arrays in a local SQL database.

(The arrays contain physics construct data. It gets rather disorganized...)

Edited by Burr, 10 August 2010 - 04:13 AM.

  • 0

#46 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 10 August 2010 - 04:14 AM

It can be used for whatever you need, of course! I've used it in Dice Game for the server list.

Just a warning, the line number reported by the DLL is not reliable, I will fix this later.
  • 2

#47 Mayhem Games

Mayhem Games

    Proud Kiwi

  • GMC Member
  • 959 posts
  • Version:GM:Studio

Posted 10 August 2010 - 08:47 AM

Very nice! I know exactly what to with it. Thanks
  • 0

For a long time it puzzled me how something so expensive, so leading edge, could be so useless, and then it occurred to me that a computer is a stupid machine with the ability to do incredibly smart things, while computer programmers are smart people with the ability to do incredibly stupid things. They are, in short, a dangerously perfect match. ~ Bill Bryson


#48 attm

attm

    GMC Member

  • New Member
  • 29 posts

Posted 19 September 2010 - 05:13 AM

@TheMagicNumber

Heh, your dll would obviously be faster since it is coded in C++, but here's how it might look in GML (GM6). Just for laffs!
Place the code in the creation even of an object and put it in a room.

font0 properties are :
Font : Arial
Size 10
Character Range : Normal 32 till 127

/*
function : tokenizes an expression and stores the tokens in an array
An expression may contain :
numbers, (, ), *, /, +, -, %, "string",variable names,
arrays,commas,function names

parentheses are used by functions, arrays, and grouping sub-expressions
* is multiplication
/ is division
+ is addition
- is subtraction
% is modulo

A string is encapsulated in double quotation marks '"' or single quotation
marks "'" at the beginning and end of the string. A string enclosed in double
quotation marks cannot have double quotation marks inside, single quotation
marks should be used.
Example of illegal string :
"this is "a" string"
Example of illegal string :
'this is 'a' string'
Example of legal string :
"this is a string"
Example of legal string :
'this is "a" string'

Function names, variable names and array names must start with an underscore
'_' chr(95) or an uppercase letter chr(65) to chr(90) or a lowercase letter
chr(97) to chr(122) followed by more uppercase letters, lowercase letters,
underscores, or numbers chr(48) to chr(57)

For numbers we look for ascii codes chr(48) to chr(57)
Note : numbers may be a float and have a decimal point so chr(46) should
be taken into account

All our tokens are stored as strings. To indicate what type of token
is stored we prefix each token with a code :
0 indicates a string value example : "text" is "0text"
1 indicates a real value example : 58.4 is "158.4"
2 indicates a variable name, or a function name, or an array name
  example : name1 is "2name1"
3 indicates operators, parentheses and commas (argument separators)
  example : ( is "3("
  another example : + is "3+"
*/

expression = 'function1(var1,var2,var3)+45.899-32+a/function2("a string")%(-82 - 5)* -75'

max_index = -1
single_char = "()*/+-%,"
number_char = "0123456789."
name_char1 = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_"
name_char2 = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_"
while (string_length(expression) > 0)
{
 char = string_char_at(expression,1)
 //error checking
 if string_count(char,single_char) = 0
 {
  if string_count(char,number_char) = 0
  {
   if string_count(char,name_char2) = 0
   {
    if char != " "
    {
     if char != '"'
     {
      if char != "'"
      {
       show_message("Error : Expression contains illegal characters.")
       exit
      }
     }
    }
   }
  }
 }
 //if it is a single token ()*/+-%,
 if string_count(char,single_char) > 0
 {
  max_index += 1
  token_array[max_index] = "3" + char
  expression = string_delete(expression,1,1)
 }
 //ignore space, do nothing
 if char = " "
 {
  expression = string_delete(expression,1,1)
 }
 //if a string using double quotes is found
 if char = '"'
 {
  pos = 1
  do
  {
   pos += 1
   if pos > string_length(expression)
   {
    show_message("Error : Illegal string.")
    exit
   }
   next_char = string_char_at(expression,pos)
  }
  until (next_char = '"')
  temp1 = string_copy(expression,1,pos)
  temp1 = string_replace_all(temp1,'"',"")
  max_index += 1
  token_array[max_index] = "0" + temp1
  expression = string_delete(expression,1,pos)
 }
 //if a string using single quotation marks
 if char = "'"
 {
  pos = 1
  do
  {
   pos += 1
   if pos > string_length(expression)
   {
    show_message("Error : Illegal string.")
    exit
   }
   next_char = string_char_at(expression,pos)
  }
  until (next_char = "'")
  temp1 = string_copy(expression,1,pos)
  temp1 = string_replace_all(temp1,"'","")
  max_index += 1
  token_array[max_index] = "0" + temp1
  expression = string_delete(expression,1,pos)
 }
 //if it is a number
 if string_count(char,number_char) > 0
 {
  pos = 1
  do
  {
   pos += 1
   if pos > string_length(expression)
   {
    pos -= 1
    break
   }
   next_char = string_char_at(expression,pos)
  }
  until (string_count(next_char,number_char) = 0)
  if string_count(next_char,number_char) > 0
  {
   temp1 = string_copy(expression,1,pos)
   expression = string_delete(expression,1,pos)
  }
  else
  {
   temp1 = string_copy(expression,1,pos-1)
   expression = string_delete(expression,1,pos-1)
  }
  if string_count(".",temp1) > 1
  {
   show_message("Error : Illegal real number.")
   exit
  }
  max_index += 1
  token_array[max_index] = "1" + temp1
 }
 //if it is a function/variable/array name
 if string_count(char,name_char1) > 0
 {
  pos = 1
  do
  {
   pos += 1
   if pos > string_length(expression)
   {
    pos -= 1
    break
   }
   next_char = string_char_at(expression,pos)
  }
  until (string_count(next_char,name_char2) = 0)
  if string_count(next_char,name_char2) > 0
  {
   temp1 = string_copy(expression,1,pos)
   expression = string_delete(expression,1,pos)
  }
  else
  {
   temp1 = string_copy(expression,1,pos-1)
   expression = string_delete(expression,1,pos-1)
  }
  max_index += 1
  token_array[max_index] = "2" + temp1
 }
}

//debugging
temp1 = 'function1(var1,var2,var3)+45.899-32+a/function2("a string")%(-82 - 5)* -75' + '#'
for (count = 0; count <= max_index; count += 1)
{
 temp1 += token_array[count] + "   "
}
draw_clear(c_black)
draw_set_font(font0)
draw_set_color(c_white)
draw_text_ext(0,0,temp1,-1,600)
set_automatic_draw(0)

Pretty neat dll. Too bad I can't find any concrete info on how to convert an infix expression to a postfix expression taking into account functions, variables and arrays. Then evaluate that postifx expression as a result. If I knew that I'm pretty sure I could make an interpreted language. :huh:

The Wikipedia articles only hint on how to go about doing this :
http://en.wikipedia....-yard_algorithm
http://en.wikipedia....Polish_notation

Edited by attm, 19 September 2010 - 05:14 AM.

  • 0

#49 Unistyle

Unistyle

    GMC Member

  • New Member
  • 4 posts

Posted 09 June 2012 - 11:32 AM

Is it possible to make a fast explode function with it?
Cause i search for a faster way, then with the normal gm Functions.

I m working on a map editor which saves a map file with 11 information per instance.
The map file will be loaded by opening the file, splitting the information in it and use it.
But already a map of 5000 x 5000 full with instances freezes the game for 5 sec when the text_file splitting.

Currently I use this script:

/*
**  Usage:
**      size = explode_string(array,sep,data);
**
**  Arguments:
**      array       name of a local array, string
**      sep         seperator character, string
**      data        array data, string
**
**  Returns:
**      size        size of the array of data
**
**  Notes:
**      Converts a string of data with elements seperated
**      by a delimiter into an array of strings.
**
**  GMLscripts.com
*/
{
    var arr,sep,dat,len,ind,pos;
    arr = argument0;
    sep = argument1;
    dat = argument2 + sep;
    len = string_length(sep);
    ind = 0;
    repeat (string_count(sep,dat)) {
        pos = string_pos(sep,dat)-1;
        variable_local_array_set(arr,ind,string_copy(dat,1,pos));
        dat = string_delete(dat,1,pos+len);
        ind += 1;
    }
    return ind;
}

I hope i didnt fully misunderstand your dll.
  • 0

#50 TheMagicNumber

TheMagicNumber

    GMC Member

  • GMC Member
  • 5247 posts
  • Version:Unknown

Posted 10 June 2012 - 12:30 AM

Is it possible to make a fast explode function with it?

No. Something like that would always be slow in pure GML. That script doesn't look too good, try making your own which doesn't store values in an array or use string_delete.
  • 0