Jump to content


Photo

Gm Database System v10.3


  • Please log in to reply
86 replies to this topic

#1 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 10 August 2009 - 07:08 AM

This extension package made exclusively in GML adds support for a new data structure in GM: tables. Tables are very similar to lists (which are built-in in GM). The major difference is that the items they contain consist of multiple fields.

Download (v10.3): http://www.benetonfi...emp/gmdb103.zip
Help File: http://www.benetonfi...mdb103_doc.html
Changes Log: http://www.benetonfi...mdb_changes.txt

NOTE: This extension was written for GM 8.1 but it should be compatible with GM: Studio. I am planning on making a new version specifically for GM: Studio in the future.


Edited by benetonmovie, 14 June 2013 - 04:19 PM.

  • 2

#2 sabriath

sabriath

    12013

  • GMC Member
  • 3197 posts

Posted 10 August 2009 - 08:22 AM

Uhm...wouldn't it be simpler to just make it a double array?

Instead of a table with first name, last name and age like in the example, it would be:

first_name = 0;
last_name = 1;
age = 2;

table[0,first_name] = "James";
table[0,last_name] = "Beethoven";
table[0,age] = 32;
Just saying, I am not seeing any differences between a table that couldn't be done in a double array that is built-in to GM. Although some of the functionalities (such as sort and shuffle) would be nice to have for both single AND double arrays, so if you can pull those out of the table functions and package them as a library for use with regular arrays, that would be great! (for others anyway, I don't need them, I use class structures in GM :P ).

Good luck!
  • 0

Tutorials of Interest:
* Multiplayer: mine or True Valhalla's

Projects:
* Net39
* My 39dll scripts
* My 39dll lib
* Multiplayer Engine
* Artificial Chemistry

Posted Image

#3 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 10 August 2009 - 06:29 PM

No, tables are much more powerful than what can be done with arrays. You can delete and insert entries at given positions and others are automatically pushed (just like lists). Also it makes things much more structured and easier to work with (in my opinion). Thanks for the reply!
  • 0

#4 sabriath

sabriath

    12013

  • GMC Member
  • 3197 posts

Posted 17 August 2009 - 07:57 AM

No, tables are much more powerful than what can be done with arrays. You can delete and insert entries at given positions and others are automatically pushed (just like lists). Also it makes things much more structured and easier to work with (in my opinion). Thanks for the reply!

Anything that can be done with a table can be done with arrays.

//'scr_new_list()'
ar_max = -1;

//'scr_add(elem)'
ar_max += 1;
ar_elem[ar_max] = argument0;

//'scr_insertafter(ind, elem)'
for (i=argument0; i<ar_max; i+=1)
{
  ar_elem[i+1] = ar_elem[i];
}
ar_elem[argument0+1] = argument1;
if (argument0 == ar_max) ar_max += 1;

//'scr_delete(ind)'
for (i=argument0; i<ar_max; i+=1)
{
  ar_elem[i] = ar_elem[i+1];
}
ar_max -= 1;

Then again, I could turn an array into a class member if I felt like it:

//'scr_class_init()'
class_first = -1;
class_last = -1;
class_max = -1;

//'scr_class_new(elems)'
i = class_max + 4;
class_max += argument0 + 3;
class_arr[i] = class_last;
class_arr[i+1] = -1;
class_arr[i+2] = argument0;
if (class_first == -1)
{
  class_first = i;
}
else
{
  class_arr[class_last+1] = i;
}
class_last = i;
return i;

//'scr_class_remove(class_ind)'
i = class_arr[argument0];
j = class_arr[argument1+1];
if (i == -1)
{
  class_first = j;
}
else
{
  class_arr[i+1] = j;
}
if (j == -1)
{
  class_last = i;
}
else
{
  class_arr[j] = i;
}

Of course this is a severely dubbed down version (doesn't even show garbage collection). I can do this with pure strings as well :) Let me know if you need any help though, I'm good at helping.
  • 0

Tutorials of Interest:
* Multiplayer: mine or True Valhalla's

Projects:
* Net39
* My 39dll scripts
* My 39dll lib
* Multiplayer Engine
* Artificial Chemistry

Posted Image

#5 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 17 August 2009 - 11:16 AM

Of course, I know it can be done somehow... This extension only uses GML, by the way. But I made it to keep it simple for everyone (myself included) and fast. It uses lists, which are (quoting the GM manual) "implemented using simple arrays but, as this is done in compiled code it is a lot faster than using an array yourself".
  • 0

#6 gmlctl

gmlctl

    GMC Member

  • GMC Member
  • 47 posts
  • Version:Unknown

Posted 24 September 2009 - 08:25 AM

Looking good. Keep up the good work!
  • 0

#7 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 21 October 2009 - 01:56 AM

Thanks! Any suggestion, anyone?
  • 0

#8 meatspider

meatspider

    GMC Member

  • New Member
  • 3 posts

Posted 11 November 2009 - 12:23 AM

i know this is a complete noob question. to use a .gex file i just put it in the extensions folder? (never used one before) i try and it gives me a cannot find extension package gm database syste, when i try to load the example.
  • 0

#9 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 11 November 2009 - 02:21 AM

Please look at this tutorial: http://gmc.yoyogames...howtopic=280300
  • 0

#10 meatspider

meatspider

    GMC Member

  • New Member
  • 3 posts

Posted 11 November 2009 - 10:40 AM

Awesome! thanks. i tried the extensions selection before but didnt see the new one listed so gave up. thanks again!
  • 0

#11 cuetzpalin

cuetzpalin

    GMC Member

  • New Member
  • 8 posts

Posted 22 November 2009 - 07:19 AM

Is it possible to sort entries based on more than one column?
  • 0

#12 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 24 November 2009 - 05:39 AM

No, not at the moment, sorry. I added it to my to-do list! :)
  • 0

#13 SimpleMugen

SimpleMugen

    GMC Member

  • New Member
  • 10 posts

Posted 26 November 2009 - 01:14 PM

nice job! :)
  • 0

#14 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 26 November 2009 - 08:12 PM

Thank you very much for the nice reply! Hoping you're finding it useful!
  • 0

#15 SimpleMugen

SimpleMugen

    GMC Member

  • New Member
  • 10 posts

Posted 26 November 2009 - 11:25 PM

you didn't add one usefull action - select row by 2 or more variables...
i made one, here it is:
/*
argument0 - your table
argument1 - column1
argument2 - column2
argument3 - column3
argument4 - variable that must = column1
argument5 - variable that must = column2
argument6 - variable that must = column3
usage:
SCRIPT_NAME(table id, column 1, column 2, column 3, var = column 1, var = column 2, var = column 3)
in this case script will delete searched entry
*/
var _list,_pos;
_list=ds_list_create()
_pos=db_entry_get_pos_all(argument0,argument1, argument4, _list)
for(i=0;i<ds_list_size(_list);i+=1)
{
	if(argument5=db_entry_get_value(argument0,argument2,ds_list_find_value(_list,i)))
	{
		if(argument6=db_entry_get_value(argument0,argument3,ds_list_find_value(_list,i)))
		{
		//If all ok, return:
		show_message(string('ok on step: '+string(i)+'#(count from zero)'))
		//delete a right entry (where argument1=argument4, argument2=argument5 and argument3=argument6)
		db_entry_delete(argument0,ds_list_find_value(_list,i))
		ds_list_clear(_list)
		exit;
		}
	}
}
hope you add this function in your new release of gex :) (certainly more flexible)
  • 0

#16 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 27 November 2009 - 12:43 AM

Nice script SimpleMugen! However there's already a way to check for more than 1 column, using the function db_entry_get_pos_ext() like this:

db_entry_get_pos_ext(table id, "column1=variable1,column2=variable2,column3=variab
le3
", 0, list, ",");


This function saves in list (must be an existing ds_list ID) all the entries that correspond to the given conditions.

Edited by benetonmovie, 27 November 2009 - 12:45 AM.

  • 0

#17 SimpleMugen

SimpleMugen

    GMC Member

  • New Member
  • 10 posts

Posted 27 November 2009 - 02:43 AM

omg :)
thanks a lot! your gex is great :)
  • 0

#18 brod

brod

    Brian RODriguez

  • GMC Member
  • 2050 posts
  • Version:GM8

Posted 27 November 2009 - 02:50 AM

Ahhh.. I can't thank you enough!! I was trying to create my own database editor, and it was a hell of an annoyance! But this... this is so incredible and simple, thank you!

A suggestion, though.
db_column_get_name/type/default()

Would make your example better, for example your example can do
var exe;
exe = "db_entry_add(" +string(table)+ ", " +string(db_column_count(table))+ ", ";
for(a=0;a<db_column_count(table);a+=1)
	{
	if(db_column_get_type(table, a)==0)
		{
		exe+= string(get_integer(db_column_get_name(table, a),db_column_get_default(table, a)));
		}
	else
		{
		exe+= '"'+get_string(db_column_get_name(table, a),db_column_get_default(table, a))+'"';
		}
	if(a<db_column_count(table)-1)exe+=", ";
	else exe+=")";
}
execute_string(exe);

EDIT: Never mind, they seem to already be in there :)
I think you should make them function-colored or whatever, they're very useful :)

I'm not even gonna bother making a new editor, I'm just going to use yours right now cause it's so user-friendly! THANKS AGAIN!

EDIT2: There seems to be a bug where decimal values won't be saved. :s

Edited by brod, 27 November 2009 - 05:07 AM.

  • 0

#19 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 27 November 2009 - 06:01 AM

A suggestion, though.
db_column_get_name/type/default()

Hmm, that shouldn't be too hard to add, but I remember deciding not to do it because you're normally supposed to know what they are. I guess if you want to give the user some control on the database structure... but then why do you need that?


EDIT2: There seems to be a bug where decimal values won't be saved. :s

Please tell me how to reproduce the bug! Thanks!

Edited by benetonmovie, 27 November 2009 - 06:07 AM.

  • 0

#20 brod

brod

    Brian RODriguez

  • GMC Member
  • 2050 posts
  • Version:GM8

Posted 27 November 2009 - 06:04 AM

Hmm, that shouldn't be too hard to add, but I remember deciding not to do it because you're normally supposed to know what they are. I guess if you want to give the user some control on the database structure... but then why do you need that?

Well in my case, I'm trying to make a complete engine where I can make tables on the fly using the exe, and not having to go into the gmk changing everything. Also, that code snippet I posted above is a good example on how to achieve that. I'm sure that other people will try to do the same thing, so it wouldn't hurt to add.

Please tell me how to reproduce the bug! Thanks!

I just put a decimal in get_integer(), and for some reason it wouldn't show up on the table.
never mind, I know what I've done wrong.

Edited by brod, 27 November 2009 - 06:10 AM.

  • 0

#21 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 27 November 2009 - 06:10 AM

Well in my case, I'm trying to make a complete engine where I can make tables on the fly using the exe, and not having to go into the gmk changing everything. Also, that code snippet I posted above is a good example on how to achieve that. I'm sure that other people will try to do the same thing, so it wouldn't hurt to add.

You're right. Well as you said the functions already exist, they're just not "public" (since I thought they would just be used internally) but they will be in the next version. :)
  • 0

#22 brod

brod

    Brian RODriguez

  • GMC Member
  • 2050 posts
  • Version:GM8

Posted 27 November 2009 - 06:25 AM

Um, a few problems I've encountered while editing.

db_entry_count(); has 3 arguments, but doesn't/shouldn't it return how many entries are in a single column? (So it should be (table id, column) I guess)

Also, I got this error

___________________________________________
ERROR in
action number 1
of Draw Event
for object obj_example:

In extension function db_entry_count:
In extension function db_entry_get_pos_all:
Error in code at line 14:
for (n = 0; n < ds_list_size(global.db_list_entries[argument0, colpos]); n += 1)

at position 65: Negative array index

I assume the error is a result of this code:
//draw the values
	for (m = 0; m < db_entry_count(table, n-1, 1); m += 1)
		{
		if (n == 0)
			{
			value = string(m);
			}
		else
			{
			value = db_entry_get_pos(table, n-1, m);
			}
		//if this entry is selected, change the color
		if (selectedrow == m)
			{
			draw_set_color(c_orange);
			if(selectedcol+1 == n && selectedcol>-1)draw_set_color(c_yellow);
			}
		draw_text(20 + n * 110, 76 + m * 22, value);
		draw_set_color(c_white);
		}

Edited by brod, 27 November 2009 - 06:29 AM.

  • 0

#23 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 27 November 2009 - 02:51 PM

As the helpfile says, the db_entry_count() function

Counts the number of entries where the value of the given column corresponds to the given value. Returns the number of entries found.


If you want to count the number of entries in a table (it will be the same for every column), use db_table_get_size().

Also can I please have your complete script so I know what the values of n are in this piece of code? Thank you!
  • 0

#24 brod

brod

    Brian RODriguez

  • GMC Member
  • 2050 posts
  • Version:GM8

Posted 27 November 2009 - 03:09 PM

Ah, that worked! Silly me :whistle:

I am having a very frustrating problem though, for some reason, this keeps on returning the default value when I try to put in a decimal value..
if (menu == 0)
		{
		//add entry
		var exe;
		exe = "db_entry_add(" +string(table)+ ", " +string(db_column_count(table))+ ", ";
		for(a=0;a<db_column_count(table);a+=1)
			{
			if(db_column_get_type(table, a)==0)
				{
				exe+= string(get_integer(db_column_get_name(table, a), 0)); //always return 0 when I put in any decimal value
				}
			else
				{
				exe+= '"'+get_string(db_column_get_name(table, a),db_column_get_default(table, a))+'"';
				}
			if(a<db_column_count(table)-1)exe+=", ";
			else exe+=")";
		}
		execute_string(exe);
		}

Edited by brod, 27 November 2009 - 03:10 PM.

  • 0

#25 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 27 November 2009 - 08:23 PM

Hey Brod, seems to be with the get_integer function... It doesn't take decimals! :whistle:
  • 0

#26 brod

brod

    Brian RODriguez

  • GMC Member
  • 2050 posts
  • Version:GM8

Posted 27 November 2009 - 11:07 PM

Posted Image

damn. Well that ruins my plans. Guess I'll have to come up with something else.

Edited by brod, 28 November 2009 - 12:13 AM.

  • 0

#27 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 27 November 2009 - 11:26 PM

Well there are other ways... for instance you could use get_string and convert the string to a real. Of course that would need some error-checking but it isn't hard. :whistle:
  • 0

#28 brod

brod

    Brian RODriguez

  • GMC Member
  • 2050 posts
  • Version:GM8

Posted 28 November 2009 - 12:14 AM

Yeah I'll just do something easier. (I was trying to make a "% per use", but instead I'll just make it "Uses to master". Same difference)
  • 0

#29 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 28 November 2009 - 01:26 AM

Lol okay. Funny how you edited your post after I replied to it.
  • 0

#30 brod

brod

    Brian RODriguez

  • GMC Member
  • 2050 posts
  • Version:GM8

Posted 28 November 2009 - 02:09 AM

Had to add the image :whistle:
But yeah, I'm done bickering and annoying you, thanks again for the awesome extension, GREAT IDEA!
  • 0

#31 Recreate

Recreate

    Furry

  • GMC Member
  • 2992 posts
  • Version:GM:Studio

Posted 28 November 2009 - 04:39 AM

Well. I would just use .php files to manipulate my database with netread, But this is still neat. :whistle:
  • 0

If the post that you are reading was created prior to 2011. For the safety of the general public, It is not to be regarded under any circumstances.
Please don't ask me to join your group at anything.


#32 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 28 November 2009 - 07:43 AM

Thanks!

I have the pleasure to announce the release of VERSION 10! :whistle: See the change log below and download on first post!

NEW in version 10.0
- Major change in the columns system to enhance flexibility: columns now have ID's, which you can save as variables to use in function calls (you can still use the column name directly, but not the position)
- New functions db_column_insert(), db_column_destroy(), many new db_column_get_* functions
- Ability to sort by more than one column with the new db_table_sort_multiple() function
- Improved the example, now has drag & drop and the ability to add, insert and destroy columns
- Solved a bug in the shuffle function, that prevented it from working under certain conditions
- Added arguments to change the size of the drawing in db_table_draw()
- Changed a bit the bahevior of the function db_table_read()
- Optimized a few scripts
- Shorter and cleaner argument names in the script editor help panel
  • 0

#33 brod

brod

    Brian RODriguez

  • GMC Member
  • 2050 posts
  • Version:GM8

Posted 28 November 2009 - 02:56 PM

GREAT!

Edited by brod, 28 November 2009 - 03:23 PM.

  • 0

#34 Universal_X

Universal_X

    GMC Member

  • New Member
  • 126 posts

Posted 07 December 2009 - 12:28 PM

This looks great!

But could you put in the table data in a .ini file?
  • 0
ѬѯҨ Cities of the Future

#35 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 07 December 2009 - 11:34 PM

Of course you can, by transforming the table into a string with the function db_table_write().
  • 0

#36 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 26 December 2009 - 05:49 AM

I would like to say that this extension is 100% compatible with GM8 (at least from a quick test & review). :snitch:
  • 0

#37 Rafferty97

Rafferty97

    GMC Member

  • New Member
  • 268 posts

Posted 10 January 2010 - 02:43 AM

cool, the gex looks very promising. I will probabily use it when I do a project requiring databases. Just from looking at this topic, I can give it 8/10.

One thing though, is it fast?

- Alex

EDIT: Awesome, compatible with GM8 ;)!

Edited by Rafferty97, 10 January 2010 - 02:45 AM.

  • 0



Want a place on the internet to put your files? Get 40mb free at Host-a.net
Gm member since 2007

#38 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 10 January 2010 - 03:29 PM

One thing though, is it fast?

Speed shouldn't be a problem!
  • 0

#39 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 14 March 2010 - 09:22 PM

Gentle bump... Any suggestions for future updates?
  • 0

#40 Antonio_moder

Antonio_moder

    GMC Member

  • New Member
  • 17 posts

Posted 05 April 2010 - 09:51 AM

Thank you very much benetonmovie ^_^
  • 0

#41 darkelfv

darkelfv

    GMC Member

  • New Member
  • 28 posts

Posted 02 May 2010 - 07:15 AM

This is awesome, Thanks for making this. I plan on using it for the rougelike game I'm working on.
  • 0

#42 Kolink

Kolink

    Almighty Shinx

  • GMC Member
  • 2351 posts

Posted 02 May 2010 - 09:26 AM

Just curious: wouldn't it be better to use a ds_grid?
  • 0

Posted Image


#43 darkelfv

darkelfv

    GMC Member

  • New Member
  • 28 posts

Posted 03 May 2010 - 04:59 PM

I just realized one thing that may hinder my development. How is the memory of the stored data used, Cleared, etc...

For example. A db structure is so we can have many data entries that are file based that means we make queries for the data and it quickly returns what we need with out having the entire thing in memory.

You have support for import export which is fine, but when it comes down to it, if i have to load the entire db file into memory to access the 3 lines i need then this isn't going to be any more efficient then keeping everything in memory.

If that is true i could break up the tables into multiple files, which would be ok. However if it takes a long time to load large tables into memory this would break the program as well. The question then is when you load a table into the game does it destroy delete the memory of the old one, assuming your using the same variable for the table? If i load 5 tables in a row, how do i go about getting rid of the old table data from memory. Do i have to call db_table_destroy(id) each time and create a new one when importing a table? Or is this handled automatically when i import to the same table with structure.

Finally my suggestion is to make a couple of calls that allow you to grab queried data into a new table that reads from a file instead of loading the table into memory then grabbing the data you want and deleting the data. I think this would complete your system and allow for minimal memory usage with large db tables.

Thanks
  • 0

#44 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 04 May 2010 - 12:31 PM

darkelfv, your reply is confusing me a bit. I don't think any table would be too big to be kept in memory (especially these days).

You have support for import export which is fine, but when it comes down to it, if i have to load the entire db file into memory to access the 3 lines i need then this isn't going to be any more efficient then keeping everything in memory.

Well normally you would just import all your tables at the beginning of the game/program, and export them before closing (if you want to save the data, that is). I can't tell you how "efficient" it is but I use GM's ds_list_read() function to read the data of every column, and it seems pretty fast.

The question then is when you load a table into the game does it destroy delete the memory of the old one, assuming your using the same variable for the table? If i load 5 tables in a row, how do i go about getting rid of the old table data from memory. Do i have to call db_table_destroy(id) each time and create a new one when importing a table? Or is this handled automatically when i import to the same table with structure.

It's handled automatically. No useless data is kept in memory.

Finally my suggestion is to make a couple of calls that allow you to grab queried data into a new table that reads from a file instead of loading the table into memory then grabbing the data you want and deleting the data. I think this would complete your system and allow for minimal memory usage with large db tables.

This is where I really don't follow you, sorry. You want everything to be stored directly in files instead of memory? :P
  • 0

#45 Schyler

Schyler

    Noskcirderf Derf

  • GMC Member
  • 2680 posts
  • Version:GM:Studio

Posted 04 May 2010 - 12:35 PM

Any chance you may support SQL, for us PHP junkies?
  • 0

#46 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 04 May 2010 - 02:38 PM

Any chance you may support SQL, for us PHP junkies?

Haha, well it would be a completely different thing. My extension is really for offline tables, and has nothing to do with SQL. Sorry, I have no idea how to do that, let alone do it securely.
  • 0

#47 darkelfv

darkelfv

    GMC Member

  • New Member
  • 28 posts

Posted 04 May 2010 - 05:04 PM

The game I'm making will expand into a large world where every mob, item, object (including trees walls) will be independently recorded into the db. This also does not include the 1000 year history I'm creating per game which could be many entries per year.

Loading from a file lets me query data i need and only load that into memory, streaming in the land, objects, object details, mobs, mob equipment, mob current status, hp stats, afflictions, bonuses.

For instance if i need to grab a time period between 200-300 of the 1000 year history, loading the entire thing alone would take a while. Assuming a bare minimum of 3 races, with a minimum of 24 history events per year each. That is 72000 records at the least. That is just one of my tables as well, the history table will have to tie into sub tables that have details on some of the records which could double or triple the amount of records. This is of course per game.

Think of a mmo - except the mobs remember how much health they have. Also if i cut a tree down it will be recorded. If a mob kills me and i make a new character. 10 years will pass and i will be able to find and kill the mob that slain my old character.

I am also doing a detailed event history of every move you make in the game to help generate a detailed story book of your adventure once you complete the game.

There is also alot of data going to be saved for main quest and sub quest generation.

I'm used to working with sql tables, or mdb access tables, File based tables allow you to store lots of records, but trying to load them all at once is not really optimal. Maybe i'm the only one that is trying to make a massive game, but the whole point of having a db imo is so you don't have to have all the data in memory at the same time.

The other reason is i intend to try to make this work on a psp at some point, They haven't released it yet but they are working on it. So making the game as fast and optimized as possible is important. Idk if gex will be supported by psp but it would be cool to get it to work on psp either way. So my target is low memory use.

I can break up my db tables into separate files, but it would be easier to directly get targeted data from a single table though.

I wonder how much 72000 records of string data would take up in memory...?

Thanks

I will do some tests when i have some time and get the full history set working. I also forgot to reply saying that i would not be loading data strictly from files, there would be some streaming and depending on where the character was and what quest he is on, there would be many tables loaded into memory. So not everything would be just from files, its more of just using what is needed in memory at the time. I think a lot of the hard core table creation / reading would be during the generation process and this is already taking a while to make the landmasses, so optimizing this is really my goal with out using alot of memory.

Edited by darkelfv, 04 May 2010 - 05:12 PM.

  • 0

#48 darkelfv

darkelfv

    GMC Member

  • New Member
  • 28 posts

Posted 05 May 2010 - 04:28 AM

Well I did some testing and I think my worries have been resolved. While I will be using files to store the tables and I will break them up, I don't see as much of a issue as i first excepted. Keeping tables around 10,000 records or less shouldn't be a issue.

Creating a table with 72000 records took about 15 seconds and took about 20 megs of memory (including standard gamemaker program)
A 10 entry was about 11 megs of memory , basically the base size for a game maker game.

Interestingly enough loading took about 20 seconds and was around 30 megs

The table was a single string table with the string
"This is a test event string for checking how much memory a string record takes"

When exported the file was around 13 megs

The program was a bit unresponsive, however i think this was because the drawing function in the step event that tries to draw the table each frame. So not a bid deal.

So while it would be neat to have a function grab a result directly from a file, its not really required.

Thanks again for making this awesome tool.
  • 0

#49 benoit.rouleau

benoit.rouleau

    GMC Member

  • GMC Member
  • 1596 posts
  • Version:GM:Studio

Posted 06 May 2010 - 01:52 PM

darkelfv, your project is very interesting! I'm glad you're using my database system. :P

So while it would be neat to have a function grab a result directly from a file, its not really required.

Well, I'm always open to suggestions, but I don't see how such a function would work. Can you explain how it would be called, what would be the arguments, etc. and I could try working on it?


Thanks again for making this awesome tool.

You're very welcome! ::lmao::
  • 0

#50 darkelfv

darkelfv

    GMC Member

  • New Member
  • 28 posts

Posted 06 May 2010 - 04:55 PM

I would do a function similar to the one you already have.

db_entry_get_pos_ext(table id, "column1=variable1,column2=variable2,column3=v
ariable3", 0, list, ",");

Table id would be file name.
Instead of list it would be a table.
It would first load the structure of the file. Then search for the columns where the requirements matched.
Then it would load the entire row where it found a match into the new table.

*Notes and possible problems. Part of the reason file based db's work this way is because they have some sort of indexed system so they can do quick searches for records with out having to go through them one by one.

Secondly > < (greater then, less then) are supported by the queries. So i could say where Column1 > 100 and column1 < 200

Thinking about it now, that would be a lot of work on top of what you already have. Not to mention i don't know your current file structure but it seems like a encrypted ini file or some sort of bin file. So indexing it, figureing out how to do that on top of having some sort of greater then less then would probably not be practical for you to add.

As far as some sort of indexing strategy i suppose having something like this might work.

Column index section - list of all columns pointing to a seconded section that has sorted column data
Sorted column data index - Sorted list of data that is easy to search through and contains the pos of the record.

At the front of the file you might have a list of the columns with a index (pos) of each of the secondary column data indexs.
These secondary index records would be the the data with the first 5 characters of a string or all of the real numbers sorted From Low to high. Then it would also contain the pos of the real row of data.

So you could search for column2 (a real, say 100) even if the data contained 1000's of other rows. It would search the beginning of the file find that column2 index is stored at file pos x, then looking at the secondary index it would find the starting data is 1, it could skip 100 records and maybe some are missing so it finds record data 105 then searches backwards till it finds 100 then grabs the record pos and returns that entire column of data from the real table.

Like i said it would be more complex. Real data would be stored as a file byte position at the start of the file as well. For where the real table data started.

Here is a example of what im talking about to try to clearify it.

Column index
Column Name sorted alphabeticly , File posistion of column index , Type
Column1 , byte 10 , real
Column2 , byte 20 , string
Column3 , byte 30 , string

Column1 index data (sorted), row posistion
5 , 1
10, 3
15, 6
20, 2
25, 4
30, 5
35, 7

Column2 index data (sorted), row posistion
a , 3
b, 1
c, 4
d, 2
e, 6
f, 5
g, 7

Column3 index data (sorted), row posistion
bike , 2
carrot, 7
jump, 3
plant, 4
quick, 6
van, 5
zebra, 1

Then the real data would be following
Pos, Column1 , column2, column3
1 , 5 , b, zebra
2 , 20, d, bike
3 , 10, a, jump
4 , 25, c, plant
5 , 30, f, van
6 , 15, e, quick
7 , 35, g, carrot
So if i wanted column1 data 30 it would know it was on row 5 and could return column1 column2 and column3 only into a new table.
pos 5 , 30, f, van

Having ranges is important to minimize how many times you would call from a table.
Also maybe a search string like column3 contains "a" which would return 1 4 5 and 7
advanced contains would be like contains "ump" which would contain 3

These are just my thoughts on the matter as i do not know exactly how they are done in the professional world.
  • 0




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users