Open Surge Forum

A fun 2D retro platformer inspired by Sonic games and a game creation system

You are not logged in.

Announcement

Our community has moved to Discord! https://discord.gg/w8JqM7m

#1 2011-07-02 13:00:45

KZR
Member
Registered: 2010-07-14
Posts: 1,447
Website

[unconfirmed] Level size limit ?

Engine Rev: 548

This is how it happened:

I was building a level, saved it and reloaded it. But it didn't reload, it only crashed the game. weird, huh? The log file didn't even know what it was. Ah, computers... so, the game restarts with no problem, but now that level crashes the game every time it is loaded. Aw! On to level selection and... Level selection crashes the game as well.

This is how I solved it:

Any text editor loads the level just fine, and there were no order/syntax mistakes, after all, the editor wrote this file. so why doesn't it load in-game? I figured it could have been some of the last changes, so i sorted bricks by number (with notepad++ textfx sort), and deleted some i knew that were passable and would not affect solid constructions. After looking at the level properties, the file size was almost 1000 KB large. When the file size dropped to < 960 KB it loaded again in-game. Built some more and the game crashed again when the level reloaded. File size was now 968 KB. So I went on again with sorting bricks and deleted some more passables. File size: 943KB , It loads again!

What I concluded:

Level size limit = ~~960 KB

For some reason the engine can't handle bigger levels (memory leak?)

Can someone please confirm?

Last edited by KZR (2011-07-02 13:01:32)


SD_sml.pngSeD_sml.pngLTot_W_sml.png
https://discord.gg/w8JqM7m ---> Open Surge's Discord server

Offline

#2 2011-07-02 13:41:32

Sugarmaster
Member
From: Barcelona, Spain
Registered: 2011-04-26
Posts: 132
Website

Re: [unconfirmed] Level size limit ?

KZR wrote:

Level size limit = ~~960 KB

For some reason the engine can't handle bigger levels (memory leak?)

Can someone please confirm?

Confirmed KZR, I got a fresh new latest nigthly builds 564, opened template.lev, copied the text of the bricks over and over till i reached ~900k filesize and game engine crashes when opening that lvl. Afterwards I erased  repetead brick text to reach ~ 800k filesize and could play the level, but performance was sluggish.


---
The other thing to have into account is that a I use Windows 7, to play the game at full speed I have to execute debugens.exe (a modified version of the gens emulator) and then the game plays perfect. But this time, even with debugens opened, the game performance was slow.

I tried different configurations with opensurge.exe, compatibility with XP, deactivate themes, etc... but I just could get the game to run smoothly using debugens along with it. I think that's because debugens deactivates things from directx to make emulation run better.


"The true master is always learning"
Avatar by Raul Sama.

Offline

#3 2011-07-02 16:35:00

Alexandre
Administrator
From: Brazil
Registered: 2009-01-27
Posts: 3,300
Website

Re: [unconfirmed] Level size limit ?

KZR wrote:

Level selection crashes the game as well.

the engine should never crash. you guys gotta help me out.

delete the big level for a moment. now, create a new object (in a new file) like this:

object bug_test {
    requires 0.2.0

    state main {
        show
        show
        show
        show
        ...
        show
    }
}

copy and paste the show command until the file reaches 900 KB. Did it work? Increase it to 1MB. Does it crash? When?

Sugarmaster wrote:

Afterwards I erased  repetead brick text to reach ~ 800k filesize and could play the level, but performance was sluggish.

if you copy and paste 10 million bricks on the same spot, whenever the camera gets close to that location, the engine will have to process 10 million bricks. so, this is expected.

Offline

#4 2011-07-03 20:57:33

KZR
Member
Registered: 2010-07-14
Posts: 1,447
Website

Re: [unconfirmed] Level size limit ?

object size: 6MB
game crashes right after the first loading screen

object size: 3MB
same effect

object size: 1MB
same effect

object size: 814KB
same effect


object size: 330KB
same effect


object size: 147KB
GAME LOADS!


SD_sml.pngSeD_sml.pngLTot_W_sml.png
https://discord.gg/w8JqM7m ---> Open Surge's Discord server

Offline

#5 2011-07-03 21:10:11

Alexandre
Administrator
From: Brazil
Registered: 2009-01-27
Posts: 3,300
Website

Re: [unconfirmed] Level size limit ?

KZR wrote:

object size: 330KB
same effect

Can you paste the log?

Offline

#6 2011-07-03 21:26:34

KZR
Member
Registered: 2010-07-14
Posts: 1,447
Website

Re: [unconfirmed] Level size limit ?

logfile stops with

Loading objects scripts...

no error message of any kind

http://dl.dropbox.com/u/20216605/logfile.zip


SD_sml.pngSeD_sml.pngLTot_W_sml.png
https://discord.gg/w8JqM7m ---> Open Surge's Discord server

Offline

#7 2011-07-03 21:57:23

Alexandre
Administrator
From: Brazil
Registered: 2009-01-27
Posts: 3,300
Website

Re: [unconfirmed] Level size limit ?

I can't reproduce that on linux, so I need help.

i saw that you put the big object in dropbox, so let's enfore some syntax error and figure out where the problem is.

I have modified one line of the obj to

{show

. that line is on the beginning of the object. Does the game crash? If not, what happens?

Next, fix that line changing it to

show

again and pick a line on a few lines ahead and switch it to

{show

. Does it crash? If not, what happens?

Repeat this until it crashes. What's the logfile before it crashes? And after? What's the exact line number that makes it crash?

Offline

#8 2011-07-10 01:05:55

KZR
Member
Registered: 2010-07-14
Posts: 1,447
Website

Re: [unconfirmed] Level size limit ?

i've been experimenting a bit more, and i found out that it is the fact that a large brick is now composed of smaller ones grouped together that causes this issue. we now need 16 lines to describe a 64x64 group, when we could use just one and describe it as a whole brick.
But then there's another issue: Memory management vs physics accuracy: Large angled bricks are more prone to collision errors/glitches. Smaller bricks need more CPU time/memory.
So what i conclude out of all this, is that bricksets need a balance between RBE-ready small bricks, and manually added large bricks. For solid walkable surfaces, 16*16 bricks are excellent, as on the other hand they are bad for covering large surfaces. That's where the manually added bricks come into play. not only they cover large surfaces, they require less CPU to scroll and less memory for that task.

I think we don't need to run any more tests on this. This procedure will not only virtually eliminate the crash possibility, it will also allow building levels almost 16 times larger (within the size limit).


SD_sml.pngSeD_sml.pngLTot_W_sml.png
https://discord.gg/w8JqM7m ---> Open Surge's Discord server

Offline

#9 2011-07-10 03:58:15

Alexandre
Administrator
From: Brazil
Registered: 2009-01-27
Posts: 3,300
Website

Re: [unconfirmed] Level size limit ?

KZR wrote:

i've been experimenting a bit more, and i found out that it is the fact that a large brick is now composed of smaller ones grouped together that causes this issue. we now need 16 lines to describe a 64x64 group, when we could use just one and describe it as a whole brick.
But then there's another issue: Memory management vs physics accuracy: Large angled bricks are more prone to collision errors/glitches. Smaller bricks need more CPU time/memory.
So what i conclude out of all this, is that bricksets need a balance between RBE-ready small bricks, and manually added large bricks. For solid walkable surfaces, 16*16 bricks are excellent, as on the other hand they are bad for covering large surfaces. That's where the manually added bricks come into play. not only they cover large surfaces, they require less CPU to scroll and less memory for that task.

I think we don't need to run any more tests on this. This procedure will not only virtually eliminate the crash possibility, it will also allow building levels almost 16 times larger (within the size limit).

Thanks for experimenting with it. smile

That's a wiser approach, thought it implies on having to add a few bricks manually to the script. Use RBE for walkable platforms, and manual editing for the rest.

Nonetheless, the crash is still hiding, and I have no idea of what it is... hmm yet.

Offline

#10 2011-07-10 15:19:09

KZR
Member
Registered: 2010-07-14
Posts: 1,447
Website

Re: [unconfirmed] Level size limit ?

I'm not experient enough in coding to have a notion of exactly where the problem is, but i'm guessing:

Levels:

*the engine loads the level and lists the position of every brick, item and object, allocating memory for their coordinates. this memory has a limit.
*there is a limit to how many bricks a level can load.

Objects:

*There is a memory limit for each object
*objects have a limited number of commands they can run

Global:

*File size limits.


Still, there is no reason why the level selection should crash when a very large level is listed. I know the headers are read for level title and act. does it read more of the file? because if it does, it may be the fact of loading a level which size is over the limit.

Last edited by KZR (2011-07-10 15:20:07)


SD_sml.pngSeD_sml.pngLTot_W_sml.png
https://discord.gg/w8JqM7m ---> Open Surge's Discord server

Offline

#11 2011-08-01 01:23:23

SilverstepP
Member
From: North Carolina
Registered: 2009-07-31
Posts: 1,545

Re: [unconfirmed] Level size limit ?

I can confirm this bug exists, Wrath showed me a (non-test) level that was around 939 kb and it does the same thing even though it worked just fine before a certain save.

Notes:

-Even with no objects or items, the level will crash if it's too big. We deleted every item and object.
-Anything above 900 kb is now pretty dangerous.

So this crashing bug is completely unrelated to objects. But still, why does it happen? When decreased to 839kb, it works.

Offline

#12 2011-08-01 13:17:56

KZR
Member
Registered: 2010-07-14
Posts: 1,447
Website

Re: [unconfirmed] Level size limit ?

i am guessing it's either an Allegro limitation or something in the engine. For now, building only with 16*16 bricks carries this risk even if the level is not that big in-game. until this is solved, i strongly advise people to use large passables.
one trick to use RBE to help with large bricks is to mark "passable" in the top left corner of the large brick, and taking a note of real size and brick number.

once the brickset is exported, you only need to search for the brick numbers you noted and add the right dimensions.

Last edited by KZR (2011-08-01 13:18:14)


SD_sml.pngSeD_sml.pngLTot_W_sml.png
https://discord.gg/w8JqM7m ---> Open Surge's Discord server

Offline

#13 2011-08-02 00:24:21

Alexandre
Administrator
From: Brazil
Registered: 2009-01-27
Posts: 3,300
Website

Re: [unconfirmed] Level size limit ?

KZR wrote:

For now, building only with 16*16 bricks carries this risk even if the level is not that big in-game. until this is solved, i strongly advise people to use large passables.

this would be wise even if no crash existed. You could just save the passables in some other .png file and script these bricks by hand, calling them (for example) brick 9000, 9001, 9002, etc. Not only this leads to smaller .lev files, but IMO it also makes level design easier. then you just use 16x16 bricks when you need accurate collision detection.

Race has been making some really awesome passable bricks for Sunshine Paradise, and I'm considering to use this method instead: manual scripting for passables and RBE for solid blocks, with solids and passables in different .png files.

Offline

#14 2011-08-02 00:29:45

SilverstepP
Member
From: North Carolina
Registered: 2009-07-31
Posts: 1,545

Re: [unconfirmed] Level size limit ?

But still, what's the bug here?

Offline

#15 2011-08-02 00:54:57

KZR
Member
Registered: 2010-07-14
Posts: 1,447
Website

Re: [unconfirmed] Level size limit ?

Alexandre wrote:

this would be wise even if no crash existed. You could just save the passables in some other .png file and script these bricks by hand, calling them (for example) brick 9000, 9001, 9002, etc. Not only this leads to smaller .lev files, but IMO it also makes level design easier. then you just use 16x16 bricks when you need accurate collision detection.

I can also add that using passables in green/yellow layers to simulate different zindex can be used with no risk at all. SD uses this technique extensively and all tests returned good results. Solids are a different thing and should be used "by the rules". This saves time when adding bricks manually, eliminating many copy-paste-edit cycles.


SD_sml.pngSeD_sml.pngLTot_W_sml.png
https://discord.gg/w8JqM7m ---> Open Surge's Discord server

Offline

#16 2011-08-02 14:03:35

Alexandre
Administrator
From: Brazil
Registered: 2009-01-27
Posts: 3,300
Website

Re: [unconfirmed] Level size limit ?

KZR wrote:

I can also add that using passables in green/yellow layers to simulate different zindex can be used with no risk at all. SD uses this technique extensively and all tests returned good results. Solids are a different thing and should be used "by the rules". This saves time when adding bricks manually, eliminating many copy-paste-edit cycles.

well, all right. smile

SilverstepP wrote:

But still, what's the bug here?

that requires a longer explanation.

I have a hypothesis. I haven't had enough time yet to prove myself right or wrong on this, but I have strong beliefs that my hypothesis is correct.

When reading any script files (.lev, .brk, .obj, etc.), we use an algorithm to identify the underlying grammar of the text, so our engine can "understand" what is written in there. This is called a recursive descendant parser, which uses mutual recursive functions to identify important elements in the text. In a computer program, there is something in the memory called execution stack. Each function call in the algorithm increases the execution stack a tiny bit. However, since the execution stack has a maximum limit, after about a million descendant function calls (1M .lev file), the stack gets crushed.

Having this complex algorithm is very important for brk, spr or obj files, which have some hierarchy associated to them (example: an script has many objects, an object has its properties and many states, states have many commands...). On the other hand, it may not be so important for .lev files, since they have no complex hierarchy, but top-level commands - a bunch of lines one after the other. We could just use a different algorithm for .lev files, without bothering with grammars and stuff.

As I already wrote here, as I'm finishing my undergraduate studies, I'm really short on free time, and this situation won't change for some months. Therefore, we have two approaches:

1. Replace the algorithm for reading the .lev files. I can`t guarantee that it will work, but I can say that with a very good probability, if we do that the bug will be fixed. That should take some days. However, if we do so, I would like that you guys created Sunshine Paradise soon, with SurgeChris` brickset, Race`s non-solid blocks and with no water for act 1 (no custom objects for water, please, due to: 1. efficiency on slow hardware and 2. accuracy).

2. Or we can use my tiny free time to develop water into the engine. I don't know how long that will take, but I can say that if you choose this approach, the bug will continue on the engine until we finish water.

I'm letting you guys choose. What do you think?

Offline

#17 2011-08-02 14:49:43

Sugarmaster
Member
From: Barcelona, Spain
Registered: 2011-04-26
Posts: 132
Website

Re: [unconfirmed] Level size limit ?

I think the water can be done via scripting so there is no need to bother doing it right now (maybe in the future, when you and the community have more free time).


Someone should be able to create some "generic" water object for the engine (there are some examples in the object section that could fit if they were readapted) and just put that object as default or a bundled object with the engine (like lady bugsy).

I would cast a vote for the algorithm thing because it's something that only you can do at the moment.


"The true master is always learning"
Avatar by Raul Sama.

Offline

#18 2011-08-02 15:08:07

SilverstepP
Member
From: North Carolina
Registered: 2009-07-31
Posts: 1,545

Re: [unconfirmed] Level size limit ?

Water is a feature...

But this is a bug.

In my opinion, bugfixes should come first. So I say we do this.

Offline

#19 2011-08-02 18:46:00

KZR
Member
Registered: 2010-07-14
Posts: 1,447
Website

Re: [unconfirmed] Level size limit ?

Bugfix! Silver's awesome water from MM can be used as a temporary replacement with virtually no lag added, since it uses a lot less objects that the "water blocks" approach.


SD_sml.pngSeD_sml.pngLTot_W_sml.png
https://discord.gg/w8JqM7m ---> Open Surge's Discord server

Offline

#20 2011-08-02 19:56:17

SilverstepP
Member
From: North Carolina
Registered: 2009-07-31
Posts: 1,545

Re: [unconfirmed] Level size limit ?

No. Just don't put water in.

My objects don't have water counters for Surge and company. Sorry...

However.

I have some water slide/waterfall objects that could be used. No lag or anything of the sort. But I'll need time to make them work for Surge and friends.

Offline

#21 2011-08-04 18:54:22

Alexandre
Administrator
From: Brazil
Registered: 2009-01-27
Posts: 3,300
Website

Re: [unconfirmed] Level size limit ?

all right.

bugfix it is. smile

Offline

#22 2011-08-11 22:41:32

SilverstepP
Member
From: North Carolina
Registered: 2009-07-31
Posts: 1,545

Re: [unconfirmed] Level size limit ?

So, any progress?

Offline

#23 2011-08-18 22:13:43

Alexandre
Administrator
From: Brazil
Registered: 2009-01-27
Posts: 3,300
Website

Re: [unconfirmed] Level size limit ?

sorry it took so long. should be fixed on 578.

i haven't tested it too much, so please backup the executable just in case.

Offline

Board footer

Powered by FluxBB  hosted by tuxfamily.org