.ini file max size limit.

Discussions about Coding and Scripting

.ini file max size limit.

Postby Wildcard » Wed Jul 09, 2014 12:02 am

I am wondering if anyone has discovered the maximum allowed file size for .ini files in UT. I am working on a serverside mod right now that stores a ridiculous amount of data and I'm at a turning point in the design here and am wondering if I should use a bunch of redundant subclasses to achieve different .ini filenames and store my data separately or if it doesn't matter and I can cram as much data as I like into a single .ini file. I have not witnessed a limit yet I had .ini files as big as or more than 13 MB in size for some of the mods I have codded in the past. This mod is going to be a monster and will be using objects to store data in the .ini file and If I proceed with the design I will have a limitless amount of data I could store in theory because the objects have unique generated names so they have their own section in the .ini file. Only big problem I see is the immense amount of data this .ini file is going to hold and objects from time to time that are stored in the .ini file will be deleted and leave their data in the .ini file abandoned and that' a lot of garbage that's going to pile up so I plan on writing some kind of program that will manually clean out abandoned entries from time to time. I really should quit rambling and ask my question.

Is there any known limit to how much data can be stored in an .ini file? any other devs hit it and noticed some weird stuff happening? I'm not talking about the amount of data in a variable that can be written correctly to an .ini file.

Just curious, Thanks
- nOs*Wildcard
Image
User avatar
Wildcard
Average
 
Posts: 46
Joined: Fri Sep 09, 2011 9:06 pm
Personal rank: Admin Wizard

Re: .ini file max size limit.

Postby Wises » Wed Jul 09, 2014 12:23 am

Not sure about .ini file size but i have hit the limits in terms of Actor/Packages a couple of times though.
Can't remeber the limit or if it's only a problem with Pure itself.
...
User avatar
Wises
Godlike
 
Posts: 1087
Joined: Sun Sep 07, 2008 10:59 am
Personal rank: ILLEGAL!

Re: .ini file max size limit.

Postby Chamberly » Wed Jul 09, 2014 12:54 am

I remember someone saying something about there is some max limit within Server Actor and Server Packages (not enough details from here), but anything else I've never heard. I guess you can add many random texts in the .ini and see when it crash but it can take awful long time to upload/download (for slow internet users) and quite a bit of time to open with notepad. lol

I guess it might be a good idea to make separate .ini file and let the mod work dynamically with that. Just to be safe than running into a lot of trouble of bugs and screw up maybe.

Good luck on your big request. :)
Image
Image
irc.globalgamers.net #uscript
http://irc.lc/globalgamers/uscript
Image
User avatar
Chamberly
Godlike
 
Posts: 1502
Joined: Sat Sep 17, 2011 4:32 pm
Location: Tennessee, USA
Personal rank: Dame. Vandora

Re: .ini file max size limit.

Postby JackGriffin » Wed Jul 09, 2014 1:04 am

I thought much as you did but I spoke to Helen about it and he is using a RE-DIC-U-LOUS-LY large array in his DM server. He had a neat way of parsing it though that would very much interest you. I don't want to say more on that because it's not my work but it would be worth your time to see if he's still active and willing to share. He's a good guy, very smart, I just don't know if he's around.

The only caveat I might add on monster-sized ini files is reading/writing to them is mostly OK if you do it right but there will be real problems if you have to reorder them on the fly or read all the lines for comparison (like for "leading scorer" or "most kills"). If you are planning that then you'd be MUCH better served with hooking to a PHP database and letting that parse your data for you then return whatever the server needed. If you don't and you let this go on during a game it's gonna lag your game..
Image

On almost all applications you'd be better served letting it write to an external DB. That's what those things were written for while UEngine was never designed for any sort of massive reading/writing of files (and it really shows).
Relocating to Cherno.... Friendly! Friendly! <BLAM BLAM>....grrrr....
JackGriffin
Godlike
 
Posts: 3045
Joined: Fri Jan 14, 2011 1:53 pm
Personal rank: Hack coder

Re: .ini file max size limit.

Postby UT99.org » Wed Jul 09, 2014 7:09 am

billybill wrote:^ What he said. Your ram will take the brunt of it, and when accessing it your CPU as well.

There's still some unknowns about ram and caching with UT, if it were a listenserver then the cached database would get consume more and more ram as you switch maps or restart the round. Thankfully this is only a problem with clients and not servers AFAIK
UT99.org
 

Re: .ini file max size limit.

Postby TheDane » Wed Jul 09, 2014 9:00 am

Well, UT is a 32bit game if I'm correct? So there's your upper theoretical limit, but that limit includes everything loaded into the RAM.

About .ini files. I think people have lost focus on what an .ini file is? A .ini file is simply just values for the variables it represent, now what does that mean? Well, it Means that every variable in your mod has a value, when you make the variable configurable you add the option to assign the initial value using a .ini file. So, the size of your .ini file will be determined by the number of variables and their assigned values. String variables take up more length than a byte variable for instance (unless the string variables value is set to "" lol). So you have absolutely no limitations to the size of you ini file in any way as long as your mod is functioning and not overloading the server itself by cycling through all the variables all the time.

Now the .ini file can cause severe lag during gameplay if you write to it during gameplay - that's a no-go unless it's a very small file. You never "read" from it during gameplay because it is only loaded into memory once when the map begins (prior to pre-beginplay). And every variable that has it's value read or changes during gameplay will not affect the .ini file itself as it will be read from memory and not the file. If you want to update the .ini file itself to reflect any changes made, you can only do that by calling the SaveConfig() function - and then the server will lag for the time it takes to write the file to the harddrive.

Except for the writing part the values of the .ini file has no other limitations but the limitations of just another regular variable value in the Uengine. It doesn't take any longer to iterate through the config variables than through any other variables - Just handle with care, know what it is, and your quite safe :tu:

I always use the HandleEndGame() function to save my config files. It will still lag, but then it won't affect the gameplay. And if you delay it a few seconds once HandleEndGame is called, then it won't lag much unless it's a veeeeeery huge file :mrgreen:
Retired UT fan.
User avatar
TheDane
Masterful
 
Posts: 535
Joined: Tue Feb 12, 2008 2:47 pm
Personal rank: Offline.....

Re: .ini file max size limit.

Postby Wildcard » Wed Jul 09, 2014 10:52 am

Here is an .ini file generated by my mod that shows the sheer amount of insanity I have planned.
http://gauntletwarriors.com/Wildcard/Ma ... le0000.zip

I notice things start lagging down once I get lots of stuff written to the .ini file but I think that could be due to all the objects I have floating around waiting to be garbage collected or something IDK.

I appreciate all the feedback on this thread as of now to me this is an interesting subject.
Perhaps sometime I can go more into details on what exactly my mod does but you may figure that out if you look at the .ini file
So much work to be done. Test 170
Image
User avatar
Wildcard
Average
 
Posts: 46
Joined: Fri Sep 09, 2011 9:06 pm
Personal rank: Admin Wizard

Re: .ini file max size limit.

Postby Sp0ngeb0b » Wed Jul 09, 2014 12:08 pm

I've been working extensively with these "ridiculous" sized config arrays in my Nexgen Player Lookup plugin. It's basically a complete player information DB in UScript, consisting of 5 string arrays with 25.000 slots each, all in one class. As the others already stated, there's no limit for the number of lines in the .ini file (as you might have summed up already, mine is 125.000 entries large), it's working as it's supposed to work.
I can confirm what Dane said: avoid the SaveConfig() call during the game, do it on game-end. In my mod I didn't browse through the database during the game (e.g. whenever a player joined), instead, I saved the new data in a temporary array and did the whole updating before saving it on gameend.

What's gonna be interesting for you, and what the others havent pointed out yet, gave me good headache for some time. It turned out that: the bigger my arrays, the higher the chance of a player timing out on the initial connection try (= they never joined the server, only "Connection failed" pops up after some time). I didn't quite get it, since the actor containing the large data was never spawned clientside. Because the Mod requires some client-side stuff in another class, I had to add it as a ServerPackage, resulting in every client loading the complete package into RAM while connecting to the server. And well yeah, large arrays seem to need some time to load, which a few clients apparently couldn't handle :lol2: I got that under control by moving the whole config arrays to a Serverside only package, so you might consider this while developing your mod.

And alllow me to also give my 2 cents on that whole DB Uscript thing: Of course, I totally agree that an external SQL database in combination with a php script is way more efficient than the UScript solution. Nonetheless, my opinion is that you can still do A LOT of DB stuff in UT without noticing the lack of efficiency, if it's done right. Todays serverhardware proofed to handle a complete iteration through these five 25.000 entries arrays in a single tick, without any noticeable lag or performance drop. Just be carefull and always keep in mind how many iterations per tick your action will take - if they are to many, split the whole thing up over several ticks.
******************************************************************************
NexgenAdvancedBanManager || NexgenPlayerLookupSystem || NexgenTeamspeak3Connector
******************************************************************************


UnrealRiderS UT4: www.unrealriders.de
User avatar
Sp0ngeb0b
Adept
 
Posts: 279
Joined: Wed Feb 13, 2008 9:16 pm
Location: Cologne

Re: .ini file max size limit.

Postby JackGriffin » Thu Jul 10, 2014 3:09 am

Sp0ngeb0b wrote:Todays serverhardware proofed to handle a complete iteration through these five 25.000 entries arrays in a single tick, without any noticeable lag or performance drop.

I'd challenge that coming from someone else but if you say it's so then I'll just shake me head in astonishment. Wow.

I spent about 6 months coding and running servers for Dayz (Epoch flavor) under the Arma engine. Literally everything that goes on in the DayZ server is a function of the database and data is transferred *constantly* between the running server and the DB. It's a very efficient way to have things although the learning curve can be steep in the beginning. If I ever worked in UScript again I'd certainly try to mimic that model to ease the load on the running server. A lot of the data the server is having to keep track of could be offloaded to a DB if you could streamline the fetching protocol good enough.
Relocating to Cherno.... Friendly! Friendly! <BLAM BLAM>....grrrr....
JackGriffin
Godlike
 
Posts: 3045
Joined: Fri Jan 14, 2011 1:53 pm
Personal rank: Hack coder

Re: .ini file max size limit.

Postby Dr.Flay » Thu Jul 10, 2014 3:55 am

I would be most worried about the large txt files being corrupted if the game crashes while writing.
User avatar
Dr.Flay
Site Staff
 
Posts: 2971
Joined: Thu Aug 04, 2011 9:26 pm
Location: Kernow, UK
Personal rank: Chaos Evangelist

Re: .ini file max size limit.

Postby Sp0ngeb0b » Thu Jul 10, 2014 9:50 am

Sp0ngeb0b wrote:Todays serverhardware proofed to handle a complete iteration through these five 25.000 entries arrays in a single tick, without any noticeable lag or performance drop.

JackGriffin wrote:I'd challenge that coming from someone else but if you say it's so then I'll just shake me head in astonishment. Wow.


For some clarification: In my mod, I tried to prevent these situations where it has to cycle through the complete DB in one tick. All searches for example are splitted up over several ticks, since they require additional, performance eating operations (each string has to be desplitted first). Nonetheless, there's still one operation which I kept in the mod which occurs during gameplay, and, giving the right conditions, will iterate 25.000 times in one tick. It compares the first 32 characters of each string with a given search mask. It's not the most performance eating stuff, but it's still something. I just checked it again online (since I wrote my above statement from memory), and providing the worst case-condition, the only difference network-wise I could see was a ping jump of about 10 when performing that operation. Unfortunately, I couldn't check it's impact on the actual gameplay right now. I could imagine it's noticeable at least a little bit, but I hope you get my point: If you are careful and think about which operations you perform when and in how many ticks, you can get along quite well with a UScript DB :)

Interesting stuff with the DayZ server Jack, indeed, an implementation with UT's TCP link is thinkable. Although I wonder how good it will handle the constant sending of data :ironic:
******************************************************************************
NexgenAdvancedBanManager || NexgenPlayerLookupSystem || NexgenTeamspeak3Connector
******************************************************************************


UnrealRiderS UT4: www.unrealriders.de
User avatar
Sp0ngeb0b
Adept
 
Posts: 279
Joined: Wed Feb 13, 2008 9:16 pm
Location: Cologne

Re: .ini file max size limit.

Postby Higor » Fri Jul 11, 2014 2:00 am

You can always create a variant of the Markers, where instead of specifying a single location you get an array of 8.
Could be autoparsed as follows:


Example 1:
Three monsters
- Spawn marker, set info.
- Spawn marker, set info.
- Spawn marker, set info.

Example 2:
Eight Monsters
- Spawn marker, set info.
- Spawn marker, set info.
- Spawn marker, set info.
- Spawn marker, set info.
- Spawn marker, set info.
- Spawn marker, set info.
- Spawn marker, set info.
- Spawn marker, set info.
- Spawn group marker, set the 'Group' name = to markers 'Group'.

In the loading process, the second case will only have one ini entry with 8 different location arrays.
In the saving process, markers that have been included in a group will not save.

Optimized saving process on markers:
- Chain all Group markers as they're created via PostBeginPlay() into a master actor (to hold the list, like the PawnList in LevelInfo), doing so will chain the markers either when adding them or loading them.
- Design in said master actor a function that loops the list for finding a suitable group.
Code: Select all
FindMarker(string GroupName, class<Actor> ActorClass, vector OtherLocation)
{
   local GroupMarker G;
   For ( G=GroupList ; G!=none ; G=G.nextGroup )
        if ( !G.IsFull() && (G.GroupName == GroupName) &&  (G.ActorClass == ActorClass) && (VSize(G.MyLocation - OtherLocation) < G.DetectionRadius) )
               return G;
}
- During save, have each individual marker call FindMarker() on the master actor to see if they belong to a group.
- If they belong to a group, register into said group's temporary marker list, and do not save.

Group code:
Code: Select all
- var class<actor> ActorClass;
- var Marker tmpMarkers[8];
- var config string GroupName;
- var config float DetectionRadius;
- var config vector MyLocation, SavedLoc[8];
- var config string PropList[8];
- var config rotator SavedRot[8];
- var config byte Count;
- var GroupMarker nextMarker;
- var MasterActor Master;

PostLoad( MasterActor Other) <<//Called my master
>> Master = Other.
>> Chain self to master's chained list
>> if Count > 0, then i was loaded, have master spawn my markers and set marker properties
>> Set Count to 0, it will re-count when saving

IsFull()
>> return Count >= 8

AttachMarker( Marker Other) //Called by marker after determining it belongs to group marker
>> Assert( Count < 8 ); //Crash the game if we're calling this on a full marker, best debugging HUEHUE (and we get a trace log)
>> tmpMarkers[Count] = Other;
>> SavedLoc[Count++] = Other.Location;

SaveData()
>> SaveConfig()
>> Clear tmpMarker array
>> Set Count to 0


That will economize up to 3xArraysize - 1 (count) lines, if our array is 8 we're saving 23 lines in a place full of markers.
Will also allow you to shrink the Members[] array.

PD: Are actors being spawned to indicate the existance and position of these data markers?
Higor
Godlike
 
Posts: 1463
Joined: Sun Mar 04, 2012 6:47 pm


Return to Coding, Scripting

Who is online

Users browsing this forum: No registered users and 1 guest