[WIP] Node UTStats 1.12C

Discussions about UT99
User avatar
UT Sniper (SJA94)
Inhuman
Posts: 755
Joined: Thu Jun 24, 2010 10:35 pm
Personal rank: Retard
Location: England

[WIP] Node UTStats 1.12C

Post by UT Sniper (SJA94) »

Node UTStats is a full remake of the old utstats system using the same UTStats mutators, coded in javascript and using more updated mysql compared to the original.

Online example: https://stats.xjon.me/

New build released: https://github.com/scottadkin/Node-UTSt ... /tag/1.12C

ReadME: https://github.com/scottadkin/Node-UTStats

New in 12C:
- Added support for 4CTF screenshots.
- Fixed map screenshot on matches page to always use the non prefix over gametype prefix names.

Image

New in 12B:
- Changed how map images are displayed, instead of just looking for an image name without a gametype prefix the system will now look for a file with the gametype prefix, if that is not found it will look for an image with the gametype prefix, and if that also fails it will display the default image.
- Changed how the recent matches display is rendered.

New this build:

Nexgen stats viewer tool, now you can easily configure how your UT server will display the data from your website, you are not restricted to just ranking points, there are many different categories you can use for any gametype.

Image

Image

Image

Supported gametypes:
- Deathmatch
- Team Deathmatch
- Capture the Flag
- Domination
- Last man standing
- Bunnytrack
- Monsterhunt
- Siege (basic support)
- Coop

Map Importer added
Every map file in your /Maps/ folder will be imported and saved with the option set to true.

Nexgen Stats viewer support added
From version beta 8 there is now support for nexgen stats viewer by using the ulr example.com:1337/nexgenstats

Ace Log+Sshot importer
You can now import ace logs and screenshots to the website.

Bunnytrack BTPlusPlus.ini Importer added
You can now import your server bunnytrack records into your nodeUtstats website by running node btimport from the server module directory.

Map screenshot packs
Place the image files in /public/files/maps on the website module.
- Default Deathmatch maps

Possible problems:
- .tmp files sometimes show errors while importing but are moved successfully.
- First import may get stuck if mapimport is set to true.

ChangeLog(From what I can remember):
Build 12 release notes(14/04/20):

- Added player name to page description and open graph data, for individual player match report.
- Moved player profile link on individual player match report to the top.
- Fixed empty pickup data being displayed if there were no pickup events during match.
- Added server hostname to server query page. (Thanks to fbraz3)
- Added files to a perfect webserver setup(Linux). (Thanks to fbraz3)
- Added Nexgen stats viewer tool to admin area.

- Build 11 release notes(09/04/20)
- The Website, Importer and Mutator are now combined into one package, you no longer need to install the server/importer module to each ut server now just add the server ftp details to the ftpServers array in Importer/api/config.js.
- Added FTP support, you can now have an array of servers for the importer to download the logs from the utservers.
- Added ACE logs importer to ftp.
- Added ACE screenshot importer to ftp.
- Added BTPlusPlus.ini importer to ftp.
- Added BTGame.ini importer to ftp.
- Ace screenshots are now moved after import to Logs/Imported/Ace-Shots/<month>-<year>.
- BT record inis are now backed up daily via the latest import of that day to BT/Imported/<day>-<month>-<year> after being imported.
- Already Imported .log/.jpg/.tmp files will not be downloaded from ftp servers.
- Imported files from utservers are now backed up by default to another local folder to speed up the later imports(.log, .tmp, .jpg). If bDeleteFilesFromFTP is set to true they will be deleted locally.
- Fixed crash while trying to import bt records.
- Fixed headshots import.
- Fixed team kills for non team games.
- Fixed some logs failing to import after a bt log.
- Fixed server MOTD lines not importing.
- Fixed player pickups count being wrong if they reconnect.
- Fixed player best spree & multi not resetting after death. (I'm an idiot :) )
- Fixed grabbed and capped player being counted as assists in the flag captures area.
- Fixed javascript error trying to access teams that don't exist for the kill graphs.
- Fixed headshots not being merged if a player reconnects.
- Fixed broken flag images on records page.
- Fixed broken flag images on player page under possible aliases.
- Fixed error message displayed if a map has had 0 matches played.
- Fixed crash on website on players page when searching by first match.
- Fixed domination point graph showing team colours that were not in the match.
- Fixed Serbia flag being missing from package.
- Added BTGame.ini records import, now both BTPlusPlus.ini and BTGame.ini records are imported via the btimport.js on the server module.
- Added upgrade.js to website module that admins can run instead of making a fresh database from upgrading from version 9 and above.
- Added a warning message if a log is a large size, so users know the difference between a crash/infinite loop and a large file.
- Added mapList import, if you run "node mapimport" in the server module it will import every single map in your servers ./Maps/ folder. It will also set the file size of maps that have been imported from logs.
- Added kill distance to the mutator, now the distance between kills are saved to the log.
- Added shortest kill distance to matches and profiles for every player.
- Added longest kill distance to matches and profiles for every player.
- Added flag base locations to the mutator, now the flag base locations are saved to the log.
- Added spawn point locations to the mutator.
- Added weapon locations to the mutator.
- Added pickup locations to the mutator.
- Added Domination point locations to the mutator.
- Added shortest and longest time between kills to the mutator.
- Added player spawn locations to mutator.
- Added extended flag kill data logging, now killer, victim, kill distance, distance to cap, and distance to base are now saved.
- Added filename to match database, this will be used to delete duplicates from the admin panel.
- Added extended flag kills information to match reports.
- Added extended kill data to match pages, once the button is clicked the page will load all kill data that includes time, killer, victim, distance, weapon.
- Added clickable headers to match sections, now you can link directly to the area you want to share.
- Added headshots to frag performance if there was a headshot in the match.
- Added player record imports to btimport, now players will have their records saved to their pages.
- Added map bt records to map pages.
- Updated style of pickup summary on match pages.
- Updated style of player serach page, removed ugly search form and replaced it with clickable arrows next to table headers to specify sort order.
- Updated style of maps serach page, added clickable arrows next to table headers to specify sort order instead of stupid dropdown box.
- Merged Best multi kills and sprees in match pages to their respected areas instead of having separate areas.
- Added match player individual stats, now if you click on a players name in a match report it will take you to their match performance page with extended data displayed.
- Added player specific kills and deaths to match player individual stats.
- Added player Killing Sprees Information to match player individual stats, this will show how long the spree was in kills, time, and how the spree ended.
- Added clickable player match links to match weapon stats.
- Added clickable player match links to domination point capture summary.
- Added clickable player match links to pickups summary.
- Added clickable player match links to ranking summary.
- Added clickable player match links to team change summary.
- Added clickable player match links to player connections summary.
- Match pickup data now only displays players that have picked up 1 or more items to save scrolling time.
- Updated style of match assault captured objectives.
- Removed player connection history from match pages.



Original Post:
Spoiler
The last few days I've been working on a new utstats system that will use nodeJS instead of php, one thing I've been struggling to make my mind up on is whether I'm going to do one complete package, or do a server module, combined with a website module.

Server module:
This will handle all the log parsing, database updates, file sorting/managament. Basically the backend of the system.
The idea is you can have this running on multiple servers, that connect to the website module. Note: This methods will not use ftp.

Website module:
The website the user can interact with that all connected servers send their data to, will include admin control panel.



Would this just be unnecessary, or would it be more convenient?


Notes of what has been done so far:

Log Importer
- Scans the logs directory for .log files, ignoring .tmp files(I can make an options that will allow the admin to import these files, but advanced stats for stats_player events won't be there).
- NUTStats keeps track of all imported files, if the file has already been imported it will be ignored to prevent duplicates.

TODO:
- Let the admin choose what happens to compiled logs, this could be deleting, moving, compressing and moving.
Last edited by UT Sniper (SJA94) on Thu Jul 22, 2021 6:18 pm, edited 55 times in total.
User avatar
Darkelarious
Skilled
Posts: 177
Joined: Sat Feb 08, 2014 12:02 pm
Personal rank: 333networks admin
Location: Phobos Moon

Re: nodeJS UTStats

Post by Darkelarious »

UT Sniper (SJA94) wrote:Would this just be unnecessary, or would it be more convenient?
This sounds a lot like the interfaces I wrote for 333networks and the masterserver.

333networks hosts (several) masterserver(s) and stores the servers in a database. Our website can access the database and display server information (close to) real-time.

We also have a jQuery frontend in combination with a json API with our masterserver database.
http://333networks.com/json --> json API documentation

Working example:
http://ubrowser.333networks.com/ut/

Is this the kind of thing you have in mind?
--Darkelarious
Image
Masterserver | Discord | Donate
User avatar
Shrimp
Adept
Posts: 299
Joined: Wed Oct 10, 2018 11:15 am
Location: Australia

Re: nodeJS UTStats

Post by Shrimp »

It's definitely a better architecture to have a separate backend/processing service, versus a combined website/backend thing.

Depending on how you want to do things, you might even consider breaking the backend into two services as well:

- import/submission/processing: this is where the logs are pushed, parsed, and written to the DB

- view/query/api service: this reads from the DB the processing service writes to, and is what the frontend connects to

Allows you to scale things (har har, not like we need to "scale" UT services for more than 5 hits a week, but whatever, its fun to design these things anyway) separately, and you can also do things like redeploy or improve the parsing stuff without breaking the website, and vice versa.

Some modern tooling would definitely be cool :thuup:
ShrimpWorks
Unreal Archive - preserving over 25 years of user-created content for the Unreal series!
User avatar
UT Sniper (SJA94)
Inhuman
Posts: 755
Joined: Thu Jun 24, 2010 10:35 pm
Personal rank: Retard
Location: England

Re: nodeJS UTStats

Post by UT Sniper (SJA94) »

Darkelarious wrote:
UT Sniper (SJA94) wrote:Would this just be unnecessary, or would it be more convenient?
This sounds a lot like the interfaces I wrote for 333networks and the masterserver.

333networks hosts (several) masterserver(s) and stores the servers in a database. Our website can access the database and display server information (close to) real-time.

We also have a jQuery frontend in combination with a json API with our masterserver database.
http://333networks.com/json --> json API documentation

Working example:
http://ubrowser.333networks.com/ut/

Is this the kind of thing you have in mind?
Yeah that's what I have in mind. It's nice to see someone else who likes to recreate UT look for webpages 8)

Nexgen system for my INI site:
Image


One of many projects I start but never finish..
Image



Shrimp wrote:It's definitely a better architecture to have a separate backend/processing service, versus a combined website/backend thing.

Depending on how you want to do things, you might even consider breaking the backend into two services as well:

- import/submission/processing: this is where the logs are pushed, parsed, and written to the DB

- view/query/api service: this reads from the DB the processing service writes to, and is what the frontend connects to

Allows you to scale things (har har, not like we need to "scale" UT services for more than 5 hits a week, but whatever, its fun to design these things anyway) separately, and you can also do things like redeploy or improve the parsing stuff without breaking the website, and vice versa.

Some modern tooling would definitely be cool :thuup:
What I was also thinking of doing is was a module that is compatible with the current utstats database, so people can still access old matches from previous versions of utstats.

I also want to make this in such a way other people can create and add their own modules to add support for custom game types if needed, as well as other stuff like I made for the current UTstats like this: https://github.com/scottadkin/UTstats-c ... ots-plugin. What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.
User avatar
esnesi
Godlike
Posts: 1038
Joined: Mon Aug 31, 2015 12:58 pm
Personal rank: Dialed in.

Re: nodeJS UTStats

Post by esnesi »

UT Sniper (SJA94) wrote:What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.
That sounds like a dream!
User avatar
UT Sniper (SJA94)
Inhuman
Posts: 755
Joined: Thu Jun 24, 2010 10:35 pm
Personal rank: Retard
Location: England

Re: nodeJS UTStats

Post by UT Sniper (SJA94) »

iSenSe wrote:
UT Sniper (SJA94) wrote:What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.
That sounds like a dream!
Very early video of what I mean, so far it only does kills... and me being me I forgot to set the player' team during import so I need to do that :loool: This is a ctf match, but because I forgot to set the players teams you have to imagine its a team scoreboard. There is no sounds yet or special events.

OY7gNHhvTa4




What I plan on doing is make it so you can cycle through the players via the canvas so you can hear their special events, as well as the match events like flag captures and so on.




-----------------------------------------------------------------------------edit------------------------------------------------

APuYSqUWtgw


-------------------------------------------------------------------------------edit----------------------------------------------------


g7Ubuz35_2o


-------------------------------------------------------------------------------edit--------------------------------------------------------


Done some more work on the replay system, system now keeps tracks of player connects/disconnects and team changes:

Sim56QEe3g4






----------------------------------------------------------edit----------------------------------------------------------------------------------

User can now cycle through all the players currently in the game, you will only hear the multi kills and spree events for the player you are currently spectating(I haven't added headshots yet). If another player gets a spree event you will hear the same sound as you do in game when another players gets a spree event.
Image


----------------------------------------------------efit---------------------------------

sSm2Lk8UMXc





---------------------------------------------------------edit--------------------------------------------

Early match report page, the players won't be like that colour wise, just did that for testing purposes.
Image
User avatar
UT Sniper (SJA94)
Inhuman
Posts: 755
Joined: Thu Jun 24, 2010 10:35 pm
Personal rank: Retard
Location: England

Re: nodeJS UTStats

Post by UT Sniper (SJA94) »

Just had a thought, would people be interested if I made a discord bot for my new utstats system.

Example of what I mean:
.stats player <playername>
<url to player page>
<player has played 237 games>
<last played 23 hours 4 minutes ago>
<link to last played game>
.stats gametype <gametype>
<url to gametype match history>
<2104 game played of <gametype>
<last played 21 minutes 5 seconds ago>
<link to last played game>

Sorry for double post, been editing last post last 3 days, wanted to ask in new post about a discord bot.




--------------------------------edit----------------------------------------

Dm scoreboard, doesn't have FPH and pings yet:
Image
Last edited by UT Sniper (SJA94) on Fri Mar 22, 2019 5:22 pm, edited 1 time in total.
User avatar
papercoffee
Godlike
Posts: 10507
Joined: Wed Jul 15, 2009 11:36 am
Personal rank: coffee addicted !!!
Location: Cologne, the city with the big cathedral.

Re: nodeJS UTStats

Post by papercoffee »

UT Sniper (SJA94) wrote: Sorry for double post, been editing last post last 3 days
That's ok ... this doesn't count as double post.
User avatar
UnrealGGecko
Godlike
Posts: 2961
Joined: Wed Feb 01, 2012 11:26 am
Personal rank: GEx the Gecko
Location: Kaunas, Lithuania

Re: nodeJS UTStats

Post by UnrealGGecko »

UT Sniper (SJA94) wrote:Just had a thought, would people be interested if I made a discord bot for my new utstats system.

Example of what I mean:
.stats player <playername>
<url to player page>
<player has played 237 games>
<last played 23 hours 4 minutes ago>
<link to last played game>
.stats gametype <gametype>
<url to gametype match history>
<2104 game played of <gametype>
<last played 21 minutes 5 seconds ago>
<link to last played game>
Considering there are some active clans that host servers and hage a Discord channel, I say there might be interest on this. Just gotta let them know.

Heck, I might try to put it in the ut99org Discord as well, if its easy to put in :mrgreen:
User avatar
UT Sniper (SJA94)
Inhuman
Posts: 755
Joined: Thu Jun 24, 2010 10:35 pm
Personal rank: Retard
Location: England

Re: nodeJS UTStats

Post by UT Sniper (SJA94) »

Just a preview video of a DM game being played at 6 seconds a second. I have made some changes to the server module which will enhance this feature more in the future which will includes pickup logging, which weapon the player killed the other player with, and also added support for smartCTF/DM which will save the players faces/netspeed/FOV which will later be implemented in SmartCTF/DM scoreboards.

The user interacting with the match replay can choose which player they want to view, which will effect what sounds will be played, hud messages, and so on. I haven't implemented pick up hud, you were killed by, as well as timers, and other stats for the gameplay modeimg.


WUqOUEZy4T4


Here is sshot at end of game from NUTStats:
Image

Here is a in game screenshot of the same match, the small difference in Oopers kills is me forgetting to making the player lose a point for suicides, atm it counts suicides as the player has killed someone.

Image







------------------------------------------edit------------------------------

System now automatically imports weapons from log files and save them to database(kill strings):

Image
User avatar
UT Sniper (SJA94)
Inhuman
Posts: 755
Joined: Thu Jun 24, 2010 10:35 pm
Personal rank: Retard
Location: England

Re: nodeJS UTStats

Post by UT Sniper (SJA94) »

Match replay now includes item pickups, which includes the sounds for that item when they are picked up. Updated gameplay HUD, as well as some other stuff for the replay system which you can see in this video:


5YV1xgEKRTg

























----------------------------------------edit------------------------------------------

- Added support for Assault.
- Added support for Domination.











--------------------------------edit------------------------

Image
User avatar
UT Sniper (SJA94)
Inhuman
Posts: 755
Joined: Thu Jun 24, 2010 10:35 pm
Personal rank: Retard
Location: England

Re: nodeJS UTStats

Post by UT Sniper (SJA94) »

Just a sneak peak of a match replay with the smartCTF scoreboard, also updated the "gameplay" HUD. Just need to add support for LMS and Assault to the match replay(won't take long), then I'll work on the design of the website.



Ignore the fact it's a CTF match with a DM image as the background, haven't added that functionality yet, I need to add import functions for map Images and faces.

wpTDN0rEL9Q



------------------------------edit-------------------------------------


Now supports DOM point captures for match replay:

Image

Also added LMS support

Image


Just need to do assault now.
User avatar
UT Sniper (SJA94)
Inhuman
Posts: 755
Joined: Thu Jun 24, 2010 10:35 pm
Personal rank: Retard
Location: England

Re: nodeJS UTStats

Post by UT Sniper (SJA94) »

Just a screenshot of a more updated homepage, was doing another project for someone so didn't put much time into nutstats for a while:

Image
User avatar
esnesi
Godlike
Posts: 1038
Joined: Mon Aug 31, 2015 12:58 pm
Personal rank: Dialed in.

Re: nodeJS UTStats

Post by esnesi »

This looks very awesome, and I am very interested in this for our server! :agree1:
User avatar
UT Sniper (SJA94)
Inhuman
Posts: 755
Joined: Thu Jun 24, 2010 10:35 pm
Personal rank: Retard
Location: England

Re: nodeJS UTStats

Post by UT Sniper (SJA94) »

Just a sshot of a page from top to bottom, I removed stuff via developer tools in browser so you get to see the whole design of a page. There are totals from both all time stats, and records, including more per category then is displayed for single matches:

Image






Callback hell :sad2: :

Code: Select all

mysql.query(killQuery,(err, result) =>{

            if(err) throw err;

            if(result.length > 0){

                for(let i = 0; i < result.length; i++){
                    this.allTimeRecords.kills.push(result[i]);
                }

                
            }

            mysql.query(deathsQuery, (err, result) =>{
                if(err) throw err;

                if(result.length > 0){
                    for(let i = 0; i < result.length; i++){
                        this.allTimeRecords.deaths.push(result[i]);
                    }
                }

                mysql.query(scoreQuery, (err, result) =>{
                    if(err) throw err;

                    if(result.length > 0){
                        for(let i = 0; i < result.length; i++){

                            this.allTimeRecords.points.push(result[i]);
                        }
                    }

                    mysql.query(suicidesQuery, (err, result) =>{
                        if(err) throw err;
    
                        if(result.length > 0){
                            for(let i = 0; i < result.length; i++){
    
                                this.allTimeRecords.suicides.push(result[i]);
                            }
                        }

                        mysql.query(headshotsQuery, (err, result) =>{
                            if(err) throw err;
        
                            if(result.length > 0){
                                for(let i = 0; i < result.length; i++){
        
                                    this.allTimeRecords.headshots.push(result[i]);
                                }
                            }

                            mysql.query(spreeQuery, (err, result) =>{
                                if(err) throw err;
            
                                if(result.length > 0){
                                    for(let i = 0; i < result.length; i++){
            
                                        this.allTimeRecords.bestSpree.push(result[i]);
                                    }
                                }
                            });
                            mysql.query(multiQuery, (err, result) =>{
                                if(err) throw err;
            
                                if(result.length > 0){
                                    for(let i = 0; i < result.length; i++){
            
                                        this.allTimeRecords.bestMulti.push(result[i]);
                                    }
                                }

                                mysql.query(damageQuery, (err, result) =>{
                                    if(err) throw err;
                
                                    if(result.length > 0){
                                        for(let i = 0; i < result.length; i++){
                
                                            this.allTimeRecords.damage.push(result[i]);
                                        }
                                    }
                                   // this.getMatchRecords();
                                    
                                   // console.log(this.allTimeRecords);
                                });
                            });
                        });
                    });
                });
            });
        });*/
Changed to this:

Code: Select all

//only render the page when all data is loaded
    getResult(query, index, bAllTime){

        let data = [];
        mysql.query(query, (err, result) =>{

            if(err) throw err;
            if(result.length > 0){
                for(let i = 0; i < result.length; i++){
                    data.push(result[i]);
                }
            }

            

            if(bAllTime){
                this.allTimeRecords[index] = data;
                this.currentAllTimeQueries++;
            }else{
                this.matchRecords[index] = data;
            }

            if(this.allTimeQueries == this.currentAllTimeQueries){
                this.res.render("totals",{"allTime":this.allTimeRecords,"match":this.matchRecords});
            }
            
            return data;
        });
    }

Code: Select all


this.getResult(killQuery,"kills", true);
this.getResult(deathsQuery,"deaths", true);
this.getResult(scoreQuery,"points", true);
this.getResult(suicidesQuery,"suicides", true);
this.getResult(headshotsQuery,"headshots", true);
this.getResult(spreeQuery,"bestSpree", true);
this.getResult(multiQuery,"bestMulti", true);
this.getResult(damageQuery,"damage", true);









-------------------edit------------------------------------

Finished player search page
Image