[WIP] Node UTStats Beta 12C (FTP) Nexgen Stats Viewer

[UT Sniper (SJA94)]

Node UTStats is a full remake of the old utstats system using the same UTStats mutators, coded in javascript and using more updated mysql compared to the original.

Online example: https://stats.xjon.me/

New build released: https://github.com/scottadkin/Node-UTSt ... /tag/1.12C

ReadME: https://github.com/scottadkin/Node-UTStats

New in 12C:
- Added support for 4CTF screenshots.
- Fixed map screenshot on matches page to always use the non prefix over gametype prefix names.



New in 12B:
- Changed how map images are displayed, instead of just looking for an image name without a gametype prefix the system will now look for a file with the gametype prefix, if that is not found it will look for an image with the gametype prefix, and if that also fails it will display the default image.
- Changed how the recent matches display is rendered.

New this build:

Nexgen stats viewer tool, now you can easily configure how your UT server will display the data from your website, you are not restricted to just ranking points, there are many different categories you can use for any gametype.







Supported gametypes:
- Deathmatch
- Team Deathmatch
- Capture the Flag
- Domination
- Last man standing
- Bunnytrack
- Monsterhunt
- Siege (basic support)
- Coop

Map Importer added
Every map file in your /Maps/ folder will be imported and saved with the option set to true.

Nexgen Stats viewer support added
From version beta 8 there is now support for nexgen stats viewer by using the ulr example.com:1337/nexgenstats

Ace Log+Sshot importer
You can now import ace logs and screenshots to the website.

Bunnytrack BTPlusPlus.ini Importer added
You can now import your server bunnytrack records into your nodeUtstats website by running node btimport from the server module directory.

Map screenshot packs
Place the image files in /public/files/maps on the website module.
- Default Deathmatch maps

Possible problems:
- .tmp files sometimes show errors while importing but are moved successfully.
- First import may get stuck if mapimport is set to true.

ChangeLog(From what I can remember):

Build 12 release notes(14/04/20):

- Added player name to page description and open graph data, for individual player match report.
- Moved player profile link on individual player match report to the top.
- Fixed empty pickup data being displayed if there were no pickup events during match.
- Added server hostname to server query page. (Thanks to fbraz3)
- Added files to a perfect webserver setup(Linux). (Thanks to fbraz3)
- Added Nexgen stats viewer tool to admin area.

- Build 11 release notes(09/04/20)
- The Website, Importer and Mutator are now combined into one package, you no longer need to install the server/importer module to each ut server now just add the server ftp details to the ftpServers array in Importer/api/config.js.
- Added FTP support, you can now have an array of servers for the importer to download the logs from the utservers.
- Added ACE logs importer to ftp.
- Added ACE screenshot importer to ftp.
- Added BTPlusPlus.ini importer to ftp.
- Added BTGame.ini importer to ftp.
- Ace screenshots are now moved after import to Logs/Imported/Ace-Shots/<month>-<year>.
- BT record inis are now backed up daily via the latest import of that day to BT/Imported/<day>-<month>-<year> after being imported.
- Already Imported .log/.jpg/.tmp files will not be downloaded from ftp servers.
- Imported files from utservers are now backed up by default to another local folder to speed up the later imports(.log, .tmp, .jpg). If bDeleteFilesFromFTP is set to true they will be deleted locally.
- Fixed crash while trying to import bt records.
- Fixed headshots import.
- Fixed team kills for non team games.
- Fixed some logs failing to import after a bt log.
- Fixed server MOTD lines not importing.
- Fixed player pickups count being wrong if they reconnect.
- Fixed player best spree & multi not resetting after death. (I'm an idiot )
- Fixed grabbed and capped player being counted as assists in the flag captures area.
- Fixed javascript error trying to access teams that don't exist for the kill graphs.
- Fixed headshots not being merged if a player reconnects.
- Fixed broken flag images on records page.
- Fixed broken flag images on player page under possible aliases.
- Fixed error message displayed if a map has had 0 matches played.
- Fixed crash on website on players page when searching by first match.
- Fixed domination point graph showing team colours that were not in the match.
- Fixed Serbia flag being missing from package.
- Added BTGame.ini records import, now both BTPlusPlus.ini and BTGame.ini records are imported via the btimport.js on the server module.
- Added upgrade.js to website module that admins can run instead of making a fresh database from upgrading from version 9 and above.
- Added a warning message if a log is a large size, so users know the difference between a crash/infinite loop and a large file.
- Added mapList import, if you run "node mapimport" in the server module it will import every single map in your servers ./Maps/ folder. It will also set the file size of maps that have been imported from logs.
- Added kill distance to the mutator, now the distance between kills are saved to the log.
- Added shortest kill distance to matches and profiles for every player.
- Added longest kill distance to matches and profiles for every player.
- Added flag base locations to the mutator, now the flag base locations are saved to the log.
- Added spawn point locations to the mutator.
- Added weapon locations to the mutator.
- Added pickup locations to the mutator.
- Added Domination point locations to the mutator.
- Added shortest and longest time between kills to the mutator.
- Added player spawn locations to mutator.
- Added extended flag kill data logging, now killer, victim, kill distance, distance to cap, and distance to base are now saved.
- Added filename to match database, this will be used to delete duplicates from the admin panel.
- Added extended flag kills information to match reports.
- Added extended kill data to match pages, once the button is clicked the page will load all kill data that includes time, killer, victim, distance, weapon.
- Added clickable headers to match sections, now you can link directly to the area you want to share.
- Added headshots to frag performance if there was a headshot in the match.
- Added player record imports to btimport, now players will have their records saved to their pages.
- Added map bt records to map pages.
- Updated style of pickup summary on match pages.
- Updated style of player serach page, removed ugly search form and replaced it with clickable arrows next to table headers to specify sort order.
- Updated style of maps serach page, added clickable arrows next to table headers to specify sort order instead of stupid dropdown box.
- Merged Best multi kills and sprees in match pages to their respected areas instead of having separate areas.
- Added match player individual stats, now if you click on a players name in a match report it will take you to their match performance page with extended data displayed.
- Added player specific kills and deaths to match player individual stats.
- Added player Killing Sprees Information to match player individual stats, this will show how long the spree was in kills, time, and how the spree ended.
- Added clickable player match links to match weapon stats.
- Added clickable player match links to domination point capture summary.
- Added clickable player match links to pickups summary.
- Added clickable player match links to ranking summary.
- Added clickable player match links to team change summary.
- Added clickable player match links to player connections summary.
- Match pickup data now only displays players that have picked up 1 or more items to save scrolling time.
- Updated style of match assault captured objectives.
- Removed player connection history from match pages.

Original Post:

Spoiler

Show

The last few days I've been working on a new utstats system that will use nodeJS instead of php, one thing I've been struggling to make my mind up on is whether I'm going to do one complete package, or do a server module, combined with a website module.

Server module:
This will handle all the log parsing, database updates, file sorting/managament. Basically the backend of the system.
The idea is you can have this running on multiple servers, that connect to the website module. Note: This methods will not use ftp.

Website module:
The website the user can interact with that all connected servers send their data to, will include admin control panel.



Would this just be unnecessary, or would it be more convenient?


Notes of what has been done so far:

Log Importer

- Scans the logs directory for .log files, ignoring .tmp files(I can make an options that will allow the admin to import these files, but advanced stats for stats_player events won't be there).
- NUTStats keeps track of all imported files, if the file has already been imported it will be ignored to prevent duplicates.

TODO:
- Let the admin choose what happens to compiled logs, this could be deleting, moving, compressing and moving.

[Darkelarious]

Would this just be unnecessary, or would it be more convenient?

This sounds a lot like the interfaces I wrote for 333networks and the masterserver.

333networks hosts (several) masterserver(s) and stores the servers in a database. Our website can access the database and display server information (close to) real-time.

We also have a jQuery frontend in combination with a json API with our masterserver database.
http://333networks.com/json --> json API documentation

Working example:
http://ubrowser.333networks.com/ut/

Is this the kind of thing you have in mind?

[Shrimp]

It's definitely a better architecture to have a separate backend/processing service, versus a combined website/backend thing.

Depending on how you want to do things, you might even consider breaking the backend into two services as well:

- import/submission/processing: this is where the logs are pushed, parsed, and written to the DB

- view/query/api service: this reads from the DB the processing service writes to, and is what the frontend connects to

Allows you to scale things (har har, not like we need to "scale" UT services for more than 5 hits a week, but whatever, its fun to design these things anyway) separately, and you can also do things like redeploy or improve the parsing stuff without breaking the website, and vice versa.

Some modern tooling would definitely be cool

[UT Sniper (SJA94)]

Would this just be unnecessary, or would it be more convenient?

This sounds a lot like the interfaces I wrote for 333networks and the masterserver.

333networks hosts (several) masterserver(s) and stores the servers in a database. Our website can access the database and display server information (close to) real-time.

We also have a jQuery frontend in combination with a json API with our masterserver database.
http://333networks.com/json --> json API documentation

Working example:
http://ubrowser.333networks.com/ut/

Is this the kind of thing you have in mind?

Yeah that's what I have in mind. It's nice to see someone else who likes to recreate UT look for webpages

Nexgen system for my INI site:



One of many projects I start but never finish..

It's definitely a better architecture to have a separate backend/processing service, versus a combined website/backend thing.

Depending on how you want to do things, you might even consider breaking the backend into two services as well:

- import/submission/processing: this is where the logs are pushed, parsed, and written to the DB

- view/query/api service: this reads from the DB the processing service writes to, and is what the frontend connects to

Allows you to scale things (har har, not like we need to "scale" UT services for more than 5 hits a week, but whatever, its fun to design these things anyway) separately, and you can also do things like redeploy or improve the parsing stuff without breaking the website, and vice versa.

Some modern tooling would definitely be cool

What I was also thinking of doing is was a module that is compatible with the current utstats database, so people can still access old matches from previous versions of utstats.

I also want to make this in such a way other people can create and add their own modules to add support for custom game types if needed, as well as other stuff like I made for the current UTstats like this: https://github.com/scottadkin/UTstats-c ... ots-plugin. What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.

[esnesi]

What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.

That sounds like a dream!

[UT Sniper (SJA94)]

What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.

That sounds like a dream!

Very early video of what I mean, so far it only does kills... and me being me I forgot to set the player' team during import so I need to do that This is a ctf match, but because I forgot to set the players teams you have to imagine its a team scoreboard. There is no sounds yet or special events.






What I plan on doing is make it so you can cycle through the players via the canvas so you can hear their special events, as well as the match events like flag captures and so on.




-----------------------------------------------------------------------------edit------------------------------------------------




-------------------------------------------------------------------------------edit----------------------------------------------------





-------------------------------------------------------------------------------edit--------------------------------------------------------


Done some more work on the replay system, system now keeps tracks of player connects/disconnects and team changes:








----------------------------------------------------------edit----------------------------------------------------------------------------------

User can now cycle through all the players currently in the game, you will only hear the multi kills and spree events for the player you are currently spectating(I haven't added headshots yet). If another player gets a spree event you will hear the same sound as you do in game when another players gets a spree event.



----------------------------------------------------efit---------------------------------







---------------------------------------------------------edit--------------------------------------------

Early match report page, the players won't be like that colour wise, just did that for testing purposes.

[UT Sniper (SJA94)]

Just had a thought, would people be interested if I made a discord bot for my new utstats system.

Example of what I mean:
.stats player <playername>

<url to player page>
<player has played 237 games>
<last played 23 hours 4 minutes ago>
<link to last played game>

.stats gametype <gametype>

<url to gametype match history>
<2104 game played of <gametype>
<last played 21 minutes 5 seconds ago>
<link to last played game>

Sorry for double post, been editing last post last 3 days, wanted to ask in new post about a discord bot.




--------------------------------edit----------------------------------------

Dm scoreboard, doesn't have FPH and pings yet:

[papercoffee]

Sorry for double post, been editing last post last 3 days

That's ok ... this doesn't count as double post.

[UnrealGGecko]

Just had a thought, would people be interested if I made a discord bot for my new utstats system.

Example of what I mean:
.stats player <playername>

<url to player page>
<player has played 237 games>
<last played 23 hours 4 minutes ago>
<link to last played game>

.stats gametype <gametype>

<url to gametype match history>
<2104 game played of <gametype>
<last played 21 minutes 5 seconds ago>
<link to last played game>

Considering there are some active clans that host servers and hage a Discord channel, I say there might be interest on this. Just gotta let them know.

Heck, I might try to put it in the ut99org Discord as well, if its easy to put in

[UT Sniper (SJA94)]

Just a preview video of a DM game being played at 6 seconds a second. I have made some changes to the server module which will enhance this feature more in the future which will includes pickup logging, which weapon the player killed the other player with, and also added support for smartCTF/DM which will save the players faces/netspeed/FOV which will later be implemented in SmartCTF/DM scoreboards.

The user interacting with the match replay can choose which player they want to view, which will effect what sounds will be played, hud messages, and so on. I haven't implemented pick up hud, you were killed by, as well as timers, and other stats for the gameplay modeimg.





Here is sshot at end of game from NUTStats:


Here is a in game screenshot of the same match, the small difference in Oopers kills is me forgetting to making the player lose a point for suicides, atm it counts suicides as the player has killed someone.









------------------------------------------edit------------------------------

System now automatically imports weapons from log files and save them to database(kill strings):

[UT Sniper (SJA94)]

Match replay now includes item pickups, which includes the sounds for that item when they are picked up. Updated gameplay HUD, as well as some other stuff for the replay system which you can see in this video:




























----------------------------------------edit------------------------------------------

- Added support for Assault.
- Added support for Domination.











--------------------------------edit------------------------

[UT Sniper (SJA94)]

Just a sneak peak of a match replay with the smartCTF scoreboard, also updated the "gameplay" HUD. Just need to add support for LMS and Assault to the match replay(won't take long), then I'll work on the design of the website.



Ignore the fact it's a CTF match with a DM image as the background, haven't added that functionality yet, I need to add import functions for map Images and faces.





------------------------------edit-------------------------------------


Now supports DOM point captures for match replay:



Also added LMS support




Just need to do assault now.

[UT Sniper (SJA94)]

Just a screenshot of a more updated homepage, was doing another project for someone so didn't put much time into nutstats for a while:

[esnesi]

This looks very awesome, and I am very interested in this for our server!

[UT Sniper (SJA94)]

Just a sshot of a page from top to bottom, I removed stuff via developer tools in browser so you get to see the whole design of a page. There are totals from both all time stats, and records, including more per category then is displayed for single matches:








Callback hell :

Code: Select all

mysql.query(killQuery,(err, result) =>{

            if(err) throw err;

            if(result.length > 0){

                for(let i = 0; i < result.length; i++){
                    this.allTimeRecords.kills.push(result[i]);
                }

                
            }

            mysql.query(deathsQuery, (err, result) =>{
                if(err) throw err;

                if(result.length > 0){
                    for(let i = 0; i < result.length; i++){
                        this.allTimeRecords.deaths.push(result[i]);
                    }
                }

                mysql.query(scoreQuery, (err, result) =>{
                    if(err) throw err;

                    if(result.length > 0){
                        for(let i = 0; i < result.length; i++){

                            this.allTimeRecords.points.push(result[i]);
                        }
                    }

                    mysql.query(suicidesQuery, (err, result) =>{
                        if(err) throw err;
    
                        if(result.length > 0){
                            for(let i = 0; i < result.length; i++){
    
                                this.allTimeRecords.suicides.push(result[i]);
                            }
                        }

                        mysql.query(headshotsQuery, (err, result) =>{
                            if(err) throw err;
        
                            if(result.length > 0){
                                for(let i = 0; i < result.length; i++){
        
                                    this.allTimeRecords.headshots.push(result[i]);
                                }
                            }

                            mysql.query(spreeQuery, (err, result) =>{
                                if(err) throw err;
            
                                if(result.length > 0){
                                    for(let i = 0; i < result.length; i++){
            
                                        this.allTimeRecords.bestSpree.push(result[i]);
                                    }
                                }
                            });
                            mysql.query(multiQuery, (err, result) =>{
                                if(err) throw err;
            
                                if(result.length > 0){
                                    for(let i = 0; i < result.length; i++){
            
                                        this.allTimeRecords.bestMulti.push(result[i]);
                                    }
                                }

                                mysql.query(damageQuery, (err, result) =>{
                                    if(err) throw err;
                
                                    if(result.length > 0){
                                        for(let i = 0; i < result.length; i++){
                
                                            this.allTimeRecords.damage.push(result[i]);
                                        }
                                    }
                                   // this.getMatchRecords();
                                    
                                   // console.log(this.allTimeRecords);
                                });
                            });
                        });
                    });
                });
            });
        });*/

Changed to this:

Code: Select all

//only render the page when all data is loaded
    getResult(query, index, bAllTime){

        let data = [];
        mysql.query(query, (err, result) =>{

            if(err) throw err;
            if(result.length > 0){
                for(let i = 0; i < result.length; i++){
                    data.push(result[i]);
                }
            }

            

            if(bAllTime){
                this.allTimeRecords[index] = data;
                this.currentAllTimeQueries++;
            }else{
                this.matchRecords[index] = data;
            }

            if(this.allTimeQueries == this.currentAllTimeQueries){
                this.res.render("totals",{"allTime":this.allTimeRecords,"match":this.matchRecords});
            }
            
            return data;
        });
    }

Code: Select all

this.getResult(killQuery,"kills", true);
this.getResult(deathsQuery,"deaths", true);
this.getResult(scoreQuery,"points", true);
this.getResult(suicidesQuery,"suicides", true);
this.getResult(headshotsQuery,"headshots", true);
this.getResult(spreeQuery,"bestSpree", true);
this.getResult(multiQuery,"bestMulti", true);
this.getResult(damageQuery,"damage", true);

-------------------edit------------------------------------

Finished player search page