[WIP] Node.js UTStats Beta 10D

[UT Sniper (SJA94)]

Node UTStats is a full remake of the old utstats system using the same UTStats mutators, coded in node.js and using updated mysql compared to the original.
There is no need for fpt access for servers as logs are directly imported to a mysql database via the server modules.



Basic Idea:

The idea is to have an instance of the server module in the main directory of your unrealtouranament server(s) 1 directory deep (e.g C:/UnrealTournament/NodeUTstatsServer/).
These modules will parse logs and send the data to a database that the website module connects to, there is no limit of how many server modules can be connected to a website module.
You could have 1 server module connected to 1 website module or have 20 servers connected to multiple website modules.

New build released: https://drive.google.com/file/d/1Sobbck ... sp=sharing

- Build 10D release notes(09/01/20)

- BT import now creates player profiles if the record owner doesn't exist.

To Import run node btimport in the command prompt(server module).

Upgrading from version 9:

Simply replace all files with the new ones, there has been no changes to the database in this build.

Change Log:

- Build 10B release notes(08/01/20)
- Fixed broken flag images on bunnytrack pages.
- Added merge player with ips to admin area.
- Fixed player pages not responding if a player has no data.

Note: Admin control panel is very basic at the moment with basic search and match delete.

Once you have installed everything you should see the website running at port 1337 unless changed (www.example.com:1337), if you can't access it from outside the server you have to port forward the port number.

Supported gametypes:
- Deathmatch
- Team Deathmatch
- Capture the Flag
- Domination
- Last man standing
- Bunnytrack
- Monsterhunt
- Siege (basic support)
- Coop

Nexgen Stats viewer support added
From version beta 8 there is now support for nexgen stats viewer by using the ulr example.com:1337/nexgenstats

Ace Log+Sshot importer
You can now import ace logs and screenshots to the website.

Bunnytrack BTPlusPlus.ini Importer added
You can now import your server bunnytrack records into your nodeUtstats website by running node btimport from the server module directory.

Map screenshot packs
Place the image files in /public/files/maps on the website module.
- Default Deathmatch maps

Possible problems:
- Some custom monsterhunt gametypes may fail to import.
- .tmp files sometimes show errors while importing but are moved successfully.


Github
- Node UTStats Server module (this will be placed in your unreal tournament server directory, there is no limit of how many of these can be connected to a website module)
- Node UTStats Website module (you can use one installation for many servers or have separate instances for different servers )


Change log:

- Build 10D release notes(09/01/20)
- BT import now creates player profiles if the record owner doesn't exist.

- Build 10C release notes(08/01/20)
- Added BTPlusPlus.ini record importer.

- Build 10B release notes(08/01/20)
- Fixed broken flag images on bunnytrack pages.
- Added merge player with ips to admin area.
- Fixed player pages not responding if a player has no data

- Build 10 release notes(05/01/20)
- Fixed efficiency displaying too many digits
- Fixed gametype name being set to unknown gametype if the gametype is set to hidden for the ranking page.
- Added scores to player search page.
- Added kills to player search page.
- Added deaths to player search page.
- Added First Match to player search page.
- Added sort by Score to player search page.
- Added sort by Kills to player search page.
- Added sort by Deaths to player search page.
- Added sort by First Match to player search page.
- Updated style of most played gametypes on homepage.
- Updated style of most popular countries on homepage.
- Updated style on rankings page.
- Updated style on player search page.
- Updated style on records page.
- Updated style on match page.
- Updated style on bunnytrack page.
- New monster images, big thanks to Krull0r.
- Added gametype search to records pages, this includes the main page, all time, and match record versions.
- Fixed crash while importing some bunnytrack matches.
- Fixed data not being displayed in bunnytrack matches.
- Added Grab to Cap Ratio to player page.

- Build 9d release notes(09/12/19)
- Added best multi kill for each player to match pages.
- Added best killing spree for each player to match pages.

- Build 9c release notes(09/12/19)
- Added flag returns table to ctf matches.

- Build 9b release notes(07/12/19)
- Fixed special events not being counted.
- Removed readme.mds

- Build 9 release notes(07/12/19):
- REQUIRES NEW DATABASE INSTALL
- Moved database install to the website module, thanks to ue for pointing how stupid that was. You now have to run "node install" for both modules to install the database(website module), and directories(server module).
- Changed time strings to my new EpochToString module, will now display years - weeks - days - minutes - seconds, it will also not display just 1 day, it will display 1 day and 32 seconds ect.
- Added tool to the admin area that creates thumbnails of all images current stored on server.
- Added bulk image upload to admin area, this can be used to easily upload map screenshot packs(all files are converted and resized to 1920x1080).
- Added single image file upload form next to each missing map name in admin panel, the image name is automatically changed to match the current selected map, and file type is converted to a jpg image.
- Added config options to hide navigation options, Bunnytrack tab is now hidden by default. You can find these in config.js by the bNavDisplay<type>: true/false, restart is required for there settings to take effect.
- Added face image upload in admin area, name is automatically set.
- Changed how monster images are displayed, each different monster needs it's own file now.
- Added monster image upload to admin area.
- Fixed error caused by no frag data in ut server query.
- Added option to merge players to admin area.
- Added option to rename players to admin area.
- Added option to delete players to admin area.
- Gametype class is now saved to database from each match, this should help badly named gametypes to be imported and rendered correctly.
- Fixed CTF events not being loaded in match page if the gametype has a custom name.
- Added ACE support, now ACE logs and screenshots will be imported to the database.
- Added alternative database install script, this can be used if your mysql user doesn't have permission to create databases.
- Fixed empty server query box displaying on home page if no servers are set to be queried.
- Added ACE kick log search to admin area.
- Added basic site analytics to admin area.
- Changed dm scoreboard if the gametype is battle royal.
- Updated the kills match up data on match reports to fit better and bee easier to read.
- Fixed missing mysql queries for installer.
- Fixed weapon stats importing, weapon names are correctly imported now.
- Fixed player profile best multi kill being 1 more than it should be.
- Added option to admin area that allows admin to choose what gametypes are displayed on the rankings page.
- Fixed green and yellow teams being displayed in the team kill graphs even if they were not in the match.
- Added option to admin area that allows admin to choose what record types to display.
- Fixed playtime record not being displayed on the records page.
- Fixed players being on wrong team on match screenshots.
- Added time and efficiency to players on smart ctf scoreboard.
- Fixed duplicate weapon stats data for reconnected players in matches.
- Fixed kills being wrong in matches.
- Fixed weapon totals not being displayed on player pages.
- Added total accuracy stat to player frag performance bar.
- Map images on many parts of the site will now be thumbnail size instead of fullsize when available.

- Build 8 release notes(10/11/19):

- Added Nexgen Stats Viewer support.
- Made the mutator log a random face texture if PlayerReplicationInfo.TalkTexture == None


- Build 7c release notes(09/11/19):

- Fixed installer script.

- Build 7b release notes(08/11/19):

- Fixed logic error for inserting kills into the database.

- Build 7 release notes(08/11/19):

- Fixed broken flag image links on rankings, records, and match pages.
- Added view image option to admin area for map images.
- Added view image option to admin area for monster images.
- Added face images to admin faces area.
- Added users can now connect directly to a utserver by clicking the server ip link on a server box on the homepage.
- Fixed country flag size on rankings page.
- Fixed country flag size and position on match frag performance area.
- Fixed country flag size and position on match Multi kills and sprees area.
- Fixed country flag size and position on match weapon stats area.
- Fixed country flag size and position on match ranking change area.
- Fixed country flag size and position on match connections area.
- Fixed country flag size and position on match kills match up area.
- Fixed country flag size and position on player search page.
- Fixed country flag size and position on records main page.
- Updated recent match boxes to include the server name (Every page).
- Fixed text overflow on gametype boxes on player pages.
- Updated the style of match pickups.

- Build 6e release notes(07/11/19):

- Fixed broken flag image links on ranking, records, and match pages.

- Build 6d release notes(06/11/19):

- Fixed broken flag image links on Linux machines.

- Build 6c release notes(05/11/19):

- Fixed match import if killer is empty in log.

- Build 6b release notes(05/11/19):

- Fixed "Can't find package Promise" even when it was installed.
- Fixed crash when no UT Servers are set to be queried.

- Build 6 release notes(04/11/19):

- Added order types to player searches, you can now order by total matches, total time, last seen, and name. This includes ASC and DESC order.
- Fixed searches not returning more that the first page or data for player searches.
- Added more options to map search, you can now search for map name, map title, and map author. You can order the search by map name, map title, map author, total matches, total playtime, last match, and first match in ASC and DESC order.
- Updated style of map search page.
- Updated style of map page.
- Updated style of player search page.
- Updated style of player profile pages.
- Updated the colours for teams throughout the site.
- Added mouse over event for every flag image that displays the country name.
- Fixed client side javascript accessed null on match page.

- Build 5 release notes:

- Fixed crash if a server is added to the query list if the server address is not a valid ip or address (Website).
- Added reimport script, if you upgrade to a new build(that requires a new database) and want to import all previous logs imported, you can
use the command "node reimport" to import all valid logs from ../Logs/imported/

- Build 4 release notes:

- Added player connection events to match pages.
- Added last import time at bottom of home page.


- Build 3 release notes:

- Added UT server queries, you can now specify a number of ut servers you want to query every X minutes.
These will then be displayed on the homepage displaying the map image, current player count, server ip:port, and when the server was last queried.
Servers that use a dynamic dns will still work without problem whenever the server ip changes, and the correct ip:port will be displayed.

- Added a new admin area for managing ut server queries.

- Fixed monster file image status in admin area.

- Added pickups data to match pages, this displays every single weapon, ammo, and powerups for every player.

- Added site hits counter at the bottom of the home page.


- Build 2 release notes

Upgrading from beta build 1:
- The only package that has changed is the website module so there is no need to replace the other packages, or reinstall the databases.

Change Log:

- Added views counter to match pages.
- Added views counter to profile pages.
- Updated the style so data is easier to read.
- Changed the DM scoreboard so it is the same order as in-game.

Readme:

=============================================================================
========================== Node UTStats Beat build 5 ========================
========================= Created By Scott Adkin 2019 =======================
=============================================================================

=============================================================================
============================== Requirements =================================
=============================================================================

- Windows or Linux system.

- Node.js

Download and install from here https://nodejs.org/en/ choose the LTS(long term support) package.

- MYSQL

Download and install from here https://dev.mysql.com/downloads/

- UTStats Mutator

Download and install from here http://ut-files.com/index.php?dir=Stats ... eta428.zip
Skip the website install as it's not needed.

Each package has it's own individual readme included.

=============================================================================
================================= About =====================================
=============================================================================

Node UTStats is a full remake of the old utstats system using the same UTStats mutators, coded in node.js and using updated mysql compared to the original.
There is no need for fpt access for servers as logs are directly imported to a mysql database via the server modules.



Basic Idea:

The idea is to have an instance of the server module in the main directory of your unrealtouranament server(s) 1 directory deep (e.g C:/UnrealTournament/NodeUTstatsServer/).
These modules will parse logs and send the data to a database that the website module connects to, there is no limit of how many server modules can be connected to a website module.
You could have 1 server module connected to 1 website module or have 20 servers connected to multiple website modules.



Your unreal tournament server must be running the UTStats server actors for the minimum support, the NodeUTStats is optional, but will log more data(spawnkills, monsterhunt kills, just to name a few) then the utstats server actors.



============================================================================
========================= Thanks to =========================================
============================================================================

The original UTStats developers zazel, AnthraX and toa.

Main tester esnesi.

Original Post:

Spoiler

Show

The last few days I've been working on a new utstats system that will use nodeJS instead of php, one thing I've been struggling to make my mind up on is whether I'm going to do one complete package, or do a server module, combined with a website module.

Server module:
This will handle all the log parsing, database updates, file sorting/managament. Basically the backend of the system.
The idea is you can have this running on multiple servers, that connect to the website module. Note: This methods will not use ftp.

Website module:
The website the user can interact with that all connected servers send their data to, will include admin control panel.



Would this just be unnecessary, or would it be more convenient?


Notes of what has been done so far:

Log Importer

- Scans the logs directory for .log files, ignoring .tmp files(I can make an options that will allow the admin to import these files, but advanced stats for stats_player events won't be there).
- NUTStats keeps track of all imported files, if the file has already been imported it will be ignored to prevent duplicates.

TODO:
- Let the admin choose what happens to compiled logs, this could be deleting, moving, compressing and moving.

[Darkelarious]

Would this just be unnecessary, or would it be more convenient?

This sounds a lot like the interfaces I wrote for 333networks and the masterserver.

333networks hosts (several) masterserver(s) and stores the servers in a database. Our website can access the database and display server information (close to) real-time.

We also have a jQuery frontend in combination with a json API with our masterserver database.
http://333networks.com/json --> json API documentation

Working example:
http://ubrowser.333networks.com/ut/

Is this the kind of thing you have in mind?

[Shrimp]

It's definitely a better architecture to have a separate backend/processing service, versus a combined website/backend thing.

Depending on how you want to do things, you might even consider breaking the backend into two services as well:

- import/submission/processing: this is where the logs are pushed, parsed, and written to the DB

- view/query/api service: this reads from the DB the processing service writes to, and is what the frontend connects to

Allows you to scale things (har har, not like we need to "scale" UT services for more than 5 hits a week, but whatever, its fun to design these things anyway) separately, and you can also do things like redeploy or improve the parsing stuff without breaking the website, and vice versa.

Some modern tooling would definitely be cool

[UT Sniper (SJA94)]

Would this just be unnecessary, or would it be more convenient?

This sounds a lot like the interfaces I wrote for 333networks and the masterserver.

333networks hosts (several) masterserver(s) and stores the servers in a database. Our website can access the database and display server information (close to) real-time.

We also have a jQuery frontend in combination with a json API with our masterserver database.
http://333networks.com/json --> json API documentation

Working example:
http://ubrowser.333networks.com/ut/

Is this the kind of thing you have in mind?

Yeah that's what I have in mind. It's nice to see someone else who likes to recreate UT look for webpages

Nexgen system for my INI site:



One of many projects I start but never finish..

It's definitely a better architecture to have a separate backend/processing service, versus a combined website/backend thing.

Depending on how you want to do things, you might even consider breaking the backend into two services as well:

- import/submission/processing: this is where the logs are pushed, parsed, and written to the DB

- view/query/api service: this reads from the DB the processing service writes to, and is what the frontend connects to

Allows you to scale things (har har, not like we need to "scale" UT services for more than 5 hits a week, but whatever, its fun to design these things anyway) separately, and you can also do things like redeploy or improve the parsing stuff without breaking the website, and vice versa.

Some modern tooling would definitely be cool

What I was also thinking of doing is was a module that is compatible with the current utstats database, so people can still access old matches from previous versions of utstats.

I also want to make this in such a way other people can create and add their own modules to add support for custom game types if needed, as well as other stuff like I made for the current UTstats like this: https://github.com/scottadkin/UTstats-c ... ots-plugin. What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.

[esnesi]

What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.

That sounds like a dream!

[UT Sniper (SJA94)]

What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.

That sounds like a dream!

Very early video of what I mean, so far it only does kills... and me being me I forgot to set the player' team during import so I need to do that This is a ctf match, but because I forgot to set the players teams you have to imagine its a team scoreboard. There is no sounds yet or special events.






What I plan on doing is make it so you can cycle through the players via the canvas so you can hear their special events, as well as the match events like flag captures and so on.




-----------------------------------------------------------------------------edit------------------------------------------------




-------------------------------------------------------------------------------edit----------------------------------------------------





-------------------------------------------------------------------------------edit--------------------------------------------------------


Done some more work on the replay system, system now keeps tracks of player connects/disconnects and team changes:








----------------------------------------------------------edit----------------------------------------------------------------------------------

User can now cycle through all the players currently in the game, you will only hear the multi kills and spree events for the player you are currently spectating(I haven't added headshots yet). If another player gets a spree event you will hear the same sound as you do in game when another players gets a spree event.



----------------------------------------------------efit---------------------------------







---------------------------------------------------------edit--------------------------------------------

Early match report page, the players won't be like that colour wise, just did that for testing purposes.

[UT Sniper (SJA94)]

Just had a thought, would people be interested if I made a discord bot for my new utstats system.

Example of what I mean:
.stats player <playername>

<url to player page>
<player has played 237 games>
<last played 23 hours 4 minutes ago>
<link to last played game>

.stats gametype <gametype>

<url to gametype match history>
<2104 game played of <gametype>
<last played 21 minutes 5 seconds ago>
<link to last played game>

Sorry for double post, been editing last post last 3 days, wanted to ask in new post about a discord bot.




--------------------------------edit----------------------------------------

Dm scoreboard, doesn't have FPH and pings yet:

[papercoffee]

Sorry for double post, been editing last post last 3 days

That's ok ... this doesn't count as double post.

[UnrealGGecko]

Just had a thought, would people be interested if I made a discord bot for my new utstats system.

Example of what I mean:
.stats player <playername>

<url to player page>
<player has played 237 games>
<last played 23 hours 4 minutes ago>
<link to last played game>

.stats gametype <gametype>

<url to gametype match history>
<2104 game played of <gametype>
<last played 21 minutes 5 seconds ago>
<link to last played game>

Considering there are some active clans that host servers and hage a Discord channel, I say there might be interest on this. Just gotta let them know.

Heck, I might try to put it in the ut99org Discord as well, if its easy to put in

[UT Sniper (SJA94)]

Just a preview video of a DM game being played at 6 seconds a second. I have made some changes to the server module which will enhance this feature more in the future which will includes pickup logging, which weapon the player killed the other player with, and also added support for smartCTF/DM which will save the players faces/netspeed/FOV which will later be implemented in SmartCTF/DM scoreboards.

The user interacting with the match replay can choose which player they want to view, which will effect what sounds will be played, hud messages, and so on. I haven't implemented pick up hud, you were killed by, as well as timers, and other stats for the gameplay modeimg.





Here is sshot at end of game from NUTStats:


Here is a in game screenshot of the same match, the small difference in Oopers kills is me forgetting to making the player lose a point for suicides, atm it counts suicides as the player has killed someone.









------------------------------------------edit------------------------------

System now automatically imports weapons from log files and save them to database(kill strings):

[UT Sniper (SJA94)]

Match replay now includes item pickups, which includes the sounds for that item when they are picked up. Updated gameplay HUD, as well as some other stuff for the replay system which you can see in this video:




























----------------------------------------edit------------------------------------------

- Added support for Assault.
- Added support for Domination.











--------------------------------edit------------------------

[UT Sniper (SJA94)]

Just a sneak peak of a match replay with the smartCTF scoreboard, also updated the "gameplay" HUD. Just need to add support for LMS and Assault to the match replay(won't take long), then I'll work on the design of the website.



Ignore the fact it's a CTF match with a DM image as the background, haven't added that functionality yet, I need to add import functions for map Images and faces.





------------------------------edit-------------------------------------


Now supports DOM point captures for match replay:



Also added LMS support




Just need to do assault now.

[UT Sniper (SJA94)]

Just a screenshot of a more updated homepage, was doing another project for someone so didn't put much time into nutstats for a while:

[esnesi]

This looks very awesome, and I am very interested in this for our server!

[UT Sniper (SJA94)]

Just a sshot of a page from top to bottom, I removed stuff via developer tools in browser so you get to see the whole design of a page. There are totals from both all time stats, and records, including more per category then is displayed for single matches:








Callback hell :

Code: Select all

mysql.query(killQuery,(err, result) =>{

            if(err) throw err;

            if(result.length > 0){

                for(let i = 0; i < result.length; i++){
                    this.allTimeRecords.kills.push(result[i]);
                }

                
            }

            mysql.query(deathsQuery, (err, result) =>{
                if(err) throw err;

                if(result.length > 0){
                    for(let i = 0; i < result.length; i++){
                        this.allTimeRecords.deaths.push(result[i]);
                    }
                }

                mysql.query(scoreQuery, (err, result) =>{
                    if(err) throw err;

                    if(result.length > 0){
                        for(let i = 0; i < result.length; i++){

                            this.allTimeRecords.points.push(result[i]);
                        }
                    }

                    mysql.query(suicidesQuery, (err, result) =>{
                        if(err) throw err;
    
                        if(result.length > 0){
                            for(let i = 0; i < result.length; i++){
    
                                this.allTimeRecords.suicides.push(result[i]);
                            }
                        }

                        mysql.query(headshotsQuery, (err, result) =>{
                            if(err) throw err;
        
                            if(result.length > 0){
                                for(let i = 0; i < result.length; i++){
        
                                    this.allTimeRecords.headshots.push(result[i]);
                                }
                            }

                            mysql.query(spreeQuery, (err, result) =>{
                                if(err) throw err;
            
                                if(result.length > 0){
                                    for(let i = 0; i < result.length; i++){
            
                                        this.allTimeRecords.bestSpree.push(result[i]);
                                    }
                                }
                            });
                            mysql.query(multiQuery, (err, result) =>{
                                if(err) throw err;
            
                                if(result.length > 0){
                                    for(let i = 0; i < result.length; i++){
            
                                        this.allTimeRecords.bestMulti.push(result[i]);
                                    }
                                }

                                mysql.query(damageQuery, (err, result) =>{
                                    if(err) throw err;
                
                                    if(result.length > 0){
                                        for(let i = 0; i < result.length; i++){
                
                                            this.allTimeRecords.damage.push(result[i]);
                                        }
                                    }
                                   // this.getMatchRecords();
                                    
                                   // console.log(this.allTimeRecords);
                                });
                            });
                        });
                    });
                });
            });
        });*/

Changed to this:

Code: Select all

//only render the page when all data is loaded
    getResult(query, index, bAllTime){

        let data = [];
        mysql.query(query, (err, result) =>{

            if(err) throw err;
            if(result.length > 0){
                for(let i = 0; i < result.length; i++){
                    data.push(result[i]);
                }
            }

            

            if(bAllTime){
                this.allTimeRecords[index] = data;
                this.currentAllTimeQueries++;
            }else{
                this.matchRecords[index] = data;
            }

            if(this.allTimeQueries == this.currentAllTimeQueries){
                this.res.render("totals",{"allTime":this.allTimeRecords,"match":this.matchRecords});
            }
            
            return data;
        });
    }

Code: Select all

this.getResult(killQuery,"kills", true);
this.getResult(deathsQuery,"deaths", true);
this.getResult(scoreQuery,"points", true);
this.getResult(suicidesQuery,"suicides", true);
this.getResult(headshotsQuery,"headshots", true);
this.getResult(spreeQuery,"bestSpree", true);
this.getResult(multiQuery,"bestMulti", true);
this.getResult(damageQuery,"damage", true);

-------------------edit------------------------------------

Finished player search page