nodeJS UTStats

Discussions about UT99

nodeJS UTStats

Postby UT Sniper (SJA94) » Tue Mar 12, 2019 2:46 am

The last few days I've been working on a new utstats system that will use nodeJS instead of php, one thing I've been struggling to make my mind up on is whether I'm going to do one complete package, or do a server module, combined with a website module.

Server module:
This will handle all the log parsing, database updates, file sorting/managament. Basically the backend of the system.
The idea is you can have this running on multiple servers, that connect to the website module. Note: This methods will not use ftp.

Website module:
The website the user can interact with that all connected servers send their data to, will include admin control panel.



Would this just be unnecessary, or would it be more convenient?


Notes of what has been done so far:

Log Importer
- Scans the logs directory for .log files, ignoring .tmp files(I can make an options that will allow the admin to import these files, but advanced stats for stats_player events won't be there).
- NUTStats keeps track of all imported files, if the file has already been imported it will be ignored to prevent duplicates.

TODO:
- Let the admin choose what happens to compiled logs, this could be deleting, moving, compressing and moving.
Last edited by UT Sniper (SJA94) on Fri Mar 22, 2019 9:48 am, edited 1 time in total.
Find the Latest player inis https://ooper.000webhostapp.com/ut/newinis/ (2019 version)
My latest UT themed game Spawn Killer 4 Game
User avatar
UT Sniper (SJA94)
Adept
 
Posts: 446
Joined: Thu Jun 24, 2010 10:35 pm
Location: England
Personal rank: noob programmer

Re: nodeJS UTStats

Postby Darkelarious » Tue Mar 12, 2019 5:26 pm

UT Sniper (SJA94) wrote:Would this just be unnecessary, or would it be more convenient?

This sounds a lot like the interfaces I wrote for 333networks and the masterserver.

333networks hosts (several) masterserver(s) and stores the servers in a database. Our website can access the database and display server information (close to) real-time.

We also have a jQuery frontend in combination with a json API with our masterserver database.
http://333networks.com/json --> json API documentation

Working example:
http://ubrowser.333networks.com/ut/

Is this the kind of thing you have in mind?
User avatar
Darkelarious
Skilled
 
Posts: 162
Joined: Sat Feb 08, 2014 12:02 pm
Location: Phobos Moon

Re: nodeJS UTStats

Postby Shrimp » Tue Mar 12, 2019 7:08 pm

It's definitely a better architecture to have a separate backend/processing service, versus a combined website/backend thing.

Depending on how you want to do things, you might even consider breaking the backend into two services as well:

- import/submission/processing: this is where the logs are pushed, parsed, and written to the DB

- view/query/api service: this reads from the DB the processing service writes to, and is what the frontend connects to

Allows you to scale things (har har, not like we need to "scale" UT services for more than 5 hits a week, but whatever, its fun to design these things anyway) separately, and you can also do things like redeploy or improve the parsing stuff without breaking the website, and vice versa.

Some modern tooling would definitely be cool :thuup:
ShrimpWorks
Unreal Archive - preserving over 20 years of user-created content for the Unreal series!
User avatar
Shrimp
Average
 
Posts: 35
Joined: Wed Oct 10, 2018 11:15 am
Location: South Africa

Re: nodeJS UTStats

Postby UT Sniper (SJA94) » Tue Mar 12, 2019 9:33 pm

Darkelarious wrote:
UT Sniper (SJA94) wrote:Would this just be unnecessary, or would it be more convenient?

This sounds a lot like the interfaces I wrote for 333networks and the masterserver.

333networks hosts (several) masterserver(s) and stores the servers in a database. Our website can access the database and display server information (close to) real-time.

We also have a jQuery frontend in combination with a json API with our masterserver database.
http://333networks.com/json --> json API documentation

Working example:
http://ubrowser.333networks.com/ut/

Is this the kind of thing you have in mind?


Yeah that's what I have in mind. It's nice to see someone else who likes to recreate UT look for webpages 8)

Nexgen system for my INI site:
Image


One of many projects I start but never finish..
Image




Shrimp wrote:It's definitely a better architecture to have a separate backend/processing service, versus a combined website/backend thing.

Depending on how you want to do things, you might even consider breaking the backend into two services as well:

- import/submission/processing: this is where the logs are pushed, parsed, and written to the DB

- view/query/api service: this reads from the DB the processing service writes to, and is what the frontend connects to

Allows you to scale things (har har, not like we need to "scale" UT services for more than 5 hits a week, but whatever, its fun to design these things anyway) separately, and you can also do things like redeploy or improve the parsing stuff without breaking the website, and vice versa.

Some modern tooling would definitely be cool :thuup:


What I was also thinking of doing is was a module that is compatible with the current utstats database, so people can still access old matches from previous versions of utstats.

I also want to make this in such a way other people can create and add their own modules to add support for custom game types if needed, as well as other stuff like I made for the current UTstats like this: https://github.com/scottadkin/UTstats-client-side-screenshots-plugin. What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.
Find the Latest player inis https://ooper.000webhostapp.com/ut/newinis/ (2019 version)
My latest UT themed game Spawn Killer 4 Game
User avatar
UT Sniper (SJA94)
Adept
 
Posts: 446
Joined: Thu Jun 24, 2010 10:35 pm
Location: England
Personal rank: noob programmer

Re: nodeJS UTStats

Postby iSenSe » Fri Mar 15, 2019 10:24 pm

UT Sniper (SJA94) wrote:What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.


That sounds like a dream!
Image
User avatar
iSenSe
Adept
 
Posts: 281
Joined: Mon Aug 31, 2015 12:58 pm

Re: nodeJS UTStats

Postby UT Sniper (SJA94) » Sun Mar 17, 2019 9:30 am

iSenSe wrote:
UT Sniper (SJA94) wrote:What I really want to do is make a match replay system, where a user can watch a match progress second by second with an interactive HTML5 canvas, where users can toggle different scoreboards, server info, and have match events played like sprees, multi kills, flag captures.. and so on.


That sounds like a dream!


Very early video of what I mean, so far it only does kills... and me being me I forgot to set the player' team during import so I need to do that :loool: This is a ctf match, but because I forgot to set the players teams you have to imagine its a team scoreboard. There is no sounds yet or special events.






What I plan on doing is make it so you can cycle through the players via the canvas so you can hear their special events, as well as the match events like flag captures and so on.




-----------------------------------------------------------------------------edit------------------------------------------------




-------------------------------------------------------------------------------edit----------------------------------------------------





-------------------------------------------------------------------------------edit--------------------------------------------------------


Done some more work on the replay system, system now keeps tracks of player connects/disconnects and team changes:








----------------------------------------------------------edit----------------------------------------------------------------------------------

User can now cycle through all the players currently in the game, you will only hear the multi kills and spree events for the player you are currently spectating(I haven't added headshots yet). If another player gets a spree event you will hear the same sound as you do in game when another players gets a spree event.
Image


----------------------------------------------------efit---------------------------------







---------------------------------------------------------edit--------------------------------------------

Early match report page, the players won't be like that colour wise, just did that for testing purposes.
Image
Find the Latest player inis https://ooper.000webhostapp.com/ut/newinis/ (2019 version)
My latest UT themed game Spawn Killer 4 Game
User avatar
UT Sniper (SJA94)
Adept
 
Posts: 446
Joined: Thu Jun 24, 2010 10:35 pm
Location: England
Personal rank: noob programmer

Re: nodeJS UTStats

Postby UT Sniper (SJA94) » Wed Mar 20, 2019 6:49 pm

Just had a thought, would people be interested if I made a discord bot for my new utstats system.

Example of what I mean:
.stats player <playername>
<url to player page>
<player has played 237 games>
<last played 23 hours 4 minutes ago>
<link to last played game>


.stats gametype <gametype>
<url to gametype match history>
<2104 game played of <gametype>
<last played 21 minutes 5 seconds ago>
<link to last played game>



Sorry for double post, been editing last post last 3 days, wanted to ask in new post about a discord bot.




--------------------------------edit----------------------------------------

Dm scoreboard, doesn't have FPH and pings yet:
Image
Last edited by UT Sniper (SJA94) on Fri Mar 22, 2019 5:22 pm, edited 1 time in total.
Find the Latest player inis https://ooper.000webhostapp.com/ut/newinis/ (2019 version)
My latest UT themed game Spawn Killer 4 Game
User avatar
UT Sniper (SJA94)
Adept
 
Posts: 446
Joined: Thu Jun 24, 2010 10:35 pm
Location: England
Personal rank: noob programmer

Re: nodeJS UTStats

Postby papercoffee » Wed Mar 20, 2019 8:16 pm

UT Sniper (SJA94) wrote:Sorry for double post, been editing last post last 3 days

That's ok ... this doesn't count as double post.
User avatar
papercoffee
Site Staff
 
Posts: 9174
Joined: Wed Jul 15, 2009 11:36 am
Location: Cologne, the city with the big cathedral.
Personal rank: coffee addicted !!!

Re: nodeJS UTStats

Postby UnrealGGecko » Wed Mar 20, 2019 9:02 pm

UT Sniper (SJA94) wrote:Just had a thought, would people be interested if I made a discord bot for my new utstats system.

Example of what I mean:
.stats player <playername>
<url to player page>
<player has played 237 games>
<last played 23 hours 4 minutes ago>
<link to last played game>


.stats gametype <gametype>
<url to gametype match history>
<2104 game played of <gametype>
<last played 21 minutes 5 seconds ago>
<link to last played game>


Considering there are some active clans that host servers and hage a Discord channel, I say there might be interest on this. Just gotta let them know.

Heck, I might try to put it in the ut99org Discord as well, if its easy to put in :mrgreen:
"NHL Rock The Rink" (PS1) announcer wrote:...And there are no stunt doubles in this game, folks!

My work for UT99: Counter-Strike VP, MaleOne+ & FemaleOne+ voicepacks (NEW!), DM-XC-NaliTreeV2 (from the ut99.org Xmas Contest mappack), my small spec of files at Google Drive

List of console converted maps, models & more!
User avatar
UnrealGGecko
Site Staff
 
Posts: 2160
Joined: Wed Feb 01, 2012 11:26 am
Location: Kaunas, Lithuania
Personal rank: GEx the Gecko

Re: nodeJS UTStats

Postby UT Sniper (SJA94) » Fri Mar 22, 2019 6:16 pm

Just a preview video of a DM game being played at 6 seconds a second. I have made some changes to the server module which will enhance this feature more in the future which will includes pickup logging, which weapon the player killed the other player with, and also added support for smartCTF/DM which will save the players faces/netspeed/FOV which will later be implemented in SmartCTF/DM scoreboards.

The user interacting with the match replay can choose which player they want to view, which will effect what sounds will be played, hud messages, and so on. I haven't implemented pick up hud, you were killed by, as well as timers, and other stats for the gameplay modeimg.





Here is sshot at end of game from NUTStats:
Image

Here is a in game screenshot of the same match, the small difference in Oopers kills is me forgetting to making the player lose a point for suicides, atm it counts suicides as the player has killed someone.

Image







------------------------------------------edit------------------------------

System now automatically imports weapons from log files and save them to database(kill strings):

Image
Find the Latest player inis https://ooper.000webhostapp.com/ut/newinis/ (2019 version)
My latest UT themed game Spawn Killer 4 Game
User avatar
UT Sniper (SJA94)
Adept
 
Posts: 446
Joined: Thu Jun 24, 2010 10:35 pm
Location: England
Personal rank: noob programmer

Re: nodeJS UTStats

Postby UT Sniper (SJA94) » Mon Mar 25, 2019 9:53 pm

Match replay now includes item pickups, which includes the sounds for that item when they are picked up. Updated gameplay HUD, as well as some other stuff for the replay system which you can see in this video:




























----------------------------------------edit------------------------------------------

- Added support for Assault.
- Added support for Domination.











--------------------------------edit------------------------

Image
Find the Latest player inis https://ooper.000webhostapp.com/ut/newinis/ (2019 version)
My latest UT themed game Spawn Killer 4 Game
User avatar
UT Sniper (SJA94)
Adept
 
Posts: 446
Joined: Thu Jun 24, 2010 10:35 pm
Location: England
Personal rank: noob programmer

Re: nodeJS UTStats

Postby UT Sniper (SJA94) » Mon Apr 01, 2019 4:03 pm

Just a sneak peak of a match replay with the smartCTF scoreboard, also updated the "gameplay" HUD. Just need to add support for LMS and Assault to the match replay(won't take long), then I'll work on the design of the website.



Ignore the fact it's a CTF match with a DM image as the background, haven't added that functionality yet, I need to add import functions for map Images and faces.





------------------------------edit-------------------------------------


Now supports DOM point captures for match replay:

Image

Also added LMS support

Image


Just need to do assault now.
Find the Latest player inis https://ooper.000webhostapp.com/ut/newinis/ (2019 version)
My latest UT themed game Spawn Killer 4 Game
User avatar
UT Sniper (SJA94)
Adept
 
Posts: 446
Joined: Thu Jun 24, 2010 10:35 pm
Location: England
Personal rank: noob programmer

Re: nodeJS UTStats

Postby UT Sniper (SJA94) » Sun Apr 14, 2019 2:33 am

Just a screenshot of a more updated homepage, was doing another project for someone so didn't put much time into nutstats for a while:

Image
Find the Latest player inis https://ooper.000webhostapp.com/ut/newinis/ (2019 version)
My latest UT themed game Spawn Killer 4 Game
User avatar
UT Sniper (SJA94)
Adept
 
Posts: 446
Joined: Thu Jun 24, 2010 10:35 pm
Location: England
Personal rank: noob programmer

Re: nodeJS UTStats

Postby iSenSe » Sun Apr 14, 2019 4:48 pm

This looks very awesome, and I am very interested in this for our server! :agree1:
Image
User avatar
iSenSe
Adept
 
Posts: 281
Joined: Mon Aug 31, 2015 12:58 pm

Re: nodeJS UTStats

Postby UT Sniper (SJA94) » Wed Apr 17, 2019 1:38 pm

Just a sshot of a page from top to bottom, I removed stuff via developer tools in browser so you get to see the whole design of a page. There are totals from both all time stats, and records, including more per category then is displayed for single matches:

Image






Callback hell :sad2: :
Code: Select all
mysql.query(killQuery,(err, result) =>{

            if(err) throw err;

            if(result.length > 0){

                for(let i = 0; i < result.length; i++){
                    this.allTimeRecords.kills.push(result[i]);
                }

               
            }

            mysql.query(deathsQuery, (err, result) =>{
                if(err) throw err;

                if(result.length > 0){
                    for(let i = 0; i < result.length; i++){
                        this.allTimeRecords.deaths.push(result[i]);
                    }
                }

                mysql.query(scoreQuery, (err, result) =>{
                    if(err) throw err;

                    if(result.length > 0){
                        for(let i = 0; i < result.length; i++){

                            this.allTimeRecords.points.push(result[i]);
                        }
                    }

                    mysql.query(suicidesQuery, (err, result) =>{
                        if(err) throw err;
   
                        if(result.length > 0){
                            for(let i = 0; i < result.length; i++){
   
                                this.allTimeRecords.suicides.push(result[i]);
                            }
                        }

                        mysql.query(headshotsQuery, (err, result) =>{
                            if(err) throw err;
       
                            if(result.length > 0){
                                for(let i = 0; i < result.length; i++){
       
                                    this.allTimeRecords.headshots.push(result[i]);
                                }
                            }

                            mysql.query(spreeQuery, (err, result) =>{
                                if(err) throw err;
           
                                if(result.length > 0){
                                    for(let i = 0; i < result.length; i++){
           
                                        this.allTimeRecords.bestSpree.push(result[i]);
                                    }
                                }
                            });
                            mysql.query(multiQuery, (err, result) =>{
                                if(err) throw err;
           
                                if(result.length > 0){
                                    for(let i = 0; i < result.length; i++){
           
                                        this.allTimeRecords.bestMulti.push(result[i]);
                                    }
                                }

                                mysql.query(damageQuery, (err, result) =>{
                                    if(err) throw err;
               
                                    if(result.length > 0){
                                        for(let i = 0; i < result.length; i++){
               
                                            this.allTimeRecords.damage.push(result[i]);
                                        }
                                    }
                                   // this.getMatchRecords();
                                   
                                   // console.log(this.allTimeRecords);
                                });
                            });
                        });
                    });
                });
            });
        });*/


Changed to this:

Code: Select all
//only render the page when all data is loaded
    getResult(query, index, bAllTime){

        let data = [];
        mysql.query(query, (err, result) =>{

            if(err) throw err;
            if(result.length > 0){
                for(let i = 0; i < result.length; i++){
                    data.push(result[i]);
                }
            }

           

            if(bAllTime){
                this.allTimeRecords[index] = data;
                this.currentAllTimeQueries++;
            }else{
                this.matchRecords[index] = data;
            }

            if(this.allTimeQueries == this.currentAllTimeQueries){
                this.res.render("totals",{"allTime":this.allTimeRecords,"match":this.matchRecords});
            }
           
            return data;
        });
    }

Code: Select all

this.getResult(killQuery,"kills", true);
this.getResult(deathsQuery,"deaths", true);
this.getResult(scoreQuery,"points", true);
this.getResult(suicidesQuery,"suicides", true);
this.getResult(headshotsQuery,"headshots", true);
this.getResult(spreeQuery,"bestSpree", true);
this.getResult(multiQuery,"bestMulti", true);
this.getResult(damageQuery,"damage", true);











-------------------edit------------------------------------

Finished player search page
Image
Find the Latest player inis https://ooper.000webhostapp.com/ut/newinis/ (2019 version)
My latest UT themed game Spawn Killer 4 Game
User avatar
UT Sniper (SJA94)
Adept
 
Posts: 446
Joined: Thu Jun 24, 2010 10:35 pm
Location: England
Personal rank: noob programmer

Next

Return to General Discussions

Who is online

Users browsing this forum: No registered users and 2 guests