@grandeto/vrchats NPM

Description

HTTP/WebSocket real-time message processor based on Socket.io

Requirements

Node.js version: refer to package.json engines.node

Features

HTTP event handler
WebSocket multicast emitter
Publishers: Trusted Origins, CIDRs, PROXY
Subscribers: WebSocket Auth, Rate Limiter

Environment

Tune OS (Linux) and Node.js for websocket connections

https://tcminh.com/600k-concurrent-websocket-connections-on-aws-using-node-js/

sudo nano /etc/security/limits.d/custom.conf

root soft nofile 1000000
root hard nofile 1000000
* soft nofile 1000000
* hard nofile 1000000

sudo nano /etc/sysctl.conf

fs.file-max = 1000000
fs.nr_open = 1000000
net.ipv4.netfilter.ip_conntrack_max = 1048576
net.nf_conntrack_max = 1048576

sudo nano /etc/sysctl.d/net.ipv4.ip_local_port_range.conf

net.ipv4.ip_local_port_range = 10000 65535

check ulimits

ulimit -Sa && echo -e "\n" && ulimit -Ha

set memlock if too low
- example of setting 15.5GB memlock in /etc/docker/daemon.json
  *NOTE: in daemon.json memlock soft:hard values should be in binary bytes representation e.g. 15872000000 B instead of 15500000 KB - https://www.gbmb.org/kilobytes
```
"default-ulimits": {
    "memlock": {
        "Hard": 15872000000,
        "Name": "memlock",
        "Soft": 15872000000
    }
}
```
  *NOTE in /etc/security/limits.d/custom.conf memlock soft:hard values should be in decimal kilobytes representation e.g. 15500000 KB
```
root soft memlock 500000
root hard memlock 500000
```
- soft memlock 15500000
- hard memlock 15500000
Create an .env file from .env_example

chmod 600 .env

Add pubkey.pem, privkey.pem, ca.pem in ./certs

chmod 600 ./certs/*.pem

Whitelist .env ports in firewall

Docker

Prod

Build an image

docker build -t "vrchats:vX.X.X" .

Start a container

docker run -d --name vrchats --restart always -p 8443:8443 -p 2053:2053 vrchats:vX.X.X

applying changes

re-run build step followed by:

docker stop vrchats && docker rm vrchats && docker run -d --name vrchats --restart always -p 8443:8443 -p 2053:2053 vrchats:vX.X.X

*NOTE: Consider adding a cronjob in order to handle https://github.com/grandeto/vrchats#pm2-knows-issues
0 0 * * * docker restart vrchats

Dev

npm run dev.run

npm run dev.remove

npm run dev.update

pm2

Save pm2 settings

pm2 save

pm2 deamon

pm2 startup

Copy and execute the output command generated by pm2 startup then:

sudo nano /etc/systemd/system/pm2-$USER.service and:

add Wants=network-online.target above After=network.target

add RequiresMountsFor=/home above After=network.target

change After=network.target to After=network.target network-online.target

change WantedBy=multi-user.target to WantedBy=multi-user.target network-online.target

add under [Service] the following env variables:

Environment=NODE_ENV=production
Environment=CLUSTER_MODE=1
Environment=CLUSTER_INSTANCES="-1"
Environment=PRODUCER_PORT=2053
Environment=CONSUMER_PORT=8443
Environment=STANDALONE_PORT=8443
Environment=TRUST_ORIGIN_LIST="https://example.com"
Environment=TRUST_CIDR_LIST="123.123.123.123/32,127.0.0.1/32,::1/128"
Environment=USE_PROXY=1
Environment=TRUST_PROXY_LIST="103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,104.16.0.0/13,104.24.0.0/14,108.162.192.0/18,131.0.72.0/22,141.101.64.0/18,162.158.0.0/15,172.64.0.0/13,173.245.48.0/20,188.114.96.0/20,190.93.240.0/20,197.234.240.0/22,198.41.128.0/17,2400:cb00::/32,2606:4700::/32,2803:f800::/32,2405:b500::/32,2405:8100::/32,2c0f:f248::/32,2a06:98c0::/29"
Environment=PUB_KEY_PATH=/var/www/vrchats/pubkey.pem
Environment=PRIV_KEY_PATH=/var/www/vrchats/privkey.pem
Environment=CA_PATH=/var/www/vrchats/ca.pem
Environment=VERIFY_ORIGIN=1
Environment=WEBSOCKET_AUTH_TOKEN_RENEW_START_HOUR=0
Environment=WEBSOCKET_AUTH_TOKEN_RENEW_INTERVAL=86400000
Environment=WEBSOCKET_AUTH_TOKEN_SECRET="some-nasty-secret"

Execute sudo systemctl daemon-reload
reboot
after reboot test the service is operational

pm2 ls - should display list of running instances having status online

pm2 applying changes

pm2 ls && pm2 stop all && pm2 delete all
pm2 save
sudo systemctl restart pm2-$USER
pm2 ls - verify pm2 started successfuly the app instances with new pids

Test connectivity

https://example.com:2053/status - should return 200 OK

https://example.com:8443 - should return 404 "Cannot GET /"

Monitoring

set ENABLE_DEBUG=1
rebuild and deploy
attach to container

pm2 monit and pm2 logs

TODO export and collect metrics
TODO collect log stream

Troubleshooting and known issues

pm2 knows issues

It seems pm2 v5.1.2 has a weird timezone behaviour, as seen from .pm2/pm2.log:
An example from sequent restarts of pm2-$USER service:
- 2021-12-19T14:53:56: PM2 log: --- New PM2 Daemon started - has the correct TZ in UTC
- 2021-12-19T18:24:12: PM2 log: --- New PM2 Daemon started - the next one executed right after the previous one has started all of the nodes, but outputs an incorrect UTC TZ
  Thus, auto-refreshing of the auth_token becomes unreliable and the simplest workaround is a root cron job scheduled to restart the pm2-$USER service daily. For instance:
  0 0 * * * systemctl restart pm2-$USER
pm2 docs - journalctl -u pm2-$USER.service
stackoverflow 43786412
If your home directory is encrypted in order to demonize with pm2 startup all the node.js, npm and app raleted files should be outside of the home dir or try link1 link2 link3