Welcome back, followers of the fearsome!
Over the past few weeks we’ve been working on making the game better, based on what was reported through crash logs, forum posts and emails coming in from helpful customers. We’ve fixed a few (rare) crash issues, and there are a few more fixes coming soon.
One of the more ‘interesting’ problems we heard of through emails were network connection issues on the Playstation 4.
Our game uses a peer-to-peer ‘mesh’ setup, which means every client needs to be able to talk to every other client. The mesh starts by one person starting in ‘host’ mode. Then up to two other players connect to this host, and to any other players already in the network.
Occasionally it turns out that while players can connect to the host, they cannot get any messages through to the other player in the game, even though the NAT negotiation library seems to indicate it can. I’m at a bit of a loss as to why exactly, since we have never seen this exact behavior in our test setups. It probably has something to do with firewalls/NAT traversal, but it’s hard to say exactly if we can’t reproduce this problem in-house. Interestingly enough, we’ve never had complaints from our Steam customers, but that is likely because Steam has their own routing-service that automatically sends messages through a ‘known’ steam server if it can’t send directly.
After a bit of investigation on what to do about this problem, I decided to send each client-client network packet directly to the remote client (if a connection is available) as well as through the host. The one sent to the host is marked as a ‘routing’ packet, meaning the host is expected to forward it to the other client. On the receiving client, because of the double message send, we now often see two of the same packets coming in at different times. This is not a problem because we automatically reject duplicate packets based on the packet ID.
There are two downsides to this approach: Double the latency, and double the network bandwidth. Luckily our game was designed to be very latency resistant. We almost always test with 250-350ms of lag in our test setups. Even with 350ms of lag, the lag is only really noticeable occasionally, and usually not really a problem.
The bandwidth of our game is quite low. The total host bandwidth with two connected clients is about 30kbps on average, which is quite low compared to most modern games. Because of the bitpacked network packets, and the use of the ‘data-blobs’ which use binary delta compression, the bandwidth is kept to a minimum. So doubling the bandwidth to around 60kbps is still well within the acceptable range.
For a minute I was thinking I should just not send client messages directly to other clients at all, and always route them through the host. This would increase latency, but the bandwidth would remain the same as before (30kbps). But because the bandwidth is already so low, I decided to always send two packets so we get lower latency for clients that are able to communicate.
I could probably improve it further by only sending messages through the host when it is detected that the client-client connection isn’t working, but that’d introduce a time frame where the game would feel ‘locked up’ until the messages were finally routed through the host, which would create a pretty bad user experience.
I might change it around for other games, depending on the requirements, but for now I think the current routing system will fix the issues with the PS4 networking. We’re submitting a patch soon, but we have lots of testing to be done to make sure it’s robust first!
Until next week, keep those packets routing!
As usual, today at 4pm there will be another Dev Stream with Jouste the Drawbarian. Don’t miss it!