Reducing lag / TCP_NODELAY

[tueidj]

Does Sylverant's ship code enable TCP_NODELAY on client connections to reduce lag?
I get pretty bad lag when picking up items, I assumed it was because my ping time is roughly 210ms but after being hit countless times because I can't attack (A button action overridden to "pick-up item") I think I need to investigate deeper.

[BlueCrab]

I've noticed a rather large amount of lag that I experience when connecting via Devolution that I don't experience with a direct connection. I couldn't ever track down the exact cause, but it is quite a noticeable difference on my end.

My best guess was that there was some kind of nasty network issue that exists somewhere along the line between the PPP emulator/server and my server. Lately, I've seen a few people where the lag gets so bad that they end up tripping the ping out timer on the server (which means there is more than 2 minutes of delay). The one common thread amongst most of them is that they're connecting via Devolution.

Perhaps we could discuss at some point how I'd go about setting one of them up on Sylverant's server itself so we could see if that fixes the lag issue?

That said, I'm not entirely convinced that TCP_NODELAY would make much difference. I could certainly try enabling it, but I don't think that's actually causing the problem, especially not with the particular behavior you've described up there. Basically, the TCP_NODELAY option really only affects situations where you have a pattern of send-send-receive. When doing anything in a game with only one client, that situation will almost never come up from the server-side. Even with multiple clients in the game, the situation doesn't frequently drop to the send-send-receive pattern that can cause issues with Nagle's algorithm (as you will often have several sends in a row due to all of the clients interacting at the same time).

[tueidj]

I discounted anything to do with Devolution because I get the same lag with Dolphin, using BBA emulation (which uses a network tap). My VPS can be unstable at times (it's a VPS after all) but when it's behaving there are no issues between it and sylverant:

PING 67.222.144.120 (67.222.144.120) 56(84) bytes of data.
64 bytes from 67.222.144.120: icmp_req=1 ttl=52 time=32.3 ms
64 bytes from 67.222.144.120: icmp_req=2 ttl=52 time=32.5 ms
64 bytes from 67.222.144.120: icmp_req=3 ttl=52 time=32.3 ms
64 bytes from 67.222.144.120: icmp_req=4 ttl=52 time=32.4 ms

However I first have to reach it from the other side of the pacific:

Pinging tueidj.net [199.195.142.242] with 32 bytes of data:
Reply from 199.195.142.242: bytes=32 time=178ms TTL=52
Reply from 199.195.142.242: bytes=32 time=179ms TTL=52
Reply from 199.195.142.242: bytes=32 time=179ms TTL=52
Reply from 199.195.142.242: bytes=32 time=178ms TTL=52

Compared to reaching Sylverant directly:

Pinging sylverant.net [67.222.144.120] with 32 bytes of data:
Reply from 67.222.144.120: bytes=32 time=238ms TTL=48
Reply from 67.222.144.120: bytes=32 time=238ms TTL=48
Reply from 67.222.144.120: bytes=32 time=239ms TTL=48
Reply from 67.222.144.120: bytes=32 time=239ms TTL=48

If you wanted to test running a ppp emulator directly on Sylverant it's pretty simple, I just use socat to listen and pass incoming connections to new slirp instances (both of which use TCP_NODELAY - socat supports it via an option, slirp had to be patched). Devolution can be told to connect to any ip:port using the dial-up number; concatenate the ip and port to form a 12 digit hexadecimal number and convert it to decimal (also remember to manually set the DNS).

I'll try spraying some logging around at both ends to try and spot exactly where the data is currently being delayed.

[BlueCrab]

Hrm... Yeah, your results for pinging sylverant from your server are consistent with mine the other way, now that I'm looking at it...

$ ping 199.195.142.242
PING 199.195.142.242 (199.195.142.242): 56 data bytes
64 bytes from 199.195.142.242: icmp_seq=0 ttl=50 time=32.410 ms
64 bytes from 199.195.142.242: icmp_seq=1 ttl=50 time=32.529 ms
64 bytes from 199.195.142.242: icmp_seq=2 ttl=50 time=32.332 ms
64 bytes from 199.195.142.242: icmp_seq=3 ttl=50 time=32.502 ms
^C
--- 199.195.142.242 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 32.332/32.443/32.529/0.078 ms

That said, you've definitely got a lot more lag connecting to the server directly than do I:

$ ping sylverant.net
PING sylverant.net (67.222.144.120): 56 data bytes
64 bytes from 67.222.144.120: icmp_seq=0 ttl=248 time=59.108 ms
64 bytes from 67.222.144.120: icmp_seq=1 ttl=248 time=53.406 ms
64 bytes from 67.222.144.120: icmp_seq=2 ttl=248 time=57.818 ms
64 bytes from 67.222.144.120: icmp_seq=3 ttl=248 time=52.181 ms
64 bytes from 67.222.144.120: icmp_seq=4 ttl=248 time=56.576 ms
^C
--- sylverant.net ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 52.181/55.818/59.108/2.625 ms

[tueidj]

That's to be expected when it's a 28,900 km round trip.

[tueidj]

Here's a basic log of what goes in/out of the wii when I pick up an item:

4163537681: 84 bytes written, 0/0
4177641833: 64 bytes received, 64/0
4179774574: 65 bytes written, 0/0
4181790276: 85 bytes written, 0/0
4204831204: 48 bytes received, 48/0

The first value is the current tick count; divide it by 60750000 to convert to seconds. The values at the end of the line are the length of the in/out queues (bytes waiting to be processed by the game/sent out on the wire).

The first line appears when I press A to pick up the item.
The next three lines occur together and the item gets picked up on screen.
The last line always shows up a short while later.

So checking the time between the first and second messages gives (4177641833-4163537681)/60750000 = 232ms, which is only a 10% difference from the expected round trip time It also matches what I get when pinging Sylverant directly or playing online using Dolphin.
This just makes me more confused why anyone in the US would see extra lag whilst using Devolution...

[BeBeZiNhA]

Sorry for butting in here, but then would that be the reason of the lag that I see in my game whenever someone enters the server?

[BlueCrab]

Huh? You mean the spinning hexagons with the "[Player] is joining, please wait" message? That has nothing to do with this, if that's what you mean.

That's the game freezing everyone else to give a chance to synchronize that new client into the game.

[BlueCrab]

Here's a basic log of what goes in/out of the wii when I pick up an item:

4163537681: 84 bytes written, 0/0
4177641833: 64 bytes received, 64/0
4179774574: 65 bytes written, 0/0
4181790276: 85 bytes written, 0/0
4204831204: 48 bytes received, 48/0

The first value is the current tick count; divide it by 60750000 to convert to seconds. The values at the end of the line are the length of the in/out queues (bytes waiting to be processed by the game/sent out on the wire).

The first line appears when I press A to pick up the item.
The next three lines occur together and the item gets picked up on screen.
The last line always shows up a short while later.

So checking the time between the first and second messages gives (4177641833-4163537681)/60750000 = 232ms, which is only a 10% difference from the expected round trip time It also matches what I get when pinging Sylverant directly or playing online using Dolphin.
This just makes me more confused why anyone in the US would see extra lag whilst using Devolution...

Very odd. I noticed a severe amount of lag the last time I tried to play for any period of time (granted, that was a few weeks back). Could have just been some kind of network hiccup along the way... It does look like the issue has mostly cleared up with people pinging out, now that I'm looking at the logs more closely.

I've been pretty stressed out and lacking time to work on things lately...

[tueidj]

The VPS does tend to "zone out" from time to time (I see it because it runs my IRC bouncers). I assume it's someone else's container on the same machine hogging all the bandwidth, it's only a $3/mth service after all.
It's also possible the ping timeouts are caused by the wii's wifi dropping out then reconnecting which can happen often if the signal's not strong.