1. These forums are still being retired! Please use GitHub discussions or Discord. You cannot create new threads or new accounts now. If you need to contact some user of the forums, you should do it sooner, rather than later. This notice was last updated on May 30th, 2021.

Strange "Deadlocks" occuring recently

Discussion in 'Help and Support' started by pink_panther, Jul 14, 2014.

  1. pink_panther

    pink_panther Level 9

    Joined:
    Oct 14, 2013
    Messages:
    370
    When the server stops responding, but doesnt actually crash. players can still "find server" but not connect.
     
  2. pink_panther

    pink_panther Level 9

    Joined:
    Oct 14, 2013
    Messages:
    370
    Since running marios build, I still haven't seen it lock up.

    What changed exactly?

    I did have some players seem to think it was being caused by 1/3/5 second timers, but I just disregarded them.
     
  3. MarioE

    TShock Admin TShock Mod Plugin Developer

    Joined:
    May 26, 2012
    Messages:
    383
    I made the wiring code thread-safe.
     
  4. pink_panther

    pink_panther Level 9

    Joined:
    Oct 14, 2013
    Messages:
    370
    thing is though, it only started recently. Also it didn "dead" lock, was it would sometimes come back to normal

    Does that sound right?
     
  5. MarioE

    TShock Admin TShock Mod Plugin Developer

    Joined:
    May 26, 2012
    Messages:
    383
    Well, thread-safety issues don't necessarily have to result in deadlocks...

    For example, suppose I have
    Code:
    public static int counter = 0;
    and thread #1 is running
    Code:
    for (; counter < 1000; counter++)
    {
        TSPlayer.All.SendInfoMessage("{0}", counter);
    }
    and thread #2 is running
    Code:
    counter = 0;
    If thread #2 executes once in the middle of the for loop in thread #1, it'll reset the loop counter and cause it to do more than it was supposed to do. It won't dead lock, however.
     
  6. pink_panther

    pink_panther Level 9

    Joined:
    Oct 14, 2013
    Messages:
    370
    That being said, could it result in halts, like im describing? Just stalling for a random amount of time, between a few seconds and a few minutes before proceeding?

    I've actualyl had some people say it is actually still happening. I didnt notice it last night, but ill watch it again tonight.
     
  7. Wolfje

    TShock Admin TShock Mod Zero Day Plugin Author

    Joined:
    Jul 2, 2013
    Messages:
    191
    It's more of a race-condition. Deadlocking occurs when two or more threads wait on eachother to advance and therefore will wait on eachother forever. What's happening is more like a race condition, being that something can change half way through executing something else at the same time, causing loops etc to whizz around far more times than they need to, blocking up the threads that are supposed to be executing other code.

    Because of the way TerrariaServerAPI's threadpool model works, the symptoms of problems like this happening are exactly the same as a deadlock because the end result is kind of the same: a thread gets clogged up doing things for far too long. This however is not a deadlocking issue, although I would have never ever picked the wiring functionality to be causing it.

    This is why people report issues with 'deadlocking', without knowing the difference, all people are going to do is report deadlocking because the symptoms are the same.

    --

    By the looks of this, only servers making heavy use of wiring will experience this?
     
  8. MarioE

    TShock Admin TShock Mod Plugin Developer

    Joined:
    May 26, 2012
    Messages:
    383
    It was pretty similar to a problem that had occurred before: https://github.com/Deathmax/TerrariaAPI-Server/pull/29

    I examined Simon311's dump during one of these "deadlock" problems and noticed that several threads were at wiring methods as well. Unfortunately, I'm not certain that only servers making heavy use of wiring will be affected by this.
     
  9. Wolfje

    TShock Admin TShock Mod Zero Day Plugin Author

    Joined:
    Jul 2, 2013
    Messages:
    191
    That's weird. None of the dumps I ever took had anything to do with the wiring mechanism at all. The main thread was usually on the update handler or some sort of I/O for TShock, a socket accept thread and a few other bits and pieces.

    I get why ThreadStatic will eliminate racing, but nothing from any of the dumps (there were about 5) would lead me anywhere in TSAPI itself.
     
  10. Wolfje

    TShock Admin TShock Mod Zero Day Plugin Author

    Joined:
    Jul 2, 2013
    Messages:
    191
    There was one thing I found with all the dumps that I took, and that's this thread from an SOS !EEStack

    Code:
    0b5be220 6f389c57 (MethodDesc 6f1c9844 +0x3b DomainBoundILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags))
    0b5be248 6f389c57 (MethodDesc 6f1c9844 +0x3b DomainBoundILStubClass.IL_STUB_PInvoke(IntPtr, Byte*, Int32, System.Net.Sockets.SocketFlags))
    0b5be264 6f357ced (MethodDesc 6f1c7278 +0xbd System.Net.Sockets.Socket.Send(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef)), calling 6f2bec40
    0b5be270 7193ae54 clr!CoUninitializeEE+0xb37c, calling clr!DllUnregisterServerInternal+0x1966
    0b5be28c 6f357c0d (MethodDesc 6f1c726c +0x1d System.Net.Sockets.Socket.Send(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags)), calling (MethodDesc 6f1c7278 +0 System.Net.Sockets.Socket.Send(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags, System.Net.Sockets.SocketError ByRef))
    0b5be2b0 6f357b33 (MethodDesc 6f1c0734 +0x83 System.Net.Sockets.NetworkStream.Write(Byte[], Int32, Int32)), calling (MethodDesc 6f1c726c +0 System.Net.Sockets.Socket.Send(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags))
    0b5be2e4 0a325423 (MethodDesc 00c1982c +0x83 TShockAPI.TShock.SendBytesBufferless(Terraria.ServerSock, Byte[]))
    0b5be334 0a32512e (MethodDesc 00c198d4 +0xe6 TShockAPI.TShock.<NetHooks_SendData>b__9(Terraria.ServerSock, System.String)), calling (MethodDesc 00c1982c +0 TShockAPI.TShock.SendBytesBufferless(Terraria.ServerSock, Byte[]))
    0b5be384 0ace5695 (MethodDesc 00c19838 +0x16d TShockAPI.TShock.NetHooks_SendData(TerrariaApi.Server.SendDataEventArgs))
    0b5be5b0 006f448c (MethodDesc 005e69c4 +0x8c TerrariaApi.Server.HandlerCollection`1[[System.__Canon, mscorlib]].Invoke(System.__Canon))
    0b5be5fc 0ace527f (MethodDesc 005e5864 +0x97 TerrariaApi.Server.HookManager.InvokeNetSendData(Int32 ByRef, Int32 ByRef, Int32 ByRef, System.String ByRef, Int32 ByRef, Single ByRef, Single ByRef, Single ByRef, Int32 ByRef)), calling (MethodDesc 005e69c4 +0 TerrariaApi.Server.HandlerCollection`1[[System.__Canon, mscorlib]].Invoke(System.__Canon))
    0b5be610 0ace0df2 (MethodDesc 09ef6a7c +0x8a Terraria.NetMessage.SendData(Int32, Int32, Int32, System.String, Int32, Single, Single, Single, Int32)), calling (MethodDesc 005e5864 +0 TerrariaApi.Server.HookManager.InvokeNetSendData(Int32 ByRef, Int32 ByRef, Int32 ByRef, System.String ByRef, Int32 ByRef, Single ByRef, Single ByRef, Single ByRef, Int32 ByRef))
    0b5be92c 0aced14c (MethodDesc 00c1a0a0 +0xa4 TShockAPI.TSPlayer.SendData(PacketTypes, System.String, Int32, Single, Single, Single, Int32)), calling (MethodDesc 09ef6a7c +0 Terraria.NetMessage.SendData(Int32, Int32, Int32, System.String, Int32, Single, Single, Single, Int32))
    0b5be978 0a325012 (MethodDesc 00c19f94 +0x4a TShockAPI.TSPlayer.Disconnect(System.String))
    0b5be9a0 0a324d90 (MethodDesc 00c1970c +0x180 TShockAPI.TShock.NetHooks_NameCollision(TerrariaApi.Server.NameCollisionEventArgs))
    0b5bea1c 006f448c (MethodDesc 005e69c4 +0x8c TerrariaApi.Server.HandlerCollection`1[[System.__Canon, mscorlib]].Invoke(System.__Canon))
    0b5bea68 0a324bf2 (MethodDesc 005e5894 +0x32 TerrariaApi.Server.HookManager.InvokeNetNameCollision(Int32, System.String)), calling (MethodDesc 005e69c4 +0 TerrariaApi.Server.HandlerCollection`1[[System.__Canon, mscorlib]].Invoke(System.__Canon))
    0b5bea78 0b9b0b91 (MethodDesc 09ef69e4 +0xb41 Terraria.MessageBuffer.GetData(Int32, Int32)), calling (MethodDesc 005e5894 +0 TerrariaApi.Server.HookManager.InvokeNetNameCollision(Int32, System.String))
    0b5bea80 7191ebe3 clr!DllUnregisterServerInternal+0x34b7, calling clr!DllUnregisterServerInternal+0x2a20
    0b5bf2b8 0acef722 (MethodDesc 09ef6ac4 +0xca Terraria.NetMessage.CheckBytes(Int32)), calling (MethodDesc 09ef69e4 +0 Terraria.MessageBuffer.GetData(Int32, Int32))
    0b5bf2f4 0ace6d3e (MethodDesc 005e3e20 +0x41e Terraria.Netplay.ServerLoop(System.Object)), calling (MethodDesc 09ef6ac4 +0 Terraria.NetMessage.CheckBytes(Int32))
    0b5bf388 70c6d30e (MethodDesc 70a21b8c +0x3e System.Threading.QueueUserWorkItemCallback.WaitCallback_Context(System.Object))
    0b5bf390 70c44157 (MethodDesc 709675c8 +0xa7 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
    0b5bf3f4 70c44096 (MethodDesc 709675bc +0x16 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)), calling (MethodDesc 709675c8 +0 System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
    0b5bf408 70c6e1a0 (MethodDesc 709782dc +0x60 System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()), calling (MethodDesc 709675bc +0 System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean))
    0b5bf424 70c6d949 (MethodDesc 709782c4 +0x149 System.Threading.ThreadPoolWorkQueue.Dispatch()), calling 001dc19a
    0b5bf474 70c6d7f5 (MethodDesc 709782f0 +0x5 System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()), calling (MethodDesc 709782c4 +0 System.Threading.ThreadPoolWorkQueue.Dispatch())
    
    All of them had a thread at Socket.Send, after a NetNameCollission event had fired. I don't know if that has anything to do with the blocking of the threadpool, but yeah. All of them taken were at a socket send after a NetNameCollission. I don't have any dumps of a normal operating server.
     
    • Like Like x 1
  11. MarioE

    TShock Admin TShock Mod Plugin Developer

    Joined:
    May 26, 2012
    Messages:
    383
    Hm... I figured disconnecting them with a message would work since it would kill the socket and possibly send a disconnect reason, but apparently that doesn't work well?

    I just pushed a change to it, perhaps that will work?
     
    • Like Like x 1
  12. Wolfje

    TShock Admin TShock Mod Zero Day Plugin Author

    Joined:
    Jul 2, 2013
    Messages:
    191

    Most interesting. What is NetNameCollission anyway, is it a handler when two people try to connect with the same name? Socket.Send() will completely block until all the bytes in the sendq have been sent. Half-open or lingering connections will block until socket timeout. It'll hang about if half the bytes are sent in the buffer and then the client isn't ready to receive bytes anymore.

    Edit: Even if the connection is active, it will still block until the other end receive()s buffered bytes, which will tell TCP to send ACK packets back to the server.

    If I figure out how to trigger it, I can try to reproduce it. The wiring fix may just be coincidence.
     
  13. MarioE

    TShock Admin TShock Mod Plugin Developer

    Joined:
    May 26, 2012
    Messages:
    383
    Yeah, it's when two people try to connect with the same name -- for example, when you lag out and attempt to reconnect. I thought sending a disconnect wouldn't hurt since the server would be trying to send crap like chat, etc to the lingering connection, but apparently it's bad.

    Triggering it's pretty easy, just disable your connection, time out, then reconnect.
     
  14. Wolfje

    TShock Admin TShock Mod Zero Day Plugin Author

    Joined:
    Jul 2, 2013
    Messages:
    191
    Harder to test locally haha
     
  15. Marcus101RR

    Marcus101RR Guest

    So these deadlocks are caused by someone trying to connect at the same time with the same name?
     
  16. MarioE

    TShock Admin TShock Mod Plugin Developer

    Joined:
    May 26, 2012
    Messages:
    383
    No, it's because of the way I was originally disconnecting the clients.
     
  17. pink_panther

    pink_panther Level 9

    Joined:
    Oct 14, 2013
    Messages:
    370
    yeeah well it looks good now. I still have yet to see it happen again running this version of yours
     
    • Like Like x 1
  18. popstarfreas

    Plugin Developer

    Joined:
    Feb 27, 2012
    Messages:
    357
    Personally using that version, I've only had it happen once while I was on. It's not as a common as I remember. However, I believe people still report it happening while I am not online.

    Edit: oh wait; am I suppose to have downloaded the latest general-devel/bamboo build? If that's so, I'll download that. But if its to do with the "connection ghosting" feature, I believe I had a "deadlock" issue before that. Regardless, I'll try.

    Edit2: On the topic of connecting; some players report issues with getting stuck on "Receiving Tile Data: Complete" and never being able to fully-join the server. They say that when they switch characters it works fine.
     
  19. pink_panther

    pink_panther Level 9

    Joined:
    Oct 14, 2013
    Messages:
    370
    No, not bamboo, just that exe mario posted.
     
  20. popstarfreas

    Plugin Developer

    Joined:
    Feb 27, 2012
    Messages:
    357
    Yes I tried that exe he posted and it didn't help much. But if you read the commit changes the comment says "Fix "deadlock" occurring?" which is why I assumed.