Find the answer to your Linux question:
Results 1 to 9 of 9
In an SFTP communication, the status of the port is always hanging in FIN_WAIT2. My understanding about FIN_WAIT2 is that this socket is waiting on other side to send final ...
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1
    Just Joined!
    Join Date
    May 2013
    Posts
    5

    FIN_WAIT2 is hanging forever


    In an SFTP communication, the status of the port is always hanging in FIN_WAIT2. My understanding about FIN_WAIT2 is that this socket is waiting on other side to send final FIN command which it is not sending due to some reason. Please correct me if my understanding is not correct ?

    My question is, is there a way these FIN_WAIT2 can be removed, I believe that this can be controlled by setting timeout in /proc/sys/net/ipv4/tcp_fin_timeout, but there is a default time of 60 sec. in there but even after 60 sec. the sockets are in the same state of FIN_WAIT2. How can we get rid of this state. Please note that we do not want to manually killing the port or restarting the application.

    Thanks in Advance.

  2. #2
    Just Joined! snelson's Avatar
    Join Date
    Nov 2006
    Posts
    3
    Take a peek at this web entry..

    1. Go to serverfault.com/questions/7689/how-do-i-get-rid-of-sockets-in-fin-wait1-state

    Also:

    2. The FIN_WAIT2 state socket remains allocated to the program until finished or aborted.
    Check for missing close statement on the FIN_WAIT2 side of the TCP/IP communication. Use lsof from a root account using the command:

    # lsof | grep tcp | grep <first network port from netstat -an in FIN_WAIT2 state>

    Hope it helps.
    Last edited by snelson; 05-03-2013 at 12:41 PM.

  3. #3
    Just Joined!
    Join Date
    May 2013
    Posts
    5
    The FIN_WAIT2 state socket remains allocated to the program until finished or aborted.
    Does this statement mean that the socket will remain in FIN_WAIT2 until the other side send the close command ?

    I read in the web entry, it looks that we can handle FIN_WAIT2 using /proc/sys/net/ipv4/tcp_keepalive_time. Does this mean that after this time FIN_WAIT2 will go away even if the other side does not close the command ?

  4. $spacer_open
    $spacer_close
  5. #4
    Just Joined! snelson's Avatar
    Join Date
    Nov 2006
    Posts
    3
    The first thing is to find out what application is causing the FIN_WAIT2 to remain active. By issuing something like:

    1. lsof -np $(pidof node) | grep tcp

    or

    2. lsof | grep tcp | grep <first network port from netstat -an in FIN_WAIT2 state>

    You will get an understanding of what application is causing the FIN_WAIT2 state. Once you understand the cause, we can work on the solution.

  6. #5
    Just Joined!
    Join Date
    May 2013
    Posts
    5
    It is given that some of the sockets are in FIN_WAIT2, our requirement is to handle FIN_WAIN2 rather than fixing the application (it is because we do not have control on that), I want to know how can we handle FIN_WAIT2 once there is huge list piled up of these FIN_WAIT2s.

    I understand that lsof will give what application is holding FIN_WAIT2, what I want to know can it be handles at all, if yes how ?
    will tcp_keepalive_time help here.

    Please note it is a given fact that the other side (in CLOSE_WAIT) has some bug and it will never send the final FIN command in this case is it possible to handle FIN_WAIT2.

  7. #6
    Just Joined! snelson's Avatar
    Join Date
    Nov 2006
    Posts
    3
    OK then.. Just remember the server is waiting for an ACK (acknowledgement) for the FIN (Finish) it sent.

    Go ahead and try adjusting the tcp_keepalive_time to something other that 2 hours. However, I suspect you may find yourself adjusting
    tcp_fin_timeout, tcp_orphan_retries and tcp_max_orphans.

    1. echo 600 > /proc/sys/net/ipv4/tcp_keepalive_time
    2. echo 15 > /proc/sys/net/ipv4/tcp_fin_timeout
    3. echo 8192 > /proc/sys/net/ipv4/tcp_max_orphans
    4. echo 1 > /proc/sys/net/ipv4/tcp_orphan_retries

    Good luck.

  8. #7
    Just Joined!
    Join Date
    May 2013
    Posts
    5
    So, if we adjust above parameter than it is possible to remove FIN_WAIT2 without even client sending the FIN command.
    But, isn't it violation of TCP protocol as in FIN_WAIT2 can be removed even when the client does not send FIN command, by adjusting the above parameter ?

    OK then.. Just remember the server is waiting for an ACK (acknowledgement) for the FIN (Finish) it sent.
    Please correct me if I am wrong, FIN_WAIT2 is the state when server gets and ACK and server gets a FIN from the client. that means here the server has already got the ACK for its FIN from the client, while it is waiting for the client to send its FIN.

    In this case also client is not sending the FIN and that is the reason it remains in the FIN_WAIT2 state.

    Thanks

  9. #8
    Just Joined!
    Join Date
    May 2013
    Posts
    5
    I create a simple client and a server, I closed the server side socket but did not close the client socket, I made sure that the client is in CLOSE_WAIT and the SERVER is in FIN_WAIT2, I saw that after /proc/sys/net/ipv4/tcp_fin_timeout (60 sec) FIN_WAIT2 were gone. This works correctly in an standalone application.

    Is there any use case where these FIN_WAIT2 will remain in this state even after the /proc/sys/net/ipv4/tcp_fin_timeout is 60 sec ? Or is it definite that after /proc/sys/net/ipv4/tcp_fin_timeout FIN_WAIT2 will surely go away no matter what ?

    Thanks

  10. #9
    Just Joined!
    Join Date
    Nov 2013
    Posts
    1
    Hello, I got stuck with the same problem. My application is using Websocket over TCP and transfers MBs of data in a go. So the connection stays for an average time of 1 minute. After that either the client OR server may initiate a dis-connection as the case may be.
    Everything is working fine upto application functionality. Problem is, my server socket stuck in FIN_WAIT2 state for life time until I kill my server process (running on port#8095). So i can see multiple connections in FIN_WAIT2 state. Following is the output of 'netstat -na | grep 8095'

    [root@localhost ~]# netstat -na | grep 8095
    tcp 0 0 ::ffff:172.31.209.18:8095 :::* LISTEN
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:59.182.155.150:20858 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.27:6071 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.27:6570 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.7:32440 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:59.182.155.150:20721 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.119:47107 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.27:15202 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.27:19023 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.119:48941 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.7:13325 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.7:10760 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.7:24342 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.12:10222 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.12:1778 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.64:39367 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.24:58245 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.12:49259 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.80:1083 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.12:2918 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.124:34061 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.12:53370 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.88:11815 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.88:47397 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.64:12556 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.64:15456 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.12:58406 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.10:53305 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.12:18495 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.80:42849 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.64:27007 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.64:60737 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.10:20746 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.64:30023 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.12:17950 FIN_WAIT2
    tcp 0 0 ::ffff:172.31.209.18:8095 ::ffff:49.32.0.40:47676 FIN_WAIT2
    The client application has been shut down after completing the application flow. Following is the TCP kernel tuning on my Red Hat Enterprise Linux Server release 5.8 (Tikanga) running on HP Proliant G8 server with 192 GB of RAM. No other process is running on this system.
    [sysctl.conf]

    # Kernel sysctl configuration file for Red Hat Linux
    #
    # For binary values, 0 is disabled, 1 is enabled. See sysctl( and
    # sysctl.conf(5) for more details.

    # Controls IP packet forwarding
    net.ipv4.ip_forward = 0

    # Controls source route verification
    net.ipv4.conf.default.rp_filter = 1

    # Do not accept source routing
    net.ipv4.conf.default.accept_source_route = 0

    # Controls the System Request debugging functionality of the kernel
    kernel.sysrq = 0

    # Controls whether core dumps will append the PID to the core filename
    # Useful for debugging multi-threaded applications
    kernel.core_uses_pid = 1

    # Controls the use of TCP syncookies
    net.ipv4.tcp_syncookies = 1


    # Controls the maximum size of a message, in bytes
    kernel.msgmnb = 65536

    # Controls the default maxmimum size of a mesage queue
    kernel.msgmax = 65536

    # Controls the maximum shared segment size, in bytes
    kernel.shmmax = 68719476736

    # Controls the maximum number of shared memory segments, in pages
    kernel.shmall = 4294967296

    # Default max queues syatem wide = 1024
    kernel.msgmni = 2048

    #Maximum number of file descriptor allowed
    fs.file-max = 1048576



    # The Linux autotuning TCP buffer limit
    # Defines per-socket memory usage for auto-tuning. The third value is the maximum send buffer space
    # (overridden by r/wmem_max).
    # The receive buffers for auto-tuning
    net.ipv4.tcp_rmem = 4096 87380 16777216

    # The write (send) buffers for auto-tuning
    net.ipv4.tcp_wmem = 4096 65536 16777216

    # Determines how the TCP stack should behave for memory usage; each count is in memory pages (typically 4KB).
    # Increase the count for large BDP (but remember, it's memory pages, not bytes).
    net.ipv4.tcp_mem = 8388608 8388608 8388608

    # For 10G NIC
    net.core.netdev_max_backlog = 32768

    net.ipv4.tcp_congestion_control = htcp

    # Recommended for hosts with jumbo frames enabled
    net.ipv4.tcp_mtu_probing = 1

    # Enables window scaling as defined by RFC 1323. Must be enabled to support windows larger than 64KB.
    net.ipv4.tcp_window_scaling = 1


    # Enables selective acknowledgment, which improves performance by selectively acknowledging packets received out of order
    # (causing the sender to retransmit only the missing segments), but it can increase CPU utilization.
    net.ipv4.tcp_sack = 1

    #
    net.ipv4.tcp_fack = 1

    net.ipv4.tcp_timestamps = 1

    net.ipv4.tcp_low_latency = 1

    net.ipv4.tcp_westwood = 1

    net.ipv4.tcp_bic = 1

    net.ipv4.tcp_max_tw_buckets = 1440000

    net.ipv4.tcp_tw_recycle = 1

    net.ipv4.tcp_tw_reuse = 1

    net.ipv4.tcp_fin_timeout = 5

    # Limit of socket listen() backlog.Defaults to 128. See also tcp_max_syn_backlog for additional tuning
    net.ipv4.somaxconn = 4096

    net.core.somaxconn = 4096

    # Maximal number of remembered connection requests, which are still did not receive an acknowledgment from connecting client
    net.ipv4.tcp_max_syn_backlog = 4096

    #
    net.ipv4.tcp_no_metrics_save = 1

    net.ipv4.tcp_keepalive_intvl = 30

    net.ipv4.tcp_keepalive_time = 400

    net.ipv4.tcp_keepalive_probes = 5

    vm.swappiness = 20
    net.ipv4.tcp_max_orphans = 5
    net.ipv4.tcp_orphan_retries = 2
    Please help and let me know if additional information is required regarding this.

    Thanx in advance.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •