Find the answer to your Linux question:
Results 1 to 2 of 2
Enjoy an ad free experience by logging in. Not a member yet? Register.
  1. #1

    Unhappy Extracting errored line from the debug output of a Linux Kernel


    I have implemented a mechanism to migrate TCP socket between different severs in Linux OS. The migration mechanism works perfectly, except when the importing server needs to close the server the whole machine freezes. I was able to dump the log file of the machine using serial connection as explained in this article Debugging Linux Kernel. From the reading of the log file, I saw that NULL pointer dereference problem is occurring in the tcp_time_wait function. So my current problem now I need to know how to figure out which data member or macro inside the tcp_time_wait is causing the problem?

    Should I recompile the Kernel with some printk to detect the exact source of error?

    Linux Kernel Dump:
    Code:
    [  225.717049] BUG: unable to handle kernel NULL pointer dereference at 0000000000000020
    [  225.720544] IP: [<ffffffff8162056c>] __inet_twsk_hashdance+0x8c/0x160    
    [  225.720544] PGD 7bbf9067 PUD 7b7fc067 PMD 0                              
    [  225.720544] Oops: 0000 [#1] SMP                         
    [  225.720544] Modules linked in: sockmi(OF) nf_conntrack(F) vesafb(F) vboxsf(OF) snd_int)
    [  225.720544] CPU 0                                       
    [  225.720544] Pid: 0, comm: swapper/0 Tainted: GF       W  O 3.8.0-29-generic #42~precisx
    [  225.720544] RIP: 0010:[<ffffffff8162056c>]  [<ffffffff8162056c>] __inet_twsk_hashdance0
    [  225.720544] RSP: 0018:ffff88007fc039d0  EFLAGS: 00010282
    [  225.720544] RAX: 0000000000000000 RBX: ffff88005cbd5000 RCX: ffffffff81e304c0
    [  225.720544] RDX: 0000000000001c37 RSI: 0000000000000082 RDI: 0000000000000009
    [  225.720544] RBP: ffff88007fc03a00 R08: 000000000000000a R09: 0000000000000000
    [  225.720544] R10: 000000000000021a R11: 0000000000000219 R12: ffff88007ba4b800         
    [  225.720544] R13: ffffc9000035a7d0 R14: ffffc90000322500 R15: ffff88007c1e3d40         
    [  225.720544] FS:  0000000000000000(0000) GS:ffff88007fc00000(0000) knlGS:000000000000000
    [  225.720544] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b                         
    [  225.720544] CR2: 0000000000000020 CR3: 000000007965f000 CR4: 00000000000006f0         
    [  225.720544] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000         
    [  225.720544] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
    [  225.720544] Process swapper/0 (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c15)
    [  225.720544] Stack:                                                                    
    [  225.720544]  ffff88007ba4b800 ffff88007ba4b800 ffff88005cbd5000 0000000000000000      
    [  225.720544]  000000000000001f 0000000000000000 ffff88007fc03a50 ffffffff8163de3c      
    [  225.720544]  ffff8800000000d9 000000065cb8b300 ffff88003652e262 ffff88007ba4b800      
    [  225.720544] Call Trace:                                                               
    [  225.720544]  <IRQ>                                                                    
    [  225.720544]  [<ffffffff8163de3c>] tcp_time_wait+0x1bc/0x290                           
    [  225.720544]  [<ffffffff8162d4fe>] tcp_fin+0x10e/0x1c0                                 
    [  225.720544]  [<ffffffff8162e078>] tcp_data_queue+0x3e8/0x580                          
    [  225.720544]  [<ffffffff81631bb5>] tcp_rcv_state_process+0x2b5/0x6b0                   
    [  225.720544]  [<ffffffff8163abc7>] tcp_v4_do_rcv+0xc7/0x220                            
    [  225.720544]  [<ffffffff8163c829>] tcp_v4_rcv+0x569/0x830                              
    [  225.720544]  [<ffffffff81616366>] ip_local_deliver_finish+0xe6/0x280                  
    [  225.720544]  [<ffffffff8161668a>] ip_local_deliver+0x4a/0x90
    [  225.720544]  [<ffffffff81616039>] ip_rcv_finish+0x119/0x360                           
    [  225.720544]  [<ffffffff816168ed>] ip_rcv+0x21d/0x300                                  
    [  225.720544]  [<ffffffff815e355a>] __netif_receive_skb+0x5fa/0x760                     
    [  225.720544]  [<ffffffff8163cbae>] ? tcp4_gro_receive+0x9e/0x110                       
    [  225.720544]  [<ffffffff815e36e3>] netif_receive_skb+0x23/0x90                         
    [  225.720544]  [<ffffffff815e3e28>] napi_gro_receive+0xe8/0x140                         
    [  225.906423]  [<ffffffffa0002f98>] e1000_clean_rx_irq+0x2b8/0x520 [e1000]              
    [  225.906423]  [<ffffffff8144df56>] ? credit_entropy_bits.part.7+0x176/0x1d0            
    [  225.906423]  [<ffffffffa0004801>] e1000_clean+0x51/0xc0 [e1000]                       
    [  225.906423]  [<ffffffff815e4cd4>] net_rx_action+0x134/0x260                           
    [  225.906423]  [<ffffffff81045136>] ? native_safe_halt+0x6/0x10                         
    [  225.906423]  [<ffffffff81062620>] __do_softirq+0xc0/0x240                             
    [  225.906423]  [<ffffffff8103cca2>] ? ack_apic_level+0x72/0x130                         
    [  225.906423]  [<ffffffff816fdd5c>] call_softirq+0x1c/0x30                              
    [  225.906423]  [<ffffffff81016775>] do_softirq+0x65/0xa0                                
    [  225.906423]  [<ffffffff810628fe>] irq_exit+0x8e/0xb0                                  
    [  225.906423]  [<ffffffff816fe5f3>] do_IRQ+0x63/0xe0                                    
    [  225.906423]  [<ffffffff816f406d>] common_interrupt+0x6d/0x6d                          
    [  225.906423]  <EOI>                                                                    
    [  225.906423]  [<ffffffff81084008>] ? hrtimer_start+0x18/0x20                           
    [  225.906423]  [<ffffffff81045136>] ? native_safe_halt+0x6/0x10
    [  225.906423]  [<ffffffff8101cc33>] default_idle+0x53/0x1f0                             
    [  225.906423]  [<ffffffff8101dad9>] cpu_idle+0xd9/0x120                                 
    [  225.906423]  [<ffffffff816c6db2>] rest_init+0x72/0x80                                 
    [  225.906423]  [<ffffffff81d05c4f>] start_kernel+0x3d1/0x3de                            
    [  225.906423]  [<ffffffff81d057ff>] ? pass_bootoption.constprop.2+0xd3/0xd3             
    [  225.906423]  [<ffffffff81d05397>] x86_64_start_reservations+0x131/0x135               
    [  225.906423]  [<ffffffff81d05120>] ? early_idt_handlers+0x120/0x120                    
    [  225.906423]  [<ffffffff81d05468>] x86_64_start_kernel+0xcd/0xdc                       
    [  225.906423] Code: 49 c1 e5 04 4c 03 6a 18 4c 89 ef e8 2f 35 0d 00 49 8b 84 24 60 03 00
    [  225.906423] RIP  [<ffffffff8162056c>] __inet_twsk_hashdance+0x8c/0x160                
    [  225.906423]  RSP <ffff88007fc039d0>                                                   
    [  225.906423] CR2: 0000000000000020                                                     
    [  226.111238] ---[ end trace 18ecfe1006daa681 ]---                                      
    [  226.111817] Kernel panic - not syncing: Fatal exception in interrupt

  2. #2
    Linux Guru Rubberman's Avatar
    Join Date
    Apr 2009
    Location
    I can be found either 40 miles west of Chicago, in Chicago, or in a galaxy far, far away.
    Posts
    12,107
    This isn't particularly helpful, other than the fact that it is a null pointer exception in the kernel (hence the kernel panic). Please post details of how you are "migrating" a socket from one server to another. I assume you had to do some kernel mods to do this? If not, then how?
    Sometimes, real fast is almost as good as real time.
    Just remember, Semper Gumbi - always be flexible!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •