Crash Every 3 Days

[ecylcje]

Hi,

I'm sure this is me doing something wrong, but I bought a SheevaPlug about 2 months back and it is running the Debian from newit build.

chris@neutron:~$ dmesg
Linux version 2.6.32.7 (newit@gemini) (gcc version 4.4.1 (Sourcery G++ Lite 2009q3-68) ) #5 PREEMPT Wed Feb 10 21:09:39 GMT 2010
CPU: Feroceon 88FR131 [56251311] revision 1 (ARMv5TE), cr=00053177
CPU: VIVT data cache, VIVT instruction cache
Machine: Marvell SheevaPlug Reference Board
Memory policy: ECC disabled, Data cache writeback
On node 0 totalpages: 131072


The box isn't doing anything special, just running mysql (reduced memory) and apache2 and that is it. I have given the box a swap file on a mounted esata drive for an extra 512Mb.

The issue is that every 3 days I get the following error. The box doesn't lockup completely, I'm able to ssh on fine. I just cannot reboot the box without pulling the plug, shutdown certain processes like apache2 and the box has a load of 49+ (seen from uptime and top). I've done an apt-get update and apt-get upgrade to get the latest patches etc.

INFO: task init:1 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
init          D c0403758     0     1      0 0x00000000
[<c0403758>] (schedule+0x364/0x3f4) from [<c0403830>] (io_schedule+0x48/0x7c)
[<c0403830>] (io_schedule+0x48/0x7c) from [<c008c7cc>] (sync_page+0x48/0x54)
[<c008c7cc>] (sync_page+0x48/0x54) from [<c0404280>] (__wait_on_bit+0x54/0x9c)
[<c0404280>] (__wait_on_bit+0x54/0x9c) from [<c008c9bc>] (wait_on_page_bit+0x90/0xa0)
[<c008c9bc>] (wait_on_page_bit+0x90/0xa0) from [<c0097fd4>] (shrink_page_list+0x144/0x550)
[<c0097fd4>] (shrink_page_list+0x144/0x550) from [<c0098898>] (shrink_list+0x4b8/0x7dc)
[<c0098898>] (shrink_list+0x4b8/0x7dc) from [<c0098e98>] (shrink_zone+0x2dc/0x394)
[<c0098e98>] (shrink_zone+0x2dc/0x394) from [<c00997e0>] (try_to_free_pages+0x1ac/0x2f4)
[<c00997e0>] (try_to_free_pages+0x1ac/0x2f4) from [<c0092720>] (__alloc_pages_nodemask+0x30c/0x584)
[<c0092720>] (__alloc_pages_nodemask+0x30c/0x584) from [<c00b77a8>] (__slab_alloc+0x1a4/0x57c)
[<c00b77a8>] (__slab_alloc+0x1a4/0x57c) from [<c00b84f8>] (kmem_cache_alloc+0x64/0x134)
[<c00b84f8>] (kmem_cache_alloc+0x64/0x134) from [<c00c5788>] (getname+0x18/0xcc)
[<c00c5788>] (getname+0x18/0xcc) from [<c00c71f4>] (user_path_at+0x1c/0x90)
[<c00c71f4>] (user_path_at+0x1c/0x90) from [<c00bf5c4>] (vfs_fstatat+0x2c/0x54)
[<c00bf5c4>] (vfs_fstatat+0x2c/0x54) from [<c00bf6c8>] (sys_stat64+0x18/0x34)
[<c00bf6c8>] (sys_stat64+0x18/0x34) from [<c002b9e0>] (ret_fast_syscall+0x0/0x28)
INFO: task init:1 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
init          D c0403758     0     1      0 0x00000000
[<c0403758>] (schedule+0x364/0x3f4) from [<c0403830>] (io_schedule+0x48/0x7c)
[<c0403830>] (io_schedule+0x48/0x7c) from [<c008c7cc>] (sync_page+0x48/0x54)
[<c008c7cc>] (sync_page+0x48/0x54) from [<c0404280>] (__wait_on_bit+0x54/0x9c)
[<c0404280>] (__wait_on_bit+0x54/0x9c) from [<c008c9bc>] (wait_on_page_bit+0x90/0xa0)
[<c008c9bc>] (wait_on_page_bit+0x90/0xa0) from [<c0097fd4>] (shrink_page_list+0x144/0x550)
[<c0097fd4>] (shrink_page_list+0x144/0x550) from [<c0098898>] (shrink_list+0x4b8/0x7dc)
[<c0098898>] (shrink_list+0x4b8/0x7dc) from [<c0098e98>] (shrink_zone+0x2dc/0x394)
[<c0098e98>] (shrink_zone+0x2dc/0x394) from [<c00997e0>] (try_to_free_pages+0x1ac/0x2f4)


Could someone explain what I've done wrong?

Thanks so much,

Chris

[NewIT_James]

Not many clues in there but vfs_fstatat is found here

 *  arch/arm/kernel/sys_oabi-compat.c
 *
 *  Compatibility wrappers for syscalls that are used from
 *  old ABI user space binaries with an EABI kernel.

That may or may not be useful to you.

NewIT Technical

[ecylcje]

Thanks for this. I'm afraid not being an expert it doesn't mean too much.

Could it be that this is related to memory usage being too great after 3 days.

1/. My swap file being a big mistake. Perhaps they don't work so well on arm?

2/. Apache2 or mysql having issues

Or perhaps I have stuffed up the linux kernel (I haven't touched it and perhaps that is the point).

Linux version 2.6.32.7 (newit@gemini) (gcc version 4.4.1 (Sourcery G++ Lite 2009q3-68) ) #5 PREEMPT Wed Feb 10 21:09:39 GMT
 2010


I have used linux a little before and there always seems to be a new kernel image every few weeks for bug fixes etc. Should I be worried there hasn't been an upgrade to this one (from Debian) in nearly a year? Could that be the issue?

Should I be doing something to have a new kernel?

Thanks again,

Chris

[ecylcje]

Hi,

The problem seems to have gone now.

I have removed a cron job which ran a php script every hour. With this disabled the system has been up for 6 days now. I suspect something in php (or the script I'm running) is at fault.

Thanks,

Chris

[ecylcje]

Hi,

Update: The cron job removal only reduced the frequency of the issue. It still crashed once a week.

The exact cause of the issue was my nfs mount to a Thecus N4200ECO box. Doing a normal NFS mount somehow causes the box to lockup after a while. The fix for this was to make the NFS mount have rsize and wsize of 32K. This improved performance and also stopped the crashing.

Chris