== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 14:40:30 (1781030430) start mds service on oleg318-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing load_modules_local oleg318-server: Loading modules from /home/green/git/lustre-release/lustre oleg318-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg318-server: detected 4 online CPUs by sysfs oleg318-server: MODOPTS_LIBCFS= oleg318-server: Force libcfs to create 2 CPU partitions oleg318-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg318-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg318-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg318-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg318-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg318-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg318-server: oleg318-server.virtnet: executing set_default_debug -1 all kernel.panic_on_warn = 0 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 1 Started lustre-OST0000 oleg318-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg318-client.virtnet: -o user_xattr,flock 192.168.203.118@tcp:/lustre /mnt/lustre kernel.panic_on_warn = 0 seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 754 (time 1781030483.47 total 10.99 last 68.64) - open/close 1538 (time 1781030493.47 total 20.99 last 78.39) - open/close 2246 (time 1781030504.40 total 31.91 last 64.80) - open/close 3043 (time 1781030514.40 total 41.91 last 79.70) - open/close 3843 (time 1781030524.43 total 51.95 last 79.72) - open/close 4519 (time 1781030535.12 total 62.63 last 63.26) - open/close 5123 (time 1781030545.74 total 73.26 last 56.85) - open/close 5641 (time 1781030556.11 total 83.62 last 49.97) - open/close 6215 (time 1781030566.61 total 94.12 last 54.69) - open/close 6585 (time 1781030576.61 total 104.12 last 37.00) - open/close 7325 (time 1781030586.61 total 114.12 last 73.99) - open/close 7980 (time 1781030597.52 total 125.03 last 60.04) - open/close 8547 (time 1781030608.01 total 135.53 last 54.02) - open/close 8860 (time 1781030618.02 total 145.53 last 31.28) - open/close 9378 (time 1781030628.37 total 155.88 last 50.07) - open/close 9845 (time 1781030638.61 total 166.12 last 45.59) - open/close 10648 (time 1781030648.65 total 176.17 last 79.94) - open/close 11271 (time 1781030659.22 total 186.73 last 58.99) - open/close 11762 (time 1781030669.65 total 197.16 last 47.08) - open/close 12587 (time 1781030679.65 total 207.16 last 82.49) - open/close 13358 (time 1781030689.68 total 217.20 last 76.84) - open/close 13944 (time 1781030700.24 total 227.75 last 55.51) - open/close 14493 (time 1781030710.67 total 238.19 last 52.61) - open/close 15080 (time 1781030721.23 total 248.74 last 55.60) - open/close 15585 (time 1781030731.60 total 259.12 last 48.69) - open/close 16187 (time 1781030742.29 total 269.80 last 56.32) - open/close 16755 (time 1781030752.85 total 280.36 last 53.81) - open/close 17607 (time 1781030762.96 total 290.47 last 84.24) - open/close 19405 (time 1781030772.96 total 300.48 last 179.77) - open/close 24357 (time 1781030782.96 total 310.48 last 495.13) - open/close 29377 (time 1781030792.96 total 320.48 last 501.99) - open/close 34251 (time 1781030802.96 total 330.48 last 487.33) - open/close 39224 (time 1781030812.97 total 340.48 last 497.24) - open/close 44179 (time 1781030822.97 total 350.48 last 495.44) - open/close 49126 (time 1781030832.97 total 360.48 last 494.62) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 361.40 seconds: 137.33 ops/second - unlinked 0 (time 1781030835 ; total 0 ; last 0) - unlinked 10000 (time 1781030845 ; total 10 ; last 10) - unlinked 20000 (time 1781030856 ; total 21 ; last 11) - unlinked 30000 (time 1781030866 ; total 31 ; last 10) - unlinked 40000 (time 1781030877 ; total 42 ; last 11) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 53 seconds: 936.433960 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg318-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg318-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg318-server stop mds service on oleg318-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg318-server unloading modules via unload_modules_local on: 'oleg318-server' oleg318-server: oleg318-server.virtnet: executing unload_modules_local oleg318-server: modules unloaded. pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-server: ssh exited with exit code 2 pdsh@oleg318-client: oleg318-client: ssh exited with exit code 2