== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:59:22 (1781611162) start mds service on oleg213-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1045: echo: write error: Device or resource busy oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg213-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg213-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg213-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg213-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 795 (time 1781611217.44 total 10.06 last 78.99) - open/close 1573 (time 1781611227.44 total 20.06 last 77.80) - open/close 2300 (time 1781611238.49 total 31.12 last 65.74) - open/close 3045 (time 1781611249.37 total 42.00 last 68.47) - open/close 3741 (time 1781611259.42 total 52.05 last 69.29) - open/close 4395 (time 1781611269.42 total 62.05 last 65.39) - open/close 5058 (time 1781611280.41 total 73.04 last 60.33) - open/close 5627 (time 1781611290.41 total 83.04 last 56.89) - open/close 6235 (time 1781611301.08 total 93.71 last 56.98) - open/close 6946 (time 1781611311.09 total 103.72 last 71.08) - open/close 7577 (time 1781611321.88 total 114.51 last 58.44) - open/close 8307 (time 1781611332.77 total 125.39 last 67.08) - open/close 9032 (time 1781611343.71 total 136.34 last 66.25) - open/close 9692 (time 1781611354.46 total 147.09 last 61.38) - open/close 10000 (time 1781611358.22 total 150.85 last 82.00) - open/close 10619 (time 1781611369.05 total 161.68 last 57.14) - open/close 11294 (time 1781611379.93 total 172.56 last 62.04) - open/close 12091 (time 1781611389.93 total 182.56 last 79.69) - open/close 12795 (time 1781611399.93 total 192.56 last 70.40) - open/close 13573 (time 1781611410.98 total 203.60 last 70.46) - open/close 14159 (time 1781611421.60 total 214.22 last 55.18) - open/close 14936 (time 1781611431.60 total 224.23 last 77.69) - open/close 15635 (time 1781611442.46 total 235.09 last 64.35) - open/close 16393 (time 1781611452.46 total 245.09 last 75.79) - open/close 17145 (time 1781611462.49 total 255.12 last 74.97) - open/close 17967 (time 1781611472.54 total 265.17 last 81.78) - open/close 18584 (time 1781611483.10 total 275.73 last 58.44) - open/close 19295 (time 1781611493.92 total 286.55 last 65.71) - open/close 19845 (time 1781611504.41 total 297.04 last 52.43) - open/close 20000 (time 1781611506.80 total 299.43 last 64.98) - open/close 20758 (time 1781611516.80 total 309.43 last 75.78) - open/close 25100 (time 1781611526.80 total 319.43 last 434.15) - open/close 29924 (time 1781611536.80 total 329.43 last 482.34) - open/close 34838 (time 1781611546.80 total 339.43 last 491.32) - open/close 39670 (time 1781611556.80 total 349.43 last 483.12) - open/close 44403 (time 1781611566.80 total 359.43 last 473.30) - open/close 49196 (time 1781611576.80 total 369.43 last 479.27) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 370.33 seconds: 134.02 ops/second - unlinked 0 (time 1781611579 ; total 1 ; last 1) - unlinked 10000 (time 1781611590 ; total 12 ; last 11) - unlinked 20000 (time 1781611602 ; total 24 ; last 12) - unlinked 30000 (time 1781611614 ; total 36 ; last 12) - unlinked 40000 (time 1781611626 ; total 48 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 59 seconds: 841.203369 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg213-server unloading modules via unload_modules_local on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing unload_modules_local oleg213-server: modules unloaded. pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2