== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 11:52:34 (1783353154) start mds service on oleg343-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing load_modules_local oleg343-server: Loading modules from /home/green/git/lustre-release/lustre oleg343-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg343-server: detected 4 online CPUs by sysfs oleg343-server: MODOPTS_LIBCFS= oleg343-server: Force libcfs to create 2 CPU partitions oleg343-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg343-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg343-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg343-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg343-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg343-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg343-server: oleg343-server.virtnet: executing set_default_debug -1 all pdsh@oleg343-client: oleg343-server: ssh exited with exit code 1 Started lustre-OST0000 oleg343-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg343-client.virtnet: -o user_xattr,flock 192.168.203.143@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 767 (time 1783353211.79 total 10.00 last 76.69) - open/close 1511 (time 1783353222.76 total 20.98 last 67.78) - open/close 2117 (time 1783353233.26 total 31.47 last 57.75) - open/close 2646 (time 1783353243.63 total 41.84 last 51.01) - open/close 3222 (time 1783353254.06 total 52.27 last 55.22) - open/close 3837 (time 1783353264.81 total 63.02 last 57.20) - open/close 4279 (time 1783353275.05 total 73.27 last 43.15) - open/close 4995 (time 1783353285.93 total 84.14 last 65.82) - open/close 5764 (time 1783353295.93 total 94.15 last 76.89) - open/close 6523 (time 1783353306.92 total 105.14 last 69.06) - open/close 7250 (time 1783353317.80 total 116.02 last 66.81) - open/close 7856 (time 1783353328.43 total 126.64 last 57.05) - open/close 8423 (time 1783353338.86 total 137.08 last 54.33) - open/close 9073 (time 1783353349.55 total 147.76 last 60.83) - open/close 9635 (time 1783353360.11 total 158.32 last 53.22) - open/close 10000 (time 1783353366.08 total 164.29 last 61.14) - open/close 10724 (time 1783353376.88 total 175.09 last 67.03) - open/close 11493 (time 1783353386.88 total 185.09 last 76.88) - open/close 12289 (time 1783353397.93 total 196.14 last 72.02) - open/close 13059 (time 1783353408.11 total 206.32 last 75.65) - open/close 13869 (time 1783353419.05 total 217.26 last 74.03) - open/close 14679 (time 1783353429.05 total 227.26 last 80.99) - open/close 15447 (time 1783353440.04 total 238.26 last 69.87) - open/close 16224 (time 1783353450.04 total 248.26 last 77.69) - open/close 17057 (time 1783353460.04 total 258.26 last 83.28) - open/close 17852 (time 1783353470.05 total 268.26 last 79.48) - open/close 18625 (time 1783353480.05 total 278.26 last 77.29) - open/close 19363 (time 1783353490.99 total 289.20 last 67.47) - open/close 20000 (time 1783353498.69 total 296.91 last 82.66) - open/close 22261 (time 1783353508.69 total 306.91 last 226.06) - open/close 27021 (time 1783353518.69 total 316.91 last 475.97) - open/close 30000 (time 1783353524.83 total 323.04 last 485.78) - open/close 35196 (time 1783353534.83 total 333.04 last 519.59) - open/close 39897 (time 1783353544.83 total 343.04 last 470.08) - open/close 44656 (time 1783353554.83 total 353.04 last 475.89) - open/close 49358 (time 1783353564.83 total 363.04 last 470.12) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 363.63 seconds: 136.49 ops/second - unlinked 0 (time 1783353566 ; total 0 ; last 0) - unlinked 10000 (time 1783353579 ; total 13 ; last 13) - unlinked 20000 (time 1783353591 ; total 25 ; last 12) - unlinked 30000 (time 1783353604 ; total 38 ; last 13) - unlinked 40000 (time 1783353616 ; total 50 ; last 12) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 61 seconds: 813.622925 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg343-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg343-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg343-server stop mds service on oleg343-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg343-server unloading modules via unload_modules_local on: 'oleg343-server' oleg343-server: oleg343-server.virtnet: executing unload_modules_local oleg343-server: modules unloaded. pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-server: ssh exited with exit code 2 pdsh@oleg343-client: oleg343-client: ssh exited with exit code 2