== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 09:52:14 (1784123534) start mds service on oleg450-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg450-server: detected 4 online CPUs by sysfs oleg450-server: MODOPTS_LIBCFS= oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg450-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg450-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg450-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg450-client.virtnet: -o user_xattr,flock 192.168.204.150@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 593 (time 1784123572.91 total 10.43 last 56.85) - open/close 1245 (time 1784123583.60 total 21.12 last 61.01) - open/close 1901 (time 1784123593.97 total 31.49 last 63.27) - open/close 2569 (time 1784123604.27 total 41.79 last 64.82) - open/close 2947 (time 1784123615.02 total 52.54 last 35.16) - open/close 3508 (time 1784123625.91 total 63.42 last 51.56) - open/close 4088 (time 1784123635.95 total 73.47 last 57.72) - open/close 4740 (time 1784123646.06 total 83.58 last 64.49) - open/close 5400 (time 1784123657.07 total 94.59 last 59.94) - open/close 5828 (time 1784123667.38 total 104.89 last 41.54) - open/close 6484 (time 1784123678.32 total 115.84 last 59.95) - open/close 7158 (time 1784123688.56 total 126.08 last 65.82) - open/close 7648 (time 1784123699.19 total 136.70 last 46.11) - open/close 8326 (time 1784123709.23 total 146.75 last 67.49) - open/close 8934 (time 1784123719.92 total 157.44 last 56.87) - open/close 9528 (time 1784123730.86 total 168.38 last 54.29) - open/close 10000 (time 1784123738.24 total 175.75 last 64.03) - open/close 10577 (time 1784123748.24 total 185.75 last 57.69) - open/close 11072 (time 1784123758.77 total 196.29 last 47.00) - open/close 11592 (time 1784123769.20 total 206.72 last 49.85) - open/close 12141 (time 1784123779.95 total 217.47 last 51.05) - open/close 12793 (time 1784123790.96 total 228.48 last 59.24) - open/close 13454 (time 1784123801.91 total 239.42 last 60.38) - open/close 13940 (time 1784123812.40 total 249.92 last 46.32) - open/close 14488 (time 1784123822.96 total 260.48 last 51.90) - open/close 14876 (time 1784123833.78 total 271.30 last 35.86) - open/close 15086 (time 1784123843.79 total 281.30 last 20.98) - open/close 15253 (time 1784123853.79 total 291.31 last 16.70) - open/close 15464 (time 1784123866.64 total 304.16 last 16.41) - open/close 15730 (time 1784123877.04 total 314.56 last 25.57) - open/close 15921 (time 1784123887.10 total 324.61 last 19.00) - open/close 16059 (time 1784123898.88 total 336.39 last 11.71) - open/close 16915 (time 1784123908.88 total 346.40 last 85.58) - open/close 18437 (time 1784123918.88 total 356.40 last 152.18) - open/close 20000 (time 1784123927.03 total 364.55 last 191.79) - open/close 21917 (time 1784123937.03 total 374.55 last 191.67) - open/close 24128 (time 1784123947.04 total 384.55 last 221.02) - open/close 26138 (time 1784123957.04 total 394.56 last 200.93) - open/close 28331 (time 1784123967.04 total 404.56 last 219.23) - open/close 30000 (time 1784123975.35 total 412.87 last 200.84) - open/close 32299 (time 1784123985.35 total 422.87 last 229.89) - open/close 34407 (time 1784123995.35 total 432.87 last 210.79) - open/close 36300 (time 1784124005.36 total 442.87 last 189.26) - open/close 38002 (time 1784124015.36 total 452.87 last 170.19) - open/close 39946 (time 1784124025.36 total 462.88 last 194.32) - open/close 42000 (time 1784124035.36 total 472.88 last 205.34) - open/close 44090 (time 1784124045.36 total 482.88 last 208.98) - open/close 45835 (time 1784124055.37 total 492.89 last 174.38) - open/close 47588 (time 1784124065.37 total 502.89 last 175.24) - open/close 49743 (time 1784124075.38 total 512.90 last 215.42) total: 50000 open/close in 513.98 seconds: 97.28 ops/second - unlinked 0 (time 1784124078 ; total 0 ; last 0) - unlinked 10000 (time 1784124111 ; total 33 ; last 33) - unlinked 20000 (time 1784124153 ; total 75 ; last 42) - unlinked 30000 (time 1784124202 ; total 124 ; last 49) - unlinked 40000 (time 1784124251 ; total 173 ; last 49) total: 50000 unlinks in 210 seconds: 238.095245 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg450-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server unloading modules via unload_modules_local on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing unload_modules_local oleg450-server: modules unloaded. pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2