== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 08:38:28 (1786192708) start mds service on oleg423-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions oleg423-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg423-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg423-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 176 (time 1786192780.85 total 13.05 last 13.49) - open/close 297 (time 1786192791.53 total 23.73 last 11.33) - open/close 528 (time 1786192802.30 total 34.50 last 21.46) - open/close 644 (time 1786192812.91 total 45.11 last 10.93) - open/close 819 (time 1786192823.95 total 56.15 last 15.86) - open/close 959 (time 1786192834.80 total 67.00 last 12.90) - open/close 1240 (time 1786192845.42 total 77.62 last 26.46) - open/close 1485 (time 1786192857.83 total 90.03 last 19.73) - open/close 1810 (time 1786192868.65 total 100.85 last 30.04) - open/close 2130 (time 1786192878.65 total 110.85 last 31.99) - open/close 2374 (time 1786192888.66 total 120.86 last 24.38) - open/close 2490 (time 1786192898.93 total 131.13 last 11.30) - open/close 2647 (time 1786192909.16 total 141.36 last 15.34) - open/close 2912 (time 1786192919.16 total 151.36 last 26.49) - open/close 3167 (time 1786192929.17 total 161.37 last 25.49) - open/close 3383 (time 1786192939.81 total 172.02 last 20.29) - open/close 3598 (time 1786192950.44 total 182.64 last 20.23) - open/close 3876 (time 1786192960.45 total 192.65 last 27.79) - open/close 4095 (time 1786192971.13 total 203.33 last 20.50) - open/close 4330 (time 1786192981.86 total 214.07 last 21.89) - open/close 4597 (time 1786192991.87 total 224.07 last 26.69) - open/close 4866 (time 1786193002.02 total 234.22 last 26.49) - open/close 5113 (time 1786193012.03 total 244.23 last 24.69) - open/close 5362 (time 1786193023.08 total 255.28 last 22.53) - open/close 5572 (time 1786193033.64 total 265.84 last 19.88) - open/close 5870 (time 1786193044.52 total 276.72 last 27.39) - open/close 5963 (time 1786193055.33 total 287.54 last 8.60) - open/close 6151 (time 1786193065.90 total 298.10 last 17.79) - open/close 6348 (time 1786193076.33 total 308.53 last 18.89) - open/close 6617 (time 1786193086.70 total 318.90 last 25.93) - open/close 6902 (time 1786193097.13 total 329.33 last 27.33) - open/close 7122 (time 1786193107.95 total 340.15 last 20.33) - open/close 7370 (time 1786193118.76 total 350.97 last 22.93) - open/close 7702 (time 1786193129.64 total 361.84 last 30.52) - open/close 7973 (time 1786193140.39 total 372.59 last 25.20) - open/close 8289 (time 1786193150.40 total 382.60 last 31.58) - open/close 8635 (time 1786193161.13 total 393.33 last 32.25) - open/close 8864 (time 1786193171.75 total 403.95 last 21.55) - open/close 9367 (time 1786193182.26 total 414.46 last 47.89) - open/close 10000 (time 1786193190.00 total 422.20 last 81.78) - open/close 12208 (time 1786193200.00 total 432.20 last 220.75) - open/close 14232 (time 1786193210.00 total 442.20 last 202.37) - open/close 16688 (time 1786193220.01 total 452.21 last 245.48) - open/close 19021 (time 1786193230.01 total 462.21 last 233.26) - open/close 20000 (time 1786193234.27 total 466.47 last 229.63) - open/close 22580 (time 1786193244.28 total 476.48 last 257.90) - open/close 24235 (time 1786193254.28 total 486.48 last 165.49) - open/close 26079 (time 1786193264.28 total 496.48 last 184.34) - open/close 27543 (time 1786193274.28 total 506.48 last 146.35) - open/close 29593 (time 1786193284.29 total 516.49 last 204.94) - open/close 31825 (time 1786193294.29 total 526.49 last 223.11) - open/close 34306 (time 1786193304.29 total 536.49 last 248.04) - open/close 36868 (time 1786193314.30 total 546.50 last 256.13) - open/close 39829 (time 1786193324.30 total 556.50 last 296.06) - open/close 43342 (time 1786193334.30 total 566.50 last 351.26) - open/close 46338 (time 1786193344.30 total 576.50 last 299.59) - open/close 49064 (time 1786193354.30 total 586.50 last 272.58) total: 50000 open/close in 589.55 seconds: 84.81 ops/second - unlinked 0 (time 1786193358 ; total 0 ; last 0) - unlinked 10000 (time 1786193383 ; total 25 ; last 25) - unlinked 20000 (time 1786193403 ; total 45 ; last 20) - unlinked 30000 (time 1786193421 ; total 63 ; last 18) - unlinked 40000 (time 1786193441 ; total 83 ; last 20) total: 50000 unlinks in 103 seconds: 485.436890 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2