== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:06:13 (1786752373) start mds service on oleg610-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg610-server' oleg610-server: oleg610-server.virtnet: executing load_modules_local oleg610-server: Loading modules from /home/green/git/lustre-release/lustre oleg610-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg610-server: detected 4 online CPUs by sysfs oleg610-server: MODOPTS_LIBCFS= oleg610-server: Force libcfs to create 2 CPU partitions oleg610-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg610-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg610-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg610-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg610-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg610-server: oleg610-server.virtnet: executing set_default_debug -1 all pdsh@oleg610-client: oleg610-server: ssh exited with exit code 1 Started lustre-OST0000 oleg610-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg610-client.virtnet: -o user_xattr,flock 192.168.206.110@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 88 (time 1786752460.63 total 10.00 last 8.80) - open/close 222 (time 1786752470.65 total 20.02 last 13.37) - open/close 372 (time 1786752480.78 total 30.15 last 14.81) - open/close 579 (time 1786752490.79 total 40.16 last 20.67) - open/close 687 (time 1786752501.19 total 50.56 last 10.39) - open/close 995 (time 1786752511.62 total 60.99 last 29.52) - open/close 1231 (time 1786752521.63 total 71.00 last 23.59) - open/close 1415 (time 1786752532.25 total 81.62 last 17.32) - open/close 1650 (time 1786752543.33 total 92.70 last 21.21) - open/close 1873 (time 1786752554.00 total 103.37 last 20.90) - open/close 2118 (time 1786752564.89 total 114.26 last 22.50) - open/close 2267 (time 1786752575.52 total 124.89 last 14.03) - open/close 2456 (time 1786752586.59 total 135.96 last 17.07) - open/close 2615 (time 1786752597.16 total 146.53 last 15.04) - open/close 2897 (time 1786752607.17 total 156.54 last 28.16) - open/close 3115 (time 1786752617.18 total 166.55 last 21.79) - open/close 3364 (time 1786752627.18 total 176.55 last 24.89) - open/close 3555 (time 1786752637.45 total 186.82 last 18.61) - open/close 3727 (time 1786752647.45 total 196.82 last 17.20) - open/close 3834 (time 1786752658.44 total 207.81 last 9.73) - open/close 4034 (time 1786752668.68 total 218.05 last 19.53) - open/close 4275 (time 1786752679.01 total 228.38 last 23.32) - open/close 4484 (time 1786752689.55 total 238.92 last 19.84) - open/close 4675 (time 1786752699.55 total 248.92 last 19.10) - open/close 4938 (time 1786752710.47 total 259.84 last 24.08) - open/close 5176 (time 1786752720.72 total 270.09 last 23.23) - open/close 5356 (time 1786752731.22 total 280.59 last 17.14) - open/close 5595 (time 1786752741.59 total 290.96 last 23.05) - open/close 5838 (time 1786752752.26 total 301.63 last 22.76) - open/close 6115 (time 1786752763.08 total 312.45 last 25.62) - open/close 6400 (time 1786752773.51 total 322.87 last 27.33) - open/close 6605 (time 1786752783.52 total 332.88 last 20.48) - open/close 6850 (time 1786752793.53 total 342.90 last 24.47) - open/close 7105 (time 1786752804.08 total 353.45 last 24.16) - open/close 7245 (time 1786752814.08 total 363.45 last 14.00) - open/close 7492 (time 1786752825.04 total 374.41 last 22.55) - open/close 7676 (time 1786752835.22 total 384.59 last 18.08) - open/close 7972 (time 1786752846.21 total 395.58 last 26.92) - open/close 8195 (time 1786752856.85 total 406.22 last 20.96) - open/close 8444 (time 1786752866.86 total 416.23 last 24.88) - open/close 8671 (time 1786752877.00 total 426.37 last 22.39) - open/close 8919 (time 1786752887.56 total 436.93 last 23.48) - open/close 9170 (time 1786752898.38 total 447.75 last 23.20) - open/close 9416 (time 1786752908.39 total 457.76 last 24.60) - open/close 9641 (time 1786752918.41 total 467.78 last 22.43) - open/close 9873 (time 1786752930.05 total 479.42 last 19.93) - open/close 10000 (time 1786752935.53 total 484.90 last 23.20) - open/close 10265 (time 1786752945.53 total 494.90 last 26.49) - open/close 10507 (time 1786752956.19 total 505.56 last 22.71) - open/close 10609 (time 1786752966.61 total 515.98 last 9.79) - open/close 10839 (time 1786752976.64 total 526.01 last 22.94) - open/close 11073 (time 1786752987.46 total 536.83 last 21.61) - open/close 11278 (time 1786752998.29 total 547.66 last 18.93) - open/close 11522 (time 1786753009.11 total 558.48 last 22.55) - open/close 11772 (time 1786753020.05 total 569.42 last 22.87) - open/close 12013 (time 1786753030.46 total 579.83 last 23.14) - open/close 12243 (time 1786753040.71 total 590.08 last 22.45) - open/close 12481 (time 1786753051.28 total 600.65 last 22.52) - open/close 12658 (time 1786753061.47 total 610.84 last 17.37) - open/close 12770 (time 1786753071.50 total 620.87 last 11.16) - open/close 12862 (time 1786753081.89 total 631.26 last 8.86) - open/close 13097 (time 1786753092.56 total 641.93 last 22.02) - open/close 13323 (time 1786753102.56 total 651.93 last 22.59) - open/close 13576 (time 1786753113.17 total 662.54 last 23.85) - open/close 13799 (time 1786753123.17 total 672.54 last 22.28) - open/close 14032 (time 1786753133.18 total 682.55 last 23.29) - open/close 14294 (time 1786753144.03 total 693.40 last 24.14) - open/close 14473 (time 1786753157.96 total 707.33 last 12.85) - open/close 14680 (time 1786753168.21 total 717.58 last 20.20) - open/close 14866 (time 1786753179.14 total 728.51 last 17.01) - open/close 15046 (time 1786753189.59 total 738.96 last 17.24) - open/close 15318 (time 1786753199.60 total 748.97 last 27.18) - open/close 15558 (time 1786753209.92 total 759.29 last 23.24) - open/close 15798 (time 1786753219.93 total 769.29 last 23.99) - open/close 16036 (time 1786753230.54 total 779.91 last 22.42) - open/close 16279 (time 1786753240.55 total 789.92 last 24.26) - open/close 16492 (time 1786753250.57 total 799.94 last 21.27) - open/close 16706 (time 1786753260.71 total 810.08 last 21.11) - open/close 16972 (time 1786753270.75 total 820.12 last 26.49) - open/close 17226 (time 1786753281.48 total 830.85 last 23.67) - open/close 17479 (time 1786753291.48 total 840.85 last 25.29) - open/close 17601 (time 1786753301.65 total 851.02 last 12.00) - open/close 17714 (time 1786753311.65 total 861.02 last 11.29) - open/close 17888 (time 1786753322.26 total 871.63 last 16.41) - open/close 18159 (time 1786753333.26 total 882.63 last 24.63) - open/close 18434 (time 1786753343.50 total 892.87 last 26.85) - open/close 18695 (time 1786753353.50 total 902.87 last 26.09) - open/close 19005 (time 1786753363.52 total 912.89 last 30.96) - open/close 19266 (time 1786753373.53 total 922.90 last 26.07) - open/close 19877 (time 1786753383.54 total 932.91 last 61.03) - open/close 20575 (time 1786753393.55 total 942.92 last 69.76) - open/close 21244 (time 1786753403.55 total 952.92 last 66.88) - open/close 21958 (time 1786753413.57 total 962.94 last 71.27) - open/close 22620 (time 1786753423.58 total 972.95 last 66.12) - open/close 23111 (time 1786753433.60 total 982.97 last 49.00) - open/close 23645 (time 1786753443.61 total 992.98 last 53.34) - open/close 24303 (time 1786753453.63 total 1003.00 last 65.70) - open/close 24966 (time 1786753463.63 total 1013.00 last 66.26) - open/close 25702 (time 1786753473.64 total 1023.01 last 73.55) - open/close 26422 (time 1786753483.64 total 1033.01 last 71.97) - open/close 27122 (time 1786753493.64 total 1043.01 last 69.99) - open/close 27854 (time 1786753503.65 total 1053.02 last 73.17) - open/close 28556 (time 1786753513.70 total 1063.07 last 69.80) - open/close 29202 (time 1786753523.71 total 1073.08 last 64.58) - open/close 29954 (time 1786753533.71 total 1083.08 last 75.16) - open/close 30627 (time 1786753543.72 total 1093.09 last 67.25) - open/close 31291 (time 1786753553.72 total 1103.09 last 66.37) - open/close 31984 (time 1786753563.73 total 1113.10 last 69.25) - open/close 32661 (time 1786753573.74 total 1123.11 last 67.64) - open/close 33279 (time 1786753583.74 total 1133.11 last 61.78) - open/close 33899 (time 1786753593.78 total 1143.15 last 61.77) - open/close 34578 (time 1786753603.80 total 1153.17 last 67.74) - open/close 35238 (time 1786753613.81 total 1163.18 last 65.95) - open/close 35930 (time 1786753623.81 total 1173.18 last 69.18) - open/close 36624 (time 1786753633.83 total 1183.20 last 69.26) - open/close 37337 (time 1786753643.85 total 1193.22 last 71.17) - open/close 38007 (time 1786753653.85 total 1203.22 last 66.99) - open/close 38640 (time 1786753663.86 total 1213.22 last 63.28) - open/close 39307 (time 1786753673.87 total 1223.24 last 66.61) - open/close 40000 (time 1786753683.34 total 1232.71 last 73.15) - open/close 40715 (time 1786753693.36 total 1242.73 last 71.37) - open/close 41416 (time 1786753703.37 total 1252.74 last 70.06) - open/close 42097 (time 1786753713.37 total 1262.74 last 68.06) - open/close 42826 (time 1786753723.38 total 1272.75 last 72.85) - open/close 43532 (time 1786753733.39 total 1282.76 last 70.56) - open/close 44209 (time 1786753743.39 total 1292.76 last 67.66) - open/close 44841 (time 1786753753.39 total 1302.76 last 63.18) - open/close 45492 (time 1786753763.40 total 1312.77 last 65.09) - open/close 46138 (time 1786753773.44 total 1322.81 last 64.34) - open/close 46814 (time 1786753783.44 total 1332.81 last 67.59) - open/close 47461 (time 1786753793.45 total 1342.82 last 64.64) - open/close 48097 (time 1786753803.45 total 1352.82 last 63.58) - open/close 48761 (time 1786753813.46 total 1362.83 last 66.34) - open/close 49424 (time 1786753823.47 total 1372.84 last 66.23) total: 50000 open/close in 1382.14 seconds: 36.18 ops/second - unlinked 0 (time 1786753836 ; total 0 ; last 0) - unlinked 10000 (time 1786753996 ; total 160 ; last 160) - unlinked 20000 (time 1786754158 ; total 322 ; last 162) - unlinked 30000 (time 1786754319 ; total 483 ; last 161) - unlinked 40000 (time 1786754480 ; total 644 ; last 161) total: 50000 unlinks in 794 seconds: 62.972294 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg610-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg610-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg610-server stop mds service on oleg610-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg610-server unloading modules via unload_modules_local on: 'oleg610-server' oleg610-server: oleg610-server.virtnet: executing unload_modules_local oleg610-server: modules unloaded. pdsh@oleg610-client: oleg610-server: ssh exited with exit code 2 pdsh@oleg610-client: oleg610-client: ssh exited with exit code 2 pdsh@oleg610-client: oleg610-client: ssh exited with exit code 2 pdsh@oleg610-client: oleg610-server: ssh exited with exit code 2