== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:03:21 (1787443401) start mds service on oleg225-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing load_modules_local oleg225-server: Loading modules from /home/green/git/lustre-release/lustre oleg225-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg225-server: detected 4 online CPUs by sysfs oleg225-server: MODOPTS_LIBCFS= oleg225-server: Force libcfs to create 2 CPU partitions oleg225-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg225-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg225-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg225-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg225-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg225-server: oleg225-server.virtnet: executing set_default_debug -1 all pdsh@oleg225-client: oleg225-server: ssh exited with exit code 1 Started lustre-OST0000 oleg225-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg225-client.virtnet: -o user_xattr,flock 192.168.202.125@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 156 (time 1787443490.06 total 10.68 last 14.60) - open/close 371 (time 1787443500.45 total 21.07 last 20.70) - open/close 633 (time 1787443510.93 total 31.55 last 24.99) - open/close 830 (time 1787443520.95 total 41.57 last 19.66) - open/close 943 (time 1787443530.97 total 51.58 last 11.29) - open/close 1005 (time 1787443540.97 total 61.59 last 6.20) - open/close 1211 (time 1787443551.18 total 71.80 last 20.18) - open/close 1378 (time 1787443561.88 total 82.50 last 15.61) - open/close 1469 (time 1787443571.94 total 92.56 last 9.04) - open/close 1682 (time 1787443582.81 total 103.43 last 19.60) - open/close 1792 (time 1787443593.71 total 114.33 last 10.09) - open/close 1928 (time 1787443604.32 total 124.94 last 12.81) - open/close 2151 (time 1787443614.34 total 134.96 last 22.26) - open/close 2244 (time 1787443624.35 total 144.97 last 9.29) - open/close 2353 (time 1787443634.38 total 155.00 last 10.86) - open/close 2468 (time 1787443645.21 total 165.83 last 10.62) - open/close 2553 (time 1787443655.82 total 176.44 last 8.01) - open/close 2713 (time 1787443666.32 total 186.94 last 15.24) - open/close 2826 (time 1787443676.33 total 196.95 last 11.29) - open/close 2926 (time 1787443686.83 total 207.45 last 9.52) - open/close 3046 (time 1787443700.11 total 220.73 last 9.04) - open/close 3173 (time 1787443712.28 total 232.90 last 10.44) - open/close 3410 (time 1787443723.30 total 243.92 last 21.51) - open/close 3652 (time 1787443736.21 total 256.83 last 18.74) - open/close 3731 (time 1787443746.65 total 267.27 last 7.57) - open/close 3916 (time 1787443759.18 total 279.80 last 14.76) - open/close 4036 (time 1787443769.20 total 289.82 last 11.98) - open/close 4154 (time 1787443779.21 total 299.82 last 11.79) - open/close 4260 (time 1787443790.10 total 310.72 last 9.73) - open/close 4406 (time 1787443800.10 total 320.72 last 14.59) - open/close 4478 (time 1787443810.17 total 330.79 last 7.15) - open/close 4746 (time 1787443820.18 total 340.80 last 26.78) - open/close 4806 (time 1787443833.20 total 353.82 last 4.61) - open/close 4958 (time 1787443843.73 total 364.35 last 14.43) - open/close 5109 (time 1787443853.76 total 374.38 last 15.04) - open/close 5239 (time 1787443864.22 total 384.84 last 12.43) - open/close 5506 (time 1787443874.23 total 394.85 last 26.67) - open/close 5698 (time 1787443884.35 total 404.97 last 18.98) - open/close 5812 (time 1787443895.33 total 415.95 last 10.38) - open/close 5979 (time 1787443905.35 total 425.97 last 16.67) - open/close 6095 (time 1787443916.17 total 436.79 last 10.72) - open/close 6256 (time 1787443929.24 total 449.86 last 12.32) - open/close 6420 (time 1787443939.26 total 459.88 last 16.37) - open/close 6669 (time 1787443949.28 total 469.90 last 24.85) - open/close 6881 (time 1787443959.59 total 480.21 last 20.57) - open/close 7046 (time 1787443969.61 total 490.23 last 16.47) - open/close 7169 (time 1787443979.62 total 500.24 last 12.29) - open/close 7254 (time 1787443989.77 total 510.39 last 8.37) - open/close 7442 (time 1787444000.08 total 520.70 last 18.24) - open/close 7626 (time 1787444011.95 total 532.57 last 15.50) - open/close 7791 (time 1787444021.95 total 542.57 last 16.50) - open/close 7911 (time 1787444032.56 total 553.18 last 11.31) - open/close 8055 (time 1787444042.57 total 563.18 last 14.39) - open/close 8198 (time 1787444053.91 total 574.53 last 12.61) - open/close 8329 (time 1787444069.74 total 590.36 last 8.27) - open/close 8558 (time 1787444080.55 total 601.16 last 21.19) - open/close 8791 (time 1787444090.57 total 611.19 last 23.24) - open/close 9007 (time 1787444101.20 total 621.82 last 20.32) - open/close 9309 (time 1787444111.21 total 631.83 last 30.17) - open/close 9493 (time 1787444121.32 total 641.93 last 18.21) - open/close 9667 (time 1787444132.00 total 652.62 last 16.29) - open/close 9775 (time 1787444142.03 total 662.65 last 10.77) - open/close 9930 (time 1787444152.61 total 673.23 last 14.65) - open/close 10000 (time 1787444154.97 total 675.59 last 29.69) - open/close 10094 (time 1787444165.53 total 686.15 last 8.90) - open/close 10308 (time 1787444175.92 total 696.54 last 20.59) - open/close 10552 (time 1787444185.93 total 706.55 last 24.39) - open/close 10724 (time 1787444196.63 total 717.25 last 16.08) - open/close 10877 (time 1787444206.90 total 727.52 last 14.89) - open/close 11084 (time 1787444217.18 total 737.80 last 20.14) - open/close 11270 (time 1787444227.49 total 748.11 last 18.05) - open/close 11430 (time 1787444237.50 total 758.12 last 15.97) - open/close 11588 (time 1787444247.50 total 768.12 last 15.80) - open/close 11711 (time 1787444258.36 total 778.98 last 11.33) - open/close 11903 (time 1787444268.65 total 789.27 last 18.66) - open/close 12110 (time 1787444279.60 total 800.21 last 18.91) - open/close 12308 (time 1787444290.45 total 811.07 last 18.23) - open/close 12369 (time 1787444300.57 total 821.19 last 6.03) - open/close 12510 (time 1787444311.66 total 832.28 last 12.72) - open/close 12589 (time 1787444323.05 total 843.67 last 6.94) - open/close 12725 (time 1787444334.03 total 854.65 last 12.38) - open/close 12859 (time 1787444344.37 total 864.99 last 12.97) - open/close 12960 (time 1787444355.76 total 876.38 last 8.86) - open/close 13033 (time 1787444368.15 total 888.77 last 5.89) - open/close 13165 (time 1787444378.68 total 899.30 last 12.54) - open/close 13342 (time 1787444389.47 total 910.09 last 16.40) - open/close 13560 (time 1787444400.47 total 921.09 last 19.82) - open/close 13680 (time 1787444410.48 total 931.10 last 11.98) - open/close 13756 (time 1787444420.49 total 941.11 last 7.60) - open/close 13989 (time 1787444430.49 total 951.11 last 23.29) - open/close 14178 (time 1787444440.59 total 961.21 last 18.71) - open/close 14363 (time 1787444450.62 total 971.24 last 18.45) - open/close 14469 (time 1787444462.29 total 982.91 last 9.09) - open/close 14605 (time 1787444473.75 total 994.37 last 11.86) - open/close 14902 (time 1787444483.85 total 1004.47 last 29.41) - open/close 15036 (time 1787444493.86 total 1014.48 last 13.39) - open/close 15126 (time 1787444504.19 total 1024.80 last 8.72) - open/close 15293 (time 1787444514.90 total 1035.52 last 15.58) - open/close 15413 (time 1787444524.95 total 1045.57 last 11.94) - open/close 15497 (time 1787444536.35 total 1056.97 last 7.37) - open/close 15641 (time 1787444546.59 total 1067.21 last 14.06) - open/close 15861 (time 1787444556.61 total 1077.23 last 21.96) - open/close 16020 (time 1787444566.62 total 1087.24 last 15.88) - open/close 16654 (time 1787444576.67 total 1097.29 last 63.11) - open/close 17228 (time 1787444586.69 total 1107.31 last 57.30) - open/close 17918 (time 1787444596.69 total 1117.31 last 68.97) - open/close 18575 (time 1787444606.72 total 1127.34 last 65.48) - open/close 19173 (time 1787444616.74 total 1137.36 last 59.73) - open/close 19800 (time 1787444626.74 total 1147.36 last 62.66) - open/close 20000 (time 1787444629.86 total 1150.48 last 64.10) - open/close 20605 (time 1787444639.88 total 1160.50 last 60.42) - open/close 21362 (time 1787444649.88 total 1170.50 last 75.64) - open/close 22193 (time 1787444659.89 total 1180.51 last 83.06) - open/close 22965 (time 1787444669.90 total 1190.52 last 77.13) - open/close 23753 (time 1787444679.91 total 1200.53 last 78.70) - open/close 24376 (time 1787444689.92 total 1210.54 last 62.22) - open/close 25035 (time 1787444699.93 total 1220.55 last 65.87) - open/close 25767 (time 1787444709.93 total 1230.55 last 73.20) - open/close 26451 (time 1787444719.94 total 1240.56 last 68.30) - open/close 27139 (time 1787444729.95 total 1250.57 last 68.78) - open/close 27720 (time 1787444739.95 total 1260.57 last 58.06) - open/close 28328 (time 1787444749.96 total 1270.58 last 60.77) - open/close 28992 (time 1787444759.97 total 1280.59 last 66.32) - open/close 29662 (time 1787444769.98 total 1290.60 last 66.96) - open/close 30000 (time 1787444775.30 total 1295.92 last 63.51) - open/close 30633 (time 1787444785.30 total 1305.92 last 63.29) - open/close 31282 (time 1787444795.31 total 1315.93 last 64.84) - open/close 31998 (time 1787444805.33 total 1325.95 last 71.45) - open/close 32590 (time 1787444815.34 total 1335.96 last 59.13) - open/close 33274 (time 1787444825.35 total 1345.97 last 68.33) - open/close 33946 (time 1787444835.36 total 1355.98 last 67.17) - open/close 34561 (time 1787444845.36 total 1365.98 last 61.50) - open/close 35207 (time 1787444855.36 total 1375.98 last 64.56) - open/close 35876 (time 1787444865.37 total 1385.98 last 66.89) - open/close 36597 (time 1787444875.38 total 1396.00 last 72.00) - open/close 37215 (time 1787444885.40 total 1406.02 last 61.65) - open/close 37953 (time 1787444895.41 total 1416.02 last 73.79) - open/close 38584 (time 1787444905.41 total 1426.03 last 63.09) - open/close 39216 (time 1787444915.41 total 1436.03 last 63.20) - open/close 39847 (time 1787444925.45 total 1446.07 last 62.84) - open/close 40000 (time 1787444927.98 total 1448.60 last 60.36) - open/close 40543 (time 1787444938.00 total 1458.62 last 54.18) - open/close 41093 (time 1787444948.02 total 1468.64 last 54.89) - open/close 41666 (time 1787444958.03 total 1478.65 last 57.23) - open/close 42326 (time 1787444968.04 total 1488.66 last 65.96) - open/close 42972 (time 1787444978.04 total 1498.66 last 64.57) - open/close 43658 (time 1787444988.05 total 1508.67 last 68.55) - open/close 44376 (time 1787444998.06 total 1518.68 last 71.72) - open/close 45021 (time 1787445008.07 total 1528.69 last 64.44) - open/close 45695 (time 1787445018.07 total 1538.69 last 67.38) - open/close 46277 (time 1787445028.09 total 1548.71 last 58.10) - open/close 46920 (time 1787445038.12 total 1558.74 last 64.14) - open/close 47510 (time 1787445048.16 total 1568.78 last 58.76) - open/close 48115 (time 1787445058.17 total 1578.79 last 60.43) - open/close 48701 (time 1787445068.19 total 1588.81 last 58.48) - open/close 49272 (time 1787445078.21 total 1598.83 last 56.98) - open/close 49945 (time 1787445088.22 total 1608.84 last 67.23) total: 50000 open/close in 1609.61 seconds: 31.06 ops/second - unlinked 0 (time 1787445093 ; total 0 ; last 0) - unlinked 10000 (time 1787445259 ; total 166 ; last 166) - unlinked 20000 (time 1787445424 ; total 331 ; last 165) - unlinked 30000 (time 1787445606 ; total 513 ; last 182) - unlinked 40000 (time 1787445767 ; total 674 ; last 161) total: 50000 unlinks in 836 seconds: 59.808613 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg225-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg225-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg225-server stop mds service on oleg225-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg225-server unloading modules via unload_modules_local on: 'oleg225-server' oleg225-server: oleg225-server.virtnet: executing unload_modules_local oleg225-server: modules unloaded. pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-client: ssh exited with exit code 2 pdsh@oleg225-client: oleg225-server: ssh exited with exit code 2