== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 00:06:49 (1786421209) start mds service on oleg127-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing load_modules_local oleg127-server: Loading modules from /home/green/git/lustre-release/lustre oleg127-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg127-server: detected 4 online CPUs by sysfs oleg127-server: MODOPTS_LIBCFS= oleg127-server: Force libcfs to create 2 CPU partitions oleg127-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg127-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg127-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg127-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg127-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg127-server: oleg127-server.virtnet: executing set_default_debug -1 all pdsh@oleg127-client: oleg127-server: ssh exited with exit code 1 Started lustre-OST0000 oleg127-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg127-client.virtnet: -o user_xattr,flock 192.168.201.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 203 (time 1786421297.28 total 11.05 last 18.38) - open/close 457 (time 1786421307.29 total 21.05 last 25.38) - open/close 658 (time 1786421317.30 total 31.06 last 20.08) - open/close 839 (time 1786421328.06 total 41.83 last 16.81) - open/close 925 (time 1786421338.75 total 52.52 last 8.05) - open/close 1114 (time 1786421348.81 total 62.58 last 18.78) - open/close 1342 (time 1786421359.10 total 72.86 last 22.17) - open/close 1562 (time 1786421369.27 total 83.03 last 21.63) - open/close 1819 (time 1786421379.73 total 93.49 last 24.57) - open/close 2088 (time 1786421390.45 total 104.22 last 25.08) - open/close 2344 (time 1786421400.47 total 114.23 last 25.57) - open/close 2563 (time 1786421410.49 total 124.26 last 21.84) - open/close 2814 (time 1786421421.14 total 134.91 last 23.57) - open/close 3045 (time 1786421431.19 total 144.96 last 22.98) - open/close 3164 (time 1786421441.24 total 155.01 last 11.84) - open/close 3339 (time 1786421451.94 total 165.70 last 16.36) - open/close 3528 (time 1786421461.94 total 175.70 last 18.90) - open/close 3627 (time 1786421471.94 total 185.71 last 9.90) - open/close 3802 (time 1786421482.74 total 196.50 last 16.21) - open/close 3992 (time 1786421493.70 total 207.47 last 17.33) - open/close 4191 (time 1786421505.53 total 219.29 last 16.83) - open/close 4395 (time 1786421515.55 total 229.31 last 20.36) - open/close 4620 (time 1786421525.82 total 239.59 last 21.90) - open/close 4851 (time 1786421535.86 total 249.62 last 23.02) - open/close 5125 (time 1786421546.05 total 259.81 last 26.90) - open/close 5307 (time 1786421556.06 total 269.82 last 18.17) - open/close 5494 (time 1786421566.59 total 280.35 last 17.77) - open/close 5637 (time 1786421577.22 total 290.99 last 13.45) - open/close 5877 (time 1786421587.58 total 301.34 last 23.18) - open/close 6098 (time 1786421597.96 total 311.72 last 21.29) - open/close 6363 (time 1786421607.97 total 321.74 last 26.46) - open/close 6625 (time 1786421617.98 total 331.74 last 26.18) - open/close 6809 (time 1786421628.06 total 341.82 last 18.25) - open/close 6968 (time 1786421638.74 total 352.50 last 14.89) - open/close 7230 (time 1786421648.90 total 362.66 last 25.78) - open/close 7501 (time 1786421659.79 total 373.55 last 24.90) - open/close 7750 (time 1786421670.85 total 384.61 last 22.51) - open/close 7971 (time 1786421681.20 total 394.96 last 21.34) - open/close 8052 (time 1786421691.60 total 405.36 last 7.79) - open/close 8196 (time 1786421701.85 total 415.61 last 14.04) - open/close 8340 (time 1786421711.86 total 425.63 last 14.38) - open/close 8543 (time 1786421721.88 total 435.65 last 20.26) - open/close 8615 (time 1786421731.89 total 445.66 last 7.19) - open/close 8877 (time 1786421741.89 total 455.66 last 26.20) - open/close 9016 (time 1786421752.69 total 466.45 last 12.88) - open/close 9260 (time 1786421763.33 total 477.10 last 22.93) - open/close 9445 (time 1786421773.34 total 487.11 last 18.48) - open/close 9576 (time 1786421783.34 total 497.11 last 13.10) - open/close 9829 (time 1786421793.86 total 507.63 last 24.05) - open/close 10000 (time 1786421802.59 total 516.36 last 19.59) - open/close 10198 (time 1786421813.06 total 526.82 last 18.92) - open/close 10379 (time 1786421823.74 total 537.51 last 16.94) - open/close 10490 (time 1786421834.40 total 548.16 last 10.42) - open/close 10648 (time 1786421844.42 total 558.18 last 15.77) - open/close 10883 (time 1786421854.97 total 568.73 last 22.26) - open/close 10954 (time 1786421864.97 total 578.74 last 7.10) - open/close 11175 (time 1786421875.79 total 589.55 last 20.44) - open/close 11346 (time 1786421886.26 total 600.02 last 16.33) - open/close 11577 (time 1786421896.26 total 610.03 last 23.09) - open/close 11800 (time 1786421906.27 total 620.04 last 22.28) - open/close 12053 (time 1786421916.28 total 630.05 last 25.27) - open/close 12283 (time 1786421926.86 total 640.63 last 21.75) - open/close 12455 (time 1786421937.33 total 651.09 last 16.43) - open/close 12640 (time 1786421947.35 total 661.11 last 18.47) - open/close 12750 (time 1786421958.34 total 672.10 last 10.01) - open/close 12875 (time 1786421968.96 total 682.73 last 11.76) - open/close 13068 (time 1786421979.38 total 693.15 last 18.52) - open/close 13158 (time 1786421989.63 total 703.40 last 8.78) - open/close 13251 (time 1786422000.58 total 714.34 last 8.50) - open/close 13469 (time 1786422010.61 total 724.38 last 21.72) - open/close 13615 (time 1786422021.52 total 735.29 last 13.38) - open/close 13775 (time 1786422032.38 total 746.15 last 14.73) - open/close 13910 (time 1786422042.63 total 756.39 last 13.18) - open/close 14007 (time 1786422052.73 total 766.49 last 9.60) - open/close 14068 (time 1786422063.12 total 776.88 last 5.87) - open/close 14178 (time 1786422073.12 total 786.88 last 11.00) - open/close 14366 (time 1786422083.15 total 796.92 last 18.74) - open/close 14593 (time 1786422093.64 total 807.40 last 21.65) - open/close 14877 (time 1786422104.69 total 818.46 last 25.68) - open/close 15066 (time 1786422115.78 total 829.54 last 17.06) - open/close 15173 (time 1786422126.61 total 840.38 last 9.87) - open/close 15364 (time 1786422137.52 total 851.28 last 17.51) - open/close 15574 (time 1786422147.53 total 861.29 last 20.98) - open/close 15790 (time 1786422157.76 total 871.53 last 21.10) - open/close 16004 (time 1786422168.15 total 881.91 last 20.61) - open/close 16207 (time 1786422178.83 total 892.59 last 19.01) - open/close 16417 (time 1786422189.79 total 903.55 last 19.17) - open/close 16472 (time 1786422200.76 total 914.53 last 5.01) - open/close 16614 (time 1786422210.83 total 924.60 last 14.10) - open/close 16848 (time 1786422221.17 total 934.94 last 22.63) - open/close 17043 (time 1786422231.20 total 944.96 last 19.45) - open/close 17236 (time 1786422241.21 total 954.97 last 19.27) - open/close 17449 (time 1786422251.21 total 964.98 last 21.29) - open/close 17703 (time 1786422261.22 total 974.99 last 25.37) - open/close 18413 (time 1786422271.23 total 984.99 last 70.97) - open/close 19086 (time 1786422281.23 total 995.00 last 67.27) - open/close 19946 (time 1786422291.24 total 1005.01 last 85.92) - open/close 20668 (time 1786422301.24 total 1015.01 last 72.19) - open/close 21448 (time 1786422311.25 total 1025.01 last 77.98) - open/close 22169 (time 1786422321.25 total 1035.01 last 72.08) - open/close 22932 (time 1786422331.26 total 1045.02 last 76.23) - open/close 23818 (time 1786422341.26 total 1055.03 last 88.57) - open/close 24739 (time 1786422351.26 total 1065.03 last 92.09) - open/close 25486 (time 1786422361.27 total 1075.04 last 74.62) - open/close 26168 (time 1786422371.28 total 1085.05 last 68.14) - open/close 27027 (time 1786422381.29 total 1095.06 last 85.81) - open/close 27837 (time 1786422391.30 total 1105.06 last 80.97) - open/close 28639 (time 1786422401.30 total 1115.07 last 80.14) - open/close 29565 (time 1786422411.31 total 1125.07 last 92.57) - open/close 30000 (time 1786422416.61 total 1130.37 last 82.10) - open/close 30939 (time 1786422426.62 total 1140.38 last 93.82) - open/close 31847 (time 1786422436.62 total 1150.39 last 90.74) - open/close 32517 (time 1786422446.63 total 1160.40 last 66.94) - open/close 33430 (time 1786422456.63 total 1170.40 last 91.28) - open/close 34243 (time 1786422466.63 total 1180.40 last 81.29) - open/close 35167 (time 1786422476.65 total 1190.41 last 92.30) - open/close 36085 (time 1786422486.72 total 1200.48 last 91.13) - open/close 36898 (time 1786422496.72 total 1210.49 last 81.25) - open/close 37730 (time 1786422506.74 total 1220.50 last 83.11) - open/close 38600 (time 1786422516.74 total 1230.51 last 86.95) - open/close 39313 (time 1786422526.74 total 1240.51 last 71.29) - open/close 40000 (time 1786422534.72 total 1248.49 last 86.09) - open/close 40916 (time 1786422544.73 total 1258.49 last 91.56) - open/close 41742 (time 1786422554.73 total 1268.49 last 82.58) - open/close 42534 (time 1786422564.74 total 1278.50 last 79.15) - open/close 43477 (time 1786422574.76 total 1288.52 last 94.10) - open/close 44366 (time 1786422584.78 total 1298.54 last 88.73) - open/close 45153 (time 1786422594.78 total 1308.55 last 78.65) - open/close 45934 (time 1786422604.79 total 1318.55 last 78.05) - open/close 46787 (time 1786422614.79 total 1328.55 last 85.30) - open/close 47402 (time 1786422624.80 total 1338.56 last 61.45) - open/close 48187 (time 1786422634.80 total 1348.56 last 78.49) - open/close 49060 (time 1786422644.80 total 1358.57 last 87.26) - open/close 49952 (time 1786422654.81 total 1368.57 last 89.12) total: 50000 open/close in 1369.02 seconds: 36.52 ops/second - unlinked 0 (time 1786422659 ; total 0 ; last 0) - unlinked 10000 (time 1786422802 ; total 143 ; last 143) - unlinked 20000 (time 1786422934 ; total 275 ; last 132) - unlinked 30000 (time 1786423077 ; total 418 ; last 143) - unlinked 40000 (time 1786423215 ; total 556 ; last 138) total: 50000 unlinks in 689 seconds: 72.568939 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg127-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg127-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg127-server stop mds service on oleg127-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg127-server unloading modules via unload_modules_local on: 'oleg127-server' oleg127-server: oleg127-server.virtnet: executing unload_modules_local oleg127-server: modules unloaded. pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-client: ssh exited with exit code 2 pdsh@oleg127-client: oleg127-server: ssh exited with exit code 2