== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 20:24:58 (1786753498) start mds service on oleg627-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing load_modules_local oleg627-server: Loading modules from /home/green/git/lustre-release/lustre oleg627-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg627-server: detected 4 online CPUs by sysfs oleg627-server: MODOPTS_LIBCFS= oleg627-server: Force libcfs to create 2 CPU partitions oleg627-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg627-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg627-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg627-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg627-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg627-server: oleg627-server.virtnet: executing set_default_debug -1 all pdsh@oleg627-client: oleg627-server: ssh exited with exit code 1 Started lustre-OST0000 oleg627-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg627-client.virtnet: -o user_xattr,flock 192.168.206.127@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 200 (time 1786753583.22 total 10.29 last 19.44) - open/close 434 (time 1786753593.23 total 20.30 last 23.37) - open/close 611 (time 1786753604.08 total 31.14 last 16.32) - open/close 871 (time 1786753614.09 total 41.16 last 25.96) - open/close 1087 (time 1786753624.25 total 51.32 last 21.26) - open/close 1365 (time 1786753634.28 total 61.34 last 27.73) - open/close 1586 (time 1786753644.29 total 71.36 last 22.07) - open/close 1671 (time 1786753654.31 total 81.38 last 8.49) - open/close 1871 (time 1786753666.29 total 93.35 last 16.70) - open/close 1968 (time 1786753676.29 total 103.35 last 9.70) - open/close 2210 (time 1786753686.30 total 113.37 last 24.16) - open/close 2405 (time 1786753696.94 total 124.01 last 18.34) - open/close 2649 (time 1786753706.94 total 134.01 last 24.39) - open/close 2864 (time 1786753717.48 total 144.55 last 20.40) - open/close 3066 (time 1786753727.90 total 154.97 last 19.39) - open/close 3289 (time 1786753737.91 total 164.97 last 22.29) - open/close 3529 (time 1786753748.83 total 175.90 last 21.96) - open/close 3711 (time 1786753758.84 total 185.91 last 18.19) - open/close 3904 (time 1786753768.85 total 195.92 last 19.27) - open/close 4083 (time 1786753779.63 total 206.70 last 16.61) - open/close 4231 (time 1786753790.32 total 217.39 last 13.85) - open/close 4412 (time 1786753802.41 total 229.48 last 14.96) - open/close 4576 (time 1786753812.42 total 239.49 last 16.39) - open/close 4818 (time 1786753822.69 total 249.76 last 23.55) - open/close 5071 (time 1786753833.39 total 260.45 last 23.66) - open/close 5277 (time 1786753843.56 total 270.63 last 20.25) - open/close 5445 (time 1786753856.50 total 283.57 last 12.98) - open/close 5539 (time 1786753867.00 total 294.07 last 8.95) - open/close 5758 (time 1786753877.01 total 304.08 last 21.88) - open/close 5872 (time 1786753887.79 total 314.86 last 10.58) - open/close 6095 (time 1786753898.55 total 325.62 last 20.71) - open/close 6298 (time 1786753908.73 total 335.80 last 19.95) - open/close 6535 (time 1786753919.26 total 346.33 last 22.51) - open/close 6755 (time 1786753929.32 total 356.39 last 21.87) - open/close 6992 (time 1786753939.33 total 366.40 last 23.66) - open/close 7253 (time 1786753950.12 total 377.18 last 24.21) - open/close 7535 (time 1786753960.12 total 387.19 last 28.17) - open/close 7763 (time 1786753970.67 total 397.74 last 21.62) - open/close 7987 (time 1786753980.87 total 407.94 last 21.95) - open/close 8240 (time 1786753991.93 total 419.00 last 22.88) - open/close 8458 (time 1786754001.97 total 429.04 last 21.71) - open/close 8646 (time 1786754012.04 total 439.11 last 18.68) - open/close 8899 (time 1786754022.05 total 449.12 last 25.26) - open/close 9129 (time 1786754032.64 total 459.71 last 21.72) - open/close 9308 (time 1786754043.56 total 470.63 last 16.39) - open/close 9448 (time 1786754055.94 total 483.01 last 11.31) - open/close 9597 (time 1786754066.18 total 493.25 last 14.56) - open/close 9841 (time 1786754076.20 total 503.27 last 24.35) - open/close 10000 (time 1786754083.15 total 510.22 last 22.85) - open/close 10213 (time 1786754093.35 total 520.42 last 20.88) - open/close 10450 (time 1786754103.36 total 530.43 last 23.68) - open/close 10661 (time 1786754113.64 total 540.71 last 20.52) - open/close 10854 (time 1786754123.69 total 550.76 last 19.21) - open/close 11002 (time 1786754133.70 total 560.77 last 14.79) - open/close 11227 (time 1786754144.39 total 571.46 last 21.04) - open/close 11490 (time 1786754154.87 total 581.94 last 25.09) - open/close 11704 (time 1786754165.41 total 592.48 last 20.30) - open/close 11945 (time 1786754175.92 total 602.99 last 22.94) - open/close 12176 (time 1786754186.55 total 613.62 last 21.73) - open/close 12413 (time 1786754196.55 total 623.62 last 23.69) - open/close 12609 (time 1786754206.56 total 633.63 last 19.59) - open/close 12836 (time 1786754216.56 total 643.63 last 22.69) - open/close 13015 (time 1786754227.26 total 654.32 last 16.74) - open/close 13216 (time 1786754237.26 total 664.33 last 20.09) - open/close 13473 (time 1786754247.47 total 674.54 last 25.18) - open/close 13717 (time 1786754257.47 total 684.54 last 24.40) - open/close 13967 (time 1786754267.62 total 694.69 last 24.63) - open/close 14263 (time 1786754277.62 total 704.69 last 29.59) - open/close 14481 (time 1786754287.65 total 714.72 last 21.74) - open/close 14688 (time 1786754297.66 total 724.73 last 20.69) - open/close 14863 (time 1786754307.72 total 734.78 last 17.40) - open/close 14959 (time 1786754318.13 total 745.20 last 9.21) - open/close 15201 (time 1786754328.83 total 755.90 last 22.63) - open/close 15401 (time 1786754339.26 total 766.33 last 19.17) - open/close 15568 (time 1786754351.15 total 778.22 last 14.04) - open/close 15751 (time 1786754361.78 total 788.85 last 17.22) - open/close 15945 (time 1786754371.80 total 798.87 last 19.36) - open/close 16230 (time 1786754381.81 total 808.88 last 28.49) - open/close 16315 (time 1786754391.82 total 818.89 last 8.49) - open/close 16525 (time 1786754401.91 total 828.98 last 20.80) - open/close 16783 (time 1786754412.91 total 839.98 last 23.46) - open/close 16966 (time 1786754423.87 total 850.93 last 16.71) - open/close 17213 (time 1786754434.09 total 861.16 last 24.16) - open/close 17345 (time 1786754444.28 total 871.35 last 12.96) - open/close 17503 (time 1786754454.29 total 881.36 last 15.79) - open/close 17597 (time 1786754464.30 total 891.37 last 9.38) - open/close 17729 (time 1786754474.99 total 902.06 last 12.36) - open/close 17936 (time 1786754485.35 total 912.42 last 19.97) - open/close 18161 (time 1786754495.36 total 922.42 last 22.50) - open/close 18420 (time 1786754506.02 total 933.09 last 24.28) - open/close 18656 (time 1786754516.29 total 943.36 last 22.99) - open/close 19395 (time 1786754526.29 total 953.36 last 73.89) - open/close 20000 (time 1786754534.95 total 962.02 last 69.84) - open/close 20780 (time 1786754544.95 total 972.02 last 77.99) - open/close 21508 (time 1786754554.96 total 982.03 last 72.76) - open/close 22219 (time 1786754564.97 total 992.04 last 71.04) - open/close 22943 (time 1786754574.97 total 1002.04 last 72.39) - open/close 23644 (time 1786754584.98 total 1012.05 last 70.01) - open/close 24300 (time 1786754594.99 total 1022.06 last 65.56) - open/close 24984 (time 1786754605.00 total 1032.07 last 68.35) - open/close 25736 (time 1786754615.04 total 1042.11 last 74.90) - open/close 26450 (time 1786754625.05 total 1052.12 last 71.32) - open/close 27092 (time 1786754635.06 total 1062.13 last 64.14) - open/close 27748 (time 1786754645.07 total 1072.14 last 65.48) - open/close 28368 (time 1786754655.09 total 1082.16 last 61.93) - open/close 29102 (time 1786754665.10 total 1092.17 last 73.28) - open/close 29771 (time 1786754675.11 total 1102.18 last 66.84) - open/close 30000 (time 1786754678.80 total 1105.87 last 62.09) - open/close 30592 (time 1786754688.80 total 1115.87 last 59.20) - open/close 31256 (time 1786754698.82 total 1125.89 last 66.25) - open/close 31862 (time 1786754708.83 total 1135.90 last 60.55) - open/close 32509 (time 1786754718.84 total 1145.91 last 64.62) - open/close 33193 (time 1786754728.85 total 1155.92 last 68.35) - open/close 33841 (time 1786754738.87 total 1165.94 last 64.69) - open/close 34527 (time 1786754748.87 total 1175.94 last 68.56) - open/close 35247 (time 1786754758.88 total 1185.95 last 71.93) - open/close 35944 (time 1786754768.88 total 1195.95 last 69.69) - open/close 36686 (time 1786754778.92 total 1205.99 last 73.93) - open/close 37353 (time 1786754788.92 total 1215.99 last 66.68) - open/close 38046 (time 1786754798.92 total 1225.99 last 69.30) - open/close 38764 (time 1786754808.92 total 1235.99 last 71.80) - open/close 39420 (time 1786754818.93 total 1246.00 last 65.53) - open/close 40000 (time 1786754827.80 total 1254.87 last 65.41) - open/close 40548 (time 1786754837.81 total 1264.88 last 54.75) - open/close 41156 (time 1786754847.81 total 1274.88 last 60.80) - open/close 41734 (time 1786754857.83 total 1284.90 last 57.69) - open/close 42276 (time 1786754867.85 total 1294.92 last 54.11) - open/close 42930 (time 1786754877.87 total 1304.94 last 65.24) - open/close 43532 (time 1786754887.88 total 1314.95 last 60.16) - open/close 44185 (time 1786754897.90 total 1324.97 last 65.15) - open/close 44942 (time 1786754907.94 total 1335.01 last 75.42) - open/close 45703 (time 1786754917.95 total 1345.01 last 76.06) - open/close 46393 (time 1786754927.95 total 1355.02 last 69.00) - open/close 47155 (time 1786754937.96 total 1365.03 last 76.08) - open/close 47897 (time 1786754947.96 total 1375.03 last 74.19) - open/close 48659 (time 1786754957.99 total 1385.06 last 76.00) - open/close 49312 (time 1786754967.99 total 1395.06 last 65.28) total: 50000 open/close in 1404.55 seconds: 35.60 ops/second - unlinked 0 (time 1786754981 ; total 0 ; last 0) - unlinked 10000 (time 1786755148 ; total 167 ; last 167) - unlinked 20000 (time 1786755300 ; total 319 ; last 152) - unlinked 30000 (time 1786755462 ; total 481 ; last 162) - unlinked 40000 (time 1786755618 ; total 637 ; last 156) total: 50000 unlinks in 795 seconds: 62.893082 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg627-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg627-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg627-server stop mds service on oleg627-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg627-server unloading modules via unload_modules_local on: 'oleg627-server' oleg627-server: oleg627-server.virtnet: executing unload_modules_local oleg627-server: modules unloaded. pdsh@oleg627-client: oleg627-client: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-client: ssh exited with exit code 2 pdsh@oleg627-client: oleg627-server: ssh exited with exit code 2