-----============= acceptance-small: conf-sanity ============----- Fri Oct 10 15:45:51 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg432-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg432-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg432-server oleg432-server: oleg432-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg432-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg432-server: oleg432-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg432-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:47:46 (1760125666) Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) oleg432-server: oleg432-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg432-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1015 (time 1760125730.90 total 10.00 last 101.48) - open/close 1976 (time 1760125740.90 total 20.00 last 96.07) - open/close 3044 (time 1760125750.91 total 30.02 last 106.69) - open/close 4200 (time 1760125760.91 total 40.02 last 115.52) - open/close 5079 (time 1760125770.92 total 50.03 last 87.84) - open/close 5995 (time 1760125780.92 total 60.03 last 91.57) - open/close 6969 (time 1760125790.93 total 70.04 last 97.35) - open/close 7896 (time 1760125800.93 total 80.04 last 92.68) - open/close 8806 (time 1760125810.94 total 90.04 last 90.98) - open/close 9760 (time 1760125820.94 total 100.05 last 95.35) - open/close 10000 (time 1760125823.19 total 102.30 last 106.51) - open/close 10854 (time 1760125833.20 total 112.31 last 85.35) - open/close 11711 (time 1760125843.21 total 122.31 last 85.65) - open/close 12661 (time 1760125853.21 total 132.31 last 95.00) - open/close 13677 (time 1760125863.21 total 142.32 last 101.57) - open/close 14796 (time 1760125873.22 total 152.32 last 111.82) - open/close 15867 (time 1760125883.22 total 162.32 last 107.09) - open/close 16935 (time 1760125893.22 total 172.33 last 106.74) - open/close 18307 (time 1760125903.22 total 182.33 last 137.20) - open/close 19889 (time 1760125913.23 total 192.34 last 158.12) - open/close 21872 (time 1760125923.23 total 202.34 last 198.26) - open/close 23509 (time 1760125933.25 total 212.36 last 163.33) - open/close 25011 (time 1760125943.26 total 222.36 last 150.17) - open/close 26536 (time 1760125953.26 total 232.37 last 152.41) - open/close 28486 (time 1760125963.27 total 242.37 last 194.91) - open/close 30000 (time 1760125972.63 total 251.74 last 161.70) - open/close 31855 (time 1760125982.63 total 261.74 last 185.43) - open/close 34197 (time 1760125992.63 total 271.74 last 234.17) - open/close 36458 (time 1760126002.64 total 281.74 last 226.05) - open/close 38589 (time 1760126012.64 total 291.75 last 213.02) - open/close 40000 (time 1760126018.25 total 297.36 last 251.31) - open/close 42203 (time 1760126028.26 total 307.36 last 220.24) - open/close 44275 (time 1760126038.26 total 317.37 last 207.12) - open/close 46379 (time 1760126048.26 total 327.37 last 210.34) - open/close 48623 (time 1760126058.27 total 337.38 last 224.26) - open/close 50000 (time 1760126064.54 total 343.65 last 219.47) - open/close 51829 (time 1760126074.55 total 353.65 last 182.84) - open/close 53389 (time 1760126084.55 total 363.66 last 155.97) - open/close 54734 (time 1760126094.55 total 373.66 last 134.49) - open/close 56121 (time 1760126104.56 total 383.67 last 138.58) - open/close 57863 (time 1760126114.56 total 393.67 last 174.19) - open/close 59367 (time 1760126124.56 total 403.67 last 150.38) - open/close 60000 (time 1760126128.76 total 407.86 last 150.81) - open/close 61728 (time 1760126138.76 total 417.87 last 172.79) - open/close 63432 (time 1760126148.76 total 427.87 last 170.40) total: 64768 open/close in 436.31 seconds: 148.44 ops/second - /unlink 4036 (time 1760126169.00 total 10.00 last 403.58) - /unlink 7498 (time 1760126179.02 total 20.02 last 345.41) - /unlink 10000 (time 1760126185.84 total 26.85 last 366.77) - /unlink 12763 (time 1760126195.84 total 36.85 last 276.30) - /unlink 15554 (time 1760126205.85 total 46.85 last 278.99) - /unlink 18720 (time 1760126215.85 total 56.85 last 316.54) - /unlink 20000 (time 1760126219.88 total 60.89 last 317.27) - /unlink 23305 (time 1760126229.88 total 70.89 last 330.40) - /unlink 26562 (time 1760126239.89 total 80.89 last 325.67) - /unlink 30000 (time 1760126249.05 total 90.05 last 375.30) - /unlink 33046 (time 1760126259.05 total 100.05 last 304.55) - /unlink 36245 (time 1760126269.05 total 110.05 last 319.86) - /unlink 40000 (time 1760126277.97 total 118.97 last 421.12) - /unlink 44283 (time 1760126287.97 total 128.97 last 428.25) - /unlink 48071 (time 1760126297.97 total 138.97 last 378.74) - /unlink 50000 (time 1760126303.24 total 144.25 last 365.77) - /unlink 53867 (time 1760126313.24 total 154.25 last 386.70) - /unlink 57640 (time 1760126323.24 total 164.25 last 377.24) - /unlink 60000 (time 1760126329.40 total 170.41 last 383.29) - /unlink 63815 (time 1760126339.40 total 180.41 last 381.43) total: 64768 /unlink in 182.69 seconds: 354.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2588 (time 1760126371.07 total 10.00 last 258.71) - open/close 5033 (time 1760126381.08 total 20.00 last 244.47) - open/close 7531 (time 1760126391.08 total 30.00 last 249.79) - open/close 9982 (time 1760126401.08 total 40.01 last 245.00) - open/close 12000 (time 1760126411.08 total 50.01 last 201.76) - open/close 12888 (time 1760126421.09 total 60.02 last 88.73) - open/close 13381 (time 1760126431.09 total 70.02 last 49.30) - open/close 13782 (time 1760126441.11 total 80.04 last 40.02) - open/close 14116 (time 1760126451.11 total 90.04 last 33.38) - open/close 14687 (time 1760126461.13 total 100.06 last 57.03) - open/close 15230 (time 1760126471.14 total 110.07 last 54.24) - open/close 15692 (time 1760126481.15 total 120.08 last 46.15) - open/close 16433 (time 1760126491.17 total 130.10 last 73.93) - open/close 17191 (time 1760126501.18 total 140.11 last 75.73) - open/close 17890 (time 1760126511.18 total 150.11 last 69.88) - open/close 18699 (time 1760126521.19 total 160.11 last 80.89) - open/close 19530 (time 1760126531.19 total 170.12 last 83.10) - open/close 20000 (time 1760126536.45 total 175.38 last 89.31) - open/close 20857 (time 1760126546.45 total 185.38 last 85.66) - open/close 21653 (time 1760126556.46 total 195.39 last 79.56) - open/close 22431 (time 1760126566.46 total 205.39 last 77.78) - open/close 23245 (time 1760126576.47 total 215.39 last 81.36) - open/close 24043 (time 1760126586.47 total 225.40 last 79.73) - open/close 24851 (time 1760126596.48 total 235.40 last 80.80) - open/close 25681 (time 1760126606.49 total 245.42 last 82.89) - open/close 26476 (time 1760126616.49 total 255.42 last 79.49) - open/close 27305 (time 1760126626.49 total 265.42 last 82.86) - open/close 28240 (time 1760126636.50 total 275.43 last 93.41) - open/close 29081 (time 1760126646.51 total 285.44 last 84.08) - open/close 29835 (time 1760126656.51 total 295.44 last 75.38) - open/close 30000 (time 1760126658.62 total 297.55 last 78.07) - open/close 30767 (time 1760126668.63 total 307.55 last 76.68) - open/close 31530 (time 1760126678.64 total 317.57 last 76.18) - open/close 32305 (time 1760126688.64 total 327.57 last 77.48) - open/close 33046 (time 1760126698.66 total 337.58 last 74.01) - open/close 33679 (time 1760126708.68 total 347.61 last 63.17) - open/close 34222 (time 1760126718.70 total 357.63 last 54.15) - open/close 34897 (time 1760126728.71 total 367.64 last 67.46) - open/close 35597 (time 1760126738.73 total 377.66 last 69.87) - open/close 36232 (time 1760126748.74 total 387.67 last 63.44) - open/close 36932 (time 1760126758.74 total 397.67 last 70.00) - open/close 37597 (time 1760126768.74 total 407.67 last 66.47) - open/close 38325 (time 1760126778.75 total 417.68 last 72.72) - open/close 39021 (time 1760126788.77 total 427.69 last 69.52) - open/close 39719 (time 1760126798.79 total 437.72 last 69.62) - open/close 40000 (time 1760126803.18 total 442.11 last 64.04) - open/close 40651 (time 1760126813.18 total 452.11 last 65.07) - open/close 41307 (time 1760126823.20 total 462.13 last 65.50) - open/close 41918 (time 1760126833.21 total 472.14 last 61.05) - open/close 42583 (time 1760126843.21 total 482.14 last 66.49) - open/close 43258 (time 1760126853.21 total 492.14 last 67.48) - open/close 43883 (time 1760126863.22 total 502.15 last 62.43) - open/close 44545 (time 1760126873.24 total 512.17 last 66.06) - open/close 45185 (time 1760126883.25 total 522.18 last 63.98) - open/close 45844 (time 1760126893.25 total 532.18 last 65.89) - open/close 46358 (time 1760126903.26 total 542.19 last 51.35) - open/close 47034 (time 1760126913.28 total 552.21 last 67.45) - open/close 47712 (time 1760126923.29 total 562.22 last 67.78) - open/close 48463 (time 1760126933.29 total 572.22 last 75.09) - open/close 49237 (time 1760126943.29 total 582.22 last 77.37) - open/close 50000 (time 1760126952.84 total 591.77 last 79.92) - open/close 50754 (time 1760126962.85 total 601.78 last 75.33) - open/close 51565 (time 1760126972.85 total 611.78 last 81.07) - open/close 52243 (time 1760126982.86 total 621.78 last 67.77) - open/close 52920 (time 1760126992.86 total 631.79 last 67.69) - open/close 53555 (time 1760127002.87 total 641.80 last 63.39) - open/close 54202 (time 1760127012.88 total 651.81 last 64.69) - open/close 54914 (time 1760127022.88 total 661.81 last 71.18) - open/close 55534 (time 1760127032.89 total 671.81 last 61.96) - open/close 56271 (time 1760127042.91 total 681.84 last 73.55) - open/close 56919 (time 1760127052.92 total 691.85 last 64.70) - open/close 57544 (time 1760127062.94 total 701.87 last 62.41) - open/close 58217 (time 1760127072.97 total 711.90 last 67.06) - open/close 58910 (time 1760127082.98 total 721.91 last 69.28) - open/close 59611 (time 1760127092.98 total 731.91 last 70.10) - open/close 60000 (time 1760127098.36 total 737.29 last 72.25) - open/close 60691 (time 1760127108.37 total 747.30 last 69.05) - open/close 61395 (time 1760127118.37 total 757.30 last 70.37) - open/close 62069 (time 1760127128.37 total 767.30 last 67.39) - open/close 62721 (time 1760127138.40 total 777.32 last 65.06) - open/close 63440 (time 1760127148.41 total 787.33 last 71.82) - open/close 64149 (time 1760127158.42 total 797.35 last 70.79) total: 64768 open/close in 806.40 seconds: 80.32 ops/second - /unlink 675 (time 1760127181.56 total 10.00 last 67.48) - /unlink 1317 (time 1760127191.56 total 20.01 last 64.16) - /unlink 2054 (time 1760127201.57 total 30.02 last 73.63) - /unlink 2789 (time 1760127211.58 total 40.02 last 73.46) - /unlink 3584 (time 1760127221.59 total 50.03 last 79.43) - /unlink 4427 (time 1760127231.59 total 60.04 last 84.23) - /unlink 5260 (time 1760127241.60 total 70.04 last 83.30) - /unlink 6031 (time 1760127251.60 total 80.05 last 77.06) - /unlink 6783 (time 1760127261.60 total 90.05 last 75.20) - /unlink 7561 (time 1760127271.63 total 100.08 last 77.56) - /unlink 8374 (time 1760127281.63 total 110.08 last 81.28) - /unlink 9172 (time 1760127291.64 total 120.09 last 79.76) - /unlink 9965 (time 1760127301.64 total 130.09 last 79.28) - /unlink 10716 (time 1760127311.64 total 140.09 last 75.10) - /unlink 11430 (time 1760127321.64 total 150.09 last 71.38) - /unlink 12123 (time 1760127331.66 total 160.11 last 69.20) - /unlink 12785 (time 1760127341.66 total 170.11 last 66.16) - /unlink 13497 (time 1760127351.67 total 180.12 last 71.15) - /unlink 14268 (time 1760127361.68 total 190.12 last 77.06) - /unlink 15050 (time 1760127371.69 total 200.13 last 78.10) - /unlink 15819 (time 1760127381.70 total 210.14 last 76.84) - /unlink 16581 (time 1760127391.70 total 220.15 last 76.15) - /unlink 17332 (time 1760127401.71 total 230.16 last 75.02) - /unlink 18070 (time 1760127411.72 total 240.17 last 73.77) - /unlink 18868 (time 1760127421.74 total 250.19 last 79.61) - /unlink 19664 (time 1760127431.75 total 260.20 last 79.53) - /unlink 20000 (time 1760127436.18 total 264.63 last 75.89) - /unlink 20811 (time 1760127446.18 total 274.63 last 81.09) - /unlink 21619 (time 1760127456.19 total 284.64 last 80.70) - /unlink 22544 (time 1760127466.20 total 294.65 last 92.44) - /unlink 23351 (time 1760127476.22 total 304.66 last 80.56) - /unlink 24109 (time 1760127486.22 total 314.67 last 75.74) - /unlink 24900 (time 1760127496.23 total 324.67 last 79.08) - /unlink 25677 (time 1760127506.23 total 334.67 last 77.69) - /unlink 26465 (time 1760127516.24 total 344.68 last 78.74) - /unlink 27257 (time 1760127526.24 total 354.69 last 79.16) - /unlink 28072 (time 1760127536.25 total 364.69 last 81.46) - /unlink 28856 (time 1760127546.26 total 374.70 last 78.32) - /unlink 29653 (time 1760127556.26 total 384.71 last 79.66) - /unlink 30000 (time 1760127561.08 total 389.53 last 71.95) - /unlink 30717 (time 1760127571.09 total 399.54 last 71.66) - /unlink 31469 (time 1760127581.10 total 409.55 last 75.09) - /unlink 32219 (time 1760127591.11 total 419.56 last 74.92) - /unlink 32942 (time 1760127601.12 total 429.57 last 72.27) - /unlink 33655 (time 1760127611.13 total 439.58 last 71.21) - /unlink 34424 (time 1760127621.13 total 449.58 last 76.89) - /unlink 35128 (time 1760127631.14 total 459.59 last 70.36) - /unlink 35865 (time 1760127641.15 total 469.59 last 73.64) - /unlink 36575 (time 1760127651.17 total 479.62 last 70.83) - /unlink 37374 (time 1760127661.18 total 489.63 last 79.79) - /unlink 38120 (time 1760127671.19 total 499.64 last 74.56) - /unlink 38886 (time 1760127681.22 total 509.67 last 76.32) - /unlink 39635 (time 1760127691.23 total 519.68 last 74.84) - /unlink 40000 (time 1760127695.87 total 524.32 last 78.62) - /unlink 40742 (time 1760127705.90 total 534.34 last 74.03) - /unlink 41527 (time 1760127715.91 total 544.36 last 78.41) - /unlink 42286 (time 1760127725.91 total 554.36 last 75.89) - /unlink 43039 (time 1760127735.92 total 564.37 last 75.23) - /unlink 43820 (time 1760127745.93 total 574.38 last 78.02) - /unlink 44579 (time 1760127755.94 total 584.39 last 75.79) - /unlink 45331 (time 1760127765.95 total 594.39 last 75.18) - /unlink 46123 (time 1760127775.95 total 604.40 last 79.14) - /unlink 46927 (time 1760127785.96 total 614.40 last 80.38) - /unlink 47708 (time 1760127795.96 total 624.41 last 78.06) - /unlink 48525 (time 1760127805.97 total 634.41 last 81.65) - /unlink 49284 (time 1760127815.98 total 644.42 last 75.83) - /unlink 50000 (time 1760127825.02 total 653.47 last 79.17) - /unlink 50827 (time 1760127835.05 total 663.49 last 82.49) - /unlink 51651 (time 1760127845.06 total 673.50 last 82.31) - /unlink 52581 (time 1760127855.06 total 683.51 last 92.98) - /unlink 53499 (time 1760127865.06 total 693.51 last 91.80) - /unlink 54311 (time 1760127875.07 total 703.51 last 81.15) - /unlink 55193 (time 1760127885.07 total 713.52 last 88.15) - /unlink 55994 (time 1760127895.08 total 723.52 last 80.05) - /unlink 56845 (time 1760127905.08 total 733.53 last 85.10) - /unlink 57823 (time 1760127915.08 total 743.53 last 97.74) - /unlink 58733 (time 1760127925.09 total 753.54 last 90.94) - /unlink 59624 (time 1760127935.09 total 763.54 last 89.07) - /unlink 60000 (time 1760127939.10 total 767.54 last 93.96) - /unlink 61020 (time 1760127949.10 total 777.55 last 101.91) - /unlink 61895 (time 1760127959.11 total 787.56 last 87.43) - /unlink 62838 (time 1760127969.12 total 797.57 last 94.22) - /unlink 63635 (time 1760127979.13 total 807.57 last 79.66) - /unlink 64505 (time 1760127989.13 total 817.58 last 86.94) total: 64768 /unlink in 820.79 seconds: 78.91 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 803 (time 1760128021.60 total 10.01 last 80.22) - open/close 1382 (time 1760128031.62 total 20.03 last 57.78) - open/close 2018 (time 1760128041.63 total 30.04 last 63.56) - open/close 2809 (time 1760128051.63 total 40.04 last 79.09) - open/close 3635 (time 1760128061.64 total 50.05 last 82.48) - open/close 4481 (time 1760128071.64 total 60.05 last 84.59) - open/close 5283 (time 1760128081.66 total 70.06 last 80.12) - open/close 6079 (time 1760128091.66 total 80.07 last 79.54) - open/close 6825 (time 1760128101.67 total 90.08 last 74.52) - open/close 7606 (time 1760128111.68 total 100.09 last 78.05) - open/close 8401 (time 1760128121.69 total 110.10 last 79.44) - open/close 9140 (time 1760128131.69 total 120.10 last 73.88) - open/close 9942 (time 1760128141.69 total 130.10 last 80.17) - open/close 10754 (time 1760128151.70 total 140.11 last 81.16) - open/close 11525 (time 1760128161.70 total 150.11 last 77.07) - open/close 12312 (time 1760128171.71 total 160.12 last 78.65) - open/close 13088 (time 1760128181.71 total 170.12 last 77.59) - open/close 13847 (time 1760128191.72 total 180.13 last 75.82) - open/close 14677 (time 1760128201.73 total 190.14 last 82.93) - open/close 15503 (time 1760128211.74 total 200.15 last 82.49) - open/close 16271 (time 1760128221.75 total 210.16 last 76.71) - open/close 17122 (time 1760128231.77 total 220.17 last 85.01) - open/close 17967 (time 1760128241.77 total 230.18 last 84.45) - open/close 18797 (time 1760128251.77 total 240.18 last 82.98) - open/close 19634 (time 1760128261.78 total 250.19 last 83.68) - open/close 20000 (time 1760128266.07 total 254.48 last 85.23) - open/close 20800 (time 1760128276.08 total 264.48 last 79.96) - open/close 21625 (time 1760128286.08 total 274.49 last 82.45) - open/close 22415 (time 1760128296.10 total 284.50 last 78.90) - open/close 23162 (time 1760128306.10 total 294.50 last 74.69) - open/close 23903 (time 1760128316.12 total 304.53 last 73.94) - open/close 24628 (time 1760128326.12 total 314.53 last 72.49) - open/close 25448 (time 1760128336.12 total 324.53 last 81.97) - open/close 26253 (time 1760128346.12 total 334.53 last 80.50) - open/close 27104 (time 1760128356.13 total 344.54 last 85.05) - open/close 27888 (time 1760128366.13 total 354.54 last 78.39) - open/close 28717 (time 1760128376.14 total 364.54 last 82.86) - open/close 29525 (time 1760128386.14 total 374.55 last 80.74) - open/close 30000 (time 1760128391.81 total 380.21 last 83.87) - open/close 30883 (time 1760128401.81 total 390.22 last 88.29) - open/close 31810 (time 1760128411.82 total 400.23 last 92.57) - open/close 32969 (time 1760128421.83 total 410.23 last 115.87) - open/close 33898 (time 1760128431.83 total 420.24 last 92.87) - open/close 34823 (time 1760128441.83 total 430.24 last 92.47) - open/close 35734 (time 1760128451.84 total 440.25 last 91.03) - open/close 36851 (time 1760128461.85 total 450.26 last 111.59) - open/close 37744 (time 1760128471.86 total 460.27 last 89.19) - open/close 38687 (time 1760128481.86 total 470.27 last 94.30) - open/close 39626 (time 1760128491.87 total 480.28 last 93.84) - open/close 40000 (time 1760128495.63 total 484.04 last 99.42) - open/close 40943 (time 1760128505.63 total 494.04 last 94.27) - open/close 41950 (time 1760128515.64 total 504.04 last 100.67) - open/close 42915 (time 1760128525.64 total 514.05 last 96.45) - open/close 43707 (time 1760128535.64 total 524.05 last 79.19) - open/close 44581 (time 1760128545.64 total 534.05 last 87.39) - open/close 45467 (time 1760128555.65 total 544.06 last 88.53) - open/close 46324 (time 1760128565.66 total 554.07 last 85.59) - open/close 47201 (time 1760128575.68 total 564.08 last 87.60) - open/close 48183 (time 1760128585.68 total 574.09 last 98.18) - open/close 49132 (time 1760128595.68 total 584.09 last 94.86) - open/close 50000 (time 1760128602.34 total 590.75 last 130.34) - open/close 51120 (time 1760128612.35 total 600.76 last 111.92) - open/close 52235 (time 1760128622.36 total 610.76 last 111.41) - open/close 53320 (time 1760128632.36 total 620.77 last 108.45) - open/close 54430 (time 1760128642.37 total 630.78 last 110.91) - open/close 55445 (time 1760128652.37 total 640.78 last 101.48) - open/close 56529 (time 1760128662.38 total 650.79 last 108.31) - open/close 57487 (time 1760128672.38 total 660.79 last 95.79) - open/close 58415 (time 1760128682.38 total 670.79 last 92.79) - open/close 59486 (time 1760128692.42 total 680.83 last 106.69) - open/close 60000 (time 1760128695.74 total 684.15 last 154.81) - open/close 61053 (time 1760128705.75 total 694.16 last 105.20) - open/close 61961 (time 1760128715.76 total 704.16 last 90.75) - open/close 62938 (time 1760128725.76 total 714.17 last 97.62) - open/close 63905 (time 1760128735.77 total 724.18 last 96.66) total: 64768 open/close in 733.22 seconds: 88.33 ops/second - /unlink 1015 (time 1760128757.47 total 10.00 last 101.47) - /unlink 1975 (time 1760128767.48 total 20.01 last 95.94) - /unlink 2983 (time 1760128777.48 total 30.01 last 100.76) - /unlink 4000 (time 1760128787.50 total 40.03 last 101.56) - /unlink 4932 (time 1760128797.50 total 50.03 last 93.14) - /unlink 6159 (time 1760128807.51 total 60.04 last 122.63) - /unlink 7381 (time 1760128817.51 total 70.04 last 122.14) - /unlink 8380 (time 1760128827.52 total 80.05 last 99.84) - /unlink 9577 (time 1760128837.53 total 90.06 last 119.55) - /unlink 10000 (time 1760128841.00 total 93.53 last 122.04) - /unlink 11222 (time 1760128851.01 total 103.53 last 122.13) - /unlink 12235 (time 1760128861.01 total 113.54 last 101.27) - /unlink 13418 (time 1760128871.01 total 123.54 last 118.26) - /unlink 14579 (time 1760128881.01 total 133.54 last 116.07) - /unlink 15759 (time 1760128891.02 total 143.55 last 117.93) - /unlink 17135 (time 1760128901.03 total 153.55 last 137.54) - /unlink 18694 (time 1760128911.03 total 163.55 last 155.89) - /unlink 19999 (time 1760128921.03 total 173.56 last 130.43) - /unlink 21288 (time 1760128931.04 total 183.56 last 128.84) - /unlink 22230 (time 1760128941.05 total 193.57 last 94.11) - /unlink 23076 (time 1760128951.05 total 203.58 last 84.55) - /unlink 23949 (time 1760128961.05 total 213.58 last 87.29) - /unlink 24735 (time 1760128971.06 total 223.59 last 78.51) - /unlink 25775 (time 1760128981.07 total 233.60 last 103.94) - /unlink 26654 (time 1760128991.07 total 243.60 last 87.89) - /unlink 27678 (time 1760129001.08 total 253.61 last 102.28) - /unlink 28480 (time 1760129011.09 total 263.62 last 80.15) - /unlink 29613 (time 1760129021.09 total 273.62 last 113.24) - /unlink 30000 (time 1760129024.98 total 277.51 last 99.61) - /unlink 30790 (time 1760129034.99 total 287.52 last 78.94) - /unlink 31601 (time 1760129044.99 total 297.52 last 81.08) - /unlink 32429 (time 1760129055.00 total 307.52 last 82.76) - /unlink 33324 (time 1760129065.00 total 317.53 last 89.45) - /unlink 34294 (time 1760129075.00 total 327.53 last 96.96) - /unlink 35244 (time 1760129085.01 total 337.54 last 94.94) - /unlink 36295 (time 1760129095.01 total 347.54 last 105.10) - /unlink 37462 (time 1760129105.02 total 357.55 last 116.62) - /unlink 38446 (time 1760129115.03 total 367.55 last 98.33) - /unlink 39485 (time 1760129125.03 total 377.56 last 103.85) - /unlink 40000 (time 1760129130.80 total 383.32 last 89.30) - /unlink 40959 (time 1760129140.80 total 393.32 last 95.90) - /unlink 42072 (time 1760129150.80 total 403.33 last 111.29) - /unlink 43370 (time 1760129160.80 total 413.33 last 129.71) - /unlink 44933 (time 1760129170.81 total 423.34 last 156.25) - /unlink 46455 (time 1760129180.81 total 433.34 last 152.14) - /unlink 47700 (time 1760129190.82 total 443.35 last 124.37) - /unlink 48839 (time 1760129200.83 total 453.35 last 113.85) - /unlink 49897 (time 1760129210.83 total 463.36 last 105.78) - /unlink 50916 (time 1760129220.83 total 473.36 last 101.87) - /unlink 51980 (time 1760129230.83 total 483.36 last 106.40) - /unlink 53196 (time 1760129240.84 total 493.37 last 121.49) - /unlink 54114 (time 1760129250.85 total 503.38 last 91.69) - /unlink 55008 (time 1760129260.87 total 513.39 last 89.28) - /unlink 55970 (time 1760129270.87 total 523.40 last 96.12) - /unlink 56859 (time 1760129280.88 total 533.41 last 88.86) - /unlink 57723 (time 1760129290.88 total 543.41 last 86.38) - /unlink 58520 (time 1760129300.88 total 553.41 last 79.70) - /unlink 59462 (time 1760129310.90 total 563.43 last 93.99) - /unlink 60000 (time 1760129317.33 total 569.86 last 83.71) - /unlink 60806 (time 1760129327.34 total 579.86 last 80.54) - /unlink 61702 (time 1760129337.34 total 589.87 last 89.58) - /unlink 62479 (time 1760129347.34 total 599.87 last 77.69) - /unlink 63285 (time 1760129357.34 total 609.87 last 80.57) - /unlink 64113 (time 1760129367.35 total 619.87 last 82.78) total: 64768 /unlink in 627.15 seconds: 103.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1040 (time 1760129402.70 total 10.01 last 103.89) - open/close 2074 (time 1760129412.71 total 20.02 last 103.35) - open/close 3070 (time 1760129422.72 total 30.03 last 99.50) - open/close 4140 (time 1760129432.72 total 40.03 last 106.99) - open/close 5234 (time 1760129442.73 total 50.03 last 109.32) - open/close 6243 (time 1760129452.73 total 60.04 last 100.87) - open/close 7383 (time 1760129462.73 total 70.04 last 113.94) - open/close 8557 (time 1760129472.74 total 80.05 last 117.32) - open/close 9710 (time 1760129482.74 total 90.05 last 115.30) - open/close 10000 (time 1760129485.36 total 92.66 last 110.95) - open/close 11339 (time 1760129495.36 total 102.67 last 133.83) - open/close 12485 (time 1760129505.37 total 112.67 last 114.53) - open/close 13510 (time 1760129515.37 total 122.68 last 102.43) - open/close 14605 (time 1760129525.38 total 132.68 last 109.46) - open/close 15676 (time 1760129535.38 total 142.69 last 107.04) - open/close 16850 (time 1760129545.38 total 152.69 last 117.38) - open/close 18183 (time 1760129555.38 total 162.69 last 133.29) - open/close 19522 (time 1760129565.39 total 172.70 last 133.82) - open/close 20000 (time 1760129569.43 total 176.74 last 118.26) - open/close 21108 (time 1760129579.44 total 186.74 last 110.75) - open/close 22347 (time 1760129589.44 total 196.74 last 123.89) - open/close 23752 (time 1760129599.44 total 206.75 last 140.43) - open/close 24966 (time 1760129609.45 total 216.75 last 121.37) - open/close 26211 (time 1760129619.45 total 226.75 last 124.47) - open/close 27496 (time 1760129629.45 total 236.76 last 128.46) - open/close 28607 (time 1760129639.45 total 246.76 last 111.07) - open/close 29655 (time 1760129649.46 total 256.76 last 104.77) - open/close 30000 (time 1760129652.01 total 259.31 last 135.27) - open/close 31434 (time 1760129662.01 total 269.32 last 143.31) - open/close 32825 (time 1760129672.02 total 279.33 last 139.00) - open/close 34224 (time 1760129682.03 total 289.33 last 139.83) - open/close 35768 (time 1760129692.03 total 299.34 last 154.32) - open/close 37296 (time 1760129702.03 total 309.34 last 152.79) - open/close 38464 (time 1760129712.04 total 319.35 last 116.68) - open/close 39896 (time 1760129722.04 total 329.35 last 143.18) - open/close 41354 (time 1760129732.05 total 339.36 last 145.71) - open/close 42938 (time 1760129742.05 total 349.36 last 158.32) - open/close 44251 (time 1760129752.06 total 359.36 last 131.28) - open/close 45499 (time 1760129762.06 total 369.36 last 124.79) - open/close 47143 (time 1760129772.06 total 379.37 last 164.31) - open/close 48186 (time 1760129782.08 total 389.38 last 104.14) - open/close 49382 (time 1760129792.08 total 399.39 last 119.56) - open/close 50000 (time 1760129797.03 total 404.34 last 124.77) - open/close 51596 (time 1760129807.04 total 414.35 last 159.44) - open/close 53061 (time 1760129817.05 total 424.35 last 146.47) - open/close 54813 (time 1760129827.05 total 434.36 last 175.14) - open/close 56564 (time 1760129837.05 total 444.36 last 175.09) - open/close 58645 (time 1760129847.05 total 454.36 last 208.02) - open/close 60000 (time 1760129855.65 total 462.96 last 157.62) - open/close 61734 (time 1760129865.65 total 472.96 last 173.35) - open/close 63331 (time 1760129875.66 total 482.96 last 159.62) total: 64768 open/close in 489.85 seconds: 132.22 ops/second - /unlink 3144 (time 1760129894.29 total 10.00 last 314.37) - /unlink 6055 (time 1760129904.30 total 20.01 last 290.98) - /unlink 9787 (time 1760129914.30 total 30.01 last 373.19) - /unlink 13366 (time 1760129924.32 total 40.03 last 357.12) - /unlink 15960 (time 1760129934.32 total 50.03 last 259.35) - /unlink 19207 (time 1760129944.32 total 60.03 last 324.57) - /unlink 20000 (time 1760129946.78 total 62.49 last 322.34) - /unlink 24250 (time 1760129956.78 total 72.49 last 424.96) - /unlink 28220 (time 1760129966.79 total 82.49 last 396.98) - /unlink 30000 (time 1760129971.90 total 87.61 last 348.26) - /unlink 33819 (time 1760129981.90 total 97.61 last 381.76) - /unlink 38554 (time 1760129991.91 total 107.62 last 473.22) - /unlink 40000 (time 1760129995.79 total 111.50 last 372.06) - /unlink 43963 (time 1760130005.79 total 121.50 last 396.20) - /unlink 46767 (time 1760130015.80 total 131.51 last 280.31) - /unlink 50000 (time 1760130024.87 total 140.58 last 356.48) - /unlink 53687 (time 1760130034.87 total 150.58 last 368.57) - /unlink 56562 (time 1760130044.87 total 160.58 last 287.44) - /unlink 59593 (time 1760130054.88 total 170.59 last 303.01) - /unlink 62511 (time 1760130064.88 total 180.59 last 291.77) total: 64768 /unlink in 186.74 seconds: 346.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1545 (time 1760130102.81 total 10.00 last 154.48) - open/close 3269 (time 1760130112.81 total 20.00 last 172.36) - open/close 4688 (time 1760130122.82 total 30.01 last 141.81) - open/close 6201 (time 1760130132.82 total 40.01 last 151.29) - open/close 7694 (time 1760130142.82 total 50.02 last 149.22) - open/close 8929 (time 1760130152.83 total 60.02 last 123.41) - open/close 10000 (time 1760130160.85 total 68.04 last 133.62) - open/close 11582 (time 1760130170.85 total 78.04 last 158.13) - open/close 13055 (time 1760130180.86 total 88.05 last 147.19) - open/close 14439 (time 1760130190.86 total 98.05 last 138.40) - open/close 16156 (time 1760130200.87 total 108.06 last 171.56) - open/close 18331 (time 1760130210.87 total 118.06 last 217.47) - open/close 20000 (time 1760130220.11 total 127.30 last 180.57) - open/close 21959 (time 1760130230.11 total 137.31 last 195.86) - open/close 23556 (time 1760130240.12 total 147.31 last 159.58) - open/close 25305 (time 1760130250.13 total 157.32 last 174.78) - open/close 27019 (time 1760130260.13 total 167.32 last 171.35) - open/close 29079 (time 1760130270.14 total 177.33 last 205.79) - open/close 30000 (time 1760130275.00 total 182.19 last 189.57) - open/close 31767 (time 1760130285.00 total 192.20 last 176.62) - open/close 34101 (time 1760130295.01 total 202.20 last 233.33) - open/close 36738 (time 1760130305.01 total 212.20 last 263.69) - open/close 39365 (time 1760130315.01 total 222.20 last 262.67) - open/close 40000 (time 1760130317.35 total 224.54 last 271.29) - open/close 42374 (time 1760130327.35 total 234.54 last 237.33) - open/close 44658 (time 1760130337.35 total 244.55 last 228.30) - open/close 47135 (time 1760130347.36 total 254.55 last 247.67) - open/close 50000 (time 1760130356.86 total 264.05 last 301.44) - open/close 53247 (time 1760130366.86 total 274.06 last 324.65) - open/close 55545 (time 1760130376.97 total 284.16 last 227.37) - open/close 58601 (time 1760130386.97 total 294.16 last 305.60) - open/close 60000 (time 1760130391.00 total 298.19 last 347.30) - open/close 62820 (time 1760130401.00 total 308.19 last 281.97) total: 64768 open/close in 315.46 seconds: 205.31 ops/second - /unlink 5016 (time 1760130419.88 total 10.00 last 501.56) - /unlink 10000 (time 1760130429.35 total 19.47 last 526.09) - /unlink 15144 (time 1760130439.35 total 29.47 last 514.39) - /unlink 19730 (time 1760130449.35 total 39.48 last 458.45) - /unlink 24055 (time 1760130459.41 total 49.53 last 430.06) - /unlink 28396 (time 1760130469.41 total 59.53 last 434.09) - /unlink 30000 (time 1760130472.65 total 62.78 last 494.72) - /unlink 35606 (time 1760130482.65 total 72.78 last 560.57) - /unlink 40000 (time 1760130490.76 total 80.88 last 542.23) - /unlink 45278 (time 1760130500.76 total 90.88 last 527.71) - /unlink 50000 (time 1760130508.03 total 98.16 last 649.14) - /unlink 56723 (time 1760130518.03 total 108.16 last 672.28) - /unlink 60000 (time 1760130523.12 total 113.24 last 644.37) total: 64768 /unlink in 120.46 seconds: 537.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local oleg432-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg432-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 PASS 106 (4900s) Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 oleg432-server: oleg432-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions oleg432-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg432-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 5040 sec ========== 17:09:52 (1760130592) === conf-sanity: start cleanup 17:09:52 (1760130592) === === conf-sanity: finish cleanup 17:09:53 (1760130593) ===