-----============= acceptance-small: conf-sanity ============----- Sun Dec 7 05:43:05 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg441-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg441-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg441-server oleg441-server: oleg441-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg441-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg441-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg441-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg441-server: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg441-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg441-server: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg441-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:45:39 (1765104339) Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f) oleg441-server: oleg441-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg441-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg441-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg441-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32714 458 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32899 643 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 917 (time 1765104428.93 total 10.02 last 91.55) - open/close 1900 (time 1765104438.94 total 20.03 last 98.21) - open/close 2936 (time 1765104448.94 total 30.03 last 103.56) - open/close 3823 (time 1765104458.95 total 40.04 last 88.65) - open/close 4830 (time 1765104468.95 total 50.04 last 100.68) - open/close 5836 (time 1765104478.95 total 60.04 last 100.55) - open/close 6837 (time 1765104488.95 total 70.04 last 100.09) - open/close 7832 (time 1765104498.96 total 80.05 last 99.48) - open/close 8893 (time 1765104508.96 total 90.05 last 106.07) - open/close 10000 (time 1765104518.68 total 99.77 last 113.92) - open/close 11114 (time 1765104528.68 total 109.77 last 111.38) - open/close 12210 (time 1765104538.68 total 119.77 last 109.57) - open/close 13257 (time 1765104548.69 total 129.78 last 104.62) - open/close 14445 (time 1765104558.70 total 139.79 last 118.66) - open/close 15534 (time 1765104568.70 total 149.79 last 108.87) - open/close 16593 (time 1765104578.71 total 159.80 last 105.87) - open/close 17636 (time 1765104588.71 total 169.80 last 104.26) - open/close 18560 (time 1765104598.72 total 179.81 last 92.34) - open/close 19615 (time 1765104608.73 total 189.82 last 105.38) - open/close 20000 (time 1765104612.75 total 193.84 last 95.68) - open/close 20855 (time 1765104622.76 total 203.85 last 85.46) - open/close 21672 (time 1765104632.76 total 213.85 last 81.66) - open/close 22575 (time 1765104642.77 total 223.86 last 90.20) - open/close 23426 (time 1765104652.78 total 233.87 last 85.02) - open/close 24314 (time 1765104662.79 total 243.88 last 88.76) - open/close 25205 (time 1765104672.80 total 253.88 last 89.03) - open/close 26069 (time 1765104682.80 total 263.89 last 86.38) - open/close 26904 (time 1765104692.81 total 273.89 last 83.43) - open/close 27737 (time 1765104702.81 total 283.90 last 83.27) - open/close 28590 (time 1765104712.82 total 293.91 last 85.20) - open/close 29423 (time 1765104722.82 total 303.91 last 83.29) - open/close 30000 (time 1765104730.12 total 311.21 last 79.08) - open/close 30871 (time 1765104740.12 total 321.21 last 87.07) - open/close 31750 (time 1765104750.13 total 331.22 last 87.86) - open/close 32711 (time 1765104760.13 total 341.22 last 96.06) - open/close 33627 (time 1765104770.13 total 351.22 last 91.57) - open/close 34612 (time 1765104780.13 total 361.22 last 98.48) - open/close 35613 (time 1765104790.14 total 371.22 last 100.08) - open/close 36568 (time 1765104800.14 total 381.23 last 95.48) - open/close 37484 (time 1765104810.14 total 391.23 last 91.55) - open/close 38354 (time 1765104820.15 total 401.24 last 86.96) - open/close 39159 (time 1765104830.15 total 411.24 last 80.48) - open/close 40000 (time 1765104839.62 total 420.71 last 88.81) - open/close 40863 (time 1765104849.62 total 430.71 last 86.28) - open/close 41679 (time 1765104859.63 total 440.72 last 81.54) - open/close 42496 (time 1765104869.64 total 450.73 last 81.58) - open/close 43391 (time 1765104879.65 total 460.74 last 89.45) - open/close 44190 (time 1765104889.66 total 470.75 last 79.82) - open/close 45051 (time 1765104899.67 total 480.76 last 86.02) - open/close 45945 (time 1765104909.67 total 490.76 last 89.37) - open/close 46796 (time 1765104919.68 total 500.77 last 85.04) - open/close 47579 (time 1765104929.68 total 510.77 last 78.27) - open/close 48324 (time 1765104939.69 total 520.78 last 74.47) - open/close 49204 (time 1765104949.69 total 530.78 last 87.98) - open/close 50000 (time 1765104957.91 total 539.00 last 96.88) - open/close 51027 (time 1765104967.91 total 549.00 last 102.62) - open/close 51841 (time 1765104977.92 total 559.01 last 81.32) - open/close 52672 (time 1765104987.93 total 569.01 last 83.09) - open/close 53553 (time 1765104997.94 total 579.03 last 87.98) - open/close 54375 (time 1765105007.94 total 589.03 last 82.17) - open/close 55234 (time 1765105017.96 total 599.05 last 85.76) - open/close 56130 (time 1765105027.97 total 609.06 last 89.52) - open/close 57103 (time 1765105037.98 total 619.07 last 97.22) - open/close 58018 (time 1765105047.98 total 629.07 last 91.43) - open/close 58945 (time 1765105057.98 total 639.07 last 92.69) - open/close 59969 (time 1765105067.98 total 649.07 last 102.40) - open/close 61081 (time 1765105077.99 total 659.08 last 111.10) - open/close 62069 (time 1765105088.00 total 669.09 last 98.77) - open/close 63035 (time 1765105098.00 total 679.09 last 96.55) - open/close 64080 (time 1765105108.01 total 689.10 last 104.39) total: 64768 open/close in 695.60 seconds: 93.11 ops/second - /unlink 1101 (time 1765105126.81 total 10.00 last 110.10) - /unlink 2193 (time 1765105136.81 total 20.00 last 109.16) - /unlink 3196 (time 1765105146.81 total 30.00 last 100.30) - /unlink 4457 (time 1765105156.82 total 40.01 last 126.03) - /unlink 5630 (time 1765105166.82 total 50.02 last 117.24) - /unlink 6536 (time 1765105176.82 total 60.02 last 90.59) - /unlink 7588 (time 1765105186.83 total 70.02 last 105.16) - /unlink 8741 (time 1765105196.83 total 80.03 last 115.23) - /unlink 9740 (time 1765105206.84 total 90.03 last 99.86) - /unlink 10000 (time 1765105209.91 total 93.10 last 84.57) - /unlink 10851 (time 1765105219.92 total 103.11 last 85.05) - /unlink 11825 (time 1765105229.92 total 113.12 last 97.34) - /unlink 12940 (time 1765105239.93 total 123.12 last 111.48) - /unlink 14130 (time 1765105249.94 total 133.13 last 118.87) - /unlink 15204 (time 1765105259.94 total 143.13 last 107.40) - /unlink 16260 (time 1765105269.94 total 153.14 last 105.53) - /unlink 17232 (time 1765105279.95 total 163.14 last 97.13) - /unlink 18406 (time 1765105289.95 total 173.15 last 117.35) - /unlink 19477 (time 1765105299.96 total 183.15 last 107.02) - /unlink 20000 (time 1765105305.24 total 188.43 last 99.04) - /unlink 20911 (time 1765105315.24 total 198.44 last 91.09) - /unlink 21981 (time 1765105325.25 total 208.44 last 106.93) - /unlink 22983 (time 1765105335.25 total 218.45 last 100.16) - /unlink 24057 (time 1765105345.27 total 228.46 last 107.22) - /unlink 25160 (time 1765105355.27 total 238.46 last 110.28) - /unlink 26355 (time 1765105365.27 total 248.47 last 119.47) - /unlink 27463 (time 1765105375.28 total 258.47 last 110.73) - /unlink 28483 (time 1765105385.28 total 268.47 last 102.00) - /unlink 29342 (time 1765105395.28 total 278.47 last 85.89) - /unlink 30000 (time 1765105402.16 total 285.35 last 95.64) - /unlink 31250 (time 1765105412.17 total 295.36 last 124.94) - /unlink 32359 (time 1765105422.17 total 305.36 last 110.87) - /unlink 33327 (time 1765105432.17 total 315.37 last 96.74) - /unlink 34384 (time 1765105442.18 total 325.37 last 105.66) - /unlink 35370 (time 1765105452.18 total 335.38 last 98.53) - /unlink 36367 (time 1765105462.19 total 345.38 last 99.65) - /unlink 37392 (time 1765105472.20 total 355.39 last 102.45) - /unlink 38368 (time 1765105482.21 total 365.40 last 97.48) - /unlink 39389 (time 1765105492.22 total 375.41 last 102.02) - /unlink 40000 (time 1765105498.78 total 381.98 last 93.04) - /unlink 40920 (time 1765105508.79 total 391.98 last 91.98) - /unlink 41901 (time 1765105518.80 total 401.99 last 98.00) - /unlink 42887 (time 1765105528.80 total 411.99 last 98.57) - /unlink 43894 (time 1765105538.81 total 422.00 last 100.60) - /unlink 44827 (time 1765105548.82 total 432.01 last 93.22) - /unlink 45696 (time 1765105558.83 total 442.02 last 86.81) - /unlink 46612 (time 1765105568.83 total 452.02 last 91.59) - /unlink 47741 (time 1765105578.83 total 462.03 last 112.84) - /unlink 48874 (time 1765105588.84 total 472.04 last 113.19) - /unlink 49756 (time 1765105598.85 total 482.04 last 88.17) - /unlink 50000 (time 1765105601.49 total 484.69 last 92.21) - /unlink 51009 (time 1765105611.53 total 494.73 last 100.50) - /unlink 52005 (time 1765105621.53 total 504.73 last 99.60) - /unlink 53234 (time 1765105631.53 total 514.73 last 122.87) - /unlink 54468 (time 1765105641.54 total 524.73 last 123.33) - /unlink 56005 (time 1765105651.55 total 534.74 last 153.61) - /unlink 57147 (time 1765105661.56 total 544.75 last 114.08) - /unlink 58113 (time 1765105671.56 total 554.75 last 96.58) - /unlink 59205 (time 1765105681.57 total 564.76 last 109.11) - /unlink 60000 (time 1765105690.26 total 573.45 last 91.48) - /unlink 61048 (time 1765105700.26 total 583.46 last 104.74) - /unlink 62116 (time 1765105710.27 total 593.46 last 106.74) - /unlink 63203 (time 1765105720.28 total 603.47 last 108.61) - /unlink 64412 (time 1765105730.28 total 613.48 last 120.82) total: 64768 /unlink in 615.86 seconds: 105.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1211 (time 1765105767.58 total 10.00 last 121.04) - open/close 2518 (time 1765105777.58 total 20.01 last 130.66) - open/close 3752 (time 1765105787.58 total 30.01 last 123.39) - open/close 5048 (time 1765105797.59 total 40.01 last 129.55) - open/close 6596 (time 1765105807.59 total 50.02 last 154.74) - open/close 8160 (time 1765105817.59 total 60.02 last 156.39) - open/close 9823 (time 1765105827.59 total 70.02 last 166.29) - open/close 11187 (time 1765105837.59 total 80.02 last 136.40) - open/close 12553 (time 1765105847.60 total 90.02 last 136.50) - open/close 14356 (time 1765105857.60 total 100.03 last 180.20) - open/close 15801 (time 1765105867.61 total 110.03 last 144.47) - open/close 17241 (time 1765105877.61 total 120.03 last 143.98) - open/close 18607 (time 1765105887.61 total 130.04 last 136.58) - open/close 20000 (time 1765105896.23 total 138.65 last 161.66) - open/close 21596 (time 1765105906.23 total 148.65 last 159.58) - open/close 23017 (time 1765105916.23 total 158.66 last 142.06) - open/close 24353 (time 1765105926.23 total 168.66 last 133.57) - open/close 25550 (time 1765105936.23 total 178.66 last 119.69) - open/close 26812 (time 1765105946.24 total 188.66 last 126.15) - open/close 28317 (time 1765105956.24 total 198.67 last 150.43) - open/close 30000 (time 1765105964.50 total 206.93 last 203.73) - open/close 32152 (time 1765105974.50 total 216.93 last 215.18) - open/close 33911 (time 1765105984.50 total 226.93 last 175.87) - open/close 35165 (time 1765105994.51 total 236.94 last 125.31) - open/close 36610 (time 1765106004.51 total 246.94 last 144.47) - open/close 37964 (time 1765106014.51 total 256.94 last 135.40) - open/close 39367 (time 1765106024.52 total 266.95 last 140.21) - open/close 40000 (time 1765106028.57 total 271.00 last 156.34) - open/close 41794 (time 1765106038.57 total 281.00 last 179.38) - open/close 43547 (time 1765106048.57 total 291.00 last 175.25) - open/close 44969 (time 1765106058.58 total 301.01 last 142.12) - open/close 46434 (time 1765106068.58 total 311.01 last 146.48) - open/close 48014 (time 1765106078.59 total 321.01 last 157.91) - open/close 49738 (time 1765106088.59 total 331.02 last 172.35) - open/close 51074 (time 1765106098.60 total 341.02 last 133.52) - open/close 52373 (time 1765106108.60 total 351.03 last 129.85) - open/close 53690 (time 1765106118.61 total 361.03 last 131.59) - open/close 55501 (time 1765106128.62 total 371.04 last 180.91) - open/close 57299 (time 1765106138.62 total 381.05 last 179.73) - open/close 59406 (time 1765106148.63 total 391.05 last 210.63) - open/close 60000 (time 1765106152.04 total 394.46 last 174.13) - open/close 61921 (time 1765106162.04 total 404.47 last 192.03) - open/close 63484 (time 1765106172.04 total 414.47 last 156.28) total: 64768 open/close in 422.53 seconds: 153.28 ops/second - /unlink 2720 (time 1765106192.01 total 10.00 last 271.96) - /unlink 5574 (time 1765106202.01 total 20.00 last 285.35) - /unlink 9035 (time 1765106212.01 total 30.01 last 346.03) - /unlink 10000 (time 1765106215.04 total 33.03 last 318.67) - /unlink 13021 (time 1765106225.04 total 43.03 last 302.08) - /unlink 16452 (time 1765106235.04 total 53.04 last 343.03) - /unlink 19828 (time 1765106245.04 total 63.04 last 337.52) - /unlink 23083 (time 1765106255.05 total 73.04 last 325.41) - /unlink 26622 (time 1765106265.05 total 83.04 last 353.84) - /unlink 30000 (time 1765106274.93 total 92.93 last 341.68) - /unlink 34183 (time 1765106284.93 total 102.93 last 418.27) - /unlink 38112 (time 1765106294.94 total 112.93 last 392.85) - /unlink 40000 (time 1765106300.25 total 118.25 last 355.16) - /unlink 43494 (time 1765106310.25 total 128.25 last 349.37) - /unlink 47723 (time 1765106320.25 total 138.25 last 422.88) - /unlink 50000 (time 1765106326.63 total 144.62 last 357.23) - /unlink 52552 (time 1765106336.63 total 154.62 last 255.16) - /unlink 55665 (time 1765106346.63 total 164.63 last 311.22) - /unlink 59531 (time 1765106356.63 total 174.63 last 386.54) - /unlink 63724 (time 1765106366.64 total 184.63 last 419.20) total: 64768 /unlink in 186.82 seconds: 346.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2440 (time 1765106400.33 total 10.00 last 243.93) - open/close 4524 (time 1765106410.33 total 20.01 last 208.34) - open/close 6801 (time 1765106420.34 total 30.01 last 227.60) - open/close 9240 (time 1765106430.34 total 40.01 last 243.86) - open/close 10000 (time 1765106432.97 total 42.64 last 289.25) - open/close 12603 (time 1765106442.97 total 52.64 last 260.18) - open/close 14895 (time 1765106452.97 total 62.65 last 229.17) - open/close 17677 (time 1765106462.97 total 72.65 last 278.17) - open/close 20000 (time 1765106468.55 total 78.22 last 416.79) - open/close 23821 (time 1765106478.55 total 88.22 last 382.07) - open/close 28284 (time 1765106488.55 total 98.22 last 446.21) - open/close 30000 (time 1765106493.08 total 102.75 last 379.16) - open/close 33901 (time 1765106503.08 total 112.75 last 389.98) - open/close 36977 (time 1765106513.08 total 122.75 last 307.57) - open/close 40000 (time 1765106521.02 total 130.69 last 380.87) - open/close 43498 (time 1765106531.02 total 140.69 last 349.79) - open/close 47003 (time 1765106541.02 total 150.69 last 350.49) - open/close 50000 (time 1765106550.12 total 159.80 last 329.14) - open/close 52782 (time 1765106560.13 total 169.80 last 278.16) - open/close 56799 (time 1765106570.13 total 179.80 last 401.63) - open/close 60000 (time 1765106578.05 total 187.72 last 404.12) - open/close 64214 (time 1765106588.05 total 197.72 last 421.34) total: 64768 open/close in 199.10 seconds: 325.31 ops/second - /unlink 6606 (time 1765106600.81 total 10.00 last 660.47) - /unlink 10000 (time 1765106606.46 total 15.65 last 600.54) - /unlink 16424 (time 1765106616.46 total 25.65 last 642.34) - /unlink 20000 (time 1765106620.53 total 29.73 last 878.10) - /unlink 26076 (time 1765106630.53 total 39.73 last 607.52) - /unlink 30000 (time 1765106637.18 total 46.37 last 590.39) - /unlink 35514 (time 1765106647.18 total 56.38 last 551.32) - /unlink 40000 (time 1765106653.88 total 63.08 last 669.65) - /unlink 45655 (time 1765106663.88 total 73.08 last 565.47) - /unlink 50000 (time 1765106670.79 total 79.98 last 629.18) - /unlink 54355 (time 1765106680.79 total 89.98 last 435.44) - /unlink 58048 (time 1765106690.79 total 99.99 last 369.21) - /unlink 60000 (time 1765106696.29 total 105.48 last 355.01) - /unlink 63585 (time 1765106706.29 total 115.49 last 358.44) total: 64768 /unlink in 118.68 seconds: 545.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2508 (time 1765106735.71 total 10.00 last 250.76) - open/close 5133 (time 1765106745.72 total 20.00 last 262.46) - open/close 8001 (time 1765106755.72 total 30.00 last 286.77) - open/close 10000 (time 1765106761.68 total 35.96 last 335.36) - open/close 12720 (time 1765106771.68 total 45.97 last 271.96) - open/close 15317 (time 1765106781.68 total 55.97 last 259.62) - open/close 17819 (time 1765106791.68 total 65.97 last 250.17) - open/close 20000 (time 1765106797.84 total 72.13 last 354.11) - open/close 23116 (time 1765106807.84 total 82.13 last 311.59) - open/close 25503 (time 1765106817.85 total 92.13 last 238.65) - open/close 28030 (time 1765106827.85 total 102.13 last 252.66) - open/close 30000 (time 1765106834.41 total 108.69 last 300.34) - open/close 33302 (time 1765106844.41 total 118.70 last 330.12) - open/close 36527 (time 1765106854.41 total 128.70 last 322.47) - open/close 40000 (time 1765106864.02 total 138.31 last 361.40) - open/close 44401 (time 1765106874.02 total 148.31 last 440.06) - open/close 48130 (time 1765106884.02 total 158.31 last 372.83) - open/close 50000 (time 1765106888.77 total 163.06 last 393.48) - open/close 54299 (time 1765106898.78 total 173.06 last 429.87) - open/close 58433 (time 1765106908.79 total 183.08 last 412.83) - open/close 60000 (time 1765106912.24 total 186.53 last 453.67) - open/close 64161 (time 1765106922.24 total 196.53 last 416.09) total: 64768 open/close in 198.12 seconds: 326.91 ops/second - /unlink 5167 (time 1765106935.31 total 10.00 last 516.62) - /unlink 10000 (time 1765106942.46 total 17.15 last 675.89) - /unlink 16169 (time 1765106952.46 total 27.15 last 616.83) - /unlink 20000 (time 1765106957.97 total 32.66 last 695.18) - /unlink 26427 (time 1765106967.97 total 42.66 last 642.66) - /unlink 30000 (time 1765106973.20 total 47.90 last 683.14) - /unlink 36237 (time 1765106983.20 total 57.90 last 623.67) - /unlink 40000 (time 1765106989.40 total 64.09 last 607.78) - /unlink 45937 (time 1765106999.40 total 74.09 last 593.66) - /unlink 50000 (time 1765107005.88 total 80.57 last 626.98) - /unlink 55979 (time 1765107015.88 total 90.57 last 597.78) - /unlink 60000 (time 1765107022.97 total 97.66 last 567.28) total: 64768 /unlink in 104.02 seconds: 622.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3498 (time 1765107060.78 total 10.00 last 349.75) - open/close 7385 (time 1765107070.78 total 20.00 last 388.68) - open/close 10000 (time 1765107077.96 total 27.18 last 364.21) - open/close 12555 (time 1765107087.96 total 37.18 last 255.47) - open/close 17207 (time 1765107097.96 total 47.18 last 465.20) - open/close 20000 (time 1765107105.01 total 54.24 last 396.01) - open/close 23830 (time 1765107115.01 total 64.24 last 382.95) - open/close 27687 (time 1765107125.01 total 74.24 last 385.67) - open/close 30000 (time 1765107130.89 total 80.12 last 393.41) - open/close 34305 (time 1765107140.89 total 90.12 last 430.48) - open/close 39042 (time 1765107150.89 total 100.12 last 473.66) - open/close 40000 (time 1765107153.15 total 102.37 last 425.06) - open/close 44235 (time 1765107163.15 total 112.37 last 423.46) - open/close 48561 (time 1765107173.15 total 122.38 last 432.54) - open/close 50000 (time 1765107176.74 total 125.96 last 400.91) - open/close 54246 (time 1765107186.74 total 135.97 last 424.57) - open/close 58818 (time 1765107196.74 total 145.97 last 457.17) - open/close 60000 (time 1765107199.45 total 148.67 last 436.92) - open/close 64541 (time 1765107209.45 total 158.67 last 454.07) total: 64768 open/close in 159.23 seconds: 406.75 ops/second - /unlink 7135 (time 1765107221.33 total 10.00 last 713.47) - /unlink 10000 (time 1765107225.06 total 13.73 last 767.69) - /unlink 17223 (time 1765107235.06 total 23.73 last 722.25) - /unlink 20000 (time 1765107238.82 total 27.49 last 739.07) - /unlink 27236 (time 1765107248.82 total 37.49 last 723.55) - /unlink 30000 (time 1765107253.24 total 41.91 last 625.57) - /unlink 37156 (time 1765107263.24 total 51.91 last 715.58) - /unlink 40000 (time 1765107267.38 total 56.06 last 685.83) - /unlink 47031 (time 1765107277.38 total 66.06 last 703.09) - /unlink 50000 (time 1765107281.22 total 69.90 last 773.14) - /unlink 56484 (time 1765107291.22 total 79.90 last 648.39) - /unlink 60000 (time 1765107296.61 total 85.28 last 652.76) total: 64768 /unlink in 92.25 seconds: 702.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg441-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing unload_modules_local oleg441-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 PASS 106 (3000s) Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 oleg441-server: oleg441-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg441-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions oleg441-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg441-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3183 sec ========== 06:36:09 (1765107369) === conf-sanity: start cleanup 06:36:09 (1765107369) === === conf-sanity: finish cleanup 06:36:10 (1765107370) ===