-----============= acceptance-small: conf-sanity ============----- Sun Dec 7 05:24:25 EST 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg103-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg103-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg103-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg103-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg103-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg103-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg103-server oleg103-server: oleg103-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing load_modules_local oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg103-server: Loading modules from /home/green/git/lustre-release/lustre oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg103-server: detected 4 online CPUs by sysfs oleg103-server: MODOPTS_LIBCFS= oleg103-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg103-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg103-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg103-server: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg103-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg103-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg103-server stop mds service on oleg103-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg103-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:27:00 (1765103220) Stopping clients: oleg103-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg103-client.virtnet /mnt/lustre2 (opts:-f) oleg103-server: oleg103-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing load_modules_local oleg103-server: Loading modules from /home/green/git/lustre-release/lustre oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg103-server: detected 4 online CPUs by sysfs oleg103-server: MODOPTS_LIBCFS= oleg103-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg103-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg103-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg103-server: oleg103-server.virtnet: executing set_default_debug -1 all pdsh@oleg103-client: oleg103-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg103-client.virtnet: -o user_xattr,flock 192.168.201.103@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 655 (time 1765103331.07 total 10.02 last 65.40) - open/close 1489 (time 1765103341.08 total 20.02 last 83.36) - open/close 2216 (time 1765103351.08 total 30.02 last 72.70) - open/close 2998 (time 1765103361.08 total 40.03 last 78.16) - open/close 3784 (time 1765103371.09 total 50.03 last 78.56) - open/close 4569 (time 1765103381.09 total 60.04 last 78.45) - open/close 5329 (time 1765103391.10 total 70.04 last 75.97) - open/close 6049 (time 1765103401.10 total 80.04 last 71.97) - open/close 6813 (time 1765103411.11 total 90.05 last 76.37) - open/close 7575 (time 1765103421.11 total 100.05 last 76.16) - open/close 8304 (time 1765103431.12 total 110.06 last 72.86) - open/close 9029 (time 1765103441.14 total 120.08 last 72.36) - open/close 9742 (time 1765103451.14 total 130.08 last 71.30) - open/close 10000 (time 1765103454.76 total 133.70 last 71.24) - open/close 10698 (time 1765103464.78 total 143.73 last 69.62) - open/close 11434 (time 1765103474.79 total 153.73 last 73.53) - open/close 12170 (time 1765103484.79 total 163.74 last 73.59) - open/close 12955 (time 1765103494.81 total 173.75 last 78.39) - open/close 13743 (time 1765103504.81 total 183.75 last 78.79) - open/close 14485 (time 1765103514.82 total 193.76 last 74.15) - open/close 15270 (time 1765103524.82 total 203.76 last 78.46) - open/close 15923 (time 1765103534.83 total 213.77 last 65.28) - open/close 16606 (time 1765103544.83 total 223.77 last 68.25) - open/close 17397 (time 1765103554.84 total 233.78 last 79.03) - open/close 18160 (time 1765103564.86 total 243.80 last 76.17) - open/close 18906 (time 1765103574.86 total 253.80 last 74.59) - open/close 19620 (time 1765103584.86 total 263.81 last 71.37) - open/close 20000 (time 1765103590.55 total 269.49 last 66.82) - open/close 20546 (time 1765103600.56 total 279.50 last 54.54) - open/close 21236 (time 1765103610.59 total 289.53 last 68.82) - open/close 22007 (time 1765103620.59 total 299.54 last 77.04) - open/close 22756 (time 1765103630.60 total 309.55 last 74.83) - open/close 23481 (time 1765103640.61 total 319.55 last 72.45) - open/close 24210 (time 1765103650.62 total 329.56 last 72.87) - open/close 24939 (time 1765103660.63 total 339.57 last 72.83) - open/close 25718 (time 1765103670.63 total 349.57 last 77.89) - open/close 26335 (time 1765103680.63 total 359.58 last 61.66) - open/close 27080 (time 1765103690.64 total 369.59 last 74.42) - open/close 27807 (time 1765103700.66 total 379.61 last 72.56) - open/close 28585 (time 1765103710.66 total 389.61 last 77.79) - open/close 29339 (time 1765103720.67 total 399.62 last 75.33) - open/close 30000 (time 1765103729.76 total 408.70 last 72.76) - open/close 30807 (time 1765103739.77 total 418.71 last 80.64) - open/close 31553 (time 1765103749.77 total 428.72 last 74.55) - open/close 32318 (time 1765103759.77 total 438.72 last 76.50) - open/close 33067 (time 1765103769.78 total 448.72 last 74.84) - open/close 33855 (time 1765103779.79 total 458.74 last 78.70) - open/close 34628 (time 1765103789.80 total 468.74 last 77.24) - open/close 35431 (time 1765103799.81 total 478.76 last 80.19) - open/close 36213 (time 1765103809.83 total 488.77 last 78.10) - open/close 36985 (time 1765103819.83 total 498.78 last 77.15) - open/close 37725 (time 1765103829.84 total 508.78 last 73.95) - open/close 38511 (time 1765103839.85 total 518.79 last 78.54) - open/close 39248 (time 1765103849.85 total 528.79 last 73.70) - open/close 40000 (time 1765103859.47 total 538.41 last 78.21) - open/close 40788 (time 1765103869.47 total 548.41 last 78.78) - open/close 41605 (time 1765103879.48 total 558.43 last 81.56) - open/close 42416 (time 1765103889.49 total 568.43 last 81.06) - open/close 43214 (time 1765103899.49 total 578.44 last 79.77) - open/close 44033 (time 1765103909.50 total 588.44 last 81.86) - open/close 44930 (time 1765103919.50 total 598.44 last 89.69) - open/close 45739 (time 1765103929.50 total 608.44 last 80.90) - open/close 46516 (time 1765103939.50 total 618.45 last 77.67) - open/close 47265 (time 1765103949.51 total 628.45 last 74.88) - open/close 48076 (time 1765103959.51 total 638.45 last 81.08) - open/close 48841 (time 1765103969.52 total 648.46 last 76.44) - open/close 49596 (time 1765103979.52 total 658.46 last 75.48) - open/close 50000 (time 1765103983.93 total 662.87 last 91.65) - open/close 50923 (time 1765103993.93 total 672.88 last 92.26) - open/close 51791 (time 1765104003.94 total 682.88 last 86.78) - open/close 52601 (time 1765104013.94 total 692.88 last 80.99) - open/close 53388 (time 1765104023.95 total 702.89 last 78.59) - open/close 54207 (time 1765104033.95 total 712.90 last 81.89) - open/close 55044 (time 1765104043.96 total 722.90 last 83.64) - open/close 55865 (time 1765104053.96 total 732.91 last 82.07) - open/close 56753 (time 1765104063.97 total 742.92 last 88.71) - open/close 57604 (time 1765104073.98 total 752.93 last 85.01) - open/close 58478 (time 1765104084.00 total 762.94 last 87.30) - open/close 59353 (time 1765104094.00 total 772.94 last 87.45) - open/close 60000 (time 1765104101.09 total 780.03 last 91.32) - open/close 60942 (time 1765104111.10 total 790.04 last 94.10) - open/close 61788 (time 1765104121.10 total 800.04 last 84.58) - open/close 62658 (time 1765104131.10 total 810.04 last 86.98) - open/close 63479 (time 1765104141.11 total 820.05 last 82.05) - open/close 64382 (time 1765104151.11 total 830.05 last 90.30) total: 64768 open/close in 833.76 seconds: 77.68 ops/second - /unlink 956 (time 1765104167.19 total 10.01 last 95.50) - /unlink 1899 (time 1765104177.20 total 20.01 last 94.26) - /unlink 2869 (time 1765104187.20 total 30.02 last 96.96) - /unlink 3842 (time 1765104197.21 total 40.03 last 97.19) - /unlink 4768 (time 1765104207.22 total 50.04 last 92.54) - /unlink 5655 (time 1765104217.23 total 60.04 last 88.63) - /unlink 6546 (time 1765104227.23 total 70.05 last 89.08) - /unlink 7390 (time 1765104237.23 total 80.05 last 84.38) - /unlink 8392 (time 1765104247.23 total 90.05 last 100.19) - /unlink 9489 (time 1765104257.23 total 100.05 last 109.69) - /unlink 10000 (time 1765104261.39 total 104.20 last 123.05) - /unlink 10984 (time 1765104271.39 total 114.21 last 98.33) - /unlink 11978 (time 1765104281.40 total 124.22 last 99.31) - /unlink 13058 (time 1765104291.41 total 134.23 last 107.95) - /unlink 14111 (time 1765104301.42 total 144.24 last 105.17) - /unlink 15073 (time 1765104311.43 total 154.25 last 96.08) - /unlink 15986 (time 1765104321.44 total 164.25 last 91.26) - /unlink 16952 (time 1765104331.45 total 174.26 last 96.50) - /unlink 18058 (time 1765104341.46 total 184.27 last 110.49) - /unlink 19257 (time 1765104351.46 total 194.28 last 119.86) - /unlink 20000 (time 1765104357.64 total 200.46 last 120.14) - /unlink 21210 (time 1765104367.65 total 210.47 last 120.97) - /unlink 22504 (time 1765104377.65 total 220.47 last 129.39) - /unlink 23795 (time 1765104387.66 total 230.47 last 128.99) - /unlink 24957 (time 1765104397.66 total 240.48 last 116.17) - /unlink 26246 (time 1765104407.66 total 250.48 last 128.87) - /unlink 27445 (time 1765104417.67 total 260.49 last 119.83) - /unlink 28441 (time 1765104427.68 total 270.50 last 99.51) - /unlink 29552 (time 1765104437.68 total 280.50 last 111.09) - /unlink 30000 (time 1765104441.36 total 284.18 last 121.63) - /unlink 31180 (time 1765104451.36 total 294.18 last 117.97) - /unlink 32125 (time 1765104461.36 total 304.18 last 94.50) - /unlink 33244 (time 1765104471.37 total 314.19 last 111.84) - /unlink 34304 (time 1765104481.37 total 324.19 last 105.97) - /unlink 35270 (time 1765104491.37 total 334.19 last 96.58) - /unlink 36303 (time 1765104501.38 total 344.20 last 103.26) - /unlink 37393 (time 1765104511.38 total 354.20 last 108.96) - /unlink 38418 (time 1765104521.39 total 364.21 last 102.40) - /unlink 39395 (time 1765104531.40 total 374.21 last 97.66) - /unlink 40000 (time 1765104537.18 total 379.99 last 104.67) - /unlink 40988 (time 1765104547.19 total 390.01 last 98.67) - /unlink 41854 (time 1765104557.21 total 400.03 last 86.41) - /unlink 42725 (time 1765104567.22 total 410.04 last 87.00) - /unlink 43743 (time 1765104577.23 total 420.04 last 101.76) - /unlink 44836 (time 1765104587.24 total 430.06 last 109.15) - /unlink 45866 (time 1765104597.24 total 440.06 last 102.98) - /unlink 46830 (time 1765104607.24 total 450.06 last 96.38) - /unlink 47808 (time 1765104617.25 total 460.06 last 97.78) - /unlink 48879 (time 1765104627.25 total 470.07 last 107.08) - /unlink 50000 (time 1765104636.36 total 479.18 last 122.99) - /unlink 51024 (time 1765104646.37 total 489.19 last 102.32) - /unlink 52223 (time 1765104656.38 total 499.20 last 119.79) - /unlink 53298 (time 1765104666.38 total 509.20 last 107.46) - /unlink 54354 (time 1765104676.39 total 519.21 last 105.52) - /unlink 55626 (time 1765104686.40 total 529.21 last 127.14) - /unlink 56823 (time 1765104696.40 total 539.22 last 119.65) - /unlink 57939 (time 1765104706.41 total 549.23 last 111.49) - /unlink 58917 (time 1765104716.41 total 559.23 last 97.79) - /unlink 59808 (time 1765104726.41 total 569.23 last 89.09) - /unlink 61004 (time 1765104736.41 total 579.23 last 119.58) - /unlink 62116 (time 1765104746.42 total 589.24 last 111.10) - /unlink 63457 (time 1765104756.42 total 599.24 last 134.09) - /unlink 64614 (time 1765104766.42 total 609.24 last 115.70) total: 64768 /unlink in 610.78 seconds: 106.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1132 (time 1765104800.33 total 10.00 last 113.19) - open/close 2318 (time 1765104810.34 total 20.01 last 118.52) - open/close 3478 (time 1765104820.35 total 30.02 last 115.91) - open/close 4515 (time 1765104830.35 total 40.02 last 103.67) - open/close 5498 (time 1765104840.36 total 50.02 last 98.25) - open/close 6587 (time 1765104850.36 total 60.02 last 108.90) - open/close 7735 (time 1765104860.36 total 70.03 last 114.73) - open/close 8792 (time 1765104870.36 total 80.03 last 105.69) - open/close 9893 (time 1765104880.37 total 90.03 last 110.08) - open/close 10925 (time 1765104890.37 total 100.04 last 103.17) - open/close 11989 (time 1765104900.37 total 110.04 last 106.38) - open/close 13046 (time 1765104910.38 total 120.05 last 105.59) - open/close 14167 (time 1765104920.39 total 130.06 last 111.97) - open/close 15258 (time 1765104930.39 total 140.06 last 109.09) - open/close 16342 (time 1765104940.39 total 150.06 last 108.38) - open/close 17643 (time 1765104950.40 total 160.06 last 130.08) - open/close 18661 (time 1765104960.40 total 170.07 last 101.72) - open/close 19744 (time 1765104970.42 total 180.09 last 108.13) - open/close 20000 (time 1765104972.54 total 182.20 last 120.95) - open/close 21087 (time 1765104982.54 total 192.21 last 108.64) - open/close 22264 (time 1765104992.55 total 202.22 last 117.63) - open/close 23442 (time 1765105002.55 total 212.22 last 117.76) - open/close 24454 (time 1765105012.55 total 222.22 last 101.20) - open/close 25463 (time 1765105022.55 total 232.22 last 100.88) - open/close 26660 (time 1765105032.56 total 242.23 last 119.64) - open/close 28014 (time 1765105042.56 total 252.23 last 135.35) - open/close 29334 (time 1765105052.56 total 262.23 last 131.98) - open/close 30000 (time 1765105057.21 total 266.88 last 143.36) - open/close 31868 (time 1765105067.21 total 276.88 last 186.72) - open/close 33443 (time 1765105077.22 total 286.88 last 157.47) - open/close 34962 (time 1765105087.22 total 296.89 last 151.84) - open/close 36169 (time 1765105097.22 total 306.89 last 120.65) - open/close 37496 (time 1765105107.22 total 316.89 last 132.68) - open/close 38677 (time 1765105117.24 total 326.90 last 117.97) - open/close 39825 (time 1765105127.24 total 336.91 last 114.73) - open/close 41060 (time 1765105137.25 total 346.91 last 123.44) - open/close 42166 (time 1765105147.25 total 356.92 last 110.59) - open/close 43374 (time 1765105157.25 total 366.92 last 120.76) - open/close 44544 (time 1765105167.26 total 376.92 last 116.95) - open/close 45820 (time 1765105177.26 total 386.93 last 127.55) - open/close 46899 (time 1765105187.26 total 396.93 last 107.85) - open/close 48125 (time 1765105197.26 total 406.93 last 122.59) - open/close 49364 (time 1765105207.28 total 416.94 last 123.75) - open/close 50000 (time 1765105213.07 total 422.73 last 109.83) - open/close 51216 (time 1765105223.08 total 432.74 last 121.49) - open/close 52414 (time 1765105233.09 total 442.75 last 119.67) - open/close 53675 (time 1765105243.09 total 452.76 last 126.07) - open/close 54999 (time 1765105253.09 total 462.76 last 132.40) - open/close 56230 (time 1765105263.09 total 472.76 last 123.10) - open/close 57524 (time 1765105273.09 total 482.76 last 129.37) - open/close 58953 (time 1765105283.09 total 492.76 last 142.87) - open/close 60000 (time 1765105290.08 total 499.75 last 149.81) - open/close 61443 (time 1765105300.09 total 509.75 last 144.24) - open/close 62877 (time 1765105310.09 total 519.76 last 143.35) - open/close 64466 (time 1765105320.09 total 529.76 last 158.90) total: 64768 open/close in 531.94 seconds: 121.76 ops/second - /unlink 2387 (time 1765105334.47 total 10.00 last 238.67) - /unlink 4756 (time 1765105344.48 total 20.00 last 236.82) - /unlink 7323 (time 1765105354.48 total 30.01 last 256.61) - /unlink 9606 (time 1765105364.48 total 40.01 last 228.27) - /unlink 11923 (time 1765105374.48 total 50.01 last 231.64) - /unlink 14476 (time 1765105384.49 total 60.02 last 255.20) - /unlink 17202 (time 1765105394.49 total 70.02 last 272.53) - /unlink 20000 (time 1765105403.01 total 78.54 last 328.21) - /unlink 22753 (time 1765105413.02 total 88.55 last 275.24) - /unlink 25800 (time 1765105423.02 total 98.55 last 304.69) - /unlink 29011 (time 1765105433.02 total 108.55 last 320.99) - /unlink 30000 (time 1765105435.82 total 111.35 last 353.60) - /unlink 33946 (time 1765105445.82 total 121.35 last 394.57) - /unlink 37093 (time 1765105455.82 total 131.35 last 314.63) - /unlink 39665 (time 1765105465.82 total 141.35 last 257.15) - /unlink 42273 (time 1765105475.82 total 151.35 last 260.78) - /unlink 44489 (time 1765105485.82 total 161.35 last 221.58) - /unlink 47062 (time 1765105495.83 total 171.35 last 257.25) - /unlink 49880 (time 1765105505.83 total 181.36 last 281.69) - /unlink 52756 (time 1765105515.83 total 191.36 last 287.59) - /unlink 55595 (time 1765105525.83 total 201.36 last 283.81) - /unlink 58068 (time 1765105535.84 total 211.36 last 247.23) - /unlink 60000 (time 1765105542.25 total 217.78 last 301.14) - /unlink 63086 (time 1765105552.25 total 227.78 last 308.52) total: 64768 /unlink in 232.77 seconds: 278.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2000 (time 1765105587.11 total 10.00 last 199.97) - open/close 3761 (time 1765105597.12 total 20.00 last 176.08) - open/close 5737 (time 1765105607.12 total 30.01 last 197.54) - open/close 7626 (time 1765105617.12 total 40.01 last 188.86) - open/close 9311 (time 1765105627.12 total 50.01 last 168.47) - open/close 10000 (time 1765105630.93 total 53.82 last 180.84) - open/close 11996 (time 1765105640.93 total 63.82 last 199.60) - open/close 13986 (time 1765105650.94 total 73.82 last 198.91) - open/close 15908 (time 1765105660.94 total 83.83 last 192.15) - open/close 18264 (time 1765105670.94 total 93.83 last 235.58) - open/close 20000 (time 1765105678.69 total 101.58 last 223.96) - open/close 21957 (time 1765105688.70 total 111.58 last 195.63) - open/close 24029 (time 1765105698.70 total 121.59 last 207.13) - open/close 26021 (time 1765105708.72 total 131.60 last 198.87) - open/close 27981 (time 1765105718.72 total 141.61 last 195.95) - open/close 29986 (time 1765105728.72 total 151.61 last 200.48) - open/close 31996 (time 1765105738.72 total 161.61 last 200.97) - open/close 33869 (time 1765105748.73 total 171.61 last 187.21) - open/close 35546 (time 1765105758.73 total 181.61 last 167.66) - open/close 37477 (time 1765105768.74 total 191.62 last 192.93) - open/close 39625 (time 1765105778.74 total 201.63 last 214.76) - open/close 42072 (time 1765105788.74 total 211.63 last 244.63) - open/close 44435 (time 1765105798.74 total 221.63 last 236.24) - open/close 46828 (time 1765105808.75 total 231.64 last 239.18) - open/close 49259 (time 1765105818.75 total 241.64 last 243.04) - open/close 50000 (time 1765105821.34 total 244.23 last 286.20) - open/close 52143 (time 1765105831.35 total 254.23 last 214.18) - open/close 53613 (time 1765105841.35 total 264.24 last 146.93) - open/close 55386 (time 1765105851.35 total 274.24 last 177.30) - open/close 57408 (time 1765105861.37 total 284.25 last 201.87) - open/close 58880 (time 1765105871.37 total 294.26 last 147.13) - open/close 60000 (time 1765105879.29 total 302.17 last 141.51) - open/close 61906 (time 1765105889.29 total 312.18 last 190.53) - open/close 63728 (time 1765105899.30 total 322.19 last 182.06) total: 64768 open/close in 329.34 seconds: 196.66 ops/second - /unlink 2794 (time 1765105918.46 total 10.00 last 279.28) - /unlink 5750 (time 1765105928.46 total 20.01 last 295.57) - /unlink 8883 (time 1765105938.47 total 30.01 last 313.23) - /unlink 10000 (time 1765105941.66 total 33.21 last 349.19) - /unlink 13686 (time 1765105951.66 total 43.21 last 368.57) - /unlink 18091 (time 1765105961.67 total 53.21 last 440.48) - /unlink 20000 (time 1765105966.45 total 57.99 last 398.86) - /unlink 23783 (time 1765105976.45 total 68.00 last 378.20) - /unlink 27088 (time 1765105986.46 total 78.00 last 330.44) - /unlink 30000 (time 1765105995.41 total 86.95 last 325.37) - /unlink 34179 (time 1765106005.41 total 96.95 last 417.89) - /unlink 38996 (time 1765106015.41 total 106.95 last 481.64) - /unlink 40000 (time 1765106017.86 total 109.40 last 409.80) - /unlink 44018 (time 1765106027.86 total 119.40 last 401.75) - /unlink 48751 (time 1765106037.86 total 129.40 last 473.25) - /unlink 50000 (time 1765106040.40 total 131.95 last 490.92) - /unlink 55147 (time 1765106050.40 total 141.95 last 514.64) - /unlink 60000 (time 1765106060.22 total 151.77 last 494.22) total: 64768 /unlink in 159.78 seconds: 405.35 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2630 (time 1765106096.97 total 10.00 last 262.99) - open/close 5026 (time 1765106106.97 total 20.00 last 239.58) - open/close 7107 (time 1765106116.97 total 30.01 last 208.01) - open/close 9867 (time 1765106126.97 total 40.01 last 275.92) - open/close 12801 (time 1765106136.98 total 50.01 last 293.35) - open/close 15656 (time 1765106146.98 total 60.02 last 285.34) - open/close 18353 (time 1765106156.98 total 70.02 last 269.64) - open/close 20000 (time 1765106163.56 total 76.59 last 250.45) - open/close 22308 (time 1765106173.56 total 86.59 last 230.78) - open/close 24555 (time 1765106183.56 total 96.60 last 224.68) - open/close 27647 (time 1765106193.57 total 106.60 last 309.06) - open/close 30000 (time 1765106201.46 total 114.49 last 298.10) - open/close 32455 (time 1765106211.46 total 124.50 last 245.40) - open/close 35605 (time 1765106221.46 total 134.50 last 314.98) - open/close 39368 (time 1765106231.47 total 144.50 last 376.27) - open/close 42739 (time 1765106241.47 total 154.50 last 337.02) - open/close 46375 (time 1765106251.47 total 164.50 last 363.53) - open/close 49935 (time 1765106261.47 total 174.51 last 355.92) - open/close 53121 (time 1765106271.47 total 184.51 last 318.55) - open/close 56689 (time 1765106281.48 total 194.51 last 356.74) - open/close 60000 (time 1765106289.12 total 202.15 last 433.07) - open/close 64084 (time 1765106299.12 total 212.16 last 408.33) total: 64768 open/close in 214.18 seconds: 302.40 ops/second - /unlink 5650 (time 1765106312.56 total 10.00 last 565.00) - /unlink 10000 (time 1765106318.76 total 16.20 last 701.06) - /unlink 15930 (time 1765106328.77 total 26.21 last 592.42) - /unlink 20000 (time 1765106335.68 total 33.12 last 589.11) - /unlink 25442 (time 1765106345.68 total 43.13 last 544.10) - /unlink 30000 (time 1765106352.99 total 50.43 last 623.87) - /unlink 36939 (time 1765106362.99 total 60.43 last 693.80) - /unlink 40000 (time 1765106367.10 total 64.55 last 744.00) - /unlink 46674 (time 1765106377.10 total 74.55 last 667.40) - /unlink 50000 (time 1765106381.83 total 79.28 last 703.13) - /unlink 56821 (time 1765106391.83 total 89.28 last 682.10) - /unlink 60000 (time 1765106396.49 total 93.94 last 682.40) total: 64768 /unlink in 102.00 seconds: 634.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4099 (time 1765106435.71 total 10.00 last 409.83) - open/close 8812 (time 1765106445.71 total 20.00 last 471.25) - open/close 10000 (time 1765106448.09 total 22.38 last 500.01) - open/close 14521 (time 1765106458.09 total 32.38 last 452.00) - open/close 19157 (time 1765106468.09 total 42.38 last 463.55) - open/close 23356 (time 1765106478.09 total 52.38 last 419.86) - open/close 27488 (time 1765106488.09 total 62.38 last 413.14) - open/close 30000 (time 1765106493.45 total 67.74 last 468.70) - open/close 35046 (time 1765106503.46 total 77.75 last 504.51) - open/close 39373 (time 1765106513.46 total 87.75 last 432.62) - open/close 43245 (time 1765106523.46 total 97.75 last 387.15) - open/close 47287 (time 1765106533.46 total 107.75 last 404.12) - open/close 50000 (time 1765106540.18 total 114.47 last 403.90) - open/close 53862 (time 1765106550.18 total 124.47 last 386.16) - open/close 57665 (time 1765106560.18 total 134.47 last 380.27) - open/close 60000 (time 1765106565.44 total 139.73 last 443.93) total: 64768 open/close in 149.43 seconds: 433.44 ops/second - /unlink 7866 (time 1765106586.43 total 10.00 last 786.54) - /unlink 10000 (time 1765106588.83 total 12.41 last 887.35) - /unlink 17562 (time 1765106598.83 total 22.41 last 756.13) - /unlink 20000 (time 1765106602.56 total 26.13 last 654.13) - /unlink 26831 (time 1765106612.56 total 36.13 last 683.07) - /unlink 30000 (time 1765106616.93 total 40.51 last 724.84) - /unlink 38382 (time 1765106626.93 total 50.51 last 838.10) - /unlink 45733 (time 1765106636.93 total 60.51 last 735.00) - /unlink 50000 (time 1765106642.62 total 66.20 last 749.83) - /unlink 58209 (time 1765106652.63 total 76.20 last 820.75) - /unlink 60000 (time 1765106654.95 total 78.53 last 769.66) total: 64768 /unlink in 84.25 seconds: 768.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg103-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg103-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg103-server stop mds service on oleg103-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg103-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing unload_modules_local oleg103-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg103-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-client: ssh exited with exit code 2 pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2 PASS 106 (3471s) Stopping clients: oleg103-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg103-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg103-client: oleg103-server: ssh exited with exit code 2 oleg103-server: oleg103-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg103-server' oleg103-server: oleg103-server.virtnet: executing load_modules_local oleg103-server: Loading modules from /home/green/git/lustre-release/lustre oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg103-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg103-server: detected 4 online CPUs by sysfs oleg103-server: MODOPTS_LIBCFS= oleg103-server: Force libcfs to create 2 CPU partitions oleg103-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg103-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3649 sec ========== 06:25:15 (1765106715) === conf-sanity: start cleanup 06:25:15 (1765106715) === === conf-sanity: finish cleanup 06:25:16 (1765106716) ===