-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 10:30:36 EDT 2025 oleg403-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg403-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg403-server oleg403-server: oleg403-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg403-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg403-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:33:24 (1761316404) Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) oleg403-server: oleg403-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg403-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg403-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock oleg403-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 765 (time 1761316518.79 total 10.00 last 76.48) - open/close 1634 (time 1761316528.79 total 20.01 last 86.83) - open/close 2484 (time 1761316538.79 total 30.01 last 85.00) - open/close 3169 (time 1761316548.80 total 40.01 last 68.48) - open/close 3800 (time 1761316558.80 total 50.02 last 63.09) - open/close 4652 (time 1761316568.80 total 60.02 last 85.16) - open/close 5470 (time 1761316578.81 total 70.03 last 81.70) - open/close 6247 (time 1761316588.83 total 80.05 last 77.56) - open/close 7146 (time 1761316598.83 total 90.05 last 89.89) - open/close 8015 (time 1761316608.84 total 100.06 last 86.85) - open/close 8889 (time 1761316618.86 total 110.07 last 87.26) - open/close 9723 (time 1761316628.86 total 120.08 last 83.38) - open/close 10000 (time 1761316632.28 total 123.49 last 81.09) - open/close 10936 (time 1761316642.29 total 133.51 last 93.43) - open/close 11827 (time 1761316652.30 total 143.51 last 89.07) - open/close 12693 (time 1761316662.30 total 153.52 last 86.59) - open/close 13483 (time 1761316672.31 total 163.53 last 78.92) - open/close 14017 (time 1761316682.34 total 173.56 last 53.23) - open/close 14549 (time 1761316692.37 total 183.58 last 53.07) - open/close 15191 (time 1761316702.37 total 193.58 last 64.20) - open/close 16084 (time 1761316712.37 total 203.59 last 89.22) - open/close 16972 (time 1761316722.39 total 213.60 last 88.70) - open/close 17908 (time 1761316732.39 total 223.61 last 93.53) - open/close 18825 (time 1761316742.40 total 233.61 last 91.68) - open/close 19636 (time 1761316752.40 total 243.62 last 81.04) - open/close 20000 (time 1761316757.12 total 248.34 last 77.11) - open/close 20825 (time 1761316767.13 total 258.34 last 82.48) - open/close 21603 (time 1761316777.13 total 268.34 last 77.80) - open/close 22486 (time 1761316787.13 total 278.35 last 88.27) - open/close 23374 (time 1761316797.14 total 288.35 last 88.75) - open/close 24243 (time 1761316807.14 total 298.36 last 86.86) - open/close 25105 (time 1761316817.14 total 308.36 last 86.16) - open/close 25920 (time 1761316827.15 total 318.37 last 81.47) - open/close 26773 (time 1761316837.15 total 328.37 last 85.30) - open/close 27651 (time 1761316847.15 total 338.37 last 87.77) - open/close 28496 (time 1761316857.16 total 348.38 last 84.42) - open/close 29320 (time 1761316867.17 total 358.38 last 82.36) - open/close 30000 (time 1761316875.21 total 366.43 last 84.56) - open/close 30781 (time 1761316885.22 total 376.43 last 78.04) - open/close 31651 (time 1761316895.22 total 386.44 last 86.94) - open/close 32479 (time 1761316905.23 total 396.44 last 82.76) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 397.77 seconds: 81.94 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1484 (time 1761316920.10 total 10.00 last 148.36) - /unlink 2792 (time 1761316930.10 total 20.01 last 130.76) - /unlink 4167 (time 1761316940.11 total 30.02 last 137.35) - /unlink 5592 (time 1761316950.11 total 40.02 last 142.46) - /unlink 6980 (time 1761316960.11 total 50.02 last 138.78) - /unlink 8423 (time 1761316970.13 total 60.03 last 144.09) - /unlink 9839 (time 1761316980.13 total 70.03 last 141.60) - /unlink 11298 (time 1761316990.13 total 80.04 last 145.85) - /unlink 12538 (time 1761317000.14 total 90.05 last 123.87) - /unlink 13942 (time 1761317010.15 total 100.05 last 140.33) - /unlink 15388 (time 1761317020.15 total 110.06 last 144.52) - /unlink 16635 (time 1761317030.16 total 120.06 last 124.64) - /unlink 18052 (time 1761317040.16 total 130.07 last 141.60) - /unlink 19403 (time 1761317050.17 total 140.08 last 135.03) - /unlink 20000 (time 1761317054.26 total 144.17 last 145.91) - /unlink 21461 (time 1761317064.27 total 154.18 last 145.95) - /unlink 22982 (time 1761317074.28 total 164.18 last 152.05) - /unlink 24535 (time 1761317084.28 total 174.19 last 155.23) - /unlink 26105 (time 1761317094.28 total 184.19 last 156.96) - /unlink 27708 (time 1761317104.29 total 194.19 last 160.21) - /unlink 29377 (time 1761317114.30 total 204.20 last 166.74) - /unlink 30000 (time 1761317117.88 total 207.79 last 173.80) - /unlink 31614 (time 1761317127.89 total 217.79 last 161.35) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 223.33 seconds: 145.95 ops/second Waiting for MDT destroys to complete - open/close 922 (time 1761317159.96 total 10.00 last 92.16) - open/close 1831 (time 1761317169.97 total 20.01 last 90.83) - open/close 2776 (time 1761317179.98 total 30.02 last 94.47) - open/close 3704 (time 1761317189.98 total 40.02 last 92.74) - open/close 4639 (time 1761317199.99 total 50.03 last 93.46) - open/close 5523 (time 1761317209.99 total 60.03 last 88.36) - open/close 6435 (time 1761317220.00 total 70.04 last 91.15) - open/close 7322 (time 1761317230.01 total 80.05 last 88.59) - open/close 8209 (time 1761317240.01 total 90.05 last 88.66) - open/close 9092 (time 1761317250.02 total 100.06 last 88.28) - open/close 9975 (time 1761317260.02 total 110.06 last 88.27) - open/close 10772 (time 1761317270.03 total 120.07 last 79.60) - open/close 11743 (time 1761317280.05 total 130.09 last 96.95) - open/close 12663 (time 1761317290.05 total 140.09 last 91.98) - open/close 13618 (time 1761317300.06 total 150.10 last 95.37) - open/close 14519 (time 1761317310.06 total 160.10 last 90.09) - open/close 15364 (time 1761317320.08 total 170.12 last 84.34) - open/close 16138 (time 1761317330.09 total 180.13 last 77.37) - open/close 16986 (time 1761317340.09 total 190.13 last 84.79) - open/close 17681 (time 1761317350.10 total 200.14 last 69.40) - open/close 18411 (time 1761317360.10 total 210.14 last 72.99) - open/close 18977 (time 1761317370.10 total 220.14 last 56.59) - open/close 19685 (time 1761317380.11 total 230.15 last 70.73) - open/close 20000 (time 1761317384.86 total 234.90 last 66.31) - open/close 20753 (time 1761317394.87 total 244.91 last 75.24) - open/close 21613 (time 1761317404.90 total 254.94 last 85.77) - open/close 22340 (time 1761317414.90 total 264.94 last 72.68) - open/close 23186 (time 1761317424.91 total 274.95 last 84.55) - open/close 24034 (time 1761317434.92 total 284.96 last 84.67) - open/close 24883 (time 1761317444.92 total 294.96 last 84.88) - open/close 25718 (time 1761317454.93 total 304.97 last 83.47) - open/close 26546 (time 1761317464.93 total 314.97 last 82.76) - open/close 27255 (time 1761317474.94 total 324.98 last 70.87) - open/close 27961 (time 1761317484.95 total 334.99 last 70.49) - open/close 28675 (time 1761317494.96 total 345.00 last 71.36) - open/close 29412 (time 1761317504.97 total 355.01 last 73.65) - open/close 30000 (time 1761317513.27 total 363.31 last 70.84) - open/close 30640 (time 1761317523.27 total 373.31 last 63.96) - open/close 31329 (time 1761317533.27 total 383.31 last 68.90) - open/close 32115 (time 1761317543.27 total 393.31 last 78.58) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 399.43 seconds: 81.60 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1340 (time 1761317563.05 total 10.00 last 133.97) - /unlink 2718 (time 1761317573.05 total 20.00 last 137.79) - /unlink 4151 (time 1761317583.05 total 30.01 last 143.26) - /unlink 5712 (time 1761317593.05 total 40.01 last 156.06) - /unlink 7150 (time 1761317603.06 total 50.01 last 143.77) - /unlink 8696 (time 1761317613.06 total 60.01 last 154.56) - /unlink 10000 (time 1761317622.53 total 69.48 last 137.70) - /unlink 11446 (time 1761317632.53 total 79.49 last 144.57) - /unlink 13065 (time 1761317642.53 total 89.49 last 161.87) - /unlink 14533 (time 1761317652.54 total 99.49 last 146.76) - /unlink 16061 (time 1761317662.54 total 109.49 last 152.74) - /unlink 17431 (time 1761317672.55 total 119.50 last 136.92) - /unlink 18767 (time 1761317682.55 total 129.51 last 133.51) - /unlink 20000 (time 1761317690.86 total 137.81 last 148.43) - /unlink 21506 (time 1761317700.86 total 147.82 last 150.55) - /unlink 22985 (time 1761317710.87 total 157.82 last 147.84) - /unlink 24554 (time 1761317720.87 total 167.82 last 156.90) - /unlink 25996 (time 1761317730.87 total 177.82 last 144.17) - /unlink 27437 (time 1761317740.87 total 187.82 last 144.08) - /unlink 28914 (time 1761317750.87 total 197.83 last 147.65) - /unlink 30000 (time 1761317758.53 total 205.48 last 141.86) - /unlink 31470 (time 1761317768.55 total 215.51 last 146.67) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 223.43 seconds: 145.88 ops/second Waiting for MDT destroys to complete - open/close 770 (time 1761317804.49 total 10.01 last 76.91) - open/close 1558 (time 1761317814.49 total 20.01 last 78.78) - open/close 2282 (time 1761317824.50 total 30.03 last 72.29) - open/close 3037 (time 1761317834.51 total 40.03 last 75.46) - open/close 3842 (time 1761317844.52 total 50.04 last 80.44) - open/close 4724 (time 1761317854.52 total 60.04 last 88.17) - open/close 5588 (time 1761317864.52 total 70.05 last 86.38) - open/close 6426 (time 1761317874.52 total 80.05 last 83.79) - open/close 7261 (time 1761317884.53 total 90.06 last 83.40) - open/close 8112 (time 1761317894.54 total 100.07 last 85.05) - open/close 9003 (time 1761317904.54 total 110.07 last 89.08) - open/close 9805 (time 1761317914.55 total 120.07 last 80.14) - open/close 10000 (time 1761317916.85 total 122.38 last 84.72) - open/close 10789 (time 1761317926.85 total 132.38 last 78.90) - open/close 11598 (time 1761317936.85 total 142.38 last 80.88) - open/close 12485 (time 1761317946.86 total 152.38 last 88.67) - open/close 13401 (time 1761317956.86 total 162.39 last 91.56) - open/close 14262 (time 1761317966.87 total 172.39 last 86.06) - open/close 15100 (time 1761317976.88 total 182.40 last 83.69) - open/close 15956 (time 1761317986.88 total 192.41 last 85.57) - open/close 16823 (time 1761317996.89 total 202.42 last 86.61) - open/close 17711 (time 1761318006.90 total 212.43 last 88.72) - open/close 18590 (time 1761318016.91 total 222.43 last 87.85) - open/close 19433 (time 1761318026.91 total 232.43 last 84.30) - open/close 20000 (time 1761318034.09 total 239.62 last 78.91) - open/close 20846 (time 1761318044.10 total 249.63 last 84.52) - open/close 21684 (time 1761318054.10 total 259.63 last 83.78) - open/close 22438 (time 1761318064.11 total 269.64 last 75.33) - open/close 23249 (time 1761318074.11 total 279.64 last 81.09) - open/close 24046 (time 1761318084.11 total 289.64 last 79.69) - open/close 24872 (time 1761318094.12 total 299.65 last 82.52) - open/close 25724 (time 1761318104.13 total 309.65 last 85.17) - open/close 26497 (time 1761318114.14 total 319.66 last 77.21) - open/close 27334 (time 1761318124.15 total 329.67 last 83.64) - open/close 28108 (time 1761318134.16 total 339.68 last 77.33) - open/close 28946 (time 1761318144.17 total 349.69 last 83.70) - open/close 29807 (time 1761318154.18 total 359.70 last 86.02) - open/close 30000 (time 1761318156.85 total 362.38 last 72.13) - open/close 30821 (time 1761318166.86 total 372.39 last 81.99) - open/close 31616 (time 1761318176.87 total 382.39 last 79.47) - open/close 32401 (time 1761318186.88 total 392.40 last 78.43) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 394.96 seconds: 82.53 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1307 (time 1761318203.34 total 10.00 last 130.67) - /unlink 2677 (time 1761318213.35 total 20.00 last 136.98) - /unlink 4028 (time 1761318223.35 total 30.01 last 135.06) - /unlink 5621 (time 1761318233.35 total 40.01 last 159.24) - /unlink 7036 (time 1761318243.36 total 50.01 last 141.47) - /unlink 8407 (time 1761318253.36 total 60.02 last 137.07) - /unlink 9806 (time 1761318263.36 total 70.02 last 139.88) - /unlink 11141 (time 1761318273.36 total 80.02 last 133.44) - /unlink 12458 (time 1761318283.36 total 90.02 last 131.68) - /unlink 13802 (time 1761318293.37 total 100.03 last 134.35) - /unlink 15199 (time 1761318303.37 total 110.03 last 139.66) - /unlink 16539 (time 1761318313.38 total 120.03 last 133.96) - /unlink 17912 (time 1761318323.38 total 130.03 last 137.28) - /unlink 19260 (time 1761318333.38 total 140.04 last 134.73) - /unlink 20000 (time 1761318339.69 total 146.35 last 117.26) - /unlink 21452 (time 1761318349.69 total 156.35 last 145.18) - /unlink 22952 (time 1761318359.69 total 166.35 last 149.99) - /unlink 24339 (time 1761318369.70 total 176.36 last 138.63) - /unlink 25718 (time 1761318379.70 total 186.36 last 137.88) - /unlink 27054 (time 1761318389.70 total 196.36 last 133.55) - /unlink 28468 (time 1761318399.71 total 206.36 last 141.37) - /unlink 29838 (time 1761318409.71 total 216.37 last 136.99) - /unlink 31281 (time 1761318419.71 total 226.37 last 144.26) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 235.59 seconds: 138.35 ops/second Waiting for MDT destroys to complete - open/close 857 (time 1761318454.47 total 10.00 last 85.69) - open/close 1705 (time 1761318464.48 total 20.01 last 84.71) - open/close 2601 (time 1761318474.48 total 30.01 last 89.59) - open/close 3510 (time 1761318484.48 total 40.02 last 90.84) - open/close 4418 (time 1761318494.50 total 50.03 last 90.69) - open/close 5329 (time 1761318504.52 total 60.05 last 90.92) - open/close 6223 (time 1761318514.52 total 70.05 last 89.38) - open/close 7080 (time 1761318524.52 total 80.06 last 85.66) - open/close 7916 (time 1761318534.52 total 90.06 last 83.60) - open/close 8755 (time 1761318544.53 total 100.06 last 83.84) - open/close 9539 (time 1761318554.54 total 110.07 last 78.34) - open/close 10000 (time 1761318560.65 total 116.18 last 75.43) - open/close 10874 (time 1761318570.65 total 126.19 last 87.38) - open/close 11735 (time 1761318580.66 total 136.19 last 86.07) - open/close 12580 (time 1761318590.66 total 146.20 last 84.46) - open/close 13409 (time 1761318600.67 total 156.20 last 82.83) - open/close 14239 (time 1761318610.68 total 166.21 last 82.91) - open/close 15025 (time 1761318620.69 total 176.22 last 78.53) - open/close 15842 (time 1761318630.69 total 186.23 last 81.67) - open/close 16609 (time 1761318640.71 total 196.24 last 76.57) - open/close 17444 (time 1761318650.72 total 206.25 last 83.43) - open/close 18123 (time 1761318660.72 total 216.26 last 67.86) - open/close 18889 (time 1761318670.73 total 226.27 last 76.54) - open/close 19682 (time 1761318680.74 total 236.28 last 79.21) - open/close 20000 (time 1761318684.49 total 240.02 last 84.86) - open/close 20830 (time 1761318694.51 total 250.04 last 82.85) - open/close 21674 (time 1761318704.51 total 260.05 last 84.37) - open/close 22495 (time 1761318714.53 total 270.06 last 81.95) - open/close 23372 (time 1761318724.54 total 280.08 last 87.59) - open/close 24225 (time 1761318734.56 total 290.09 last 85.19) - open/close 25077 (time 1761318744.56 total 300.10 last 85.14) - open/close 25931 (time 1761318754.57 total 310.11 last 85.31) - open/close 26721 (time 1761318764.58 total 320.12 last 78.90) - open/close 27495 (time 1761318774.59 total 330.12 last 77.37) - open/close 28276 (time 1761318784.59 total 340.13 last 78.07) - open/close 29141 (time 1761318794.59 total 350.13 last 86.49) - open/close 29976 (time 1761318804.59 total 360.13 last 83.49) - open/close 30821 (time 1761318814.60 total 370.13 last 84.49) - open/close 31569 (time 1761318824.60 total 380.14 last 74.74) - open/close 32390 (time 1761318834.61 total 390.14 last 82.09) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 392.87 seconds: 82.96 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1381 (time 1761318851.15 total 10.01 last 138.01) - /unlink 2835 (time 1761318861.15 total 20.01 last 145.36) - /unlink 4354 (time 1761318871.15 total 30.01 last 151.86) - /unlink 5826 (time 1761318881.16 total 40.02 last 147.14) - /unlink 7326 (time 1761318891.17 total 50.02 last 149.86) - /unlink 8698 (time 1761318901.17 total 60.03 last 137.16) - /unlink 10000 (time 1761318910.06 total 68.92 last 146.46) - /unlink 11501 (time 1761318920.07 total 78.93 last 149.98) - /unlink 12877 (time 1761318930.08 total 88.93 last 137.50) - /unlink 14351 (time 1761318940.08 total 98.93 last 147.38) - /unlink 15707 (time 1761318950.08 total 108.94 last 135.52) - /unlink 17264 (time 1761318960.09 total 118.94 last 155.65) - /unlink 18612 (time 1761318970.09 total 128.95 last 134.76) - /unlink 20000 (time 1761318979.12 total 137.98 last 153.65) - /unlink 21516 (time 1761318989.12 total 147.98 last 151.57) - /unlink 23005 (time 1761318999.13 total 157.98 last 148.87) - /unlink 24602 (time 1761319009.13 total 167.99 last 159.58) - /unlink 26055 (time 1761319019.14 total 178.00 last 145.24) - /unlink 27565 (time 1761319029.14 total 188.00 last 150.93) - /unlink 29022 (time 1761319039.15 total 198.00 last 145.67) - /unlink 30000 (time 1761319045.35 total 204.21 last 157.58) - /unlink 31467 (time 1761319055.36 total 214.22 last 146.57) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 222.00 seconds: 146.82 ops/second Waiting for MDT destroys to complete - open/close 883 (time 1761319085.04 total 10.01 last 88.23) - open/close 1777 (time 1761319095.05 total 20.02 last 89.32) - open/close 2635 (time 1761319105.05 total 30.02 last 85.78) - open/close 3422 (time 1761319115.05 total 40.02 last 78.67) - open/close 4308 (time 1761319125.06 total 50.03 last 88.52) - open/close 5086 (time 1761319135.07 total 60.03 last 77.78) - open/close 6035 (time 1761319145.08 total 70.05 last 94.76) - open/close 6885 (time 1761319155.09 total 80.06 last 84.94) - open/close 7775 (time 1761319165.10 total 90.07 last 88.89) - open/close 8671 (time 1761319175.11 total 100.08 last 89.52) - open/close 9576 (time 1761319185.12 total 110.09 last 90.40) - open/close 10000 (time 1761319189.89 total 114.85 last 88.98) - open/close 10913 (time 1761319199.90 total 124.86 last 91.20) - open/close 11868 (time 1761319209.90 total 134.86 last 95.50) - open/close 12809 (time 1761319219.90 total 144.87 last 94.06) - open/close 13835 (time 1761319229.91 total 154.88 last 102.50) - open/close 14750 (time 1761319239.92 total 164.88 last 91.45) - open/close 15716 (time 1761319249.92 total 174.88 last 96.59) - open/close 16723 (time 1761319259.92 total 184.89 last 100.64) - open/close 17794 (time 1761319269.93 total 194.90 last 107.00) - open/close 18990 (time 1761319279.94 total 204.90 last 119.55) - open/close 19986 (time 1761319289.96 total 214.93 last 99.35) - open/close 20951 (time 1761319299.96 total 224.93 last 96.47) - open/close 21976 (time 1761319309.96 total 234.93 last 102.49) - open/close 23024 (time 1761319319.98 total 244.95 last 104.67) - open/close 24015 (time 1761319329.99 total 254.95 last 99.01) - open/close 25052 (time 1761319339.99 total 264.96 last 103.64) - open/close 26078 (time 1761319350.00 total 274.96 last 102.55) - open/close 27087 (time 1761319360.00 total 284.97 last 100.82) - open/close 28177 (time 1761319370.02 total 294.98 last 108.88) - open/close 29153 (time 1761319380.02 total 304.99 last 97.53) - open/close 30000 (time 1761319387.84 total 312.81 last 108.31) - open/close 31110 (time 1761319397.85 total 322.81 last 110.96) - open/close 32237 (time 1761319407.85 total 332.82 last 112.68) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 335.78 seconds: 97.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2550 (time 1761319423.28 total 10.00 last 254.98) - /unlink 4816 (time 1761319433.29 total 20.01 last 226.49) - /unlink 7117 (time 1761319443.29 total 30.01 last 230.10) - /unlink 9345 (time 1761319453.29 total 40.01 last 222.77) - /unlink 10000 (time 1761319456.05 total 42.77 last 237.11) - /unlink 12377 (time 1761319466.06 total 52.77 last 237.61) - /unlink 15404 (time 1761319476.06 total 62.78 last 302.49) - /unlink 18330 (time 1761319486.07 total 72.78 last 292.51) - /unlink 20000 (time 1761319491.52 total 78.23 last 306.43) - /unlink 22721 (time 1761319501.52 total 88.24 last 271.93) - /unlink 25266 (time 1761319511.53 total 98.24 last 254.40) - /unlink 28011 (time 1761319521.53 total 108.25 last 274.45) - /unlink 30000 (time 1761319528.21 total 114.93 last 297.70) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 123.66 seconds: 263.58 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg403-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local oleg403-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg403-server: LNET ready to unload oleg403-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 PASS 106 (3181s) Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 oleg403-server: oleg403-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg403-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg403-server: detected 4 online CPUs by sysfs oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3416 sec ========== 11:27:34 (1761319654)