-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:43:42 EDT 2025 oleg207-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg207-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg207-server oleg207-server: oleg207-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg207-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg207-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg207-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg207-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg207-server: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg207-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:46:24 (1761313584) Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) oleg207-server: oleg207-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg207-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg207-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg207-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock oleg207-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 886 (time 1761313706.66 total 10.00 last 88.56) - open/close 1891 (time 1761313716.67 total 20.01 last 100.43) - open/close 2762 (time 1761313726.68 total 30.02 last 87.01) - open/close 3649 (time 1761313736.68 total 40.02 last 88.69) - open/close 4441 (time 1761313746.70 total 50.04 last 79.06) - open/close 5263 (time 1761313756.70 total 60.05 last 82.16) - open/close 6020 (time 1761313766.72 total 70.06 last 75.60) - open/close 6830 (time 1761313776.72 total 80.06 last 80.98) - open/close 7492 (time 1761313786.72 total 90.07 last 66.17) - open/close 8260 (time 1761313796.73 total 100.07 last 76.77) - open/close 9060 (time 1761313806.73 total 110.07 last 79.98) - open/close 9835 (time 1761313816.74 total 120.09 last 77.39) - open/close 10670 (time 1761313826.75 total 130.09 last 83.44) - open/close 11492 (time 1761313836.76 total 140.10 last 82.11) - open/close 12289 (time 1761313846.76 total 150.11 last 79.67) - open/close 13203 (time 1761313856.78 total 160.12 last 91.27) - open/close 14028 (time 1761313866.79 total 170.13 last 82.41) - open/close 14750 (time 1761313876.79 total 180.14 last 72.18) - open/close 15453 (time 1761313886.80 total 190.14 last 70.26) - open/close 16230 (time 1761313896.80 total 200.15 last 77.68) - open/close 16992 (time 1761313906.82 total 210.16 last 76.08) - open/close 17717 (time 1761313916.82 total 220.17 last 72.47) - open/close 18553 (time 1761313926.83 total 230.17 last 83.56) - open/close 19422 (time 1761313936.83 total 240.17 last 86.88) - open/close 20000 (time 1761313943.52 total 246.86 last 86.44) - open/close 20873 (time 1761313953.53 total 256.87 last 87.21) - open/close 21785 (time 1761313963.53 total 266.87 last 91.19) - open/close 22657 (time 1761313973.53 total 276.87 last 87.17) - open/close 23606 (time 1761313983.54 total 286.88 last 94.85) - open/close 24475 (time 1761313993.55 total 296.89 last 86.79) - open/close 25385 (time 1761314003.55 total 306.90 last 90.95) - open/close 26257 (time 1761314013.56 total 316.90 last 87.15) - open/close 27175 (time 1761314023.57 total 326.91 last 91.73) - open/close 28063 (time 1761314033.58 total 336.92 last 88.71) - open/close 28924 (time 1761314043.58 total 346.93 last 86.05) - open/close 29803 (time 1761314053.58 total 356.93 last 87.90) - open/close 30000 (time 1761314055.85 total 359.20 last 86.75) - open/close 30890 (time 1761314065.86 total 369.20 last 88.97) - open/close 31783 (time 1761314075.86 total 379.20 last 89.29) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 388.18 seconds: 83.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1392 (time 1761314098.75 total 10.01 last 139.05) - /unlink 2682 (time 1761314108.75 total 20.02 last 128.93) - /unlink 3874 (time 1761314118.76 total 30.02 last 119.15) - /unlink 5262 (time 1761314128.77 total 40.03 last 138.69) - /unlink 6659 (time 1761314138.77 total 50.03 last 139.68) - /unlink 8082 (time 1761314148.77 total 60.03 last 142.27) - /unlink 9443 (time 1761314158.77 total 70.03 last 136.06) - /unlink 10000 (time 1761314162.49 total 73.75 last 149.89) - /unlink 11444 (time 1761314172.49 total 83.75 last 144.34) - /unlink 12816 (time 1761314182.50 total 93.76 last 137.15) - /unlink 14216 (time 1761314192.50 total 103.76 last 139.97) - /unlink 15651 (time 1761314202.50 total 113.76 last 143.49) - /unlink 17146 (time 1761314212.50 total 123.77 last 149.43) - /unlink 18638 (time 1761314222.50 total 133.77 last 149.19) - /unlink 20000 (time 1761314232.27 total 143.53 last 139.48) - /unlink 21476 (time 1761314242.27 total 153.54 last 147.51) - /unlink 22868 (time 1761314252.28 total 163.54 last 139.20) - /unlink 24132 (time 1761314262.29 total 173.55 last 126.20) - /unlink 25523 (time 1761314272.30 total 183.56 last 139.04) - /unlink 27049 (time 1761314282.30 total 193.56 last 152.58) - /unlink 28468 (time 1761314292.30 total 203.56 last 141.87) - /unlink 29936 (time 1761314302.30 total 213.56 last 146.80) - /unlink 31414 (time 1761314312.30 total 223.56 last 147.75) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 231.85 seconds: 140.58 ops/second Waiting for MDT destroys to complete - open/close 906 (time 1761314343.54 total 10.00 last 90.57) - open/close 1784 (time 1761314353.55 total 20.01 last 87.73) - open/close 2616 (time 1761314363.55 total 30.02 last 83.16) - open/close 3492 (time 1761314373.56 total 40.02 last 87.54) - open/close 4294 (time 1761314383.56 total 50.02 last 80.19) - open/close 5173 (time 1761314393.57 total 60.03 last 87.83) - open/close 6035 (time 1761314403.57 total 70.04 last 86.16) - open/close 6832 (time 1761314413.58 total 80.05 last 79.61) - open/close 7679 (time 1761314423.58 total 90.05 last 84.70) - open/close 8544 (time 1761314433.59 total 100.05 last 86.44) - open/close 9399 (time 1761314443.59 total 110.06 last 85.48) - open/close 10000 (time 1761314451.34 total 117.81 last 77.51) - open/close 10860 (time 1761314461.35 total 127.82 last 85.93) - open/close 11655 (time 1761314471.36 total 137.83 last 79.42) - open/close 12481 (time 1761314481.37 total 147.83 last 82.56) - open/close 13361 (time 1761314491.37 total 157.83 last 88.00) - open/close 14249 (time 1761314501.38 total 167.84 last 88.72) - open/close 15121 (time 1761314511.38 total 177.85 last 87.14) - open/close 16001 (time 1761314521.39 total 187.86 last 87.95) - open/close 16802 (time 1761314531.40 total 197.86 last 80.05) - open/close 17541 (time 1761314541.41 total 207.87 last 73.81) - open/close 18436 (time 1761314551.41 total 217.88 last 89.49) - open/close 19259 (time 1761314561.41 total 227.88 last 82.26) - open/close 20000 (time 1761314570.99 total 237.46 last 77.36) - open/close 20733 (time 1761314581.00 total 247.46 last 73.27) - open/close 21631 (time 1761314591.00 total 257.47 last 89.73) - open/close 22510 (time 1761314601.01 total 267.48 last 87.84) - open/close 23490 (time 1761314611.02 total 277.48 last 97.95) - open/close 24364 (time 1761314621.02 total 287.49 last 87.34) - open/close 25243 (time 1761314631.03 total 297.49 last 87.86) - open/close 26096 (time 1761314641.05 total 307.52 last 85.10) - open/close 26981 (time 1761314651.06 total 317.52 last 88.45) - open/close 27883 (time 1761314661.06 total 327.53 last 90.17) - open/close 28760 (time 1761314671.07 total 337.53 last 87.62) - open/close 29612 (time 1761314681.09 total 347.55 last 85.03) - open/close 30000 (time 1761314685.85 total 352.32 last 81.46) - open/close 30831 (time 1761314695.86 total 362.32 last 83.06) - open/close 31720 (time 1761314705.86 total 372.32 last 88.90) - open/close 32508 (time 1761314715.86 total 382.33 last 78.74) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 383.44 seconds: 85.00 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1421 (time 1761314730.88 total 10.00 last 142.10) - /unlink 2889 (time 1761314740.89 total 20.00 last 146.78) - /unlink 4430 (time 1761314750.89 total 30.00 last 154.05) - /unlink 6017 (time 1761314760.89 total 40.01 last 158.66) - /unlink 7397 (time 1761314770.90 total 50.01 last 137.90) - /unlink 8885 (time 1761314780.90 total 60.02 last 148.74) - /unlink 10000 (time 1761314788.80 total 67.92 last 141.13) - /unlink 11456 (time 1761314798.81 total 77.92 last 145.55) - /unlink 12868 (time 1761314808.81 total 87.92 last 141.18) - /unlink 14254 (time 1761314818.81 total 97.93 last 138.58) - /unlink 15704 (time 1761314828.81 total 107.93 last 144.96) - /unlink 17131 (time 1761314838.82 total 117.94 last 142.59) - /unlink 18633 (time 1761314848.82 total 127.94 last 150.16) - /unlink 20000 (time 1761314857.96 total 137.07 last 149.67) - /unlink 21511 (time 1761314867.96 total 147.07 last 151.08) - /unlink 22936 (time 1761314877.96 total 157.08 last 142.43) - /unlink 24421 (time 1761314887.97 total 167.08 last 148.43) - /unlink 25863 (time 1761314897.97 total 177.09 last 144.15) - /unlink 27321 (time 1761314907.97 total 187.09 last 145.74) - /unlink 28798 (time 1761314917.98 total 197.09 last 147.65) - /unlink 30000 (time 1761314926.64 total 205.75 last 138.81) - /unlink 31408 (time 1761314936.64 total 215.76 last 140.69) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 223.87 seconds: 145.60 ops/second Waiting for MDT destroys to complete - open/close 870 (time 1761314971.48 total 10.00 last 87.00) - open/close 1758 (time 1761314981.48 total 20.01 last 88.76) - open/close 2606 (time 1761314991.49 total 30.01 last 84.76) - open/close 3503 (time 1761315001.49 total 40.01 last 89.69) - open/close 4359 (time 1761315011.49 total 50.01 last 85.60) - open/close 5195 (time 1761315021.49 total 60.01 last 83.60) - open/close 6063 (time 1761315031.49 total 70.01 last 86.79) - open/close 6847 (time 1761315041.50 total 80.02 last 78.32) - open/close 7687 (time 1761315051.51 total 90.03 last 83.91) - open/close 8565 (time 1761315061.52 total 100.04 last 87.73) - open/close 9408 (time 1761315071.53 total 110.05 last 84.23) - open/close 10000 (time 1761315078.04 total 116.56 last 90.92) - open/close 10837 (time 1761315088.04 total 126.57 last 83.64) - open/close 11746 (time 1761315098.05 total 136.58 last 90.83) - open/close 12600 (time 1761315108.06 total 146.59 last 85.30) - open/close 13397 (time 1761315118.08 total 156.60 last 79.60) - open/close 14232 (time 1761315128.09 total 166.61 last 83.43) - open/close 15099 (time 1761315138.09 total 176.61 last 86.68) - open/close 15931 (time 1761315148.09 total 186.61 last 83.18) - open/close 16752 (time 1761315158.10 total 196.62 last 82.05) - open/close 17609 (time 1761315168.10 total 206.62 last 85.67) - open/close 18533 (time 1761315178.11 total 216.63 last 92.34) - open/close 19439 (time 1761315188.11 total 226.63 last 90.57) - open/close 20000 (time 1761315194.49 total 233.01 last 87.93) - open/close 20856 (time 1761315204.50 total 243.03 last 85.49) - open/close 21665 (time 1761315214.50 total 253.03 last 80.88) - open/close 22426 (time 1761315224.51 total 263.03 last 76.05) - open/close 23269 (time 1761315234.51 total 273.04 last 84.27) - open/close 24039 (time 1761315244.52 total 283.04 last 76.98) - open/close 24897 (time 1761315254.52 total 293.05 last 85.74) - open/close 25791 (time 1761315264.53 total 303.06 last 89.33) - open/close 26618 (time 1761315274.54 total 313.06 last 82.65) - open/close 27304 (time 1761315284.54 total 323.07 last 68.56) - open/close 28069 (time 1761315294.55 total 333.08 last 76.43) - open/close 28971 (time 1761315304.56 total 343.09 last 90.12) - open/close 29813 (time 1761315314.56 total 353.09 last 84.19) - open/close 30000 (time 1761315317.25 total 355.77 last 69.66) - open/close 30716 (time 1761315327.25 total 365.77 last 71.60) - open/close 31484 (time 1761315337.26 total 375.78 last 76.73) - open/close 32218 (time 1761315347.27 total 385.80 last 73.30) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 390.53 seconds: 83.46 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1408 (time 1761315365.61 total 10.00 last 140.75) - /unlink 2797 (time 1761315375.61 total 20.00 last 138.89) - /unlink 4154 (time 1761315385.61 total 30.00 last 135.70) - /unlink 5573 (time 1761315395.61 total 40.01 last 141.88) - /unlink 7052 (time 1761315405.62 total 50.01 last 147.77) - /unlink 8521 (time 1761315415.62 total 60.02 last 146.88) - /unlink 9999 (time 1761315425.62 total 70.02 last 147.75) - /unlink 11416 (time 1761315435.63 total 80.02 last 141.63) - /unlink 12853 (time 1761315445.63 total 90.03 last 143.66) - /unlink 14292 (time 1761315455.64 total 100.04 last 143.77) - /unlink 15562 (time 1761315465.64 total 110.04 last 126.97) - /unlink 16938 (time 1761315475.65 total 120.05 last 137.47) - /unlink 18321 (time 1761315485.66 total 130.05 last 138.24) - /unlink 19924 (time 1761315495.66 total 140.05 last 160.28) - /unlink 21479 (time 1761315505.66 total 150.06 last 155.44) - /unlink 23001 (time 1761315515.66 total 160.06 last 152.13) - /unlink 24471 (time 1761315525.67 total 170.07 last 146.91) - /unlink 25954 (time 1761315535.69 total 180.08 last 148.05) - /unlink 27440 (time 1761315545.69 total 190.09 last 148.52) - /unlink 28869 (time 1761315555.70 total 200.09 last 142.83) - /unlink 30000 (time 1761315563.53 total 207.93 last 144.40) - /unlink 31487 (time 1761315573.53 total 217.93 last 148.65) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 226.07 seconds: 144.18 ops/second Waiting for MDT destroys to complete - open/close 897 (time 1761315602.25 total 10.01 last 89.63) - open/close 1772 (time 1761315612.26 total 20.01 last 87.47) - open/close 2591 (time 1761315622.26 total 30.02 last 81.84) - open/close 3396 (time 1761315632.28 total 40.03 last 80.40) - open/close 4252 (time 1761315642.28 total 50.03 last 85.57) - open/close 5102 (time 1761315652.29 total 60.04 last 84.93) - open/close 5965 (time 1761315662.30 total 70.06 last 86.18) - open/close 6864 (time 1761315672.30 total 80.06 last 89.86) - open/close 7737 (time 1761315682.32 total 90.07 last 87.19) - open/close 8627 (time 1761315692.34 total 100.10 last 88.80) - open/close 9511 (time 1761315702.35 total 110.11 last 88.32) - open/close 10000 (time 1761315707.97 total 115.72 last 87.04) - open/close 10865 (time 1761315717.97 total 125.73 last 86.45) - open/close 11756 (time 1761315727.98 total 135.74 last 89.02) - open/close 12598 (time 1761315737.98 total 145.74 last 84.19) - open/close 13502 (time 1761315747.99 total 155.75 last 90.34) - open/close 14403 (time 1761315758.00 total 165.75 last 90.04) - open/close 15248 (time 1761315768.01 total 175.77 last 84.36) - open/close 16062 (time 1761315778.02 total 185.78 last 81.34) - open/close 16902 (time 1761315788.02 total 195.78 last 83.97) - open/close 17702 (time 1761315798.03 total 205.79 last 79.95) - open/close 18504 (time 1761315808.03 total 215.79 last 80.16) - open/close 19373 (time 1761315818.04 total 225.80 last 86.82) - open/close 20000 (time 1761315825.18 total 232.93 last 87.88) - open/close 20866 (time 1761315835.18 total 242.93 last 86.60) - open/close 21726 (time 1761315845.18 total 252.94 last 85.96) - open/close 22594 (time 1761315855.19 total 262.95 last 86.73) - open/close 23437 (time 1761315865.20 total 272.96 last 84.21) - open/close 24325 (time 1761315875.20 total 282.96 last 88.77) - open/close 25153 (time 1761315885.21 total 292.96 last 82.79) - open/close 26023 (time 1761315895.21 total 302.97 last 86.97) - open/close 26885 (time 1761315905.21 total 312.97 last 86.17) - open/close 27722 (time 1761315915.22 total 322.98 last 83.62) - open/close 28556 (time 1761315925.23 total 332.98 last 83.37) - open/close 29425 (time 1761315935.23 total 342.99 last 86.88) - open/close 30000 (time 1761315942.17 total 349.93 last 82.80) - open/close 30871 (time 1761315952.18 total 359.94 last 87.02) - open/close 31747 (time 1761315962.19 total 369.95 last 87.50) - open/close 32561 (time 1761315972.19 total 379.95 last 81.40) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 380.30 seconds: 85.71 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1450 (time 1761315986.15 total 10.00 last 144.97) - /unlink 2826 (time 1761315996.16 total 20.01 last 137.52) - /unlink 4309 (time 1761316006.16 total 30.01 last 148.23) - /unlink 5838 (time 1761316016.17 total 40.02 last 152.82) - /unlink 7321 (time 1761316026.17 total 50.02 last 148.28) - /unlink 8791 (time 1761316036.17 total 60.02 last 146.97) - /unlink 10000 (time 1761316044.52 total 68.38 last 144.71) - /unlink 11388 (time 1761316054.53 total 78.38 last 138.75) - /unlink 12927 (time 1761316064.54 total 88.39 last 153.72) - /unlink 14376 (time 1761316074.55 total 98.40 last 144.75) - /unlink 15850 (time 1761316084.55 total 108.41 last 147.34) - /unlink 17365 (time 1761316094.55 total 118.41 last 151.49) - /unlink 18879 (time 1761316104.56 total 128.41 last 151.38) - /unlink 20000 (time 1761316112.30 total 136.15 last 144.80) - /unlink 21512 (time 1761316122.30 total 146.15 last 151.13) - /unlink 22833 (time 1761316132.30 total 156.16 last 132.07) - /unlink 24012 (time 1761316142.31 total 166.16 last 117.84) - /unlink 25432 (time 1761316152.32 total 176.17 last 141.91) - /unlink 26970 (time 1761316162.32 total 186.18 last 153.67) - /unlink 28535 (time 1761316172.33 total 196.18 last 156.47) - /unlink 30000 (time 1761316182.31 total 206.16 last 146.73) - /unlink 31455 (time 1761316192.31 total 216.17 last 145.45) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 223.69 seconds: 145.71 ops/second Waiting for MDT destroys to complete - open/close 872 (time 1761316227.96 total 10.01 last 87.12) - open/close 1700 (time 1761316237.97 total 20.02 last 82.71) - open/close 2502 (time 1761316247.98 total 30.02 last 80.17) - open/close 3340 (time 1761316257.99 total 40.03 last 83.73) - open/close 4079 (time 1761316267.99 total 50.04 last 73.86) - open/close 4813 (time 1761316277.99 total 60.04 last 73.38) - open/close 5650 (time 1761316288.01 total 70.05 last 83.60) - open/close 6359 (time 1761316298.01 total 80.06 last 70.88) - open/close 7177 (time 1761316308.02 total 90.07 last 81.69) - open/close 7918 (time 1761316318.04 total 100.08 last 74.00) - open/close 8737 (time 1761316328.05 total 110.10 last 81.78) - open/close 9587 (time 1761316338.05 total 120.10 last 84.99) - open/close 10000 (time 1761316343.52 total 125.57 last 75.53) - open/close 10788 (time 1761316353.52 total 135.57 last 78.78) - open/close 11649 (time 1761316363.52 total 145.57 last 86.09) - open/close 12555 (time 1761316373.53 total 155.58 last 90.53) - open/close 13355 (time 1761316383.54 total 165.58 last 79.94) - open/close 13977 (time 1761316393.55 total 175.60 last 62.13) - open/close 14768 (time 1761316403.55 total 185.60 last 79.09) - open/close 15570 (time 1761316413.59 total 195.64 last 79.88) - open/close 16394 (time 1761316423.61 total 205.65 last 82.28) - open/close 17196 (time 1761316433.62 total 215.67 last 80.07) - open/close 18028 (time 1761316443.63 total 225.68 last 83.11) - open/close 18855 (time 1761316453.64 total 235.68 last 82.67) - open/close 19648 (time 1761316463.65 total 245.70 last 79.19) - open/close 20000 (time 1761316468.01 total 250.06 last 80.69) - open/close 20750 (time 1761316478.02 total 260.07 last 74.96) - open/close 21656 (time 1761316488.03 total 270.07 last 90.52) - open/close 22484 (time 1761316498.04 total 280.09 last 82.69) - open/close 23327 (time 1761316508.05 total 290.09 last 84.27) - open/close 24091 (time 1761316518.05 total 300.09 last 76.40) - open/close 24817 (time 1761316528.06 total 310.10 last 72.51) - open/close 25567 (time 1761316538.07 total 320.12 last 74.89) - open/close 26381 (time 1761316548.08 total 330.12 last 81.36) - open/close 27185 (time 1761316558.08 total 340.13 last 80.38) - open/close 27996 (time 1761316568.09 total 350.13 last 81.04) - open/close 28856 (time 1761316578.10 total 360.14 last 85.91) - open/close 29721 (time 1761316588.10 total 370.15 last 86.49) - open/close 30000 (time 1761316591.27 total 373.31 last 88.07) - open/close 30814 (time 1761316601.27 total 383.31 last 81.40) - open/close 31684 (time 1761316611.27 total 393.32 last 86.95) - open/close 32543 (time 1761316621.29 total 403.33 last 85.78) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 403.91 seconds: 80.70 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1473 (time 1761316635.16 total 10.00 last 147.26) - /unlink 3220 (time 1761316645.16 total 20.01 last 174.61) - /unlink 4786 (time 1761316655.16 total 30.01 last 156.55) - /unlink 6222 (time 1761316665.16 total 40.01 last 143.58) - /unlink 7611 (time 1761316675.17 total 50.01 last 138.88) - /unlink 9058 (time 1761316685.17 total 60.02 last 144.66) - /unlink 10000 (time 1761316692.62 total 67.47 last 126.44) - /unlink 11410 (time 1761316702.62 total 77.47 last 140.96) - /unlink 12870 (time 1761316712.63 total 87.48 last 145.93) - /unlink 14379 (time 1761316722.63 total 97.48 last 150.82) - /unlink 15797 (time 1761316732.63 total 107.48 last 141.79) - /unlink 17032 (time 1761316742.64 total 117.48 last 123.47) - /unlink 18414 (time 1761316752.64 total 127.49 last 138.15) - /unlink 19988 (time 1761316762.64 total 137.49 last 157.35) - /unlink 21529 (time 1761316772.65 total 147.49 last 154.06) - /unlink 23136 (time 1761316782.65 total 157.50 last 160.66) - /unlink 24803 (time 1761316792.65 total 167.50 last 166.69) - /unlink 26352 (time 1761316802.65 total 177.50 last 154.86) - /unlink 27779 (time 1761316812.66 total 187.51 last 142.61) - /unlink 29280 (time 1761316822.66 total 197.51 last 150.01) - /unlink 30000 (time 1761316827.88 total 202.73 last 137.96) - /unlink 31437 (time 1761316837.89 total 212.74 last 143.62) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 221.63 seconds: 147.07 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg207-server: LNET ready to unload oleg207-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 PASS 106 (3325s) Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 oleg207-server: oleg207-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg207-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg207-server: detected 4 online CPUs by sysfs oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3551 sec ========== 10:42:55 (1761316975)