-----============= acceptance-small: conf-sanity ============----- Tue Apr 14 16:55:56 EDT 2026 oleg216-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg216-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg216-server oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg216-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg216-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg216-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg216-server: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg216-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg216-server: oleg216-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg216-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:59:18 (1776200358) Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg216-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg216-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg216-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg216-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg216-server: oleg216-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg216-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg216-client.virtnet: -o user_xattr,flock oleg216-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 864 (time 1776200499.66 total 10.00 last 86.37) - open/close 1605 (time 1776200509.67 total 20.01 last 74.03) - open/close 2335 (time 1776200519.67 total 30.02 last 72.98) - open/close 3040 (time 1776200529.68 total 40.02 last 70.45) - open/close 3807 (time 1776200539.68 total 50.02 last 76.68) - open/close 4600 (time 1776200549.69 total 60.04 last 79.21) - open/close 5310 (time 1776200559.69 total 70.04 last 70.98) - open/close 6069 (time 1776200569.69 total 80.04 last 75.90) - open/close 6757 (time 1776200579.71 total 90.05 last 68.72) - open/close 7355 (time 1776200589.72 total 100.06 last 59.74) - open/close 8076 (time 1776200599.72 total 110.06 last 72.09) - open/close 8629 (time 1776200609.73 total 120.07 last 55.24) - open/close 9195 (time 1776200619.74 total 130.08 last 56.54) - open/close 9911 (time 1776200629.74 total 140.09 last 71.56) - open/close 10722 (time 1776200639.75 total 150.10 last 81.02) - open/close 11475 (time 1776200649.76 total 160.11 last 75.25) - open/close 12291 (time 1776200659.77 total 170.11 last 81.53) - open/close 13049 (time 1776200669.78 total 180.13 last 75.68) - open/close 13798 (time 1776200679.80 total 190.14 last 74.80) - open/close 14576 (time 1776200689.80 total 200.15 last 77.75) - open/close 15406 (time 1776200699.81 total 210.16 last 82.95) - open/close 16213 (time 1776200709.82 total 220.16 last 80.63) - open/close 16907 (time 1776200719.82 total 230.17 last 69.37) - open/close 17816 (time 1776200729.83 total 240.18 last 90.83) - open/close 18469 (time 1776200739.84 total 250.19 last 65.23) - open/close 19148 (time 1776200749.84 total 260.19 last 67.89) - open/close 19906 (time 1776200759.85 total 270.19 last 75.76) - open/close 20621 (time 1776200769.85 total 280.20 last 71.48) - open/close 21362 (time 1776200779.85 total 290.20 last 74.08) - open/close 22060 (time 1776200789.86 total 300.21 last 69.75) - open/close 22733 (time 1776200799.87 total 310.21 last 67.26) - open/close 23423 (time 1776200809.88 total 320.22 last 68.94) - open/close 24050 (time 1776200819.89 total 330.23 last 62.65) - open/close 24778 (time 1776200829.89 total 340.24 last 72.73) - open/close 25524 (time 1776200839.90 total 350.24 last 74.60) - open/close 26301 (time 1776200849.90 total 360.25 last 77.62) - open/close 26918 (time 1776200859.91 total 370.26 last 61.64) - open/close 27599 (time 1776200869.94 total 380.28 last 67.93) - open/close 28391 (time 1776200879.95 total 390.30 last 79.08) - open/close 29180 (time 1776200889.96 total 400.30 last 78.88) - open/close 29831 (time 1776200899.97 total 410.31 last 65.03) - open/close 30000 (time 1776200902.44 total 412.78 last 68.44) - open/close 30612 (time 1776200912.44 total 422.78 last 61.19) - open/close 31157 (time 1776200922.45 total 432.80 last 54.43) - open/close 31706 (time 1776200932.45 total 442.80 last 54.88) - open/close 32501 (time 1776200942.46 total 452.81 last 79.42) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 453.96 seconds: 71.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1384 (time 1776200956.87 total 10.00 last 138.34) - /unlink 2540 (time 1776200966.87 total 20.01 last 115.56) - /unlink 3757 (time 1776200976.87 total 30.01 last 121.65) - /unlink 5040 (time 1776200986.88 total 40.02 last 128.18) - /unlink 6191 (time 1776200996.88 total 50.02 last 115.10) - /unlink 7392 (time 1776201006.89 total 60.03 last 120.05) - /unlink 8518 (time 1776201016.89 total 70.03 last 112.55) - /unlink 9738 (time 1776201026.89 total 80.03 last 121.97) - /unlink 10000 (time 1776201029.55 total 82.69 last 98.79) - /unlink 11184 (time 1776201039.55 total 92.69 last 118.38) - /unlink 12421 (time 1776201049.56 total 102.70 last 123.56) - /unlink 13599 (time 1776201059.56 total 112.70 last 117.79) - /unlink 14799 (time 1776201069.56 total 122.70 last 119.95) - /unlink 15928 (time 1776201079.57 total 132.71 last 112.85) - /unlink 17283 (time 1776201089.57 total 142.71 last 135.45) - /unlink 18588 (time 1776201099.58 total 152.72 last 130.42) - /unlink 20000 (time 1776201108.21 total 161.35 last 163.54) - /unlink 21395 (time 1776201118.22 total 171.36 last 139.38) - /unlink 22663 (time 1776201128.22 total 181.36 last 126.78) - /unlink 23874 (time 1776201138.23 total 191.37 last 121.02) - /unlink 24969 (time 1776201148.24 total 201.38 last 109.42) - /unlink 26075 (time 1776201158.24 total 211.38 last 110.58) - /unlink 27494 (time 1776201168.24 total 221.38 last 141.86) - /unlink 28951 (time 1776201178.24 total 231.38 last 145.65) - /unlink 30000 (time 1776201185.97 total 239.11 last 135.69) - /unlink 31320 (time 1776201195.98 total 249.12 last 131.97) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 258.38 seconds: 126.15 ops/second Waiting for MDT destroys to complete - open/close 817 (time 1776201230.15 total 10.01 last 81.65) - open/close 1583 (time 1776201240.15 total 20.01 last 76.59) - open/close 2319 (time 1776201250.16 total 30.02 last 73.52) - open/close 3089 (time 1776201260.17 total 40.02 last 76.97) - open/close 4001 (time 1776201270.17 total 50.02 last 91.19) - open/close 4936 (time 1776201280.17 total 60.02 last 93.50) - open/close 5606 (time 1776201290.17 total 70.03 last 66.98) - open/close 6225 (time 1776201300.18 total 80.04 last 61.83) - open/close 7013 (time 1776201310.19 total 90.05 last 78.75) - open/close 7829 (time 1776201320.20 total 100.05 last 81.53) - open/close 8679 (time 1776201330.21 total 110.07 last 84.88) - open/close 9417 (time 1776201340.22 total 120.07 last 73.75) - open/close 10000 (time 1776201349.59 total 129.45 last 62.20) - open/close 10924 (time 1776201359.60 total 139.45 last 92.36) - open/close 11787 (time 1776201369.62 total 149.47 last 86.13) - open/close 12564 (time 1776201379.63 total 159.48 last 77.61) - open/close 13354 (time 1776201389.63 total 169.49 last 78.95) - open/close 14080 (time 1776201399.64 total 179.50 last 72.52) - open/close 14701 (time 1776201409.65 total 189.51 last 62.04) - open/close 15425 (time 1776201419.66 total 199.51 last 72.38) - open/close 16206 (time 1776201429.66 total 209.52 last 78.09) - open/close 16866 (time 1776201439.66 total 219.52 last 65.98) - open/close 17524 (time 1776201449.67 total 229.52 last 65.76) - open/close 18258 (time 1776201459.68 total 239.53 last 73.33) - open/close 19098 (time 1776201469.70 total 249.56 last 83.82) - open/close 19839 (time 1776201479.72 total 259.58 last 73.95) - open/close 20000 (time 1776201482.28 total 262.14 last 62.84) - open/close 20813 (time 1776201492.28 total 272.14 last 81.29) - open/close 21592 (time 1776201502.29 total 282.15 last 77.82) - open/close 22288 (time 1776201512.31 total 292.17 last 69.45) - open/close 22928 (time 1776201522.34 total 302.20 last 63.84) - open/close 23474 (time 1776201532.37 total 312.22 last 54.45) - open/close 23969 (time 1776201542.37 total 322.23 last 49.47) - open/close 24680 (time 1776201552.38 total 332.23 last 71.09) - open/close 25404 (time 1776201562.38 total 342.24 last 72.35) - open/close 26093 (time 1776201572.39 total 352.25 last 68.86) - open/close 26660 (time 1776201582.39 total 362.25 last 56.67) - open/close 27324 (time 1776201592.40 total 372.25 last 66.39) - open/close 27959 (time 1776201602.41 total 382.27 last 63.40) - open/close 28657 (time 1776201612.42 total 392.28 last 69.72) - open/close 29378 (time 1776201622.43 total 402.28 last 72.08) - open/close 30000 (time 1776201629.99 total 409.84 last 82.27) - open/close 30717 (time 1776201639.99 total 419.85 last 71.67) - open/close 31339 (time 1776201649.99 total 429.85 last 62.17) - open/close 31981 (time 1776201660.00 total 439.85 last 64.19) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 448.02 seconds: 72.75 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1243 (time 1776201682.79 total 10.01 last 124.19) - /unlink 2398 (time 1776201692.80 total 20.02 last 115.34) - /unlink 3502 (time 1776201702.80 total 30.02 last 110.39) - /unlink 4699 (time 1776201712.81 total 40.03 last 119.67) - /unlink 5895 (time 1776201722.81 total 50.03 last 119.51) - /unlink 7046 (time 1776201732.81 total 60.03 last 115.09) - /unlink 8236 (time 1776201742.81 total 70.03 last 119.00) - /unlink 9474 (time 1776201752.82 total 80.04 last 123.77) - /unlink 10000 (time 1776201758.01 total 85.23 last 101.35) - /unlink 11135 (time 1776201768.02 total 95.24 last 113.35) - /unlink 12364 (time 1776201778.02 total 105.24 last 122.88) - /unlink 13567 (time 1776201788.04 total 115.26 last 120.12) - /unlink 14883 (time 1776201798.04 total 125.26 last 131.59) - /unlink 16148 (time 1776201808.04 total 135.26 last 126.46) - /unlink 17126 (time 1776201818.04 total 145.26 last 97.76) - /unlink 18098 (time 1776201828.05 total 155.27 last 97.10) - /unlink 19453 (time 1776201838.05 total 165.27 last 135.49) - /unlink 20000 (time 1776201841.85 total 169.07 last 144.25) - /unlink 20945 (time 1776201851.85 total 179.07 last 94.46) - /unlink 22155 (time 1776201861.85 total 189.07 last 120.99) - /unlink 23330 (time 1776201871.86 total 199.08 last 117.41) - /unlink 24590 (time 1776201881.86 total 209.08 last 125.99) - /unlink 25769 (time 1776201891.88 total 219.10 last 117.61) - /unlink 27003 (time 1776201901.89 total 229.11 last 123.31) - /unlink 28280 (time 1776201911.89 total 239.11 last 127.67) - /unlink 29625 (time 1776201921.90 total 249.12 last 134.42) - /unlink 30000 (time 1776201924.89 total 252.11 last 125.34) - /unlink 31308 (time 1776201934.89 total 262.11 last 130.75) - /unlink 32569 (time 1776201944.90 total 272.12 last 126.06) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 272.38 seconds: 119.66 ops/second Waiting for MDT destroys to complete - open/close 603 (time 1776201974.58 total 10.02 last 60.19) - open/close 1243 (time 1776201984.59 total 20.02 last 63.98) - open/close 2000 (time 1776201994.59 total 30.03 last 75.67) - open/close 2757 (time 1776202004.60 total 40.03 last 75.65) - open/close 3462 (time 1776202014.60 total 50.03 last 70.48) - open/close 4209 (time 1776202024.60 total 60.04 last 74.68) - open/close 4848 (time 1776202034.62 total 70.05 last 63.80) - open/close 5560 (time 1776202044.62 total 80.06 last 71.19) - open/close 6373 (time 1776202054.62 total 90.06 last 81.29) - open/close 6942 (time 1776202064.62 total 100.06 last 56.90) - open/close 7622 (time 1776202074.66 total 110.10 last 67.74) - open/close 8226 (time 1776202084.67 total 120.10 last 60.36) - open/close 8898 (time 1776202094.67 total 130.11 last 67.16) - open/close 9581 (time 1776202104.68 total 140.11 last 68.27) - open/close 10000 (time 1776202110.36 total 145.79 last 73.81) - open/close 10760 (time 1776202120.36 total 155.79 last 75.99) - open/close 11568 (time 1776202130.36 total 165.80 last 80.75) - open/close 12205 (time 1776202140.38 total 175.81 last 63.62) - open/close 12653 (time 1776202150.38 total 185.81 last 44.80) - open/close 13248 (time 1776202160.39 total 195.82 last 59.44) - open/close 13919 (time 1776202170.39 total 205.83 last 67.05) - open/close 14650 (time 1776202180.40 total 215.84 last 73.04) - open/close 15269 (time 1776202190.44 total 225.87 last 61.69) - open/close 15966 (time 1776202200.46 total 235.89 last 69.56) - open/close 16574 (time 1776202210.48 total 245.91 last 60.68) - open/close 17261 (time 1776202220.49 total 255.92 last 68.63) - open/close 18040 (time 1776202230.50 total 265.93 last 77.82) - open/close 18766 (time 1776202240.51 total 275.95 last 72.50) - open/close 19518 (time 1776202250.53 total 285.96 last 75.08) - open/close 20000 (time 1776202257.57 total 293.01 last 68.43) - open/close 20678 (time 1776202267.58 total 303.02 last 67.73) - open/close 21382 (time 1776202277.59 total 313.03 last 70.33) - open/close 22110 (time 1776202287.60 total 323.04 last 72.72) - open/close 22871 (time 1776202297.60 total 333.04 last 76.09) - open/close 23610 (time 1776202307.61 total 343.05 last 73.83) - open/close 24368 (time 1776202317.64 total 353.07 last 75.62) - open/close 25057 (time 1776202327.65 total 363.08 last 68.81) - open/close 25729 (time 1776202337.66 total 373.10 last 67.11) - open/close 26500 (time 1776202347.67 total 383.11 last 77.03) - open/close 27270 (time 1776202357.68 total 393.11 last 76.94) - open/close 27999 (time 1776202367.68 total 403.12 last 72.88) - open/close 28715 (time 1776202377.69 total 413.12 last 71.57) - open/close 29400 (time 1776202387.69 total 423.13 last 68.46) - open/close 30000 (time 1776202397.38 total 432.82 last 61.92) - open/close 30684 (time 1776202407.39 total 442.82 last 68.36) - open/close 31325 (time 1776202417.41 total 452.85 last 63.94) - open/close 31982 (time 1776202427.44 total 462.87 last 65.56) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 472.49 seconds: 68.98 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1146 (time 1776202451.95 total 10.00 last 114.58) - /unlink 2203 (time 1776202461.95 total 20.01 last 105.62) - /unlink 3231 (time 1776202471.96 total 30.02 last 102.72) - /unlink 4181 (time 1776202481.97 total 40.03 last 94.91) - /unlink 5240 (time 1776202491.98 total 50.03 last 105.85) - /unlink 6421 (time 1776202501.98 total 60.04 last 118.04) - /unlink 7692 (time 1776202511.98 total 70.04 last 127.06) - /unlink 8865 (time 1776202521.99 total 80.04 last 117.28) - /unlink 10000 (time 1776202531.20 total 89.26 last 123.13) - /unlink 11285 (time 1776202541.21 total 99.26 last 128.45) - /unlink 12592 (time 1776202551.22 total 109.27 last 130.60) - /unlink 13648 (time 1776202561.22 total 119.27 last 105.57) - /unlink 14679 (time 1776202571.22 total 129.28 last 103.03) - /unlink 15716 (time 1776202581.23 total 139.29 last 103.62) - /unlink 16801 (time 1776202591.24 total 149.29 last 108.46) - /unlink 17861 (time 1776202601.24 total 159.30 last 105.95) - /unlink 18913 (time 1776202611.25 total 169.31 last 105.09) - /unlink 19999 (time 1776202621.26 total 179.32 last 108.48) - /unlink 21111 (time 1776202631.27 total 189.32 last 111.13) - /unlink 22293 (time 1776202641.27 total 199.33 last 118.14) - /unlink 23466 (time 1776202651.27 total 209.33 last 117.29) - /unlink 24887 (time 1776202661.28 total 219.33 last 142.05) - /unlink 26063 (time 1776202671.28 total 229.34 last 117.53) - /unlink 27465 (time 1776202681.28 total 239.34 last 140.20) - /unlink 28562 (time 1776202691.29 total 249.34 last 109.63) - /unlink 29712 (time 1776202701.29 total 259.35 last 114.97) - /unlink 30000 (time 1776202703.94 total 261.99 last 108.91) - /unlink 31262 (time 1776202713.94 total 271.99 last 126.15) - /unlink 32410 (time 1776202723.96 total 282.01 last 114.62) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 283.28 seconds: 115.06 ops/second Waiting for MDT destroys to complete - open/close 778 (time 1776202752.23 total 10.00 last 77.78) - open/close 1663 (time 1776202762.25 total 20.01 last 88.39) - open/close 2470 (time 1776202772.25 total 30.02 last 80.62) - open/close 3177 (time 1776202782.28 total 40.05 last 70.53) - open/close 3863 (time 1776202792.28 total 50.05 last 68.58) - open/close 4373 (time 1776202802.29 total 60.06 last 50.95) - open/close 5093 (time 1776202812.30 total 70.07 last 71.95) - open/close 5792 (time 1776202822.30 total 80.07 last 69.89) - open/close 6554 (time 1776202832.32 total 90.09 last 76.06) - open/close 7269 (time 1776202842.33 total 100.09 last 71.44) - open/close 8058 (time 1776202852.34 total 110.11 last 78.78) - open/close 8804 (time 1776202862.34 total 120.11 last 74.57) - open/close 9537 (time 1776202872.35 total 130.12 last 73.22) - open/close 10000 (time 1776202878.59 total 136.36 last 74.28) - open/close 10733 (time 1776202888.60 total 146.37 last 73.22) - open/close 11485 (time 1776202898.60 total 156.37 last 75.17) - open/close 12304 (time 1776202908.62 total 166.39 last 81.72) - open/close 13038 (time 1776202918.63 total 176.39 last 73.40) - open/close 13765 (time 1776202928.63 total 186.40 last 72.67) - open/close 14361 (time 1776202938.63 total 196.40 last 59.57) - open/close 15020 (time 1776202948.64 total 206.41 last 65.83) - open/close 15646 (time 1776202958.66 total 216.43 last 62.50) - open/close 16309 (time 1776202968.70 total 226.47 last 66.04) - open/close 17078 (time 1776202978.71 total 236.48 last 76.79) - open/close 17612 (time 1776202988.72 total 246.48 last 53.39) - open/close 18334 (time 1776202998.73 total 256.50 last 72.12) - open/close 19118 (time 1776203008.73 total 266.50 last 78.35) - open/close 19901 (time 1776203018.75 total 276.51 last 78.21) - open/close 20655 (time 1776203028.75 total 286.52 last 75.33) - open/close 21297 (time 1776203038.76 total 296.53 last 64.17) - open/close 21813 (time 1776203048.82 total 306.58 last 51.31) - open/close 22292 (time 1776203058.83 total 316.60 last 47.84) - open/close 23041 (time 1776203068.84 total 326.61 last 74.81) - open/close 23781 (time 1776203078.86 total 336.63 last 73.85) - open/close 24565 (time 1776203088.88 total 346.65 last 78.25) - open/close 25298 (time 1776203098.89 total 356.66 last 73.21) - open/close 25941 (time 1776203108.89 total 366.66 last 64.29) - open/close 26598 (time 1776203118.90 total 376.67 last 65.65) - open/close 27402 (time 1776203128.90 total 386.67 last 80.37) - open/close 28206 (time 1776203138.91 total 396.68 last 80.34) - open/close 28875 (time 1776203148.93 total 406.70 last 66.75) - open/close 29531 (time 1776203158.94 total 416.71 last 65.54) - open/close 30000 (time 1776203165.55 total 423.32 last 71.02) - open/close 30655 (time 1776203175.56 total 433.33 last 65.43) - open/close 31455 (time 1776203185.56 total 443.33 last 79.98) - open/close 32105 (time 1776203195.56 total 453.33 last 64.97) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 460.50 seconds: 70.78 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1341 (time 1776203217.72 total 10.00 last 134.06) - /unlink 2622 (time 1776203227.72 total 20.01 last 128.03) - /unlink 3927 (time 1776203237.73 total 30.01 last 130.43) - /unlink 5178 (time 1776203247.73 total 40.02 last 125.06) - /unlink 6606 (time 1776203257.73 total 50.02 last 142.78) - /unlink 7868 (time 1776203267.73 total 60.02 last 126.19) - /unlink 9228 (time 1776203277.74 total 70.02 last 135.92) - /unlink 10000 (time 1776203284.67 total 76.96 last 111.38) - /unlink 11078 (time 1776203294.68 total 86.97 last 107.66) - /unlink 12232 (time 1776203304.69 total 96.97 last 115.38) - /unlink 13647 (time 1776203314.70 total 106.98 last 141.35) - /unlink 15055 (time 1776203324.70 total 116.99 last 140.73) - /unlink 16379 (time 1776203334.70 total 126.99 last 132.38) - /unlink 17571 (time 1776203344.71 total 136.99 last 119.13) - /unlink 18673 (time 1776203354.71 total 147.00 last 110.14) - /unlink 19825 (time 1776203364.72 total 157.01 last 115.13) - /unlink 21209 (time 1776203374.73 total 167.01 last 138.26) - /unlink 22651 (time 1776203384.73 total 177.02 last 144.20) - /unlink 23760 (time 1776203394.73 total 187.02 last 110.88) - /unlink 24985 (time 1776203404.74 total 197.02 last 122.41) - /unlink 26095 (time 1776203414.76 total 207.05 last 110.76) - /unlink 27209 (time 1776203424.76 total 217.05 last 111.37) - /unlink 28334 (time 1776203434.77 total 227.05 last 112.48) - /unlink 29533 (time 1776203444.77 total 237.05 last 119.90) - /unlink 30000 (time 1776203448.77 total 241.05 last 116.68) - /unlink 31124 (time 1776203458.77 total 251.06 last 112.37) - /unlink 32271 (time 1776203468.78 total 261.07 last 114.58) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 263.84 seconds: 123.54 ops/second Waiting for MDT destroys to complete - open/close 702 (time 1776203500.93 total 10.00 last 70.17) - open/close 1462 (time 1776203510.94 total 20.01 last 75.93) - open/close 2212 (time 1776203520.96 total 30.03 last 74.90) - open/close 3058 (time 1776203530.96 total 40.03 last 84.59) - open/close 3767 (time 1776203540.97 total 50.04 last 70.84) - open/close 4536 (time 1776203550.97 total 60.04 last 76.88) - open/close 5340 (time 1776203560.97 total 70.04 last 80.38) - open/close 6120 (time 1776203570.98 total 80.05 last 77.96) - open/close 6925 (time 1776203580.98 total 90.05 last 80.47) - open/close 7659 (time 1776203590.99 total 100.06 last 73.35) - open/close 8415 (time 1776203601.01 total 110.08 last 75.40) - open/close 9225 (time 1776203611.04 total 120.11 last 80.78) - open/close 10000 (time 1776203620.40 total 129.47 last 82.80) - open/close 10804 (time 1776203630.40 total 139.48 last 80.37) - open/close 11564 (time 1776203640.41 total 149.48 last 75.97) - open/close 12240 (time 1776203650.41 total 159.48 last 67.58) - open/close 12930 (time 1776203660.41 total 169.48 last 68.98) - open/close 13552 (time 1776203670.43 total 179.50 last 62.13) - open/close 14213 (time 1776203680.43 total 189.50 last 66.05) - open/close 14766 (time 1776203690.44 total 199.51 last 55.23) - open/close 15414 (time 1776203700.44 total 209.52 last 64.79) - open/close 16312 (time 1776203710.45 total 219.52 last 89.76) - open/close 17166 (time 1776203720.46 total 229.53 last 85.33) - open/close 17871 (time 1776203730.46 total 239.53 last 70.48) - open/close 18558 (time 1776203740.47 total 249.54 last 68.63) - open/close 19216 (time 1776203750.48 total 259.55 last 65.75) - open/close 19903 (time 1776203760.49 total 269.56 last 68.65) - open/close 20715 (time 1776203770.49 total 279.56 last 81.17) - open/close 21466 (time 1776203780.50 total 289.57 last 75.06) - open/close 22286 (time 1776203790.50 total 299.57 last 81.93) - open/close 22972 (time 1776203800.51 total 309.58 last 68.57) - open/close 23726 (time 1776203810.51 total 319.59 last 75.36) - open/close 24354 (time 1776203820.52 total 329.59 last 62.79) - open/close 25168 (time 1776203830.53 total 339.60 last 81.28) - open/close 25958 (time 1776203840.54 total 349.61 last 78.91) - open/close 26723 (time 1776203850.55 total 359.62 last 76.42) - open/close 27491 (time 1776203860.57 total 369.64 last 76.67) - open/close 28210 (time 1776203870.57 total 379.64 last 71.88) - open/close 28946 (time 1776203880.58 total 389.65 last 73.53) - open/close 29585 (time 1776203890.59 total 399.66 last 63.85) - open/close 30000 (time 1776203896.78 total 405.85 last 67.08) - open/close 30711 (time 1776203906.78 total 415.85 last 71.08) - open/close 31446 (time 1776203916.78 total 425.85 last 73.49) - open/close 32197 (time 1776203926.78 total 435.85 last 75.10) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 440.45 seconds: 74.00 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1211 (time 1776203944.89 total 10.00 last 121.06) - /unlink 2309 (time 1776203954.90 total 20.01 last 109.73) - /unlink 3437 (time 1776203964.90 total 30.01 last 112.76) - /unlink 4626 (time 1776203974.90 total 40.02 last 118.84) - /unlink 5708 (time 1776203984.91 total 50.03 last 108.10) - /unlink 6827 (time 1776203994.91 total 60.03 last 111.90) - /unlink 7984 (time 1776204004.93 total 70.04 last 115.55) - /unlink 9098 (time 1776204014.93 total 80.04 last 111.37) - /unlink 10000 (time 1776204022.44 total 87.56 last 120.03) - /unlink 11145 (time 1776204032.45 total 97.56 last 114.42) - /unlink 12386 (time 1776204042.45 total 107.56 last 124.10) - /unlink 13621 (time 1776204052.45 total 117.57 last 123.50) - /unlink 14718 (time 1776204062.45 total 127.57 last 109.70) - /unlink 15908 (time 1776204072.45 total 137.57 last 119.00) - /unlink 17340 (time 1776204082.46 total 147.57 last 143.12) - /unlink 18487 (time 1776204092.46 total 157.57 last 114.66) - /unlink 19737 (time 1776204102.46 total 167.58 last 124.97) - /unlink 21080 (time 1776204112.47 total 177.58 last 134.24) - /unlink 22464 (time 1776204122.47 total 187.59 last 138.32) - /unlink 23852 (time 1776204132.48 total 197.59 last 138.76) - /unlink 25195 (time 1776204142.48 total 207.60 last 134.19) - /unlink 26507 (time 1776204152.50 total 217.61 last 131.03) - /unlink 27690 (time 1776204162.51 total 227.62 last 118.18) - /unlink 28860 (time 1776204172.51 total 237.62 last 116.98) - /unlink 30000 (time 1776204182.47 total 247.58 last 114.48) - /unlink 31142 (time 1776204192.47 total 257.58 last 114.15) - /unlink 32334 (time 1776204202.48 total 267.59 last 119.12) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 269.85 seconds: 120.78 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg216-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg216-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg216-server stop mds service on oleg216-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg216-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing unload_modules_local oleg216-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg216-server: LNET ready to unload oleg216-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-client: ssh exited with exit code 2 pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 PASS 106 (3905s) Stopping clients: oleg216-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg216-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg216-client: oleg216-server: ssh exited with exit code 2 oleg216-server: oleg216-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg216-server' oleg216-server: oleg216-server.virtnet: executing load_modules_local oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg216-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg216-server: Loading modules from /home/green/git/lustre-release/lustre oleg216-server: detected 4 online CPUs by sysfs oleg216-server: Force libcfs to create 2 CPU partitions oleg216-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg216-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4172 sec ========== 18:05:30 (1776204330)