-----============= acceptance-small: conf-sanity ============----- Wed Apr 15 11:07:13 EDT 2026 oleg447-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg447-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg447-server oleg447-server: oleg447-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg447-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg447-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg447-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg447-server: oleg447-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg447-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg447-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:10:50 (1776265850) Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) oleg447-server: oleg447-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg447-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg447-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg447-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg447-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg447-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg447-server: oleg447-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg447-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg447-client.virtnet: -o user_xattr,flock oleg447-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 564 (time 1776266004.80 total 10.03 last 56.25) - open/close 1181 (time 1776266014.82 total 20.04 last 61.60) - open/close 1703 (time 1776266024.82 total 30.05 last 52.16) - open/close 2406 (time 1776266034.82 total 40.05 last 70.30) - open/close 3001 (time 1776266044.83 total 50.06 last 59.46) - open/close 3451 (time 1776266054.84 total 60.07 last 44.95) - open/close 3987 (time 1776266064.86 total 70.08 last 53.51) - open/close 4585 (time 1776266074.86 total 80.09 last 59.78) - open/close 5123 (time 1776266084.89 total 90.12 last 53.64) - open/close 5707 (time 1776266094.89 total 100.12 last 58.38) - open/close 6377 (time 1776266104.90 total 110.12 last 66.99) - open/close 6950 (time 1776266114.91 total 120.13 last 57.24) - open/close 7593 (time 1776266124.91 total 130.14 last 64.25) - open/close 8228 (time 1776266134.92 total 140.15 last 63.43) - open/close 8821 (time 1776266144.93 total 150.16 last 59.26) - open/close 9446 (time 1776266154.94 total 160.17 last 62.43) - open/close 10000 (time 1776266163.14 total 168.36 last 67.59) - open/close 10609 (time 1776266173.15 total 178.37 last 60.85) - open/close 11281 (time 1776266183.16 total 188.38 last 67.14) - open/close 11882 (time 1776266193.16 total 198.38 last 60.09) - open/close 12456 (time 1776266203.17 total 208.39 last 57.34) - open/close 13105 (time 1776266213.17 total 218.39 last 64.90) - open/close 13785 (time 1776266223.18 total 228.41 last 67.89) - open/close 14383 (time 1776266233.19 total 238.41 last 59.78) - open/close 15030 (time 1776266243.19 total 248.41 last 64.70) - open/close 15673 (time 1776266253.19 total 258.42 last 64.29) - open/close 16435 (time 1776266263.21 total 268.44 last 76.05) - open/close 17054 (time 1776266273.24 total 278.47 last 61.70) - open/close 17645 (time 1776266283.25 total 288.48 last 59.05) - open/close 18321 (time 1776266293.26 total 298.49 last 67.50) - open/close 18963 (time 1776266303.27 total 308.50 last 64.13) - open/close 19595 (time 1776266313.29 total 318.52 last 63.09) - open/close 20000 (time 1776266319.26 total 324.48 last 67.92) - open/close 20711 (time 1776266329.26 total 334.49 last 71.05) - open/close 21369 (time 1776266339.29 total 344.51 last 65.64) - open/close 21983 (time 1776266349.29 total 354.52 last 61.37) - open/close 22606 (time 1776266359.29 total 364.52 last 62.29) - open/close 23297 (time 1776266369.30 total 374.53 last 69.03) - open/close 23992 (time 1776266379.31 total 384.54 last 69.45) - open/close 24601 (time 1776266389.32 total 394.54 last 60.86) - open/close 25177 (time 1776266399.33 total 404.56 last 57.52) - open/close 25748 (time 1776266409.35 total 414.57 last 57.02) - open/close 26353 (time 1776266419.35 total 424.57 last 60.48) - open/close 27006 (time 1776266429.36 total 434.58 last 65.26) - open/close 27656 (time 1776266439.36 total 444.58 last 64.98) - open/close 28279 (time 1776266449.37 total 454.59 last 62.26) - open/close 28866 (time 1776266459.37 total 464.60 last 58.65) - open/close 29401 (time 1776266469.38 total 474.61 last 53.45) - open/close 29929 (time 1776266479.40 total 484.63 last 52.72) - open/close 30465 (time 1776266489.40 total 494.63 last 53.57) - open/close 31037 (time 1776266499.43 total 504.65 last 57.07) - open/close 31578 (time 1776266509.45 total 514.67 last 54.00) - open/close 32155 (time 1776266519.45 total 524.68 last 57.65) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 532.12 seconds: 61.25 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1094 (time 1776266542.72 total 10.01 last 109.30) - /unlink 2144 (time 1776266552.73 total 20.01 last 104.99) - /unlink 3186 (time 1776266562.73 total 30.01 last 104.16) - /unlink 4217 (time 1776266572.74 total 40.02 last 102.99) - /unlink 5134 (time 1776266582.75 total 50.03 last 91.65) - /unlink 6130 (time 1776266592.75 total 60.04 last 99.53) - /unlink 7123 (time 1776266602.76 total 70.04 last 99.24) - /unlink 8249 (time 1776266612.76 total 80.04 last 112.59) - /unlink 9376 (time 1776266622.76 total 90.05 last 112.64) - /unlink 10000 (time 1776266628.58 total 95.86 last 107.30) - /unlink 10997 (time 1776266638.58 total 105.87 last 99.67) - /unlink 12113 (time 1776266648.59 total 115.87 last 111.56) - /unlink 13225 (time 1776266658.59 total 125.88 last 111.11) - /unlink 14184 (time 1776266668.61 total 135.89 last 95.78) - /unlink 15206 (time 1776266678.61 total 145.89 last 102.18) - /unlink 16113 (time 1776266688.62 total 155.90 last 90.61) - /unlink 17124 (time 1776266698.63 total 165.91 last 101.00) - /unlink 18139 (time 1776266708.64 total 175.92 last 101.42) - /unlink 19210 (time 1776266718.66 total 185.94 last 106.91) - /unlink 20000 (time 1776266725.98 total 193.26 last 107.88) - /unlink 21098 (time 1776266735.98 total 203.26 last 109.79) - /unlink 22181 (time 1776266745.98 total 213.27 last 108.28) - /unlink 23312 (time 1776266755.99 total 223.27 last 113.03) - /unlink 24366 (time 1776266765.99 total 233.28 last 105.34) - /unlink 25430 (time 1776266776.00 total 243.28 last 106.36) - /unlink 26554 (time 1776266786.00 total 253.28 last 112.37) - /unlink 27490 (time 1776266796.01 total 263.29 last 93.52) - /unlink 28201 (time 1776266806.01 total 273.29 last 71.09) - /unlink 29126 (time 1776266816.02 total 283.30 last 92.44) - /unlink 30000 (time 1776266825.44 total 292.73 last 92.69) - /unlink 31096 (time 1776266835.45 total 302.74 last 109.51) - /unlink 32128 (time 1776266845.45 total 312.74 last 103.19) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 316.97 seconds: 102.83 ops/second Waiting for MDT destroys to complete - open/close 571 (time 1776266881.33 total 10.01 last 57.06) - open/close 1238 (time 1776266891.35 total 20.03 last 66.56) - open/close 1806 (time 1776266901.36 total 30.03 last 56.78) - open/close 2389 (time 1776266911.37 total 40.05 last 58.21) - open/close 3055 (time 1776266921.38 total 50.06 last 66.54) - open/close 3644 (time 1776266931.38 total 60.06 last 58.89) - open/close 4243 (time 1776266941.40 total 70.08 last 59.78) - open/close 4893 (time 1776266951.41 total 80.09 last 64.94) - open/close 5588 (time 1776266961.42 total 90.09 last 69.47) - open/close 6107 (time 1776266971.42 total 100.09 last 51.90) - open/close 6732 (time 1776266981.45 total 110.13 last 62.31) - open/close 7328 (time 1776266991.45 total 120.13 last 59.58) - open/close 7858 (time 1776267001.45 total 130.13 last 52.98) - open/close 8492 (time 1776267011.46 total 140.13 last 63.39) - open/close 9173 (time 1776267021.46 total 150.14 last 68.08) - open/close 9856 (time 1776267031.47 total 160.15 last 68.23) - open/close 10000 (time 1776267033.88 total 162.56 last 59.72) - open/close 10723 (time 1776267043.88 total 172.56 last 72.29) - open/close 11376 (time 1776267053.89 total 182.57 last 65.26) - open/close 12016 (time 1776267063.90 total 192.58 last 63.93) - open/close 12522 (time 1776267073.90 total 202.58 last 50.57) - open/close 13021 (time 1776267083.92 total 212.60 last 49.83) - open/close 13507 (time 1776267093.93 total 222.60 last 48.56) - open/close 14139 (time 1776267103.93 total 232.61 last 63.18) - open/close 14679 (time 1776267113.94 total 242.61 last 53.98) - open/close 15236 (time 1776267123.94 total 252.62 last 55.68) - open/close 15838 (time 1776267133.95 total 262.62 last 60.16) - open/close 16410 (time 1776267143.96 total 272.64 last 57.09) - open/close 17001 (time 1776267153.97 total 282.64 last 59.09) - open/close 17533 (time 1776267163.99 total 292.66 last 53.09) - open/close 18042 (time 1776267173.99 total 302.67 last 50.88) - open/close 18545 (time 1776267184.00 total 312.68 last 50.26) - open/close 19087 (time 1776267194.00 total 322.68 last 54.20) - open/close 19642 (time 1776267204.02 total 332.69 last 55.41) - open/close 20000 (time 1776267209.91 total 338.58 last 60.78) - open/close 20592 (time 1776267219.93 total 348.61 last 59.06) - open/close 21149 (time 1776267229.94 total 358.62 last 55.64) - open/close 21782 (time 1776267239.95 total 368.63 last 63.21) - open/close 22400 (time 1776267249.97 total 378.64 last 61.73) - open/close 22961 (time 1776267259.98 total 388.65 last 56.04) - open/close 23491 (time 1776267269.99 total 398.67 last 52.91) - open/close 24007 (time 1776267280.01 total 408.68 last 51.54) - open/close 24566 (time 1776267290.01 total 418.69 last 55.88) - open/close 25091 (time 1776267300.02 total 428.70 last 52.45) - open/close 25631 (time 1776267310.02 total 438.70 last 53.99) - open/close 26249 (time 1776267320.03 total 448.70 last 61.77) - open/close 26807 (time 1776267330.03 total 458.71 last 55.77) - open/close 27440 (time 1776267340.03 total 468.71 last 63.27) - open/close 28135 (time 1776267350.05 total 478.73 last 69.37) - open/close 28821 (time 1776267360.06 total 488.74 last 68.57) - open/close 29390 (time 1776267370.06 total 498.74 last 56.86) - open/close 29989 (time 1776267380.07 total 508.75 last 59.87) - open/close 30597 (time 1776267390.08 total 518.76 last 60.72) - open/close 31099 (time 1776267400.09 total 528.77 last 50.16) - open/close 31615 (time 1776267410.10 total 538.77 last 51.57) - open/close 32105 (time 1776267420.11 total 548.78 last 48.96) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 557.61 seconds: 58.45 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1204 (time 1776267443.98 total 10.00 last 120.38) - /unlink 2313 (time 1776267453.99 total 20.00 last 110.86) - /unlink 3225 (time 1776267463.99 total 30.01 last 91.17) - /unlink 4260 (time 1776267473.99 total 40.01 last 103.49) - /unlink 5207 (time 1776267483.99 total 50.01 last 94.67) - /unlink 6139 (time 1776267494.00 total 60.01 last 93.19) - /unlink 7020 (time 1776267504.00 total 70.02 last 88.03) - /unlink 7838 (time 1776267514.01 total 80.03 last 81.75) - /unlink 8637 (time 1776267524.01 total 90.03 last 79.88) - /unlink 9439 (time 1776267534.02 total 100.04 last 80.13) - /unlink 10000 (time 1776267541.95 total 107.97 last 70.74) - /unlink 11017 (time 1776267551.96 total 117.97 last 101.65) - /unlink 11981 (time 1776267561.96 total 127.98 last 96.34) - /unlink 13008 (time 1776267571.96 total 137.98 last 102.68) - /unlink 14166 (time 1776267581.97 total 147.99 last 115.73) - /unlink 15338 (time 1776267591.98 total 158.00 last 117.07) - /unlink 16445 (time 1776267601.98 total 168.00 last 110.67) - /unlink 17640 (time 1776267611.99 total 178.00 last 119.48) - /unlink 18722 (time 1776267622.00 total 188.01 last 108.08) - /unlink 19827 (time 1776267632.00 total 198.02 last 110.43) - /unlink 20787 (time 1776267642.01 total 208.03 last 95.92) - /unlink 21864 (time 1776267652.01 total 218.03 last 107.66) - /unlink 22933 (time 1776267662.03 total 228.04 last 106.78) - /unlink 24048 (time 1776267672.03 total 238.05 last 111.46) - /unlink 25186 (time 1776267682.03 total 248.05 last 113.79) - /unlink 26331 (time 1776267692.03 total 258.05 last 114.47) - /unlink 27471 (time 1776267702.04 total 268.06 last 113.94) - /unlink 28512 (time 1776267712.04 total 278.06 last 104.08) - /unlink 29693 (time 1776267722.05 total 288.07 last 117.98) - /unlink 30000 (time 1776267724.79 total 290.81 last 112.16) - /unlink 30905 (time 1776267734.79 total 300.81 last 90.47) - /unlink 32002 (time 1776267744.79 total 310.81 last 109.68) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 316.18 seconds: 103.09 ops/second Waiting for MDT destroys to complete - open/close 695 (time 1776267774.28 total 10.03 last 69.30) - open/close 1410 (time 1776267784.29 total 20.03 last 71.48) - open/close 2038 (time 1776267794.30 total 30.04 last 62.74) - open/close 2645 (time 1776267804.30 total 40.04 last 60.69) - open/close 3188 (time 1776267814.31 total 50.05 last 54.26) - open/close 3730 (time 1776267824.33 total 60.07 last 54.09) - open/close 4198 (time 1776267834.34 total 70.08 last 46.74) - open/close 4703 (time 1776267844.35 total 80.09 last 50.46) - open/close 5248 (time 1776267854.35 total 90.10 last 54.46) - open/close 5803 (time 1776267864.36 total 100.11 last 55.46) - open/close 6340 (time 1776267874.36 total 110.11 last 53.69) - open/close 6857 (time 1776267884.39 total 120.14 last 51.55) - open/close 7368 (time 1776267894.39 total 130.14 last 51.09) - open/close 7874 (time 1776267904.40 total 140.15 last 50.54) - open/close 8356 (time 1776267914.41 total 150.15 last 48.19) - open/close 8839 (time 1776267924.42 total 160.17 last 48.23) - open/close 9313 (time 1776267934.44 total 170.18 last 47.31) - open/close 9716 (time 1776267944.46 total 180.21 last 40.20) - open/close 10000 (time 1776267949.77 total 185.52 last 53.48) - open/close 10515 (time 1776267959.79 total 195.53 last 51.43) - open/close 10993 (time 1776267969.79 total 205.54 last 47.77) - open/close 11597 (time 1776267979.80 total 215.55 last 60.35) - open/close 12138 (time 1776267989.82 total 225.56 last 54.03) - open/close 12761 (time 1776267999.82 total 235.57 last 62.27) - open/close 13291 (time 1776268009.83 total 245.58 last 52.95) - open/close 13813 (time 1776268019.84 total 255.58 last 52.16) - open/close 14416 (time 1776268029.85 total 265.59 last 60.25) - open/close 15083 (time 1776268039.85 total 275.59 last 66.69) - open/close 15678 (time 1776268049.88 total 285.62 last 59.33) - open/close 16299 (time 1776268059.88 total 295.62 last 62.10) - open/close 16862 (time 1776268069.89 total 305.63 last 56.25) - open/close 17451 (time 1776268079.90 total 315.64 last 58.84) - open/close 18057 (time 1776268089.92 total 325.66 last 60.48) - open/close 18618 (time 1776268099.93 total 335.67 last 56.03) - open/close 19139 (time 1776268109.94 total 345.69 last 52.03) - open/close 19739 (time 1776268119.95 total 355.69 last 59.97) - open/close 20000 (time 1776268124.08 total 359.82 last 63.16) - open/close 20549 (time 1776268134.08 total 369.83 last 54.87) - open/close 21098 (time 1776268144.09 total 379.83 last 54.86) - open/close 21730 (time 1776268154.11 total 389.85 last 63.08) - open/close 22270 (time 1776268164.12 total 399.86 last 53.95) - open/close 22819 (time 1776268174.14 total 409.88 last 54.80) - open/close 23272 (time 1776268184.16 total 419.90 last 45.21) - open/close 23793 (time 1776268194.17 total 429.91 last 52.03) - open/close 24304 (time 1776268204.17 total 439.92 last 51.09) - open/close 24883 (time 1776268214.18 total 449.93 last 57.83) - open/close 25498 (time 1776268224.20 total 459.94 last 61.41) - open/close 26045 (time 1776268234.20 total 469.95 last 54.68) - open/close 26693 (time 1776268244.21 total 479.95 last 64.74) - open/close 27373 (time 1776268254.22 total 489.96 last 67.96) - open/close 27901 (time 1776268264.23 total 499.98 last 52.72) - open/close 28462 (time 1776268274.24 total 509.98 last 56.05) - open/close 29037 (time 1776268284.24 total 519.99 last 57.49) - open/close 29669 (time 1776268294.25 total 529.99 last 63.15) - open/close 30000 (time 1776268299.07 total 534.81 last 68.68) - open/close 30630 (time 1776268309.07 total 544.82 last 62.97) - open/close 31265 (time 1776268319.08 total 554.83 last 63.44) - open/close 31670 (time 1776268329.12 total 564.87 last 40.34) - open/close 32122 (time 1776268339.12 total 574.87 last 45.19) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 583.11 seconds: 55.90 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 907 (time 1776268362.98 total 10.01 last 90.60) - /unlink 1900 (time 1776268372.98 total 20.01 last 99.29) - /unlink 2851 (time 1776268382.98 total 30.01 last 95.09) - /unlink 3805 (time 1776268392.99 total 40.02 last 95.34) - /unlink 4889 (time 1776268402.99 total 50.02 last 108.38) - /unlink 5802 (time 1776268413.00 total 60.03 last 91.22) - /unlink 6775 (time 1776268423.01 total 70.04 last 97.23) - /unlink 7723 (time 1776268433.01 total 80.04 last 94.78) - /unlink 8633 (time 1776268443.01 total 90.04 last 90.96) - /unlink 9668 (time 1776268453.02 total 100.05 last 103.42) - /unlink 10000 (time 1776268456.19 total 103.22 last 104.75) - /unlink 11087 (time 1776268466.20 total 113.22 last 108.65) - /unlink 12031 (time 1776268476.20 total 123.23 last 94.37) - /unlink 13095 (time 1776268486.20 total 133.23 last 106.39) - /unlink 14224 (time 1776268496.20 total 143.23 last 112.89) - /unlink 15306 (time 1776268506.21 total 153.24 last 108.14) - /unlink 16314 (time 1776268516.21 total 163.24 last 100.78) - /unlink 17351 (time 1776268526.21 total 173.24 last 103.69) - /unlink 18368 (time 1776268536.22 total 183.25 last 101.57) - /unlink 19410 (time 1776268546.23 total 193.26 last 104.13) - /unlink 20000 (time 1776268551.74 total 198.77 last 107.04) - /unlink 21118 (time 1776268561.75 total 208.78 last 111.74) - /unlink 22128 (time 1776268571.75 total 218.78 last 100.96) - /unlink 23098 (time 1776268581.75 total 228.78 last 96.96) - /unlink 24074 (time 1776268591.76 total 238.78 last 97.59) - /unlink 25008 (time 1776268601.76 total 248.79 last 93.39) - /unlink 25887 (time 1776268611.78 total 258.81 last 87.69) - /unlink 26610 (time 1776268621.80 total 268.83 last 72.14) - /unlink 27385 (time 1776268631.80 total 278.83 last 77.50) - /unlink 28190 (time 1776268641.81 total 288.83 last 80.47) - /unlink 29222 (time 1776268651.81 total 298.84 last 103.16) - /unlink 30000 (time 1776268659.81 total 306.83 last 97.31) - /unlink 30952 (time 1776268669.81 total 316.84 last 95.15) - /unlink 31967 (time 1776268679.81 total 326.84 last 101.49) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 334.65 seconds: 97.40 ops/second Waiting for MDT destroys to complete - open/close 489 (time 1776268714.38 total 10.02 last 48.82) - open/close 944 (time 1776268724.39 total 20.02 last 45.47) - open/close 1378 (time 1776268734.40 total 30.03 last 43.38) - open/close 1929 (time 1776268744.42 total 40.05 last 54.96) - open/close 2478 (time 1776268754.43 total 50.07 last 54.83) - open/close 3031 (time 1776268764.44 total 60.07 last 55.27) - open/close 3640 (time 1776268774.45 total 70.08 last 60.86) - open/close 4180 (time 1776268784.47 total 80.10 last 53.87) - open/close 4769 (time 1776268794.48 total 90.11 last 58.84) - open/close 5382 (time 1776268804.49 total 100.12 last 61.27) - open/close 5913 (time 1776268814.49 total 110.12 last 53.07) - open/close 6392 (time 1776268824.50 total 120.13 last 47.85) - open/close 6982 (time 1776268834.50 total 130.13 last 58.99) - open/close 7518 (time 1776268844.52 total 140.15 last 53.53) - open/close 8083 (time 1776268854.52 total 150.15 last 56.48) - open/close 8569 (time 1776268864.52 total 160.15 last 48.58) - open/close 9143 (time 1776268874.53 total 170.16 last 57.37) - open/close 9823 (time 1776268884.53 total 180.17 last 67.96) - open/close 10000 (time 1776268887.11 total 182.75 last 68.62) - open/close 10722 (time 1776268897.14 total 192.77 last 72.05) - open/close 11334 (time 1776268907.15 total 202.78 last 61.12) - open/close 11991 (time 1776268917.16 total 212.79 last 65.65) - open/close 12654 (time 1776268927.18 total 222.82 last 66.11) - open/close 13272 (time 1776268937.19 total 232.82 last 61.75) - open/close 13875 (time 1776268947.22 total 242.85 last 60.13) - open/close 14451 (time 1776268957.23 total 252.86 last 57.56) - open/close 15111 (time 1776268967.24 total 262.87 last 65.90) - open/close 15760 (time 1776268977.25 total 272.88 last 64.84) - open/close 16412 (time 1776268987.26 total 282.89 last 65.14) - open/close 17091 (time 1776268997.27 total 292.90 last 67.85) - open/close 17627 (time 1776269007.28 total 302.91 last 53.53) - open/close 18090 (time 1776269017.31 total 312.94 last 46.18) - open/close 18719 (time 1776269027.32 total 322.95 last 62.81) - open/close 19251 (time 1776269037.32 total 332.95 last 53.19) - open/close 19675 (time 1776269047.33 total 342.96 last 42.38) - open/close 20000 (time 1776269055.83 total 351.46 last 38.23) - open/close 20589 (time 1776269065.83 total 361.46 last 58.89) - open/close 21160 (time 1776269075.83 total 371.46 last 57.08) - open/close 21825 (time 1776269085.84 total 381.47 last 66.44) - open/close 22482 (time 1776269095.85 total 391.48 last 65.65) - open/close 23118 (time 1776269105.87 total 401.50 last 63.50) - open/close 23655 (time 1776269115.88 total 411.51 last 53.63) - open/close 24268 (time 1776269125.89 total 421.52 last 61.22) - open/close 24883 (time 1776269135.90 total 431.53 last 61.44) - open/close 25553 (time 1776269145.92 total 441.55 last 66.87) - open/close 26158 (time 1776269155.94 total 451.57 last 60.37) - open/close 26778 (time 1776269165.95 total 461.58 last 61.95) - open/close 27482 (time 1776269175.96 total 471.59 last 70.31) - open/close 28171 (time 1776269185.97 total 481.60 last 68.83) - open/close 28745 (time 1776269195.98 total 491.61 last 57.36) - open/close 29403 (time 1776269206.00 total 501.63 last 65.67) - open/close 30000 (time 1776269215.10 total 510.73 last 65.63) - open/close 30581 (time 1776269225.10 total 520.73 last 58.07) - open/close 31146 (time 1776269235.11 total 530.74 last 56.45) - open/close 31664 (time 1776269245.13 total 540.76 last 51.69) - open/close 32196 (time 1776269255.14 total 550.77 last 53.14) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 559.10 seconds: 58.30 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1028 (time 1776269279.16 total 10.00 last 102.76) - /unlink 2082 (time 1776269289.16 total 20.00 last 105.39) - /unlink 3112 (time 1776269299.17 total 30.01 last 102.96) - /unlink 4112 (time 1776269309.18 total 40.02 last 99.92) - /unlink 5055 (time 1776269319.18 total 50.02 last 94.26) - /unlink 5981 (time 1776269329.18 total 60.02 last 92.59) - /unlink 6981 (time 1776269339.19 total 70.03 last 99.90) - /unlink 8043 (time 1776269349.20 total 80.04 last 106.09) - /unlink 9185 (time 1776269359.20 total 90.04 last 114.18) - /unlink 10000 (time 1776269366.64 total 97.48 last 109.66) - /unlink 11103 (time 1776269376.64 total 107.48 last 110.23) - /unlink 12089 (time 1776269386.65 total 117.49 last 98.56) - /unlink 13155 (time 1776269396.65 total 127.49 last 106.59) - /unlink 14295 (time 1776269406.65 total 137.49 last 113.97) - /unlink 15180 (time 1776269416.65 total 147.49 last 88.48) - /unlink 16133 (time 1776269426.66 total 157.50 last 95.23) - /unlink 17077 (time 1776269436.67 total 167.51 last 94.34) - /unlink 18113 (time 1776269446.67 total 177.51 last 103.55) - /unlink 19161 (time 1776269456.68 total 187.52 last 104.74) - /unlink 20000 (time 1776269464.14 total 194.98 last 112.46) - /unlink 21104 (time 1776269474.14 total 204.98 last 110.32) - /unlink 22105 (time 1776269484.15 total 214.99 last 100.07) - /unlink 23094 (time 1776269494.15 total 224.99 last 98.90) - /unlink 24108 (time 1776269504.15 total 234.99 last 101.36) - /unlink 25172 (time 1776269514.16 total 245.00 last 106.33) - /unlink 26245 (time 1776269524.16 total 255.00 last 107.29) - /unlink 27272 (time 1776269534.16 total 265.00 last 102.67) - /unlink 28222 (time 1776269544.17 total 275.01 last 94.92) - /unlink 29171 (time 1776269554.18 total 285.02 last 94.82) - /unlink 30000 (time 1776269562.03 total 292.87 last 105.60) - /unlink 31002 (time 1776269572.03 total 302.87 last 100.18) - /unlink 31958 (time 1776269582.04 total 312.88 last 95.56) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 319.42 seconds: 102.04 ops/second Waiting for MDT destroys to complete - open/close 544 (time 1776269613.04 total 10.00 last 54.39) - open/close 1124 (time 1776269623.05 total 20.01 last 57.94) - open/close 1738 (time 1776269633.06 total 30.02 last 61.34) - open/close 2333 (time 1776269643.08 total 40.04 last 59.38) - open/close 2974 (time 1776269653.10 total 50.06 last 64.00) - open/close 3598 (time 1776269663.10 total 60.06 last 62.37) - open/close 4176 (time 1776269673.12 total 70.08 last 57.70) - open/close 4773 (time 1776269683.13 total 80.09 last 59.65) - open/close 5377 (time 1776269693.14 total 90.10 last 60.31) - open/close 5983 (time 1776269703.15 total 100.11 last 60.57) - open/close 6524 (time 1776269713.16 total 110.12 last 54.06) - open/close 7162 (time 1776269723.18 total 120.14 last 63.68) - open/close 7784 (time 1776269733.20 total 130.16 last 62.07) - open/close 8375 (time 1776269743.20 total 140.16 last 59.09) - open/close 9044 (time 1776269753.21 total 150.17 last 66.82) - open/close 9603 (time 1776269763.22 total 160.18 last 55.85) - open/close 10000 (time 1776269769.03 total 165.99 last 68.37) - open/close 10539 (time 1776269779.04 total 176.00 last 53.80) - open/close 11110 (time 1776269789.06 total 186.02 last 57.02) - open/close 11741 (time 1776269799.07 total 196.03 last 63.05) - open/close 12344 (time 1776269809.08 total 206.04 last 60.22) - open/close 12902 (time 1776269819.09 total 216.05 last 55.75) - open/close 13512 (time 1776269829.09 total 226.05 last 60.99) - open/close 14089 (time 1776269839.09 total 236.05 last 57.69) - open/close 14614 (time 1776269849.11 total 246.07 last 52.41) - open/close 15203 (time 1776269859.11 total 256.07 last 58.86) - open/close 15848 (time 1776269869.12 total 266.08 last 64.49) - open/close 16473 (time 1776269879.13 total 276.09 last 62.42) - open/close 17083 (time 1776269889.16 total 286.12 last 60.81) - open/close 17735 (time 1776269899.17 total 296.13 last 65.12) - open/close 18321 (time 1776269909.18 total 306.14 last 58.57) - open/close 18906 (time 1776269919.20 total 316.16 last 58.38) - open/close 19488 (time 1776269929.21 total 326.17 last 58.14) - open/close 20000 (time 1776269938.32 total 335.28 last 56.17) - open/close 20635 (time 1776269948.32 total 345.28 last 63.49) - open/close 21276 (time 1776269958.34 total 355.30 last 64.00) - open/close 21898 (time 1776269968.34 total 365.30 last 62.20) - open/close 22482 (time 1776269978.35 total 375.31 last 58.36) - open/close 23143 (time 1776269988.36 total 385.32 last 66.05) - open/close 23760 (time 1776269998.37 total 395.33 last 61.63) - open/close 24394 (time 1776270008.37 total 405.33 last 63.37) - open/close 25068 (time 1776270018.38 total 415.34 last 67.36) - open/close 25720 (time 1776270028.38 total 425.34 last 65.18) - open/close 26389 (time 1776270038.38 total 435.34 last 66.89) - open/close 27018 (time 1776270048.38 total 445.34 last 62.89) - open/close 27618 (time 1776270058.40 total 455.36 last 59.91) - open/close 28296 (time 1776270068.41 total 465.37 last 67.75) - open/close 28944 (time 1776270078.41 total 475.37 last 64.76) - open/close 29548 (time 1776270088.41 total 485.37 last 60.39) - open/close 30000 (time 1776270095.26 total 492.22 last 66.07) - open/close 30649 (time 1776270105.27 total 502.23 last 64.84) - open/close 31323 (time 1776270115.27 total 512.23 last 67.38) - open/close 32023 (time 1776270125.27 total 522.23 last 69.95) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 529.96 seconds: 61.50 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1013 (time 1776270146.69 total 10.00 last 101.27) - /unlink 2097 (time 1776270156.70 total 20.01 last 108.36) - /unlink 3158 (time 1776270166.70 total 30.01 last 106.08) - /unlink 4354 (time 1776270176.70 total 40.01 last 119.58) - /unlink 5553 (time 1776270186.70 total 50.01 last 119.85) - /unlink 6686 (time 1776270196.71 total 60.02 last 113.26) - /unlink 7900 (time 1776270206.71 total 70.02 last 121.32) - /unlink 9305 (time 1776270216.72 total 80.03 last 140.44) - /unlink 10000 (time 1776270222.07 total 85.38 last 129.89) - /unlink 11328 (time 1776270232.07 total 95.38 last 132.74) - /unlink 12591 (time 1776270242.08 total 105.39 last 126.18) - /unlink 13769 (time 1776270252.09 total 115.40 last 117.76) - /unlink 14945 (time 1776270262.11 total 125.42 last 117.36) - /unlink 16106 (time 1776270272.11 total 135.42 last 116.08) - /unlink 17397 (time 1776270282.11 total 145.42 last 129.07) - /unlink 18543 (time 1776270292.11 total 155.42 last 114.55) - /unlink 19694 (time 1776270302.12 total 165.43 last 115.02) - /unlink 20000 (time 1776270304.40 total 167.71 last 134.29) - /unlink 21231 (time 1776270314.41 total 177.72 last 122.93) - /unlink 22456 (time 1776270324.41 total 187.73 last 122.48) - /unlink 23749 (time 1776270334.42 total 197.73 last 129.24) - /unlink 24994 (time 1776270344.43 total 207.74 last 124.39) - /unlink 26378 (time 1776270354.43 total 217.74 last 138.31) - /unlink 27629 (time 1776270364.44 total 227.75 last 125.04) - /unlink 28900 (time 1776270374.44 total 237.75 last 127.10) - /unlink 30000 (time 1776270383.09 total 246.40 last 127.15) - /unlink 31249 (time 1776270393.09 total 256.40 last 124.88) - /unlink 32421 (time 1776270403.10 total 266.41 last 117.12) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 268.02 seconds: 121.61 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg447-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg447-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg447-server stop mds service on oleg447-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg447-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing unload_modules_local oleg447-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg447-server: LNET ready to unload oleg447-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-client: ssh exited with exit code 2 pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 PASS 106 (4616s) Stopping clients: oleg447-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg447-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg447-client: oleg447-server: ssh exited with exit code 2 oleg447-server: oleg447-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg447-server' oleg447-server: oleg447-server.virtnet: executing load_modules_local oleg447-server: Loading modules from /home/green/git/lustre-release/lustre oleg447-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg447-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg447-server: detected 4 online CPUs by sysfs oleg447-server: Force libcfs to create 2 CPU partitions oleg447-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg447-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4897 sec ========== 12:28:51 (1776270531)