-----============= acceptance-small: conf-sanity ============----- Tue Apr 14 14:31:39 EDT 2026 oleg229-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg229-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg229-server oleg229-server: oleg229-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg229-server: detected 4 online CPUs by sysfs oleg229-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg229-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg229-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg229-server: oleg229-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg229-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 14:35:26 (1776191726) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) oleg229-server: oleg229-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg229-server: detected 4 online CPUs by sysfs oleg229-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg229-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg229-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock oleg229-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 754 (time 1776191870.52 total 10.00 last 75.38) - open/close 1403 (time 1776191880.54 total 20.03 last 64.75) - open/close 2048 (time 1776191890.55 total 30.03 last 64.47) - open/close 2849 (time 1776191900.57 total 40.05 last 79.95) - open/close 3486 (time 1776191910.58 total 50.06 last 63.61) - open/close 4259 (time 1776191920.59 total 60.07 last 77.25) - open/close 5177 (time 1776191930.59 total 70.07 last 91.76) - open/close 6011 (time 1776191940.59 total 80.07 last 83.40) - open/close 6767 (time 1776191950.60 total 90.08 last 75.51) - open/close 7615 (time 1776191960.61 total 100.09 last 84.75) - open/close 8400 (time 1776191970.62 total 110.10 last 78.42) - open/close 9268 (time 1776191980.63 total 120.11 last 86.72) - open/close 10000 (time 1776191989.91 total 129.39 last 78.88) - open/close 10891 (time 1776191999.92 total 139.40 last 89.03) - open/close 11739 (time 1776192009.92 total 149.41 last 84.74) - open/close 12452 (time 1776192019.94 total 159.43 last 71.16) - open/close 13178 (time 1776192029.95 total 169.43 last 72.56) - open/close 13977 (time 1776192039.95 total 179.43 last 79.89) - open/close 14954 (time 1776192049.97 total 189.45 last 97.50) - open/close 15932 (time 1776192059.98 total 199.46 last 97.73) - open/close 16869 (time 1776192069.99 total 209.47 last 93.63) - open/close 17760 (time 1776192080.00 total 219.48 last 89.02) - open/close 18638 (time 1776192090.00 total 229.48 last 87.75) - open/close 19633 (time 1776192100.01 total 239.49 last 99.45) - open/close 20000 (time 1776192103.50 total 242.99 last 104.92) - open/close 20872 (time 1776192113.52 total 253.00 last 87.09) - open/close 21829 (time 1776192123.52 total 263.00 last 95.69) - open/close 22720 (time 1776192133.53 total 273.01 last 88.97) - open/close 23682 (time 1776192143.54 total 283.02 last 96.15) - open/close 24631 (time 1776192153.54 total 293.02 last 94.90) - open/close 25628 (time 1776192163.55 total 303.03 last 99.57) - open/close 26501 (time 1776192173.55 total 313.04 last 87.28) - open/close 27381 (time 1776192183.55 total 323.04 last 87.99) - open/close 28229 (time 1776192193.56 total 333.04 last 84.75) - open/close 29126 (time 1776192203.57 total 343.05 last 89.61) - open/close 30000 (time 1776192212.67 total 352.15 last 96.06) - open/close 30935 (time 1776192222.67 total 362.15 last 93.49) - open/close 31894 (time 1776192232.68 total 372.16 last 95.79) - open/close 32417 (time 1776192242.69 total 382.17 last 52.27) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 385.58 seconds: 84.53 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 965 (time 1776192262.29 total 10.00 last 96.46) - /unlink 2147 (time 1776192272.29 total 20.01 last 118.14) - /unlink 3240 (time 1776192282.29 total 30.01 last 109.27) - /unlink 4601 (time 1776192292.30 total 40.02 last 136.04) - /unlink 5844 (time 1776192302.30 total 50.02 last 124.24) - /unlink 6995 (time 1776192312.31 total 60.02 last 115.09) - /unlink 8191 (time 1776192322.31 total 70.03 last 119.54) - /unlink 9549 (time 1776192332.32 total 80.03 last 135.73) - /unlink 10000 (time 1776192335.63 total 83.35 last 135.87) - /unlink 11196 (time 1776192345.64 total 93.36 last 119.54) - /unlink 12697 (time 1776192355.64 total 103.36 last 150.09) - /unlink 14009 (time 1776192365.64 total 113.36 last 131.15) - /unlink 15371 (time 1776192375.65 total 123.37 last 136.14) - /unlink 16445 (time 1776192385.65 total 133.37 last 107.40) - /unlink 17915 (time 1776192395.65 total 143.37 last 146.98) - /unlink 19382 (time 1776192405.65 total 153.37 last 146.64) - /unlink 20000 (time 1776192410.00 total 157.72 last 142.10) - /unlink 21699 (time 1776192420.02 total 167.74 last 169.62) - /unlink 23266 (time 1776192430.03 total 177.75 last 156.58) - /unlink 24833 (time 1776192440.03 total 187.75 last 156.67) - /unlink 26357 (time 1776192450.04 total 197.76 last 152.23) - /unlink 28063 (time 1776192460.04 total 207.76 last 170.54) - /unlink 29455 (time 1776192470.05 total 217.77 last 139.09) - /unlink 30000 (time 1776192473.83 total 221.54 last 144.46) - /unlink 31579 (time 1776192483.83 total 231.55 last 157.82) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 238.06 seconds: 136.92 ops/second Waiting for MDT destroys to complete - open/close 796 (time 1776192518.53 total 10.00 last 79.59) - open/close 1708 (time 1776192528.53 total 20.01 last 91.15) - open/close 2661 (time 1776192538.55 total 30.03 last 95.11) - open/close 3649 (time 1776192548.56 total 40.04 last 98.68) - open/close 4341 (time 1776192558.58 total 50.06 last 69.09) - open/close 4988 (time 1776192568.59 total 60.07 last 64.63) - open/close 5951 (time 1776192578.60 total 70.07 last 96.23) - open/close 6867 (time 1776192588.61 total 80.09 last 91.45) - open/close 7845 (time 1776192598.62 total 90.09 last 97.79) - open/close 8874 (time 1776192608.62 total 100.09 last 102.86) - open/close 9932 (time 1776192618.62 total 110.09 last 105.79) - open/close 10716 (time 1776192628.63 total 120.10 last 78.35) - open/close 11659 (time 1776192638.63 total 130.11 last 94.26) - open/close 12359 (time 1776192648.65 total 140.13 last 69.83) - open/close 13287 (time 1776192658.66 total 150.13 last 92.79) - open/close 14180 (time 1776192668.66 total 160.14 last 89.25) - open/close 14942 (time 1776192678.69 total 170.17 last 75.95) - open/close 15713 (time 1776192688.71 total 180.18 last 77.02) - open/close 16406 (time 1776192698.71 total 190.18 last 69.29) - open/close 17408 (time 1776192708.71 total 200.19 last 100.13) - open/close 18175 (time 1776192718.72 total 210.20 last 76.63) - open/close 18695 (time 1776192728.73 total 220.21 last 51.95) - open/close 19405 (time 1776192738.75 total 230.22 last 70.89) - open/close 20000 (time 1776192745.84 total 237.31 last 83.92) - open/close 20871 (time 1776192755.85 total 247.32 last 87.02) - open/close 21684 (time 1776192765.85 total 257.33 last 81.24) - open/close 22660 (time 1776192775.86 total 267.33 last 97.58) - open/close 23438 (time 1776192785.86 total 277.33 last 77.78) - open/close 24051 (time 1776192795.87 total 287.34 last 61.24) - open/close 24739 (time 1776192805.88 total 297.36 last 68.71) - open/close 25368 (time 1776192815.90 total 307.38 last 62.75) - open/close 25942 (time 1776192825.91 total 317.38 last 57.38) - open/close 26787 (time 1776192835.92 total 327.39 last 84.43) - open/close 27728 (time 1776192845.92 total 337.39 last 94.07) - open/close 28568 (time 1776192855.92 total 347.40 last 83.97) - open/close 29383 (time 1776192865.92 total 357.40 last 81.49) - open/close 30000 (time 1776192871.86 total 363.33 last 104.00) - open/close 30861 (time 1776192881.87 total 373.35 last 85.97) - open/close 31748 (time 1776192891.88 total 383.36 last 88.63) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 390.87 seconds: 83.39 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1270 (time 1776192913.78 total 10.01 last 126.88) - /unlink 2701 (time 1776192923.78 total 20.01 last 143.08) - /unlink 4166 (time 1776192933.78 total 30.02 last 146.42) - /unlink 5542 (time 1776192943.79 total 40.02 last 137.58) - /unlink 7380 (time 1776192953.79 total 50.02 last 183.76) - /unlink 8626 (time 1776192963.79 total 60.02 last 124.59) - /unlink 10000 (time 1776192970.34 total 66.57 last 209.85) - /unlink 11229 (time 1776192980.34 total 76.57 last 122.89) - /unlink 12599 (time 1776192990.36 total 86.59 last 136.72) - /unlink 13894 (time 1776193000.36 total 96.60 last 129.41) - /unlink 15434 (time 1776193010.37 total 106.60 last 153.94) - /unlink 16970 (time 1776193020.37 total 116.60 last 153.56) - /unlink 18464 (time 1776193030.37 total 126.61 last 149.36) - /unlink 20000 (time 1776193040.24 total 136.47 last 155.73) - /unlink 21567 (time 1776193050.24 total 146.47 last 156.64) - /unlink 22925 (time 1776193060.24 total 156.47 last 135.79) - /unlink 24092 (time 1776193070.24 total 166.47 last 116.70) - /unlink 25577 (time 1776193080.24 total 176.48 last 148.46) - /unlink 27224 (time 1776193090.25 total 186.48 last 164.64) - /unlink 28670 (time 1776193100.25 total 196.48 last 144.59) - /unlink 30000 (time 1776193108.38 total 204.61 last 163.62) - /unlink 31588 (time 1776193118.38 total 214.61 last 158.77) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 221.95 seconds: 146.86 ops/second Waiting for MDT destroys to complete - open/close 646 (time 1776193155.37 total 10.00 last 64.57) - open/close 1481 (time 1776193165.38 total 20.01 last 83.46) - open/close 2393 (time 1776193175.38 total 30.01 last 91.20) - open/close 3309 (time 1776193185.38 total 40.01 last 91.55) - open/close 4180 (time 1776193195.39 total 50.02 last 87.06) - open/close 5044 (time 1776193205.39 total 60.02 last 86.37) - open/close 5993 (time 1776193215.40 total 70.03 last 94.88) - open/close 6799 (time 1776193225.40 total 80.03 last 80.59) - open/close 7808 (time 1776193235.40 total 90.03 last 100.90) - open/close 8738 (time 1776193245.41 total 100.04 last 92.87) - open/close 9597 (time 1776193255.41 total 110.04 last 85.89) - open/close 10000 (time 1776193259.45 total 114.08 last 99.72) - open/close 10746 (time 1776193269.47 total 124.10 last 74.47) - open/close 11610 (time 1776193279.48 total 134.11 last 86.31) - open/close 12386 (time 1776193289.48 total 144.11 last 77.60) - open/close 13260 (time 1776193299.48 total 154.11 last 87.37) - open/close 14091 (time 1776193309.49 total 164.12 last 83.08) - open/close 14913 (time 1776193319.49 total 174.12 last 82.14) - open/close 15758 (time 1776193329.52 total 184.15 last 84.24) - open/close 16535 (time 1776193339.53 total 194.16 last 77.67) - open/close 17333 (time 1776193349.54 total 204.17 last 79.72) - open/close 18182 (time 1776193359.55 total 214.19 last 84.76) - open/close 19035 (time 1776193369.56 total 224.19 last 85.27) - open/close 19752 (time 1776193379.56 total 234.19 last 71.66) - open/close 20000 (time 1776193382.73 total 237.36 last 78.23) - open/close 20751 (time 1776193392.74 total 247.37 last 75.06) - open/close 21528 (time 1776193402.75 total 257.38 last 77.64) - open/close 22320 (time 1776193412.76 total 267.39 last 79.07) - open/close 23199 (time 1776193422.77 total 277.40 last 87.82) - open/close 24007 (time 1776193432.78 total 287.41 last 80.70) - open/close 24844 (time 1776193442.80 total 297.43 last 83.54) - open/close 25708 (time 1776193452.82 total 307.45 last 86.29) - open/close 26619 (time 1776193462.82 total 317.46 last 91.01) - open/close 27594 (time 1776193472.84 total 327.47 last 97.39) - open/close 28609 (time 1776193482.84 total 337.47 last 101.46) - open/close 29529 (time 1776193492.86 total 347.49 last 91.79) - open/close 30000 (time 1776193498.54 total 353.17 last 83.01) - open/close 30849 (time 1776193508.54 total 363.17 last 84.84) - open/close 31885 (time 1776193518.55 total 373.18 last 103.51) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 381.44 seconds: 85.45 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1429 (time 1776193540.37 total 10.01 last 142.74) - /unlink 2652 (time 1776193550.37 total 20.01 last 122.27) - /unlink 4069 (time 1776193560.38 total 30.02 last 141.66) - /unlink 5546 (time 1776193570.38 total 40.02 last 147.68) - /unlink 6910 (time 1776193580.38 total 50.02 last 136.37) - /unlink 8523 (time 1776193590.38 total 60.02 last 161.23) - /unlink 9594 (time 1776193600.39 total 70.03 last 107.07) - /unlink 10000 (time 1776193603.50 total 73.14 last 130.34) - /unlink 11456 (time 1776193613.51 total 83.15 last 145.42) - /unlink 13040 (time 1776193623.52 total 93.16 last 158.31) - /unlink 14415 (time 1776193633.55 total 103.19 last 137.13) - /unlink 15926 (time 1776193643.55 total 113.19 last 151.03) - /unlink 17318 (time 1776193653.55 total 123.19 last 139.15) - /unlink 18457 (time 1776193663.56 total 133.20 last 113.79) - /unlink 19853 (time 1776193673.57 total 143.21 last 139.57) - /unlink 21224 (time 1776193683.57 total 153.21 last 137.01) - /unlink 22763 (time 1776193693.57 total 163.21 last 153.90) - /unlink 24067 (time 1776193703.57 total 173.21 last 130.38) - /unlink 25509 (time 1776193713.58 total 183.22 last 144.19) - /unlink 27083 (time 1776193723.58 total 193.22 last 157.33) - /unlink 28454 (time 1776193733.59 total 203.23 last 137.00) - /unlink 29813 (time 1776193743.59 total 213.23 last 135.83) - /unlink 31223 (time 1776193753.60 total 223.24 last 140.85) - /unlink 32473 (time 1776193763.61 total 233.25 last 124.94) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 233.96 seconds: 139.31 ops/second Waiting for MDT destroys to complete - open/close 978 (time 1776193792.05 total 10.01 last 97.75) - open/close 1814 (time 1776193802.05 total 20.01 last 83.56) - open/close 2730 (time 1776193812.06 total 30.02 last 91.52) - open/close 3655 (time 1776193822.07 total 40.03 last 92.44) - open/close 4610 (time 1776193832.07 total 50.03 last 95.48) - open/close 5467 (time 1776193842.08 total 60.04 last 85.63) - open/close 6284 (time 1776193852.08 total 70.04 last 81.67) - open/close 7288 (time 1776193862.08 total 80.04 last 100.39) - open/close 8208 (time 1776193872.09 total 90.05 last 91.94) - open/close 9071 (time 1776193882.11 total 100.07 last 86.13) - open/close 9791 (time 1776193892.12 total 110.07 last 71.96) - open/close 10000 (time 1776193894.60 total 112.56 last 84.20) - open/close 10850 (time 1776193904.60 total 122.56 last 84.97) - open/close 11736 (time 1776193914.61 total 132.57 last 88.54) - open/close 12641 (time 1776193924.61 total 142.57 last 90.49) - open/close 13562 (time 1776193934.61 total 152.57 last 92.06) - open/close 14488 (time 1776193944.62 total 162.58 last 92.49) - open/close 15333 (time 1776193954.63 total 172.59 last 84.46) - open/close 16185 (time 1776193964.64 total 182.60 last 85.09) - open/close 16920 (time 1776193974.64 total 192.60 last 73.48) - open/close 17796 (time 1776193984.65 total 202.61 last 87.55) - open/close 18684 (time 1776193994.65 total 212.61 last 88.78) - open/close 19623 (time 1776194004.65 total 222.61 last 93.90) - open/close 20000 (time 1776194008.15 total 226.11 last 107.83) - open/close 21020 (time 1776194018.15 total 236.11 last 101.98) - open/close 21957 (time 1776194028.16 total 246.12 last 93.62) - open/close 22811 (time 1776194038.18 total 256.14 last 85.20) - open/close 23519 (time 1776194048.19 total 266.15 last 70.77) - open/close 24363 (time 1776194058.19 total 276.15 last 84.38) - open/close 25227 (time 1776194068.20 total 286.15 last 86.35) - open/close 26100 (time 1776194078.20 total 296.16 last 87.26) - open/close 27021 (time 1776194088.21 total 306.16 last 92.04) - open/close 27862 (time 1776194098.21 total 316.17 last 84.04) - open/close 28673 (time 1776194108.22 total 326.18 last 81.04) - open/close 29411 (time 1776194118.23 total 336.18 last 73.76) - open/close 30000 (time 1776194124.71 total 342.67 last 90.78) - open/close 30772 (time 1776194134.72 total 352.67 last 77.18) - open/close 31660 (time 1776194144.72 total 362.68 last 88.79) - open/close 32499 (time 1776194154.72 total 372.68 last 83.90) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 373.62 seconds: 87.24 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1385 (time 1776194170.22 total 10.00 last 138.49) - /unlink 2685 (time 1776194180.22 total 20.00 last 130.00) - /unlink 4052 (time 1776194190.23 total 30.01 last 136.62) - /unlink 5421 (time 1776194200.23 total 40.01 last 136.87) - /unlink 6920 (time 1776194210.23 total 50.01 last 149.82) - /unlink 8427 (time 1776194220.24 total 60.02 last 150.65) - /unlink 9963 (time 1776194230.24 total 70.02 last 153.55) - /unlink 11372 (time 1776194240.25 total 80.03 last 140.78) - /unlink 12769 (time 1776194250.25 total 90.03 last 139.69) - /unlink 14092 (time 1776194260.25 total 100.03 last 132.24) - /unlink 15315 (time 1776194270.26 total 110.04 last 122.20) - /unlink 16521 (time 1776194280.26 total 120.04 last 120.60) - /unlink 17996 (time 1776194290.26 total 130.05 last 147.46) - /unlink 19472 (time 1776194300.27 total 140.05 last 147.53) - /unlink 20000 (time 1776194303.68 total 143.46 last 154.87) - /unlink 21493 (time 1776194313.68 total 153.46 last 149.28) - /unlink 22823 (time 1776194323.68 total 163.46 last 132.95) - /unlink 24226 (time 1776194333.69 total 173.47 last 140.22) - /unlink 25456 (time 1776194343.69 total 183.47 last 123.00) - /unlink 26801 (time 1776194353.70 total 193.48 last 134.39) - /unlink 28000 (time 1776194363.70 total 203.48 last 119.88) - /unlink 29515 (time 1776194373.70 total 213.48 last 151.47) - /unlink 30000 (time 1776194377.34 total 217.12 last 133.39) - /unlink 31425 (time 1776194387.34 total 227.12 last 142.40) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 234.87 seconds: 138.78 ops/second Waiting for MDT destroys to complete - open/close 942 (time 1776194421.26 total 10.01 last 94.12) - open/close 1844 (time 1776194431.27 total 20.01 last 90.18) - open/close 2806 (time 1776194441.27 total 30.02 last 96.13) - open/close 3674 (time 1776194451.29 total 40.03 last 86.69) - open/close 4642 (time 1776194461.30 total 50.05 last 96.64) - open/close 5437 (time 1776194471.31 total 60.05 last 79.45) - open/close 6356 (time 1776194481.31 total 70.06 last 91.87) - open/close 7198 (time 1776194491.32 total 80.06 last 84.16) - open/close 8040 (time 1776194501.33 total 90.07 last 84.12) - open/close 8923 (time 1776194511.34 total 100.08 last 88.20) - open/close 9748 (time 1776194521.34 total 110.08 last 82.50) - open/close 10000 (time 1776194524.31 total 113.06 last 84.79) - open/close 10738 (time 1776194534.31 total 123.06 last 73.77) - open/close 11601 (time 1776194544.33 total 133.08 last 86.13) - open/close 12512 (time 1776194554.36 total 143.10 last 90.91) - open/close 13350 (time 1776194564.36 total 153.10 last 83.79) - open/close 14239 (time 1776194574.37 total 163.11 last 88.79) - open/close 15085 (time 1776194584.38 total 173.12 last 84.54) - open/close 15974 (time 1776194594.38 total 183.13 last 88.83) - open/close 16943 (time 1776194604.39 total 193.13 last 96.87) - open/close 17907 (time 1776194614.40 total 203.14 last 96.28) - open/close 18783 (time 1776194624.41 total 213.15 last 87.52) - open/close 19626 (time 1776194634.42 total 223.16 last 84.23) - open/close 20000 (time 1776194639.52 total 228.27 last 73.20) - open/close 20621 (time 1776194649.53 total 238.28 last 62.06) - open/close 21467 (time 1776194659.53 total 248.28 last 84.56) - open/close 22363 (time 1776194669.56 total 258.30 last 89.41) - open/close 23269 (time 1776194679.56 total 268.30 last 90.58) - open/close 23914 (time 1776194689.57 total 278.31 last 64.45) - open/close 24767 (time 1776194699.57 total 288.32 last 85.23) - open/close 25792 (time 1776194709.58 total 298.32 last 102.49) - open/close 26757 (time 1776194719.58 total 308.33 last 96.44) - open/close 27612 (time 1776194729.59 total 318.33 last 85.46) - open/close 28416 (time 1776194739.59 total 328.33 last 80.37) - open/close 29318 (time 1776194749.60 total 338.34 last 90.11) - open/close 30000 (time 1776194756.75 total 345.49 last 95.41) - open/close 30952 (time 1776194766.75 total 355.49 last 95.18) - open/close 31877 (time 1776194776.76 total 365.50 last 92.42) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 372.91 seconds: 87.40 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1690 (time 1776194797.96 total 10.00 last 168.99) - /unlink 3160 (time 1776194807.96 total 20.00 last 146.99) - /unlink 4736 (time 1776194817.96 total 30.00 last 157.60) - /unlink 6104 (time 1776194827.96 total 40.00 last 136.78) - /unlink 7551 (time 1776194837.96 total 50.00 last 144.70) - /unlink 9030 (time 1776194847.97 total 60.01 last 147.80) - /unlink 10000 (time 1776194855.68 total 67.72 last 125.84) - /unlink 11443 (time 1776194865.68 total 77.72 last 144.29) - /unlink 12969 (time 1776194875.68 total 87.72 last 152.57) - /unlink 14188 (time 1776194885.68 total 97.72 last 121.85) - /unlink 15552 (time 1776194895.68 total 107.73 last 136.37) - /unlink 17077 (time 1776194905.69 total 117.73 last 152.38) - /unlink 18378 (time 1776194915.69 total 127.74 last 130.09) - /unlink 19664 (time 1776194925.69 total 137.74 last 128.60) - /unlink 20000 (time 1776194928.07 total 140.11 last 141.40) - /unlink 21146 (time 1776194938.07 total 150.12 last 114.56) - /unlink 22301 (time 1776194948.08 total 160.12 last 115.44) - /unlink 23704 (time 1776194958.08 total 170.12 last 140.25) - /unlink 25057 (time 1776194968.08 total 180.12 last 135.30) - /unlink 26456 (time 1776194978.09 total 190.13 last 139.79) - /unlink 27733 (time 1776194988.09 total 200.14 last 127.66) - /unlink 29153 (time 1776194998.10 total 210.14 last 141.87) - /unlink 30000 (time 1776195004.13 total 216.17 last 140.45) - /unlink 31511 (time 1776195014.14 total 226.19 last 150.92) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 235.05 seconds: 138.67 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg229-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local oleg229-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg229-server: LNET ready to unload oleg229-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 PASS 106 (3357s) Stopping clients: oleg229-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg229-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 oleg229-server: oleg229-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg229-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg229-server: detected 4 online CPUs by sysfs oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3659 sec ========== 15:32:40 (1776195160)