-----============= acceptance-small: conf-sanity ============----- Tue Apr 14 17:15:02 EDT 2026 oleg433-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg433-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg433-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg433-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg433-server oleg433-server: oleg433-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: detected 4 online CPUs by sysfs oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg433-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg433-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg433-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg433-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg433-server: oleg433-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg433-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg433-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:18:22 (1776201502) Stopping clients: oleg433-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg433-client.virtnet /mnt/lustre2 (opts:-f) oleg433-server: oleg433-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg433-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg433-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg433-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg433-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg433-server: oleg433-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg433-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg433-client.virtnet: -o user_xattr,flock oleg433-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 600 (time 1776201640.51 total 10.00 last 59.97) - open/close 1242 (time 1776201650.52 total 20.01 last 64.17) - open/close 1822 (time 1776201660.52 total 30.01 last 57.99) - open/close 2472 (time 1776201670.53 total 40.02 last 64.94) - open/close 3136 (time 1776201680.53 total 50.03 last 66.36) - open/close 3774 (time 1776201690.54 total 60.03 last 63.77) - open/close 4364 (time 1776201700.55 total 70.05 last 58.91) - open/close 5025 (time 1776201710.56 total 80.05 last 66.08) - open/close 5670 (time 1776201720.57 total 90.06 last 64.43) - open/close 6292 (time 1776201730.58 total 100.07 last 62.15) - open/close 6903 (time 1776201740.59 total 110.08 last 61.02) - open/close 7464 (time 1776201750.59 total 120.08 last 56.09) - open/close 8115 (time 1776201760.60 total 130.09 last 65.04) - open/close 8740 (time 1776201770.61 total 140.10 last 62.44) - open/close 9259 (time 1776201780.63 total 150.13 last 51.79) - open/close 9755 (time 1776201790.65 total 160.15 last 49.49) - open/close 10000 (time 1776201794.49 total 163.98 last 63.92) - open/close 10559 (time 1776201804.49 total 173.99 last 55.86) - open/close 11178 (time 1776201814.50 total 184.00 last 61.84) - open/close 11775 (time 1776201824.52 total 194.01 last 59.61) - open/close 12400 (time 1776201834.53 total 204.03 last 62.42) - open/close 13036 (time 1776201844.55 total 214.04 last 63.51) - open/close 13710 (time 1776201854.55 total 224.04 last 67.39) - open/close 14377 (time 1776201864.56 total 234.05 last 66.62) - open/close 14969 (time 1776201874.57 total 244.06 last 59.16) - open/close 15594 (time 1776201884.58 total 254.07 last 62.44) - open/close 16196 (time 1776201894.58 total 264.08 last 60.15) - open/close 16809 (time 1776201904.59 total 274.09 last 61.25) - open/close 17402 (time 1776201914.59 total 284.09 last 59.30) - open/close 18019 (time 1776201924.60 total 294.10 last 61.64) - open/close 18653 (time 1776201934.60 total 304.10 last 63.39) - open/close 19257 (time 1776201944.61 total 314.10 last 60.36) - open/close 19950 (time 1776201954.62 total 324.11 last 69.24) - open/close 20549 (time 1776201964.62 total 334.12 last 59.89) - open/close 21135 (time 1776201974.64 total 344.13 last 58.52) - open/close 21828 (time 1776201984.64 total 354.14 last 69.26) - open/close 22534 (time 1776201994.65 total 364.14 last 70.55) - open/close 23205 (time 1776202004.66 total 374.15 last 67.05) - open/close 23860 (time 1776202014.67 total 384.16 last 65.44) - open/close 24495 (time 1776202024.67 total 394.16 last 63.50) - open/close 25163 (time 1776202034.67 total 404.17 last 66.77) - open/close 25838 (time 1776202044.68 total 414.18 last 67.43) - open/close 26429 (time 1776202054.69 total 424.19 last 59.03) - open/close 27046 (time 1776202064.70 total 434.19 last 61.67) - open/close 27544 (time 1776202074.71 total 444.20 last 49.77) - open/close 28179 (time 1776202084.71 total 454.20 last 63.47) - open/close 28865 (time 1776202094.72 total 464.21 last 68.54) - open/close 29529 (time 1776202104.73 total 474.22 last 66.36) - open/close 30000 (time 1776202111.88 total 481.37 last 65.86) - open/close 30647 (time 1776202121.88 total 491.37 last 64.68) - open/close 31240 (time 1776202131.89 total 501.39 last 59.23) - open/close 31812 (time 1776202141.90 total 511.40 last 57.13) - open/close 32471 (time 1776202151.91 total 521.40 last 65.86) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 523.03 seconds: 62.32 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1052 (time 1776202168.78 total 10.01 last 105.10) - /unlink 2218 (time 1776202178.78 total 20.02 last 116.53) - /unlink 3307 (time 1776202188.80 total 30.03 last 108.77) - /unlink 4470 (time 1776202198.81 total 40.05 last 116.09) - /unlink 5471 (time 1776202208.83 total 50.06 last 99.98) - /unlink 6633 (time 1776202218.83 total 60.06 last 116.17) - /unlink 7709 (time 1776202228.83 total 70.06 last 107.57) - /unlink 8870 (time 1776202238.84 total 80.07 last 116.00) - /unlink 9898 (time 1776202248.85 total 90.08 last 102.68) - /unlink 10813 (time 1776202258.86 total 100.09 last 91.41) - /unlink 11906 (time 1776202268.86 total 110.10 last 109.27) - /unlink 12879 (time 1776202278.87 total 120.10 last 97.28) - /unlink 13952 (time 1776202288.88 total 130.11 last 107.13) - /unlink 15075 (time 1776202298.89 total 140.12 last 112.26) - /unlink 16129 (time 1776202308.89 total 150.12 last 105.39) - /unlink 17245 (time 1776202318.89 total 160.12 last 111.55) - /unlink 18289 (time 1776202328.89 total 170.12 last 104.40) - /unlink 19295 (time 1776202338.90 total 180.13 last 100.55) - /unlink 20000 (time 1776202345.28 total 186.51 last 110.46) - /unlink 21082 (time 1776202355.29 total 196.52 last 108.14) - /unlink 22239 (time 1776202365.29 total 206.52 last 115.64) - /unlink 23585 (time 1776202375.30 total 216.53 last 134.52) - /unlink 24762 (time 1776202385.30 total 226.53 last 117.65) - /unlink 25874 (time 1776202395.30 total 236.54 last 111.16) - /unlink 26956 (time 1776202405.31 total 246.55 last 108.09) - /unlink 28070 (time 1776202415.32 total 256.55 last 111.36) - /unlink 29154 (time 1776202425.32 total 266.55 last 108.38) - /unlink 30000 (time 1776202433.25 total 274.48 last 106.70) - /unlink 30963 (time 1776202443.25 total 284.48 last 96.25) - /unlink 31993 (time 1776202453.26 total 294.49 last 102.96) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 299.77 seconds: 108.73 ops/second Waiting for MDT destroys to complete - open/close 647 (time 1776202485.97 total 10.01 last 64.66) - open/close 1260 (time 1776202495.97 total 20.01 last 61.27) - open/close 1912 (time 1776202505.98 total 30.02 last 65.14) - open/close 2505 (time 1776202516.00 total 40.03 last 59.22) - open/close 3104 (time 1776202526.00 total 50.04 last 59.86) - open/close 3678 (time 1776202536.01 total 60.04 last 57.38) - open/close 4298 (time 1776202546.01 total 70.05 last 61.99) - open/close 4963 (time 1776202556.02 total 80.05 last 66.45) - open/close 5514 (time 1776202566.02 total 90.06 last 55.08) - open/close 6164 (time 1776202576.03 total 100.06 last 64.97) - open/close 6828 (time 1776202586.04 total 110.07 last 66.32) - open/close 7506 (time 1776202596.05 total 120.09 last 67.70) - open/close 8144 (time 1776202606.08 total 130.11 last 63.65) - open/close 8787 (time 1776202616.08 total 140.11 last 64.30) - open/close 9449 (time 1776202626.08 total 150.11 last 66.19) - open/close 10000 (time 1776202635.01 total 159.05 last 61.68) - open/close 10639 (time 1776202645.02 total 169.05 last 63.87) - open/close 11273 (time 1776202655.04 total 179.08 last 63.25) - open/close 11929 (time 1776202665.05 total 189.09 last 65.52) - open/close 12638 (time 1776202675.06 total 199.10 last 70.85) - open/close 13352 (time 1776202685.08 total 209.11 last 71.26) - open/close 14057 (time 1776202695.08 total 219.12 last 70.47) - open/close 14817 (time 1776202705.09 total 229.13 last 75.92) - open/close 15517 (time 1776202715.11 total 239.15 last 69.85) - open/close 16170 (time 1776202725.12 total 249.15 last 65.28) - open/close 16855 (time 1776202735.13 total 259.16 last 68.43) - open/close 17512 (time 1776202745.14 total 269.18 last 65.61) - open/close 18197 (time 1776202755.15 total 279.19 last 68.43) - open/close 18894 (time 1776202765.15 total 289.19 last 69.70) - open/close 19545 (time 1776202775.16 total 299.20 last 65.03) - open/close 20000 (time 1776202782.22 total 306.26 last 64.44) - open/close 20630 (time 1776202792.24 total 316.27 last 62.91) - open/close 21278 (time 1776202802.27 total 326.30 last 64.61) - open/close 21871 (time 1776202812.27 total 336.31 last 59.26) - open/close 22597 (time 1776202822.28 total 346.31 last 72.57) - open/close 23280 (time 1776202832.30 total 356.33 last 68.17) - open/close 23941 (time 1776202842.30 total 366.34 last 66.06) - open/close 24610 (time 1776202852.32 total 376.35 last 66.81) - open/close 25267 (time 1776202862.32 total 386.36 last 65.65) - open/close 25915 (time 1776202872.35 total 396.38 last 64.67) - open/close 26586 (time 1776202882.35 total 406.39 last 67.04) - open/close 27299 (time 1776202892.36 total 416.39 last 71.28) - open/close 27965 (time 1776202902.36 total 426.40 last 66.56) - open/close 28477 (time 1776202912.37 total 436.40 last 51.18) - open/close 29162 (time 1776202922.37 total 446.41 last 68.44) - open/close 29816 (time 1776202932.38 total 456.42 last 65.36) - open/close 30000 (time 1776202935.28 total 459.32 last 63.38) - open/close 30699 (time 1776202945.29 total 469.33 last 69.85) - open/close 31334 (time 1776202955.31 total 479.34 last 63.40) - open/close 32009 (time 1776202965.32 total 489.36 last 67.40) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 499.28 seconds: 65.28 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1180 (time 1776202990.12 total 10.00 last 117.95) - /unlink 2350 (time 1776203000.12 total 20.00 last 116.99) - /unlink 3483 (time 1776203010.13 total 30.01 last 113.24) - /unlink 4649 (time 1776203020.13 total 40.01 last 116.58) - /unlink 5740 (time 1776203030.13 total 50.02 last 109.04) - /unlink 6846 (time 1776203040.14 total 60.02 last 110.55) - /unlink 8008 (time 1776203050.14 total 70.03 last 116.15) - /unlink 9170 (time 1776203060.14 total 80.03 last 116.17) - /unlink 10000 (time 1776203067.94 total 87.82 last 106.47) - /unlink 11181 (time 1776203077.94 total 97.83 last 118.08) - /unlink 12259 (time 1776203087.95 total 107.83 last 107.76) - /unlink 13295 (time 1776203097.95 total 117.84 last 103.55) - /unlink 14261 (time 1776203107.95 total 127.84 last 96.60) - /unlink 15135 (time 1776203117.95 total 137.84 last 87.38) - /unlink 16275 (time 1776203127.95 total 147.84 last 113.99) - /unlink 17363 (time 1776203137.95 total 157.84 last 108.79) - /unlink 18515 (time 1776203147.95 total 167.84 last 115.19) - /unlink 19558 (time 1776203157.96 total 177.85 last 104.22) - /unlink 20000 (time 1776203161.98 total 181.87 last 109.90) - /unlink 21016 (time 1776203171.99 total 191.87 last 101.54) - /unlink 22126 (time 1776203181.99 total 201.88 last 110.97) - /unlink 23161 (time 1776203192.00 total 211.88 last 103.45) - /unlink 24318 (time 1776203202.00 total 221.89 last 115.64) - /unlink 25617 (time 1776203212.01 total 231.89 last 129.82) - /unlink 26746 (time 1776203222.01 total 241.90 last 112.88) - /unlink 27776 (time 1776203232.02 total 251.90 last 102.92) - /unlink 28892 (time 1776203242.03 total 261.91 last 111.46) - /unlink 30000 (time 1776203251.32 total 271.21 last 119.24) - /unlink 31206 (time 1776203261.33 total 281.21 last 120.55) - /unlink 32343 (time 1776203271.33 total 291.22 last 113.62) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 293.30 seconds: 111.13 ops/second Waiting for MDT destroys to complete - open/close 615 (time 1776203300.40 total 10.02 last 61.39) - open/close 1234 (time 1776203310.42 total 20.05 last 61.73) - open/close 1893 (time 1776203320.43 total 30.05 last 65.87) - open/close 2596 (time 1776203330.45 total 40.07 last 70.17) - open/close 3218 (time 1776203340.45 total 50.08 last 62.15) - open/close 3758 (time 1776203350.46 total 60.08 last 53.98) - open/close 4384 (time 1776203360.47 total 70.09 last 62.53) - open/close 5061 (time 1776203370.47 total 80.09 last 67.70) - open/close 5716 (time 1776203380.48 total 90.10 last 65.45) - open/close 6290 (time 1776203390.48 total 100.11 last 57.37) - open/close 6890 (time 1776203400.50 total 110.12 last 59.90) - open/close 7419 (time 1776203410.50 total 120.13 last 52.88) - open/close 8028 (time 1776203420.51 total 130.13 last 60.87) - open/close 8677 (time 1776203430.51 total 140.14 last 64.86) - open/close 9268 (time 1776203440.52 total 150.14 last 59.08) - open/close 9810 (time 1776203450.52 total 160.14 last 54.17) - open/close 10000 (time 1776203453.70 total 163.32 last 59.87) - open/close 10463 (time 1776203463.70 total 173.32 last 46.30) - open/close 10976 (time 1776203473.70 total 183.32 last 51.28) - open/close 11621 (time 1776203483.71 total 193.33 last 64.46) - open/close 12263 (time 1776203493.71 total 203.34 last 64.15) - open/close 12838 (time 1776203503.71 total 213.34 last 57.50) - open/close 13384 (time 1776203513.73 total 223.35 last 54.54) - open/close 13916 (time 1776203523.75 total 233.37 last 53.09) - open/close 14519 (time 1776203533.76 total 243.38 last 60.23) - open/close 15074 (time 1776203543.78 total 253.40 last 55.38) - open/close 15696 (time 1776203553.79 total 263.41 last 62.15) - open/close 16354 (time 1776203563.79 total 273.41 last 65.78) - open/close 17026 (time 1776203573.80 total 283.42 last 67.16) - open/close 17598 (time 1776203583.81 total 293.44 last 57.10) - open/close 18220 (time 1776203593.84 total 303.46 last 62.04) - open/close 18805 (time 1776203603.85 total 313.48 last 58.41) - open/close 19473 (time 1776203613.86 total 323.48 last 66.75) - open/close 20000 (time 1776203621.85 total 331.47 last 66.02) - open/close 20676 (time 1776203631.85 total 341.47 last 67.56) - open/close 21289 (time 1776203641.86 total 351.48 last 61.26) - open/close 21909 (time 1776203651.86 total 361.48 last 62.00) - open/close 22545 (time 1776203661.88 total 371.50 last 63.47) - open/close 23112 (time 1776203671.91 total 381.53 last 56.54) - open/close 23731 (time 1776203681.92 total 391.54 last 61.84) - open/close 24384 (time 1776203691.92 total 401.54 last 65.27) - open/close 25042 (time 1776203701.94 total 411.56 last 65.70) - open/close 25639 (time 1776203711.97 total 421.60 last 59.48) - open/close 26168 (time 1776203721.98 total 431.61 last 52.84) - open/close 26758 (time 1776203732.00 total 441.62 last 58.94) - open/close 27386 (time 1776203742.00 total 451.62 last 62.78) - open/close 27956 (time 1776203752.01 total 461.64 last 56.91) - open/close 28543 (time 1776203762.03 total 471.65 last 58.61) - open/close 29059 (time 1776203772.03 total 481.65 last 51.60) - open/close 29585 (time 1776203782.03 total 491.65 last 52.59) - open/close 30000 (time 1776203789.54 total 499.17 last 55.24) - open/close 30553 (time 1776203799.55 total 509.17 last 55.27) - open/close 31221 (time 1776203809.58 total 519.20 last 66.62) - open/close 31768 (time 1776203819.59 total 529.21 last 54.61) - open/close 32371 (time 1776203829.61 total 539.23 last 60.18) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 542.34 seconds: 60.10 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 1151 (time 1776203846.77 total 10.00 last 115.05) - /unlink 2380 (time 1776203856.77 total 20.01 last 122.82) - /unlink 3534 (time 1776203866.78 total 30.02 last 115.28) - /unlink 4606 (time 1776203876.79 total 40.03 last 107.14) - /unlink 5749 (time 1776203886.79 total 50.03 last 114.27) - /unlink 6859 (time 1776203896.79 total 60.03 last 110.97) - /unlink 7943 (time 1776203906.80 total 70.04 last 108.35) - /unlink 8998 (time 1776203916.81 total 80.05 last 105.38) - /unlink 10000 (time 1776203925.84 total 89.08 last 110.92) - /unlink 11092 (time 1776203935.85 total 99.09 last 109.15) - /unlink 12204 (time 1776203945.85 total 109.09 last 111.16) - /unlink 13287 (time 1776203955.85 total 119.09 last 108.29) - /unlink 14346 (time 1776203965.85 total 129.09 last 105.89) - /unlink 15325 (time 1776203975.86 total 139.10 last 97.84) - /unlink 16379 (time 1776203985.87 total 149.11 last 105.24) - /unlink 17454 (time 1776203995.88 total 159.12 last 107.41) - /unlink 18636 (time 1776204005.89 total 169.12 last 118.17) - /unlink 19772 (time 1776204015.89 total 179.13 last 113.52) - /unlink 20915 (time 1776204025.90 total 189.14 last 114.24) - /unlink 21947 (time 1776204035.90 total 199.14 last 103.18) - /unlink 23022 (time 1776204045.90 total 209.14 last 107.49) - /unlink 24093 (time 1776204055.91 total 219.15 last 106.99) - /unlink 25227 (time 1776204065.92 total 229.16 last 113.32) - /unlink 26414 (time 1776204075.92 total 239.16 last 118.70) - /unlink 27558 (time 1776204085.92 total 249.16 last 114.37) - /unlink 28698 (time 1776204095.92 total 259.16 last 113.99) - /unlink 29715 (time 1776204105.93 total 269.17 last 101.62) - /unlink 30000 (time 1776204108.84 total 272.08 last 97.91) - /unlink 31248 (time 1776204118.84 total 282.08 last 124.76) - /unlink 32321 (time 1776204128.85 total 292.09 last 107.24) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 294.35 seconds: 110.73 ops/second Waiting for MDT destroys to complete - open/close 592 (time 1776204158.67 total 10.01 last 59.15) - open/close 1198 (time 1776204168.67 total 20.01 last 60.59) - open/close 1816 (time 1776204178.69 total 30.03 last 61.68) - open/close 2420 (time 1776204188.70 total 40.04 last 60.35) - open/close 3035 (time 1776204198.70 total 50.05 last 61.46) - open/close 3617 (time 1776204208.71 total 60.05 last 58.19) - open/close 4257 (time 1776204218.71 total 70.06 last 63.94) - open/close 4932 (time 1776204228.72 total 80.06 last 67.47) - open/close 5552 (time 1776204238.73 total 90.07 last 61.93) - open/close 6234 (time 1776204248.73 total 100.07 last 68.20) - open/close 6896 (time 1776204258.74 total 110.08 last 66.13) - open/close 7542 (time 1776204268.76 total 120.10 last 64.47) - open/close 8172 (time 1776204278.77 total 130.12 last 62.92) - open/close 8791 (time 1776204288.79 total 140.13 last 61.80) - open/close 9444 (time 1776204298.80 total 150.14 last 65.25) - open/close 10000 (time 1776204307.76 total 159.10 last 62.06) - open/close 10684 (time 1776204317.78 total 169.12 last 68.25) - open/close 11351 (time 1776204327.79 total 179.13 last 66.62) - open/close 11973 (time 1776204337.79 total 189.13 last 62.18) - open/close 12605 (time 1776204347.80 total 199.14 last 63.15) - open/close 13206 (time 1776204357.81 total 209.15 last 60.06) - open/close 13803 (time 1776204367.81 total 219.15 last 59.66) - open/close 14339 (time 1776204377.82 total 229.16 last 53.57) - open/close 14934 (time 1776204387.82 total 239.16 last 59.49) - open/close 15512 (time 1776204397.84 total 249.18 last 57.67) - open/close 16204 (time 1776204407.85 total 259.19 last 69.16) - open/close 16841 (time 1776204417.86 total 269.20 last 63.62) - open/close 17482 (time 1776204427.87 total 279.21 last 64.07) - open/close 18101 (time 1776204437.88 total 289.22 last 61.79) - open/close 18718 (time 1776204447.91 total 299.25 last 61.53) - open/close 19397 (time 1776204457.93 total 309.27 last 67.79) - open/close 20000 (time 1776204466.40 total 317.74 last 71.18) - open/close 20654 (time 1776204476.40 total 327.75 last 65.35) - open/close 21344 (time 1776204486.42 total 337.76 last 68.91) - open/close 21937 (time 1776204496.43 total 347.78 last 59.20) - open/close 22558 (time 1776204506.45 total 357.79 last 62.01) - open/close 23180 (time 1776204516.46 total 367.81 last 62.10) - open/close 23841 (time 1776204526.47 total 377.81 last 66.07) - open/close 24495 (time 1776204536.48 total 387.82 last 65.35) - open/close 25074 (time 1776204546.48 total 397.82 last 57.87) - open/close 25787 (time 1776204556.50 total 407.84 last 71.21) - open/close 26420 (time 1776204566.52 total 417.86 last 63.16) - open/close 27048 (time 1776204576.53 total 427.87 last 62.74) - open/close 27730 (time 1776204586.54 total 437.88 last 68.12) - open/close 28430 (time 1776204596.55 total 447.89 last 69.95) - open/close 29096 (time 1776204606.56 total 457.90 last 66.51) - open/close 29669 (time 1776204616.58 total 467.92 last 57.19) - open/close 30000 (time 1776204622.34 total 473.68 last 57.46) - open/close 30620 (time 1776204632.35 total 483.69 last 61.97) - open/close 31330 (time 1776204642.36 total 493.70 last 70.92) - open/close 31997 (time 1776204652.36 total 503.70 last 66.66) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 512.66 seconds: 63.58 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 970 (time 1776204676.58 total 10.00 last 96.97) - /unlink 2031 (time 1776204686.59 total 20.01 last 106.05) - /unlink 2992 (time 1776204696.59 total 30.01 last 96.06) - /unlink 4067 (time 1776204706.59 total 40.01 last 107.50) - /unlink 5202 (time 1776204716.60 total 50.02 last 113.43) - /unlink 6384 (time 1776204726.61 total 60.02 last 118.12) - /unlink 7454 (time 1776204736.61 total 70.03 last 106.91) - /unlink 8564 (time 1776204746.63 total 80.05 last 110.82) - /unlink 9681 (time 1776204756.64 total 90.06 last 111.64) - /unlink 10000 (time 1776204759.45 total 92.87 last 113.19) - /unlink 11080 (time 1776204769.46 total 102.88 last 107.94) - /unlink 12076 (time 1776204779.46 total 112.88 last 99.56) - /unlink 13108 (time 1776204789.47 total 122.89 last 103.14) - /unlink 14198 (time 1776204799.47 total 132.89 last 108.96) - /unlink 15177 (time 1776204809.48 total 142.90 last 97.82) - /unlink 16253 (time 1776204819.49 total 152.91 last 107.51) - /unlink 17269 (time 1776204829.49 total 162.91 last 101.59) - /unlink 18354 (time 1776204839.49 total 172.91 last 108.48) - /unlink 19467 (time 1776204849.50 total 182.91 last 111.28) - /unlink 20000 (time 1776204854.60 total 188.02 last 104.37) - /unlink 21045 (time 1776204864.60 total 198.02 last 104.49) - /unlink 22188 (time 1776204874.60 total 208.02 last 114.28) - /unlink 23236 (time 1776204884.61 total 218.03 last 104.76) - /unlink 24319 (time 1776204894.61 total 228.03 last 108.27) - /unlink 25499 (time 1776204904.62 total 238.04 last 117.90) - /unlink 26594 (time 1776204914.62 total 248.04 last 109.43) - /unlink 27786 (time 1776204924.64 total 258.06 last 119.05) - /unlink 28839 (time 1776204934.65 total 268.07 last 105.20) - /unlink 30000 (time 1776204944.47 total 277.89 last 118.19) - /unlink 31023 (time 1776204954.47 total 287.89 last 102.25) - /unlink 32100 (time 1776204964.48 total 297.90 last 107.60) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 302.63 seconds: 107.70 ops/second Waiting for MDT destroys to complete - open/close 615 (time 1776205000.09 total 10.01 last 61.44) - open/close 1288 (time 1776205010.10 total 20.02 last 67.24) - open/close 1941 (time 1776205020.11 total 30.03 last 65.25) - open/close 2560 (time 1776205030.13 total 40.06 last 61.73) - open/close 3117 (time 1776205040.15 total 50.07 last 55.62) - open/close 3726 (time 1776205050.15 total 60.07 last 60.90) - open/close 4325 (time 1776205060.15 total 70.07 last 59.87) - open/close 4929 (time 1776205070.16 total 80.08 last 60.38) - open/close 5585 (time 1776205080.16 total 90.08 last 65.58) - open/close 6211 (time 1776205090.16 total 100.08 last 62.59) - open/close 6826 (time 1776205100.18 total 110.10 last 61.38) - open/close 7486 (time 1776205110.21 total 120.13 last 65.80) - open/close 8114 (time 1776205120.22 total 130.14 last 62.73) - open/close 8683 (time 1776205130.22 total 140.14 last 56.89) - open/close 9229 (time 1776205140.22 total 150.14 last 54.60) - open/close 9781 (time 1776205150.23 total 160.15 last 55.17) - open/close 10000 (time 1776205154.01 total 163.93 last 57.94) - open/close 10589 (time 1776205164.02 total 173.94 last 58.84) - open/close 11147 (time 1776205174.03 total 183.95 last 55.72) - open/close 11804 (time 1776205184.05 total 193.97 last 65.58) - open/close 12459 (time 1776205194.07 total 203.99 last 65.40) - open/close 13125 (time 1776205204.08 total 214.00 last 66.51) - open/close 13722 (time 1776205214.09 total 224.01 last 59.65) - open/close 14375 (time 1776205224.09 total 234.01 last 65.28) - open/close 15065 (time 1776205234.10 total 244.02 last 68.90) - open/close 15692 (time 1776205244.11 total 254.03 last 62.64) - open/close 16354 (time 1776205254.12 total 264.04 last 66.15) - open/close 17047 (time 1776205264.13 total 274.05 last 69.26) - open/close 17693 (time 1776205274.13 total 284.05 last 64.56) - open/close 18344 (time 1776205284.13 total 294.05 last 65.09) - open/close 18934 (time 1776205294.14 total 304.06 last 58.99) - open/close 19536 (time 1776205304.14 total 314.07 last 60.14) - open/close 20000 (time 1776205311.57 total 321.49 last 62.49) - open/close 20598 (time 1776205321.58 total 331.50 last 59.77) - open/close 21170 (time 1776205331.58 total 341.50 last 57.15) - open/close 21833 (time 1776205341.59 total 351.51 last 66.25) - open/close 22443 (time 1776205351.61 total 361.53 last 60.89) - open/close 23089 (time 1776205361.62 total 371.54 last 64.57) - open/close 23787 (time 1776205371.62 total 381.54 last 69.80) - open/close 24436 (time 1776205381.64 total 391.57 last 64.72) - open/close 25163 (time 1776205391.66 total 401.58 last 72.58) - open/close 25818 (time 1776205401.68 total 411.60 last 65.39) - open/close 26458 (time 1776205411.68 total 421.60 last 63.97) - open/close 27022 (time 1776205421.69 total 431.61 last 56.38) - open/close 27657 (time 1776205431.69 total 441.61 last 63.45) - open/close 28318 (time 1776205441.72 total 451.64 last 65.93) - open/close 28963 (time 1776205451.74 total 461.66 last 64.38) - open/close 29550 (time 1776205461.75 total 471.67 last 58.62) - open/close 30000 (time 1776205470.73 total 480.65 last 50.13) - open/close 30630 (time 1776205480.73 total 490.66 last 62.97) - open/close 31159 (time 1776205490.74 total 500.66 last 52.85) - open/close 31759 (time 1776205500.75 total 510.67 last 59.95) - open/close 32329 (time 1776205510.78 total 520.70 last 56.83) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 525.57 seconds: 62.02 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 846 (time 1776205531.63 total 10.00 last 84.58) - /unlink 1902 (time 1776205541.64 total 20.01 last 105.54) - /unlink 2823 (time 1776205551.64 total 30.01 last 92.09) - /unlink 3763 (time 1776205561.64 total 40.01 last 93.96) - /unlink 4743 (time 1776205571.65 total 50.02 last 97.95) - /unlink 5807 (time 1776205581.65 total 60.02 last 106.35) - /unlink 6841 (time 1776205591.66 total 70.03 last 103.33) - /unlink 7866 (time 1776205601.67 total 80.04 last 102.37) - /unlink 8923 (time 1776205611.67 total 90.04 last 105.68) - /unlink 9945 (time 1776205621.68 total 100.05 last 102.16) - /unlink 11026 (time 1776205631.68 total 110.05 last 108.08) - /unlink 12067 (time 1776205641.68 total 120.05 last 104.08) - /unlink 13192 (time 1776205651.68 total 130.05 last 112.48) - /unlink 14276 (time 1776205661.69 total 140.06 last 108.30) - /unlink 15420 (time 1776205671.70 total 150.07 last 114.33) - /unlink 16482 (time 1776205681.70 total 160.07 last 106.16) - /unlink 17573 (time 1776205691.71 total 170.08 last 108.99) - /unlink 18712 (time 1776205701.71 total 180.08 last 113.90) - /unlink 19926 (time 1776205711.73 total 190.10 last 121.21) - /unlink 21146 (time 1776205721.73 total 200.10 last 121.93) - /unlink 22285 (time 1776205731.73 total 210.11 last 113.89) - /unlink 23314 (time 1776205741.74 total 220.11 last 102.80) - /unlink 24364 (time 1776205751.75 total 230.12 last 104.94) - /unlink 25300 (time 1776205761.75 total 240.12 last 93.58) - /unlink 26265 (time 1776205771.76 total 250.13 last 96.44) - /unlink 27295 (time 1776205781.77 total 260.14 last 102.91) - /unlink 28441 (time 1776205791.77 total 270.14 last 114.56) - /unlink 29546 (time 1776205801.77 total 280.14 last 110.48) - /unlink 30000 (time 1776205806.15 total 284.52 last 103.76) - /unlink 31083 (time 1776205816.16 total 294.53 last 108.15) - /unlink 32252 (time 1776205826.16 total 304.53 last 116.87) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 308.32 seconds: 105.72 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg433-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg433-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg433-server stop mds service on oleg433-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg433-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing unload_modules_local oleg433-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg433-server: LNET ready to unload oleg433-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-client: ssh exited with exit code 2 pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 PASS 106 (4405s) Stopping clients: oleg433-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg433-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg433-client: oleg433-server: ssh exited with exit code 2 oleg433-server: oleg433-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg433-server' oleg433-server: oleg433-server.virtnet: executing load_modules_local oleg433-server: Loading modules from /home/green/git/lustre-release/lustre oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg433-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg433-server: detected 4 online CPUs by sysfs oleg433-server: Force libcfs to create 2 CPU partitions oleg433-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg433-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 4687 sec ========== 18:33:11 (1776205991)