-----============= acceptance-small: conf-sanity ============----- Fri May 16 08:45:46 EDT 2025 oleg244-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg244-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg244-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg244-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg244-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg244-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg244-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg244-server oleg244-server: oleg244-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg244-server' oleg244-server: oleg244-server.virtnet: executing load_modules_local oleg244-server: Loading modules from /home/green/git/lustre-release/lustre oleg244-server: detected 4 online CPUs by sysfs oleg244-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg244-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg244-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg244-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg244-server: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg244-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg244-server: oleg244-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg244-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg244-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg244-server stop mds service on oleg244-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg244-server stop mds service on oleg244-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg244-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:47:32 (1747399652) Stopping clients: oleg244-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg244-client.virtnet /mnt/lustre2 (opts:-f) oleg244-server: oleg244-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg244-server' oleg244-server: oleg244-server.virtnet: executing load_modules_local oleg244-server: Loading modules from /home/green/git/lustre-release/lustre oleg244-server: detected 4 online CPUs by sysfs oleg244-server: Force libcfs to create 2 CPU partitions oleg244-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg244-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg244-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg244-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg244-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg244-server: oleg244-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg244-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg244-client.virtnet: -o user_xattr,flock oleg244-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 334 32562 2% /mnt/lustre[OST:0] filesystem_summary: 33091 529 32562 2% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1576 (time 1747399713.37 total 10.00 last 157.53) - open/close 3180 (time 1747399723.37 total 20.01 last 160.31) - open/close 4729 (time 1747399733.38 total 30.02 last 154.78) - open/close 6618 (time 1747399743.38 total 40.02 last 188.88) - open/close 8521 (time 1747399753.39 total 50.03 last 190.18) - open/close 10000 (time 1747399760.16 total 56.79 last 218.56) - open/close 12038 (time 1747399770.16 total 66.80 last 203.73) - open/close 13926 (time 1747399780.16 total 76.80 last 188.76) - open/close 15770 (time 1747399790.17 total 86.80 last 184.30) - open/close 17602 (time 1747399800.17 total 96.81 last 183.17) - open/close 19403 (time 1747399810.17 total 106.81 last 180.01) - open/close 20000 (time 1747399813.34 total 109.98 last 188.41) - open/close 21875 (time 1747399823.35 total 119.98 last 187.40) - open/close 23743 (time 1747399833.36 total 129.99 last 186.65) - open/close 25640 (time 1747399843.36 total 140.00 last 189.64) - open/close 27409 (time 1747399853.36 total 150.00 last 176.84) - open/close 29068 (time 1747399863.37 total 160.00 last 165.83) - open/close 30000 (time 1747399868.86 total 165.49 last 169.77) - open/close 31519 (time 1747399878.86 total 175.50 last 151.86) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 182.64 seconds: 178.46 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2863 (time 1747399897.69 total 10.00 last 286.21) - /unlink 5864 (time 1747399907.69 total 20.00 last 300.09) - /unlink 8955 (time 1747399917.69 total 30.01 last 309.03) - /unlink 10000 (time 1747399921.29 total 33.61 last 290.15) - /unlink 12846 (time 1747399931.30 total 43.61 last 284.56) - /unlink 15785 (time 1747399941.30 total 53.61 last 293.86) - /unlink 18850 (time 1747399951.30 total 63.61 last 306.49) - /unlink 20000 (time 1747399954.84 total 67.16 last 324.39) - /unlink 23185 (time 1747399964.85 total 77.16 last 318.38) - /unlink 25916 (time 1747399974.85 total 87.16 last 272.95) - /unlink 29390 (time 1747399984.85 total 97.17 last 347.33) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 106.82 seconds: 305.12 ops/second Waiting for MDT destroys to complete - open/close 1687 (time 1747400015.50 total 10.01 last 168.57) - open/close 3391 (time 1747400025.50 total 20.01 last 170.39) - open/close 5128 (time 1747400035.50 total 30.01 last 173.65) - open/close 6911 (time 1747400045.51 total 40.02 last 178.16) - open/close 8689 (time 1747400055.51 total 50.02 last 177.74) - open/close 10000 (time 1747400062.40 total 56.90 last 190.49) - open/close 11760 (time 1747400072.40 total 66.91 last 175.94) - open/close 13380 (time 1747400082.40 total 76.91 last 161.95) - open/close 15171 (time 1747400092.40 total 86.91 last 179.09) - open/close 17120 (time 1747400102.40 total 96.91 last 194.87) - open/close 19274 (time 1747400112.41 total 106.92 last 215.34) - open/close 20000 (time 1747400115.95 total 110.46 last 204.92) - open/close 22264 (time 1747400125.95 total 120.46 last 226.38) - open/close 24151 (time 1747400135.96 total 130.46 last 188.59) - open/close 25959 (time 1747400145.96 total 140.47 last 180.75) - open/close 27782 (time 1747400155.96 total 150.47 last 182.23) - open/close 29535 (time 1747400165.96 total 160.47 last 175.28) - open/close 30000 (time 1747400168.59 total 163.10 last 176.96) - open/close 32171 (time 1747400178.59 total 173.10 last 217.10) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 175.13 seconds: 186.11 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4477 (time 1747400192.26 total 10.00 last 447.69) - /unlink 8881 (time 1747400202.27 total 20.00 last 440.29) - /unlink 10000 (time 1747400204.68 total 22.42 last 462.60) - /unlink 13908 (time 1747400214.69 total 32.42 last 390.73) - /unlink 17548 (time 1747400224.69 total 42.42 last 363.99) - /unlink 20000 (time 1747400232.62 total 50.36 last 308.95) - /unlink 23261 (time 1747400242.62 total 60.36 last 326.03) - /unlink 27099 (time 1747400252.63 total 70.36 last 383.77) - /unlink 30000 (time 1747400260.65 total 78.39 last 361.51) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 85.22 seconds: 382.48 ops/second Waiting for MDT destroys to complete - open/close 2255 (time 1747400285.37 total 10.00 last 225.44) - open/close 4386 (time 1747400295.37 total 20.00 last 213.08) - open/close 6624 (time 1747400305.37 total 30.01 last 223.76) - open/close 8972 (time 1747400315.37 total 40.01 last 234.78) - open/close 10000 (time 1747400319.49 total 44.13 last 249.58) - open/close 11925 (time 1747400329.49 total 54.13 last 192.49) - open/close 14079 (time 1747400339.50 total 64.13 last 215.31) - open/close 16969 (time 1747400349.50 total 74.13 last 288.96) - open/close 19779 (time 1747400359.50 total 84.14 last 280.88) - open/close 22606 (time 1747400369.50 total 94.14 last 282.69) - open/close 25013 (time 1747400379.50 total 104.14 last 240.66) - open/close 27581 (time 1747400389.51 total 114.15 last 256.56) - open/close 29619 (time 1747400399.52 total 124.15 last 203.74) - open/close 31926 (time 1747400409.52 total 134.15 last 230.70) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 137.29 seconds: 237.41 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 4176 (time 1747400423.86 total 10.00 last 417.53) - /unlink 7840 (time 1747400433.86 total 20.00 last 366.29) - /unlink 10000 (time 1747400439.39 total 25.53 last 391.04) - /unlink 14488 (time 1747400449.39 total 35.53 last 448.77) - /unlink 19171 (time 1747400459.39 total 45.53 last 468.29) - /unlink 24720 (time 1747400469.39 total 55.53 last 554.88) - /unlink 29856 (time 1747400479.39 total 65.53 last 513.54) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 71.61 seconds: 455.17 ops/second Waiting for MDT destroys to complete - open/close 3877 (time 1747400502.22 total 10.00 last 387.64) - open/close 7629 (time 1747400512.22 total 20.00 last 375.18) - open/close 10000 (time 1747400519.52 total 27.30 last 324.76) - open/close 13268 (time 1747400529.52 total 37.31 last 326.64) - open/close 16196 (time 1747400539.53 total 47.31 last 292.72) - open/close 19409 (time 1747400549.53 total 57.31 last 321.24) - open/close 22969 (time 1747400559.53 total 67.31 last 355.99) - open/close 26452 (time 1747400569.53 total 77.31 last 348.22) - open/close 30000 (time 1747400579.36 total 87.14 last 361.00) - open/close 32370 (time 1747400589.36 total 97.14 last 236.96) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 98.30 seconds: 331.58 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 2514 (time 1747400602.35 total 10.00 last 251.38) - /unlink 7090 (time 1747400612.36 total 20.00 last 457.55) - /unlink 10000 (time 1747400619.52 total 27.17 last 406.16) - /unlink 14324 (time 1747400629.52 total 37.17 last 432.35) - /unlink 17794 (time 1747400639.52 total 47.17 last 346.96) - /unlink 20000 (time 1747400645.22 total 52.86 last 387.43) - /unlink 23660 (time 1747400655.22 total 62.86 last 365.98) - /unlink 27287 (time 1747400665.22 total 72.86 last 362.67) - /unlink 30000 (time 1747400674.24 total 81.89 last 300.60) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 89.29 seconds: 365.04 ops/second Waiting for MDT destroys to complete - open/close 1718 (time 1747400701.79 total 10.00 last 171.79) - open/close 3355 (time 1747400711.79 total 20.00 last 163.65) - open/close 4920 (time 1747400721.79 total 30.01 last 156.48) - open/close 6524 (time 1747400731.80 total 40.01 last 160.37) - open/close 8571 (time 1747400741.80 total 50.01 last 204.62) - open/close 10000 (time 1747400748.80 total 57.01 last 204.08) - open/close 12186 (time 1747400758.81 total 67.02 last 218.54) - open/close 14853 (time 1747400768.81 total 77.02 last 266.57) - open/close 17130 (time 1747400778.81 total 87.02 last 227.69) - open/close 19790 (time 1747400788.81 total 97.02 last 265.99) - open/close 23710 (time 1747400798.81 total 107.02 last 391.89) - open/close 27099 (time 1747400808.82 total 117.03 last 338.81) - open/close 30000 (time 1747400818.61 total 126.82 last 296.33) - open/close 32136 (time 1747400828.61 total 136.82 last 213.54) open(/mnt/lustre/d106.conf-sanity/f-32594) error: No space left on device total: 32594 open/close in 138.62 seconds: 235.13 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32869 47123 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33124 33124 0 100% /mnt/lustre - /unlink 6372 (time 1747400841.50 total 10.00 last 637.19) - /unlink 10000 (time 1747400848.08 total 16.58 last 551.17) - /unlink 15714 (time 1747400858.08 total 26.58 last 571.32) - /unlink 20000 (time 1747400866.48 total 34.98 last 510.53) - /unlink 25496 (time 1747400876.48 total 44.98 last 549.32) - /unlink 30000 (time 1747400886.06 total 54.56 last 470.28) unlink(/mnt/lustre/d106.conf-sanity/f-32594) error: No such file or directory total: 32594 /unlink in 58.84 seconds: 553.99 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg244-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg244-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg244-server stop mds service on oleg244-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg244-server stop mds service on oleg244-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg244-server LNET ready to unload unloading modules on: 'oleg244-server' oleg244-server: oleg244-server.virtnet: executing unload_modules_local oleg244-server: LNET ready to unload modules unloaded. pdsh@oleg244-client: oleg244-client: ssh exited with exit code 2 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 2 pdsh@oleg244-client: oleg244-client: ssh exited with exit code 2 pdsh@oleg244-client: oleg244-server: ssh exited with exit code 2 PASS 106 (1270s) Stopping clients: oleg244-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg244-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg244-client: oleg244-server: ssh exited with exit code 2 oleg244-server: oleg244-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg244-server' oleg244-server: oleg244-server.virtnet: executing load_modules_local oleg244-server: Loading modules from /home/green/git/lustre-release/lustre oleg244-server: detected 4 online CPUs by sysfs oleg244-server: Force libcfs to create 2 CPU partitions oleg244-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg244-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1398 sec ========== 09:09:05 (1747400945)