-----============= acceptance-small: conf-sanity ============----- Tue Mar 11 07:18:02 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg459-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg459-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg459-server oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: MODOPTS_LIBCFS= oleg459-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg459-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg459-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg459-server: oleg459-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg459-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:19:28 (1741691968) Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: MODOPTS_LIBCFS= oleg459-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg459-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg459-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg459-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5540 (time 1741692006.36 total 10.00 last 553.97) - open/close 10000 (time 1741692013.90 total 17.54 last 591.19) - open/close 15414 (time 1741692023.91 total 27.55 last 541.36) - open/close 20000 (time 1741692032.32 total 35.95 last 545.33) - open/close 25517 (time 1741692042.32 total 45.96 last 551.64) - open/close 30000 (time 1741692050.47 total 54.11 last 549.94) - open/close 35449 (time 1741692060.47 total 64.11 last 544.80) - open/close 40000 (time 1741692069.55 total 73.19 last 501.40) - open/close 44885 (time 1741692079.55 total 83.19 last 488.47) - open/close 50000 (time 1741692089.25 total 92.89 last 527.14) - open/close 55367 (time 1741692099.25 total 102.89 last 536.63) - open/close 60000 (time 1741692107.79 total 111.43 last 542.41) total: 64768 open/close in 120.29 seconds: 538.42 ops/second - /unlink 9730 (time 1741692127.54 total 10.00 last 972.93) - /unlink 19565 (time 1741692137.54 total 20.00 last 983.44) - /unlink 29191 (time 1741692147.54 total 30.00 last 962.49) - /unlink 38857 (time 1741692157.54 total 40.00 last 966.60) - /unlink 48592 (time 1741692167.54 total 50.00 last 973.46) - /unlink 58082 (time 1741692177.54 total 60.00 last 948.93) - /unlink 60000 (time 1741692179.62 total 62.08 last 923.58) total: 64768 /unlink in 66.90 seconds: 968.16 ops/second Waiting for MDT destroys to complete oleg459-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg459-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 - open/close 5498 (time 1741692290.87 total 10.00 last 549.74) - open/close 10000 (time 1741692299.11 total 18.24 last 546.27) - open/close 15607 (time 1741692309.12 total 28.24 last 560.64) - open/close 20000 (time 1741692317.26 total 36.38 last 539.64) - open/close 25343 (time 1741692327.26 total 46.39 last 534.23) - open/close 30000 (time 1741692336.05 total 55.18 last 529.39) - open/close 35298 (time 1741692346.05 total 65.18 last 529.79) - open/close 40000 (time 1741692354.60 total 73.72 last 550.53) - open/close 45546 (time 1741692364.60 total 83.72 last 554.52) - open/close 50000 (time 1741692372.71 total 91.84 last 548.88) - open/close 55460 (time 1741692382.71 total 101.84 last 545.91) - open/close 60000 (time 1741692390.98 total 110.10 last 549.48) total: 64768 open/close in 118.92 seconds: 544.63 ops/second - /unlink 9735 (time 1741692410.64 total 10.00 last 973.41) - /unlink 19741 (time 1741692420.64 total 20.00 last 1000.52) - /unlink 29546 (time 1741692430.64 total 30.00 last 980.49) - /unlink 39526 (time 1741692440.64 total 40.00 last 997.94) - /unlink 49528 (time 1741692450.64 total 50.00 last 1000.12) - /unlink 59342 (time 1741692460.65 total 60.00 last 981.39) total: 64768 /unlink in 65.46 seconds: 989.49 ops/second Waiting for MDT destroys to complete oleg459-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg459-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 - open/close 5568 (time 1741692571.64 total 10.00 last 556.72) - open/close 10000 (time 1741692579.70 total 18.06 last 550.27) - open/close 14990 (time 1741692589.70 total 28.06 last 498.97) - open/close 19978 (time 1741692599.70 total 38.06 last 498.72) - open/close 24353 (time 1741692609.70 total 48.06 last 437.47) - open/close 28186 (time 1741692619.70 total 58.06 last 383.29) - open/close 30000 (time 1741692623.33 total 61.68 last 500.53) - open/close 35184 (time 1741692633.33 total 71.68 last 518.32) - open/close 40000 (time 1741692642.76 total 81.12 last 510.39) - open/close 44527 (time 1741692652.76 total 91.12 last 452.70) - open/close 49327 (time 1741692662.76 total 101.12 last 479.96) - open/close 53965 (time 1741692672.77 total 111.12 last 463.77) - open/close 58316 (time 1741692682.77 total 121.12 last 435.02) - open/close 60000 (time 1741692686.83 total 125.19 last 414.11) - open/close 64021 (time 1741692696.84 total 135.19 last 402.02) total: 64768 open/close in 137.19 seconds: 472.12 ops/second - /unlink 7934 (time 1741692709.87 total 10.00 last 793.34) - /unlink 10000 (time 1741692712.56 total 12.69 last 767.32) - /unlink 18062 (time 1741692722.56 total 22.69 last 806.16) - /unlink 20000 (time 1741692724.61 total 24.73 last 949.79) - /unlink 28369 (time 1741692734.61 total 34.73 last 836.86) - /unlink 37749 (time 1741692744.61 total 44.74 last 937.90) - /unlink 40000 (time 1741692747.18 total 47.30 last 876.33) - /unlink 49625 (time 1741692757.18 total 57.30 last 962.47) - /unlink 59368 (time 1741692767.18 total 67.31 last 974.25) total: 64768 /unlink in 72.72 seconds: 890.66 ops/second Waiting for MDT destroys to complete oleg459-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg459-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 - open/close 3235 (time 1741692872.01 total 10.00 last 323.49) - open/close 6795 (time 1741692882.01 total 20.00 last 355.89) - open/close 9921 (time 1741692892.02 total 30.01 last 312.54) - open/close 13256 (time 1741692902.02 total 40.01 last 333.43) - open/close 16838 (time 1741692912.02 total 50.01 last 358.15) - open/close 19756 (time 1741692922.02 total 60.01 last 291.76) - open/close 22960 (time 1741692932.02 total 70.01 last 320.34) - open/close 26071 (time 1741692942.03 total 80.01 last 311.02) - open/close 28864 (time 1741692952.03 total 90.02 last 279.26) - open/close 30000 (time 1741692956.33 total 94.32 last 264.17) - open/close 32629 (time 1741692966.33 total 104.32 last 262.85) - open/close 35221 (time 1741692976.33 total 114.32 last 259.11) - open/close 37754 (time 1741692986.33 total 124.32 last 253.29) - open/close 40000 (time 1741692994.67 total 132.66 last 269.44) - open/close 42766 (time 1741693004.67 total 142.66 last 276.59) - open/close 45414 (time 1741693014.67 total 152.66 last 264.70) - open/close 48024 (time 1741693024.68 total 162.67 last 260.91) - open/close 50000 (time 1741693031.11 total 169.10 last 307.33) - open/close 52803 (time 1741693041.11 total 179.10 last 280.27) - open/close 55410 (time 1741693051.11 total 189.10 last 260.62) - open/close 58029 (time 1741693061.11 total 199.10 last 261.85) - open/close 60000 (time 1741693067.73 total 205.72 last 297.75) - open/close 62777 (time 1741693077.73 total 215.72 last 277.65) total: 64768 open/close in 223.03 seconds: 290.41 ops/second - /unlink 4610 (time 1741693096.37 total 10.00 last 460.97) - /unlink 9357 (time 1741693106.37 total 20.00 last 474.67) - /unlink 14289 (time 1741693116.37 total 30.00 last 493.11) - /unlink 20000 (time 1741693125.96 total 39.59 last 595.43) - /unlink 27014 (time 1741693135.97 total 49.60 last 701.31) - /unlink 30000 (time 1741693142.12 total 55.75 last 484.92) - /unlink 34758 (time 1741693152.13 total 65.75 last 475.74) - /unlink 39583 (time 1741693162.13 total 75.76 last 482.42) - /unlink 44246 (time 1741693172.13 total 85.76 last 466.23) - /unlink 49337 (time 1741693182.13 total 95.76 last 509.08) - /unlink 54602 (time 1741693192.13 total 105.76 last 526.45) - /unlink 59665 (time 1741693202.13 total 115.76 last 506.21) total: 64768 /unlink in 125.62 seconds: 515.60 ops/second Waiting for MDT destroys to complete oleg459-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg459-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 - open/close 2756 (time 1741693228.15 total 10.00 last 275.51) - open/close 5351 (time 1741693238.15 total 20.01 last 259.40) - open/close 8171 (time 1741693248.15 total 30.01 last 281.98) - open/close 10000 (time 1741693254.27 total 36.13 last 298.83) - open/close 13216 (time 1741693264.28 total 46.13 last 321.51) - open/close 16177 (time 1741693274.28 total 56.13 last 296.08) - open/close 20000 (time 1741693284.22 total 66.07 last 384.63) - open/close 24692 (time 1741693294.22 total 76.07 last 469.19) - open/close 29556 (time 1741693304.22 total 86.07 last 486.31) - open/close 34022 (time 1741693314.22 total 96.07 last 446.53) - open/close 39074 (time 1741693324.22 total 106.08 last 505.13) - open/close 42812 (time 1741693334.22 total 116.08 last 373.66) - open/close 45829 (time 1741693344.22 total 126.08 last 301.67) - open/close 49346 (time 1741693354.23 total 136.08 last 351.62) - open/close 53045 (time 1741693364.23 total 146.08 last 369.88) - open/close 56280 (time 1741693374.23 total 156.09 last 323.44) - open/close 59404 (time 1741693384.23 total 166.09 last 312.35) - open/close 60000 (time 1741693386.52 total 168.37 last 260.87) - open/close 63273 (time 1741693396.52 total 178.37 last 327.22) total: 64768 open/close in 183.04 seconds: 353.85 ops/second - /unlink 6993 (time 1741693412.36 total 10.00 last 699.27) - /unlink 10000 (time 1741693416.14 total 13.78 last 796.58) - /unlink 18087 (time 1741693426.14 total 23.78 last 808.65) - /unlink 20000 (time 1741693428.72 total 26.36 last 740.37) - /unlink 26098 (time 1741693438.72 total 36.36 last 609.80) - /unlink 30000 (time 1741693446.44 total 44.08 last 505.56) - /unlink 35412 (time 1741693456.44 total 54.08 last 541.18) - /unlink 40000 (time 1741693465.37 total 63.01 last 513.54) - /unlink 49473 (time 1741693475.37 total 73.01 last 947.26) - /unlink 56218 (time 1741693485.38 total 83.01 last 674.45) - /unlink 60000 (time 1741693490.24 total 87.88 last 777.87) total: 64768 /unlink in 93.94 seconds: 689.49 ops/second Waiting for MDT destroys to complete oleg459-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg459-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg459-server unloading modules via unload_modules_local on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 PASS 106 (1611s) Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 oleg459-server: oleg459-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: detected 4 online CPUs by sysfs oleg459-server: MODOPTS_LIBCFS= oleg459-server: Force libcfs to create 2 CPU partitions oleg459-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg459-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1718 sec ========== 07:46:40 (1741693600) === conf-sanity: start cleanup 07:46:40 (1741693600) === === conf-sanity: finish cleanup 07:46:41 (1741693601) ===