-----============= acceptance-small: conf-sanity ============----- Wed May 7 09:45:49 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg152-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg152-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg152-server oleg152-server: oleg152-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg152-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg152-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg152-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg152-server: oleg152-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg152-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:47:14 (1746625634) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) oleg152-server: oleg152-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg152-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg152-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg152-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg152-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 279 79713 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 257 79735 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65976 536 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2888 (time 1746625673.84 total 10.00 last 288.75) - open/close 4341 (time 1746625683.85 total 20.01 last 145.24) - open/close 5746 (time 1746625693.85 total 30.01 last 140.47) - open/close 8177 (time 1746625703.86 total 40.01 last 242.94) - open/close 10000 (time 1746625710.30 total 46.46 last 282.96) - open/close 12810 (time 1746625720.30 total 56.46 last 280.84) - open/close 14721 (time 1746625730.31 total 66.47 last 190.97) - open/close 17191 (time 1746625740.32 total 76.47 last 246.88) - open/close 20000 (time 1746625748.63 total 84.79 last 337.98) - open/close 24852 (time 1746625758.63 total 94.79 last 485.14) - open/close 28759 (time 1746625768.63 total 104.79 last 390.68) - open/close 30000 (time 1746625772.90 total 109.06 last 290.33) - open/close 32857 (time 1746625782.90 total 119.06 last 285.65) - open/close 36056 (time 1746625792.91 total 129.06 last 319.88) - open/close 40000 (time 1746625801.91 total 138.07 last 437.87) - open/close 43001 (time 1746625811.91 total 148.07 last 300.05) - open/close 46140 (time 1746625821.92 total 158.07 last 313.86) - open/close 50000 (time 1746625830.26 total 166.42 last 462.32) - open/close 54076 (time 1746625840.27 total 176.42 last 407.55) - open/close 58363 (time 1746625850.27 total 186.43 last 428.64) - open/close 60000 (time 1746625853.72 total 189.88 last 474.10) total: 64768 open/close in 198.54 seconds: 326.23 ops/second - /unlink 10000 (time 1746625873.08 total 9.88 last 1012.16) - /unlink 20000 (time 1746625882.88 total 19.68 last 1020.45) - /unlink 29118 (time 1746625892.88 total 29.68 last 911.74) - /unlink 38062 (time 1746625902.88 total 39.68 last 894.39) - /unlink 48492 (time 1746625912.88 total 49.68 last 1042.98) - /unlink 58386 (time 1746625922.88 total 59.68 last 989.35) total: 64768 /unlink in 66.73 seconds: 970.62 ops/second Waiting for MDT destroys to complete - open/close 5680 (time 1746626037.17 total 10.00 last 567.87) - open/close 10000 (time 1746626045.47 total 18.30 last 520.37) - open/close 15768 (time 1746626055.47 total 28.30 last 576.79) - open/close 20000 (time 1746626064.26 total 37.09 last 481.44) - open/close 24057 (time 1746626074.26 total 47.10 last 405.65) - open/close 27163 (time 1746626084.27 total 57.10 last 310.56) - open/close 30000 (time 1746626093.32 total 66.15 last 313.34) - open/close 33135 (time 1746626103.32 total 76.16 last 313.35) - open/close 36805 (time 1746626113.33 total 86.16 last 366.93) - open/close 39707 (time 1746626123.35 total 96.18 last 289.54) - open/close 44232 (time 1746626133.35 total 106.18 last 452.43) - open/close 48347 (time 1746626143.35 total 116.18 last 411.47) - open/close 50000 (time 1746626147.25 total 120.08 last 423.95) - open/close 53127 (time 1746626157.25 total 130.08 last 312.66) - open/close 56071 (time 1746626167.25 total 140.08 last 294.37) - open/close 59604 (time 1746626177.25 total 150.09 last 353.24) - open/close 62639 (time 1746626187.26 total 160.09 last 303.46) total: 64768 open/close in 165.76 seconds: 390.73 ops/second - /unlink 6391 (time 1746626203.97 total 10.00 last 639.07) - /unlink 10000 (time 1746626209.11 total 15.14 last 701.58) - /unlink 19171 (time 1746626219.11 total 25.15 last 917.00) - /unlink 25154 (time 1746626229.12 total 35.15 last 598.25) - /unlink 30000 (time 1746626235.21 total 41.24 last 794.74) - /unlink 39319 (time 1746626245.21 total 51.24 last 931.88) - /unlink 48271 (time 1746626255.21 total 61.24 last 895.19) - /unlink 50000 (time 1746626257.33 total 63.36 last 816.05) - /unlink 56966 (time 1746626267.33 total 73.36 last 696.56) - /unlink 60000 (time 1746626272.48 total 78.51 last 589.58) total: 64768 /unlink in 84.71 seconds: 764.55 ops/second Waiting for MDT destroys to complete - open/close 4249 (time 1746626368.38 total 10.00 last 424.84) - open/close 8966 (time 1746626378.38 total 20.00 last 471.66) - open/close 10000 (time 1746626380.65 total 22.27 last 456.48) - open/close 14807 (time 1746626390.65 total 32.27 last 480.64) - open/close 19551 (time 1746626400.65 total 42.27 last 474.34) - open/close 24829 (time 1746626410.65 total 52.27 last 527.77) - open/close 30000 (time 1746626419.78 total 61.40 last 566.46) - open/close 35990 (time 1746626429.78 total 71.40 last 598.99) - open/close 40000 (time 1746626437.24 total 78.86 last 537.49) - open/close 45678 (time 1746626447.24 total 88.86 last 567.77) - open/close 50000 (time 1746626454.47 total 96.09 last 598.23) - open/close 55702 (time 1746626464.47 total 106.09 last 570.12) - open/close 60000 (time 1746626471.74 total 113.36 last 590.79) total: 64768 open/close in 121.44 seconds: 533.34 ops/second - /unlink 9642 (time 1746626490.64 total 10.00 last 964.15) - /unlink 18215 (time 1746626500.64 total 20.00 last 857.21) - /unlink 28058 (time 1746626510.64 total 30.00 last 984.27) - /unlink 39005 (time 1746626520.64 total 40.00 last 1094.68) - /unlink 50000 (time 1746626530.45 total 49.81 last 1121.10) - /unlink 60000 (time 1746626539.75 total 59.12 last 1074.57) total: 64768 /unlink in 63.42 seconds: 1021.27 ops/second Waiting for MDT destroys to complete - open/close 5138 (time 1746626652.94 total 10.00 last 513.71) - open/close 10000 (time 1746626662.69 total 19.75 last 498.67) - open/close 15171 (time 1746626672.69 total 29.75 last 517.01) - open/close 20000 (time 1746626682.05 total 39.11 last 515.90) - open/close 22982 (time 1746626692.05 total 49.12 last 298.12) - open/close 24756 (time 1746626702.05 total 59.12 last 177.40) - open/close 27753 (time 1746626712.05 total 69.12 last 299.70) - open/close 30000 (time 1746626717.20 total 74.26 last 436.54) - open/close 33917 (time 1746626727.20 total 84.26 last 391.66) - open/close 36502 (time 1746626737.21 total 94.27 last 258.39) - open/close 39847 (time 1746626747.21 total 104.27 last 334.48) - open/close 44176 (time 1746626757.21 total 114.27 last 432.85) - open/close 47901 (time 1746626767.21 total 124.27 last 372.45) - open/close 50000 (time 1746626772.22 total 129.29 last 418.61) - open/close 53200 (time 1746626782.23 total 139.29 last 319.75) - open/close 54863 (time 1746626792.23 total 149.30 last 166.26) - open/close 57716 (time 1746626802.23 total 159.30 last 285.28) - open/close 60000 (time 1746626809.35 total 166.41 last 320.93) total: 64768 open/close in 176.05 seconds: 367.89 ops/second - /unlink 8894 (time 1746626829.87 total 10.00 last 889.33) - /unlink 18113 (time 1746626839.87 total 20.00 last 921.85) - /unlink 20000 (time 1746626841.95 total 22.08 last 906.80) - /unlink 28842 (time 1746626851.95 total 32.08 last 884.20) - /unlink 38061 (time 1746626861.95 total 42.08 last 921.81) - /unlink 40000 (time 1746626864.39 total 44.52 last 796.47) - /unlink 48883 (time 1746626874.39 total 54.52 last 888.27) - /unlink 58057 (time 1746626884.39 total 64.52 last 917.31) - /unlink 60000 (time 1746626886.56 total 66.69 last 895.59) total: 64768 /unlink in 72.18 seconds: 897.36 ops/second Waiting for MDT destroys to complete - open/close 4803 (time 1746626988.91 total 10.00 last 480.22) - open/close 9386 (time 1746626998.91 total 20.00 last 458.26) - open/close 13543 (time 1746627008.91 total 30.00 last 415.66) - open/close 17799 (time 1746627018.91 total 40.00 last 425.58) - open/close 20000 (time 1746627023.17 total 44.26 last 516.61) - open/close 24737 (time 1746627033.18 total 54.27 last 473.64) - open/close 30000 (time 1746627042.79 total 63.88 last 547.34) - open/close 34532 (time 1746627052.79 total 73.88 last 453.14) - open/close 38882 (time 1746627062.79 total 83.88 last 434.91) - open/close 40000 (time 1746627065.39 total 86.48 last 431.14) - open/close 44158 (time 1746627075.39 total 96.48 last 415.78) - open/close 49302 (time 1746627085.39 total 106.48 last 514.39) - open/close 54491 (time 1746627095.39 total 116.48 last 518.89) - open/close 59351 (time 1746627105.39 total 126.48 last 485.98) - open/close 64364 (time 1746627115.39 total 136.48 last 501.28) total: 64768 open/close in 137.25 seconds: 471.91 ops/second - /unlink 8927 (time 1746627127.04 total 10.00 last 892.63) - /unlink 17821 (time 1746627137.04 total 20.00 last 889.36) - /unlink 20000 (time 1746627139.45 total 22.40 last 907.25) - /unlink 29317 (time 1746627149.45 total 32.40 last 931.62) - /unlink 38329 (time 1746627159.45 total 42.41 last 901.06) - /unlink 46245 (time 1746627169.45 total 52.41 last 791.53) - /unlink 50000 (time 1746627173.76 total 56.72 last 870.11) - /unlink 59108 (time 1746627183.76 total 66.72 last 910.79) total: 64768 /unlink in 72.65 seconds: 891.50 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 PASS 106 (1666s) Stopping clients: oleg152-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg152-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 oleg152-server: oleg152-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1769 sec ========== 10:15:18 (1746627318) === conf-sanity: start cleanup 10:15:18 (1746627318) === === conf-sanity: finish cleanup 10:15:19 (1746627319) ===