-----============= acceptance-small: conf-sanity ============----- Tue Apr 8 23:44:36 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg348-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg348-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg348-server oleg348-server: oleg348-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg348-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg348-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg348-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg348-server: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg348-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg348-server: oleg348-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg348-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:46:03 (1744170363) Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:-f) oleg348-server: oleg348-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg348-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg348-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg348-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg348-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg348-server: oleg348-server.virtnet: executing set_default_debug -1 all pdsh@oleg348-client: oleg348-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg348-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg348-client.virtnet: -o user_xattr,flock 192.168.203.148@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 279 79713 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 257 79735 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65976 536 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4381 (time 1744170403.96 total 10.00 last 438.03) - open/close 8988 (time 1744170413.96 total 20.00 last 460.70) - open/close 10000 (time 1744170415.97 total 22.00 last 505.16) - open/close 14827 (time 1744170425.97 total 32.01 last 482.62) - open/close 19056 (time 1744170435.97 total 42.01 last 422.86) - open/close 20000 (time 1744170438.05 total 44.09 last 453.07) - open/close 25558 (time 1744170448.06 total 54.09 last 555.71) - open/close 30000 (time 1744170456.92 total 62.96 last 501.12) - open/close 35479 (time 1744170466.92 total 72.96 last 547.89) - open/close 40000 (time 1744170475.74 total 81.77 last 512.75) - open/close 44196 (time 1744170485.74 total 91.77 last 419.57) - open/close 48248 (time 1744170495.74 total 101.78 last 405.15) - open/close 50000 (time 1744170499.90 total 105.93 last 421.53) - open/close 54193 (time 1744170509.90 total 115.93 last 419.25) - open/close 58074 (time 1744170519.90 total 125.93 last 388.07) - open/close 60000 (time 1744170524.40 total 130.43 last 428.17) - open/close 63829 (time 1744170534.40 total 140.43 last 382.85) total: 64768 open/close in 142.51 seconds: 454.49 ops/second - /unlink 8340 (time 1744170547.34 total 10.00 last 833.94) - /unlink 17230 (time 1744170557.34 total 20.00 last 888.98) - /unlink 20000 (time 1744170560.20 total 22.86 last 968.80) - /unlink 29222 (time 1744170570.20 total 32.86 last 922.15) - /unlink 39389 (time 1744170580.20 total 42.86 last 1016.65) - /unlink 48783 (time 1744170590.20 total 52.86 last 939.32) - /unlink 58233 (time 1744170600.20 total 62.86 last 944.93) total: 64768 /unlink in 69.74 seconds: 928.71 ops/second Waiting for MDT destroys to complete - open/close 5565 (time 1744170706.46 total 10.00 last 556.48) - open/close 10000 (time 1744170714.52 total 18.06 last 550.36) - open/close 15625 (time 1744170724.52 total 28.06 last 562.50) - open/close 20000 (time 1744170732.94 total 36.48 last 519.63) - open/close 25355 (time 1744170742.94 total 46.48 last 535.48) - open/close 30000 (time 1744170751.63 total 55.17 last 534.50) - open/close 35526 (time 1744170761.63 total 65.17 last 552.52) - open/close 40000 (time 1744170769.65 total 73.19 last 557.94) - open/close 45359 (time 1744170779.65 total 83.19 last 535.83) - open/close 50000 (time 1744170788.48 total 92.02 last 525.85) - open/close 55430 (time 1744170798.48 total 102.02 last 542.99) - open/close 60000 (time 1744170806.89 total 110.43 last 543.27) total: 64768 open/close in 119.31 seconds: 542.85 ops/second - /unlink 9580 (time 1744170826.64 total 10.00 last 957.97) - /unlink 19590 (time 1744170836.64 total 20.00 last 1000.95) - /unlink 29030 (time 1744170846.64 total 30.00 last 943.93) - /unlink 38925 (time 1744170856.64 total 40.00 last 989.41) - /unlink 49101 (time 1744170866.64 total 50.00 last 1017.59) - /unlink 59270 (time 1744170876.64 total 60.00 last 1016.86) total: 64768 /unlink in 65.65 seconds: 986.51 ops/second Waiting for MDT destroys to complete - open/close 5270 (time 1744170983.40 total 10.00 last 526.98) - open/close 10000 (time 1744170992.48 total 19.08 last 521.16) - open/close 14669 (time 1744171002.48 total 29.08 last 466.83) - open/close 19819 (time 1744171012.48 total 39.08 last 514.93) - open/close 24154 (time 1744171022.48 total 49.08 last 433.50) - open/close 29442 (time 1744171032.48 total 59.08 last 528.75) - open/close 34824 (time 1744171042.48 total 69.08 last 538.18) - open/close 40000 (time 1744171052.19 total 78.79 last 533.06) - open/close 45274 (time 1744171062.19 total 88.79 last 527.39) - open/close 50000 (time 1744171071.65 total 98.25 last 499.37) - open/close 54165 (time 1744171081.66 total 108.26 last 416.42) - open/close 57643 (time 1744171091.66 total 118.26 last 347.78) - open/close 60000 (time 1744171097.77 total 124.37 last 385.46) - open/close 62942 (time 1744171107.78 total 134.38 last 294.09) total: 64768 open/close in 138.82 seconds: 466.57 ops/second - /unlink 7677 (time 1744171123.17 total 10.00 last 767.69) - /unlink 10000 (time 1744171125.90 total 12.72 last 852.53) - /unlink 19338 (time 1744171135.90 total 22.72 last 933.79) - /unlink 27281 (time 1744171145.90 total 32.73 last 794.23) - /unlink 30000 (time 1744171149.62 total 36.45 last 730.13) - /unlink 36036 (time 1744171159.62 total 46.45 last 603.57) - /unlink 40000 (time 1744171164.58 total 51.41 last 799.00) - /unlink 49140 (time 1744171174.58 total 61.41 last 913.92) - /unlink 58455 (time 1744171184.58 total 71.41 last 931.47) total: 64768 /unlink in 77.73 seconds: 833.19 ops/second Waiting for MDT destroys to complete - open/close 3858 (time 1744171278.49 total 10.00 last 385.76) - open/close 8142 (time 1744171288.49 total 20.00 last 428.30) - open/close 10000 (time 1744171293.41 total 24.93 last 377.14) - open/close 14389 (time 1744171303.42 total 34.93 last 438.84) - open/close 18057 (time 1744171313.42 total 44.93 last 366.75) - open/close 20000 (time 1744171319.62 total 51.14 last 313.02) - open/close 24220 (time 1744171329.63 total 61.14 last 421.91) - open/close 28989 (time 1744171339.63 total 71.14 last 476.84) - open/close 30000 (time 1744171341.98 total 73.50 last 429.15) - open/close 33866 (time 1744171351.99 total 83.50 last 386.49) - open/close 37682 (time 1744171361.99 total 93.50 last 381.57) - open/close 40000 (time 1744171369.26 total 100.77 last 318.94) - open/close 43126 (time 1744171379.26 total 110.77 last 312.52) - open/close 46662 (time 1744171389.26 total 120.77 last 353.55) - open/close 50000 (time 1744171398.98 total 130.49 last 343.50) - open/close 53633 (time 1744171408.98 total 140.49 last 363.27) - open/close 56506 (time 1744171418.98 total 150.49 last 287.28) - open/close 59148 (time 1744171428.98 total 160.49 last 264.20) - open/close 60000 (time 1744171432.31 total 163.83 last 255.64) - open/close 64017 (time 1744171442.31 total 173.83 last 401.65) total: 64768 open/close in 175.86 seconds: 368.29 ops/second - /unlink 5223 (time 1744171455.52 total 10.00 last 522.22) - /unlink 10000 (time 1744171463.30 total 17.78 last 614.19) - /unlink 15061 (time 1744171473.30 total 27.78 last 506.08) - /unlink 20000 (time 1744171483.11 total 37.59 last 503.48) - /unlink 25188 (time 1744171493.11 total 47.59 last 518.79) - /unlink 30000 (time 1744171502.35 total 56.83 last 520.75) - /unlink 35419 (time 1744171512.35 total 66.83 last 541.80) - /unlink 40000 (time 1744171520.47 total 74.95 last 564.30) - /unlink 45352 (time 1744171530.47 total 84.95 last 535.09) - /unlink 50000 (time 1744171539.67 total 94.16 last 504.87) - /unlink 55301 (time 1744171549.68 total 104.16 last 530.00) - /unlink 60000 (time 1744171557.35 total 111.83 last 612.38) total: 64768 /unlink in 120.52 seconds: 537.39 ops/second Waiting for MDT destroys to complete - open/close 2828 (time 1744171584.25 total 10.00 last 282.77) - open/close 6289 (time 1744171594.25 total 20.00 last 346.06) - open/close 9238 (time 1744171604.25 total 30.00 last 294.90) - open/close 10000 (time 1744171606.74 total 32.49 last 305.78) - open/close 13092 (time 1744171616.74 total 42.49 last 309.18) - open/close 15991 (time 1744171626.74 total 52.50 last 289.86) - open/close 18921 (time 1744171636.74 total 62.50 last 292.92) - open/close 20000 (time 1744171639.92 total 65.67 last 340.07) - open/close 22593 (time 1744171649.92 total 75.67 last 259.24) - open/close 25339 (time 1744171659.92 total 85.68 last 274.58) - open/close 27841 (time 1744171669.92 total 95.68 last 250.14) - open/close 30000 (time 1744171678.76 total 104.52 last 244.19) - open/close 32705 (time 1744171688.77 total 114.52 last 270.47) - open/close 35545 (time 1744171698.77 total 124.52 last 283.94) - open/close 38985 (time 1744171708.77 total 134.52 last 343.91) - open/close 40000 (time 1744171711.19 total 136.94 last 420.08) - open/close 42608 (time 1744171721.19 total 146.94 last 260.78) - open/close 45222 (time 1744171731.19 total 156.94 last 261.36) - open/close 48046 (time 1744171741.19 total 166.94 last 282.40) - open/close 50000 (time 1744171748.62 total 174.37 last 263.06) - open/close 52655 (time 1744171758.62 total 184.37 last 265.46) - open/close 55423 (time 1744171768.62 total 194.37 last 276.76) - open/close 58363 (time 1744171778.62 total 204.38 last 293.95) - open/close 60000 (time 1744171784.67 total 210.42 last 270.66) - open/close 63803 (time 1744171794.67 total 220.43 last 380.25) total: 64768 open/close in 223.80 seconds: 289.40 ops/second - /unlink 5616 (time 1744171809.25 total 10.00 last 561.58) - /unlink 10000 (time 1744171818.23 total 18.98 last 488.38) - /unlink 16308 (time 1744171828.23 total 28.98 last 630.74) - /unlink 20000 (time 1744171836.15 total 36.89 last 466.46) - /unlink 24761 (time 1744171846.15 total 46.89 last 476.09) - /unlink 29984 (time 1744171856.15 total 56.89 last 522.28) - /unlink 35366 (time 1744171866.15 total 66.89 last 538.15) - /unlink 40000 (time 1744171875.54 total 76.28 last 493.59) - /unlink 45383 (time 1744171885.54 total 86.28 last 538.24) - /unlink 50000 (time 1744171891.35 total 92.10 last 794.03) - /unlink 59287 (time 1744171901.35 total 102.10 last 928.62) total: 64768 /unlink in 108.03 seconds: 599.51 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg348-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg348-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg348-server stop mds service on oleg348-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg348-server unloading modules via unload_modules_local on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-client: ssh exited with exit code 2 pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 PASS 106 (1594s) Stopping clients: oleg348-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg348-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg348-client: oleg348-server: ssh exited with exit code 2 oleg348-server: oleg348-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg348-server' oleg348-server: oleg348-server.virtnet: executing load_modules_local oleg348-server: Loading modules from /home/green/git/lustre-release/lustre oleg348-server: detected 4 online CPUs by sysfs oleg348-server: MODOPTS_LIBCFS= oleg348-server: Force libcfs to create 2 CPU partitions oleg348-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg348-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg348-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg348-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg348-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1703 sec ========== 00:12:59 (1744171979) === conf-sanity: start cleanup 00:12:59 (1744171979) === === conf-sanity: finish cleanup 00:13:00 (1744171980) ===