-----============= acceptance-small: conf-sanity ============----- Fri Mar 28 16:40:55 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg441-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg441-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg441-server oleg441-server: oleg441-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions oleg441-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg441-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg441-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg441-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg441-server: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg441-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg441-server: oleg441-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg441-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:42:26 (1743194546) Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f) oleg441-server: oleg441-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg441-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg441-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg441-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg441-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg441-server: oleg441-server.virtnet: executing set_default_debug -1 all pdsh@oleg441-client: oleg441-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg441-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg441-client.virtnet: -o user_xattr,flock 192.168.204.141@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4678 (time 1743194588.74 total 10.00 last 467.79) - open/close 8033 (time 1743194598.75 total 20.00 last 335.40) - open/close 10000 (time 1743194602.86 total 24.12 last 477.79) - open/close 13938 (time 1743194612.87 total 34.12 last 393.78) - open/close 17301 (time 1743194622.87 total 44.12 last 336.29) - open/close 20000 (time 1743194630.13 total 51.38 last 371.78) - open/close 23583 (time 1743194640.13 total 61.38 last 358.19) - open/close 28911 (time 1743194650.13 total 71.38 last 532.72) - open/close 30000 (time 1743194652.15 total 73.41 last 538.82) - open/close 34741 (time 1743194662.15 total 83.41 last 473.95) - open/close 39426 (time 1743194672.15 total 93.41 last 468.46) - open/close 44691 (time 1743194682.16 total 103.41 last 526.46) - open/close 49308 (time 1743194692.16 total 113.41 last 461.55) - open/close 54150 (time 1743194702.16 total 123.41 last 484.19) - open/close 59312 (time 1743194712.16 total 133.41 last 516.18) - open/close 64747 (time 1743194722.16 total 143.42 last 543.44) total: 64768 open/close in 143.45 seconds: 451.49 ops/second - /unlink 9761 (time 1743194733.03 total 10.00 last 976.08) - /unlink 19751 (time 1743194743.03 total 20.00 last 999.00) - /unlink 29877 (time 1743194753.03 total 30.00 last 1012.51) - /unlink 39871 (time 1743194763.03 total 40.00 last 999.34) - /unlink 49819 (time 1743194773.03 total 50.00 last 994.76) - /unlink 59798 (time 1743194783.03 total 60.00 last 997.87) total: 64768 /unlink in 64.78 seconds: 999.74 ops/second Waiting for MDT destroys to complete - open/close 5721 (time 1743194896.39 total 10.00 last 572.09) - open/close 10000 (time 1743194903.93 total 17.54 last 567.61) - open/close 15558 (time 1743194913.93 total 27.54 last 555.72) - open/close 20000 (time 1743194921.84 total 35.45 last 561.75) - open/close 25445 (time 1743194931.84 total 45.45 last 544.43) - open/close 30000 (time 1743194940.15 total 53.76 last 548.11) - open/close 35562 (time 1743194950.15 total 63.76 last 556.17) - open/close 40000 (time 1743194958.05 total 71.66 last 561.73) - open/close 45644 (time 1743194968.05 total 81.66 last 564.34) - open/close 50000 (time 1743194975.95 total 89.56 last 551.72) - open/close 55454 (time 1743194985.95 total 99.56 last 545.36) - open/close 60000 (time 1743194994.20 total 107.81 last 550.78) total: 64768 open/close in 116.66 seconds: 555.18 ops/second - /unlink 9642 (time 1743195013.92 total 10.00 last 964.20) - /unlink 19691 (time 1743195023.92 total 20.00 last 1004.85) - /unlink 29670 (time 1743195033.92 total 30.00 last 997.77) - /unlink 39720 (time 1743195043.92 total 40.00 last 1004.92) - /unlink 49889 (time 1743195053.92 total 50.00 last 1016.80) - /unlink 60000 (time 1743195063.83 total 59.91 last 1020.79) total: 64768 /unlink in 64.57 seconds: 1003.04 ops/second Waiting for MDT destroys to complete - open/close 4808 (time 1743195177.81 total 10.00 last 480.80) - open/close 9731 (time 1743195187.81 total 20.00 last 492.22) - open/close 14613 (time 1743195197.81 total 30.00 last 488.14) - open/close 20000 (time 1743195207.59 total 39.79 last 550.62) - open/close 23644 (time 1743195217.59 total 49.79 last 364.33) - open/close 26711 (time 1743195227.60 total 59.79 last 306.66) - open/close 30000 (time 1743195236.65 total 68.85 last 363.09) - open/close 34158 (time 1743195246.65 total 78.85 last 415.80) - open/close 37590 (time 1743195256.66 total 88.85 last 343.10) - open/close 40000 (time 1743195262.50 total 94.69 last 412.39) - open/close 44446 (time 1743195272.50 total 104.69 last 444.59) - open/close 49026 (time 1743195282.50 total 114.70 last 457.91) - open/close 50000 (time 1743195285.10 total 117.29 last 375.74) - open/close 53763 (time 1743195295.10 total 127.29 last 376.20) - open/close 56992 (time 1743195305.10 total 137.29 last 322.80) - open/close 60000 (time 1743195313.00 total 145.19 last 380.95) - open/close 63152 (time 1743195323.00 total 155.19 last 315.14) total: 64768 open/close in 160.24 seconds: 404.19 ops/second - /unlink 6731 (time 1743195339.25 total 10.00 last 673.04) - /unlink 10000 (time 1743195344.05 total 14.80 last 681.59) - /unlink 17645 (time 1743195354.05 total 24.80 last 764.49) - /unlink 20000 (time 1743195357.58 total 28.33 last 666.85) - /unlink 27248 (time 1743195367.58 total 38.33 last 724.66) - /unlink 30000 (time 1743195372.67 total 43.42 last 540.98) - /unlink 35459 (time 1743195382.67 total 53.42 last 545.88) - /unlink 40000 (time 1743195389.90 total 60.65 last 627.83) - /unlink 47282 (time 1743195399.90 total 70.65 last 728.14) - /unlink 50000 (time 1743195403.52 total 74.27 last 751.86) - /unlink 55771 (time 1743195413.52 total 84.27 last 577.08) - /unlink 60000 (time 1743195420.59 total 91.34 last 598.00) total: 64768 /unlink in 100.18 seconds: 646.52 ops/second Waiting for MDT destroys to complete - open/close 3567 (time 1743195487.69 total 10.00 last 356.61) - open/close 6487 (time 1743195497.69 total 20.00 last 291.97) - open/close 9328 (time 1743195507.69 total 30.00 last 284.09) - open/close 10000 (time 1743195510.14 total 32.45 last 274.43) - open/close 13483 (time 1743195520.14 total 42.45 last 348.29) - open/close 18611 (time 1743195530.14 total 52.45 last 512.80) - open/close 20000 (time 1743195532.81 total 55.13 last 519.78) - open/close 25178 (time 1743195542.81 total 65.13 last 517.79) - open/close 29820 (time 1743195552.81 total 75.13 last 464.16) - open/close 34705 (time 1743195562.81 total 85.13 last 488.45) - open/close 39735 (time 1743195572.81 total 95.13 last 502.91) - open/close 44623 (time 1743195582.82 total 105.13 last 488.76) - open/close 49566 (time 1743195592.82 total 115.13 last 494.29) - open/close 54564 (time 1743195602.82 total 125.13 last 499.76) - open/close 59557 (time 1743195612.82 total 135.13 last 499.26) - open/close 64511 (time 1743195622.82 total 145.13 last 495.34) total: 64768 open/close in 145.68 seconds: 444.60 ops/second - /unlink 8708 (time 1743195634.14 total 10.00 last 870.78) - /unlink 17799 (time 1743195644.14 total 20.00 last 909.06) - /unlink 20000 (time 1743195646.55 total 22.41 last 914.80) - /unlink 29370 (time 1743195656.55 total 32.41 last 936.94) - /unlink 38738 (time 1743195666.55 total 42.41 last 936.74) - /unlink 48337 (time 1743195676.55 total 52.41 last 959.81) - /unlink 57522 (time 1743195686.55 total 62.41 last 918.42) - /unlink 60000 (time 1743195689.18 total 65.04 last 941.90) total: 64768 /unlink in 70.00 seconds: 925.28 ops/second Waiting for MDT destroys to complete - open/close 3108 (time 1743195794.25 total 10.00 last 310.80) - open/close 6418 (time 1743195804.25 total 20.00 last 330.98) - open/close 9093 (time 1743195814.25 total 30.00 last 267.45) - open/close 10000 (time 1743195816.76 total 32.51 last 361.73) - open/close 12771 (time 1743195826.76 total 42.51 last 277.07) - open/close 15676 (time 1743195836.76 total 52.51 last 290.41) - open/close 19297 (time 1743195846.76 total 62.51 last 362.10) - open/close 20000 (time 1743195849.15 total 64.90 last 294.49) - open/close 22844 (time 1743195859.15 total 74.90 last 284.35) - open/close 26638 (time 1743195869.15 total 84.91 last 379.29) - open/close 29504 (time 1743195879.15 total 94.91 last 286.58) - open/close 33009 (time 1743195889.16 total 104.91 last 350.48) - open/close 37140 (time 1743195899.16 total 114.91 last 413.04) - open/close 40000 (time 1743195907.73 total 123.48 last 333.67) - open/close 42706 (time 1743195917.73 total 133.48 last 270.52) - open/close 45465 (time 1743195927.73 total 143.48 last 275.89) - open/close 49768 (time 1743195937.73 total 153.48 last 430.29) - open/close 52950 (time 1743195947.73 total 163.48 last 318.20) - open/close 56456 (time 1743195957.73 total 173.48 last 350.58) - open/close 59611 (time 1743195967.74 total 183.49 last 315.41) - open/close 62374 (time 1743195977.74 total 193.49 last 276.30) total: 64768 open/close in 199.49 seconds: 324.67 ops/second - /unlink 7922 (time 1743195994.59 total 10.00 last 792.18) - /unlink 10000 (time 1743195997.99 total 13.41 last 610.09) - /unlink 16627 (time 1743196007.99 total 23.41 last 662.66) - /unlink 20000 (time 1743196012.05 total 27.47 last 830.49) - /unlink 27402 (time 1743196022.06 total 37.47 last 740.10) - /unlink 30000 (time 1743196026.91 total 42.32 last 535.19) - /unlink 35911 (time 1743196036.91 total 52.32 last 591.06) - /unlink 40000 (time 1743196043.77 total 59.18 last 596.59) - /unlink 45130 (time 1743196053.77 total 69.18 last 512.99) - /unlink 50000 (time 1743196061.48 total 76.89 last 631.67) - /unlink 56824 (time 1743196071.48 total 86.89 last 682.40) - /unlink 60000 (time 1743196075.05 total 90.46 last 888.58) total: 64768 /unlink in 97.33 seconds: 665.43 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg441-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg441-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg441-server stop mds service on oleg441-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg441-server unloading modules via unload_modules_local on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-client: ssh exited with exit code 2 pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 PASS 106 (1611s) Stopping clients: oleg441-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg441-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg441-client: oleg441-server: ssh exited with exit code 2 oleg441-server: oleg441-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg441-server' oleg441-server: oleg441-server.virtnet: executing load_modules_local oleg441-server: Loading modules from /home/green/git/lustre-release/lustre oleg441-server: detected 4 online CPUs by sysfs oleg441-server: MODOPTS_LIBCFS= oleg441-server: Force libcfs to create 2 CPU partitions oleg441-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg441-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg441-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1724 sec ========== 17:09:40 (1743196180) === conf-sanity: start cleanup 17:09:41 (1743196181) === === conf-sanity: finish cleanup 17:09:42 (1743196182) ===