-----============= acceptance-small: conf-sanity ============----- Mon Jun 9 10:12:36 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg310-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg310-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg310-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg310-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg310-server oleg310-server: oleg310-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions oleg310-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg310-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg310-server: oleg310-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg310-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:13:32 (1749478412) Stopping clients: oleg310-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg310-client.virtnet /mnt/lustre2 (opts:-f) oleg310-server: oleg310-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg310-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5359 (time 1749478447.51 total 10.00 last 535.87) - open/close 10000 (time 1749478456.47 total 18.97 last 517.62) - open/close 16286 (time 1749478466.47 total 28.97 last 628.54) - open/close 20000 (time 1749478472.74 total 35.23 last 593.00) - open/close 25608 (time 1749478482.74 total 45.23 last 560.78) - open/close 30000 (time 1749478489.92 total 52.41 last 611.53) - open/close 35432 (time 1749478499.92 total 62.41 last 543.10) - open/close 40000 (time 1749478508.56 total 71.05 last 528.83) - open/close 45321 (time 1749478518.56 total 81.05 last 532.09) - open/close 50000 (time 1749478527.46 total 89.95 last 525.89) - open/close 55237 (time 1749478537.46 total 99.95 last 523.69) - open/close 60000 (time 1749478546.67 total 109.16 last 517.16) total: 64768 open/close in 118.44 seconds: 546.84 ops/second - /unlink 8696 (time 1749478566.80 total 10.00 last 869.50) - /unlink 17129 (time 1749478576.80 total 20.00 last 843.24) - /unlink 20000 (time 1749478581.45 total 24.65 last 617.90) - /unlink 27894 (time 1749478591.45 total 34.65 last 789.33) - /unlink 30000 (time 1749478593.86 total 37.07 last 871.62) - /unlink 38789 (time 1749478603.86 total 47.07 last 878.80) - /unlink 47696 (time 1749478613.86 total 57.07 last 890.70) - /unlink 50000 (time 1749478616.47 total 59.67 last 884.03) - /unlink 59087 (time 1749478626.47 total 69.67 last 908.69) total: 64768 /unlink in 75.76 seconds: 854.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5420 (time 1749478666.24 total 10.04 last 539.77) - open/close 10000 (time 1749478674.75 total 18.56 last 537.94) - open/close 15129 (time 1749478684.75 total 28.56 last 512.87) - open/close 20000 (time 1749478694.26 total 38.06 last 512.42) - open/close 25271 (time 1749478704.26 total 48.06 last 527.08) - open/close 30000 (time 1749478713.21 total 57.01 last 528.46) - open/close 35125 (time 1749478723.21 total 67.01 last 512.45) - open/close 40000 (time 1749478732.66 total 76.47 last 515.67) - open/close 45174 (time 1749478742.67 total 86.47 last 517.31) - open/close 50000 (time 1749478752.64 total 96.44 last 483.71) - open/close 54270 (time 1749478762.65 total 106.45 last 426.89) - open/close 59085 (time 1749478772.65 total 116.45 last 481.42) - open/close 64228 (time 1749478782.65 total 126.45 last 514.25) total: 64768 open/close in 127.57 seconds: 507.72 ops/second - /unlink 8846 (time 1749478794.60 total 10.00 last 884.52) - /unlink 17726 (time 1749478804.61 total 20.00 last 887.88) - /unlink 20000 (time 1749478807.03 total 22.43 last 937.83) - /unlink 28873 (time 1749478817.03 total 32.43 last 887.25) - /unlink 38007 (time 1749478827.03 total 42.43 last 913.37) - /unlink 40000 (time 1749478829.26 total 44.66 last 893.00) - /unlink 49072 (time 1749478839.26 total 54.66 last 907.17) - /unlink 58302 (time 1749478849.26 total 64.66 last 922.92) total: 64768 /unlink in 71.57 seconds: 905.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5275 (time 1749478891.40 total 10.00 last 527.45) - open/close 9168 (time 1749478901.40 total 20.00 last 389.29) - open/close 10000 (time 1749478903.66 total 22.26 last 368.78) - open/close 14807 (time 1749478913.66 total 32.26 last 480.61) - open/close 18810 (time 1749478923.66 total 42.26 last 400.25) - open/close 20000 (time 1749478927.16 total 45.75 last 340.61) - open/close 23170 (time 1749478937.16 total 55.76 last 316.93) - open/close 27012 (time 1749478947.16 total 65.76 last 384.12) - open/close 30000 (time 1749478953.35 total 71.95 last 482.77) - open/close 34891 (time 1749478963.35 total 81.95 last 489.05) - open/close 39380 (time 1749478973.35 total 91.95 last 448.85) - open/close 44525 (time 1749478983.35 total 101.95 last 514.45) - open/close 49155 (time 1749478993.35 total 111.95 last 462.97) - open/close 53650 (time 1749479003.36 total 121.95 last 449.36) - open/close 58710 (time 1749479013.36 total 131.96 last 505.91) - open/close 60000 (time 1749479016.77 total 135.37 last 377.74) - open/close 63753 (time 1749479026.78 total 145.37 last 375.23) total: 64768 open/close in 147.49 seconds: 439.14 ops/second - /unlink 7994 (time 1749479039.68 total 10.00 last 799.32) - /unlink 10000 (time 1749479042.06 total 12.39 last 840.86) - /unlink 18244 (time 1749479052.06 total 22.39 last 824.36) - /unlink 20000 (time 1749479054.44 total 24.76 last 740.29) - /unlink 27020 (time 1749479064.44 total 34.76 last 701.98) - /unlink 30000 (time 1749479068.33 total 38.66 last 764.62) - /unlink 39841 (time 1749479078.33 total 48.66 last 984.06) - /unlink 48750 (time 1749479088.33 total 58.66 last 890.89) - /unlink 57187 (time 1749479098.33 total 68.66 last 843.69) - /unlink 60000 (time 1749479101.72 total 72.04 last 830.76) total: 64768 /unlink in 77.41 seconds: 836.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5193 (time 1749479141.61 total 10.00 last 519.19) - open/close 10000 (time 1749479151.23 total 19.63 last 499.47) - open/close 14780 (time 1749479161.24 total 29.63 last 477.93) - open/close 19391 (time 1749479171.24 total 39.63 last 461.04) - open/close 24005 (time 1749479181.24 total 49.63 last 461.39) - open/close 28589 (time 1749479191.24 total 59.63 last 458.36) - open/close 30000 (time 1749479194.17 total 62.56 last 481.09) - open/close 34665 (time 1749479204.17 total 72.56 last 466.45) - open/close 39020 (time 1749479214.19 total 82.58 last 434.68) - open/close 40000 (time 1749479216.28 total 84.67 last 469.99) - open/close 44782 (time 1749479226.28 total 94.67 last 478.13) - open/close 49413 (time 1749479236.28 total 104.67 last 463.03) - open/close 54198 (time 1749479246.28 total 114.67 last 478.45) - open/close 58996 (time 1749479256.28 total 124.67 last 479.74) - open/close 60000 (time 1749479258.57 total 126.96 last 439.59) total: 64768 open/close in 136.46 seconds: 474.63 ops/second - /unlink 8665 (time 1749479278.97 total 10.00 last 866.42) - /unlink 17222 (time 1749479288.97 total 20.00 last 855.65) - /unlink 20000 (time 1749479292.20 total 23.23 last 859.38) - /unlink 28333 (time 1749479302.20 total 33.23 last 833.26) - /unlink 36937 (time 1749479312.20 total 43.23 last 860.38) - /unlink 40000 (time 1749479315.84 total 46.87 last 842.56) - /unlink 48718 (time 1749479325.84 total 56.87 last 871.77) - /unlink 57256 (time 1749479335.84 total 66.87 last 853.70) - /unlink 60000 (time 1749479338.84 total 69.87 last 915.77) total: 64768 /unlink in 75.11 seconds: 862.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5279 (time 1749479377.40 total 10.00 last 527.80) - open/close 10000 (time 1749479386.31 total 18.91 last 530.03) - open/close 15266 (time 1749479396.31 total 28.91 last 526.58) - open/close 20000 (time 1749479405.32 total 37.92 last 525.25) - open/close 24903 (time 1749479415.32 total 47.92 last 490.25) - open/close 30000 (time 1749479425.15 total 57.75 last 518.75) - open/close 35355 (time 1749479435.15 total 67.75 last 535.48) - open/close 39821 (time 1749479445.15 total 77.75 last 446.53) - open/close 43876 (time 1749479455.15 total 87.75 last 405.46) - open/close 48094 (time 1749479465.15 total 97.75 last 421.79) - open/close 50000 (time 1749479468.67 total 101.28 last 540.75) - open/close 55022 (time 1749479478.68 total 111.28 last 502.13) - open/close 59861 (time 1749479488.68 total 121.28 last 483.80) total: 64768 open/close in 130.55 seconds: 496.11 ops/second - /unlink 8836 (time 1749479508.79 total 10.00 last 883.59) - /unlink 17902 (time 1749479518.79 total 20.00 last 906.58) - /unlink 20000 (time 1749479521.31 total 22.52 last 832.20) - /unlink 28872 (time 1749479531.31 total 32.52 last 887.18) - /unlink 37238 (time 1749479541.31 total 42.52 last 836.52) - /unlink 40000 (time 1749479544.30 total 45.51 last 925.19) - /unlink 48840 (time 1749479554.30 total 55.51 last 883.96) - /unlink 57052 (time 1749479564.30 total 65.51 last 821.09) - /unlink 60000 (time 1749479567.45 total 68.66 last 936.66) total: 64768 /unlink in 75.24 seconds: 860.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 PASS 106 (1205s) Stopping clients: oleg310-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg310-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 oleg310-server: oleg310-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions oleg310-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg310-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1278 sec ========== 10:33:55 (1749479635) === conf-sanity: start cleanup 10:33:55 (1749479635) === === conf-sanity: finish cleanup 10:33:56 (1749479636) ===