-----============= acceptance-small: conf-sanity ============----- Thu Jun 5 07:07:08 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg116-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg116-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg116-server oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg116-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg116-server: oleg116-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg116-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:08:47 (1749121727) Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg116-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg116-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg116-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg116-server: oleg116-server.virtnet: executing set_default_debug -1 all pdsh@oleg116-client: oleg116-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg116-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg116-client.virtnet: -o user_xattr,flock 192.168.201.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 999 (time 1749121792.84 total 10.00 last 99.89) - open/close 1971 (time 1749121802.85 total 20.01 last 97.13) - open/close 2975 (time 1749121812.86 total 30.02 last 100.32) - open/close 3944 (time 1749121822.86 total 40.02 last 96.86) - open/close 4898 (time 1749121832.89 total 50.04 last 95.17) - open/close 5867 (time 1749121842.89 total 60.05 last 96.86) - open/close 6960 (time 1749121852.90 total 70.05 last 109.22) - open/close 8065 (time 1749121862.90 total 80.06 last 110.47) - open/close 9138 (time 1749121872.91 total 90.07 last 107.19) - open/close 10000 (time 1749121881.33 total 98.49 last 102.40) - open/close 11111 (time 1749121891.34 total 108.49 last 111.01) - open/close 12164 (time 1749121901.34 total 118.50 last 105.25) - open/close 13223 (time 1749121911.34 total 128.50 last 105.89) - open/close 14257 (time 1749121921.34 total 138.50 last 103.38) - open/close 15287 (time 1749121931.35 total 148.51 last 102.93) - open/close 16338 (time 1749121941.36 total 158.51 last 105.03) - open/close 17410 (time 1749121951.37 total 168.52 last 107.09) - open/close 18598 (time 1749121961.39 total 178.54 last 118.59) - open/close 19786 (time 1749121971.39 total 188.55 last 118.72) - open/close 20925 (time 1749121981.40 total 198.55 last 113.87) - open/close 22097 (time 1749121991.40 total 208.56 last 117.13) - open/close 23288 (time 1749122001.41 total 218.56 last 119.05) - open/close 24532 (time 1749122011.41 total 228.56 last 124.39) - open/close 25810 (time 1749122021.41 total 238.57 last 127.78) - open/close 27004 (time 1749122031.41 total 248.57 last 119.38) - open/close 28212 (time 1749122041.41 total 258.57 last 120.77) - open/close 29601 (time 1749122051.41 total 268.57 last 138.89) - open/close 30000 (time 1749122054.27 total 271.42 last 139.88) - open/close 31336 (time 1749122064.27 total 281.42 last 133.59) - open/close 32643 (time 1749122074.27 total 291.42 last 130.69) - open/close 34140 (time 1749122084.27 total 301.43 last 149.66) - open/close 35593 (time 1749122094.27 total 311.43 last 145.29) - open/close 37067 (time 1749122104.27 total 321.43 last 147.36) - open/close 38549 (time 1749122114.27 total 331.43 last 148.20) - open/close 39974 (time 1749122124.27 total 341.43 last 142.48) - open/close 41497 (time 1749122134.28 total 351.44 last 152.22) - open/close 43108 (time 1749122144.29 total 361.44 last 160.97) - open/close 44696 (time 1749122154.29 total 371.44 last 158.79) - open/close 46116 (time 1749122164.29 total 381.45 last 141.92) - open/close 47469 (time 1749122174.29 total 391.45 last 135.28) - open/close 48494 (time 1749122184.30 total 401.45 last 102.49) - open/close 49714 (time 1749122194.30 total 411.46 last 121.96) - open/close 50000 (time 1749122196.45 total 413.60 last 133.08) - open/close 51411 (time 1749122206.45 total 423.61 last 141.08) - open/close 52916 (time 1749122216.45 total 433.61 last 150.49) - open/close 54483 (time 1749122226.46 total 443.62 last 156.47) - open/close 56091 (time 1749122236.46 total 453.62 last 160.79) - open/close 57795 (time 1749122246.46 total 463.62 last 170.40) - open/close 59527 (time 1749122256.47 total 473.62 last 173.19) - open/close 60000 (time 1749122259.34 total 476.50 last 164.41) - open/close 61631 (time 1749122269.34 total 486.50 last 163.08) - open/close 63219 (time 1749122279.35 total 496.50 last 158.73) total: 64768 open/close in 505.32 seconds: 128.17 ops/second - /unlink 3058 (time 1749122299.48 total 10.00 last 305.71) - /unlink 5940 (time 1749122309.48 total 20.01 last 288.11) - /unlink 9199 (time 1749122319.49 total 30.01 last 325.86) - /unlink 10000 (time 1749122322.35 total 32.87 last 279.83) - /unlink 13005 (time 1749122332.35 total 42.87 last 300.44) - /unlink 16023 (time 1749122342.35 total 52.87 last 301.77) - /unlink 19450 (time 1749122352.36 total 62.88 last 342.56) - /unlink 23724 (time 1749122362.36 total 72.88 last 427.32) - /unlink 28923 (time 1749122372.36 total 82.88 last 519.85) - /unlink 30000 (time 1749122374.68 total 85.20 last 464.12) - /unlink 34204 (time 1749122384.68 total 95.20 last 420.39) - /unlink 38675 (time 1749122394.68 total 105.20 last 447.08) - /unlink 40000 (time 1749122397.64 total 108.16 last 447.41) - /unlink 44032 (time 1749122407.64 total 118.16 last 403.15) - /unlink 47756 (time 1749122417.64 total 128.16 last 372.37) - /unlink 50000 (time 1749122422.99 total 133.51 last 419.85) - /unlink 54775 (time 1749122432.99 total 143.51 last 477.45) - /unlink 60000 (time 1749122442.82 total 153.34 last 531.69) total: 64768 /unlink in 161.51 seconds: 401.03 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2593 (time 1749122481.31 total 10.01 last 259.07) - open/close 4634 (time 1749122491.32 total 20.01 last 204.01) - open/close 7461 (time 1749122501.32 total 30.02 last 282.64) - open/close 10000 (time 1749122508.55 total 37.24 last 351.20) - open/close 13360 (time 1749122518.55 total 47.25 last 335.89) - open/close 16376 (time 1749122528.55 total 57.25 last 301.54) - open/close 19961 (time 1749122538.56 total 67.25 last 358.47) - open/close 23368 (time 1749122548.56 total 77.25 last 340.68) - open/close 26470 (time 1749122558.56 total 87.25 last 310.17) - open/close 29695 (time 1749122568.56 total 97.25 last 322.46) - open/close 33065 (time 1749122578.56 total 107.26 last 336.94) - open/close 36049 (time 1749122588.56 total 117.26 last 298.35) - open/close 39259 (time 1749122598.56 total 127.26 last 320.96) - open/close 40000 (time 1749122600.63 total 129.33 last 358.59) - open/close 43932 (time 1749122610.63 total 139.33 last 393.16) - open/close 47761 (time 1749122620.63 total 149.33 last 382.86) - open/close 50000 (time 1749122625.83 total 154.52 last 431.05) - open/close 54237 (time 1749122635.83 total 164.52 last 423.57) - open/close 58248 (time 1749122645.83 total 174.53 last 401.05) - open/close 60000 (time 1749122650.05 total 178.75 last 414.80) - open/close 63626 (time 1749122660.06 total 188.75 last 362.45) total: 64768 open/close in 192.62 seconds: 336.24 ops/second - /unlink 6669 (time 1749122674.89 total 10.00 last 666.88) - /unlink 10000 (time 1749122679.27 total 14.38 last 760.96) - /unlink 17653 (time 1749122689.27 total 24.38 last 765.21) - /unlink 20000 (time 1749122691.96 total 27.08 last 870.11) - /unlink 28341 (time 1749122701.96 total 37.08 last 834.08) - /unlink 30000 (time 1749122704.05 total 39.16 last 795.15) - /unlink 35967 (time 1749122714.05 total 49.16 last 596.61) - /unlink 40000 (time 1749122719.09 total 54.21 last 799.78) - /unlink 47751 (time 1749122729.09 total 64.21 last 775.10) - /unlink 50000 (time 1749122731.95 total 67.06 last 787.47) - /unlink 56334 (time 1749122741.95 total 77.06 last 633.38) - /unlink 60000 (time 1749122747.18 total 82.29 last 700.79) total: 64768 /unlink in 88.84 seconds: 729.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3836 (time 1749122781.88 total 10.00 last 383.54) - open/close 7823 (time 1749122791.88 total 20.00 last 398.62) - open/close 10000 (time 1749122798.18 total 26.31 last 345.43) - open/close 13874 (time 1749122808.18 total 36.31 last 387.31) - open/close 16958 (time 1749122818.19 total 46.31 last 308.31) - open/close 19979 (time 1749122828.19 total 56.31 last 302.08) - open/close 22754 (time 1749122838.19 total 66.31 last 277.47) - open/close 25634 (time 1749122848.19 total 76.31 last 287.99) - open/close 28638 (time 1749122858.19 total 86.31 last 300.36) - open/close 30000 (time 1749122861.49 total 89.62 last 412.51) - open/close 34439 (time 1749122871.49 total 99.62 last 443.82) - open/close 38286 (time 1749122881.50 total 109.62 last 384.65) - open/close 40000 (time 1749122885.77 total 113.89 last 401.07) - open/close 44836 (time 1749122895.77 total 123.89 last 483.60) - open/close 49404 (time 1749122905.81 total 133.93 last 455.16) - open/close 53835 (time 1749122915.81 total 143.93 last 443.02) - open/close 58006 (time 1749122925.81 total 153.93 last 417.07) - open/close 60000 (time 1749122930.23 total 158.36 last 450.67) - open/close 64631 (time 1749122940.23 total 168.36 last 463.00) total: 64768 open/close in 168.76 seconds: 383.78 ops/second - /unlink 8148 (time 1749122951.51 total 10.00 last 814.74) - /unlink 10000 (time 1749122953.82 total 12.31 last 802.71) - /unlink 18372 (time 1749122963.82 total 22.31 last 837.17) - /unlink 26891 (time 1749122973.82 total 32.31 last 851.81) - /unlink 30000 (time 1749122977.80 total 36.29 last 781.79) - /unlink 37740 (time 1749122987.80 total 46.29 last 773.99) - /unlink 40000 (time 1749122990.21 total 48.70 last 938.21) - /unlink 48630 (time 1749123000.21 total 58.70 last 862.97) - /unlink 57102 (time 1749123010.21 total 68.70 last 847.20) - /unlink 60000 (time 1749123013.63 total 72.11 last 847.71) total: 64768 /unlink in 77.27 seconds: 838.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5268 (time 1749123052.15 total 10.00 last 526.72) - open/close 10000 (time 1749123061.34 total 19.19 last 515.02) - open/close 14691 (time 1749123071.34 total 29.19 last 469.07) - open/close 19349 (time 1749123081.34 total 39.19 last 465.73) - open/close 23465 (time 1749123091.34 total 49.19 last 411.57) - open/close 26837 (time 1749123101.34 total 59.19 last 337.18) - open/close 30000 (time 1749123109.22 total 67.07 last 401.51) - open/close 33808 (time 1749123119.22 total 77.07 last 380.68) - open/close 38517 (time 1749123129.23 total 87.08 last 470.85) - open/close 40000 (time 1749123131.95 total 89.80 last 543.44) - open/close 45036 (time 1749123141.95 total 99.80 last 503.59) - open/close 50000 (time 1749123151.38 total 109.23 last 526.90) - open/close 54859 (time 1749123161.38 total 119.23 last 485.82) - open/close 60000 (time 1749123170.76 total 128.61 last 547.86) total: 64768 open/close in 138.32 seconds: 468.26 ops/second - /unlink 9739 (time 1749123191.30 total 10.00 last 973.82) - /unlink 19860 (time 1749123201.30 total 20.00 last 1012.05) - /unlink 28144 (time 1749123211.30 total 30.00 last 828.40) - /unlink 30000 (time 1749123213.46 total 32.15 last 862.31) - /unlink 38376 (time 1749123223.46 total 42.15 last 837.51) - /unlink 46606 (time 1749123233.46 total 52.16 last 822.98) - /unlink 50000 (time 1749123237.56 total 56.26 last 827.32) - /unlink 58467 (time 1749123247.56 total 66.26 last 846.65) total: 64768 /unlink in 73.53 seconds: 880.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5071 (time 1749123292.63 total 10.00 last 507.07) - open/close 9968 (time 1749123302.63 total 20.00 last 489.68) - open/close 14997 (time 1749123312.63 total 30.00 last 502.83) - open/close 20000 (time 1749123322.35 total 39.72 last 514.79) - open/close 24983 (time 1749123332.35 total 49.72 last 498.25) - open/close 29958 (time 1749123342.35 total 59.72 last 497.46) - open/close 35022 (time 1749123352.35 total 69.72 last 506.31) - open/close 40000 (time 1749123362.17 total 79.54 last 506.91) - open/close 44952 (time 1749123372.17 total 89.55 last 495.15) - open/close 49921 (time 1749123382.17 total 99.55 last 496.89) - open/close 54633 (time 1749123392.20 total 109.57 last 470.05) - open/close 59524 (time 1749123402.20 total 119.57 last 489.09) - open/close 64437 (time 1749123412.23 total 129.61 last 489.56) total: 64768 open/close in 130.29 seconds: 497.09 ops/second - /unlink 8039 (time 1749123423.78 total 10.00 last 803.88) - /unlink 10000 (time 1749123426.11 total 12.33 last 840.04) - /unlink 18460 (time 1749123436.11 total 22.33 last 845.99) - /unlink 27170 (time 1749123446.11 total 32.34 last 870.91) - /unlink 30000 (time 1749123449.34 total 35.57 last 876.31) - /unlink 38707 (time 1749123459.34 total 45.57 last 870.68) - /unlink 47391 (time 1749123469.34 total 55.57 last 868.32) - /unlink 50000 (time 1749123472.33 total 58.56 last 872.48) - /unlink 58859 (time 1749123482.33 total 68.56 last 885.85) total: 64768 /unlink in 75.14 seconds: 862.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg116-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg116-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg116-server stop mds service on oleg116-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg116-server unloading modules via unload_modules_local on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-client: ssh exited with exit code 2 pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 PASS 106 (1805s) Stopping clients: oleg116-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg116-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg116-client: oleg116-server: ssh exited with exit code 2 oleg116-server: oleg116-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg116-server' oleg116-server: oleg116-server.virtnet: executing load_modules_local oleg116-server: Loading modules from /home/green/git/lustre-release/lustre oleg116-server: detected 4 online CPUs by sysfs oleg116-server: MODOPTS_LIBCFS= oleg116-server: Force libcfs to create 2 CPU partitions oleg116-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg116-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1921 sec ========== 07:39:10 (1749123550) === conf-sanity: start cleanup 07:39:10 (1749123550) === === conf-sanity: finish cleanup 07:39:11 (1749123551) ===