-----============= acceptance-small: conf-sanity ============----- Wed Apr 23 21:02:32 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg339-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg339-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg339-server oleg339-server: oleg339-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg339-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg339-server: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg339-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:03:26 (1745456606) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) oleg339-server: oleg339-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg339-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg339-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg339-server: oleg339-server.virtnet: executing set_default_debug -1 all pdsh@oleg339-client: oleg339-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg339-client.virtnet: -o user_xattr,flock 192.168.203.139@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4303 (time 1745456640.71 total 10.00 last 430.28) - open/close 9315 (time 1745456650.71 total 20.00 last 501.16) - open/close 14647 (time 1745456660.71 total 30.00 last 533.14) - open/close 19173 (time 1745456670.71 total 40.00 last 452.59) - open/close 20000 (time 1745456673.59 total 42.88 last 287.78) - open/close 21186 (time 1745456683.60 total 52.88 last 118.51) - open/close 22464 (time 1745456693.60 total 62.89 last 127.78) - open/close 25402 (time 1745456703.60 total 72.89 last 293.78) - open/close 28003 (time 1745456713.60 total 82.89 last 260.02) - open/close 30000 (time 1745456722.89 total 92.18 last 214.98) - open/close 32192 (time 1745456732.89 total 102.18 last 219.12) - open/close 35401 (time 1745456742.89 total 112.18 last 320.87) - open/close 38636 (time 1745456752.90 total 122.19 last 323.42) - open/close 40000 (time 1745456755.90 total 125.19 last 453.51) - open/close 43304 (time 1745456765.91 total 135.19 last 330.36) - open/close 46142 (time 1745456775.91 total 145.20 last 283.77) - open/close 49262 (time 1745456785.91 total 155.20 last 311.97) - open/close 50000 (time 1745456789.47 total 158.76 last 206.90) - open/close 52468 (time 1745456799.48 total 168.77 last 246.66) - open/close 54781 (time 1745456809.48 total 178.77 last 231.26) - open/close 57346 (time 1745456819.49 total 188.77 last 256.41) - open/close 59086 (time 1745456829.49 total 198.78 last 173.93) - open/close 60000 (time 1745456834.69 total 203.98 last 175.83) - open/close 61777 (time 1745456844.69 total 213.98 last 177.64) - open/close 63807 (time 1745456854.69 total 223.98 last 202.94) total: 64768 open/close in 228.59 seconds: 283.33 ops/second - /unlink 3162 (time 1745456870.45 total 10.00 last 316.17) - /unlink 6309 (time 1745456880.46 total 20.00 last 314.68) - /unlink 9639 (time 1745456890.46 total 30.00 last 332.99) - /unlink 12824 (time 1745456900.46 total 40.01 last 318.26) - /unlink 16262 (time 1745456910.47 total 50.01 last 343.73) - /unlink 19910 (time 1745456920.47 total 60.02 last 364.67) - /unlink 24146 (time 1745456930.47 total 70.02 last 423.50) - /unlink 27947 (time 1745456940.47 total 80.02 last 380.00) - /unlink 30000 (time 1745456946.44 total 85.98 last 344.35) - /unlink 34022 (time 1745456956.44 total 95.98 last 402.15) - /unlink 37808 (time 1745456966.44 total 105.99 last 378.50) - /unlink 40000 (time 1745456972.32 total 111.86 last 372.90) - /unlink 42761 (time 1745456982.32 total 121.87 last 276.06) - /unlink 46194 (time 1745456992.32 total 131.87 last 343.28) - /unlink 49786 (time 1745457002.32 total 141.87 last 359.19) - /unlink 52590 (time 1745457012.32 total 151.87 last 280.40) - /unlink 56109 (time 1745457022.32 total 161.87 last 351.76) - /unlink 59676 (time 1745457032.32 total 171.87 last 356.69) - /unlink 64581 (time 1745457042.33 total 181.87 last 490.41) total: 64768 /unlink in 182.26 seconds: 355.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2449 (time 1745457070.70 total 10.00 last 244.88) - open/close 5385 (time 1745457080.70 total 20.00 last 293.51) - open/close 7849 (time 1745457090.70 total 30.01 last 246.37) - open/close 10000 (time 1745457098.08 total 37.38 last 291.63) - open/close 13287 (time 1745457108.08 total 47.39 last 328.54) - open/close 15976 (time 1745457118.08 total 57.39 last 268.88) - open/close 19038 (time 1745457128.08 total 67.39 last 306.15) - open/close 20000 (time 1745457131.00 total 70.30 last 329.84) - open/close 23040 (time 1745457141.00 total 80.31 last 303.96) - open/close 26392 (time 1745457151.00 total 90.31 last 335.16) - open/close 30000 (time 1745457160.83 total 100.14 last 366.98) - open/close 33513 (time 1745457170.84 total 110.14 last 351.23) - open/close 36422 (time 1745457180.84 total 120.14 last 290.90) - open/close 40000 (time 1745457190.28 total 129.59 last 378.75) - open/close 43636 (time 1745457200.28 total 139.59 last 363.59) - open/close 47391 (time 1745457210.28 total 149.59 last 375.50) - open/close 50000 (time 1745457218.73 total 158.04 last 308.76) - open/close 53891 (time 1745457228.74 total 168.04 last 389.02) - open/close 57468 (time 1745457238.74 total 178.04 last 357.63) - open/close 60000 (time 1745457246.77 total 186.08 last 315.16) - open/close 63664 (time 1745457256.77 total 196.08 last 366.39) total: 64768 open/close in 198.95 seconds: 325.55 ops/second - /unlink 8614 (time 1745457270.54 total 10.00 last 861.33) - /unlink 16184 (time 1745457280.55 total 20.00 last 756.96) - /unlink 20000 (time 1745457284.61 total 24.06 last 939.13) - /unlink 28858 (time 1745457294.61 total 34.07 last 885.77) - /unlink 37075 (time 1745457304.61 total 44.07 last 821.63) - /unlink 40000 (time 1745457308.31 total 47.76 last 791.34) - /unlink 47363 (time 1745457318.31 total 57.76 last 736.28) - /unlink 50000 (time 1745457321.67 total 61.13 last 782.89) - /unlink 58339 (time 1745457331.68 total 71.13 last 833.85) - /unlink 60000 (time 1745457333.75 total 73.20 last 802.10) total: 64768 /unlink in 79.77 seconds: 811.92 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5507 (time 1745457370.38 total 10.00 last 550.65) - open/close 10000 (time 1745457378.06 total 17.68 last 584.93) - open/close 15853 (time 1745457388.06 total 27.68 last 585.26) - open/close 20000 (time 1745457395.60 total 35.23 last 549.80) - open/close 25999 (time 1745457405.60 total 45.23 last 599.89) - open/close 30000 (time 1745457412.70 total 52.32 last 563.90) - open/close 35645 (time 1745457422.70 total 62.32 last 564.45) - open/close 40000 (time 1745457430.30 total 69.92 last 572.91) - open/close 45027 (time 1745457440.30 total 79.92 last 502.69) - open/close 50000 (time 1745457450.29 total 89.91 last 497.84) - open/close 55365 (time 1745457460.29 total 99.91 last 536.45) - open/close 60000 (time 1745457470.16 total 109.79 last 469.36) - open/close 64629 (time 1745457480.17 total 119.79 last 462.79) total: 64768 open/close in 120.19 seconds: 538.88 ops/second - /unlink 7282 (time 1745457491.43 total 10.00 last 728.19) - /unlink 10000 (time 1745457495.02 total 13.59 last 757.71) - /unlink 17017 (time 1745457505.02 total 23.59 last 701.67) - /unlink 20000 (time 1745457509.06 total 27.63 last 737.61) - /unlink 29062 (time 1745457519.07 total 37.63 last 906.10) - /unlink 37181 (time 1745457529.08 total 47.64 last 811.15) - /unlink 40000 (time 1745457532.02 total 50.58 last 958.62) - /unlink 49522 (time 1745457542.02 total 60.58 last 952.13) - /unlink 56822 (time 1745457552.02 total 70.58 last 729.98) - /unlink 60000 (time 1745457555.57 total 74.14 last 893.75) total: 64768 /unlink in 79.80 seconds: 811.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5077 (time 1745457595.66 total 10.00 last 507.69) - open/close 10000 (time 1745457604.56 total 18.90 last 552.93) - open/close 15263 (time 1745457614.56 total 28.90 last 526.27) - open/close 20000 (time 1745457623.61 total 37.95 last 523.61) - open/close 25508 (time 1745457633.61 total 47.95 last 550.79) - open/close 29588 (time 1745457643.61 total 57.95 last 407.94) - open/close 33953 (time 1745457653.61 total 67.95 last 436.45) - open/close 38360 (time 1745457663.61 total 77.95 last 440.70) - open/close 40000 (time 1745457667.51 total 81.85 last 420.84) - open/close 44649 (time 1745457677.51 total 91.85 last 464.83) - open/close 49658 (time 1745457687.51 total 101.85 last 500.84) - open/close 54731 (time 1745457697.55 total 111.89 last 505.21) - open/close 60000 (time 1745457707.24 total 121.58 last 544.09) total: 64768 open/close in 131.37 seconds: 493.04 ops/second - /unlink 7595 (time 1745457727.92 total 10.00 last 759.42) - /unlink 10000 (time 1745457731.10 total 13.18 last 757.43) - /unlink 18241 (time 1745457741.10 total 23.18 last 824.06) - /unlink 27901 (time 1745457751.10 total 33.18 last 965.92) - /unlink 30000 (time 1745457753.48 total 35.56 last 881.94) - /unlink 39370 (time 1745457763.48 total 45.56 last 936.94) - /unlink 48208 (time 1745457773.48 total 55.56 last 883.71) - /unlink 50000 (time 1745457775.83 total 57.90 last 764.26) - /unlink 58195 (time 1745457785.83 total 67.90 last 819.43) total: 64768 /unlink in 75.16 seconds: 861.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4585 (time 1745457825.79 total 10.00 last 458.50) - open/close 9274 (time 1745457835.79 total 20.00 last 468.83) - open/close 14611 (time 1745457845.80 total 30.00 last 533.63) - open/close 20000 (time 1745457855.15 total 39.36 last 575.87) - open/close 25710 (time 1745457865.15 total 49.36 last 570.94) - open/close 30000 (time 1745457873.50 total 57.70 last 514.29) - open/close 35385 (time 1745457883.50 total 67.70 last 538.47) - open/close 40000 (time 1745457891.81 total 76.02 last 555.13) - open/close 44789 (time 1745457901.81 total 86.02 last 478.83) - open/close 49965 (time 1745457911.81 total 96.02 last 517.56) - open/close 55540 (time 1745457921.81 total 106.02 last 557.43) - open/close 59978 (time 1745457931.81 total 116.02 last 443.76) total: 64768 open/close in 124.87 seconds: 518.70 ops/second - /unlink 8412 (time 1745457951.53 total 10.00 last 841.13) - /unlink 16166 (time 1745457961.53 total 20.00 last 775.39) - /unlink 20000 (time 1745457966.33 total 24.80 last 798.89) - /unlink 27920 (time 1745457976.33 total 34.80 last 791.99) - /unlink 30000 (time 1745457978.71 total 37.18 last 873.91) - /unlink 37976 (time 1745457988.71 total 47.18 last 797.59) - /unlink 40000 (time 1745457991.18 total 49.65 last 818.78) - /unlink 48676 (time 1745458001.19 total 59.65 last 867.55) - /unlink 57705 (time 1745458011.19 total 69.65 last 902.82) - /unlink 60000 (time 1745458013.77 total 72.24 last 888.11) total: 64768 /unlink in 77.51 seconds: 835.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg339-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg339-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg339-server stop mds service on oleg339-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg339-server unloading modules via unload_modules_local on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-client: ssh exited with exit code 2 pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 PASS 106 (1455s) Stopping clients: oleg339-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg339-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg339-client: oleg339-server: ssh exited with exit code 2 oleg339-server: oleg339-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg339-server' oleg339-server: oleg339-server.virtnet: executing load_modules_local oleg339-server: Loading modules from /home/green/git/lustre-release/lustre oleg339-server: detected 4 online CPUs by sysfs oleg339-server: MODOPTS_LIBCFS= oleg339-server: Force libcfs to create 2 CPU partitions oleg339-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg339-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1527 sec ========== 21:27:59 (1745458079) === conf-sanity: start cleanup 21:27:59 (1745458079) === === conf-sanity: finish cleanup 21:28:00 (1745458080) ===