-----============= acceptance-small: conf-sanity ============----- Thu May 1 11:07:18 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg437-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg437-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg437-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg437-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg437-server oleg437-server: oleg437-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg437-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg437-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg437-server: oleg437-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg437-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:08:22 (1746112102) Stopping clients: oleg437-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg437-client.virtnet /mnt/lustre2 (opts:-f) oleg437-server: oleg437-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg437-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg437-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4341 (time 1746112136.48 total 10.00 last 434.07) - open/close 9544 (time 1746112146.48 total 20.00 last 520.27) - open/close 14933 (time 1746112156.48 total 30.00 last 538.85) - open/close 20000 (time 1746112166.37 total 39.90 last 512.18) - open/close 25193 (time 1746112176.37 total 49.90 last 519.29) - open/close 30000 (time 1746112185.54 total 59.07 last 524.12) - open/close 34949 (time 1746112195.55 total 69.07 last 494.82) - open/close 40000 (time 1746112205.54 total 79.06 last 505.63) - open/close 44775 (time 1746112215.54 total 89.06 last 477.47) - open/close 48900 (time 1746112225.54 total 99.06 last 412.50) - open/close 50000 (time 1746112227.87 total 101.39 last 471.41) - open/close 53879 (time 1746112237.87 total 111.39 last 387.86) - open/close 58585 (time 1746112247.87 total 121.39 last 470.52) - open/close 60000 (time 1746112250.83 total 124.35 last 478.65) - open/close 64419 (time 1746112260.83 total 134.35 last 441.89) total: 64768 open/close in 135.11 seconds: 479.38 ops/second - /unlink 7288 (time 1746112272.42 total 10.00 last 728.79) - /unlink 10000 (time 1746112276.58 total 14.17 last 651.12) - /unlink 18051 (time 1746112286.58 total 24.17 last 805.01) - /unlink 20000 (time 1746112289.19 total 26.78 last 746.34) - /unlink 26616 (time 1746112299.20 total 36.78 last 661.55) - /unlink 30000 (time 1746112302.94 total 40.52 last 903.28) - /unlink 38890 (time 1746112312.94 total 50.53 last 888.96) - /unlink 47411 (time 1746112322.94 total 60.53 last 852.09) - /unlink 50000 (time 1746112325.90 total 63.48 last 876.60) - /unlink 59221 (time 1746112335.90 total 73.48 last 922.01) total: 64768 /unlink in 79.38 seconds: 815.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5332 (time 1746112375.32 total 10.00 last 533.11) - open/close 10000 (time 1746112384.22 total 18.90 last 524.49) - open/close 14596 (time 1746112394.22 total 28.90 last 459.55) - open/close 19326 (time 1746112404.22 total 38.90 last 472.99) - open/close 23450 (time 1746112414.22 total 48.90 last 412.33) - open/close 27495 (time 1746112424.22 total 58.90 last 404.48) - open/close 30000 (time 1746112430.16 total 64.85 last 421.60) - open/close 34897 (time 1746112440.17 total 74.85 last 489.63) - open/close 39507 (time 1746112450.17 total 84.85 last 460.98) - open/close 43375 (time 1746112460.17 total 94.85 last 386.78) - open/close 48470 (time 1746112470.17 total 104.85 last 509.49) - open/close 50000 (time 1746112473.17 total 107.85 last 510.07) - open/close 55226 (time 1746112483.17 total 117.85 last 522.57) - open/close 60000 (time 1746112492.34 total 127.02 last 520.60) total: 64768 open/close in 136.14 seconds: 475.76 ops/second - /unlink 9381 (time 1746112512.28 total 10.00 last 938.09) - /unlink 18626 (time 1746112522.28 total 20.00 last 924.49) - /unlink 27544 (time 1746112532.28 total 30.00 last 891.71) - /unlink 30000 (time 1746112535.04 total 32.77 last 887.55) - /unlink 37591 (time 1746112545.04 total 42.77 last 759.08) - /unlink 40000 (time 1746112547.56 total 45.29 last 955.88) - /unlink 49148 (time 1746112557.56 total 55.29 last 914.76) - /unlink 55391 (time 1746112567.57 total 65.29 last 624.27) - /unlink 60000 (time 1746112576.24 total 73.96 last 531.57) total: 64768 /unlink in 80.06 seconds: 808.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2882 (time 1746112609.70 total 10.00 last 288.19) - open/close 6388 (time 1746112619.70 total 20.00 last 350.51) - open/close 9318 (time 1746112629.70 total 30.00 last 293.00) - open/close 10000 (time 1746112632.28 total 32.59 last 264.11) - open/close 12818 (time 1746112642.28 total 42.59 last 281.75) - open/close 15847 (time 1746112652.28 total 52.59 last 302.90) - open/close 19010 (time 1746112662.28 total 62.59 last 316.27) - open/close 20000 (time 1746112664.43 total 64.73 last 462.24) - open/close 24343 (time 1746112674.43 total 74.73 last 434.29) - open/close 27939 (time 1746112684.43 total 84.73 last 359.59) - open/close 30000 (time 1746112690.81 total 91.11 last 322.86) - open/close 34005 (time 1746112700.81 total 101.11 last 400.48) - open/close 38960 (time 1746112710.81 total 111.12 last 495.48) - open/close 40000 (time 1746112713.15 total 113.45 last 445.02) - open/close 43177 (time 1746112723.15 total 123.46 last 317.59) - open/close 45815 (time 1746112733.15 total 133.46 last 263.74) - open/close 49106 (time 1746112743.15 total 143.46 last 329.09) - open/close 50000 (time 1746112745.48 total 145.78 last 384.24) - open/close 54321 (time 1746112755.48 total 155.79 last 432.04) - open/close 57933 (time 1746112765.48 total 165.79 last 361.15) - open/close 60000 (time 1746112772.00 total 172.31 last 317.12) total: 64768 open/close in 182.05 seconds: 355.78 ops/second - /unlink 7759 (time 1746112792.54 total 10.00 last 775.87) - /unlink 10000 (time 1746112796.17 total 13.63 last 616.89) - /unlink 15798 (time 1746112806.17 total 23.63 last 579.75) - /unlink 20000 (time 1746112813.24 total 30.71 last 594.21) - /unlink 25354 (time 1746112823.25 total 40.71 last 535.37) - /unlink 30000 (time 1746112832.47 total 49.93 last 503.77) - /unlink 35479 (time 1746112842.47 total 59.93 last 547.82) - /unlink 40000 (time 1746112850.48 total 67.94 last 564.37) - /unlink 45523 (time 1746112860.48 total 77.94 last 552.25) - /unlink 50000 (time 1746112868.51 total 85.97 last 557.87) - /unlink 55665 (time 1746112878.51 total 95.97 last 566.42) - /unlink 60000 (time 1746112886.61 total 104.07 last 535.01) total: 64768 /unlink in 111.97 seconds: 578.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2765 (time 1746112921.46 total 10.00 last 276.46) - open/close 5349 (time 1746112931.46 total 20.01 last 258.30) - open/close 7915 (time 1746112941.46 total 30.01 last 256.60) - open/close 10000 (time 1746112949.54 total 38.08 last 258.10) - open/close 12562 (time 1746112959.54 total 48.08 last 256.18) - open/close 15067 (time 1746112969.54 total 58.08 last 250.50) - open/close 17665 (time 1746112979.55 total 68.09 last 259.77) - open/close 20000 (time 1746112987.94 total 76.48 last 278.07) - open/close 22865 (time 1746112997.95 total 86.49 last 286.41) - open/close 25559 (time 1746113007.95 total 96.49 last 269.35) - open/close 28632 (time 1746113017.95 total 106.49 last 307.20) - open/close 30000 (time 1746113023.24 total 111.78 last 258.82) - open/close 32536 (time 1746113033.24 total 121.78 last 253.58) - open/close 35027 (time 1746113043.24 total 131.78 last 249.07) - open/close 37550 (time 1746113053.24 total 141.78 last 252.30) - open/close 40000 (time 1746113062.53 total 151.07 last 263.70) - open/close 42500 (time 1746113072.53 total 161.07 last 249.93) - open/close 44992 (time 1746113082.53 total 171.07 last 249.15) - open/close 47448 (time 1746113092.54 total 181.08 last 245.54) - open/close 49956 (time 1746113102.54 total 191.08 last 250.78) - open/close 52472 (time 1746113112.54 total 201.08 last 251.59) - open/close 55137 (time 1746113122.54 total 211.08 last 266.49) - open/close 57657 (time 1746113132.54 total 221.08 last 251.98) - open/close 60000 (time 1746113140.93 total 229.47 last 279.12) - open/close 62693 (time 1746113150.94 total 239.48 last 269.19) total: 64768 open/close in 247.83 seconds: 261.34 ops/second - /unlink 4841 (time 1746113170.55 total 10.00 last 484.05) - /unlink 9874 (time 1746113180.55 total 20.00 last 503.23) - /unlink 14752 (time 1746113190.55 total 30.00 last 487.78) - /unlink 19945 (time 1746113200.56 total 40.00 last 519.27) - /unlink 24833 (time 1746113210.56 total 50.00 last 488.79) - /unlink 29748 (time 1746113220.56 total 60.00 last 491.49) - /unlink 34969 (time 1746113230.56 total 70.00 last 522.08) - /unlink 40000 (time 1746113239.53 total 78.98 last 560.41) - /unlink 45056 (time 1746113249.53 total 88.98 last 505.56) - /unlink 49834 (time 1746113259.53 total 98.98 last 477.78) - /unlink 54697 (time 1746113269.54 total 108.98 last 486.26) - /unlink 59737 (time 1746113279.54 total 118.98 last 503.95) - /unlink 64744 (time 1746113289.54 total 128.99 last 500.59) total: 64768 /unlink in 129.04 seconds: 501.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2510 (time 1746113317.10 total 10.00 last 250.92) - open/close 5111 (time 1746113327.10 total 20.01 last 260.05) - open/close 7681 (time 1746113337.11 total 30.01 last 256.96) - open/close 10000 (time 1746113346.22 total 39.12 last 254.42) - open/close 12558 (time 1746113356.22 total 49.12 last 255.75) - open/close 15206 (time 1746113366.22 total 59.13 last 264.75) - open/close 17901 (time 1746113376.23 total 69.13 last 269.48) - open/close 20000 (time 1746113384.07 total 76.97 last 267.43) - open/close 22758 (time 1746113394.07 total 86.98 last 275.79) - open/close 25518 (time 1746113404.08 total 96.98 last 276.00) - open/close 28339 (time 1746113414.08 total 106.98 last 282.08) - open/close 30000 (time 1746113419.44 total 112.34 last 309.77) - open/close 33303 (time 1746113429.44 total 122.34 last 330.29) - open/close 36490 (time 1746113439.44 total 132.34 last 318.68) - open/close 39354 (time 1746113449.44 total 142.34 last 286.31) - open/close 40000 (time 1746113451.84 total 144.74 last 269.68) - open/close 42744 (time 1746113461.84 total 154.74 last 274.37) - open/close 45425 (time 1746113471.84 total 164.74 last 268.08) - open/close 47997 (time 1746113481.84 total 174.74 last 257.10) - open/close 50000 (time 1746113489.97 total 182.87 last 246.46) - open/close 52476 (time 1746113499.97 total 192.87 last 247.55) - open/close 54966 (time 1746113509.97 total 202.87 last 248.98) - open/close 57484 (time 1746113519.98 total 212.88 last 251.70) - open/close 59917 (time 1746113529.98 total 222.88 last 243.23) - open/close 62420 (time 1746113539.98 total 232.88 last 250.24) total: 64768 open/close in 241.96 seconds: 267.69 ops/second - /unlink 5246 (time 1746113560.35 total 10.00 last 524.56) - /unlink 10000 (time 1746113570.24 total 19.90 last 480.47) - /unlink 15020 (time 1746113580.24 total 29.90 last 501.92) - /unlink 20000 (time 1746113587.05 total 36.71 last 731.25) - /unlink 28531 (time 1746113597.05 total 46.71 last 853.02) - /unlink 38113 (time 1746113607.06 total 56.71 last 958.14) - /unlink 40000 (time 1746113609.12 total 58.77 last 915.48) - /unlink 49102 (time 1746113619.12 total 68.77 last 910.15) - /unlink 58046 (time 1746113629.12 total 78.77 last 894.38) - /unlink 60000 (time 1746113631.27 total 80.93 last 906.00) total: 64768 /unlink in 86.00 seconds: 753.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 PASS 106 (1579s) Stopping clients: oleg437-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg437-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 oleg437-server: oleg437-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions oleg437-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg437-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg437-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1660 sec ========== 11:34:59 (1746113699) === conf-sanity: start cleanup 11:34:59 (1746113699) === === conf-sanity: finish cleanup 11:35:00 (1746113700) ===