-----============= acceptance-small: conf-sanity ============----- Sat Sep 14 04:49:56 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg416-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg416-server oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg416-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg416-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg416-server: oleg416-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg416-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:51:11 (1726303871) Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg416-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg416-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg416-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg416-server: oleg416-server.virtnet: executing set_default_debug -1 all pdsh@oleg416-client: oleg416-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg416-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg416-client.virtnet: -o user_xattr,flock 192.168.204.116@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2109 (time 1726303915.28 total 10.00 last 210.89) - open/close 4536 (time 1726303925.28 total 20.00 last 242.64) - open/close 6789 (time 1726303935.29 total 30.01 last 225.21) - open/close 8992 (time 1726303945.29 total 40.01 last 220.29) - open/close 10000 (time 1726303949.57 total 44.29 last 235.39) - open/close 11772 (time 1726303959.57 total 54.29 last 177.19) - open/close 13743 (time 1726303969.57 total 64.29 last 197.03) - open/close 15621 (time 1726303979.57 total 74.30 last 187.76) - open/close 17779 (time 1726303989.58 total 84.30 last 215.76) - open/close 20000 (time 1726303997.76 total 92.48 last 271.46) - open/close 22850 (time 1726304007.76 total 102.48 last 284.94) - open/close 25759 (time 1726304017.76 total 112.48 last 290.87) - open/close 28815 (time 1726304027.76 total 122.48 last 305.58) - open/close 30000 (time 1726304031.04 total 125.76 last 361.41) - open/close 33221 (time 1726304041.04 total 135.76 last 322.02) - open/close 36802 (time 1726304051.04 total 145.77 last 358.01) - open/close 40000 (time 1726304058.90 total 153.62 last 407.21) - open/close 44102 (time 1726304068.90 total 163.62 last 410.14) - open/close 48321 (time 1726304078.90 total 173.62 last 421.90) - open/close 50000 (time 1726304083.88 total 178.61 last 336.86) - open/close 52949 (time 1726304093.89 total 188.61 last 294.79) - open/close 55124 (time 1726304103.89 total 198.61 last 217.47) - open/close 57612 (time 1726304113.89 total 208.61 last 248.72) - open/close 60000 (time 1726304121.83 total 216.55 last 300.72) - open/close 63705 (time 1726304131.83 total 226.56 last 370.49) total: 64768 open/close in 229.78 seconds: 281.87 ops/second - /unlink 7009 (time 1726304146.03 total 10.00 last 700.90) - /unlink 10000 (time 1726304149.64 total 13.62 last 827.26) - /unlink 17759 (time 1726304159.64 total 23.62 last 775.88) - /unlink 20000 (time 1726304162.93 total 26.90 last 681.66) - /unlink 29147 (time 1726304172.93 total 36.90 last 914.66) - /unlink 38440 (time 1726304182.93 total 46.90 last 929.24) - /unlink 46817 (time 1726304192.93 total 56.91 last 837.65) - /unlink 50000 (time 1726304196.49 total 60.47 last 893.87) - /unlink 58000 (time 1726304206.49 total 70.47 last 800.00) - /unlink 60000 (time 1726304208.96 total 72.94 last 808.73) total: 64768 /unlink in 77.52 seconds: 835.53 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4517 (time 1726304242.97 total 10.00 last 451.70) - open/close 10000 (time 1726304252.74 total 19.77 last 561.17) - open/close 14694 (time 1726304262.75 total 29.77 last 469.35) - open/close 19153 (time 1726304272.75 total 39.77 last 445.87) - open/close 24706 (time 1726304282.75 total 49.77 last 555.23) - open/close 29597 (time 1726304292.75 total 59.78 last 489.04) - open/close 35234 (time 1726304302.75 total 69.78 last 563.69) - open/close 40000 (time 1726304311.45 total 78.47 last 547.94) - open/close 45068 (time 1726304321.45 total 88.47 last 506.74) - open/close 49732 (time 1726304331.45 total 98.48 last 466.30) - open/close 54985 (time 1726304341.45 total 108.48 last 525.28) - open/close 60000 (time 1726304351.38 total 118.40 last 505.21) total: 64768 open/close in 127.80 seconds: 506.78 ops/second - /unlink 8581 (time 1726304371.68 total 10.00 last 858.05) - /unlink 18741 (time 1726304381.68 total 20.00 last 1015.94) - /unlink 28672 (time 1726304391.68 total 30.00 last 993.05) - /unlink 38420 (time 1726304401.68 total 40.00 last 974.77) - /unlink 47674 (time 1726304411.68 total 50.00 last 925.31) - /unlink 50000 (time 1726304414.53 total 52.85 last 817.87) - /unlink 58563 (time 1726304424.53 total 62.85 last 856.29) total: 64768 /unlink in 69.52 seconds: 931.60 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5118 (time 1726304472.96 total 10.00 last 511.74) - open/close 10000 (time 1726304482.63 total 19.67 last 504.93) - open/close 15109 (time 1726304492.63 total 29.67 last 510.84) - open/close 20000 (time 1726304502.21 total 39.25 last 510.47) - open/close 24736 (time 1726304512.21 total 49.25 last 473.58) - open/close 29365 (time 1726304522.21 total 59.25 last 462.87) - open/close 34431 (time 1726304532.21 total 69.25 last 506.59) - open/close 39351 (time 1726304542.21 total 79.25 last 491.98) - open/close 44462 (time 1726304552.21 total 89.25 last 511.09) - open/close 49622 (time 1726304562.21 total 99.25 last 515.98) - open/close 54573 (time 1726304572.22 total 109.26 last 495.05) - open/close 59497 (time 1726304582.22 total 119.26 last 492.36) - open/close 64369 (time 1726304592.22 total 129.26 last 487.19) total: 64768 open/close in 130.10 seconds: 497.81 ops/second - /unlink 8430 (time 1726304603.93 total 10.00 last 842.99) - /unlink 17089 (time 1726304613.94 total 20.00 last 865.80) - /unlink 20000 (time 1726304617.19 total 23.25 last 894.96) - /unlink 29036 (time 1726304627.19 total 33.25 last 903.51) - /unlink 37938 (time 1726304637.19 total 43.26 last 890.11) - /unlink 40000 (time 1726304639.51 total 45.57 last 889.31) - /unlink 49074 (time 1726304649.51 total 55.58 last 907.32) - /unlink 57980 (time 1726304659.51 total 65.58 last 890.52) - /unlink 60000 (time 1726304661.71 total 67.78 last 917.17) total: 64768 /unlink in 72.94 seconds: 887.98 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5189 (time 1726304703.36 total 10.00 last 518.89) - open/close 10000 (time 1726304712.88 total 19.53 last 505.08) - open/close 15071 (time 1726304722.88 total 29.53 last 507.03) - open/close 19858 (time 1726304732.88 total 39.53 last 478.64) - open/close 25136 (time 1726304742.88 total 49.53 last 527.77) - open/close 30000 (time 1726304751.98 total 58.63 last 534.52) - open/close 35194 (time 1726304761.99 total 68.63 last 519.32) - open/close 40000 (time 1726304771.13 total 77.77 last 525.59) - open/close 45038 (time 1726304781.13 total 87.77 last 503.79) - open/close 50000 (time 1726304790.85 total 97.50 last 510.31) - open/close 54724 (time 1726304800.85 total 107.50 last 472.34) - open/close 60000 (time 1726304810.16 total 116.80 last 567.17) total: 64768 open/close in 124.44 seconds: 520.46 ops/second - /unlink 9596 (time 1726304828.56 total 10.00 last 959.58) - /unlink 19448 (time 1726304838.57 total 20.00 last 985.12) - /unlink 28777 (time 1726304848.57 total 30.00 last 932.88) - /unlink 38125 (time 1726304858.57 total 40.00 last 934.79) - /unlink 47983 (time 1726304868.57 total 50.00 last 985.79) - /unlink 57767 (time 1726304878.57 total 60.00 last 978.33) - /unlink 60000 (time 1726304880.86 total 62.29 last 975.80) total: 64768 /unlink in 66.74 seconds: 970.44 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5183 (time 1726304928.51 total 10.00 last 518.26) - open/close 10000 (time 1726304937.56 total 19.05 last 532.59) - open/close 15346 (time 1726304947.56 total 29.05 last 534.55) - open/close 20000 (time 1726304956.85 total 38.34 last 500.91) - open/close 25347 (time 1726304966.85 total 48.34 last 534.64) - open/close 30000 (time 1726304975.75 total 57.24 last 522.94) - open/close 35161 (time 1726304985.75 total 67.24 last 516.08) - open/close 40000 (time 1726304995.60 total 77.09 last 490.99) - open/close 45288 (time 1726305005.60 total 87.09 last 528.79) - open/close 50000 (time 1726305014.73 total 96.22 last 516.30) - open/close 55154 (time 1726305024.73 total 106.22 last 515.32) - open/close 60000 (time 1726305034.14 total 115.63 last 515.05) total: 64768 open/close in 125.45 seconds: 516.28 ops/second - /unlink 9053 (time 1726305054.79 total 10.00 last 905.28) - /unlink 18184 (time 1726305064.79 total 20.00 last 913.02) - /unlink 27414 (time 1726305074.79 total 30.00 last 922.96) - /unlink 30000 (time 1726305077.55 total 32.76 last 936.78) - /unlink 38452 (time 1726305087.55 total 42.76 last 845.09) - /unlink 47249 (time 1726305097.55 total 52.76 last 879.67) - /unlink 50000 (time 1726305100.67 total 55.88 last 883.22) - /unlink 58952 (time 1726305110.67 total 65.88 last 895.10) total: 64768 /unlink in 72.32 seconds: 895.53 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg416-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg416-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg416-server stop mds service on oleg416-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg416-server unloading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 pdsh@oleg416-client: oleg416-client: ssh exited with exit code 2 PASS 106 (1292s) Stopping clients: oleg416-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg416-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg416-client: oleg416-server: ssh exited with exit code 2 oleg416-server: oleg416-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg416-server' oleg416-server: oleg416-server.virtnet: executing load_modules_local oleg416-server: Loading modules from /home/green/git/lustre-release/lustre oleg416-server: detected 4 online CPUs by sysfs oleg416-server: Force libcfs to create 2 CPU partitions oleg416-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg416-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1385 sec ========== 05:13:01 (1726305181) === conf-sanity: start cleanup 05:13:01 (1726305181) === === conf-sanity: finish cleanup 05:13:01 (1726305181) ===