-----============= acceptance-small: conf-sanity ============----- Tue Aug 27 11:41:44 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg222-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg222-server oleg222-server: oleg222-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg222-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg222-server: oleg222-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg222-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:42:37 (1724773357) Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) oleg222-server: oleg222-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg222-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg222-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg222-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg222-server: oleg222-server.virtnet: executing set_default_debug -1 all pdsh@oleg222-client: oleg222-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg222-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg222-client.virtnet: -o user_xattr,flock 192.168.202.122@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5489 (time 1724773392.09 total 10.00 last 548.88) - open/close 10000 (time 1724773400.52 total 18.43 last 535.28) - open/close 14971 (time 1724773410.52 total 28.43 last 497.07) - open/close 20000 (time 1724773420.09 total 38.00 last 525.42) - open/close 24973 (time 1724773430.09 total 48.00 last 497.22) - open/close 29491 (time 1724773440.09 total 58.00 last 451.72) - open/close 33602 (time 1724773450.09 total 68.00 last 411.09) - open/close 38364 (time 1724773460.10 total 78.00 last 476.12) - open/close 40000 (time 1724773462.91 total 80.82 last 581.42) - open/close 45488 (time 1724773472.91 total 90.82 last 548.70) - open/close 50000 (time 1724773481.14 total 99.04 last 548.63) - open/close 54963 (time 1724773491.14 total 109.05 last 496.26) - open/close 60000 (time 1724773500.67 total 118.58 last 528.41) total: 64768 open/close in 127.38 seconds: 508.45 ops/second - /unlink 8173 (time 1724773520.37 total 10.00 last 817.30) - /unlink 10000 (time 1724773522.80 total 12.43 last 753.35) - /unlink 18351 (time 1724773532.80 total 22.43 last 835.05) - /unlink 20000 (time 1724773534.88 total 24.51 last 791.28) - /unlink 28616 (time 1724773544.89 total 34.51 last 861.50) - /unlink 36449 (time 1724773554.89 total 44.51 last 783.26) - /unlink 40000 (time 1724773558.78 total 48.41 last 911.69) - /unlink 48376 (time 1724773568.78 total 58.41 last 837.55) - /unlink 56495 (time 1724773578.78 total 68.41 last 811.86) - /unlink 60000 (time 1724773582.89 total 72.51 last 853.76) total: 64768 /unlink in 78.44 seconds: 825.65 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5038 (time 1724773621.14 total 10.00 last 503.77) - open/close 10000 (time 1724773630.40 total 19.26 last 535.71) - open/close 14705 (time 1724773640.41 total 29.27 last 470.39) - open/close 19930 (time 1724773650.41 total 39.27 last 522.49) - open/close 25646 (time 1724773660.41 total 49.27 last 571.57) - open/close 30000 (time 1724773667.43 total 56.29 last 619.73) - open/close 36067 (time 1724773677.43 total 66.29 last 606.69) - open/close 40000 (time 1724773684.88 total 73.74 last 528.14) - open/close 45167 (time 1724773694.88 total 83.74 last 516.63) - open/close 50000 (time 1724773704.15 total 93.01 last 521.51) - open/close 55300 (time 1724773714.15 total 103.01 last 529.94) - open/close 60000 (time 1724773723.31 total 112.17 last 512.87) total: 64768 open/close in 121.97 seconds: 531.04 ops/second - /unlink 8653 (time 1724773743.97 total 10.00 last 865.23) - /unlink 16688 (time 1724773753.97 total 20.00 last 803.48) - /unlink 20000 (time 1724773757.87 total 23.90 last 849.47) - /unlink 28846 (time 1724773767.87 total 33.90 last 884.59) - /unlink 37321 (time 1724773777.87 total 43.90 last 847.47) - /unlink 40000 (time 1724773781.31 total 47.34 last 779.97) - /unlink 49595 (time 1724773791.31 total 57.34 last 959.48) - /unlink 59854 (time 1724773801.31 total 67.34 last 1025.84) total: 64768 /unlink in 72.72 seconds: 890.61 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5619 (time 1724773841.70 total 10.00 last 561.86) - open/close 10000 (time 1724773850.25 total 18.55 last 512.66) - open/close 14984 (time 1724773860.25 total 28.55 last 498.35) - open/close 20000 (time 1724773870.17 total 38.46 last 505.81) - open/close 25638 (time 1724773880.17 total 48.46 last 563.79) - open/close 30000 (time 1724773888.65 total 56.94 last 514.35) - open/close 34805 (time 1724773898.65 total 66.95 last 480.42) - open/close 40000 (time 1724773908.29 total 76.58 last 539.13) - open/close 45793 (time 1724773918.29 total 86.58 last 579.21) - open/close 50000 (time 1724773926.83 total 95.13 last 492.35) - open/close 54708 (time 1724773936.83 total 105.13 last 470.73) - open/close 59601 (time 1724773946.84 total 115.13 last 489.25) - open/close 64324 (time 1724773956.84 total 125.13 last 472.23) total: 64768 open/close in 126.24 seconds: 513.04 ops/second - /unlink 8955 (time 1724773968.88 total 10.00 last 895.47) - /unlink 19259 (time 1724773978.88 total 20.00 last 1030.37) - /unlink 28893 (time 1724773988.88 total 30.00 last 963.32) - /unlink 38172 (time 1724773998.88 total 40.00 last 927.86) - /unlink 40000 (time 1724774001.03 total 42.15 last 851.04) - /unlink 49062 (time 1724774011.03 total 52.15 last 906.12) - /unlink 58482 (time 1724774021.03 total 62.15 last 941.97) total: 64768 /unlink in 68.94 seconds: 939.54 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5318 (time 1724774067.15 total 10.00 last 531.73) - open/close 10000 (time 1724774076.89 total 19.73 last 481.07) - open/close 15004 (time 1724774086.89 total 29.74 last 500.31) - open/close 20000 (time 1724774096.76 total 39.60 last 506.23) - open/close 24559 (time 1724774106.76 total 49.61 last 455.84) - open/close 29520 (time 1724774116.76 total 59.61 last 496.08) - open/close 34731 (time 1724774126.80 total 69.65 last 518.67) - open/close 39941 (time 1724774136.81 total 79.65 last 520.97) - open/close 45167 (time 1724774146.81 total 89.66 last 522.52) - open/close 50000 (time 1724774155.85 total 98.69 last 534.71) - open/close 55216 (time 1724774165.85 total 108.69 last 521.54) - open/close 60000 (time 1724774175.82 total 118.67 last 479.47) total: 64768 open/close in 128.35 seconds: 504.62 ops/second - /unlink 7663 (time 1724774196.41 total 10.00 last 766.24) - /unlink 10000 (time 1724774199.12 total 12.71 last 861.34) - /unlink 18805 (time 1724774209.12 total 22.71 last 880.48) - /unlink 27700 (time 1724774219.12 total 32.71 last 889.49) - /unlink 30000 (time 1724774221.69 total 35.28 last 896.46) - /unlink 38734 (time 1724774231.69 total 45.28 last 873.30) - /unlink 46679 (time 1724774241.69 total 55.28 last 794.45) - /unlink 50000 (time 1724774245.90 total 59.49 last 788.32) - /unlink 57197 (time 1724774255.90 total 69.49 last 719.68) - /unlink 60000 (time 1724774259.59 total 73.18 last 760.00) total: 64768 /unlink in 79.19 seconds: 817.88 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4169 (time 1724774297.37 total 10.00 last 416.86) - open/close 8895 (time 1724774307.37 total 20.00 last 472.54) - open/close 10000 (time 1724774309.47 total 22.10 last 526.20) - open/close 15447 (time 1724774319.47 total 32.10 last 544.62) - open/close 20000 (time 1724774328.31 total 40.94 last 515.23) - open/close 24921 (time 1724774338.31 total 50.94 last 492.10) - open/close 29856 (time 1724774348.31 total 60.94 last 493.42) - open/close 34586 (time 1724774358.31 total 70.94 last 472.98) - open/close 39429 (time 1724774368.31 total 80.94 last 484.28) - open/close 44860 (time 1724774378.31 total 90.94 last 543.10) - open/close 50000 (time 1724774387.94 total 100.57 last 533.99) - open/close 55106 (time 1724774397.94 total 110.57 last 510.52) - open/close 60000 (time 1724774407.33 total 119.96 last 520.97) total: 64768 open/close in 128.99 seconds: 502.10 ops/second - /unlink 9114 (time 1724774427.21 total 10.00 last 911.36) - /unlink 17948 (time 1724774437.21 total 20.00 last 883.35) - /unlink 20000 (time 1724774439.43 total 22.22 last 923.98) - /unlink 29078 (time 1724774449.43 total 32.22 last 907.76) - /unlink 38280 (time 1724774459.43 total 42.22 last 920.18) - /unlink 47453 (time 1724774469.43 total 52.22 last 917.30) - /unlink 50000 (time 1724774472.21 total 55.00 last 917.51) - /unlink 59339 (time 1724774482.21 total 65.00 last 933.87) total: 64768 /unlink in 70.72 seconds: 915.86 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg222-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg222-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg222-server stop mds service on oleg222-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg222-server unloading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-client: ssh exited with exit code 2 pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 PASS 106 (1179s) Stopping clients: oleg222-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg222-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg222-client: oleg222-server: ssh exited with exit code 2 oleg222-server: oleg222-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg222-server' oleg222-server: oleg222-server.virtnet: executing load_modules_local oleg222-server: Loading modules from /home/green/git/lustre-release/lustre oleg222-server: detected 4 online CPUs by sysfs oleg222-server: Force libcfs to create 2 CPU partitions oleg222-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg222-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1249 sec ========== 12:02:34 (1724774554) === conf-sanity: start cleanup 12:02:34 (1724774554) === === conf-sanity: finish cleanup 12:02:34 (1724774554) ===