-----============= acceptance-small: conf-sanity ============----- Thu Dec 5 06:16:10 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg137-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg137-server oleg137-server: oleg137-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: detected 4 online CPUs by sysfs oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg137-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg137-server: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg137-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 06:17:07 (1733397427) Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) oleg137-server: oleg137-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: detected 4 online CPUs by sysfs oleg137-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg137-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock 192.168.201.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4316 (time 1733397460.55 total 10.00 last 431.57) - open/close 8449 (time 1733397470.55 total 20.00 last 413.27) - open/close 10000 (time 1733397473.66 total 23.12 last 497.75) - open/close 15090 (time 1733397483.66 total 33.12 last 508.97) - open/close 20000 (time 1733397493.38 total 42.84 last 505.13) - open/close 25051 (time 1733397503.39 total 52.84 last 505.03) - open/close 30000 (time 1733397512.79 total 62.24 last 526.31) - open/close 35595 (time 1733397522.79 total 72.24 last 559.45) - open/close 40000 (time 1733397530.66 total 80.11 last 559.88) - open/close 45253 (time 1733397540.66 total 90.11 last 525.21) - open/close 50000 (time 1733397549.61 total 99.06 last 530.60) - open/close 54906 (time 1733397559.61 total 109.06 last 490.60) - open/close 60000 (time 1733397568.93 total 118.39 last 546.08) total: 64768 open/close in 127.45 seconds: 508.20 ops/second - /unlink 9262 (time 1733397588.84 total 10.00 last 926.13) - /unlink 19647 (time 1733397598.84 total 20.00 last 1038.41) - /unlink 28495 (time 1733397608.84 total 30.00 last 884.78) - /unlink 36758 (time 1733397618.84 total 40.00 last 826.24) - /unlink 40000 (time 1733397622.37 total 43.54 last 917.58) - /unlink 49399 (time 1733397632.37 total 53.54 last 939.86) - /unlink 58929 (time 1733397642.37 total 63.54 last 952.97) total: 64768 /unlink in 69.30 seconds: 934.55 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5097 (time 1733397689.03 total 10.00 last 509.66) - open/close 10000 (time 1733397698.18 total 19.15 last 535.75) - open/close 15242 (time 1733397708.19 total 29.15 last 524.16) - open/close 20000 (time 1733397717.31 total 38.28 last 521.53) - open/close 25350 (time 1733397727.31 total 48.28 last 534.97) - open/close 30000 (time 1733397736.10 total 57.07 last 528.80) - open/close 35226 (time 1733397746.10 total 67.07 last 522.52) - open/close 40000 (time 1733397755.48 total 76.45 last 508.93) - open/close 45071 (time 1733397765.49 total 86.45 last 507.04) - open/close 50000 (time 1733397774.93 total 95.90 last 521.84) - open/close 55335 (time 1733397784.93 total 105.90 last 533.42) - open/close 60000 (time 1733397793.55 total 114.52 last 541.12) total: 64768 open/close in 123.45 seconds: 524.64 ops/second - /unlink 8990 (time 1733397813.35 total 10.00 last 898.95) - /unlink 18095 (time 1733397823.35 total 20.00 last 910.48) - /unlink 20000 (time 1733397825.37 total 22.02 last 941.75) - /unlink 29116 (time 1733397835.37 total 32.02 last 911.50) - /unlink 38255 (time 1733397845.37 total 42.03 last 913.82) - /unlink 47505 (time 1733397855.37 total 52.03 last 924.93) - /unlink 50000 (time 1733397858.07 total 54.72 last 925.32) - /unlink 59121 (time 1733397868.07 total 64.72 last 912.07) total: 64768 /unlink in 70.98 seconds: 912.49 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5189 (time 1733397914.35 total 10.00 last 518.88) - open/close 10000 (time 1733397923.52 total 19.17 last 524.69) - open/close 15449 (time 1733397933.52 total 29.17 last 544.90) - open/close 20000 (time 1733397941.77 total 37.43 last 551.28) - open/close 25470 (time 1733397951.78 total 47.43 last 546.94) - open/close 30000 (time 1733397960.44 total 56.09 last 522.64) - open/close 35501 (time 1733397970.44 total 66.10 last 550.03) - open/close 40000 (time 1733397978.79 total 74.44 last 538.98) - open/close 45283 (time 1733397988.79 total 84.44 last 528.25) - open/close 50000 (time 1733397997.83 total 93.48 last 521.72) - open/close 55260 (time 1733398007.84 total 103.49 last 525.91) - open/close 60000 (time 1733398016.98 total 112.63 last 518.50) total: 64768 open/close in 121.49 seconds: 533.13 ops/second - /unlink 9204 (time 1733398036.69 total 10.00 last 920.34) - /unlink 18509 (time 1733398046.69 total 20.00 last 930.45) - /unlink 26958 (time 1733398056.69 total 30.00 last 844.85) - /unlink 30000 (time 1733398060.17 total 33.48 last 874.37) - /unlink 38892 (time 1733398070.17 total 43.48 last 889.12) - /unlink 47713 (time 1733398080.17 total 53.48 last 882.05) - /unlink 50000 (time 1733398082.85 total 56.16 last 853.93) - /unlink 58943 (time 1733398092.85 total 66.16 last 894.25) total: 64768 /unlink in 73.08 seconds: 886.23 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4501 (time 1733398135.37 total 10.00 last 450.04) - open/close 8878 (time 1733398145.37 total 20.00 last 437.61) - open/close 10000 (time 1733398148.09 total 22.72 last 413.36) - open/close 14291 (time 1733398158.09 total 32.72 last 429.05) - open/close 18266 (time 1733398168.09 total 42.72 last 397.46) - open/close 20000 (time 1733398171.56 total 46.19 last 499.60) - open/close 24755 (time 1733398181.56 total 56.19 last 475.46) - open/close 29590 (time 1733398191.56 total 66.19 last 483.48) - open/close 34294 (time 1733398201.56 total 76.19 last 470.40) - open/close 39362 (time 1733398211.56 total 86.19 last 506.75) - open/close 43746 (time 1733398221.56 total 96.19 last 438.34) - open/close 48524 (time 1733398231.56 total 106.20 last 477.73) - open/close 50000 (time 1733398234.65 total 109.28 last 478.03) - open/close 53915 (time 1733398244.66 total 119.29 last 391.37) - open/close 58433 (time 1733398254.66 total 129.29 last 451.78) - open/close 60000 (time 1733398258.89 total 133.52 last 370.13) - open/close 63997 (time 1733398268.89 total 143.52 last 399.65) total: 64768 open/close in 145.22 seconds: 446.00 ops/second - /unlink 9317 (time 1733398281.46 total 10.00 last 931.63) - /unlink 18000 (time 1733398291.46 total 20.00 last 868.25) - /unlink 20000 (time 1733398293.82 total 22.36 last 847.03) - /unlink 28821 (time 1733398303.82 total 32.36 last 882.10) - /unlink 38026 (time 1733398313.82 total 42.36 last 920.42) - /unlink 40000 (time 1733398316.04 total 44.58 last 890.68) - /unlink 48929 (time 1733398326.04 total 54.58 last 892.84) - /unlink 57554 (time 1733398336.04 total 64.58 last 862.48) - /unlink 60000 (time 1733398338.76 total 67.30 last 899.28) total: 64768 /unlink in 72.34 seconds: 895.33 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5161 (time 1733398380.24 total 10.00 last 516.05) - open/close 10000 (time 1733398390.03 total 19.79 last 494.57) - open/close 15047 (time 1733398400.03 total 29.79 last 504.64) - open/close 19974 (time 1733398410.03 total 39.79 last 492.61) - open/close 25221 (time 1733398420.03 total 49.79 last 524.70) - open/close 30000 (time 1733398429.46 total 59.22 last 506.74) - open/close 35001 (time 1733398439.46 total 69.22 last 500.04) - open/close 39927 (time 1733398449.47 total 79.22 last 492.57) - open/close 44528 (time 1733398459.47 total 89.22 last 460.09) - open/close 49422 (time 1733398469.47 total 99.22 last 489.36) - open/close 54353 (time 1733398479.47 total 109.22 last 493.05) - open/close 58936 (time 1733398489.47 total 119.22 last 458.29) - open/close 60000 (time 1733398491.54 total 121.30 last 512.47) total: 64768 open/close in 130.93 seconds: 494.67 ops/second - /unlink 8067 (time 1733398512.07 total 10.00 last 806.62) - /unlink 10000 (time 1733398514.33 total 12.26 last 855.72) - /unlink 18444 (time 1733398524.33 total 22.26 last 844.29) - /unlink 27095 (time 1733398534.33 total 32.26 last 864.92) - /unlink 30000 (time 1733398537.70 total 35.63 last 862.20) - /unlink 38807 (time 1733398547.70 total 45.63 last 880.64) - /unlink 47734 (time 1733398557.71 total 55.63 last 892.61) - /unlink 50000 (time 1733398560.33 total 58.26 last 863.90) - /unlink 58745 (time 1733398570.33 total 68.26 last 874.49) total: 64768 /unlink in 74.63 seconds: 867.86 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server unloading modules via unload_modules_local on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 PASS 106 (1193s) Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 oleg137-server: oleg137-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: detected 4 online CPUs by sysfs oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg137-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg137-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1268 sec ========== 06:37:18 (1733398638) === conf-sanity: start cleanup 06:37:18 (1733398638) === === conf-sanity: finish cleanup 06:37:19 (1733398639) ===