-----============= acceptance-small: conf-sanity ============----- Wed Nov 13 12:08:28 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg105-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg105-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg105-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg105-server oleg105-server: oleg105-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: detected 4 online CPUs by sysfs oleg105-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg105-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg105-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg105-server: oleg105-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg105-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:09:28 (1731517768) Stopping clients: oleg105-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg105-client.virtnet /mnt/lustre2 (opts:-f) oleg105-server: oleg105-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: detected 4 online CPUs by sysfs oleg105-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg105-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg105-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg105-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg105-server: oleg105-server.virtnet: executing set_default_debug -1 all pdsh@oleg105-client: oleg105-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg105-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg105-client.virtnet: -o user_xattr,flock 192.168.201.105@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 6394 (time 1731517801.43 total 10.00 last 639.35) - open/close 10000 (time 1731517807.42 total 15.99 last 602.35) - open/close 16188 (time 1731517817.42 total 25.99 last 618.75) - open/close 20000 (time 1731517824.41 total 32.98 last 545.20) - open/close 25430 (time 1731517834.41 total 42.98 last 542.95) - open/close 30000 (time 1731517842.83 total 51.40 last 543.05) - open/close 35097 (time 1731517852.83 total 61.40 last 509.69) - open/close 40000 (time 1731517862.17 total 70.73 last 525.06) - open/close 45122 (time 1731517872.17 total 80.74 last 512.12) - open/close 50000 (time 1731517881.57 total 90.14 last 518.71) - open/close 55758 (time 1731517891.58 total 100.14 last 575.73) - open/close 60000 (time 1731517899.80 total 108.37 last 515.47) total: 64768 open/close in 117.68 seconds: 550.36 ops/second - /unlink 9107 (time 1731517920.00 total 10.00 last 910.66) - /unlink 18106 (time 1731517930.00 total 20.00 last 899.83) - /unlink 20000 (time 1731517932.07 total 22.07 last 914.18) - /unlink 28774 (time 1731517942.07 total 32.07 last 877.40) - /unlink 37388 (time 1731517952.07 total 42.07 last 861.31) - /unlink 40000 (time 1731517955.15 total 45.15 last 850.00) - /unlink 48369 (time 1731517965.15 total 55.15 last 836.89) - /unlink 57121 (time 1731517975.15 total 65.15 last 875.16) - /unlink 60000 (time 1731517978.37 total 68.37 last 894.29) total: 64768 /unlink in 73.44 seconds: 881.93 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5026 (time 1731518020.60 total 10.00 last 502.52) - open/close 9580 (time 1731518030.61 total 20.01 last 455.21) - open/close 13200 (time 1731518040.61 total 30.01 last 361.96) - open/close 17235 (time 1731518050.61 total 40.01 last 403.44) - open/close 20000 (time 1731518056.51 total 45.91 last 468.65) - open/close 25325 (time 1731518066.51 total 55.91 last 532.48) - open/close 30000 (time 1731518075.34 total 64.74 last 529.24) - open/close 35189 (time 1731518085.34 total 74.74 last 518.87) - open/close 40000 (time 1731518094.34 total 83.74 last 534.60) - open/close 45311 (time 1731518104.34 total 93.74 last 531.02) - open/close 50000 (time 1731518113.30 total 102.70 last 523.39) - open/close 55301 (time 1731518123.30 total 112.70 last 530.10) - open/close 60000 (time 1731518132.50 total 121.90 last 510.91) total: 64768 open/close in 131.23 seconds: 493.53 ops/second - /unlink 9334 (time 1731518152.69 total 10.00 last 933.39) - /unlink 18773 (time 1731518162.69 total 20.00 last 943.85) - /unlink 27851 (time 1731518172.69 total 30.00 last 907.71) - /unlink 30000 (time 1731518175.27 total 32.59 last 831.15) - /unlink 38509 (time 1731518185.27 total 42.59 last 850.89) - /unlink 47955 (time 1731518195.27 total 52.59 last 944.52) - /unlink 50000 (time 1731518197.38 total 54.70 last 969.63) - /unlink 59438 (time 1731518207.38 total 64.70 last 943.72) total: 64768 /unlink in 70.63 seconds: 917.05 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4602 (time 1731518251.46 total 10.00 last 460.18) - open/close 9341 (time 1731518261.46 total 20.00 last 473.89) - open/close 14911 (time 1731518271.46 total 30.00 last 556.93) - open/close 20000 (time 1731518281.21 total 39.75 last 522.14) - open/close 24703 (time 1731518291.21 total 49.75 last 470.30) - open/close 28290 (time 1731518301.21 total 59.75 last 358.63) - open/close 30000 (time 1731518305.41 total 63.95 last 407.07) - open/close 33353 (time 1731518315.41 total 73.95 last 335.26) - open/close 35933 (time 1731518325.42 total 83.96 last 257.90) - open/close 38605 (time 1731518335.42 total 93.96 last 267.18) - open/close 40000 (time 1731518340.64 total 99.17 last 267.38) - open/close 44075 (time 1731518350.64 total 109.17 last 407.48) - open/close 49091 (time 1731518360.64 total 119.18 last 501.50) - open/close 52872 (time 1731518370.64 total 129.18 last 378.04) - open/close 55942 (time 1731518380.64 total 139.18 last 306.93) - open/close 60000 (time 1731518390.45 total 148.99 last 413.57) total: 64768 open/close in 157.72 seconds: 410.64 ops/second - /unlink 7018 (time 1731518410.05 total 10.00 last 701.79) - /unlink 10000 (time 1731518414.99 total 14.94 last 603.51) - /unlink 16056 (time 1731518424.99 total 24.94 last 605.57) - /unlink 20000 (time 1731518431.40 total 31.35 last 615.10) - /unlink 28745 (time 1731518441.40 total 41.35 last 874.45) - /unlink 37391 (time 1731518451.40 total 51.36 last 864.53) - /unlink 40000 (time 1731518455.14 total 55.09 last 698.94) - /unlink 45336 (time 1731518465.14 total 65.09 last 533.59) - /unlink 50000 (time 1731518473.43 total 73.39 last 562.02) - /unlink 55818 (time 1731518483.43 total 83.39 last 581.76) - /unlink 60000 (time 1731518489.75 total 89.70 last 662.41) total: 64768 /unlink in 94.81 seconds: 683.12 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3878 (time 1731518526.71 total 10.00 last 387.69) - open/close 7738 (time 1731518536.71 total 20.00 last 385.94) - open/close 10000 (time 1731518542.82 total 26.12 last 370.11) - open/close 14685 (time 1731518552.83 total 36.12 last 468.23) - open/close 17596 (time 1731518562.83 total 46.12 last 291.08) - open/close 20000 (time 1731518568.66 total 51.96 last 412.14) - open/close 25615 (time 1731518578.66 total 61.96 last 561.47) - open/close 30000 (time 1731518586.54 total 69.84 last 556.46) - open/close 35046 (time 1731518596.54 total 79.84 last 504.57) - open/close 40000 (time 1731518606.41 total 89.70 last 502.11) - open/close 44994 (time 1731518616.41 total 99.70 last 499.33) - open/close 50000 (time 1731518626.28 total 109.57 last 507.38) - open/close 54978 (time 1731518636.28 total 119.57 last 497.78) - open/close 60000 (time 1731518646.01 total 129.30 last 516.00) total: 64768 open/close in 138.26 seconds: 468.46 ops/second - /unlink 8967 (time 1731518665.83 total 10.00 last 896.67) - /unlink 18576 (time 1731518675.83 total 20.00 last 960.81) - /unlink 27937 (time 1731518685.83 total 30.00 last 936.03) - /unlink 30000 (time 1731518687.88 total 32.05 last 1006.68) - /unlink 39045 (time 1731518697.88 total 42.05 last 904.43) - /unlink 48219 (time 1731518707.88 total 52.05 last 917.35) - /unlink 50000 (time 1731518710.01 total 54.18 last 836.88) - /unlink 59247 (time 1731518720.01 total 64.18 last 924.69) total: 64768 /unlink in 70.02 seconds: 925.04 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5178 (time 1731518766.75 total 10.00 last 517.73) - open/close 9170 (time 1731518776.75 total 20.00 last 399.09) - open/close 10000 (time 1731518779.39 total 22.64 last 314.45) - open/close 14190 (time 1731518789.39 total 32.65 last 418.93) - open/close 17840 (time 1731518799.39 total 42.65 last 364.94) - open/close 20000 (time 1731518806.47 total 49.73 last 305.06) - open/close 23947 (time 1731518816.47 total 59.73 last 394.66) - open/close 28163 (time 1731518826.48 total 69.73 last 421.57) - open/close 30000 (time 1731518831.12 total 74.38 last 395.08) - open/close 34103 (time 1731518841.13 total 84.38 last 410.25) - open/close 37806 (time 1731518851.13 total 94.38 last 370.19) - open/close 40000 (time 1731518857.40 total 100.66 last 349.70) - open/close 42583 (time 1731518867.41 total 110.66 last 258.20) - open/close 45094 (time 1731518877.41 total 120.66 last 251.08) - open/close 47638 (time 1731518887.41 total 130.67 last 254.32) - open/close 50000 (time 1731518896.84 total 140.10 last 250.44) - open/close 52513 (time 1731518906.85 total 150.10 last 251.22) - open/close 55063 (time 1731518916.85 total 160.10 last 254.94) - open/close 57495 (time 1731518926.85 total 170.10 last 243.19) - open/close 59852 (time 1731518936.85 total 180.11 last 235.63) - open/close 62332 (time 1731518946.85 total 190.11 last 247.96) total: 64768 open/close in 199.91 seconds: 323.98 ops/second - /unlink 4973 (time 1731518967.99 total 10.00 last 497.23) - /unlink 10000 (time 1731518976.30 total 18.31 last 604.93) - /unlink 14691 (time 1731518986.30 total 28.31 last 469.02) - /unlink 19592 (time 1731518996.31 total 38.31 last 490.02) - /unlink 24376 (time 1731519006.31 total 48.31 last 478.39) - /unlink 29091 (time 1731519016.31 total 58.32 last 471.43) - /unlink 34064 (time 1731519026.31 total 68.32 last 497.22) - /unlink 38869 (time 1731519036.31 total 78.32 last 480.45) - /unlink 40000 (time 1731519038.67 total 80.68 last 478.52) - /unlink 44789 (time 1731519048.68 total 90.68 last 478.82) - /unlink 50000 (time 1731519057.03 total 99.04 last 623.73) - /unlink 54526 (time 1731519067.03 total 109.04 last 452.57) - /unlink 59820 (time 1731519077.03 total 119.04 last 529.37) - /unlink 64740 (time 1731519087.03 total 129.04 last 491.90) total: 64768 /unlink in 129.10 seconds: 501.68 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg105-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg105-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg105-server stop mds service on oleg105-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg105-server unloading modules via unload_modules_local on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-client: ssh exited with exit code 2 pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 PASS 106 (1361s) Stopping clients: oleg105-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg105-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg105-client: oleg105-server: ssh exited with exit code 2 oleg105-server: oleg105-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg105-server' oleg105-server: oleg105-server.virtnet: executing load_modules_local oleg105-server: Loading modules from /home/green/git/lustre-release/lustre oleg105-server: detected 4 online CPUs by sysfs oleg105-server: Force libcfs to create 2 CPU partitions oleg105-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg105-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1445 sec ========== 12:32:33 (1731519153) === conf-sanity: start cleanup 12:32:34 (1731519154) === === conf-sanity: finish cleanup 12:32:35 (1731519155) ===