-----============= acceptance-small: conf-sanity ============----- Fri May 10 08:07:34 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg414-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg414-server oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg414-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg414-server: oleg414-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg414-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:08:32 (1715342912) Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg414-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock oleg414-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5580 (time 1715342947.02 total 10.00 last 557.93) - open/close 10000 (time 1715342954.90 total 17.88 last 560.99) - open/close 15466 (time 1715342964.90 total 27.88 last 546.58) - open/close 20000 (time 1715342973.22 total 36.21 last 544.55) - open/close 25449 (time 1715342983.23 total 46.21 last 544.86) - open/close 30000 (time 1715342991.34 total 54.32 last 561.13) - open/close 35219 (time 1715343001.34 total 64.32 last 521.90) - open/close 40000 (time 1715343009.85 total 72.83 last 561.51) - open/close 45437 (time 1715343019.85 total 82.83 last 543.67) - open/close 50000 (time 1715343028.64 total 91.62 last 519.28) - open/close 55191 (time 1715343038.64 total 101.62 last 519.06) - open/close 60000 (time 1715343048.56 total 111.55 last 484.56) - open/close 64140 (time 1715343058.56 total 121.55 last 413.99) total: 64768 open/close in 122.79 seconds: 527.47 ops/second - /unlink 7597 (time 1715343070.84 total 10.00 last 759.67) - /unlink 10000 (time 1715343073.65 total 12.82 last 852.71) - /unlink 17163 (time 1715343083.66 total 22.82 last 716.22) - /unlink 20000 (time 1715343086.80 total 25.97 last 900.70) - /unlink 28558 (time 1715343096.81 total 35.97 last 855.74) - /unlink 37609 (time 1715343106.81 total 45.97 last 905.04) - /unlink 40000 (time 1715343109.48 total 48.64 last 895.74) - /unlink 49102 (time 1715343119.48 total 58.64 last 910.19) - /unlink 58256 (time 1715343129.48 total 68.64 last 915.40) total: 64768 /unlink in 75.58 seconds: 856.92 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5519 (time 1715343170.99 total 10.00 last 551.86) - open/close 10000 (time 1715343178.99 total 18.01 last 559.55) - open/close 15199 (time 1715343189.00 total 28.01 last 519.84) - open/close 19670 (time 1715343199.00 total 38.01 last 447.07) - open/close 24817 (time 1715343209.00 total 48.01 last 514.65) - open/close 30000 (time 1715343218.51 total 57.53 last 544.69) - open/close 35285 (time 1715343228.51 total 67.53 last 528.48) - open/close 40000 (time 1715343237.88 total 76.90 last 503.10) - open/close 45235 (time 1715343247.89 total 86.90 last 523.42) - open/close 50000 (time 1715343257.82 total 96.83 last 479.78) - open/close 54630 (time 1715343267.82 total 106.83 last 462.95) - open/close 59312 (time 1715343277.82 total 116.83 last 468.14) - open/close 64272 (time 1715343287.82 total 126.84 last 495.94) total: 64768 open/close in 127.83 seconds: 506.66 ops/second - /unlink 7970 (time 1715343299.73 total 10.00 last 796.94) - /unlink 10000 (time 1715343302.99 total 13.26 last 622.16) - /unlink 18176 (time 1715343312.99 total 23.26 last 817.58) - /unlink 20000 (time 1715343315.07 total 25.34 last 877.24) - /unlink 27465 (time 1715343325.07 total 35.34 last 746.38) - /unlink 30000 (time 1715343328.91 total 39.19 last 659.33) - /unlink 37145 (time 1715343338.92 total 49.19 last 714.44) - /unlink 40000 (time 1715343342.89 total 53.16 last 718.36) - /unlink 47521 (time 1715343352.89 total 63.17 last 752.05) - /unlink 50000 (time 1715343355.95 total 66.22 last 811.04) - /unlink 58692 (time 1715343365.95 total 76.22 last 869.14) total: 64768 /unlink in 82.96 seconds: 780.71 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4904 (time 1715343401.64 total 10.00 last 490.38) - open/close 8113 (time 1715343411.64 total 20.00 last 320.79) - open/close 10000 (time 1715343415.10 total 23.46 last 545.53) - open/close 15131 (time 1715343425.10 total 33.46 last 513.06) - open/close 19771 (time 1715343435.10 total 43.46 last 463.98) - open/close 24163 (time 1715343445.10 total 53.47 last 439.13) - open/close 28654 (time 1715343455.10 total 63.47 last 449.02) - open/close 30000 (time 1715343458.85 total 67.21 last 359.49) - open/close 33542 (time 1715343468.85 total 77.21 last 354.14) - open/close 37542 (time 1715343478.85 total 87.21 last 399.97) - open/close 40000 (time 1715343487.00 total 95.37 last 301.45) - open/close 43504 (time 1715343497.01 total 105.37 last 350.30) - open/close 47123 (time 1715343507.01 total 115.37 last 361.85) - open/close 50000 (time 1715343514.70 total 123.06 last 374.21) - open/close 53008 (time 1715343524.70 total 133.06 last 300.78) - open/close 56190 (time 1715343534.70 total 143.06 last 318.13) - open/close 59562 (time 1715343544.70 total 153.07 last 337.13) - open/close 63158 (time 1715343554.70 total 163.07 last 359.56) total: 64768 open/close in 166.81 seconds: 388.27 ops/second - /unlink 6874 (time 1715343569.29 total 10.00 last 687.29) - /unlink 10000 (time 1715343573.71 total 14.41 last 708.68) - /unlink 17609 (time 1715343583.71 total 24.41 last 760.85) - /unlink 20000 (time 1715343587.64 total 28.34 last 608.18) - /unlink 26770 (time 1715343597.64 total 38.35 last 676.93) - /unlink 30000 (time 1715343602.55 total 43.26 last 657.91) - /unlink 35479 (time 1715343612.55 total 53.26 last 547.86) - /unlink 40000 (time 1715343619.48 total 60.19 last 652.21) - /unlink 45293 (time 1715343629.48 total 70.19 last 529.22) - /unlink 50000 (time 1715343636.99 total 77.69 last 627.27) - /unlink 56357 (time 1715343646.99 total 87.69 last 635.66) - /unlink 60000 (time 1715343652.74 total 93.44 last 633.47) total: 64768 /unlink in 101.77 seconds: 636.44 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3022 (time 1715343686.50 total 10.00 last 302.09) - open/close 5793 (time 1715343696.50 total 20.01 last 277.00) - open/close 8807 (time 1715343706.51 total 30.01 last 301.31) - open/close 10000 (time 1715343710.00 total 33.51 last 341.25) - open/close 12842 (time 1715343720.00 total 43.51 last 284.17) - open/close 15580 (time 1715343730.01 total 53.51 last 273.71) - open/close 18450 (time 1715343740.01 total 63.51 last 286.92) - open/close 20000 (time 1715343745.93 total 69.43 last 261.89) - open/close 22701 (time 1715343755.93 total 79.43 last 270.03) - open/close 25258 (time 1715343765.93 total 89.43 last 255.69) - open/close 28379 (time 1715343775.93 total 99.44 last 312.05) - open/close 30000 (time 1715343781.95 total 105.45 last 269.34) - open/close 32696 (time 1715343791.95 total 115.46 last 269.53) - open/close 35613 (time 1715343801.95 total 125.46 last 291.67) - open/close 39033 (time 1715343811.95 total 135.46 last 342.00) - open/close 40000 (time 1715343814.29 total 137.80 last 413.14) - open/close 43483 (time 1715343824.29 total 147.80 last 348.28) - open/close 45988 (time 1715343834.30 total 157.80 last 250.43) - open/close 48838 (time 1715343844.30 total 167.80 last 284.94) - open/close 50000 (time 1715343848.45 total 171.96 last 279.72) - open/close 52702 (time 1715343858.46 total 181.96 last 270.11) - open/close 56006 (time 1715343868.46 total 191.97 last 330.28) - open/close 59326 (time 1715343878.46 total 201.97 last 331.87) - open/close 60000 (time 1715343880.60 total 204.10 last 315.88) - open/close 63086 (time 1715343890.60 total 214.11 last 308.47) total: 64768 open/close in 218.94 seconds: 295.82 ops/second - /unlink 6822 (time 1715343906.60 total 10.00 last 682.17) - /unlink 10000 (time 1715343912.15 total 15.54 last 573.22) - /unlink 14705 (time 1715343922.15 total 25.55 last 470.45) - /unlink 19926 (time 1715343932.15 total 35.55 last 522.06) - /unlink 24915 (time 1715343942.15 total 45.55 last 498.89) - /unlink 29883 (time 1715343952.15 total 55.55 last 496.78) - /unlink 34743 (time 1715343962.15 total 65.55 last 485.94) - /unlink 40000 (time 1715343971.66 total 75.06 last 552.53) - /unlink 45314 (time 1715343981.66 total 85.06 last 531.39) - /unlink 50000 (time 1715343988.35 total 91.75 last 701.12) - /unlink 55659 (time 1715343998.35 total 101.75 last 565.80) - /unlink 60000 (time 1715344006.86 total 110.26 last 510.03) total: 64768 /unlink in 119.42 seconds: 542.34 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2931 (time 1715344041.91 total 10.00 last 293.05) - open/close 6346 (time 1715344051.92 total 20.00 last 341.40) - open/close 9964 (time 1715344061.92 total 30.01 last 361.72) - open/close 12908 (time 1715344071.92 total 40.01 last 294.36) - open/close 15838 (time 1715344081.92 total 50.01 last 292.97) - open/close 18691 (time 1715344091.92 total 60.01 last 285.30) - open/close 20000 (time 1715344096.05 total 64.13 last 317.32) - open/close 22845 (time 1715344106.05 total 74.14 last 284.43) - open/close 25700 (time 1715344116.05 total 84.14 last 285.48) - open/close 28896 (time 1715344126.05 total 94.14 last 319.52) - open/close 30000 (time 1715344130.40 total 98.48 last 254.10) - open/close 32500 (time 1715344140.40 total 108.49 last 249.91) - open/close 35072 (time 1715344150.40 total 118.49 last 257.15) - open/close 37577 (time 1715344160.40 total 128.49 last 250.44) - open/close 40000 (time 1715344169.90 total 137.99 last 255.04) - open/close 42638 (time 1715344179.91 total 148.00 last 263.73) - open/close 45097 (time 1715344189.91 total 158.00 last 245.85) - open/close 47659 (time 1715344199.91 total 168.00 last 256.12) - open/close 50000 (time 1715344208.08 total 176.17 last 286.61) - open/close 52970 (time 1715344218.08 total 186.17 last 296.89) - open/close 55458 (time 1715344228.09 total 196.18 last 248.73) - open/close 58061 (time 1715344238.09 total 206.18 last 260.21) - open/close 60000 (time 1715344245.83 total 213.92 last 250.40) - open/close 62455 (time 1715344255.84 total 223.92 last 245.47) total: 64768 open/close in 233.01 seconds: 277.96 ops/second - /unlink 5630 (time 1715344276.11 total 10.00 last 562.99) - /unlink 10000 (time 1715344284.03 total 17.92 last 551.97) - /unlink 14750 (time 1715344294.03 total 27.92 last 475.00) - /unlink 20000 (time 1715344303.69 total 37.58 last 543.27) - /unlink 24777 (time 1715344313.69 total 47.58 last 477.63) - /unlink 29431 (time 1715344323.69 total 57.58 last 465.30) - /unlink 34293 (time 1715344333.70 total 67.59 last 486.11) - /unlink 39985 (time 1715344343.70 total 77.59 last 569.13) - /unlink 44958 (time 1715344353.70 total 87.59 last 497.28) - /unlink 50000 (time 1715344363.25 total 97.14 last 528.08) - /unlink 54858 (time 1715344373.25 total 107.14 last 485.77) - /unlink 59676 (time 1715344383.25 total 117.14 last 481.77) - /unlink 64598 (time 1715344393.25 total 127.14 last 492.19) total: 64768 /unlink in 127.52 seconds: 507.90 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server unloading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 PASS 106 (1522s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg414-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg414-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1602 sec ========== 08:34:17 (1715344457) === conf-sanity: start cleanup 08:34:18 (1715344458) === === conf-sanity: finish cleanup 08:34:18 (1715344458) ===