-----============= acceptance-small: conf-sanity ============----- Mon May 13 03:59:55 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg159-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg159-server oleg159-server: oleg159-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: detected 4 online CPUs by sysfs oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg159-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:00:53 (1715587253) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) oleg159-server: oleg159-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: detected 4 online CPUs by sysfs oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg159-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg159-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock oleg159-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5637 (time 1715587287.76 total 10.00 last 563.60) - open/close 10000 (time 1715587296.38 total 18.62 last 506.34) - open/close 15782 (time 1715587306.38 total 28.62 last 578.15) - open/close 20000 (time 1715587313.31 total 35.55 last 608.70) - open/close 25311 (time 1715587323.31 total 45.55 last 531.05) - open/close 30000 (time 1715587331.76 total 53.99 last 555.23) - open/close 34835 (time 1715587341.76 total 64.00 last 483.45) - open/close 39755 (time 1715587351.76 total 74.00 last 491.94) - open/close 45418 (time 1715587361.76 total 84.00 last 566.23) - open/close 50000 (time 1715587369.95 total 92.19 last 559.32) - open/close 55160 (time 1715587379.95 total 102.19 last 515.90) - open/close 59966 (time 1715587389.96 total 112.19 last 480.51) - open/close 64658 (time 1715587399.96 total 122.20 last 469.16) total: 64768 open/close in 122.40 seconds: 529.14 ops/second - /unlink 7660 (time 1715587411.04 total 10.00 last 765.97) - /unlink 10000 (time 1715587413.95 total 12.91 last 804.81) - /unlink 17884 (time 1715587423.95 total 22.91 last 788.34) - /unlink 20000 (time 1715587426.67 total 25.63 last 778.68) - /unlink 27689 (time 1715587436.67 total 35.63 last 768.82) - /unlink 30000 (time 1715587439.46 total 38.42 last 827.27) - /unlink 37898 (time 1715587449.46 total 48.42 last 789.79) - /unlink 40000 (time 1715587452.00 total 50.96 last 829.12) - /unlink 48552 (time 1715587462.00 total 60.96 last 855.14) - /unlink 56980 (time 1715587472.00 total 70.96 last 842.70) - /unlink 60000 (time 1715587475.66 total 74.62 last 824.92) total: 64768 /unlink in 79.84 seconds: 811.20 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5388 (time 1715587511.54 total 10.00 last 538.75) - open/close 10000 (time 1715587520.65 total 19.11 last 506.15) - open/close 14763 (time 1715587530.65 total 29.11 last 476.27) - open/close 19694 (time 1715587540.65 total 39.11 last 493.07) - open/close 24535 (time 1715587550.66 total 49.12 last 484.04) - open/close 29490 (time 1715587560.66 total 59.12 last 495.42) - open/close 34355 (time 1715587570.66 total 69.12 last 486.43) - open/close 39239 (time 1715587580.66 total 79.12 last 488.38) - open/close 44289 (time 1715587590.66 total 89.12 last 504.93) - open/close 49032 (time 1715587600.66 total 99.12 last 474.23) - open/close 50000 (time 1715587602.80 total 101.26 last 452.15) - open/close 54713 (time 1715587612.80 total 111.26 last 471.29) - open/close 60000 (time 1715587622.24 total 120.70 last 560.07) total: 64768 open/close in 128.81 seconds: 502.84 ops/second - /unlink 10000 (time 1715587641.08 total 9.95 last 1004.55) - /unlink 19024 (time 1715587651.08 total 19.96 last 902.32) - /unlink 27164 (time 1715587661.08 total 29.96 last 813.91) - /unlink 30000 (time 1715587664.08 total 32.96 last 945.53) - /unlink 39438 (time 1715587674.08 total 42.96 last 943.75) - /unlink 49292 (time 1715587684.08 total 52.96 last 985.38) - /unlink 58875 (time 1715587694.08 total 62.96 last 958.29) total: 64768 /unlink in 68.70 seconds: 942.75 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4901 (time 1715587741.22 total 10.00 last 490.01) - open/close 9578 (time 1715587751.22 total 20.00 last 467.68) - open/close 14816 (time 1715587761.22 total 30.00 last 523.71) - open/close 20000 (time 1715587770.98 total 39.76 last 531.17) - open/close 25156 (time 1715587780.99 total 49.76 last 515.56) - open/close 29974 (time 1715587790.99 total 59.77 last 481.76) - open/close 34982 (time 1715587800.99 total 69.77 last 500.77) - open/close 40000 (time 1715587810.83 total 79.61 last 509.99) - open/close 45021 (time 1715587820.83 total 89.61 last 502.02) - open/close 49766 (time 1715587830.83 total 99.61 last 474.44) - open/close 54627 (time 1715587840.83 total 109.61 last 486.06) - open/close 59504 (time 1715587850.83 total 119.61 last 487.63) - open/close 64494 (time 1715587860.83 total 129.61 last 498.96) total: 64768 open/close in 130.25 seconds: 497.25 ops/second - /unlink 8077 (time 1715587872.37 total 10.00 last 807.60) - /unlink 10000 (time 1715587874.59 total 12.21 last 868.82) - /unlink 18465 (time 1715587884.59 total 22.22 last 846.41) - /unlink 26762 (time 1715587894.59 total 32.22 last 829.65) - /unlink 30000 (time 1715587898.50 total 36.13 last 828.26) - /unlink 38415 (time 1715587908.50 total 46.13 last 841.40) - /unlink 40000 (time 1715587910.56 total 48.19 last 767.63) - /unlink 48710 (time 1715587920.56 total 58.19 last 870.91) - /unlink 57543 (time 1715587930.56 total 68.19 last 883.29) - /unlink 60000 (time 1715587933.21 total 70.84 last 927.13) total: 64768 /unlink in 76.14 seconds: 850.62 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5420 (time 1715587972.05 total 10.00 last 541.96) - open/close 10000 (time 1715587980.88 total 18.83 last 518.80) - open/close 15201 (time 1715587990.88 total 28.83 last 520.01) - open/close 20000 (time 1715588000.21 total 38.16 last 514.39) - open/close 25130 (time 1715588010.21 total 48.16 last 512.94) - open/close 30000 (time 1715588019.54 total 57.49 last 521.94) - open/close 35251 (time 1715588029.54 total 67.49 last 525.05) - open/close 40000 (time 1715588038.70 total 76.65 last 518.33) - open/close 45093 (time 1715588048.71 total 86.66 last 509.21) - open/close 50000 (time 1715588058.31 total 96.26 last 510.78) - open/close 55245 (time 1715588068.31 total 106.27 last 524.41) - open/close 60000 (time 1715588077.26 total 115.21 last 531.62) total: 64768 open/close in 124.34 seconds: 520.88 ops/second - /unlink 8438 (time 1715588097.24 total 10.00 last 843.78) - /unlink 17223 (time 1715588107.24 total 20.00 last 878.39) - /unlink 20000 (time 1715588110.41 total 23.17 last 876.30) - /unlink 28635 (time 1715588120.41 total 33.17 last 863.39) - /unlink 37505 (time 1715588130.41 total 43.17 last 886.93) - /unlink 40000 (time 1715588133.23 total 45.98 last 887.55) - /unlink 48853 (time 1715588143.23 total 55.98 last 885.24) - /unlink 57336 (time 1715588153.23 total 65.99 last 848.24) - /unlink 60000 (time 1715588156.81 total 69.57 last 743.73) total: 64768 /unlink in 75.80 seconds: 854.50 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5458 (time 1715588197.36 total 10.00 last 545.76) - open/close 10000 (time 1715588206.54 total 19.18 last 494.90) - open/close 15273 (time 1715588216.54 total 29.18 last 527.28) - open/close 20000 (time 1715588225.47 total 38.11 last 529.35) - open/close 25118 (time 1715588235.47 total 48.11 last 511.75) - open/close 29588 (time 1715588245.47 total 58.11 last 446.99) - open/close 33796 (time 1715588255.47 total 68.11 last 420.77) - open/close 38020 (time 1715588265.47 total 78.11 last 422.38) - open/close 40000 (time 1715588269.33 total 81.97 last 512.80) - open/close 44850 (time 1715588279.33 total 91.97 last 484.92) - open/close 49356 (time 1715588289.33 total 101.97 last 450.55) - open/close 54323 (time 1715588299.33 total 111.98 last 496.62) - open/close 58812 (time 1715588309.33 total 121.98 last 448.85) - open/close 60000 (time 1715588311.90 total 124.54 last 463.60) - open/close 64616 (time 1715588321.90 total 134.54 last 461.57) total: 64768 open/close in 134.80 seconds: 480.48 ops/second - /unlink 6171 (time 1715588333.02 total 10.00 last 617.08) - /unlink 10000 (time 1715588337.85 total 14.83 last 792.56) - /unlink 17102 (time 1715588347.86 total 24.83 last 710.14) - /unlink 20000 (time 1715588352.72 total 29.70 last 595.54) - /unlink 26990 (time 1715588362.72 total 39.70 last 698.97) - /unlink 30000 (time 1715588366.45 total 43.42 last 807.98) - /unlink 37241 (time 1715588376.45 total 53.42 last 724.09) - /unlink 40000 (time 1715588379.56 total 56.54 last 886.33) - /unlink 46188 (time 1715588389.56 total 66.54 last 618.80) - /unlink 50000 (time 1715588394.64 total 71.62 last 750.48) - /unlink 56595 (time 1715588404.64 total 81.62 last 659.50) - /unlink 60000 (time 1715588409.02 total 86.00 last 776.50) total: 64768 /unlink in 92.48 seconds: 700.35 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server unloading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 PASS 106 (1199s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg159-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg159-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 oleg159-server: oleg159-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: detected 4 online CPUs by sysfs oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1277 sec ========== 04:21:13 (1715588473) === conf-sanity: start cleanup 04:21:13 (1715588473) === === conf-sanity: finish cleanup 04:21:13 (1715588473) ===