-----============= acceptance-small: conf-sanity ============----- Wed Apr 10 10:58:43 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg421-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg421-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg421-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg421-server oleg421-server: oleg421-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: detected 4 online CPUs by sysfs oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg421-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg421-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg421-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg421-server: oleg421-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg421-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:00:25 (1712761225) Stopping clients: oleg421-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg421-client.virtnet /mnt/lustre2 (opts:-f) oleg421-server: oleg421-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: detected 4 online CPUs by sysfs oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg421-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg421-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg421-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg421-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg421-server: oleg421-server.virtnet: executing set_default_debug -1 all pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg421-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg421-client.virtnet: -o user_xattr,flock oleg421-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2361 (time 1712761273.53 total 10.00 last 236.10) - open/close 4768 (time 1712761283.53 total 20.00 last 240.64) - open/close 6916 (time 1712761293.54 total 30.00 last 214.75) - open/close 9127 (time 1712761303.54 total 40.01 last 221.02) - open/close 10000 (time 1712761306.81 total 43.28 last 267.19) - open/close 12349 (time 1712761316.81 total 53.28 last 234.89) - open/close 14555 (time 1712761326.81 total 63.28 last 220.59) - open/close 16764 (time 1712761336.81 total 73.28 last 220.83) - open/close 19181 (time 1712761346.81 total 83.28 last 241.68) - open/close 20000 (time 1712761349.98 total 86.44 last 258.85) - open/close 22862 (time 1712761359.98 total 96.44 last 286.20) - open/close 26521 (time 1712761369.98 total 106.45 last 365.84) - open/close 29556 (time 1712761379.98 total 116.45 last 303.50) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 124.64 seconds: 261.23 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5904 (time 1712761399.14 total 10.00 last 590.33) - /unlink 10000 (time 1712761404.09 total 14.95 last 827.81) - /unlink 18140 (time 1712761414.09 total 24.95 last 813.92) - /unlink 20000 (time 1712761416.60 total 27.47 last 739.19) - /unlink 26604 (time 1712761426.60 total 37.47 last 660.34) - /unlink 30000 (time 1712761431.74 total 42.60 last 661.70) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 46.29 seconds: 703.42 ops/second Waiting for MDT destroys to complete - open/close 2021 (time 1712761481.48 total 10.00 last 202.03) - open/close 4146 (time 1712761491.49 total 20.00 last 212.47) - open/close 6855 (time 1712761501.49 total 30.01 last 270.81) - open/close 9173 (time 1712761511.49 total 40.01 last 231.74) - open/close 10000 (time 1712761514.93 total 43.45 last 240.44) - open/close 12918 (time 1712761524.93 total 53.45 last 291.71) - open/close 15505 (time 1712761534.94 total 63.46 last 258.59) - open/close 17969 (time 1712761544.94 total 73.46 last 246.28) - open/close 20000 (time 1712761553.56 total 82.08 last 235.65) - open/close 22551 (time 1712761563.56 total 92.08 last 255.05) - open/close 24822 (time 1712761573.57 total 102.09 last 226.98) - open/close 27503 (time 1712761583.57 total 112.09 last 267.99) - open/close 29830 (time 1712761593.58 total 122.10 last 232.63) - open/close 32084 (time 1712761603.58 total 132.10 last 225.29) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 134.07 seconds: 242.87 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4815 (time 1712761616.67 total 10.00 last 481.39) - /unlink 9417 (time 1712761626.67 total 20.00 last 460.16) - /unlink 14127 (time 1712761636.67 total 30.00 last 470.93) - /unlink 18451 (time 1712761646.67 total 40.01 last 432.37) - /unlink 20000 (time 1712761649.84 total 43.18 last 488.02) - /unlink 27027 (time 1712761659.84 total 53.18 last 702.63) - /unlink 30000 (time 1712761663.82 total 57.16 last 746.97) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 62.06 seconds: 524.63 ops/second Waiting for MDT destroys to complete - open/close 2630 (time 1712761691.60 total 10.00 last 262.97) - open/close 6406 (time 1712761701.60 total 20.00 last 377.59) - open/close 10000 (time 1712761711.48 total 29.88 last 363.65) - open/close 13995 (time 1712761721.48 total 39.88 last 399.48) - open/close 19049 (time 1712761731.48 total 49.89 last 505.32) - open/close 23710 (time 1712761741.48 total 59.89 last 466.05) - open/close 27003 (time 1712761751.48 total 69.89 last 329.26) - open/close 30000 (time 1712761759.23 total 77.64 last 386.85) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 83.43 seconds: 390.29 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7328 (time 1712761775.92 total 10.00 last 732.73) - /unlink 10000 (time 1712761779.68 total 13.76 last 711.36) - /unlink 17071 (time 1712761789.68 total 23.76 last 707.03) - /unlink 20000 (time 1712761793.57 total 27.64 last 753.70) - /unlink 26477 (time 1712761803.57 total 37.65 last 647.64) - /unlink 30000 (time 1712761809.15 total 43.22 last 631.40) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 47.55 seconds: 684.80 ops/second Waiting for MDT destroys to complete - open/close 3408 (time 1712761845.25 total 10.00 last 340.80) - open/close 7601 (time 1712761855.25 total 20.00 last 419.24) - open/close 10000 (time 1712761862.41 total 27.16 last 335.15) - open/close 13339 (time 1712761872.41 total 37.16 last 333.82) - open/close 16362 (time 1712761882.41 total 47.16 last 302.21) - open/close 19563 (time 1712761892.41 total 57.17 last 320.09) - open/close 22839 (time 1712761902.41 total 67.17 last 327.51) - open/close 26469 (time 1712761912.42 total 77.17 last 362.89) - open/close 30000 (time 1712761920.90 total 85.65 last 416.48) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 92.67 seconds: 351.37 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7934 (time 1712761938.86 total 10.00 last 793.39) - /unlink 10000 (time 1712761941.02 total 12.15 last 959.86) - /unlink 18977 (time 1712761951.02 total 22.15 last 897.69) - /unlink 24492 (time 1712761961.02 total 32.15 last 551.38) - /unlink 30000 (time 1712761968.77 total 39.91 last 710.36) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 42.74 seconds: 761.87 ops/second Waiting for MDT destroys to complete - open/close 5111 (time 1712761994.73 total 10.00 last 511.02) - open/close 8873 (time 1712762004.74 total 20.00 last 376.12) - open/close 10000 (time 1712762007.91 total 23.18 last 354.69) - open/close 14048 (time 1712762017.92 total 33.18 last 404.75) - open/close 18131 (time 1712762027.92 total 43.18 last 408.21) - open/close 20000 (time 1712762031.93 total 47.20 last 465.93) - open/close 24367 (time 1712762041.93 total 57.20 last 436.61) - open/close 28619 (time 1712762051.93 total 67.20 last 425.18) - open/close 30000 (time 1712762055.27 total 70.54 last 413.64) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 76.50 seconds: 425.65 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8844 (time 1712762072.13 total 10.00 last 884.34) - /unlink 17367 (time 1712762082.13 total 20.00 last 852.18) - /unlink 20000 (time 1712762086.37 total 24.24 last 620.78) - /unlink 28014 (time 1712762096.38 total 34.24 last 801.34) - /unlink 30000 (time 1712762099.23 total 37.10 last 695.50) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 41.00 seconds: 794.15 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg421-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg421-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg421-server stop mds service on oleg421-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg421-server unloading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-client: ssh exited with exit code 2 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 PASS 106 (937s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg421-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg421-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg421-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg421-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg421-client: oleg421-server: ssh exited with exit code 2 oleg421-server: oleg421-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: detected 4 online CPUs by sysfs oleg421-server: Force libcfs to create 2 CPU partitions oleg421-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg421-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg421-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1057 sec ========== 11:16:21 (1712762181) === conf-sanity: start cleanup 11:16:22 (1712762182) === === conf-sanity: finish cleanup 11:16:22 (1712762182) ===