-----============= acceptance-small: conf-sanity ============----- Sun Apr 14 02:45:05 EDT 2024 excepting tests: 32newtarball 110 41c Stopping clients: oleg108-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg108-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg108-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg108-server oleg108-server: oleg108-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: Force libcfs to create 2 CPU partitions oleg108-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg108-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg108-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg108-server: oleg108-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg108-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg108-server debug_raw_pointers=Y debug_raw_pointers=Y == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:46:28 (1713077188) Stopping clients: oleg108-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg108-client.virtnet /mnt/lustre2 (opts:-f) oleg108-server: oleg108-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: Force libcfs to create 2 CPU partitions oleg108-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg108-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg108-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg108-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg108-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg108-server: oleg108-server.virtnet: executing set_default_debug -1 all pdsh@oleg108-client: oleg108-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg108-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg108-client.virtnet: -o user_xattr,flock oleg108-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5056 (time 1713077228.24 total 10.00 last 505.50) - open/close 9487 (time 1713077238.24 total 20.00 last 443.07) - open/close 13681 (time 1713077248.24 total 30.00 last 419.40) - open/close 18837 (time 1713077258.24 total 40.00 last 515.57) - open/close 20000 (time 1713077260.77 total 42.54 last 458.85) - open/close 24584 (time 1713077270.77 total 52.54 last 458.34) - open/close 28257 (time 1713077280.77 total 62.54 last 367.27) - open/close 30000 (time 1713077285.58 total 67.35 last 362.59) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 75.12 seconds: 433.46 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32793 47199 41% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33048 33048 0 100% /mnt/lustre - /unlink 6074 (time 1713077304.63 total 10.00 last 607.39) - /unlink 10000 (time 1713077310.87 total 16.24 last 628.95) - /unlink 15929 (time 1713077320.87 total 26.24 last 592.86) - /unlink 20000 (time 1713077328.05 total 33.42 last 567.44) - /unlink 26155 (time 1713077338.05 total 43.42 last 615.40) - /unlink 30000 (time 1713077342.93 total 48.30 last 787.90) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 52.05 seconds: 625.51 ops/second Waiting for MDT destroys to complete - open/close 4014 (time 1713077371.17 total 10.00 last 401.38) - open/close 8558 (time 1713077381.17 total 20.00 last 454.35) - open/close 10000 (time 1713077386.36 total 25.19 last 277.80) - open/close 12656 (time 1713077396.36 total 35.19 last 265.58) - open/close 15847 (time 1713077406.36 total 45.19 last 319.09) - open/close 18906 (time 1713077416.36 total 55.20 last 305.79) - open/close 20000 (time 1713077420.65 total 59.48 last 255.19) - open/close 22714 (time 1713077430.65 total 69.49 last 271.36) - open/close 26234 (time 1713077440.65 total 79.49 last 351.96) - open/close 29575 (time 1713077450.66 total 89.49 last 333.99) - open/close 32345 (time 1713077460.66 total 99.49 last 276.91) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 100.34 seconds: 324.50 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6262 (time 1713077472.73 total 10.00 last 626.19) - /unlink 10000 (time 1713077476.72 total 13.99 last 936.07) - /unlink 18022 (time 1713077486.72 total 23.99 last 802.11) - /unlink 20000 (time 1713077490.56 total 27.83 last 515.07) - /unlink 26065 (time 1713077500.57 total 37.84 last 606.36) - /unlink 30000 (time 1713077504.90 total 42.17 last 908.33) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 44.81 seconds: 726.55 ops/second Waiting for MDT destroys to complete - open/close 2736 (time 1713077539.55 total 10.00 last 273.55) - open/close 5409 (time 1713077549.55 total 20.00 last 267.28) - open/close 8168 (time 1713077559.55 total 30.00 last 275.88) - open/close 10000 (time 1713077566.07 total 36.52 last 281.14) - open/close 13075 (time 1713077576.07 total 46.52 last 307.42) - open/close 15917 (time 1713077586.07 total 56.52 last 284.14) - open/close 18536 (time 1713077596.07 total 66.53 last 261.87) - open/close 20000 (time 1713077601.41 total 71.87 last 274.14) - open/close 22930 (time 1713077611.41 total 81.87 last 292.98) - open/close 25954 (time 1713077621.42 total 91.87 last 302.37) - open/close 29037 (time 1713077631.42 total 101.87 last 308.29) - open/close 30000 (time 1713077633.50 total 103.96 last 461.17) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 110.11 seconds: 295.69 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6589 (time 1713077650.98 total 10.00 last 658.88) - /unlink 10000 (time 1713077655.22 total 14.23 last 805.81) - /unlink 17670 (time 1713077665.22 total 24.23 last 766.98) - /unlink 20000 (time 1713077668.73 total 27.74 last 663.78) - /unlink 25559 (time 1713077678.73 total 37.75 last 555.82) - /unlink 30000 (time 1713077686.51 total 45.53 last 570.40) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 50.50 seconds: 644.76 ops/second Waiting for MDT destroys to complete - open/close 3320 (time 1713077718.40 total 10.00 last 331.92) - open/close 6005 (time 1713077728.40 total 20.00 last 268.45) - open/close 9834 (time 1713077738.40 total 30.00 last 382.89) - open/close 12406 (time 1713077748.40 total 40.00 last 257.19) - open/close 15040 (time 1713077758.40 total 50.01 last 263.37) - open/close 17783 (time 1713077768.40 total 60.01 last 274.29) - open/close 20000 (time 1713077776.47 total 68.07 last 274.89) - open/close 22591 (time 1713077786.47 total 78.07 last 259.01) - open/close 25170 (time 1713077796.47 total 88.08 last 257.87) - open/close 28026 (time 1713077806.47 total 98.08 last 285.49) - open/close 30000 (time 1713077811.89 total 103.50 last 364.22) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 110.78 seconds: 293.93 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6215 (time 1713077830.10 total 10.00 last 621.37) - /unlink 10000 (time 1713077836.42 total 16.32 last 598.98) - /unlink 19766 (time 1713077846.42 total 26.32 last 976.47) - /unlink 25535 (time 1713077856.42 total 36.32 last 576.90) - /unlink 30000 (time 1713077865.70 total 45.60 last 481.46) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 49.62 seconds: 656.16 ops/second Waiting for MDT destroys to complete - open/close 2821 (time 1713077892.60 total 10.00 last 282.03) - open/close 5430 (time 1713077902.60 total 20.01 last 260.80) - open/close 9244 (time 1713077912.60 total 30.01 last 381.38) - open/close 10000 (time 1713077914.79 total 32.19 last 346.14) - open/close 12812 (time 1713077924.79 total 42.19 last 281.15) - open/close 15583 (time 1713077934.79 total 52.19 last 277.08) - open/close 18516 (time 1713077944.79 total 62.20 last 293.19) - open/close 20000 (time 1713077950.38 total 67.79 last 265.43) - open/close 22689 (time 1713077960.38 total 77.79 last 268.86) - open/close 25513 (time 1713077970.38 total 87.79 last 282.40) - open/close 28028 (time 1713077980.39 total 97.79 last 251.47) - open/close 30000 (time 1713077988.17 total 105.58 last 253.23) - open/close 32546 (time 1713077998.17 total 115.58 last 254.57) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 115.64 seconds: 281.56 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4498 (time 1713078009.52 total 10.00 last 449.77) - /unlink 8948 (time 1713078019.52 total 20.00 last 444.92) - /unlink 10000 (time 1713078021.86 total 22.34 last 449.82) - /unlink 14886 (time 1713078031.86 total 32.34 last 488.53) - /unlink 19861 (time 1713078041.86 total 42.34 last 497.50) - /unlink 25305 (time 1713078051.87 total 52.34 last 544.35) - /unlink 30000 (time 1713078060.97 total 61.45 last 515.62) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 65.94 seconds: 493.76 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg108-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg108-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg108-server stop mds service on oleg108-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg108-server unloading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 pdsh@oleg108-client: oleg108-client: ssh exited with exit code 2 PASS 106 (906s) error: set_param: param_path 'debug_raw_pointers': No such file or directory error: set_param: setting 'debug_raw_pointers'='0': No such file or directory oleg108-server: error: set_param: param_path 'debug_raw_pointers': No such file or directory oleg108-server: error: set_param: setting 'debug_raw_pointers'='0': No such file or directory Stopping clients: oleg108-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg108-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg108-client: oleg108-server: ssh exited with exit code 2 oleg108-server: oleg108-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg108-server' oleg108-server: oleg108-server.virtnet: executing load_modules_local oleg108-server: Loading modules from /home/green/git/lustre-release/lustre oleg108-server: detected 4 online CPUs by sysfs oleg108-server: Force libcfs to create 2 CPU partitions oleg108-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg108-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1014 sec ========== 03:01:59 (1713078119) === conf-sanity: start cleanup 03:01:59 (1713078119) === === conf-sanity: finish cleanup 03:01:59 (1713078119) ===