-----============= acceptance-small: conf-sanity ============----- Wed Jun 19 22:48:51 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg401-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg401-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg401-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg401-server oleg401-server: oleg401-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing load_modules_local oleg401-server: Loading modules from /home/green/git/lustre-release/lustre oleg401-server: detected 4 online CPUs by sysfs oleg401-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg401-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg401-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg401-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg401-server: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg401-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg401-server: oleg401-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg401-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg401-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:50:20 (1718851820) Stopping clients: oleg401-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg401-client.virtnet /mnt/lustre2 (opts:-f) oleg401-server: oleg401-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing load_modules_local oleg401-server: Loading modules from /home/green/git/lustre-release/lustre oleg401-server: detected 4 online CPUs by sysfs oleg401-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg401-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg401-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg401-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg401-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg401-server: oleg401-server.virtnet: executing set_default_debug -1 all pdsh@oleg401-client: oleg401-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg401-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg401-client.virtnet: -o user_xattr,flock oleg401-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4991 (time 1718851860.91 total 10.00 last 499.01) - open/close 10000 (time 1718851870.10 total 19.19 last 544.94) - open/close 15051 (time 1718851880.10 total 29.19 last 505.07) - open/close 20000 (time 1718851889.69 total 38.78 last 516.10) - open/close 25224 (time 1718851899.69 total 48.78 last 522.32) - open/close 30000 (time 1718851908.59 total 57.69 last 536.45) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 62.41 seconds: 521.70 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9522 (time 1718851924.20 total 10.00 last 952.14) - /unlink 18866 (time 1718851934.20 total 20.00 last 934.30) - /unlink 28534 (time 1718851944.20 total 30.00 last 966.80) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.29 seconds: 949.67 ops/second Waiting for MDT destroys to complete - open/close 4034 (time 1718851985.38 total 10.00 last 403.36) - open/close 9224 (time 1718851995.38 total 20.00 last 518.99) - open/close 14600 (time 1718852005.38 total 30.00 last 537.59) - open/close 20000 (time 1718852015.22 total 39.84 last 549.09) - open/close 25460 (time 1718852025.22 total 49.84 last 546.00) - open/close 30000 (time 1718852033.47 total 58.09 last 550.24) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 62.86 seconds: 518.02 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9663 (time 1718852049.12 total 10.00 last 966.26) - /unlink 19570 (time 1718852059.12 total 20.00 last 990.60) - /unlink 29376 (time 1718852069.12 total 30.00 last 980.54) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.20 seconds: 980.73 ops/second Waiting for MDT destroys to complete - open/close 5482 (time 1718852109.26 total 10.00 last 548.19) - open/close 10000 (time 1718852117.74 total 18.48 last 532.95) - open/close 15090 (time 1718852127.74 total 28.48 last 509.00) - open/close 20000 (time 1718852137.71 total 38.45 last 492.37) - open/close 25372 (time 1718852147.72 total 48.45 last 537.12) - open/close 30000 (time 1718852156.03 total 56.77 last 556.64) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.99 seconds: 525.20 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9445 (time 1718852172.13 total 10.00 last 944.44) - /unlink 18088 (time 1718852182.14 total 20.00 last 864.21) - /unlink 20000 (time 1718852184.14 total 22.00 last 954.89) - /unlink 30000 (time 1718852194.13 total 32.00 last 1000.52) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.68 seconds: 938.91 ops/second Waiting for MDT destroys to complete - open/close 4349 (time 1718852237.70 total 10.00 last 434.87) - open/close 8558 (time 1718852247.70 total 20.00 last 420.89) - open/close 10000 (time 1718852251.97 total 24.27 last 337.52) - open/close 12948 (time 1718852261.98 total 34.28 last 294.72) - open/close 15919 (time 1718852271.98 total 44.28 last 297.02) - open/close 20000 (time 1718852281.36 total 53.66 last 435.08) - open/close 24685 (time 1718852291.36 total 63.66 last 468.39) - open/close 28360 (time 1718852301.36 total 73.66 last 367.46) - open/close 30000 (time 1718852305.78 total 78.08 last 371.29) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 85.89 seconds: 379.08 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7449 (time 1718852324.88 total 10.00 last 744.90) - /unlink 10000 (time 1718852327.82 total 12.94 last 867.02) - /unlink 16747 (time 1718852337.82 total 22.94 last 674.66) - /unlink 20000 (time 1718852341.11 total 26.23 last 989.70) - /unlink 29479 (time 1718852351.11 total 36.23 last 947.86) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 39.34 seconds: 827.59 ops/second Waiting for MDT destroys to complete - open/close 5210 (time 1718852392.17 total 10.00 last 520.91) - open/close 10000 (time 1718852401.11 total 18.94 last 536.03) - open/close 15180 (time 1718852411.11 total 28.94 last 517.89) - open/close 20000 (time 1718852420.86 total 38.69 last 494.16) - open/close 25030 (time 1718852430.87 total 48.69 last 502.97) - open/close 29417 (time 1718852440.87 total 58.70 last 438.60) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 67.30 seconds: 483.84 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7932 (time 1718852460.53 total 10.00 last 793.12) - /unlink 10000 (time 1718852463.72 total 13.20 last 647.18) - /unlink 18413 (time 1718852473.72 total 23.20 last 841.30) - /unlink 25999 (time 1718852483.72 total 33.20 last 758.60) - /unlink 30000 (time 1718852490.26 total 39.73 last 612.09) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 43.53 seconds: 747.97 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg401-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg401-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg401-server stop mds service on oleg401-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg401-server unloading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-client: ssh exited with exit code 2 pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 PASS 106 (719s) Stopping clients: oleg401-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg401-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg401-client: oleg401-server: ssh exited with exit code 2 oleg401-server: oleg401-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg401-server' oleg401-server: oleg401-server.virtnet: executing load_modules_local oleg401-server: Loading modules from /home/green/git/lustre-release/lustre oleg401-server: detected 4 online CPUs by sysfs oleg401-server: Force libcfs to create 2 CPU partitions oleg401-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg401-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg401-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 831 sec =========== 23:02:43 (1718852563) === conf-sanity: start cleanup 23:02:44 (1718852564) === === conf-sanity: finish cleanup 23:02:44 (1718852564) ===