-----============= acceptance-small: conf-sanity ============----- Fri Dec 13 13:42:00 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg434-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 110 Stopping clients: oleg434-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg434-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg434-server oleg434-server: oleg434-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: MODOPTS_LIBCFS= oleg434-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg434-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg434-server: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg434-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg434-server: oleg434-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg434-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:43:27 (1734115407) Stopping clients: oleg434-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:-f) oleg434-server: oleg434-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: MODOPTS_LIBCFS= oleg434-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg434-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg434-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4976 (time 1734115447.24 total 10.00 last 497.54) - open/close 10000 (time 1734115456.93 total 19.69 last 518.59) - open/close 15534 (time 1734115466.93 total 29.69 last 553.34) - open/close 20000 (time 1734115474.92 total 37.67 last 559.48) - open/close 25131 (time 1734115484.92 total 47.67 last 513.10) - open/close 30000 (time 1734115493.69 total 56.45 last 554.90) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 61.22 seconds: 531.86 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9701 (time 1734115509.35 total 10.00 last 970.10) - /unlink 19897 (time 1734115519.35 total 20.00 last 1019.58) - /unlink 29142 (time 1734115529.35 total 30.00 last 924.36) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 33.50 seconds: 971.91 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 - open/close 4634 (time 1734115591.13 total 10.00 last 463.34) - open/close 9288 (time 1734115601.13 total 20.00 last 465.35) - open/close 13454 (time 1734115611.13 total 30.00 last 416.56) - open/close 17599 (time 1734115621.13 total 40.00 last 414.48) - open/close 20000 (time 1734115627.19 total 46.07 last 396.11) - open/close 24276 (time 1734115637.19 total 56.07 last 427.59) - open/close 28578 (time 1734115647.19 total 66.07 last 430.14) - open/close 30000 (time 1734115651.01 total 69.89 last 372.06) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 76.96 seconds: 423.10 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7727 (time 1734115669.04 total 10.00 last 772.64) - /unlink 10000 (time 1734115671.99 total 12.95 last 769.80) - /unlink 18027 (time 1734115681.99 total 22.95 last 802.67) - /unlink 20000 (time 1734115684.55 total 25.51 last 770.67) - /unlink 27228 (time 1734115694.56 total 35.51 last 722.78) - /unlink 30000 (time 1734115697.86 total 38.82 last 838.72) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 42.09 seconds: 773.55 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 - open/close 4346 (time 1734115740.73 total 10.00 last 434.55) - open/close 9005 (time 1734115750.73 total 20.00 last 465.86) - open/close 10000 (time 1734115753.07 total 22.34 last 425.93) - open/close 15196 (time 1734115763.07 total 32.34 last 519.58) - open/close 20000 (time 1734115772.17 total 41.44 last 527.76) - open/close 24417 (time 1734115782.17 total 51.44 last 441.69) - open/close 29743 (time 1734115792.17 total 61.44 last 532.52) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 66.61 seconds: 488.83 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734115808.14 total 9.94 last 1006.46) - /unlink 19470 (time 1734115818.14 total 19.94 last 946.92) - /unlink 30000 (time 1734115828.12 total 29.92 last 1054.87) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.30 seconds: 1008.16 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 - open/close 5962 (time 1734115871.26 total 10.00 last 596.15) - open/close 10000 (time 1734115878.30 total 17.04 last 574.02) - open/close 15859 (time 1734115888.30 total 27.04 last 585.82) - open/close 20000 (time 1734115894.99 total 33.73 last 619.00) - open/close 25705 (time 1734115904.99 total 43.73 last 570.46) - open/close 30000 (time 1734115911.78 total 50.52 last 632.24) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 54.87 seconds: 593.36 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734115926.55 total 9.59 last 1042.93) - /unlink 19407 (time 1734115936.55 total 19.59 last 940.66) - /unlink 29428 (time 1734115946.55 total 29.59 last 1002.09) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 32.47 seconds: 1002.84 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 - open/close 5976 (time 1734115990.73 total 10.00 last 597.58) - open/close 10000 (time 1734115996.88 total 16.15 last 654.29) - open/close 16337 (time 1734116006.88 total 26.15 last 633.66) - open/close 20000 (time 1734116012.95 total 32.22 last 603.30) - open/close 26199 (time 1734116022.95 total 42.22 last 619.90) - open/close 30000 (time 1734116028.79 total 48.06 last 651.72) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 52.16 seconds: 624.23 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734116043.34 total 9.63 last 1037.92) - /unlink 20000 (time 1734116052.47 total 18.76 last 1095.51) - /unlink 30000 (time 1734116061.54 total 27.84 last 1102.18) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 30.10 seconds: 1081.75 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server unloading modules via unload_modules_local on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 PASS 106 (708s) Stopping clients: oleg434-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 oleg434-server: oleg434-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: MODOPTS_LIBCFS= oleg434-server: Force libcfs to create 2 CPU partitions oleg434-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg434-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 812 sec =========== 13:55:33 (1734116133) === conf-sanity: start cleanup 13:55:34 (1734116134) === === conf-sanity: finish cleanup 13:55:35 (1734116135) ===