-----============= acceptance-small: conf-sanity ============----- Tue Jun 11 13:27:43 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg450-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg450-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg450-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg450-server oleg450-server: oleg450-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: detected 4 online CPUs by sysfs oleg450-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg450-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg450-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg450-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg450-server: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg450-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg450-server: oleg450-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg450-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg450-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:29:00 (1718126940) Stopping clients: oleg450-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg450-client.virtnet /mnt/lustre2 (opts:-f) oleg450-server: oleg450-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: detected 4 online CPUs by sysfs oleg450-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg450-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg450-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg450-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg450-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg450-server: oleg450-server.virtnet: executing set_default_debug -1 all pdsh@oleg450-client: oleg450-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg450-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg450-client.virtnet: -o user_xattr,flock oleg450-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3239 (time 1718126986.62 total 10.00 last 323.87) - open/close 6587 (time 1718126996.62 total 20.00 last 334.72) - open/close 9718 (time 1718127006.62 total 30.00 last 313.06) - open/close 13732 (time 1718127016.62 total 40.01 last 401.33) - open/close 17866 (time 1718127026.62 total 50.01 last 413.39) - open/close 20000 (time 1718127032.18 total 55.56 last 384.06) - open/close 23610 (time 1718127042.18 total 65.57 last 360.89) - open/close 27057 (time 1718127052.18 total 75.57 last 344.62) - open/close 30000 (time 1718127061.38 total 84.77 last 319.94) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 92.13 seconds: 353.43 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6911 (time 1718127079.78 total 10.00 last 690.98) - /unlink 10000 (time 1718127085.25 total 15.47 last 564.61) - /unlink 15126 (time 1718127095.25 total 25.47 last 512.52) - /unlink 20000 (time 1718127103.33 total 33.56 last 603.04) - /unlink 25234 (time 1718127113.33 total 43.56 last 523.37) - /unlink 30000 (time 1718127121.91 total 52.14 last 555.47) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 56.20 seconds: 579.34 ops/second Waiting for MDT destroys to complete - open/close 2648 (time 1718127144.77 total 10.00 last 264.80) - open/close 5151 (time 1718127154.77 total 20.00 last 250.20) - open/close 7723 (time 1718127164.77 total 30.00 last 257.18) - open/close 10000 (time 1718127172.70 total 37.93 last 287.29) - open/close 12882 (time 1718127182.70 total 47.93 last 288.15) - open/close 16199 (time 1718127192.70 total 57.94 last 331.59) - open/close 18717 (time 1718127202.71 total 67.94 last 251.72) - open/close 20000 (time 1718127207.47 total 72.71 last 269.16) - open/close 22472 (time 1718127217.47 total 82.71 last 247.18) - open/close 25058 (time 1718127227.47 total 92.71 last 258.59) - open/close 27528 (time 1718127237.48 total 102.71 last 246.94) - open/close 30000 (time 1718127247.44 total 112.67 last 248.22) - open/close 32376 (time 1718127257.44 total 122.67 last 237.56) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 123.46 seconds: 263.73 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4528 (time 1718127269.47 total 10.00 last 452.76) - /unlink 9039 (time 1718127279.47 total 20.00 last 451.04) - /unlink 13687 (time 1718127289.47 total 30.00 last 464.79) - /unlink 18225 (time 1718127299.47 total 40.00 last 453.75) - /unlink 20000 (time 1718127303.44 total 43.97 last 447.23) - /unlink 24563 (time 1718127313.44 total 53.97 last 456.28) - /unlink 29199 (time 1718127323.44 total 63.97 last 463.56) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 71.39 seconds: 456.06 ops/second Waiting for MDT destroys to complete - open/close 4418 (time 1718127352.38 total 10.00 last 441.64) - open/close 7256 (time 1718127362.38 total 20.00 last 283.80) - open/close 10000 (time 1718127369.66 total 27.28 last 377.31) - open/close 14474 (time 1718127379.66 total 37.28 last 447.40) - open/close 18376 (time 1718127389.66 total 47.28 last 390.12) - open/close 20000 (time 1718127393.88 total 51.50 last 384.79) - open/close 25060 (time 1718127403.88 total 61.50 last 505.94) - open/close 29821 (time 1718127413.88 total 71.50 last 476.06) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 76.82 seconds: 423.85 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8548 (time 1718127429.98 total 10.00 last 854.74) - /unlink 10000 (time 1718127432.07 total 12.08 last 697.61) - /unlink 18002 (time 1718127442.07 total 22.08 last 800.16) - /unlink 20000 (time 1718127444.22 total 24.24 last 927.15) - /unlink 28555 (time 1718127454.22 total 34.24 last 855.47) - /unlink 30000 (time 1718127456.29 total 36.31 last 697.82) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 40.81 seconds: 797.86 ops/second Waiting for MDT destroys to complete - open/close 4531 (time 1718127495.00 total 10.00 last 453.10) - open/close 8848 (time 1718127505.00 total 20.00 last 431.66) - open/close 10000 (time 1718127507.83 total 22.83 last 406.82) - open/close 14277 (time 1718127517.83 total 32.83 last 427.62) - open/close 19281 (time 1718127527.83 total 42.84 last 500.35) - open/close 24097 (time 1718127537.83 total 52.84 last 481.55) - open/close 29386 (time 1718127547.83 total 62.84 last 528.88) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 68.93 seconds: 472.35 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8920 (time 1718127564.71 total 10.00 last 891.91) - /unlink 18134 (time 1718127574.71 total 20.00 last 921.39) - /unlink 20000 (time 1718127577.28 total 22.57 last 726.20) - /unlink 28831 (time 1718127587.28 total 32.57 last 883.07) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 36.77 seconds: 885.59 ops/second Waiting for MDT destroys to complete - open/close 4353 (time 1718127616.23 total 10.00 last 435.25) - open/close 8929 (time 1718127626.23 total 20.00 last 457.54) - open/close 10000 (time 1718127629.34 total 23.11 last 344.39) - open/close 13428 (time 1718127639.34 total 33.11 last 342.77) - open/close 16227 (time 1718127649.34 total 43.11 last 279.87) - open/close 19537 (time 1718127659.34 total 53.12 last 330.94) - open/close 24187 (time 1718127669.34 total 63.12 last 464.98) - open/close 28715 (time 1718127679.35 total 73.12 last 452.76) - open/close 30000 (time 1718127681.86 total 75.63 last 511.44) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 80.56 seconds: 404.15 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6003 (time 1718127697.56 total 10.00 last 600.24) - /unlink 10000 (time 1718127703.13 total 15.57 last 717.86) - /unlink 15844 (time 1718127713.13 total 25.57 last 584.33) - /unlink 20000 (time 1718127721.81 total 34.24 last 479.16) - /unlink 25383 (time 1718127731.81 total 44.24 last 538.29) - /unlink 30000 (time 1718127736.96 total 49.40 last 896.11) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 52.22 seconds: 623.53 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg450-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg450-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg450-server stop mds service on oleg450-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg450-server unloading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-client: ssh exited with exit code 2 pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 PASS 106 (830s) Stopping clients: oleg450-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg450-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg450-client: oleg450-server: ssh exited with exit code 2 oleg450-server: oleg450-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg450-server' oleg450-server: oleg450-server.virtnet: executing load_modules_local oleg450-server: Loading modules from /home/green/git/lustre-release/lustre oleg450-server: detected 4 online CPUs by sysfs oleg450-server: Force libcfs to create 2 CPU partitions oleg450-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg450-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 928 sec =========== 13:43:11 (1718127791) === conf-sanity: start cleanup 13:43:12 (1718127792) === === conf-sanity: finish cleanup 13:43:12 (1718127792) ===