-----============= acceptance-small: conf-sanity ============----- Sat Nov 16 04:36:30 EST 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 excepting tests: 32newtarball 110 Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg160-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg160-server oleg160-server: oleg160-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: detected 4 online CPUs by sysfs oleg160-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg160-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg160-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg160-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg160-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg160-server: oleg160-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg160-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:38:00 (1731749880) Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f) oleg160-server: oleg160-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: detected 4 online CPUs by sysfs oleg160-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg160-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg160-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg160-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg160-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg160-server: oleg160-server.virtnet: executing set_default_debug -1 all pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg160-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg160-client.virtnet: -o user_xattr,flock 192.168.201.160@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1596 (time 1731749929.56 total 10.00 last 159.56) - open/close 3642 (time 1731749939.56 total 20.00 last 204.59) - open/close 5617 (time 1731749949.56 total 30.01 last 197.46) - open/close 7947 (time 1731749959.57 total 40.01 last 232.93) - open/close 10000 (time 1731749966.55 total 46.99 last 294.11) - open/close 13394 (time 1731749976.55 total 56.99 last 339.30) - open/close 15587 (time 1731749986.55 total 66.99 last 219.25) - open/close 17508 (time 1731749996.55 total 76.99 last 192.09) - open/close 19489 (time 1731750006.55 total 87.00 last 198.08) - open/close 20000 (time 1731750009.30 total 89.75 last 185.74) - open/close 21654 (time 1731750019.31 total 99.75 last 165.35) - open/close 23248 (time 1731750029.31 total 109.75 last 159.36) - open/close 24912 (time 1731750039.32 total 119.76 last 166.32) - open/close 26611 (time 1731750049.32 total 129.76 last 169.88) - open/close 28146 (time 1731750059.32 total 139.76 last 153.41) - open/close 29787 (time 1731750069.32 total 149.77 last 164.08) - open/close 31619 (time 1731750079.33 total 159.77 last 183.13) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 165.03 seconds: 197.30 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3209 (time 1731750095.98 total 10.00 last 320.87) - /unlink 6992 (time 1731750105.98 total 20.00 last 378.22) - /unlink 10000 (time 1731750113.82 total 27.85 last 383.45) - /unlink 13489 (time 1731750123.82 total 37.85 last 348.88) - /unlink 16902 (time 1731750133.83 total 47.85 last 341.22) - /unlink 19927 (time 1731750143.83 total 57.85 last 302.44) - /unlink 23580 (time 1731750153.83 total 67.85 last 365.29) - /unlink 27615 (time 1731750163.83 total 77.85 last 403.49) - /unlink 30000 (time 1731750169.92 total 83.95 last 391.37) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 92.00 seconds: 353.93 ops/second Waiting for MDT destroys to complete oleg160-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg160-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 - open/close 1914 (time 1731750200.00 total 10.00 last 191.38) - open/close 3882 (time 1731750210.01 total 20.01 last 196.66) - open/close 5553 (time 1731750220.01 total 30.01 last 167.02) - open/close 7330 (time 1731750230.02 total 40.02 last 177.61) - open/close 9343 (time 1731750240.02 total 50.02 last 201.27) - open/close 10000 (time 1731750242.68 total 52.68 last 247.32) - open/close 12460 (time 1731750252.68 total 62.68 last 245.95) - open/close 14980 (time 1731750262.68 total 72.68 last 251.93) - open/close 17267 (time 1731750272.69 total 82.69 last 228.54) - open/close 20000 (time 1731750281.89 total 91.88 last 297.16) - open/close 22945 (time 1731750291.89 total 101.89 last 294.48) - open/close 25918 (time 1731750301.89 total 111.89 last 297.22) - open/close 29012 (time 1731750311.89 total 121.89 last 309.40) - open/close 30000 (time 1731750314.90 total 124.90 last 327.99) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 134.15 seconds: 242.71 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4882 (time 1731750335.28 total 10.00 last 488.18) - /unlink 10000 (time 1731750345.04 total 19.76 last 524.50) - /unlink 15226 (time 1731750355.04 total 29.76 last 522.53) - /unlink 20000 (time 1731750364.95 total 39.67 last 481.71) - /unlink 24610 (time 1731750374.95 total 49.67 last 460.97) - /unlink 29598 (time 1731750384.95 total 59.67 last 498.76) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 66.14 seconds: 492.28 ops/second Waiting for MDT destroys to complete oleg160-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg160-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 - open/close 2637 (time 1731750409.80 total 10.00 last 263.60) - open/close 5328 (time 1731750419.81 total 20.01 last 269.02) - open/close 7797 (time 1731750429.81 total 30.01 last 246.82) - open/close 10000 (time 1731750437.82 total 38.03 last 274.82) - open/close 12665 (time 1731750447.83 total 48.03 last 266.43) - open/close 15566 (time 1731750457.83 total 58.03 last 290.09) - open/close 18500 (time 1731750467.83 total 68.03 last 293.35) - open/close 20000 (time 1731750474.23 total 74.43 last 234.26) - open/close 23007 (time 1731750484.24 total 84.44 last 300.64) - open/close 26073 (time 1731750494.24 total 94.44 last 306.60) - open/close 29379 (time 1731750504.24 total 104.44 last 330.59) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 112.64 seconds: 289.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6258 (time 1731750523.43 total 10.00 last 625.78) - /unlink 10000 (time 1731750529.32 total 15.89 last 635.60) - /unlink 16856 (time 1731750539.32 total 25.89 last 685.56) - /unlink 20000 (time 1731750543.52 total 30.09 last 747.55) - /unlink 26235 (time 1731750553.52 total 40.09 last 623.45) - /unlink 30000 (time 1731750560.15 total 46.72 last 568.15) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 50.50 seconds: 644.75 ops/second Waiting for MDT destroys to complete oleg160-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg160-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 - open/close 3144 (time 1731750599.64 total 10.00 last 314.33) - open/close 6937 (time 1731750609.65 total 20.00 last 379.22) - open/close 10000 (time 1731750616.24 total 26.60 last 464.65) - open/close 13898 (time 1731750626.24 total 36.60 last 389.73) - open/close 16914 (time 1731750636.24 total 46.60 last 301.57) - open/close 20000 (time 1731750646.08 total 56.43 last 313.75) - open/close 24309 (time 1731750656.08 total 66.44 last 430.89) - open/close 28937 (time 1731750666.08 total 76.44 last 462.64) - open/close 30000 (time 1731750668.19 total 78.54 last 504.69) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 84.54 seconds: 385.16 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7901 (time 1731750685.10 total 10.01 last 789.58) - /unlink 10000 (time 1731750688.12 total 13.02 last 696.04) - /unlink 18610 (time 1731750698.12 total 23.02 last 860.92) - /unlink 27934 (time 1731750708.12 total 33.02 last 932.35) - /unlink 30000 (time 1731750710.48 total 35.38 last 874.99) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 38.51 seconds: 845.51 ops/second Waiting for MDT destroys to complete oleg160-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg160-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 - open/close 4297 (time 1731750750.64 total 10.00 last 429.59) - open/close 8232 (time 1731750760.64 total 20.00 last 393.41) - open/close 10000 (time 1731750765.22 total 24.58 last 386.16) - open/close 14063 (time 1731750775.22 total 34.58 last 406.26) - open/close 18310 (time 1731750785.22 total 44.59 last 424.60) - open/close 20000 (time 1731750789.30 total 48.66 last 414.62) - open/close 24590 (time 1731750799.30 total 58.66 last 458.91) - open/close 29158 (time 1731750809.30 total 68.67 last 456.71) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 75.22 seconds: 432.87 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1731750826.63 total 9.93 last 1007.30) - /unlink 19951 (time 1731750836.63 total 19.93 last 995.03) - /unlink 30000 (time 1731750845.93 total 29.22 last 1081.44) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 31.51 seconds: 1033.21 ops/second Waiting for MDT destroys to complete oleg160-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg160-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg160-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg160-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg160-server stop mds service on oleg160-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg160-server unloading modules via unload_modules_local on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-client: ssh exited with exit code 2 pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 PASS 106 (1031s) Stopping clients: oleg160-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg160-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg160-client: oleg160-server: ssh exited with exit code 2 oleg160-server: oleg160-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg160-server' oleg160-server: oleg160-server.virtnet: executing load_modules_local oleg160-server: Loading modules from /home/green/git/lustre-release/lustre oleg160-server: detected 4 online CPUs by sysfs oleg160-server: Force libcfs to create 2 CPU partitions oleg160-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg160-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1140 sec ========== 04:55:30 (1731750930) === conf-sanity: start cleanup 04:55:30 (1731750930) === === conf-sanity: finish cleanup 04:55:31 (1731750931) ===