== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:51:48 (1726303908) Stopping clients: oleg226-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg226-client.virtnet /mnt/lustre2 (opts:-f) oleg226-server: oleg226-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing load_modules_local oleg226-server: Loading modules from /home/green/git/lustre-release/lustre oleg226-server: detected 4 online CPUs by sysfs oleg226-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg226-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg226-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg226-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg226-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg226-server: oleg226-server.virtnet: executing set_default_debug -1 all pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg226-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg226-client.virtnet: -o user_xattr,flock 192.168.202.126@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2050 (time 1726303956.32 total 10.01 last 204.88) - open/close 3788 (time 1726303966.32 total 20.01 last 173.74) - open/close 5618 (time 1726303976.32 total 30.01 last 182.96) - open/close 7735 (time 1726303986.32 total 40.01 last 211.64) - open/close 10000 (time 1726303994.64 total 48.33 last 272.34) - open/close 12503 (time 1726304004.64 total 58.33 last 250.25) - open/close 15008 (time 1726304014.64 total 68.33 last 250.48) - open/close 17488 (time 1726304024.65 total 78.34 last 247.96) - open/close 19988 (time 1726304034.65 total 88.34 last 249.97) - open/close 22170 (time 1726304044.65 total 98.34 last 218.12) - open/close 24329 (time 1726304054.65 total 108.34 last 215.88) - open/close 26713 (time 1726304064.65 total 118.34 last 238.38) - open/close 29040 (time 1726304074.65 total 128.34 last 232.65) - open/close 30000 (time 1726304079.92 total 133.61 last 182.42) - open/close 32158 (time 1726304089.92 total 143.61 last 215.77) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 145.54 seconds: 223.72 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4243 (time 1726304103.27 total 10.00 last 424.25) - /unlink 8364 (time 1726304113.27 total 20.00 last 412.04) - /unlink 10000 (time 1726304117.15 total 23.89 last 421.24) - /unlink 13889 (time 1726304127.15 total 33.89 last 388.84) - /unlink 17603 (time 1726304137.15 total 43.89 last 371.36) - /unlink 20000 (time 1726304143.82 total 50.56 last 359.55) - /unlink 23856 (time 1726304153.82 total 60.56 last 385.48) - /unlink 27944 (time 1726304163.85 total 70.59 last 407.62) - /unlink 30000 (time 1726304168.65 total 75.39 last 428.39) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 80.65 seconds: 403.74 ops/second Waiting for MDT destroys to complete oleg226-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg226-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 - open/close 2681 (time 1726304191.17 total 10.00 last 268.04) - open/close 5179 (time 1726304201.17 total 20.00 last 249.78) - open/close 7490 (time 1726304211.17 total 30.01 last 231.04) - open/close 9875 (time 1726304221.17 total 40.01 last 238.49) - open/close 12137 (time 1726304231.17 total 50.01 last 226.17) - open/close 14120 (time 1726304241.17 total 60.01 last 198.26) - open/close 16166 (time 1726304251.18 total 70.01 last 204.51) - open/close 18578 (time 1726304261.18 total 80.01 last 241.18) - open/close 20000 (time 1726304267.50 total 86.33 last 225.11) - open/close 22118 (time 1726304277.50 total 96.33 last 211.75) - open/close 24509 (time 1726304287.50 total 106.34 last 238.98) - open/close 26648 (time 1726304297.50 total 116.34 last 213.89) - open/close 28953 (time 1726304307.51 total 126.34 last 230.43) - open/close 30000 (time 1726304311.46 total 130.30 last 264.68) - open/close 32438 (time 1726304321.46 total 140.30 last 243.80) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 140.84 seconds: 231.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5493 (time 1726304333.16 total 10.00 last 549.29) - /unlink 10000 (time 1726304339.93 total 16.76 last 666.46) - /unlink 15472 (time 1726304349.93 total 26.76 last 547.15) - /unlink 20000 (time 1726304356.74 total 33.58 last 664.65) - /unlink 25638 (time 1726304366.74 total 43.58 last 563.72) - /unlink 30000 (time 1726304374.45 total 51.28 last 566.04) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 56.59 seconds: 575.32 ops/second Waiting for MDT destroys to complete oleg226-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg226-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 - open/close 3093 (time 1726304399.37 total 10.00 last 309.29) - open/close 6289 (time 1726304409.37 total 20.00 last 319.59) - open/close 9791 (time 1726304419.37 total 30.00 last 350.09) - open/close 13553 (time 1726304429.37 total 40.01 last 376.13) - open/close 17236 (time 1726304439.39 total 50.03 last 367.45) - open/close 20000 (time 1726304446.05 total 56.68 last 415.46) - open/close 23519 (time 1726304456.05 total 66.68 last 351.79) - open/close 27039 (time 1726304466.05 total 76.69 last 351.94) - open/close 30000 (time 1726304474.41 total 85.05 last 354.11) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 93.48 seconds: 348.31 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6753 (time 1726304493.88 total 10.00 last 675.19) - /unlink 10000 (time 1726304499.03 total 15.15 last 631.01) - /unlink 16614 (time 1726304509.03 total 25.15 last 661.38) - /unlink 20000 (time 1726304513.89 total 30.01 last 695.98) - /unlink 25796 (time 1726304523.89 total 40.01 last 579.58) - /unlink 30000 (time 1726304530.67 total 46.79 last 620.48) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 50.91 seconds: 639.61 ops/second Waiting for MDT destroys to complete oleg226-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg226-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 - open/close 3985 (time 1726304573.86 total 10.00 last 398.45) - open/close 8263 (time 1726304583.86 total 20.00 last 427.70) - open/close 10000 (time 1726304587.92 total 24.06 last 428.37) - open/close 13857 (time 1726304597.92 total 34.06 last 385.64) - open/close 18200 (time 1726304607.92 total 44.06 last 434.13) - open/close 20000 (time 1726304612.60 total 48.74 last 384.85) - open/close 24964 (time 1726304622.60 total 58.74 last 496.40) - open/close 29004 (time 1726304632.60 total 68.74 last 403.95) - open/close 30000 (time 1726304635.14 total 71.28 last 392.95) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 76.62 seconds: 424.97 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8539 (time 1726304651.36 total 10.00 last 853.81) - /unlink 16794 (time 1726304661.36 total 20.00 last 825.47) - /unlink 20000 (time 1726304665.69 total 24.33 last 740.83) - /unlink 27454 (time 1726304675.69 total 34.33 last 745.35) - /unlink 30000 (time 1726304679.75 total 38.39 last 627.34) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 42.44 seconds: 767.14 ops/second Waiting for MDT destroys to complete oleg226-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg226-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 - open/close 4828 (time 1726304706.11 total 10.00 last 482.72) - open/close 9006 (time 1726304716.11 total 20.00 last 417.76) - open/close 10000 (time 1726304719.06 total 22.95 last 336.72) - open/close 14064 (time 1726304729.06 total 32.96 last 406.34) - open/close 18567 (time 1726304739.06 total 42.96 last 450.24) - open/close 20000 (time 1726304742.37 total 46.27 last 432.99) - open/close 23259 (time 1726304752.37 total 56.27 last 325.86) - open/close 26369 (time 1726304762.38 total 66.27 last 310.89) - open/close 30000 (time 1726304771.64 total 75.54 last 391.82) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 82.06 seconds: 396.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6372 (time 1726304789.15 total 10.00 last 637.12) - /unlink 10000 (time 1726304793.65 total 14.50 last 806.09) - /unlink 19685 (time 1726304803.65 total 24.50 last 968.43) - /unlink 27899 (time 1726304813.65 total 34.50 last 821.33) - /unlink 30000 (time 1726304816.04 total 36.89 last 880.31) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 39.39 seconds: 826.58 ops/second Waiting for MDT destroys to complete oleg226-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg226-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg226-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg226-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg226-server stop mds service on oleg226-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg226-server unloading modules on: 'oleg226-server' oleg226-server: oleg226-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-server: ssh exited with exit code 2 pdsh@oleg226-client: oleg226-client: ssh exited with exit code 2