== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:06:04 (1717556764) Stopping clients: oleg412-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg412-client.virtnet /mnt/lustre2 (opts:-f) oleg412-server: oleg412-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing load_modules_local oleg412-server: Loading modules from /home/green/git/lustre-release/lustre oleg412-server: detected 4 online CPUs by sysfs oleg412-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg412-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg412-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg412-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg412-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg412-server: oleg412-server.virtnet: executing set_default_debug -1 all pdsh@oleg412-client: oleg412-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg412-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg412-client.virtnet: -o user_xattr,flock oleg412-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5715 (time 1717556802.65 total 10.00 last 571.50) - open/close 10000 (time 1717556810.48 total 17.84 last 546.77) - open/close 15790 (time 1717556820.48 total 27.84 last 578.92) - open/close 20000 (time 1717556827.80 total 35.16 last 575.35) - open/close 25565 (time 1717556837.80 total 45.16 last 556.43) - open/close 30000 (time 1717556845.74 total 53.10 last 558.52) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 57.97 seconds: 561.67 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1717556861.34 total 9.85 last 1015.39) - /unlink 18413 (time 1717556871.34 total 19.85 last 841.23) - /unlink 27478 (time 1717556881.34 total 29.85 last 906.44) - /unlink 30000 (time 1717556883.85 total 32.36 last 1003.06) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 34.89 seconds: 933.21 ops/second Waiting for MDT destroys to complete - open/close 5739 (time 1717556921.59 total 10.00 last 573.86) - open/close 10000 (time 1717556929.01 total 17.42 last 574.11) - open/close 15793 (time 1717556939.01 total 27.42 last 579.26) - open/close 20000 (time 1717556946.39 total 34.80 last 570.41) - open/close 25643 (time 1717556956.39 total 44.80 last 564.27) - open/close 29161 (time 1717556966.39 total 54.80 last 351.75) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 60.58 seconds: 537.51 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1717556982.64 total 9.63 last 1038.69) - /unlink 19932 (time 1717556992.64 total 19.63 last 993.16) - /unlink 30000 (time 1717557002.14 total 29.13 last 1059.45) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 31.51 seconds: 1033.29 ops/second Waiting for MDT destroys to complete - open/close 5747 (time 1717557046.29 total 10.00 last 574.66) - open/close 10000 (time 1717557053.60 total 17.31 last 581.53) - open/close 15813 (time 1717557063.60 total 27.32 last 581.22) - open/close 20000 (time 1717557071.31 total 35.02 last 543.35) - open/close 24703 (time 1717557081.31 total 45.02 last 470.28) - open/close 30000 (time 1717557090.46 total 54.17 last 578.92) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 58.93 seconds: 552.49 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8705 (time 1717557106.11 total 10.00 last 870.43) - /unlink 18810 (time 1717557116.11 total 20.00 last 1010.47) - /unlink 28379 (time 1717557126.11 total 30.00 last 956.88) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 35.32 seconds: 921.83 ops/second Waiting for MDT destroys to complete - open/close 5789 (time 1717557176.72 total 10.00 last 578.89) - open/close 10000 (time 1717557184.95 total 18.23 last 511.68) - open/close 15307 (time 1717557194.96 total 28.23 last 530.62) - open/close 20000 (time 1717557202.84 total 36.12 last 595.07) - open/close 25928 (time 1717557212.84 total 46.12 last 592.71) - open/close 30000 (time 1717557220.21 total 53.49 last 552.67) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 58.55 seconds: 556.08 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7913 (time 1717557236.14 total 10.00 last 791.28) - /unlink 10000 (time 1717557238.38 total 12.24 last 930.30) - /unlink 19630 (time 1717557248.38 total 22.24 last 962.94) - /unlink 29455 (time 1717557258.38 total 32.24 last 982.47) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 36.05 seconds: 903.13 ops/second Waiting for MDT destroys to complete - open/close 4000 (time 1717557311.55 total 10.00 last 399.96) - open/close 8099 (time 1717557321.55 total 20.00 last 409.79) - open/close 10000 (time 1717557325.64 total 24.09 last 465.19) - open/close 13489 (time 1717557335.64 total 34.09 last 348.88) - open/close 16600 (time 1717557345.64 total 44.09 last 311.10) - open/close 19293 (time 1717557355.64 total 54.09 last 269.22) - open/close 20000 (time 1717557358.17 total 56.62 last 279.63) - open/close 23707 (time 1717557368.17 total 66.62 last 370.60) - open/close 26668 (time 1717557378.18 total 76.63 last 296.06) - open/close 29538 (time 1717557388.18 total 86.63 last 286.91) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 93.82 seconds: 347.05 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8735 (time 1717557406.52 total 10.00 last 873.43) - /unlink 15926 (time 1717557416.52 total 20.00 last 719.00) - /unlink 20000 (time 1717557423.24 total 26.72 last 606.42) - /unlink 24987 (time 1717557433.24 total 36.72 last 498.68) - /unlink 30000 (time 1717557439.80 total 43.28 last 764.26) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 45.75 seconds: 711.75 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg412-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg412-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg412-server stop mds service on oleg412-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg412-server unloading modules on: 'oleg412-server' oleg412-server: oleg412-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-client: ssh exited with exit code 2 pdsh@oleg412-client: oleg412-server: ssh exited with exit code 2