== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:27:33 (1736785653) Stopping clients: oleg141-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg141-client.virtnet /mnt/lustre2 (opts:-f) oleg141-server: oleg141-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: MODOPTS_LIBCFS= oleg141-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg141-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg141-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2733 (time 1736785703.80 total 10.00 last 273.22) - open/close 5632 (time 1736785713.80 total 20.00 last 289.86) - open/close 8870 (time 1736785723.80 total 30.01 last 323.77) - open/close 10000 (time 1736785728.09 total 34.29 last 263.66) - open/close 12734 (time 1736785738.09 total 44.29 last 273.30) - open/close 15443 (time 1736785748.09 total 54.30 last 270.87) - open/close 18208 (time 1736785758.10 total 64.30 last 276.46) - open/close 20000 (time 1736785764.52 total 70.72 last 278.92) - open/close 22620 (time 1736785774.52 total 80.72 last 261.95) - open/close 25234 (time 1736785784.52 total 90.72 last 261.38) - open/close 27846 (time 1736785794.53 total 100.73 last 261.16) - open/close 30000 (time 1736785803.21 total 109.41 last 248.08) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 118.88 seconds: 273.90 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6067 (time 1736785824.04 total 10.00 last 606.69) - /unlink 10000 (time 1736785829.85 total 15.80 last 677.70) - /unlink 16139 (time 1736785839.85 total 25.80 last 613.87) - /unlink 20000 (time 1736785846.32 total 32.28 last 596.20) - /unlink 25654 (time 1736785856.32 total 42.28 last 565.32) - /unlink 30000 (time 1736785863.91 total 49.87 last 572.48) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 53.49 seconds: 608.71 ops/second Waiting for MDT destroys to complete oleg141-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 oleg141-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 - open/close 2978 (time 1736785905.56 total 10.00 last 297.79) - open/close 6931 (time 1736785915.56 total 20.00 last 395.25) - open/close 10000 (time 1736785924.90 total 29.33 last 328.83) - open/close 13070 (time 1736785934.90 total 39.34 last 306.95) - open/close 15822 (time 1736785944.90 total 49.34 last 275.15) - open/close 18672 (time 1736785954.90 total 59.34 last 284.93) - open/close 20000 (time 1736785958.02 total 62.46 last 425.73) - open/close 22906 (time 1736785968.02 total 72.46 last 290.57) - open/close 25525 (time 1736785978.03 total 82.46 last 261.82) - open/close 28342 (time 1736785988.03 total 92.46 last 281.67) - open/close 30000 (time 1736785994.35 total 98.78 last 262.39) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 108.14 seconds: 301.08 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4759 (time 1736786014.54 total 10.00 last 475.86) - /unlink 9581 (time 1736786024.54 total 20.00 last 482.20) - /unlink 14416 (time 1736786034.54 total 30.00 last 483.47) - /unlink 20000 (time 1736786044.02 total 39.49 last 588.76) - /unlink 25238 (time 1736786054.02 total 49.49 last 523.74) - /unlink 30000 (time 1736786063.91 total 59.37 last 481.60) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 64.38 seconds: 505.79 ops/second Waiting for MDT destroys to complete oleg141-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg141-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 - open/close 2998 (time 1736786086.71 total 10.00 last 299.79) - open/close 5618 (time 1736786096.71 total 20.00 last 261.94) - open/close 8260 (time 1736786106.72 total 30.01 last 264.09) - open/close 10000 (time 1736786113.00 total 36.28 last 277.16) - open/close 13383 (time 1736786123.00 total 46.29 last 338.24) - open/close 15968 (time 1736786133.00 total 56.29 last 258.47) - open/close 18616 (time 1736786143.00 total 66.29 last 264.73) - open/close 20000 (time 1736786147.72 total 71.01 last 293.07) - open/close 22555 (time 1736786157.73 total 81.01 last 255.48) - open/close 25132 (time 1736786167.73 total 91.01 last 257.69) - open/close 28041 (time 1736786177.73 total 101.02 last 290.86) - open/close 30000 (time 1736786184.05 total 107.34 last 309.62) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 116.92 seconds: 278.47 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7104 (time 1736786204.77 total 10.00 last 710.29) - /unlink 10000 (time 1736786207.92 total 13.15 last 920.12) - /unlink 18189 (time 1736786217.92 total 23.15 last 818.80) - /unlink 20000 (time 1736786220.14 total 25.37 last 815.82) - /unlink 28090 (time 1736786230.14 total 35.37 last 808.94) - /unlink 30000 (time 1736786232.35 total 37.58 last 865.03) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 40.45 seconds: 805.03 ops/second Waiting for MDT destroys to complete oleg141-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg141-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 - open/close 2692 (time 1736786275.49 total 10.00 last 269.20) - open/close 5290 (time 1736786285.49 total 20.00 last 259.76) - open/close 8342 (time 1736786295.50 total 30.00 last 305.15) - open/close 10000 (time 1736786300.71 total 35.22 last 317.95) - open/close 13072 (time 1736786310.71 total 45.22 last 307.19) - open/close 16046 (time 1736786320.71 total 55.22 last 297.32) - open/close 18735 (time 1736786330.72 total 65.22 last 268.85) - open/close 20000 (time 1736786334.32 total 68.83 last 350.91) - open/close 23941 (time 1736786344.32 total 78.83 last 394.06) - open/close 27183 (time 1736786354.32 total 88.83 last 324.16) - open/close 30000 (time 1736786361.63 total 96.14 last 385.49) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 105.19 seconds: 309.52 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4814 (time 1736786382.05 total 10.00 last 481.39) - /unlink 10000 (time 1736786391.19 total 19.13 last 567.77) - /unlink 15480 (time 1736786401.19 total 29.14 last 547.93) - /unlink 20000 (time 1736786408.99 total 36.94 last 579.27) - /unlink 25245 (time 1736786418.99 total 46.94 last 524.49) - /unlink 30000 (time 1736786428.36 total 56.30 last 507.73) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 61.32 seconds: 531.00 ops/second Waiting for MDT destroys to complete oleg141-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg141-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 - open/close 2726 (time 1736786455.30 total 10.00 last 272.51) - open/close 5544 (time 1736786465.31 total 20.00 last 281.77) - open/close 8593 (time 1736786475.31 total 30.01 last 304.79) - open/close 10000 (time 1736786479.85 total 34.55 last 309.68) - open/close 13023 (time 1736786489.85 total 44.55 last 302.26) - open/close 16491 (time 1736786499.86 total 54.55 last 346.75) - open/close 19523 (time 1736786509.86 total 64.56 last 303.14) - open/close 22226 (time 1736786519.86 total 74.56 last 270.29) - open/close 25538 (time 1736786529.86 total 84.56 last 331.19) - open/close 28490 (time 1736786539.86 total 94.56 last 295.19) - open/close 30000 (time 1736786544.97 total 99.67 last 295.39) open(/mnt/lustre/d106.conf-sanity/f-32560) error: No space left on device total: 32560 open/close in 105.85 seconds: 307.62 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7499 (time 1736786562.03 total 10.00 last 749.84) - /unlink 10000 (time 1736786566.55 total 14.52 last 553.70) - /unlink 14754 (time 1736786576.55 total 24.52 last 475.40) - /unlink 19550 (time 1736786586.55 total 34.52 last 479.51) - /unlink 24179 (time 1736786596.55 total 44.52 last 462.80) - /unlink 28896 (time 1736786606.55 total 54.52 last 471.68) - /unlink 30000 (time 1736786608.97 total 56.94 last 457.04) unlink(/mnt/lustre/d106.conf-sanity/f-32560) error: No such file or directory total: 32560 /unlink in 62.25 seconds: 523.08 ops/second Waiting for MDT destroys to complete oleg141-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg141-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg141-server unloading modules via unload_modules_local on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2