== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:50:01 (1734259801) Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 oleg344-server: oleg344-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: Force libcfs to create 2 CPU partitions oleg344-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg344-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg344-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg344-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2498 (time 1734259862.06 total 10.00 last 249.73) - open/close 4728 (time 1734259872.06 total 20.01 last 222.94) - open/close 6921 (time 1734259882.07 total 30.01 last 219.26) - open/close 8971 (time 1734259892.07 total 40.01 last 204.94) - open/close 10000 (time 1734259897.31 total 45.25 last 196.18) - open/close 12085 (time 1734259907.32 total 55.26 last 208.47) - open/close 14423 (time 1734259917.32 total 65.26 last 233.72) - open/close 16934 (time 1734259927.32 total 75.26 last 251.06) - open/close 19291 (time 1734259937.32 total 85.26 last 235.62) - open/close 20000 (time 1734259940.46 total 88.40 last 225.83) - open/close 22326 (time 1734259950.46 total 98.40 last 232.59) - open/close 24722 (time 1734259960.46 total 108.41 last 239.58) - open/close 26997 (time 1734259970.47 total 118.41 last 227.45) - open/close 29166 (time 1734259980.47 total 128.41 last 216.89) - open/close 30000 (time 1734259984.35 total 132.29 last 214.63) - open/close 32312 (time 1734259994.35 total 142.30 last 231.17) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 143.26 seconds: 227.29 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5211 (time 1734260006.45 total 10.00 last 521.05) - /unlink 10000 (time 1734260016.24 total 19.79 last 489.15) - /unlink 14629 (time 1734260026.24 total 29.79 last 462.85) - /unlink 19485 (time 1734260036.24 total 39.80 last 485.45) - /unlink 23932 (time 1734260046.24 total 49.80 last 444.59) - /unlink 29275 (time 1734260056.24 total 59.80 last 534.26) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 64.49 seconds: 504.89 ops/second Waiting for MDT destroys to complete oleg344-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg344-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 - open/close 2826 (time 1734260080.52 total 10.00 last 282.57) - open/close 5642 (time 1734260090.53 total 20.00 last 281.58) - open/close 8323 (time 1734260100.53 total 30.00 last 268.03) - open/close 10000 (time 1734260106.32 total 35.80 last 289.36) - open/close 13045 (time 1734260116.32 total 45.80 last 304.46) - open/close 15859 (time 1734260126.33 total 55.80 last 281.36) - open/close 18476 (time 1734260136.33 total 65.81 last 261.59) - open/close 20000 (time 1734260142.20 total 71.67 last 259.82) - open/close 22954 (time 1734260152.20 total 81.68 last 295.32) - open/close 26202 (time 1734260162.20 total 91.68 last 324.76) - open/close 29216 (time 1734260172.20 total 101.68 last 301.37) - open/close 30000 (time 1734260174.98 total 104.46 last 282.06) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 113.29 seconds: 287.41 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5837 (time 1734260194.84 total 10.00 last 583.65) - /unlink 10000 (time 1734260203.12 total 18.28 last 503.02) - /unlink 15271 (time 1734260213.12 total 28.28 last 527.04) - /unlink 20000 (time 1734260222.28 total 37.44 last 516.16) - /unlink 24943 (time 1734260232.28 total 47.44 last 494.29) - /unlink 30000 (time 1734260242.20 total 57.36 last 509.92) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 62.48 seconds: 521.12 ops/second Waiting for MDT destroys to complete oleg344-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg344-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 - open/close 3220 (time 1734260265.57 total 10.00 last 322.00) - open/close 6055 (time 1734260275.57 total 20.00 last 283.43) - open/close 8779 (time 1734260285.58 total 30.00 last 272.38) - open/close 10000 (time 1734260289.39 total 33.82 last 319.86) - open/close 12476 (time 1734260299.40 total 43.82 last 247.49) - open/close 14993 (time 1734260309.40 total 53.83 last 251.61) - open/close 17552 (time 1734260319.40 total 63.83 last 255.82) - open/close 20000 (time 1734260328.11 total 72.54 last 281.25) - open/close 22658 (time 1734260338.11 total 82.54 last 265.75) - open/close 25364 (time 1734260348.11 total 92.54 last 270.54) - open/close 28281 (time 1734260358.11 total 102.54 last 291.61) - open/close 30000 (time 1734260364.71 total 109.13 last 260.81) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 118.56 seconds: 274.62 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4620 (time 1734260385.29 total 10.00 last 461.94) - /unlink 9081 (time 1734260395.29 total 20.00 last 446.07) - /unlink 13379 (time 1734260405.30 total 30.00 last 429.73) - /unlink 17958 (time 1734260415.30 total 40.01 last 457.82) - /unlink 20000 (time 1734260419.52 total 44.23 last 483.90) - /unlink 24789 (time 1734260429.52 total 54.23 last 478.85) - /unlink 29378 (time 1734260439.52 total 64.23 last 458.90) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 71.24 seconds: 457.06 ops/second Waiting for MDT destroys to complete oleg344-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 oleg344-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 - open/close 2709 (time 1734260460.98 total 10.00 last 270.81) - open/close 5500 (time 1734260470.99 total 20.01 last 279.01) - open/close 8642 (time 1734260480.99 total 30.01 last 314.12) - open/close 10000 (time 1734260485.29 total 34.31 last 315.77) - open/close 12974 (time 1734260495.29 total 44.31 last 297.39) - open/close 16053 (time 1734260505.29 total 54.31 last 307.82) - open/close 19119 (time 1734260515.29 total 64.31 last 306.55) - open/close 20000 (time 1734260518.06 total 67.08 last 318.09) - open/close 23280 (time 1734260528.07 total 77.09 last 327.90) - open/close 26701 (time 1734260538.07 total 87.09 last 342.05) - open/close 29497 (time 1734260548.07 total 97.09 last 279.58) - open/close 32463 (time 1734260558.07 total 107.09 last 296.51) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 107.45 seconds: 303.01 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5579 (time 1734260569.47 total 10.00 last 557.88) - /unlink 10000 (time 1734260576.94 total 17.47 last 591.72) - /unlink 16247 (time 1734260586.94 total 27.47 last 624.65) - /unlink 20000 (time 1734260592.53 total 33.06 last 671.62) - /unlink 25827 (time 1734260602.53 total 43.06 last 582.68) - /unlink 30000 (time 1734260610.08 total 50.61 last 553.11) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 54.81 seconds: 594.03 ops/second Waiting for MDT destroys to complete oleg344-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg344-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 - open/close 2987 (time 1734260636.76 total 10.00 last 298.65) - open/close 6332 (time 1734260646.77 total 20.00 last 334.39) - open/close 9977 (time 1734260656.77 total 30.01 last 364.40) - open/close 13821 (time 1734260666.77 total 40.01 last 384.32) - open/close 17130 (time 1734260676.77 total 50.01 last 330.85) - open/close 20000 (time 1734260685.22 total 58.46 last 339.59) - open/close 23565 (time 1734260695.22 total 68.46 last 356.48) - open/close 26823 (time 1734260705.22 total 78.46 last 325.78) - open/close 30000 (time 1734260714.35 total 87.59 last 348.09) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 94.74 seconds: 343.68 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 7141 (time 1734260732.44 total 10.00 last 714.03) - /unlink 10000 (time 1734260736.59 total 14.15 last 688.38) - /unlink 17713 (time 1734260746.59 total 24.15 last 771.24) - /unlink 20000 (time 1734260749.40 total 26.96 last 815.20) - /unlink 27100 (time 1734260759.40 total 36.96 last 709.92) - /unlink 30000 (time 1734260762.71 total 40.28 last 874.08) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 43.64 seconds: 746.04 ops/second Waiting for MDT destroys to complete oleg344-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg344-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg344-server unloading modules via unload_modules_local on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2