== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:49:40 (1734259780) Stopping clients: oleg356-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg356-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 oleg356-server: oleg356-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing load_modules_local oleg356-server: Loading modules from /home/green/git/lustre-release/lustre oleg356-server: detected 4 online CPUs by sysfs oleg356-server: Force libcfs to create 2 CPU partitions oleg356-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg356-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg356-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg356-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg356-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg356-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg356-server: oleg356-server.virtnet: executing set_default_debug -1 all pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg356-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg356-client.virtnet: -o user_xattr,flock 192.168.203.156@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2356 (time 1734259842.95 total 10.00 last 235.58) - open/close 4813 (time 1734259852.95 total 20.01 last 245.60) - open/close 7125 (time 1734259862.95 total 30.01 last 231.11) - open/close 9181 (time 1734259872.96 total 40.01 last 205.57) - open/close 10000 (time 1734259877.00 total 44.06 last 202.43) - open/close 11986 (time 1734259887.00 total 54.06 last 198.53) - open/close 13830 (time 1734259897.01 total 64.06 last 184.40) - open/close 15724 (time 1734259907.01 total 74.06 last 189.38) - open/close 17842 (time 1734259917.01 total 84.07 last 211.70) - open/close 20000 (time 1734259926.59 total 93.65 last 225.20) - open/close 22162 (time 1734259936.60 total 103.65 last 216.14) - open/close 24205 (time 1734259946.60 total 113.65 last 204.28) - open/close 26327 (time 1734259956.60 total 123.66 last 212.13) - open/close 28347 (time 1734259966.60 total 133.66 last 201.99) - open/close 30000 (time 1734259974.47 total 141.52 last 210.14) - open/close 32070 (time 1734259984.47 total 151.52 last 206.96) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 153.74 seconds: 211.78 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4282 (time 1734259997.93 total 10.00 last 428.18) - /unlink 9402 (time 1734260007.93 total 20.00 last 511.92) - /unlink 14056 (time 1734260017.93 total 30.00 last 465.38) - /unlink 18408 (time 1734260027.93 total 40.00 last 435.12) - /unlink 20000 (time 1734260031.13 total 43.20 last 498.20) - /unlink 24632 (time 1734260041.13 total 53.20 last 463.06) - /unlink 29173 (time 1734260051.13 total 63.20 last 454.06) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 69.55 seconds: 468.13 ops/second Waiting for MDT destroys to complete oleg356-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg356-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 - open/close 2780 (time 1734260075.45 total 10.00 last 277.90) - open/close 5552 (time 1734260085.45 total 20.00 last 277.18) - open/close 8234 (time 1734260095.45 total 30.00 last 268.20) - open/close 10000 (time 1734260102.05 total 36.60 last 267.57) - open/close 12859 (time 1734260112.05 total 46.60 last 285.89) - open/close 15560 (time 1734260122.05 total 56.60 last 270.10) - open/close 18092 (time 1734260132.05 total 66.61 last 253.16) - open/close 20000 (time 1734260139.94 total 74.50 last 241.84) - open/close 22742 (time 1734260149.95 total 84.50 last 274.15) - open/close 25567 (time 1734260159.95 total 94.50 last 282.45) - open/close 28482 (time 1734260169.95 total 104.50 last 291.44) - open/close 30000 (time 1734260175.63 total 110.18 last 267.34) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 119.56 seconds: 272.33 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5326 (time 1734260196.07 total 10.00 last 532.53) - /unlink 10000 (time 1734260205.94 total 19.88 last 473.24) - /unlink 14935 (time 1734260215.94 total 29.88 last 493.47) - /unlink 19720 (time 1734260225.94 total 39.88 last 478.41) - /unlink 24562 (time 1734260235.95 total 49.88 last 484.17) - /unlink 29152 (time 1734260245.95 total 59.88 last 458.91) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 66.54 seconds: 489.34 ops/second Waiting for MDT destroys to complete oleg356-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg356-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 - open/close 2895 (time 1734260268.09 total 10.00 last 289.42) - open/close 5338 (time 1734260278.09 total 20.00 last 244.28) - open/close 7812 (time 1734260288.09 total 30.01 last 247.34) - open/close 10000 (time 1734260297.15 total 39.06 last 241.65) - open/close 12169 (time 1734260307.15 total 49.06 last 216.88) - open/close 14616 (time 1734260317.15 total 59.06 last 244.69) - open/close 17095 (time 1734260327.15 total 69.06 last 247.86) - open/close 19618 (time 1734260337.15 total 79.07 last 252.19) - open/close 22161 (time 1734260347.15 total 89.07 last 254.28) - open/close 24832 (time 1734260357.16 total 99.07 last 267.04) - open/close 27270 (time 1734260367.16 total 109.07 last 243.74) - open/close 29669 (time 1734260377.16 total 119.08 last 239.84) - open/close 31885 (time 1734260387.16 total 129.08 last 221.60) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 131.95 seconds: 246.77 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4028 (time 1734260401.15 total 10.00 last 402.75) - /unlink 7739 (time 1734260411.15 total 20.00 last 371.04) - /unlink 10000 (time 1734260416.50 total 25.35 last 423.20) - /unlink 14281 (time 1734260426.50 total 35.35 last 428.06) - /unlink 18529 (time 1734260436.50 total 45.35 last 424.74) - /unlink 20000 (time 1734260440.15 total 49.00 last 402.77) - /unlink 23956 (time 1734260450.15 total 59.00 last 395.54) - /unlink 28344 (time 1734260460.15 total 69.00 last 438.75) - /unlink 30000 (time 1734260463.65 total 72.50 last 473.93) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 78.06 seconds: 417.14 ops/second Waiting for MDT destroys to complete oleg356-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 oleg356-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 - open/close 2805 (time 1734260490.24 total 10.00 last 280.46) - open/close 5526 (time 1734260500.24 total 20.00 last 272.03) - open/close 8447 (time 1734260510.25 total 30.01 last 292.01) - open/close 10000 (time 1734260515.96 total 35.72 last 271.89) - open/close 12947 (time 1734260525.96 total 45.72 last 294.65) - open/close 16303 (time 1734260535.96 total 55.72 last 335.47) - open/close 18973 (time 1734260545.97 total 65.73 last 266.90) - open/close 20000 (time 1734260550.58 total 70.34 last 222.79) - open/close 22558 (time 1734260560.58 total 80.34 last 255.76) - open/close 25275 (time 1734260570.58 total 90.34 last 271.69) - open/close 28037 (time 1734260580.58 total 100.34 last 276.12) - open/close 30000 (time 1734260587.10 total 106.87 last 300.95) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 115.66 seconds: 281.51 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4693 (time 1734260606.95 total 10.00 last 469.26) - /unlink 9639 (time 1734260616.95 total 20.00 last 494.60) - /unlink 14938 (time 1734260626.95 total 30.00 last 529.82) - /unlink 19777 (time 1734260636.95 total 40.00 last 483.87) - /unlink 24976 (time 1734260646.95 total 50.00 last 519.88) - /unlink 30000 (time 1734260656.01 total 59.06 last 554.56) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 63.20 seconds: 515.22 ops/second Waiting for MDT destroys to complete oleg356-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg356-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 - open/close 2791 (time 1734260680.90 total 10.00 last 279.04) - open/close 5766 (time 1734260690.91 total 20.01 last 297.41) - open/close 8797 (time 1734260700.91 total 30.01 last 303.07) - open/close 10000 (time 1734260705.04 total 34.14 last 290.80) - open/close 13277 (time 1734260715.05 total 44.15 last 327.63) - open/close 16804 (time 1734260725.05 total 54.15 last 352.65) - open/close 20000 (time 1734260734.02 total 63.12 last 356.23) - open/close 23782 (time 1734260744.02 total 73.12 last 378.16) - open/close 27600 (time 1734260754.02 total 83.12 last 381.80) - open/close 30000 (time 1734260760.69 total 89.79 last 359.56) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 96.12 seconds: 338.73 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9699 (time 1734260777.94 total 10.00 last 969.81) - /unlink 18820 (time 1734260787.94 total 20.00 last 912.09) - /unlink 29942 (time 1734260797.94 total 30.00 last 1112.17) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 32.26 seconds: 1009.28 ops/second Waiting for MDT destroys to complete oleg356-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg356-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg356-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg356-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg356-server stop mds service on oleg356-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg356-server unloading modules via unload_modules_local on: 'oleg356-server' oleg356-server: oleg356-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-client: ssh exited with exit code 2 pdsh@oleg356-client: oleg356-server: ssh exited with exit code 2