== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:47:54 (1734259674) Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: Force libcfs to create 2 CPU partitions oleg151-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg151-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg151-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg151-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1860 (time 1734259741.69 total 10.00 last 185.98) - open/close 3833 (time 1734259751.70 total 20.00 last 197.26) - open/close 5695 (time 1734259761.70 total 30.01 last 186.16) - open/close 7523 (time 1734259771.70 total 40.01 last 182.76) - open/close 9728 (time 1734259781.70 total 50.01 last 220.44) - open/close 11951 (time 1734259791.70 total 60.01 last 222.25) - open/close 14023 (time 1734259801.71 total 70.01 last 207.19) - open/close 16054 (time 1734259811.71 total 80.02 last 203.04) - open/close 18193 (time 1734259821.71 total 90.02 last 213.85) - open/close 20000 (time 1734259830.96 total 99.26 last 195.47) - open/close 21828 (time 1734259840.96 total 109.27 last 182.64) - open/close 23660 (time 1734259850.97 total 119.27 last 183.16) - open/close 25645 (time 1734259860.97 total 129.28 last 198.47) - open/close 27381 (time 1734259870.97 total 139.28 last 173.52) - open/close 29184 (time 1734259880.97 total 149.28 last 180.30) - open/close 30000 (time 1734259885.15 total 153.46 last 195.16) - open/close 31994 (time 1734259895.16 total 163.47 last 199.32) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 166.47 seconds: 195.59 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 3854 (time 1734259909.56 total 10.00 last 385.39) - /unlink 7895 (time 1734259919.56 total 20.00 last 404.05) - /unlink 10000 (time 1734259924.68 total 25.12 last 411.12) - /unlink 14113 (time 1734259934.68 total 35.12 last 411.27) - /unlink 18120 (time 1734259944.68 total 45.13 last 400.60) - /unlink 20000 (time 1734259949.36 total 49.80 last 401.95) - /unlink 24313 (time 1734259959.36 total 59.80 last 431.24) - /unlink 28601 (time 1734259969.37 total 69.81 last 428.66) - /unlink 30000 (time 1734259972.96 total 73.40 last 389.66) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 79.19 seconds: 411.16 ops/second Waiting for MDT destroys to complete oleg151-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg151-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 - open/close 2388 (time 1734259999.88 total 10.01 last 238.65) - open/close 4849 (time 1734260009.89 total 20.01 last 246.00) - open/close 7225 (time 1734260019.89 total 30.01 last 237.52) - open/close 9579 (time 1734260029.89 total 40.01 last 235.37) - open/close 11898 (time 1734260039.89 total 50.02 last 231.84) - open/close 14238 (time 1734260049.89 total 60.02 last 234.00) - open/close 16448 (time 1734260059.89 total 70.02 last 221.00) - open/close 18427 (time 1734260069.90 total 80.02 last 197.84) - open/close 20000 (time 1734260077.80 total 87.92 last 199.09) - open/close 22043 (time 1734260087.80 total 97.93 last 204.20) - open/close 24216 (time 1734260097.80 total 107.93 last 217.27) - open/close 26294 (time 1734260107.81 total 117.93 last 207.77) - open/close 28439 (time 1734260117.81 total 127.93 last 214.47) - open/close 30000 (time 1734260124.81 total 134.93 last 223.05) - open/close 32377 (time 1734260134.81 total 144.93 last 237.63) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 145.66 seconds: 223.53 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4419 (time 1734260146.65 total 10.00 last 441.87) - /unlink 9022 (time 1734260156.65 total 20.00 last 460.22) - /unlink 13965 (time 1734260166.65 total 30.00 last 494.22) - /unlink 18608 (time 1734260176.65 total 40.00 last 464.27) - /unlink 20000 (time 1734260179.23 total 42.58 last 539.53) - /unlink 25549 (time 1734260189.24 total 52.59 last 554.80) - /unlink 30000 (time 1734260196.99 total 60.34 last 574.06) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 65.22 seconds: 499.21 ops/second Waiting for MDT destroys to complete oleg151-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg151-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 - open/close 3389 (time 1734260220.12 total 10.00 last 338.80) - open/close 7052 (time 1734260230.12 total 20.00 last 366.24) - open/close 10000 (time 1734260238.49 total 28.38 last 352.17) - open/close 12976 (time 1734260248.50 total 38.38 last 297.50) - open/close 16091 (time 1734260258.50 total 48.38 last 311.41) - open/close 19330 (time 1734260268.50 total 58.38 last 323.89) - open/close 20000 (time 1734260270.79 total 60.67 last 293.00) - open/close 23499 (time 1734260280.79 total 70.67 last 349.88) - open/close 26841 (time 1734260290.79 total 80.67 last 334.11) - open/close 30000 (time 1734260300.21 total 90.09 last 335.29) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 99.52 seconds: 327.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4545 (time 1734260320.72 total 10.00 last 454.49) - /unlink 9431 (time 1734260330.72 total 20.00 last 488.45) - /unlink 13900 (time 1734260340.72 total 30.01 last 446.76) - /unlink 18842 (time 1734260350.72 total 40.01 last 494.19) - /unlink 20000 (time 1734260352.98 total 42.27 last 512.12) - /unlink 26298 (time 1734260362.98 total 52.27 last 629.75) - /unlink 30000 (time 1734260368.79 total 58.07 last 637.79) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 62.32 seconds: 522.50 ops/second Waiting for MDT destroys to complete oleg151-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 oleg151-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 - open/close 3267 (time 1734260391.25 total 10.00 last 326.69) - open/close 6604 (time 1734260401.25 total 20.00 last 333.66) - open/close 9855 (time 1734260411.25 total 30.00 last 325.10) - open/close 12619 (time 1734260421.26 total 40.00 last 276.36) - open/close 15546 (time 1734260431.26 total 50.00 last 292.66) - open/close 18206 (time 1734260441.26 total 60.01 last 265.92) - open/close 20000 (time 1734260448.63 total 67.37 last 243.52) - open/close 22503 (time 1734260458.63 total 77.38 last 250.26) - open/close 25004 (time 1734260468.63 total 87.38 last 250.01) - open/close 27208 (time 1734260478.64 total 97.38 last 220.33) - open/close 29641 (time 1734260488.64 total 107.38 last 243.27) - open/close 31902 (time 1734260498.64 total 117.39 last 225.97) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 120.06 seconds: 271.19 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4881 (time 1734260512.44 total 10.00 last 488.07) - /unlink 10000 (time 1734260521.99 total 19.55 last 536.01) - /unlink 15804 (time 1734260531.99 total 29.55 last 580.31) - /unlink 20000 (time 1734260539.47 total 37.03 last 561.49) - /unlink 25338 (time 1734260549.47 total 47.03 last 533.71) - /unlink 30000 (time 1734260558.52 total 56.08 last 515.21) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 60.52 seconds: 538.00 ops/second Waiting for MDT destroys to complete oleg151-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg151-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 - open/close 3481 (time 1734260578.15 total 10.00 last 348.05) - open/close 7075 (time 1734260588.15 total 20.00 last 359.40) - open/close 9936 (time 1734260598.15 total 30.00 last 286.07) - open/close 13283 (time 1734260608.16 total 40.01 last 334.59) - open/close 17279 (time 1734260618.16 total 50.01 last 399.60) - open/close 20000 (time 1734260626.23 total 58.07 last 337.23) - open/close 24575 (time 1734260636.23 total 68.08 last 457.44) - open/close 28353 (time 1734260646.23 total 78.08 last 377.74) - open/close 30000 (time 1734260652.23 total 84.08 last 274.25) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 92.87 seconds: 350.61 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5355 (time 1734260672.14 total 10.00 last 535.41) - /unlink 10000 (time 1734260680.67 total 18.53 last 544.82) - /unlink 16312 (time 1734260690.67 total 28.53 last 631.13) - /unlink 20000 (time 1734260696.95 total 34.81 last 587.45) - /unlink 25981 (time 1734260706.95 total 44.81 last 598.04) - /unlink 30000 (time 1734260713.24 total 51.10 last 639.18) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 55.13 seconds: 590.55 ops/second Waiting for MDT destroys to complete oleg151-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg151-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg151-server unloading modules via unload_modules_local on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2