== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:50:17 (1734259817) Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 oleg204-server: oleg204-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: detected 4 online CPUs by sysfs oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg204-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg204-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg204-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2059 (time 1734259878.51 total 10.00 last 205.89) - open/close 4119 (time 1734259888.51 total 20.00 last 205.98) - open/close 6562 (time 1734259898.51 total 30.00 last 244.28) - open/close 8706 (time 1734259908.52 total 40.00 last 214.39) - open/close 10000 (time 1734259913.97 total 45.45 last 237.43) - open/close 12564 (time 1734259923.97 total 55.46 last 256.32) - open/close 15506 (time 1734259933.97 total 65.46 last 294.15) - open/close 18531 (time 1734259943.97 total 75.46 last 302.46) - open/close 20000 (time 1734259948.84 total 80.33 last 301.88) - open/close 22948 (time 1734259958.84 total 90.33 last 294.59) - open/close 25948 (time 1734259968.85 total 100.34 last 299.92) - open/close 29021 (time 1734259978.85 total 110.34 last 307.25) - open/close 30000 (time 1734259982.15 total 113.64 last 296.37) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 121.71 seconds: 267.52 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5145 (time 1734260001.25 total 10.00 last 514.46) - /unlink 10000 (time 1734260009.19 total 17.94 last 611.21) - /unlink 15375 (time 1734260019.19 total 27.94 last 537.45) - /unlink 20000 (time 1734260027.19 total 35.95 last 577.89) - /unlink 26057 (time 1734260037.20 total 45.95 last 605.67) - /unlink 30000 (time 1734260044.16 total 52.92 last 565.83) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 58.71 seconds: 554.56 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 2947 (time 1734260083.89 total 10.00 last 294.68) - open/close 6002 (time 1734260093.89 total 20.00 last 305.43) - open/close 8904 (time 1734260103.89 total 30.00 last 290.20) - open/close 10000 (time 1734260107.67 total 33.78 last 290.00) - open/close 13155 (time 1734260117.67 total 43.78 last 315.50) - open/close 16032 (time 1734260127.67 total 53.78 last 287.65) - open/close 18743 (time 1734260137.68 total 63.79 last 271.07) - open/close 20000 (time 1734260141.87 total 67.98 last 299.34) - open/close 23040 (time 1734260151.88 total 77.99 last 303.93) - open/close 26225 (time 1734260161.88 total 87.99 last 318.41) - open/close 29309 (time 1734260171.88 total 97.99 last 308.40) - open/close 30000 (time 1734260174.38 total 100.49 last 275.97) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 109.35 seconds: 297.76 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4712 (time 1734260194.41 total 10.00 last 471.11) - /unlink 10000 (time 1734260203.75 total 19.34 last 566.17) - /unlink 15436 (time 1734260213.75 total 29.34 last 543.53) - /unlink 20000 (time 1734260222.52 total 38.11 last 520.55) - /unlink 24485 (time 1734260232.52 total 48.11 last 448.47) - /unlink 29929 (time 1734260242.52 total 58.11 last 544.32) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 63.07 seconds: 516.22 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg204-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 3052 (time 1734260267.10 total 10.00 last 305.19) - open/close 6278 (time 1734260277.11 total 20.00 last 322.51) - open/close 9309 (time 1734260287.11 total 30.00 last 303.07) - open/close 10000 (time 1734260289.15 total 32.05 last 337.68) - open/close 13000 (time 1734260299.16 total 42.05 last 299.97) - open/close 15567 (time 1734260309.16 total 52.05 last 256.63) - open/close 18061 (time 1734260319.16 total 62.06 last 249.33) - open/close 20000 (time 1734260326.33 total 69.23 last 270.30) - open/close 22621 (time 1734260336.34 total 79.23 last 262.00) - open/close 25051 (time 1734260346.34 total 89.23 last 242.99) - open/close 27535 (time 1734260356.34 total 99.24 last 248.34) - open/close 30000 (time 1734260365.86 total 108.75 last 259.06) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 118.31 seconds: 275.22 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4676 (time 1734260386.46 total 10.00 last 467.53) - /unlink 9675 (time 1734260396.46 total 20.00 last 499.82) - /unlink 15200 (time 1734260406.46 total 30.00 last 552.42) - /unlink 20000 (time 1734260415.52 total 39.06 last 530.14) - /unlink 25224 (time 1734260425.52 total 49.06 last 522.33) - /unlink 30000 (time 1734260434.70 total 58.24 last 520.11) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 64.05 seconds: 508.35 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg204-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 3507 (time 1734260464.05 total 10.00 last 350.58) - open/close 6630 (time 1734260474.05 total 20.00 last 312.25) - open/close 9402 (time 1734260484.05 total 30.01 last 277.18) - open/close 10000 (time 1734260486.52 total 32.48 last 242.07) - open/close 12738 (time 1734260496.52 total 42.48 last 273.74) - open/close 16044 (time 1734260506.52 total 52.48 last 330.57) - open/close 19004 (time 1734260516.53 total 62.48 last 295.91) - open/close 20000 (time 1734260520.13 total 66.09 last 276.26) - open/close 22668 (time 1734260530.13 total 76.09 last 266.78) - open/close 25472 (time 1734260540.14 total 86.09 last 280.29) - open/close 28193 (time 1734260550.14 total 96.09 last 272.02) - open/close 30000 (time 1734260556.94 total 102.90 last 265.61) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 112.39 seconds: 289.71 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5392 (time 1734260577.50 total 10.00 last 539.19) - /unlink 10000 (time 1734260585.84 total 18.34 last 552.80) - /unlink 16931 (time 1734260595.84 total 28.34 last 692.97) - /unlink 20000 (time 1734260600.24 total 32.74 last 697.75) - /unlink 26332 (time 1734260610.24 total 42.74 last 633.10) - /unlink 30000 (time 1734260616.94 total 49.44 last 547.11) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 53.52 seconds: 608.38 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg204-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 3094 (time 1734260654.78 total 10.00 last 309.37) - open/close 7013 (time 1734260664.78 total 20.00 last 391.82) - open/close 10000 (time 1734260673.28 total 28.50 last 351.41) - open/close 13777 (time 1734260683.28 total 38.50 last 377.68) - open/close 16786 (time 1734260693.29 total 48.51 last 300.76) - open/close 19784 (time 1734260703.29 total 58.51 last 299.71) - open/close 22985 (time 1734260713.29 total 68.51 last 320.06) - open/close 26376 (time 1734260723.29 total 78.51 last 339.02) - open/close 29742 (time 1734260733.29 total 88.52 last 336.57) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 97.95 seconds: 332.41 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5436 (time 1734260753.73 total 10.00 last 543.51) - /unlink 10000 (time 1734260761.58 total 17.85 last 581.69) - /unlink 16216 (time 1734260771.58 total 27.85 last 621.53) - /unlink 20000 (time 1734260776.79 total 33.05 last 726.97) - /unlink 26813 (time 1734260786.79 total 43.05 last 681.28) - /unlink 30000 (time 1734260791.75 total 48.02 last 642.35) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 51.58 seconds: 631.20 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2