== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:48:50 (1734259730) Stopping clients: oleg337-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg337-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2 oleg337-server: oleg337-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing load_modules_local oleg337-server: Loading modules from /home/green/git/lustre-release/lustre oleg337-server: detected 4 online CPUs by sysfs oleg337-server: Force libcfs to create 2 CPU partitions oleg337-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg337-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg337-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg337-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg337-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2515 (time 1734259792.80 total 10.00 last 251.43) - open/close 4654 (time 1734259802.80 total 20.01 last 213.76) - open/close 6702 (time 1734259812.80 total 30.01 last 204.80) - open/close 9022 (time 1734259822.80 total 40.01 last 231.98) - open/close 10000 (time 1734259826.48 total 43.69 last 265.91) - open/close 12640 (time 1734259836.49 total 53.69 last 263.92) - open/close 15354 (time 1734259846.49 total 63.69 last 271.36) - open/close 18049 (time 1734259856.49 total 73.70 last 269.43) - open/close 20000 (time 1734259863.95 total 81.15 last 261.67) - open/close 22311 (time 1734259873.95 total 91.15 last 231.09) - open/close 24565 (time 1734259883.95 total 101.16 last 225.24) - open/close 26681 (time 1734259893.96 total 111.16 last 211.54) - open/close 28819 (time 1734259903.96 total 121.17 last 213.71) - open/close 30000 (time 1734259909.26 total 126.47 last 222.72) - open/close 32449 (time 1734259919.26 total 136.47 last 244.87) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 136.92 seconds: 237.80 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 4453 (time 1734259930.81 total 10.00 last 445.26) - /unlink 8857 (time 1734259940.82 total 20.00 last 440.39) - /unlink 10000 (time 1734259943.44 total 22.63 last 435.58) - /unlink 14240 (time 1734259953.44 total 32.63 last 423.94) - /unlink 18354 (time 1734259963.44 total 42.63 last 411.35) - /unlink 20000 (time 1734259967.48 total 46.66 last 408.03) - /unlink 24394 (time 1734259977.48 total 56.66 last 439.36) - /unlink 29115 (time 1734259987.48 total 66.66 last 472.07) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 73.68 seconds: 441.91 ops/second Waiting for MDT destroys to complete oleg337-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg337-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 - open/close 3174 (time 1734260012.51 total 10.00 last 317.32) - open/close 5977 (time 1734260022.51 total 20.00 last 280.23) - open/close 8892 (time 1734260032.51 total 30.01 last 291.41) - open/close 10000 (time 1734260036.52 total 34.01 last 276.84) - open/close 12604 (time 1734260046.52 total 44.01 last 260.34) - open/close 15870 (time 1734260056.52 total 54.01 last 326.59) - open/close 19510 (time 1734260066.52 total 64.01 last 363.92) - open/close 22526 (time 1734260076.52 total 74.02 last 301.59) - open/close 25535 (time 1734260086.52 total 84.02 last 300.87) - open/close 28471 (time 1734260096.52 total 94.02 last 293.57) - open/close 30000 (time 1734260101.89 total 99.39 last 284.81) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 107.68 seconds: 302.38 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32880 16 100% /mnt/lustre[OST:0] filesystem_summary: 33106 33090 16 100% /mnt/lustre - /unlink 5330 (time 1734260121.24 total 10.00 last 532.94) - /unlink 10000 (time 1734260130.14 total 18.90 last 524.88) - /unlink 14920 (time 1734260140.14 total 28.90 last 491.96) - /unlink 20000 (time 1734260148.76 total 37.52 last 589.01) - /unlink 26115 (time 1734260158.76 total 47.53 last 611.43) - /unlink 30000 (time 1734260164.30 total 53.06 last 702.29) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 57.19 seconds: 569.30 ops/second Waiting for MDT destroys to complete oleg337-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg337-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 - open/close 3378 (time 1734260187.65 total 10.00 last 337.73) - open/close 6623 (time 1734260197.65 total 20.00 last 324.42) - open/close 9747 (time 1734260207.65 total 30.01 last 312.36) - open/close 12909 (time 1734260217.65 total 40.01 last 316.13) - open/close 15880 (time 1734260227.65 total 50.01 last 297.03) - open/close 18751 (time 1734260237.66 total 60.01 last 287.07) - open/close 20000 (time 1734260241.81 total 64.17 last 300.46) - open/close 23030 (time 1734260251.82 total 74.17 last 302.88) - open/close 26541 (time 1734260261.82 total 84.18 last 350.97) - open/close 29546 (time 1734260271.82 total 94.18 last 300.47) - open/close 32223 (time 1734260281.82 total 104.18 last 267.67) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 105.43 seconds: 308.84 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5120 (time 1734260294.22 total 10.00 last 511.94) - /unlink 9480 (time 1734260304.22 total 20.00 last 435.96) - /unlink 14618 (time 1734260314.22 total 30.00 last 513.69) - /unlink 19634 (time 1734260324.22 total 40.01 last 501.53) - /unlink 25043 (time 1734260334.22 total 50.01 last 540.81) - /unlink 30000 (time 1734260343.31 total 59.10 last 545.34) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 63.85 seconds: 509.91 ops/second Waiting for MDT destroys to complete oleg337-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg337-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 - open/close 2818 (time 1734260366.06 total 10.00 last 281.75) - open/close 5675 (time 1734260376.06 total 20.00 last 285.68) - open/close 8294 (time 1734260386.06 total 30.00 last 261.86) - open/close 10000 (time 1734260392.40 total 36.34 last 269.33) - open/close 12475 (time 1734260402.40 total 46.34 last 247.45) - open/close 14902 (time 1734260412.40 total 56.34 last 242.67) - open/close 17617 (time 1734260422.40 total 66.35 last 271.39) - open/close 20000 (time 1734260431.40 total 75.34 last 264.84) - open/close 22522 (time 1734260441.40 total 85.34 last 252.16) - open/close 24992 (time 1734260451.41 total 95.35 last 246.94) - open/close 27600 (time 1734260461.41 total 105.35 last 260.67) - open/close 30000 (time 1734260470.08 total 114.02 last 276.83) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 122.02 seconds: 266.84 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 5836 (time 1734260489.07 total 10.00 last 583.56) - /unlink 10000 (time 1734260496.42 total 17.35 last 566.55) - /unlink 16272 (time 1734260506.42 total 27.35 last 627.18) - /unlink 20000 (time 1734260512.63 total 33.56 last 599.97) - /unlink 26293 (time 1734260522.63 total 43.56 last 629.27) - /unlink 30000 (time 1734260527.97 total 48.90 last 694.51) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 52.35 seconds: 622.02 ops/second Waiting for MDT destroys to complete oleg337-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg337-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 - open/close 2902 (time 1734260555.75 total 10.00 last 290.19) - open/close 5844 (time 1734260565.76 total 20.00 last 294.18) - open/close 9118 (time 1734260575.76 total 30.00 last 327.30) - open/close 10000 (time 1734260578.71 total 32.96 last 298.91) - open/close 13450 (time 1734260588.71 total 42.96 last 344.96) - open/close 16702 (time 1734260598.71 total 52.96 last 325.17) - open/close 19622 (time 1734260608.71 total 62.96 last 291.95) - open/close 22785 (time 1734260618.71 total 72.96 last 316.26) - open/close 25894 (time 1734260628.71 total 82.96 last 310.89) - open/close 28796 (time 1734260638.72 total 92.96 last 290.19) - open/close 30000 (time 1734260642.60 total 96.85 last 309.63) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 104.83 seconds: 310.61 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 6386 (time 1734260661.57 total 10.00 last 638.57) - /unlink 10000 (time 1734260667.38 total 15.81 last 622.50) - /unlink 15629 (time 1734260677.38 total 25.81 last 562.87) - /unlink 20000 (time 1734260684.68 total 33.11 last 598.81) - /unlink 26453 (time 1734260694.68 total 43.11 last 645.26) - /unlink 30000 (time 1734260700.26 total 48.69 last 635.73) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 52.79 seconds: 616.80 ops/second Waiting for MDT destroys to complete oleg337-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg337-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg337-server unloading modules via unload_modules_local on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2