== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:42:10 (1734259330) Stopping clients: oleg439-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg439-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 oleg439-server: oleg439-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing load_modules_local oleg439-server: Loading modules from /home/green/git/lustre-release/lustre oleg439-server: detected 4 online CPUs by sysfs oleg439-server: Force libcfs to create 2 CPU partitions oleg439-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg439-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg439-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg439-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg439-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg439-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg439-server: oleg439-server.virtnet: executing set_default_debug -1 all pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg439-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg439-client.virtnet: -o user_xattr,flock 192.168.204.139@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4684 (time 1734259382.43 total 10.00 last 468.37) - open/close 9176 (time 1734259392.43 total 20.00 last 449.18) - open/close 13763 (time 1734259402.43 total 30.00 last 458.69) - open/close 18350 (time 1734259412.43 total 40.00 last 458.64) - open/close 20000 (time 1734259416.02 total 43.59 last 459.91) - open/close 24635 (time 1734259426.02 total 53.59 last 463.44) - open/close 29708 (time 1734259436.02 total 63.59 last 507.28) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 69.30 seconds: 469.84 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8985 (time 1734259452.62 total 10.00 last 898.44) - /unlink 18596 (time 1734259462.62 total 20.00 last 961.01) - /unlink 27621 (time 1734259472.62 total 30.00 last 902.44) - /unlink 30000 (time 1734259475.15 total 32.53 last 939.50) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 35.24 seconds: 923.84 ops/second Waiting for MDT destroys to complete oleg439-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg439-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 - open/close 6144 (time 1734259536.34 total 10.00 last 614.26) - open/close 10000 (time 1734259542.38 total 16.04 last 638.28) - open/close 16282 (time 1734259552.38 total 26.04 last 628.19) - open/close 20000 (time 1734259558.26 total 31.92 last 632.47) - open/close 26478 (time 1734259568.26 total 41.92 last 647.79) - open/close 30000 (time 1734259573.70 total 47.37 last 646.51) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 51.42 seconds: 633.27 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259587.98 total 9.39 last 1064.97) - /unlink 20000 (time 1734259596.99 total 18.40 last 1110.31) - /unlink 30000 (time 1734259605.73 total 27.13 last 1144.56) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 29.41 seconds: 1106.95 ops/second Waiting for MDT destroys to complete oleg439-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg439-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 - open/close 5598 (time 1734259653.70 total 10.00 last 559.74) - open/close 10000 (time 1734259661.86 total 18.17 last 539.02) - open/close 15381 (time 1734259671.86 total 28.17 last 538.10) - open/close 20000 (time 1734259681.11 total 37.42 last 499.41) - open/close 24705 (time 1734259691.11 total 47.42 last 470.44) - open/close 29841 (time 1734259701.11 total 57.42 last 513.58) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 62.78 seconds: 518.66 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259716.16 total 8.87 last 1126.77) - /unlink 20000 (time 1734259725.19 total 17.91 last 1107.03) - /unlink 29912 (time 1734259735.19 total 27.91 last 991.14) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 30.42 seconds: 1070.19 ops/second Waiting for MDT destroys to complete oleg439-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg439-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 - open/close 5257 (time 1734259781.79 total 10.00 last 525.61) - open/close 10000 (time 1734259790.55 total 18.76 last 541.58) - open/close 15785 (time 1734259800.55 total 28.76 last 578.42) - open/close 20000 (time 1734259807.87 total 36.08 last 576.03) - open/close 25617 (time 1734259817.87 total 46.08 last 561.67) - open/close 30000 (time 1734259825.80 total 54.01 last 552.68) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 58.63 seconds: 555.31 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9475 (time 1734259841.32 total 10.00 last 947.46) - /unlink 20000 (time 1734259850.82 total 19.51 last 1107.12) - /unlink 30000 (time 1734259860.57 total 29.25 last 1026.53) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 31.78 seconds: 1024.47 ops/second Waiting for MDT destroys to complete oleg439-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg439-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 - open/close 5660 (time 1734259922.09 total 10.00 last 566.00) - open/close 10000 (time 1734259929.93 total 17.84 last 553.29) - open/close 15702 (time 1734259939.93 total 27.84 last 570.16) - open/close 20000 (time 1734259947.40 total 35.31 last 575.79) - open/close 25772 (time 1734259957.40 total 45.31 last 577.20) - open/close 30000 (time 1734259964.85 total 52.76 last 567.44) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 57.18 seconds: 569.42 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259980.10 total 9.95 last 1004.58) - /unlink 19995 (time 1734259990.10 total 19.96 last 999.44) - /unlink 30000 (time 1734259999.83 total 29.69 last 1027.35) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 32.16 seconds: 1012.45 ops/second Waiting for MDT destroys to complete oleg439-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg439-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg439-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg439-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg439-server stop mds service on oleg439-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg439-server unloading modules via unload_modules_local on: 'oleg439-server' oleg439-server: oleg439-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-client: ssh exited with exit code 2 pdsh@oleg439-client: oleg439-server: ssh exited with exit code 2