== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:41:27 (1734259287) Stopping clients: oleg422-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg422-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2 oleg422-server: oleg422-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing load_modules_local oleg422-server: Loading modules from /home/green/git/lustre-release/lustre oleg422-server: detected 4 online CPUs by sysfs oleg422-server: Force libcfs to create 2 CPU partitions oleg422-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg422-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg422-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg422-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg422-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg422-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg422-server: oleg422-server.virtnet: executing set_default_debug -1 all pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg422-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg422-client.virtnet: -o user_xattr,flock 192.168.204.122@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5173 (time 1734259338.30 total 10.00 last 517.28) - open/close 10000 (time 1734259348.08 total 19.78 last 493.52) - open/close 14919 (time 1734259358.08 total 29.78 last 491.87) - open/close 20000 (time 1734259366.53 total 38.23 last 601.71) - open/close 25300 (time 1734259376.53 total 48.23 last 529.94) - open/close 29579 (time 1734259386.53 total 58.23 last 427.83) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 64.86 seconds: 502.02 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8946 (time 1734259404.09 total 10.00 last 894.57) - /unlink 18201 (time 1734259414.09 total 20.00 last 925.40) - /unlink 20000 (time 1734259416.37 total 22.29 last 787.62) - /unlink 29729 (time 1734259426.37 total 32.29 last 972.84) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 35.07 seconds: 928.56 ops/second Waiting for MDT destroys to complete oleg422-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg422-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 - open/close 5025 (time 1734259466.01 total 10.00 last 502.49) - open/close 10000 (time 1734259475.95 total 19.94 last 500.36) - open/close 15241 (time 1734259485.95 total 29.94 last 524.08) - open/close 20000 (time 1734259493.68 total 37.67 last 615.60) - open/close 26097 (time 1734259503.68 total 47.68 last 609.61) - open/close 30000 (time 1734259510.14 total 54.13 last 604.36) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 58.51 seconds: 556.46 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259524.64 total 9.32 last 1073.21) - /unlink 20000 (time 1734259534.08 total 18.76 last 1059.23) - /unlink 30000 (time 1734259542.90 total 27.58 last 1133.97) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 29.80 seconds: 1092.51 ops/second Waiting for MDT destroys to complete oleg422-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg422-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 - open/close 5843 (time 1734259585.56 total 10.00 last 584.27) - open/close 10000 (time 1734259592.78 total 17.22 last 575.52) - open/close 16134 (time 1734259602.79 total 27.22 last 613.34) - open/close 20000 (time 1734259608.86 total 33.30 last 636.19) - open/close 26123 (time 1734259618.86 total 43.30 last 612.21) - open/close 30000 (time 1734259625.03 total 49.47 last 628.68) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 53.55 seconds: 608.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259639.08 total 9.14 last 1093.56) - /unlink 20000 (time 1734259648.74 total 18.81 last 1035.11) - /unlink 30000 (time 1734259658.60 total 28.66 last 1014.23) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 31.43 seconds: 1036.04 ops/second Waiting for MDT destroys to complete oleg422-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg422-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 - open/close 6416 (time 1734259722.07 total 10.00 last 641.60) - open/close 10000 (time 1734259728.19 total 16.12 last 585.36) - open/close 15334 (time 1734259738.19 total 26.12 last 533.35) - open/close 20000 (time 1734259745.60 total 33.53 last 629.85) - open/close 26111 (time 1734259755.60 total 43.53 last 611.03) - open/close 30000 (time 1734259762.78 total 50.71 last 541.80) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 55.98 seconds: 581.59 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9058 (time 1734259778.97 total 10.00 last 905.77) - /unlink 18212 (time 1734259788.97 total 20.00 last 915.31) - /unlink 28259 (time 1734259798.97 total 30.00 last 1004.68) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 34.14 seconds: 953.80 ops/second Waiting for MDT destroys to complete oleg422-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg422-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 - open/close 5353 (time 1734259837.37 total 10.00 last 535.22) - open/close 10000 (time 1734259846.49 total 19.11 last 509.94) - open/close 15690 (time 1734259856.49 total 29.12 last 568.95) - open/close 20000 (time 1734259864.16 total 36.78 last 562.03) - open/close 25417 (time 1734259874.16 total 46.79 last 541.63) - open/close 30000 (time 1734259882.40 total 55.03 last 555.82) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 59.70 seconds: 545.36 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32760 47232 41% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33015 33015 0 100% /mnt/lustre - /unlink 9712 (time 1734259897.97 total 10.00 last 971.13) - /unlink 19771 (time 1734259907.97 total 20.00 last 1005.85) - /unlink 29806 (time 1734259917.97 total 30.00 last 1003.50) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 32.69 seconds: 996.08 ops/second Waiting for MDT destroys to complete oleg422-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg422-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg422-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg422-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg422-server stop mds service on oleg422-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg422-server unloading modules via unload_modules_local on: 'oleg422-server' oleg422-server: oleg422-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-client: ssh exited with exit code 2 pdsh@oleg422-client: oleg422-server: ssh exited with exit code 2