== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:42:12 (1734259332) Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 oleg440-server: oleg440-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg440-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock 192.168.204.140@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 368 32528 2% /mnt/lustre[OST:0] filesystem_summary: 33057 529 32528 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4082 (time 1734259384.25 total 10.00 last 408.18) - open/close 8018 (time 1734259394.25 total 20.00 last 393.57) - open/close 10000 (time 1734259398.87 total 24.62 last 429.47) - open/close 14210 (time 1734259408.87 total 34.62 last 420.99) - open/close 18252 (time 1734259418.87 total 44.62 last 404.15) - open/close 20000 (time 1734259422.48 total 48.23 last 483.46) - open/close 24668 (time 1734259432.48 total 58.23 last 466.74) - open/close 29378 (time 1734259442.49 total 68.24 last 470.96) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 75.81 seconds: 429.49 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 8880 (time 1734259460.93 total 10.00 last 887.94) - /unlink 16771 (time 1734259470.93 total 20.00 last 789.08) - /unlink 20000 (time 1734259474.54 total 23.62 last 893.15) - /unlink 28589 (time 1734259484.54 total 33.62 last 858.83) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 37.23 seconds: 874.67 ops/second Waiting for MDT destroys to complete oleg440-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg440-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 - open/close 5613 (time 1734259527.15 total 10.00 last 561.22) - open/close 10000 (time 1734259535.03 total 17.88 last 556.72) - open/close 15951 (time 1734259545.03 total 27.88 last 595.08) - open/close 20000 (time 1734259552.00 total 34.85 last 581.20) - open/close 26070 (time 1734259562.00 total 44.85 last 606.98) - open/close 30000 (time 1734259568.34 total 51.18 last 620.28) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 55.36 seconds: 588.16 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259583.02 total 9.69 last 1032.24) - /unlink 20000 (time 1734259592.85 total 19.52 last 1017.27) - /unlink 30000 (time 1734259601.89 total 28.55 last 1106.98) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 30.98 seconds: 1051.10 ops/second Waiting for MDT destroys to complete oleg440-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg440-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 - open/close 5353 (time 1734259648.53 total 10.00 last 535.25) - open/close 10000 (time 1734259657.47 total 18.94 last 519.76) - open/close 14934 (time 1734259667.47 total 28.94 last 493.37) - open/close 19789 (time 1734259677.48 total 38.94 last 485.47) - open/close 24252 (time 1734259687.48 total 48.94 last 446.25) - open/close 29142 (time 1734259697.48 total 58.95 last 488.94) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 66.17 seconds: 492.07 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259714.87 total 9.26 last 1079.72) - /unlink 20000 (time 1734259724.31 total 18.70 last 1058.95) - /unlink 29064 (time 1734259734.31 total 28.71 last 906.39) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 32.16 seconds: 1012.51 ops/second Waiting for MDT destroys to complete oleg440-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg440-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 - open/close 5423 (time 1734259796.77 total 10.00 last 542.28) - open/close 10000 (time 1734259804.85 total 18.08 last 566.65) - open/close 15557 (time 1734259814.85 total 28.08 last 555.59) - open/close 20000 (time 1734259823.26 total 36.49 last 528.47) - open/close 25346 (time 1734259833.26 total 46.49 last 534.51) - open/close 30000 (time 1734259842.04 total 55.26 last 530.31) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 59.60 seconds: 546.32 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 10000 (time 1734259857.05 total 9.87 last 1013.60) - /unlink 20000 (time 1734259866.99 total 19.80 last 1006.17) - /unlink 29874 (time 1734259876.99 total 29.80 last 987.39) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 32.45 seconds: 1003.37 ops/second Waiting for MDT destroys to complete oleg440-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg440-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 - open/close 5442 (time 1734259936.09 total 10.00 last 544.11) - open/close 10000 (time 1734259944.33 total 18.24 last 553.12) - open/close 15608 (time 1734259954.33 total 28.24 last 560.72) - open/close 20000 (time 1734259962.28 total 36.20 last 552.30) - open/close 25550 (time 1734259972.28 total 46.20 last 554.95) - open/close 30000 (time 1734259980.36 total 54.27 last 551.10) open(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No space left on device total: 32560 open/close in 59.10 seconds: 550.95 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32835 47157 42% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 33090 33090 0 100% /mnt/lustre - /unlink 9515 (time 1734259996.11 total 10.00 last 951.48) - /unlink 19381 (time 1734260006.11 total 20.00 last 986.55) - /unlink 28102 (time 1734260016.11 total 30.00 last 872.07) - /unlink 30000 (time 1734260018.24 total 32.13 last 889.98) unlink(/mnt/lustre/d106.conf-sanity-1/f-32560) error: No such file or directory total: 32560 /unlink in 34.90 seconds: 932.94 ops/second Waiting for MDT destroys to complete oleg440-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg440-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server unloading modules via unload_modules_local on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2