== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:25:53 (1734258353) Stopping clients: oleg434-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg434-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 oleg434-server: oleg434-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing load_modules_local oleg434-server: Loading modules from /home/green/git/lustre-release/lustre oleg434-server: detected 4 online CPUs by sysfs oleg434-server: Force libcfs to create 2 CPU partitions oleg434-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg434-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg434-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg434-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4336 (time 1734258393.79 total 10.00 last 433.53) - open/close 8254 (time 1734258403.80 total 20.00 last 391.73) - open/close 10000 (time 1734258407.28 total 23.49 last 501.09) - open/close 14180 (time 1734258417.28 total 33.49 last 418.00) - open/close 17686 (time 1734258427.28 total 43.49 last 350.59) - open/close 20000 (time 1734258432.91 total 49.12 last 411.13) - open/close 23154 (time 1734258442.91 total 59.12 last 315.31) - open/close 25573 (time 1734258452.91 total 69.12 last 241.87) - open/close 28172 (time 1734258462.92 total 79.13 last 259.70) - open/close 30000 (time 1734258468.54 total 84.75 last 325.40) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 91.82 seconds: 355.32 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 6381 (time 1734258486.61 total 10.00 last 638.02) - /unlink 10000 (time 1734258491.72 total 15.11 last 709.06) - /unlink 15757 (time 1734258501.72 total 25.11 last 575.69) - /unlink 20000 (time 1734258507.95 total 31.34 last 681.08) - /unlink 25724 (time 1734258517.95 total 41.34 last 572.36) - /unlink 30000 (time 1734258524.87 total 48.26 last 617.89) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 53.31 seconds: 612.04 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 - open/close 3149 (time 1734258552.88 total 10.00 last 314.89) - open/close 5872 (time 1734258562.89 total 20.00 last 272.24) - open/close 9970 (time 1734258572.89 total 30.00 last 409.74) - open/close 14447 (time 1734258582.89 total 40.01 last 447.59) - open/close 18042 (time 1734258592.89 total 50.01 last 359.47) - open/close 20000 (time 1734258598.98 total 56.10 last 321.35) - open/close 22885 (time 1734258608.98 total 66.10 last 288.46) - open/close 25470 (time 1734258618.99 total 76.10 last 258.43) - open/close 28569 (time 1734258628.99 total 86.10 last 309.89) - open/close 30000 (time 1734258632.86 total 89.98 last 369.32) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 97.51 seconds: 334.58 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5688 (time 1734258651.49 total 10.00 last 568.76) - /unlink 10000 (time 1734258658.14 total 16.65 last 648.80) - /unlink 16928 (time 1734258668.14 total 26.65 last 692.74) - /unlink 20000 (time 1734258671.93 total 30.44 last 810.18) - /unlink 27467 (time 1734258681.93 total 40.44 last 746.67) - /unlink 30000 (time 1734258685.59 total 44.10 last 692.09) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 48.71 seconds: 669.76 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 - open/close 2646 (time 1734258719.19 total 10.00 last 264.53) - open/close 5863 (time 1734258729.19 total 20.00 last 321.68) - open/close 9611 (time 1734258739.20 total 30.00 last 374.74) - open/close 13490 (time 1734258749.20 total 40.01 last 387.82) - open/close 17290 (time 1734258759.20 total 50.01 last 379.97) - open/close 20000 (time 1734258765.96 total 56.77 last 400.87) - open/close 23693 (time 1734258775.96 total 66.77 last 369.24) - open/close 27494 (time 1734258785.96 total 76.77 last 380.03) - open/close 30000 (time 1734258793.19 total 84.00 last 346.63) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 93.13 seconds: 350.33 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 6184 (time 1734258813.30 total 10.00 last 618.30) - /unlink 10000 (time 1734258819.87 total 16.56 last 581.41) - /unlink 17092 (time 1734258829.87 total 26.57 last 709.13) - /unlink 20000 (time 1734258834.33 total 31.03 last 652.06) - /unlink 24622 (time 1734258844.33 total 41.03 last 462.16) - /unlink 29762 (time 1734258854.33 total 51.03 last 513.99) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 55.90 seconds: 583.62 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 - open/close 3448 (time 1734258874.59 total 10.00 last 344.77) - open/close 7172 (time 1734258884.59 total 20.00 last 372.31) - open/close 10000 (time 1734258892.93 total 28.34 last 339.22) - open/close 13195 (time 1734258902.93 total 38.34 last 319.49) - open/close 16789 (time 1734258912.93 total 48.34 last 359.29) - open/close 19728 (time 1734258922.93 total 58.34 last 293.89) - open/close 24113 (time 1734258932.93 total 68.34 last 438.48) - open/close 28791 (time 1734258942.94 total 78.35 last 467.71) - open/close 30000 (time 1734258946.21 total 81.62 last 369.08) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 88.26 seconds: 369.68 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 7977 (time 1734258963.76 total 10.00 last 797.68) - /unlink 10000 (time 1734258966.74 total 12.98 last 678.76) - /unlink 16916 (time 1734258976.74 total 22.98 last 691.55) - /unlink 20000 (time 1734258981.30 total 27.54 last 676.52) - /unlink 26814 (time 1734258991.30 total 37.54 last 681.36) - /unlink 30000 (time 1734258995.87 total 42.11 last 697.55) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 45.76 seconds: 712.95 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 - open/close 3576 (time 1734259025.23 total 10.00 last 357.55) - open/close 7126 (time 1734259035.23 total 20.00 last 354.95) - open/close 10000 (time 1734259043.55 total 28.32 last 345.38) - open/close 13360 (time 1734259053.55 total 38.33 last 335.96) - open/close 16385 (time 1734259063.56 total 48.33 last 302.41) - open/close 20000 (time 1734259073.30 total 58.07 last 371.02) - open/close 23960 (time 1734259083.30 total 68.08 last 395.84) - open/close 27613 (time 1734259093.31 total 78.08 last 365.19) - open/close 30000 (time 1734259101.19 total 85.97 last 302.63) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 93.09 seconds: 350.49 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 7905 (time 1734259119.22 total 10.00 last 790.48) - /unlink 10000 (time 1734259121.69 total 12.47 last 848.49) - /unlink 18644 (time 1734259131.69 total 22.47 last 864.35) - /unlink 27842 (time 1734259141.69 total 32.47 last 919.78) - /unlink 30000 (time 1734259144.10 total 34.88 last 896.98) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 38.12 seconds: 855.90 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server unloading modules via unload_modules_local on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2