== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:26:10 (1734258370) Stopping clients: oleg414-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg414-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 oleg414-server: oleg414-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing load_modules_local oleg414-server: Loading modules from /home/green/git/lustre-release/lustre oleg414-server: detected 4 online CPUs by sysfs oleg414-server: Force libcfs to create 2 CPU partitions oleg414-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg414-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg414-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg414-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg414-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg414-server: oleg414-server.virtnet: executing set_default_debug -1 all pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg414-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg414-client.virtnet: -o user_xattr,flock 192.168.204.114@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3685 (time 1734258416.80 total 10.00 last 368.46) - open/close 6729 (time 1734258426.80 total 20.00 last 304.39) - open/close 10000 (time 1734258436.42 total 29.62 last 340.00) - open/close 12549 (time 1734258446.42 total 39.62 last 254.90) - open/close 14817 (time 1734258456.42 total 49.63 last 226.72) - open/close 17457 (time 1734258466.42 total 59.63 last 263.92) - open/close 20000 (time 1734258473.93 total 67.14 last 338.68) - open/close 23002 (time 1734258483.93 total 77.14 last 300.14) - open/close 26056 (time 1734258493.94 total 87.14 last 305.26) - open/close 28808 (time 1734258503.94 total 97.15 last 275.12) - open/close 30000 (time 1734258507.92 total 101.13 last 299.34) - open/close 32589 (time 1734258517.93 total 111.13 last 258.85) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 111.28 seconds: 293.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4944 (time 1734258529.18 total 10.00 last 494.38) - /unlink 10000 (time 1734258537.26 total 18.08 last 626.01) - /unlink 15340 (time 1734258547.26 total 28.08 last 533.94) - /unlink 20000 (time 1734258555.83 total 36.65 last 543.57) - /unlink 25182 (time 1734258565.83 total 46.65 last 518.20) - /unlink 30000 (time 1734258571.75 total 52.57 last 814.50) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 55.65 seconds: 586.27 ops/second Waiting for MDT destroys to complete oleg414-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg414-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 - open/close 3074 (time 1734258595.39 total 10.00 last 307.39) - open/close 5702 (time 1734258605.40 total 20.00 last 262.72) - open/close 8134 (time 1734258615.40 total 30.01 last 243.12) - open/close 10000 (time 1734258623.32 total 37.93 last 235.59) - open/close 13196 (time 1734258633.32 total 47.93 last 319.52) - open/close 16128 (time 1734258643.32 total 57.93 last 293.16) - open/close 18798 (time 1734258653.32 total 67.93 last 266.99) - open/close 20000 (time 1734258657.61 total 72.21 last 280.76) - open/close 23064 (time 1734258667.61 total 82.21 last 306.34) - open/close 26484 (time 1734258677.61 total 92.22 last 341.93) - open/close 29392 (time 1734258687.61 total 102.22 last 290.69) - open/close 30000 (time 1734258690.09 total 104.70 last 245.56) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 114.04 seconds: 286.08 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4360 (time 1734258710.48 total 10.00 last 435.96) - /unlink 8918 (time 1734258720.48 total 20.00 last 455.80) - /unlink 14404 (time 1734258730.48 total 30.00 last 548.56) - /unlink 20000 (time 1734258738.92 total 38.45 last 662.63) - /unlink 27027 (time 1734258748.93 total 48.45 last 702.62) - /unlink 30000 (time 1734258753.26 total 52.78 last 685.56) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 56.54 seconds: 577.01 ops/second Waiting for MDT destroys to complete oleg414-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg414-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 - open/close 3212 (time 1734258778.87 total 10.00 last 321.15) - open/close 6951 (time 1734258788.87 total 20.00 last 373.81) - open/close 9763 (time 1734258798.87 total 30.00 last 281.18) - open/close 12893 (time 1734258808.87 total 40.01 last 312.96) - open/close 15883 (time 1734258818.87 total 50.01 last 298.94) - open/close 19590 (time 1734258828.87 total 60.01 last 370.66) - open/close 22653 (time 1734258838.87 total 70.01 last 306.28) - open/close 25148 (time 1734258848.87 total 80.01 last 249.49) - open/close 28123 (time 1734258858.88 total 90.01 last 297.49) - open/close 30000 (time 1734258864.97 total 96.11 last 307.88) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 103.24 seconds: 316.03 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 7528 (time 1734258883.11 total 10.00 last 752.79) - /unlink 10000 (time 1734258886.95 total 13.84 last 644.15) - /unlink 17176 (time 1734258896.95 total 23.84 last 717.29) - /unlink 20000 (time 1734258901.51 total 28.40 last 619.88) - /unlink 27966 (time 1734258911.51 total 38.40 last 796.49) - /unlink 30000 (time 1734258914.30 total 41.19 last 729.84) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 45.16 seconds: 722.39 ops/second Waiting for MDT destroys to complete oleg414-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg414-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 - open/close 4374 (time 1734258947.66 total 10.00 last 437.35) - open/close 8583 (time 1734258957.66 total 20.00 last 420.86) - open/close 10000 (time 1734258961.04 total 23.38 last 419.76) - open/close 14128 (time 1734258971.04 total 33.38 last 412.77) - open/close 18042 (time 1734258981.04 total 43.38 last 391.36) - open/close 20000 (time 1734258986.56 total 48.89 last 355.01) - open/close 23502 (time 1734258996.56 total 58.90 last 350.19) - open/close 27219 (time 1734259006.57 total 68.90 last 371.37) - open/close 30000 (time 1734259013.33 total 75.66 last 411.46) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 83.20 seconds: 392.14 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 7628 (time 1734259031.91 total 10.00 last 762.71) - /unlink 10000 (time 1734259035.56 total 13.65 last 650.22) - /unlink 16717 (time 1734259045.56 total 23.65 last 671.56) - /unlink 20000 (time 1734259050.38 total 28.47 last 681.93) - /unlink 26548 (time 1734259060.38 total 38.47 last 654.73) - /unlink 30000 (time 1734259064.97 total 43.06 last 751.73) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 46.38 seconds: 703.52 ops/second Waiting for MDT destroys to complete oleg414-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg414-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 - open/close 3993 (time 1734259091.48 total 10.00 last 399.18) - open/close 7394 (time 1734259101.48 total 20.00 last 340.07) - open/close 10000 (time 1734259107.62 total 26.14 last 424.75) - open/close 14441 (time 1734259117.62 total 36.14 last 444.08) - open/close 18974 (time 1734259127.62 total 46.14 last 453.28) - open/close 20000 (time 1734259129.73 total 48.25 last 486.25) - open/close 24834 (time 1734259139.73 total 58.25 last 483.34) - open/close 29276 (time 1734259149.73 total 68.25 last 444.18) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 74.63 seconds: 437.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 9246 (time 1734259167.08 total 10.00 last 924.54) - /unlink 19100 (time 1734259177.08 total 20.00 last 985.31) - /unlink 28585 (time 1734259187.08 total 30.00 last 948.48) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 33.78 seconds: 965.76 ops/second Waiting for MDT destroys to complete oleg414-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg414-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg414-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg414-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg414-server stop mds service on oleg414-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg414-server unloading modules via unload_modules_local on: 'oleg414-server' oleg414-server: oleg414-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-client: ssh exited with exit code 2 pdsh@oleg414-client: oleg414-server: ssh exited with exit code 2