== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:30:13 (1734258613) Stopping clients: oleg215-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg215-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 oleg215-server: oleg215-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing load_modules_local oleg215-server: Loading modules from /home/green/git/lustre-release/lustre oleg215-server: detected 4 online CPUs by sysfs oleg215-server: Force libcfs to create 2 CPU partitions oleg215-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg215-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg215-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg215-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg215-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg215-server: oleg215-server.virtnet: executing set_default_debug -1 all pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg215-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg215-client.virtnet: -o user_xattr,flock 192.168.202.115@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1321 (time 1734258673.45 total 10.00 last 132.06) - open/close 3075 (time 1734258683.45 total 20.01 last 175.30) - open/close 4670 (time 1734258693.46 total 30.01 last 159.47) - open/close 6352 (time 1734258703.46 total 40.02 last 168.10) - open/close 8233 (time 1734258713.46 total 50.02 last 188.06) - open/close 9715 (time 1734258723.47 total 60.02 last 148.15) - open/close 10000 (time 1734258725.50 total 62.05 last 140.25) - open/close 11560 (time 1734258735.50 total 72.06 last 155.96) - open/close 12963 (time 1734258745.50 total 82.06 last 140.28) - open/close 14607 (time 1734258755.51 total 92.06 last 164.35) - open/close 16586 (time 1734258765.51 total 102.06 last 197.85) - open/close 18424 (time 1734258775.51 total 112.07 last 183.73) - open/close 20000 (time 1734258784.11 total 120.66 last 183.39) - open/close 21893 (time 1734258794.11 total 130.66 last 189.24) - open/close 23809 (time 1734258804.11 total 140.67 last 191.54) - open/close 25236 (time 1734258814.12 total 150.67 last 142.64) - open/close 26855 (time 1734258824.12 total 160.67 last 161.85) - open/close 28572 (time 1734258834.12 total 170.68 last 171.65) - open/close 30000 (time 1734258843.31 total 179.86 last 155.43) - open/close 31970 (time 1734258853.31 total 189.87 last 196.92) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 193.15 seconds: 168.92 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4621 (time 1734258867.92 total 10.00 last 462.06) - /unlink 9467 (time 1734258877.92 total 20.00 last 484.48) - /unlink 13081 (time 1734258887.92 total 30.01 last 361.32) - /unlink 17148 (time 1734258897.92 total 40.01 last 406.69) - /unlink 20000 (time 1734258906.87 total 48.95 last 318.72) - /unlink 23271 (time 1734258916.87 total 58.96 last 327.07) - /unlink 26295 (time 1734258926.87 total 68.96 last 302.34) - /unlink 30000 (time 1734258936.25 total 78.33 last 395.24) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 85.04 seconds: 383.68 ops/second Waiting for MDT destroys to complete oleg215-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg215-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 - open/close 2495 (time 1734258961.97 total 10.00 last 249.47) - open/close 4440 (time 1734258971.98 total 20.01 last 194.24) - open/close 6575 (time 1734258981.98 total 30.02 last 213.42) - open/close 8595 (time 1734258991.98 total 40.02 last 201.98) - open/close 10000 (time 1734258998.01 total 46.04 last 233.19) - open/close 12360 (time 1734259008.01 total 56.05 last 235.95) - open/close 14559 (time 1734259018.01 total 66.05 last 219.90) - open/close 16329 (time 1734259028.01 total 76.05 last 177.00) - open/close 18116 (time 1734259038.02 total 86.05 last 178.65) - open/close 20000 (time 1734259047.71 total 95.74 last 194.32) - open/close 21894 (time 1734259057.71 total 105.74 last 189.39) - open/close 23875 (time 1734259067.71 total 115.75 last 198.06) - open/close 26015 (time 1734259077.71 total 125.75 last 213.96) - open/close 27756 (time 1734259087.72 total 135.75 last 174.06) - open/close 29599 (time 1734259097.72 total 145.75 last 184.29) - open/close 31386 (time 1734259107.72 total 155.76 last 178.61) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 161.66 seconds: 201.82 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4250 (time 1734259124.77 total 10.00 last 424.95) - /unlink 7610 (time 1734259134.77 total 20.00 last 335.95) - /unlink 10000 (time 1734259141.41 total 26.64 last 360.20) - /unlink 13806 (time 1734259151.41 total 36.64 last 380.56) - /unlink 17436 (time 1734259161.41 total 46.64 last 362.99) - /unlink 20000 (time 1734259167.55 total 52.78 last 417.87) - /unlink 23648 (time 1734259177.55 total 62.78 last 364.72) - /unlink 27130 (time 1734259187.55 total 72.78 last 348.15) - /unlink 30000 (time 1734259193.88 total 79.11 last 453.44) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 86.93 seconds: 375.33 ops/second Waiting for MDT destroys to complete oleg215-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg215-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 - open/close 2183 (time 1734259222.37 total 10.00 last 218.28) - open/close 3843 (time 1734259232.37 total 20.00 last 165.98) - open/close 5753 (time 1734259242.37 total 30.00 last 190.99) - open/close 7791 (time 1734259252.37 total 40.00 last 203.77) - open/close 9827 (time 1734259262.37 total 50.00 last 203.58) - open/close 11959 (time 1734259272.37 total 60.01 last 213.15) - open/close 14457 (time 1734259282.37 total 70.01 last 249.77) - open/close 17010 (time 1734259292.37 total 80.01 last 255.25) - open/close 18996 (time 1734259302.38 total 90.01 last 198.54) - open/close 20000 (time 1734259307.32 total 94.95 last 203.19) - open/close 21840 (time 1734259317.33 total 104.96 last 183.84) - open/close 23748 (time 1734259327.33 total 114.96 last 190.78) - open/close 25747 (time 1734259337.33 total 124.97 last 199.87) - open/close 27920 (time 1734259347.33 total 134.97 last 217.23) - open/close 30000 (time 1734259356.75 total 144.39 last 220.90) - open/close 32278 (time 1734259366.75 total 154.39 last 227.77) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 156.21 seconds: 208.85 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3825 (time 1734259379.92 total 10.00 last 382.39) - /unlink 7888 (time 1734259389.92 total 20.00 last 406.30) - /unlink 10000 (time 1734259395.42 total 25.50 last 384.31) - /unlink 13959 (time 1734259405.42 total 35.50 last 395.86) - /unlink 18090 (time 1734259415.42 total 45.50 last 412.98) - /unlink 20000 (time 1734259420.51 total 50.59 last 375.37) - /unlink 23504 (time 1734259430.51 total 60.59 last 350.39) - /unlink 27437 (time 1734259440.51 total 70.59 last 393.23) - /unlink 30000 (time 1734259448.23 total 78.31 last 332.04) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 86.32 seconds: 377.96 ops/second Waiting for MDT destroys to complete oleg215-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg215-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 - open/close 2270 (time 1734259476.88 total 10.00 last 226.89) - open/close 4201 (time 1734259486.88 total 20.01 last 193.08) - open/close 6221 (time 1734259496.88 total 30.01 last 201.97) - open/close 8060 (time 1734259506.88 total 40.01 last 183.81) - open/close 10000 (time 1734259516.06 total 49.19 last 211.36) - open/close 12170 (time 1734259526.07 total 59.20 last 216.84) - open/close 13951 (time 1734259536.07 total 69.20 last 178.06) - open/close 15863 (time 1734259546.08 total 79.20 last 191.14) - open/close 17788 (time 1734259556.08 total 89.21 last 192.46) - open/close 19891 (time 1734259566.08 total 99.21 last 210.24) - open/close 21947 (time 1734259576.08 total 109.21 last 205.58) - open/close 23947 (time 1734259586.08 total 119.21 last 199.97) - open/close 26223 (time 1734259596.09 total 129.21 last 227.56) - open/close 28460 (time 1734259606.09 total 139.21 last 223.67) - open/close 30000 (time 1734259612.94 total 146.06 last 224.82) - open/close 32263 (time 1734259622.94 total 156.07 last 226.27) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 157.77 seconds: 206.79 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5025 (time 1734259635.80 total 10.00 last 502.46) - /unlink 10000 (time 1734259645.04 total 19.23 last 538.79) - /unlink 14903 (time 1734259655.04 total 29.24 last 490.24) - /unlink 19254 (time 1734259665.04 total 39.24 last 435.07) - /unlink 23260 (time 1734259675.04 total 49.24 last 400.51) - /unlink 28181 (time 1734259685.04 total 59.24 last 492.09) - /unlink 30000 (time 1734259689.38 total 63.57 last 419.68) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 68.59 seconds: 475.64 ops/second Waiting for MDT destroys to complete oleg215-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg215-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 - open/close 2790 (time 1734259712.77 total 10.00 last 278.98) - open/close 5031 (time 1734259722.77 total 20.00 last 224.06) - open/close 7668 (time 1734259732.77 total 30.00 last 263.68) - open/close 9560 (time 1734259742.77 total 40.00 last 189.18) - open/close 10000 (time 1734259744.84 total 42.07 last 213.00) - open/close 12400 (time 1734259754.84 total 52.07 last 239.97) - open/close 14375 (time 1734259764.84 total 62.07 last 197.44) - open/close 16240 (time 1734259774.85 total 72.08 last 186.42) - open/close 18203 (time 1734259784.85 total 82.09 last 196.16) - open/close 20000 (time 1734259793.42 total 90.65 last 209.88) - open/close 22407 (time 1734259803.42 total 100.65 last 240.67) - open/close 24995 (time 1734259813.42 total 110.65 last 258.75) - open/close 27653 (time 1734259823.42 total 120.65 last 265.80) - open/close 30000 (time 1734259832.92 total 130.16 last 246.93) - open/close 32398 (time 1734259842.93 total 140.16 last 239.75) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 141.03 seconds: 231.33 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4228 (time 1734259854.88 total 10.00 last 422.78) - /unlink 8170 (time 1734259864.88 total 20.00 last 394.15) - /unlink 10000 (time 1734259868.99 total 24.11 last 445.40) - /unlink 14078 (time 1734259878.99 total 34.11 last 407.72) - /unlink 18395 (time 1734259888.99 total 44.11 last 431.67) - /unlink 20000 (time 1734259891.92 total 47.05 last 546.92) - /unlink 24653 (time 1734259901.92 total 57.05 last 465.26) - /unlink 29374 (time 1734259911.93 total 67.05 last 471.99) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 73.81 seconds: 442.05 ops/second Waiting for MDT destroys to complete oleg215-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg215-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg215-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg215-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg215-server stop mds service on oleg215-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg215-server unloading modules via unload_modules_local on: 'oleg215-server' oleg215-server: oleg215-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-client: ssh exited with exit code 2 pdsh@oleg215-client: oleg215-server: ssh exited with exit code 2