== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:29:15 (1734258555) Stopping clients: oleg301-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg301-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 oleg301-server: oleg301-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing load_modules_local oleg301-server: Loading modules from /home/green/git/lustre-release/lustre oleg301-server: detected 4 online CPUs by sysfs oleg301-server: Force libcfs to create 2 CPU partitions oleg301-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg301-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg301-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg301-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 272 79720 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 302 32594 1% /mnt/lustre[OST:0] filesystem_summary: 32866 272 32594 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1654 (time 1734258606.80 total 10.00 last 165.38) - open/close 3318 (time 1734258616.80 total 20.01 last 166.31) - open/close 5276 (time 1734258626.81 total 30.01 last 195.73) - open/close 7081 (time 1734258636.81 total 40.02 last 180.39) - open/close 8591 (time 1734258646.82 total 50.02 last 150.96) - open/close 10000 (time 1734258656.39 total 59.59 last 147.16) - open/close 11646 (time 1734258666.40 total 69.60 last 164.51) - open/close 13037 (time 1734258676.40 total 79.60 last 139.08) - open/close 14525 (time 1734258686.40 total 89.60 last 148.79) - open/close 16094 (time 1734258696.41 total 99.61 last 156.76) - open/close 17587 (time 1734258706.41 total 109.61 last 149.26) - open/close 19066 (time 1734258716.41 total 119.61 last 147.88) - open/close 20000 (time 1734258722.32 total 125.52 last 158.17) - open/close 21688 (time 1734258732.32 total 135.52 last 168.77) - open/close 23359 (time 1734258742.32 total 145.53 last 166.99) - open/close 25113 (time 1734258752.33 total 155.53 last 175.32) - open/close 26775 (time 1734258762.33 total 165.54 last 166.14) - open/close 28595 (time 1734258772.33 total 175.54 last 181.96) - open/close 30000 (time 1734258781.53 total 184.73 last 152.79) - open/close 31496 (time 1734258791.54 total 194.74 last 149.52) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 202.42 seconds: 161.18 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 2955 (time 1734258810.99 total 10.00 last 295.47) - /unlink 6450 (time 1734258820.99 total 20.00 last 349.50) - /unlink 10000 (time 1734258830.46 total 29.48 last 374.56) - /unlink 13821 (time 1734258840.47 total 39.48 last 382.09) - /unlink 17405 (time 1734258850.47 total 49.48 last 358.34) - /unlink 20000 (time 1734258856.78 total 55.80 last 410.91) - /unlink 23816 (time 1734258866.78 total 65.80 last 381.60) - /unlink 27056 (time 1734258876.78 total 75.80 last 323.99) - /unlink 30000 (time 1734258886.26 total 85.27 last 310.62) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 93.94 seconds: 347.32 ops/second Waiting for MDT destroys to complete oleg301-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg301-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 - open/close 1608 (time 1734258912.29 total 10.00 last 160.76) - open/close 3118 (time 1734258922.29 total 20.01 last 150.94) - open/close 4672 (time 1734258932.29 total 30.01 last 155.36) - open/close 6228 (time 1734258942.30 total 40.02 last 155.47) - open/close 7802 (time 1734258952.31 total 50.02 last 157.32) - open/close 9374 (time 1734258962.31 total 60.03 last 157.12) - open/close 10000 (time 1734258966.20 total 63.91 last 161.13) - open/close 11570 (time 1734258976.20 total 73.92 last 156.95) - open/close 13248 (time 1734258986.20 total 83.92 last 167.76) - open/close 15077 (time 1734258996.20 total 93.92 last 182.86) - open/close 17007 (time 1734259006.21 total 103.92 last 192.94) - open/close 18908 (time 1734259016.21 total 113.93 last 190.02) - open/close 20000 (time 1734259022.31 total 120.02 last 179.14) - open/close 21646 (time 1734259032.31 total 130.03 last 164.50) - open/close 23269 (time 1734259042.32 total 140.03 last 162.21) - open/close 24881 (time 1734259052.32 total 150.04 last 161.11) - open/close 26527 (time 1734259062.33 total 160.05 last 164.50) - open/close 28039 (time 1734259072.33 total 170.05 last 151.14) - open/close 29530 (time 1734259082.34 total 180.06 last 149.01) - open/close 30000 (time 1734259085.36 total 183.08 last 155.57) - open/close 31493 (time 1734259095.36 total 193.08 last 149.23) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 200.67 seconds: 162.58 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3021 (time 1734259114.57 total 10.00 last 302.07) - /unlink 6579 (time 1734259124.57 total 20.00 last 355.74) - /unlink 9551 (time 1734259134.58 total 30.00 last 297.14) - /unlink 12476 (time 1734259144.58 total 40.00 last 292.49) - /unlink 15502 (time 1734259154.58 total 50.01 last 302.56) - /unlink 18873 (time 1734259164.58 total 60.01 last 337.05) - /unlink 20000 (time 1734259167.73 total 63.16 last 357.13) - /unlink 23494 (time 1734259177.74 total 73.17 last 349.33) - /unlink 26692 (time 1734259187.74 total 83.17 last 319.72) - /unlink 29794 (time 1734259197.74 total 93.17 last 310.15) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 101.59 seconds: 321.15 ops/second Waiting for MDT destroys to complete oleg301-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 oleg301-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 - open/close 1684 (time 1734259226.25 total 10.01 last 168.27) - open/close 3397 (time 1734259236.25 total 20.01 last 171.23) - open/close 5419 (time 1734259246.25 total 30.01 last 202.18) - open/close 7676 (time 1734259256.25 total 40.01 last 225.66) - open/close 9781 (time 1734259266.26 total 50.02 last 210.44) - open/close 11428 (time 1734259276.26 total 60.02 last 164.66) - open/close 13396 (time 1734259286.26 total 70.02 last 196.77) - open/close 15248 (time 1734259296.26 total 80.02 last 185.17) - open/close 17066 (time 1734259306.26 total 90.03 last 181.74) - open/close 18919 (time 1734259316.27 total 100.03 last 185.25) - open/close 20000 (time 1734259322.46 total 106.22 last 174.52) - open/close 22240 (time 1734259332.46 total 116.23 last 223.92) - open/close 24307 (time 1734259342.47 total 126.23 last 206.65) - open/close 26090 (time 1734259352.47 total 136.24 last 178.20) - open/close 27757 (time 1734259362.48 total 146.24 last 166.64) - open/close 29470 (time 1734259372.48 total 156.24 last 171.25) - open/close 30000 (time 1734259375.33 total 159.09 last 186.07) - open/close 31703 (time 1734259385.33 total 169.09 last 170.24) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 174.36 seconds: 187.12 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 3496 (time 1734259401.91 total 10.00 last 349.57) - /unlink 6813 (time 1734259411.92 total 20.00 last 331.66) - /unlink 10000 (time 1734259421.65 total 29.74 last 327.38) - /unlink 13363 (time 1734259431.66 total 39.74 last 336.11) - /unlink 16746 (time 1734259441.66 total 49.74 last 338.29) - /unlink 20000 (time 1734259450.71 total 58.80 last 359.39) - /unlink 23227 (time 1734259460.71 total 68.80 last 322.69) - /unlink 26660 (time 1734259470.71 total 78.80 last 343.21) - /unlink 30000 (time 1734259479.87 total 87.95 last 364.87) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 94.29 seconds: 346.02 ops/second Waiting for MDT destroys to complete oleg301-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 oleg301-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 - open/close 1871 (time 1734259502.87 total 10.00 last 187.05) - open/close 3635 (time 1734259512.87 total 20.01 last 176.34) - open/close 5768 (time 1734259522.87 total 30.01 last 213.25) - open/close 8285 (time 1734259532.87 total 40.01 last 251.64) - open/close 10000 (time 1734259540.64 total 47.77 last 220.94) - open/close 11992 (time 1734259550.64 total 57.78 last 199.14) - open/close 14122 (time 1734259560.64 total 67.78 last 212.95) - open/close 16057 (time 1734259570.64 total 77.78 last 193.47) - open/close 17982 (time 1734259580.65 total 87.78 last 192.42) - open/close 20000 (time 1734259589.28 total 96.42 last 233.77) - open/close 22271 (time 1734259599.28 total 106.42 last 227.06) - open/close 25029 (time 1734259609.28 total 116.42 last 275.78) - open/close 27599 (time 1734259619.29 total 126.42 last 256.86) - open/close 29950 (time 1734259629.29 total 136.43 last 235.05) - open/close 32282 (time 1734259639.29 total 146.43 last 233.15) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 147.94 seconds: 220.54 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 4509 (time 1734259651.89 total 10.00 last 450.86) - /unlink 9083 (time 1734259661.89 total 20.00 last 457.30) - /unlink 10000 (time 1734259663.98 total 22.09 last 439.51) - /unlink 14503 (time 1734259673.98 total 32.09 last 450.27) - /unlink 19089 (time 1734259683.98 total 42.09 last 458.50) - /unlink 23838 (time 1734259693.98 total 52.09 last 474.89) - /unlink 29224 (time 1734259703.98 total 62.09 last 538.57) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 68.84 seconds: 473.92 ops/second Waiting for MDT destroys to complete oleg301-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 oleg301-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 - open/close 2360 (time 1734259725.13 total 10.00 last 235.96) - open/close 4450 (time 1734259735.13 total 20.00 last 208.95) - open/close 6709 (time 1734259745.13 total 30.01 last 225.82) - open/close 8694 (time 1734259755.14 total 40.01 last 198.38) - open/close 10000 (time 1734259761.07 total 45.95 last 220.10) - open/close 12443 (time 1734259771.08 total 55.95 last 244.22) - open/close 14978 (time 1734259781.08 total 65.96 last 253.38) - open/close 17632 (time 1734259791.08 total 75.96 last 265.33) - open/close 19897 (time 1734259801.09 total 85.96 last 226.44) - open/close 22045 (time 1734259811.09 total 95.96 last 214.72) - open/close 24442 (time 1734259821.09 total 105.97 last 239.67) - open/close 27284 (time 1734259831.09 total 115.97 last 284.13) - open/close 30000 (time 1734259840.93 total 125.81 last 276.11) open(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No space left on device total: 32626 open/close in 135.01 seconds: 241.66 ops/second UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 32899 47093 42% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32896 32896 0 100% /mnt/lustre[OST:0] filesystem_summary: 32899 32899 0 100% /mnt/lustre - /unlink 5314 (time 1734259861.19 total 10.00 last 531.33) - /unlink 10000 (time 1734259871.07 total 19.89 last 474.01) - /unlink 14717 (time 1734259881.07 total 29.89 last 471.69) - /unlink 19120 (time 1734259891.07 total 39.89 last 440.18) - /unlink 20000 (time 1734259893.16 total 41.97 last 422.94) - /unlink 24254 (time 1734259903.16 total 51.97 last 425.40) - /unlink 28962 (time 1734259913.16 total 61.97 last 470.75) - /unlink 30000 (time 1734259915.17 total 63.98 last 516.03) unlink(/mnt/lustre/d106.conf-sanity-1/f-32626) error: No such file or directory total: 32626 /unlink in 69.02 seconds: 472.69 ops/second Waiting for MDT destroys to complete oleg301-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 oleg301-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2