== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:36:17 (1734258977) Stopping clients: oleg232-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg232-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 oleg232-server: oleg232-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing load_modules_local oleg232-server: Loading modules from /home/green/git/lustre-release/lustre oleg232-server: detected 4 online CPUs by sysfs oleg232-server: Force libcfs to create 2 CPU partitions oleg232-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg232-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg232-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg232-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg232-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg232-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg232-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg232-server: oleg232-server.virtnet: executing set_default_debug -1 all pdsh@oleg232-client: oleg232-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg232-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg232-client.virtnet: -o user_xattr,flock 192.168.202.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16483 643 15840 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1654 (time 1734259037.99 total 10.00 last 165.36) - open/close 3477 (time 1734259047.99 total 20.00 last 182.28) - open/close 5184 (time 1734259057.99 total 30.01 last 170.64) - open/close 6936 (time 1734259068.00 total 40.01 last 175.12) - open/close 8759 (time 1734259078.00 total 50.01 last 182.24) - open/close 10000 (time 1734259085.67 total 57.68 last 161.84) - open/close 11631 (time 1734259095.67 total 67.68 last 163.06) - open/close 13286 (time 1734259105.67 total 77.69 last 165.48) - open/close 15018 (time 1734259115.67 total 87.69 last 173.17) - open/close 16939 (time 1734259125.67 total 97.69 last 192.10) - open/close 18614 (time 1734259135.67 total 107.69 last 167.48) - open/close 20000 (time 1734259143.77 total 115.79 last 171.18) - open/close 21779 (time 1734259153.77 total 125.79 last 177.87) - open/close 23618 (time 1734259163.77 total 135.79 last 183.90) - open/close 25479 (time 1734259173.78 total 145.79 last 186.06) - open/close 27115 (time 1734259183.78 total 155.79 last 163.58) - open/close 29086 (time 1734259193.78 total 165.80 last 196.98) - open/close 30000 (time 1734259199.10 total 171.11 last 171.97) - open/close 31679 (time 1734259209.10 total 181.11 last 167.88) - open/close 33874 (time 1734259219.10 total 191.11 last 219.48) - open/close 35555 (time 1734259229.10 total 201.12 last 168.03) - open/close 37254 (time 1734259239.11 total 211.12 last 169.84) - open/close 39131 (time 1734259249.11 total 221.12 last 187.64) - open/close 40000 (time 1734259253.45 total 225.46 last 200.20) - open/close 41884 (time 1734259263.45 total 235.47 last 188.34) - open/close 43822 (time 1734259273.45 total 245.47 last 193.79) - open/close 46188 (time 1734259283.46 total 255.47 last 236.52) - open/close 48393 (time 1734259293.46 total 265.47 last 220.48) - open/close 50000 (time 1734259301.36 total 273.38 last 203.28) - open/close 51806 (time 1734259311.38 total 283.39 last 180.36) - open/close 53393 (time 1734259321.38 total 293.39 last 158.66) - open/close 55234 (time 1734259331.38 total 303.40 last 184.01) - open/close 57174 (time 1734259341.39 total 313.40 last 193.89) - open/close 59224 (time 1734259351.39 total 323.41 last 204.95) - open/close 60000 (time 1734259354.76 total 326.77 last 230.47) - open/close 62141 (time 1734259364.76 total 336.78 last 213.99) - open/close 63972 (time 1734259374.77 total 346.78 last 183.08) total: 64768 open/close in 350.91 seconds: 184.57 ops/second - /unlink 3237 (time 1734259390.10 total 10.00 last 323.68) - /unlink 6536 (time 1734259400.10 total 20.00 last 329.85) - /unlink 9723 (time 1734259410.10 total 30.00 last 318.68) - /unlink 12802 (time 1734259420.10 total 40.00 last 307.87) - /unlink 15759 (time 1734259430.10 total 50.00 last 295.70) - /unlink 18853 (time 1734259440.11 total 60.01 last 309.31) - /unlink 20000 (time 1734259444.34 total 64.24 last 271.13) - /unlink 22646 (time 1734259454.34 total 74.24 last 264.51) - /unlink 25423 (time 1734259464.34 total 84.24 last 277.62) - /unlink 28784 (time 1734259474.35 total 94.25 last 336.05) - /unlink 30000 (time 1734259477.90 total 97.80 last 342.42) - /unlink 33197 (time 1734259487.90 total 107.80 last 319.61) - /unlink 36889 (time 1734259497.90 total 117.80 last 369.10) - /unlink 40000 (time 1734259507.78 total 127.68 last 314.84) - /unlink 43857 (time 1734259517.79 total 137.69 last 385.62) - /unlink 47322 (time 1734259527.79 total 147.69 last 346.47) - /unlink 50000 (time 1734259536.08 total 155.98 last 322.97) - /unlink 53339 (time 1734259546.08 total 165.98 last 333.89) - /unlink 56794 (time 1734259556.08 total 175.98 last 345.42) - /unlink 60000 (time 1734259565.00 total 184.90 last 359.47) - /unlink 63710 (time 1734259575.00 total 194.90 last 370.97) total: 64768 /unlink in 198.05 seconds: 327.03 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2419 (time 1734259605.22 total 10.00 last 241.82) - open/close 4685 (time 1734259615.22 total 20.01 last 226.46) - open/close 7011 (time 1734259625.23 total 30.01 last 232.52) - open/close 9727 (time 1734259635.23 total 40.01 last 271.58) - open/close 12457 (time 1734259645.23 total 50.02 last 272.94) - open/close 15233 (time 1734259655.23 total 60.02 last 277.59) - open/close 17607 (time 1734259665.23 total 70.02 last 237.34) - open/close 20000 (time 1734259675.21 total 79.99 last 239.90) - open/close 22526 (time 1734259685.21 total 90.00 last 252.54) - open/close 25190 (time 1734259695.21 total 100.00 last 266.39) - open/close 28137 (time 1734259705.21 total 110.00 last 294.57) - open/close 30000 (time 1734259712.21 total 117.00 last 266.26) - open/close 32346 (time 1734259722.21 total 127.00 last 234.58) - open/close 34947 (time 1734259732.21 total 137.00 last 260.08) - open/close 37220 (time 1734259742.22 total 147.00 last 227.18) - open/close 39628 (time 1734259752.28 total 157.06 last 239.40) - open/close 42108 (time 1734259762.28 total 167.07 last 247.95) - open/close 44194 (time 1734259772.28 total 177.07 last 208.54) - open/close 46331 (time 1734259782.28 total 187.07 last 213.70) - open/close 48679 (time 1734259792.29 total 197.07 last 234.69) - open/close 50000 (time 1734259797.98 total 202.77 last 231.97) - open/close 52826 (time 1734259807.98 total 212.77 last 282.60) - open/close 55716 (time 1734259817.98 total 222.77 last 288.94) - open/close 58382 (time 1734259827.99 total 232.77 last 266.54) - open/close 60000 (time 1734259833.68 total 238.46 last 284.36) - open/close 62645 (time 1734259843.68 total 248.46 last 264.46) total: 64768 open/close in 256.33 seconds: 252.68 ops/second - /unlink 3685 (time 1734259862.64 total 10.00 last 368.50) - /unlink 8057 (time 1734259872.64 total 20.00 last 437.13) - /unlink 10000 (time 1734259877.49 total 24.85 last 400.89) - /unlink 14200 (time 1734259887.49 total 34.85 last 419.91) - /unlink 18719 (time 1734259897.49 total 44.85 last 451.85) - /unlink 20000 (time 1734259900.56 total 47.92 last 417.62) - /unlink 24639 (time 1734259910.56 total 57.92 last 463.82) - /unlink 28941 (time 1734259920.56 total 67.92 last 430.18) - /unlink 30000 (time 1734259922.86 total 70.22 last 460.85) - /unlink 35166 (time 1734259932.86 total 80.22 last 516.56) - /unlink 40000 (time 1734259942.48 total 89.84 last 502.58) - /unlink 45230 (time 1734259952.48 total 99.84 last 522.95) - /unlink 50000 (time 1734259962.42 total 109.78 last 479.87) - /unlink 55274 (time 1734259972.42 total 119.78 last 527.33) - /unlink 60000 (time 1734259980.74 total 128.10 last 567.95) total: 64768 /unlink in 136.13 seconds: 475.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3044 (time 1734260015.20 total 10.00 last 304.40) - open/close 6066 (time 1734260025.20 total 20.00 last 302.19) - open/close 9139 (time 1734260035.20 total 30.00 last 307.21) - open/close 10000 (time 1734260037.96 total 32.76 last 312.23) - open/close 12718 (time 1734260047.96 total 42.76 last 271.74) - open/close 15293 (time 1734260057.96 total 52.77 last 257.45) - open/close 18511 (time 1734260067.97 total 62.77 last 321.73) - open/close 20000 (time 1734260072.71 total 67.51 last 313.97) - open/close 22816 (time 1734260082.71 total 77.51 last 281.52) - open/close 25596 (time 1734260092.71 total 87.51 last 277.94) - open/close 28238 (time 1734260102.72 total 97.52 last 264.10) - open/close 30000 (time 1734260109.28 total 104.08 last 268.60) - open/close 33012 (time 1734260119.28 total 114.08 last 301.13) - open/close 35797 (time 1734260129.28 total 124.08 last 278.49) - open/close 38426 (time 1734260139.28 total 134.08 last 262.86) - open/close 40000 (time 1734260144.89 total 139.69 last 280.62) - open/close 42829 (time 1734260154.89 total 149.69 last 282.81) - open/close 45726 (time 1734260164.90 total 159.70 last 289.67) - open/close 48393 (time 1734260174.90 total 169.70 last 266.63) - open/close 50000 (time 1734260180.62 total 175.42 last 280.66) - open/close 52325 (time 1734260190.63 total 185.43 last 232.46) - open/close 54770 (time 1734260200.63 total 195.43 last 244.45) - open/close 57477 (time 1734260210.63 total 205.43 last 270.62) - open/close 59924 (time 1734260220.63 total 215.43 last 244.64) - open/close 62213 (time 1734260230.64 total 225.44 last 228.77) total: 64768 open/close in 235.20 seconds: 275.37 ops/second - /unlink 4605 (time 1734260251.47 total 10.00 last 460.46) - /unlink 9089 (time 1734260261.47 total 20.00 last 448.40) - /unlink 13959 (time 1734260271.47 total 30.00 last 486.93) - /unlink 19047 (time 1734260281.47 total 40.00 last 508.77) - /unlink 20000 (time 1734260283.93 total 42.46 last 387.27) - /unlink 24417 (time 1734260293.93 total 52.47 last 441.64) - /unlink 28340 (time 1734260303.93 total 62.47 last 392.26) - /unlink 30000 (time 1734260308.84 total 67.38 last 338.00) - /unlink 33484 (time 1734260318.84 total 77.38 last 348.36) - /unlink 37186 (time 1734260328.85 total 87.38 last 370.12) - /unlink 40000 (time 1734260336.24 total 94.78 last 380.51) - /unlink 43250 (time 1734260346.24 total 104.78 last 324.96) - /unlink 47512 (time 1734260356.24 total 114.78 last 426.15) - /unlink 50000 (time 1734260361.96 total 120.49 last 435.34) - /unlink 54610 (time 1734260371.96 total 130.49 last 460.96) - /unlink 58952 (time 1734260381.96 total 140.50 last 434.11) - /unlink 60000 (time 1734260384.39 total 142.92 last 432.12) - /unlink 64644 (time 1734260394.39 total 152.92 last 464.35) total: 64768 /unlink in 153.24 seconds: 422.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2738 (time 1734260421.25 total 10.00 last 273.76) - open/close 5581 (time 1734260431.26 total 20.01 last 284.09) - open/close 7905 (time 1734260441.26 total 30.01 last 232.31) - open/close 10000 (time 1734260448.71 total 37.46 last 281.33) - open/close 13419 (time 1734260458.71 total 47.46 last 341.84) - open/close 16536 (time 1734260468.71 total 57.46 last 311.67) - open/close 19298 (time 1734260478.71 total 67.46 last 276.16) - open/close 20000 (time 1734260481.51 total 70.26 last 250.71) - open/close 22426 (time 1734260491.51 total 80.26 last 242.57) - open/close 25009 (time 1734260501.51 total 90.26 last 258.30) - open/close 27985 (time 1734260511.52 total 100.27 last 297.58) - open/close 30000 (time 1734260519.15 total 107.90 last 263.76) - open/close 32665 (time 1734260529.16 total 117.91 last 266.42) - open/close 35380 (time 1734260539.16 total 127.91 last 271.49) - open/close 37980 (time 1734260549.16 total 137.91 last 259.91) - open/close 40000 (time 1734260557.04 total 145.79 last 256.48) - open/close 41969 (time 1734260567.04 total 155.79 last 196.83) - open/close 44264 (time 1734260577.04 total 165.79 last 229.44) - open/close 46286 (time 1734260587.05 total 175.80 last 202.13) - open/close 48765 (time 1734260597.05 total 185.80 last 247.84) - open/close 50000 (time 1734260601.29 total 190.04 last 290.99) - open/close 52232 (time 1734260611.30 total 200.05 last 223.08) - open/close 54525 (time 1734260621.30 total 210.05 last 229.30) - open/close 57104 (time 1734260631.30 total 220.05 last 257.80) - open/close 59932 (time 1734260641.30 total 230.05 last 282.79) - open/close 62381 (time 1734260651.31 total 240.06 last 244.82) total: 64768 open/close in 249.16 seconds: 259.95 ops/second - /unlink 4547 (time 1734260671.39 total 10.00 last 454.61) - /unlink 9354 (time 1734260681.39 total 20.00 last 480.66) - /unlink 13469 (time 1734260691.39 total 30.01 last 411.41) - /unlink 17592 (time 1734260701.39 total 40.01 last 412.19) - /unlink 20000 (time 1734260707.65 total 46.26 last 385.06) - /unlink 24303 (time 1734260717.65 total 56.26 last 430.27) - /unlink 28613 (time 1734260727.65 total 66.26 last 430.95) - /unlink 30000 (time 1734260730.86 total 69.48 last 431.31) - /unlink 34039 (time 1734260740.87 total 79.48 last 403.86) - /unlink 37962 (time 1734260750.87 total 89.48 last 392.27) - /unlink 40000 (time 1734260755.51 total 94.13 last 438.83) - /unlink 44114 (time 1734260765.51 total 104.13 last 411.36) - /unlink 48572 (time 1734260775.51 total 114.13 last 445.70) - /unlink 50000 (time 1734260778.06 total 116.68 last 560.00) - /unlink 54813 (time 1734260788.07 total 126.68 last 481.25) - /unlink 59816 (time 1734260798.07 total 136.68 last 500.29) total: 64768 /unlink in 145.16 seconds: 446.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2973 (time 1734260837.18 total 10.00 last 297.25) - open/close 6058 (time 1734260847.18 total 20.00 last 308.48) - open/close 9247 (time 1734260857.18 total 30.00 last 318.89) - open/close 10000 (time 1734260859.67 total 32.49 last 303.22) - open/close 12985 (time 1734260869.67 total 42.49 last 298.45) - open/close 16120 (time 1734260879.67 total 52.49 last 313.43) - open/close 19836 (time 1734260889.67 total 62.49 last 371.54) - open/close 23756 (time 1734260899.67 total 72.49 last 391.97) - open/close 27391 (time 1734260909.67 total 82.49 last 363.43) - open/close 30000 (time 1734260917.84 total 90.66 last 319.51) - open/close 33092 (time 1734260927.84 total 100.66 last 309.13) - open/close 36019 (time 1734260937.85 total 110.66 last 292.63) - open/close 39303 (time 1734260947.85 total 120.67 last 328.30) - open/close 40000 (time 1734260950.13 total 122.95 last 305.51) - open/close 43160 (time 1734260960.13 total 132.95 last 315.96) - open/close 46100 (time 1734260970.13 total 142.95 last 293.96) - open/close 49599 (time 1734260980.13 total 152.95 last 349.89) - open/close 53373 (time 1734260990.13 total 162.95 last 377.36) - open/close 56572 (time 1734261000.14 total 172.96 last 319.81) - open/close 59444 (time 1734261010.14 total 182.96 last 287.09) - open/close 63285 (time 1734261020.14 total 192.96 last 384.05) total: 64768 open/close in 196.99 seconds: 328.79 ops/second - /unlink 5843 (time 1734261035.10 total 10.00 last 584.24) - /unlink 10000 (time 1734261041.44 total 16.35 last 654.92) - /unlink 16973 (time 1734261051.45 total 26.35 last 697.20) - /unlink 20000 (time 1734261055.80 total 30.70 last 695.26) - /unlink 28381 (time 1734261065.80 total 40.70 last 838.02) - /unlink 37371 (time 1734261075.80 total 50.70 last 898.97) - /unlink 40000 (time 1734261078.70 total 53.60 last 906.79) - /unlink 48538 (time 1734261088.70 total 63.60 last 853.72) - /unlink 57359 (time 1734261098.70 total 73.61 last 882.02) - /unlink 60000 (time 1734261102.04 total 76.95 last 790.15) total: 64768 /unlink in 82.26 seconds: 787.35 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg232-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg232-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg232-server stop mds service on oleg232-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg232-server unloading modules via unload_modules_local on: 'oleg232-server' oleg232-server: oleg232-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-client: ssh exited with exit code 2 pdsh@oleg232-client: oleg232-server: ssh exited with exit code 2