== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:18:01 (1734257881) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 oleg255-server: oleg255-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: Force libcfs to create 2 CPU partitions oleg255-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg255-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16200 392 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1525 (time 1734257953.89 total 10.00 last 152.50) - open/close 2931 (time 1734257963.89 total 20.01 last 140.52) - open/close 4467 (time 1734257973.91 total 30.02 last 153.40) - open/close 6153 (time 1734257983.91 total 40.02 last 168.60) - open/close 7645 (time 1734257993.91 total 50.02 last 149.18) - open/close 9141 (time 1734258003.91 total 60.03 last 149.52) - open/close 10000 (time 1734258011.45 total 67.57 last 113.90) - open/close 11084 (time 1734258021.47 total 77.58 last 108.28) - open/close 12428 (time 1734258031.47 total 87.58 last 134.40) - open/close 13620 (time 1734258041.47 total 97.58 last 119.14) - open/close 14784 (time 1734258051.48 total 107.59 last 116.35) - open/close 16065 (time 1734258061.48 total 117.59 last 128.06) - open/close 17323 (time 1734258071.48 total 127.60 last 125.74) - open/close 18525 (time 1734258081.49 total 137.60 last 120.16) - open/close 19793 (time 1734258091.49 total 147.61 last 126.73) - open/close 20946 (time 1734258101.50 total 157.61 last 115.25) - open/close 22159 (time 1734258111.50 total 167.62 last 121.23) - open/close 23319 (time 1734258121.51 total 177.62 last 115.92) - open/close 24502 (time 1734258131.51 total 187.62 last 118.28) - open/close 25729 (time 1734258141.51 total 197.63 last 122.68) - open/close 27070 (time 1734258151.51 total 207.63 last 134.09) - open/close 28480 (time 1734258161.53 total 217.64 last 140.83) - open/close 29656 (time 1734258171.53 total 227.64 last 117.57) - open/close 30000 (time 1734258174.25 total 230.37 last 126.27) - open/close 31399 (time 1734258184.26 total 240.37 last 139.86) - open/close 32869 (time 1734258194.26 total 250.37 last 146.99) - open/close 34154 (time 1734258204.26 total 260.38 last 128.39) - open/close 35505 (time 1734258214.27 total 270.38 last 135.08) - open/close 36764 (time 1734258224.27 total 280.39 last 125.80) - open/close 38258 (time 1734258234.28 total 290.39 last 149.33) - open/close 39697 (time 1734258244.28 total 300.39 last 143.88) - open/close 40000 (time 1734258246.85 total 302.96 last 118.15) - open/close 41315 (time 1734258256.85 total 312.96 last 131.48) - open/close 42803 (time 1734258266.85 total 322.97 last 148.73) - open/close 44024 (time 1734258276.85 total 332.97 last 122.09) - open/close 45359 (time 1734258286.86 total 342.97 last 133.44) - open/close 46762 (time 1734258296.86 total 352.97 last 140.30) - open/close 48120 (time 1734258306.87 total 362.99 last 135.61) - open/close 49502 (time 1734258316.87 total 372.99 last 138.18) - open/close 50000 (time 1734258320.49 total 376.60 last 137.72) - open/close 51225 (time 1734258330.50 total 386.61 last 122.41) - open/close 52519 (time 1734258340.50 total 396.61 last 129.40) - open/close 53909 (time 1734258350.50 total 406.62 last 138.91) - open/close 55237 (time 1734258360.51 total 416.62 last 132.78) - open/close 56609 (time 1734258370.51 total 426.62 last 137.13) - open/close 57921 (time 1734258380.51 total 436.63 last 131.17) - open/close 59301 (time 1734258390.52 total 446.63 last 137.91) - open/close 60000 (time 1734258394.84 total 450.95 last 161.91) - open/close 61708 (time 1734258404.84 total 460.96 last 170.66) - open/close 63321 (time 1734258414.85 total 470.96 last 161.22) total: 64768 open/close in 478.74 seconds: 135.29 ops/second - /unlink 2531 (time 1734258434.06 total 10.00 last 253.05) - /unlink 5472 (time 1734258444.06 total 20.00 last 294.10) - /unlink 7900 (time 1734258454.06 total 30.00 last 242.76) - /unlink 10000 (time 1734258462.10 total 38.04 last 261.21) - /unlink 12855 (time 1734258472.11 total 48.05 last 285.41) - /unlink 14937 (time 1734258482.11 total 58.05 last 208.13) - /unlink 17187 (time 1734258492.12 total 68.06 last 224.82) - /unlink 19749 (time 1734258502.12 total 78.06 last 256.17) - /unlink 21940 (time 1734258512.12 total 88.06 last 219.07) - /unlink 24310 (time 1734258522.12 total 98.06 last 236.91) - /unlink 26558 (time 1734258532.13 total 108.07 last 224.77) - /unlink 28619 (time 1734258542.13 total 118.07 last 206.04) - /unlink 30000 (time 1734258548.17 total 124.11 last 228.76) - /unlink 32383 (time 1734258558.17 total 134.11 last 238.16) - /unlink 34518 (time 1734258568.17 total 144.11 last 213.48) - /unlink 36779 (time 1734258578.17 total 154.11 last 226.07) - /unlink 39160 (time 1734258588.18 total 164.12 last 238.05) - /unlink 40000 (time 1734258592.19 total 168.13 last 209.02) - /unlink 42424 (time 1734258602.20 total 178.13 last 242.39) - /unlink 44921 (time 1734258612.20 total 188.14 last 249.64) - /unlink 47383 (time 1734258622.20 total 198.14 last 246.13) - /unlink 49757 (time 1734258632.21 total 208.15 last 237.17) - /unlink 51837 (time 1734258642.21 total 218.15 last 207.97) - /unlink 54123 (time 1734258652.21 total 228.15 last 228.55) - /unlink 56429 (time 1734258662.21 total 238.15 last 230.58) - /unlink 58513 (time 1734258672.22 total 248.16 last 208.39) - /unlink 60000 (time 1734258678.37 total 254.31 last 241.49) - /unlink 63046 (time 1734258688.37 total 264.31 last 304.60) total: 64768 /unlink in 271.21 seconds: 238.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1387 (time 1734258721.80 total 10.00 last 138.69) - open/close 2778 (time 1734258731.80 total 20.01 last 139.04) - open/close 4190 (time 1734258741.81 total 30.01 last 141.13) - open/close 5516 (time 1734258751.81 total 40.01 last 132.56) - open/close 7338 (time 1734258761.82 total 50.02 last 182.14) - open/close 9155 (time 1734258771.82 total 60.02 last 181.61) - open/close 10000 (time 1734258777.46 total 65.66 last 149.78) - open/close 11675 (time 1734258787.47 total 75.67 last 167.42) - open/close 13602 (time 1734258797.47 total 85.67 last 192.59) - open/close 15159 (time 1734258807.47 total 95.67 last 155.68) - open/close 16488 (time 1734258817.48 total 105.68 last 132.85) - open/close 18172 (time 1734258827.48 total 115.68 last 168.31) - open/close 19713 (time 1734258837.48 total 125.69 last 154.08) - open/close 20000 (time 1734258839.52 total 127.72 last 141.20) - open/close 21767 (time 1734258849.52 total 137.72 last 176.69) - open/close 23538 (time 1734258859.52 total 147.72 last 177.04) - open/close 25885 (time 1734258869.52 total 157.72 last 234.67) - open/close 27989 (time 1734258879.53 total 167.73 last 210.33) - open/close 29728 (time 1734258889.53 total 177.73 last 173.88) - open/close 31665 (time 1734258899.53 total 187.73 last 193.68) - open/close 33219 (time 1734258909.53 total 197.73 last 155.35) - open/close 34789 (time 1734258919.54 total 207.74 last 156.91) - open/close 36432 (time 1734258929.54 total 217.74 last 164.24) - open/close 38298 (time 1734258939.54 total 227.75 last 186.53) - open/close 40000 (time 1734258948.87 total 237.07 last 182.59) - open/close 42059 (time 1734258958.87 total 247.07 last 205.84) - open/close 43948 (time 1734258968.87 total 257.07 last 188.87) - open/close 45717 (time 1734258978.87 total 267.07 last 176.86) - open/close 47587 (time 1734258988.87 total 277.07 last 186.98) - open/close 49661 (time 1734258998.88 total 287.08 last 207.33) - open/close 51753 (time 1734259008.88 total 297.08 last 209.18) - open/close 53726 (time 1734259018.88 total 307.08 last 197.30) - open/close 55363 (time 1734259028.88 total 317.08 last 163.68) - open/close 56981 (time 1734259038.88 total 327.08 last 161.77) - open/close 58858 (time 1734259048.88 total 337.08 last 187.69) - open/close 60000 (time 1734259055.57 total 343.78 last 170.63) - open/close 61882 (time 1734259065.58 total 353.78 last 188.13) - open/close 63861 (time 1734259075.58 total 363.78 last 197.87) total: 64768 open/close in 368.95 seconds: 175.55 ops/second - /unlink 2947 (time 1734259092.05 total 10.00 last 294.58) - /unlink 6229 (time 1734259102.05 total 20.00 last 328.19) - /unlink 9176 (time 1734259112.05 total 30.01 last 294.68) - /unlink 10000 (time 1734259114.12 total 32.07 last 399.46) - /unlink 13769 (time 1734259124.12 total 42.07 last 376.85) - /unlink 16849 (time 1734259134.12 total 52.07 last 307.87) - /unlink 20000 (time 1734259143.88 total 61.83 last 322.95) - /unlink 23286 (time 1734259153.88 total 71.83 last 328.54) - /unlink 26541 (time 1734259163.88 total 81.83 last 325.45) - /unlink 30000 (time 1734259173.79 total 91.74 last 349.18) - /unlink 32901 (time 1734259183.79 total 101.74 last 290.10) - /unlink 36565 (time 1734259193.79 total 111.74 last 366.38) - /unlink 39401 (time 1734259203.79 total 121.74 last 283.52) - /unlink 42842 (time 1734259213.79 total 131.74 last 344.06) - /unlink 46218 (time 1734259223.80 total 141.75 last 337.40) - /unlink 48988 (time 1734259233.80 total 151.75 last 276.98) - /unlink 50000 (time 1734259237.66 total 155.61 last 262.32) - /unlink 53360 (time 1734259247.66 total 165.61 last 335.97) - /unlink 56888 (time 1734259257.66 total 175.61 last 352.73) - /unlink 60000 (time 1734259266.66 total 184.61 last 345.85) - /unlink 63642 (time 1734259276.66 total 194.61 last 364.19) total: 64768 /unlink in 197.17 seconds: 328.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1828 (time 1734259307.90 total 10.00 last 182.77) - open/close 3505 (time 1734259317.90 total 20.01 last 167.64) - open/close 5322 (time 1734259327.91 total 30.01 last 181.59) - open/close 7248 (time 1734259337.91 total 40.02 last 192.50) - open/close 9298 (time 1734259347.92 total 50.02 last 204.96) - open/close 10000 (time 1734259351.64 total 53.74 last 188.64) - open/close 12336 (time 1734259361.64 total 63.74 last 233.55) - open/close 14168 (time 1734259371.64 total 73.75 last 183.14) - open/close 16162 (time 1734259381.64 total 83.75 last 199.38) - open/close 18151 (time 1734259391.65 total 93.75 last 198.84) - open/close 20000 (time 1734259401.27 total 103.37 last 192.18) - open/close 22124 (time 1734259411.28 total 113.38 last 212.27) - open/close 23930 (time 1734259421.28 total 123.38 last 180.54) - open/close 25735 (time 1734259431.28 total 133.38 last 180.47) - open/close 27559 (time 1734259441.29 total 143.39 last 182.23) - open/close 29206 (time 1734259451.29 total 153.39 last 164.65) - open/close 30000 (time 1734259456.38 total 158.49 last 155.91) - open/close 31759 (time 1734259466.39 total 168.49 last 175.87) - open/close 33852 (time 1734259476.39 total 178.49 last 209.29) - open/close 35699 (time 1734259486.39 total 188.49 last 184.67) - open/close 37591 (time 1734259496.39 total 198.49 last 189.13) - open/close 39319 (time 1734259506.39 total 208.49 last 172.80) - open/close 40000 (time 1734259509.77 total 211.87 last 201.67) - open/close 42003 (time 1734259519.77 total 221.87 last 200.26) - open/close 43909 (time 1734259529.78 total 231.88 last 190.49) - open/close 45588 (time 1734259539.78 total 241.88 last 167.84) - open/close 47407 (time 1734259549.78 total 251.88 last 181.88) - open/close 49217 (time 1734259559.78 total 261.88 last 181.00) - open/close 50000 (time 1734259563.63 total 265.73 last 203.39) - open/close 52049 (time 1734259573.64 total 275.74 last 204.82) - open/close 53806 (time 1734259583.64 total 285.74 last 175.66) - open/close 55838 (time 1734259593.64 total 295.74 last 203.19) - open/close 58038 (time 1734259603.64 total 305.74 last 219.91) - open/close 60000 (time 1734259612.99 total 315.09 last 209.98) - open/close 62113 (time 1734259622.99 total 325.09 last 211.28) - open/close 64470 (time 1734259632.99 total 335.09 last 235.68) total: 64768 open/close in 336.53 seconds: 192.46 ops/second - /unlink 4641 (time 1734259645.52 total 10.00 last 464.07) - /unlink 8857 (time 1734259655.52 total 20.00 last 421.52) - /unlink 10000 (time 1734259658.31 total 22.80 last 408.67) - /unlink 13464 (time 1734259668.31 total 32.80 last 346.39) - /unlink 17024 (time 1734259678.31 total 42.80 last 356.00) - /unlink 20000 (time 1734259685.04 total 49.53 last 442.27) - /unlink 23986 (time 1734259695.04 total 59.53 last 398.59) - /unlink 28820 (time 1734259705.04 total 69.53 last 483.38) - /unlink 30000 (time 1734259707.28 total 71.77 last 526.91) - /unlink 34021 (time 1734259717.28 total 81.77 last 402.07) - /unlink 37984 (time 1734259727.29 total 91.77 last 396.16) - /unlink 40000 (time 1734259731.83 total 96.31 last 444.30) - /unlink 43167 (time 1734259741.83 total 106.31 last 316.60) - /unlink 46694 (time 1734259751.83 total 116.31 last 352.66) - /unlink 50000 (time 1734259761.17 total 125.66 last 353.83) - /unlink 53045 (time 1734259771.17 total 135.66 last 304.49) - /unlink 56356 (time 1734259781.18 total 145.66 last 331.04) - /unlink 59656 (time 1734259791.18 total 155.66 last 329.99) - /unlink 63657 (time 1734259801.18 total 165.66 last 400.09) total: 64768 /unlink in 167.90 seconds: 385.75 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2419 (time 1734259832.93 total 10.00 last 241.85) - open/close 4722 (time 1734259842.93 total 20.00 last 230.30) - open/close 6967 (time 1734259852.93 total 30.01 last 224.41) - open/close 8791 (time 1734259862.93 total 40.01 last 182.35) - open/close 10000 (time 1734259868.39 total 45.46 last 221.63) - open/close 12005 (time 1734259878.39 total 55.47 last 200.44) - open/close 14049 (time 1734259888.40 total 65.47 last 204.32) - open/close 16419 (time 1734259898.40 total 75.47 last 236.93) - open/close 18484 (time 1734259908.40 total 85.48 last 206.44) - open/close 20000 (time 1734259915.35 total 92.42 last 218.27) - open/close 22609 (time 1734259925.35 total 102.42 last 260.82) - open/close 25400 (time 1734259935.35 total 112.42 last 279.10) - open/close 28125 (time 1734259945.35 total 122.43 last 272.49) - open/close 30000 (time 1734259951.36 total 128.44 last 311.92) - open/close 32807 (time 1734259961.36 total 138.44 last 280.65) - open/close 35800 (time 1734259971.37 total 148.44 last 299.23) - open/close 38811 (time 1734259981.37 total 158.44 last 300.98) - open/close 40000 (time 1734259985.13 total 162.20 last 316.35) - open/close 42843 (time 1734259995.13 total 172.20 last 284.26) - open/close 45859 (time 1734260005.13 total 182.21 last 301.51) - open/close 48762 (time 1734260015.13 total 192.21 last 290.24) - open/close 50000 (time 1734260019.36 total 196.43 last 292.97) - open/close 53063 (time 1734260029.36 total 206.44 last 306.29) - open/close 56061 (time 1734260039.36 total 216.44 last 299.80) - open/close 58630 (time 1734260049.37 total 226.44 last 256.77) - open/close 60000 (time 1734260054.55 total 231.62 last 264.30) - open/close 63084 (time 1734260064.55 total 241.62 last 308.39) total: 64768 open/close in 246.50 seconds: 262.75 ops/second - /unlink 5588 (time 1734260080.36 total 10.00 last 558.67) - /unlink 10000 (time 1734260089.03 total 18.67 last 509.15) - /unlink 15303 (time 1734260099.03 total 28.67 last 530.28) - /unlink 20000 (time 1734260108.44 total 38.08 last 498.94) - /unlink 25434 (time 1734260118.44 total 48.08 last 543.28) - /unlink 30000 (time 1734260127.30 total 56.94 last 515.65) - /unlink 34873 (time 1734260137.30 total 66.94 last 487.30) - /unlink 40000 (time 1734260146.66 total 76.30 last 547.48) - /unlink 45229 (time 1734260156.66 total 86.30 last 522.85) - /unlink 50000 (time 1734260164.97 total 94.61 last 574.33) - /unlink 55438 (time 1734260174.97 total 104.61 last 543.72) - /unlink 60000 (time 1734260183.64 total 113.28 last 526.38) - /unlink 64720 (time 1734260193.64 total 123.28 last 471.92) total: 64768 /unlink in 123.38 seconds: 524.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2789 (time 1734260224.49 total 10.00 last 278.81) - open/close 5191 (time 1734260234.49 total 20.01 last 240.10) - open/close 8129 (time 1734260244.49 total 30.01 last 293.70) - open/close 10000 (time 1734260250.98 total 36.50 last 288.41) - open/close 12792 (time 1734260260.98 total 46.50 last 279.14) - open/close 15836 (time 1734260270.99 total 56.50 last 304.35) - open/close 19027 (time 1734260280.99 total 66.50 last 319.02) - open/close 20000 (time 1734260284.61 total 70.12 last 268.81) - open/close 23090 (time 1734260294.61 total 80.12 last 309.00) - open/close 25723 (time 1734260304.61 total 90.13 last 263.29) - open/close 28140 (time 1734260314.61 total 100.13 last 241.63) - open/close 30000 (time 1734260321.90 total 107.42 last 255.21) - open/close 32648 (time 1734260331.90 total 117.42 last 264.72) - open/close 35083 (time 1734260341.90 total 127.42 last 243.46) - open/close 37443 (time 1734260351.91 total 137.42 last 235.95) - open/close 39912 (time 1734260361.91 total 147.43 last 246.80) - open/close 42608 (time 1734260371.91 total 157.43 last 269.55) - open/close 45195 (time 1734260381.91 total 167.43 last 258.69) - open/close 47727 (time 1734260391.91 total 177.43 last 253.17) - open/close 50000 (time 1734260399.67 total 185.18 last 293.18) - open/close 52841 (time 1734260409.67 total 195.19 last 284.03) - open/close 55494 (time 1734260419.67 total 205.19 last 265.23) - open/close 58262 (time 1734260429.67 total 215.19 last 276.78) - open/close 60000 (time 1734260436.75 total 222.27 last 245.46) - open/close 62711 (time 1734260446.76 total 232.27 last 271.02) total: 64768 open/close in 238.12 seconds: 272.00 ops/second - /unlink 5987 (time 1734260463.55 total 10.00 last 598.70) - /unlink 10000 (time 1734260470.92 total 17.36 last 545.00) - /unlink 14988 (time 1734260480.92 total 27.36 last 498.79) - /unlink 19538 (time 1734260490.92 total 37.36 last 454.99) - /unlink 24582 (time 1734260500.92 total 47.36 last 504.37) - /unlink 30000 (time 1734260510.19 total 56.64 last 583.97) - /unlink 34945 (time 1734260520.20 total 66.64 last 494.42) - /unlink 39714 (time 1734260530.20 total 76.64 last 476.83) - /unlink 44755 (time 1734260540.20 total 86.65 last 504.08) - /unlink 49723 (time 1734260550.20 total 96.65 last 496.75) - /unlink 54616 (time 1734260560.20 total 106.65 last 489.20) - /unlink 59712 (time 1734260570.20 total 116.65 last 509.57) total: 64768 /unlink in 125.47 seconds: 516.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server unloading modules via unload_modules_local on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2