== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:18:21 (1734257901) Stopping clients: oleg201-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg201-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 oleg201-server: oleg201-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing load_modules_local oleg201-server: Loading modules from /home/green/git/lustre-release/lustre oleg201-server: detected 4 online CPUs by sysfs oleg201-server: Force libcfs to create 2 CPU partitions oleg201-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg201-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg201-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg201-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1843 (time 1734257967.98 total 10.00 last 184.29) - open/close 3696 (time 1734257977.98 total 20.00 last 185.25) - open/close 5778 (time 1734257987.99 total 30.01 last 208.08) - open/close 7386 (time 1734257997.99 total 40.01 last 160.75) - open/close 8890 (time 1734258007.99 total 50.02 last 150.36) - open/close 10000 (time 1734258017.55 total 59.58 last 116.08) - open/close 11355 (time 1734258027.56 total 69.58 last 135.49) - open/close 12732 (time 1734258037.56 total 79.58 last 137.67) - open/close 14052 (time 1734258047.57 total 89.59 last 131.88) - open/close 15399 (time 1734258057.57 total 99.59 last 134.69) - open/close 16740 (time 1734258067.57 total 109.59 last 134.07) - open/close 18126 (time 1734258077.57 total 119.60 last 138.55) - open/close 19452 (time 1734258087.58 total 129.60 last 132.53) - open/close 20000 (time 1734258091.86 total 133.88 last 127.97) - open/close 21275 (time 1734258101.86 total 143.89 last 127.48) - open/close 22619 (time 1734258111.86 total 153.89 last 134.39) - open/close 23895 (time 1734258121.87 total 163.90 last 127.47) - open/close 25301 (time 1734258131.88 total 173.90 last 140.52) - open/close 26746 (time 1734258141.88 total 183.91 last 144.46) - open/close 28324 (time 1734258151.88 total 193.91 last 157.77) - open/close 29899 (time 1734258161.89 total 203.91 last 157.45) - open/close 31306 (time 1734258171.89 total 213.91 last 140.68) - open/close 32781 (time 1734258181.91 total 223.93 last 147.19) - open/close 34571 (time 1734258191.92 total 233.94 last 178.84) - open/close 36125 (time 1734258201.92 total 243.94 last 155.37) - open/close 37783 (time 1734258211.93 total 253.95 last 165.73) - open/close 39235 (time 1734258221.93 total 263.95 last 145.11) - open/close 40000 (time 1734258226.29 total 268.32 last 175.36) - open/close 41720 (time 1734258236.31 total 278.33 last 171.77) - open/close 43384 (time 1734258246.32 total 288.34 last 166.26) - open/close 44927 (time 1734258256.32 total 298.34 last 154.29) - open/close 46578 (time 1734258266.32 total 308.34 last 165.06) - open/close 48156 (time 1734258276.32 total 318.34 last 157.76) - open/close 49729 (time 1734258286.33 total 328.35 last 157.23) - open/close 51389 (time 1734258296.33 total 338.35 last 165.97) - open/close 53160 (time 1734258306.33 total 348.35 last 177.08) - open/close 54787 (time 1734258316.33 total 358.35 last 162.68) - open/close 56334 (time 1734258326.33 total 368.35 last 154.69) - open/close 57856 (time 1734258336.34 total 378.36 last 152.12) - open/close 59304 (time 1734258346.34 total 388.37 last 144.70) - open/close 60000 (time 1734258350.83 total 392.85 last 155.25) - open/close 61653 (time 1734258360.83 total 402.85 last 165.23) - open/close 63238 (time 1734258370.83 total 412.85 last 158.49) total: 64768 open/close in 422.63 seconds: 153.25 ops/second - /unlink 3126 (time 1734258392.14 total 10.00 last 312.52) - /unlink 6719 (time 1734258402.14 total 20.00 last 359.27) - /unlink 10000 (time 1734258411.50 total 29.37 last 350.46) - /unlink 13869 (time 1734258421.51 total 39.37 last 386.81) - /unlink 17186 (time 1734258431.51 total 49.37 last 331.69) - /unlink 20000 (time 1734258439.61 total 57.48 last 347.05) - /unlink 23122 (time 1734258449.61 total 67.48 last 312.17) - /unlink 26213 (time 1734258459.62 total 77.48 last 309.04) - /unlink 29395 (time 1734258469.62 total 87.48 last 318.08) - /unlink 30000 (time 1734258471.96 total 89.82 last 258.43) - /unlink 32467 (time 1734258481.96 total 99.83 last 246.64) - /unlink 35022 (time 1734258491.97 total 109.83 last 255.46) - /unlink 37932 (time 1734258501.97 total 119.83 last 290.92) - /unlink 40000 (time 1734258509.80 total 127.66 last 264.10) - /unlink 42609 (time 1734258519.80 total 137.67 last 260.79) - /unlink 45193 (time 1734258529.81 total 147.67 last 258.32) - /unlink 47497 (time 1734258539.81 total 157.67 last 230.38) - /unlink 49997 (time 1734258549.81 total 167.67 last 249.92) - /unlink 52778 (time 1734258559.81 total 177.67 last 278.06) - /unlink 55402 (time 1734258569.81 total 187.68 last 262.36) - /unlink 57970 (time 1734258579.81 total 197.68 last 256.77) - /unlink 60000 (time 1734258586.56 total 204.42 last 301.12) - /unlink 62765 (time 1734258596.56 total 214.42 last 276.43) total: 64768 /unlink in 221.17 seconds: 292.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1419 (time 1734258633.36 total 10.00 last 141.85) - open/close 2740 (time 1734258643.36 total 20.00 last 132.10) - open/close 4178 (time 1734258653.36 total 30.00 last 143.79) - open/close 5678 (time 1734258663.36 total 40.01 last 149.99) - open/close 7177 (time 1734258673.36 total 50.01 last 149.84) - open/close 8991 (time 1734258683.36 total 60.01 last 181.36) - open/close 10000 (time 1734258688.97 total 65.62 last 179.84) - open/close 11765 (time 1734258698.98 total 75.63 last 176.41) - open/close 13683 (time 1734258708.98 total 85.63 last 191.79) - open/close 15638 (time 1734258718.98 total 95.63 last 195.48) - open/close 17207 (time 1734258728.99 total 105.64 last 156.76) - open/close 18914 (time 1734258739.00 total 115.64 last 170.61) - open/close 20000 (time 1734258745.28 total 121.92 last 172.90) - open/close 21669 (time 1734258755.28 total 131.93 last 166.84) - open/close 23818 (time 1734258765.29 total 141.93 last 214.75) - open/close 25952 (time 1734258775.29 total 151.94 last 213.36) - open/close 27825 (time 1734258785.29 total 161.94 last 187.28) - open/close 30000 (time 1734258795.05 total 171.70 last 222.81) - open/close 32047 (time 1734258805.05 total 181.70 last 204.63) - open/close 33651 (time 1734258815.06 total 191.71 last 160.30) - open/close 35378 (time 1734258825.06 total 201.71 last 172.67) - open/close 37322 (time 1734258835.07 total 211.71 last 194.32) - open/close 39017 (time 1734258845.07 total 221.72 last 169.45) - open/close 40000 (time 1734258849.38 total 226.03 last 228.11) - open/close 42236 (time 1734258859.38 total 236.03 last 223.56) - open/close 44813 (time 1734258869.38 total 246.03 last 257.63) - open/close 47633 (time 1734258879.39 total 256.03 last 281.93) - open/close 49731 (time 1734258889.39 total 266.03 last 209.77) - open/close 52052 (time 1734258899.39 total 276.04 last 232.09) - open/close 53964 (time 1734258909.39 total 286.04 last 191.19) - open/close 55707 (time 1734258919.39 total 296.04 last 174.27) - open/close 57752 (time 1734258929.39 total 306.04 last 204.50) - open/close 59887 (time 1734258939.39 total 316.04 last 213.49) - open/close 62390 (time 1734258949.39 total 326.04 last 250.21) total: 64768 open/close in 335.85 seconds: 192.85 ops/second - /unlink 4282 (time 1734258970.26 total 10.00 last 428.09) - /unlink 8334 (time 1734258980.27 total 20.01 last 405.10) - /unlink 10000 (time 1734258984.43 total 24.17 last 399.84) - /unlink 14297 (time 1734258994.43 total 34.17 last 429.66) - /unlink 18968 (time 1734259004.44 total 44.17 last 467.02) - /unlink 23840 (time 1734259014.44 total 54.17 last 487.17) - /unlink 27694 (time 1734259024.44 total 64.18 last 385.38) - /unlink 30000 (time 1734259031.10 total 70.84 last 346.03) - /unlink 33702 (time 1734259041.10 total 80.84 last 370.12) - /unlink 37701 (time 1734259051.10 total 90.84 last 399.85) - /unlink 40000 (time 1734259057.89 total 97.63 last 338.74) - /unlink 44271 (time 1734259067.89 total 107.63 last 427.04) - /unlink 48516 (time 1734259077.89 total 117.63 last 424.49) - /unlink 50000 (time 1734259081.49 total 121.23 last 412.32) - /unlink 53885 (time 1734259091.49 total 131.23 last 388.50) - /unlink 57835 (time 1734259101.49 total 141.23 last 394.90) - /unlink 60000 (time 1734259107.57 total 147.31 last 356.47) - /unlink 64605 (time 1734259117.57 total 157.31 last 460.44) total: 64768 /unlink in 157.63 seconds: 410.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2214 (time 1734259143.32 total 10.00 last 221.31) - open/close 4496 (time 1734259153.33 total 20.01 last 228.15) - open/close 6864 (time 1734259163.33 total 30.01 last 236.74) - open/close 9202 (time 1734259173.33 total 40.01 last 233.73) - open/close 10000 (time 1734259176.82 total 43.50 last 228.96) - open/close 12099 (time 1734259186.82 total 53.50 last 209.90) - open/close 14653 (time 1734259196.82 total 63.50 last 255.26) - open/close 16822 (time 1734259206.83 total 73.51 last 216.85) - open/close 19320 (time 1734259216.83 total 83.51 last 249.67) - open/close 20000 (time 1734259219.54 total 86.22 last 250.88) - open/close 22043 (time 1734259229.54 total 96.22 last 204.28) - open/close 24070 (time 1734259239.54 total 106.22 last 202.69) - open/close 26278 (time 1734259249.55 total 116.23 last 220.72) - open/close 28632 (time 1734259259.55 total 126.23 last 235.36) - open/close 30000 (time 1734259266.36 total 133.04 last 200.71) - open/close 32260 (time 1734259276.36 total 143.04 last 225.98) - open/close 34968 (time 1734259286.37 total 153.05 last 270.72) - open/close 37384 (time 1734259296.37 total 163.05 last 241.58) - open/close 39632 (time 1734259306.38 total 173.05 last 224.66) - open/close 41564 (time 1734259316.38 total 183.06 last 193.12) - open/close 43600 (time 1734259326.38 total 193.06 last 203.52) - open/close 45700 (time 1734259336.39 total 203.07 last 209.89) - open/close 48135 (time 1734259346.39 total 213.07 last 243.48) - open/close 50000 (time 1734259353.87 total 220.55 last 249.14) - open/close 52492 (time 1734259363.88 total 230.56 last 249.16) - open/close 54680 (time 1734259373.88 total 240.56 last 218.70) - open/close 57035 (time 1734259383.88 total 250.56 last 235.43) - open/close 59428 (time 1734259393.89 total 260.57 last 239.21) - open/close 60000 (time 1734259396.45 total 263.13 last 222.94) - open/close 62204 (time 1734259406.46 total 273.13 last 220.37) - open/close 64512 (time 1734259416.46 total 283.14 last 230.76) total: 64768 open/close in 284.24 seconds: 227.86 ops/second - /unlink 3692 (time 1734259428.62 total 10.00 last 369.16) - /unlink 7652 (time 1734259438.62 total 20.00 last 395.95) - /unlink 10000 (time 1734259445.55 total 26.93 last 338.84) - /unlink 13644 (time 1734259455.55 total 36.93 last 364.31) - /unlink 17513 (time 1734259465.55 total 46.94 last 386.86) - /unlink 20000 (time 1734259471.11 total 52.49 last 447.66) - /unlink 24389 (time 1734259481.11 total 62.49 last 438.87) - /unlink 28411 (time 1734259491.11 total 72.49 last 402.10) - /unlink 30000 (time 1734259494.79 total 76.17 last 431.84) - /unlink 33761 (time 1734259504.79 total 86.18 last 376.03) - /unlink 37865 (time 1734259514.79 total 96.18 last 410.37) - /unlink 40000 (time 1734259519.76 total 101.14 last 429.91) - /unlink 43788 (time 1734259529.76 total 111.15 last 378.69) - /unlink 47467 (time 1734259539.77 total 121.15 last 367.85) - /unlink 50000 (time 1734259546.74 total 128.12 last 363.20) - /unlink 53826 (time 1734259556.74 total 138.12 last 382.59) - /unlink 57949 (time 1734259566.74 total 148.12 last 412.25) - /unlink 60000 (time 1734259571.80 total 153.18 last 405.19) - /unlink 64100 (time 1734259581.80 total 163.19 last 409.93) total: 64768 /unlink in 164.75 seconds: 393.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2657 (time 1734259609.04 total 10.00 last 265.68) - open/close 5141 (time 1734259619.04 total 20.00 last 248.32) - open/close 7994 (time 1734259629.05 total 30.01 last 285.23) - open/close 10000 (time 1734259635.90 total 36.86 last 292.64) - open/close 13020 (time 1734259645.90 total 46.86 last 301.98) - open/close 15966 (time 1734259655.90 total 56.86 last 294.57) - open/close 18416 (time 1734259665.90 total 66.86 last 244.99) - open/close 20000 (time 1734259672.37 total 73.34 last 244.71) - open/close 22663 (time 1734259682.38 total 83.34 last 266.29) - open/close 25275 (time 1734259692.38 total 93.34 last 261.16) - open/close 28556 (time 1734259702.38 total 103.34 last 328.04) - open/close 30000 (time 1734259707.37 total 108.33 last 289.37) - open/close 32638 (time 1734259717.37 total 118.33 last 263.77) - open/close 35140 (time 1734259727.37 total 128.33 last 250.12) - open/close 37694 (time 1734259737.38 total 138.34 last 255.33) - open/close 39991 (time 1734259747.38 total 148.34 last 229.57) - open/close 42488 (time 1734259757.38 total 158.34 last 249.66) - open/close 44840 (time 1734259767.39 total 168.35 last 235.13) - open/close 46920 (time 1734259777.39 total 178.35 last 207.98) - open/close 49193 (time 1734259787.39 total 188.35 last 227.25) - open/close 50000 (time 1734259790.69 total 191.65 last 244.48) - open/close 52370 (time 1734259800.69 total 201.65 last 236.97) - open/close 55446 (time 1734259810.69 total 211.65 last 307.56) - open/close 58247 (time 1734259820.69 total 221.65 last 280.09) - open/close 60000 (time 1734259826.97 total 227.93 last 279.44) - open/close 62850 (time 1734259836.97 total 237.93 last 284.96) total: 64768 open/close in 245.21 seconds: 264.14 ops/second - /unlink 4710 (time 1734259855.27 total 10.00 last 470.94) - /unlink 9226 (time 1734259865.27 total 20.00 last 451.58) - /unlink 13850 (time 1734259875.28 total 30.00 last 462.31) - /unlink 18210 (time 1734259885.28 total 40.00 last 435.99) - /unlink 20000 (time 1734259888.54 total 43.27 last 548.68) - /unlink 24877 (time 1734259898.54 total 53.27 last 487.64) - /unlink 29650 (time 1734259908.54 total 63.27 last 477.21) - /unlink 34563 (time 1734259918.54 total 73.27 last 491.19) - /unlink 39875 (time 1734259928.55 total 83.27 last 531.10) - /unlink 45434 (time 1734259938.55 total 93.27 last 555.87) - /unlink 50000 (time 1734259947.04 total 101.77 last 537.58) - /unlink 55831 (time 1734259957.04 total 111.77 last 582.94) - /unlink 60000 (time 1734259964.83 total 119.55 last 535.56) total: 64768 /unlink in 127.42 seconds: 508.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2962 (time 1734260000.67 total 10.00 last 296.11) - open/close 6243 (time 1734260010.67 total 20.00 last 328.10) - open/close 9190 (time 1734260020.67 total 30.01 last 294.62) - open/close 10000 (time 1734260023.51 total 32.85 last 285.24) - open/close 13252 (time 1734260033.52 total 42.85 last 325.12) - open/close 16304 (time 1734260043.52 total 52.85 last 305.16) - open/close 18811 (time 1734260053.52 total 62.85 last 250.63) - open/close 20000 (time 1734260057.63 total 66.96 last 289.63) - open/close 23346 (time 1734260067.63 total 76.96 last 334.57) - open/close 26543 (time 1734260077.63 total 86.96 last 319.66) - open/close 29286 (time 1734260087.63 total 96.96 last 274.28) - open/close 30000 (time 1734260089.98 total 99.31 last 303.71) - open/close 32784 (time 1734260099.98 total 109.31 last 278.40) - open/close 35595 (time 1734260109.98 total 119.31 last 281.01) - open/close 38684 (time 1734260119.99 total 129.32 last 308.85) - open/close 40000 (time 1734260124.77 total 134.10 last 274.84) - open/close 42832 (time 1734260134.78 total 144.11 last 283.14) - open/close 45652 (time 1734260144.78 total 154.11 last 281.96) - open/close 48554 (time 1734260154.78 total 164.11 last 290.11) - open/close 50000 (time 1734260159.38 total 168.71 last 314.53) - open/close 53089 (time 1734260169.38 total 178.71 last 308.86) - open/close 55940 (time 1734260179.38 total 188.71 last 285.05) - open/close 58430 (time 1734260189.38 total 198.71 last 248.97) - open/close 60000 (time 1734260195.88 total 205.22 last 241.43) - open/close 62886 (time 1734260205.89 total 215.22 last 288.53) total: 64768 open/close in 222.23 seconds: 291.44 ops/second - /unlink 4821 (time 1734260223.91 total 10.00 last 481.99) - /unlink 8962 (time 1734260233.92 total 20.00 last 414.09) - /unlink 10000 (time 1734260236.02 total 22.11 last 492.99) - /unlink 15239 (time 1734260246.02 total 32.11 last 523.89) - /unlink 20000 (time 1734260255.69 total 41.77 last 492.54) - /unlink 25182 (time 1734260265.69 total 51.78 last 518.14) - /unlink 30000 (time 1734260274.74 total 60.83 last 532.05) - /unlink 35359 (time 1734260284.74 total 70.83 last 535.83) - /unlink 40000 (time 1734260292.49 total 78.58 last 598.96) - /unlink 45242 (time 1734260302.49 total 88.58 last 524.20) - /unlink 49674 (time 1734260312.50 total 98.58 last 443.12) - /unlink 54461 (time 1734260322.50 total 108.58 last 478.65) - /unlink 59585 (time 1734260332.50 total 118.58 last 512.39) - /unlink 64248 (time 1734260342.50 total 128.58 last 466.28) total: 64768 /unlink in 129.97 seconds: 498.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server unloading modules via unload_modules_local on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2