== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:13:03 (1726852383) Stopping clients: oleg436-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg436-client.virtnet /mnt/lustre2 (opts:-f) oleg436-server: oleg436-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing load_modules_local oleg436-server: Loading modules from /home/green/git/lustre-release/lustre oleg436-server: detected 4 online CPUs by sysfs oleg436-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg436-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg436-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg436-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg436-server: oleg436-server.virtnet: executing set_default_debug -1 all pdsh@oleg436-client: oleg436-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg436-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg436-client.virtnet: -o user_xattr,flock 192.168.204.136@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1339 (time 1726852440.81 total 10.01 last 133.77) - open/close 2471 (time 1726852450.82 total 20.02 last 113.10) - open/close 3679 (time 1726852460.82 total 30.02 last 120.75) - open/close 4965 (time 1726852470.83 total 40.03 last 128.48) - open/close 6207 (time 1726852480.84 total 50.04 last 124.14) - open/close 7231 (time 1726852490.85 total 60.04 last 102.34) - open/close 8430 (time 1726852500.85 total 70.05 last 119.83) - open/close 9586 (time 1726852510.85 total 80.05 last 115.58) - open/close 10000 (time 1726852514.19 total 83.39 last 124.17) - open/close 11207 (time 1726852524.19 total 93.39 last 120.65) - open/close 12382 (time 1726852534.19 total 103.39 last 117.48) - open/close 13456 (time 1726852544.20 total 113.40 last 107.34) - open/close 14660 (time 1726852554.20 total 123.40 last 120.34) - open/close 15850 (time 1726852564.21 total 133.40 last 118.96) - open/close 16989 (time 1726852574.22 total 143.42 last 113.76) - open/close 18228 (time 1726852584.23 total 153.42 last 123.79) - open/close 19457 (time 1726852594.24 total 163.44 last 122.77) - open/close 20000 (time 1726852599.01 total 168.21 last 113.83) - open/close 21175 (time 1726852609.01 total 178.21 last 117.45) - open/close 22311 (time 1726852619.02 total 188.21 last 113.54) - open/close 23641 (time 1726852629.03 total 198.22 last 132.88) - open/close 24931 (time 1726852639.03 total 208.23 last 128.96) - open/close 26245 (time 1726852649.04 total 218.23 last 131.31) - open/close 27610 (time 1726852659.04 total 228.24 last 136.45) - open/close 28881 (time 1726852669.05 total 238.24 last 127.02) - open/close 30000 (time 1726852676.91 total 246.11 last 142.23) - open/close 31243 (time 1726852686.91 total 256.11 last 124.27) - open/close 32528 (time 1726852696.92 total 266.12 last 128.45) - open/close 34001 (time 1726852706.92 total 276.12 last 147.24) - open/close 35328 (time 1726852716.93 total 286.13 last 132.57) - open/close 36771 (time 1726852726.94 total 296.14 last 144.22) - open/close 38128 (time 1726852736.94 total 306.14 last 135.68) - open/close 39372 (time 1726852746.94 total 316.14 last 124.35) - open/close 40000 (time 1726852751.21 total 320.41 last 147.24) - open/close 41415 (time 1726852761.21 total 330.41 last 141.47) - open/close 42573 (time 1726852771.21 total 340.41 last 115.77) - open/close 43811 (time 1726852781.22 total 350.42 last 123.75) - open/close 45330 (time 1726852791.23 total 360.42 last 151.79) - open/close 46445 (time 1726852801.23 total 370.43 last 111.48) - open/close 47906 (time 1726852811.23 total 380.43 last 146.05) - open/close 49218 (time 1726852821.23 total 390.43 last 131.15) - open/close 50000 (time 1726852827.56 total 396.75 last 123.69) - open/close 51286 (time 1726852837.56 total 406.76 last 128.57) - open/close 52556 (time 1726852847.56 total 416.76 last 126.95) - open/close 53780 (time 1726852857.57 total 426.77 last 122.33) - open/close 55216 (time 1726852867.57 total 436.77 last 143.54) - open/close 56620 (time 1726852877.58 total 446.77 last 140.35) - open/close 57817 (time 1726852887.58 total 456.78 last 119.61) - open/close 59084 (time 1726852897.59 total 466.78 last 126.67) - open/close 60000 (time 1726852904.96 total 474.16 last 124.24) - open/close 61397 (time 1726852914.96 total 484.16 last 139.65) - open/close 62729 (time 1726852924.97 total 494.17 last 133.12) - open/close 63844 (time 1726852934.97 total 504.17 last 111.43) total: 64768 open/close in 512.21 seconds: 126.45 ops/second - /unlink 2176 (time 1726852955.24 total 10.00 last 217.56) - /unlink 4077 (time 1726852965.24 total 20.00 last 190.05) - /unlink 5943 (time 1726852975.24 total 30.01 last 186.52) - /unlink 7889 (time 1726852985.25 total 40.01 last 194.60) - /unlink 10000 (time 1726852995.19 total 49.96 last 212.19) - /unlink 12363 (time 1726853005.20 total 59.96 last 236.21) - /unlink 14702 (time 1726853015.20 total 69.96 last 233.86) - /unlink 16940 (time 1726853025.20 total 79.97 last 223.70) - /unlink 19071 (time 1726853035.21 total 89.97 last 213.03) - /unlink 20000 (time 1726853039.46 total 94.23 last 218.21) - /unlink 22298 (time 1726853049.47 total 104.23 last 229.65) - /unlink 24352 (time 1726853059.47 total 114.24 last 205.34) - /unlink 26761 (time 1726853069.48 total 124.24 last 240.87) - /unlink 28764 (time 1726853079.48 total 134.24 last 200.28) - /unlink 30000 (time 1726853086.13 total 140.90 last 185.71) - /unlink 31857 (time 1726853096.13 total 150.90 last 185.66) - /unlink 33783 (time 1726853106.14 total 160.90 last 192.55) - /unlink 35736 (time 1726853116.14 total 170.90 last 195.30) - /unlink 37724 (time 1726853126.14 total 180.91 last 198.68) - /unlink 40000 (time 1726853135.61 total 190.37 last 240.41) - /unlink 42259 (time 1726853145.61 total 200.37 last 225.89) - /unlink 44288 (time 1726853155.61 total 210.38 last 202.85) - /unlink 46238 (time 1726853165.62 total 220.38 last 194.88) - /unlink 48266 (time 1726853175.62 total 230.39 last 202.72) - /unlink 50000 (time 1726853183.81 total 238.57 last 211.79) - /unlink 51971 (time 1726853193.81 total 248.58 last 197.08) - /unlink 53991 (time 1726853203.81 total 258.58 last 201.99) - /unlink 55910 (time 1726853213.81 total 268.58 last 191.87) - /unlink 58007 (time 1726853223.84 total 278.61 last 209.12) - /unlink 60000 (time 1726853232.44 total 287.21 last 231.70) - /unlink 61985 (time 1726853242.45 total 297.21 last 198.45) - /unlink 64258 (time 1726853252.45 total 307.21 last 227.30) total: 64768 /unlink in 309.41 seconds: 209.33 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1397 (time 1726853282.59 total 10.01 last 139.58) - open/close 2734 (time 1726853292.59 total 20.01 last 133.67) - open/close 4045 (time 1726853302.59 total 30.01 last 131.06) - open/close 5312 (time 1726853312.60 total 40.02 last 126.66) - open/close 6667 (time 1726853322.60 total 50.02 last 135.45) - open/close 7939 (time 1726853332.60 total 60.02 last 127.18) - open/close 9156 (time 1726853342.61 total 70.03 last 121.64) - open/close 10000 (time 1726853349.57 total 76.99 last 121.26) - open/close 11304 (time 1726853359.57 total 86.99 last 130.40) - open/close 12480 (time 1726853369.57 total 96.99 last 117.55) - open/close 13598 (time 1726853379.58 total 107.00 last 111.74) - open/close 14775 (time 1726853389.58 total 117.00 last 117.64) - open/close 16106 (time 1726853399.59 total 127.00 last 133.08) - open/close 17223 (time 1726853409.59 total 137.00 last 111.70) - open/close 18414 (time 1726853419.59 total 147.01 last 119.03) - open/close 19624 (time 1726853429.59 total 157.01 last 120.97) - open/close 20000 (time 1726853432.49 total 159.91 last 129.77) - open/close 21268 (time 1726853442.50 total 169.92 last 126.68) - open/close 22466 (time 1726853452.50 total 179.92 last 119.77) - open/close 23888 (time 1726853462.51 total 189.93 last 142.15) - open/close 25197 (time 1726853472.51 total 199.93 last 130.90) - open/close 26545 (time 1726853482.51 total 209.93 last 134.80) - open/close 27897 (time 1726853492.51 total 219.93 last 135.19) - open/close 29090 (time 1726853502.51 total 229.93 last 119.22) - open/close 30000 (time 1726853509.39 total 236.81 last 132.34) - open/close 31352 (time 1726853519.39 total 246.81 last 135.17) - open/close 32465 (time 1726853529.40 total 256.82 last 111.19) - open/close 33780 (time 1726853539.40 total 266.82 last 131.48) - open/close 35063 (time 1726853549.41 total 276.83 last 128.28) - open/close 36290 (time 1726853559.41 total 286.83 last 122.68) - open/close 37604 (time 1726853569.42 total 296.83 last 131.30) - open/close 38843 (time 1726853579.42 total 306.83 last 123.89) - open/close 40000 (time 1726853588.14 total 315.55 last 132.69) - open/close 41344 (time 1726853598.14 total 325.56 last 134.38) - open/close 42764 (time 1726853608.14 total 335.56 last 141.94) - open/close 44189 (time 1726853618.15 total 345.57 last 142.39) - open/close 45781 (time 1726853628.15 total 355.57 last 159.19) - open/close 47359 (time 1726853638.15 total 365.57 last 157.78) - open/close 48951 (time 1726853648.15 total 375.57 last 159.15) - open/close 50000 (time 1726853654.11 total 381.53 last 176.11) - open/close 51654 (time 1726853664.12 total 391.54 last 165.21) - open/close 53149 (time 1726853674.12 total 401.54 last 149.46) - open/close 54791 (time 1726853684.13 total 411.55 last 164.13) - open/close 56394 (time 1726853694.14 total 421.56 last 160.11) - open/close 58171 (time 1726853704.14 total 431.56 last 177.64) - open/close 60000 (time 1726853713.75 total 441.17 last 190.31) - open/close 62049 (time 1726853723.75 total 451.17 last 204.90) - open/close 63870 (time 1726853733.76 total 461.17 last 182.07) total: 64768 open/close in 465.99 seconds: 138.99 ops/second - /unlink 2689 (time 1726853749.91 total 10.00 last 268.77) - /unlink 5418 (time 1726853759.91 total 20.01 last 272.89) - /unlink 8285 (time 1726853769.91 total 30.01 last 286.64) - /unlink 10000 (time 1726853776.20 total 36.30 last 272.72) - /unlink 12528 (time 1726853786.20 total 46.30 last 252.79) - /unlink 15142 (time 1726853796.20 total 56.30 last 261.34) - /unlink 17996 (time 1726853806.21 total 66.30 last 285.28) - /unlink 20000 (time 1726853814.18 total 74.27 last 251.45) - /unlink 22598 (time 1726853824.18 total 84.27 last 259.80) - /unlink 25385 (time 1726853834.18 total 94.27 last 278.66) - /unlink 27933 (time 1726853844.18 total 104.27 last 254.79) - /unlink 30000 (time 1726853852.60 total 112.70 last 245.43) - /unlink 32500 (time 1726853862.61 total 122.70 last 249.86) - /unlink 35163 (time 1726853872.61 total 132.71 last 266.21) - /unlink 37559 (time 1726853882.62 total 142.71 last 239.48) - /unlink 40000 (time 1726853891.52 total 151.61 last 274.15) - /unlink 42599 (time 1726853901.52 total 161.62 last 259.86) - /unlink 45238 (time 1726853911.52 total 171.62 last 263.85) - /unlink 47986 (time 1726853921.52 total 181.62 last 274.75) - /unlink 50000 (time 1726853928.63 total 188.73 last 283.26) - /unlink 52983 (time 1726853938.64 total 198.73 last 298.26) - /unlink 55698 (time 1726853948.64 total 208.73 last 271.45) - /unlink 58597 (time 1726853958.64 total 218.73 last 289.84) - /unlink 60000 (time 1726853963.51 total 223.60 last 288.25) - /unlink 63227 (time 1726853973.51 total 233.60 last 322.60) total: 64768 /unlink in 237.65 seconds: 272.54 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 1677 (time 1726854007.19 total 10.03 last 167.22) - open/close 3158 (time 1726854017.19 total 20.03 last 148.04) - open/close 4778 (time 1726854027.20 total 30.04 last 161.96) - open/close 6717 (time 1726854037.20 total 40.04 last 193.87) - open/close 8608 (time 1726854047.20 total 50.04 last 189.08) - open/close 10000 (time 1726854054.59 total 57.43 last 188.21) - open/close 11865 (time 1726854064.60 total 67.44 last 186.38) - open/close 13730 (time 1726854074.61 total 77.45 last 186.39) - open/close 15543 (time 1726854084.61 total 87.45 last 181.22) - open/close 17264 (time 1726854094.62 total 97.46 last 172.01) - open/close 19127 (time 1726854104.62 total 107.46 last 186.23) - open/close 20000 (time 1726854110.39 total 113.23 last 151.39) - open/close 21627 (time 1726854120.39 total 123.23 last 162.67) - open/close 23214 (time 1726854130.39 total 133.23 last 158.65) - open/close 24993 (time 1726854140.40 total 143.23 last 177.84) - open/close 26521 (time 1726854150.40 total 153.24 last 152.73) - open/close 28043 (time 1726854160.41 total 163.24 last 152.10) - open/close 29713 (time 1726854170.41 total 173.25 last 166.90) - open/close 31379 (time 1726854180.42 total 183.26 last 166.50) - open/close 33168 (time 1726854190.42 total 193.26 last 178.86) - open/close 34817 (time 1726854200.42 total 203.26 last 164.87) - open/close 36631 (time 1726854210.43 total 213.26 last 181.35) - open/close 38215 (time 1726854220.43 total 223.27 last 158.27) - open/close 39832 (time 1726854230.44 total 233.27 last 161.66) - open/close 41644 (time 1726854240.44 total 243.28 last 181.13) - open/close 43306 (time 1726854250.44 total 253.28 last 166.12) - open/close 45016 (time 1726854260.45 total 263.28 last 170.96) - open/close 46787 (time 1726854270.45 total 273.28 last 177.10) - open/close 48592 (time 1726854280.45 total 283.29 last 180.47) - open/close 50000 (time 1726854289.81 total 292.65 last 150.44) - open/close 51665 (time 1726854299.81 total 302.65 last 166.46) - open/close 53303 (time 1726854309.81 total 312.65 last 163.73) - open/close 54733 (time 1726854319.82 total 322.66 last 142.93) - open/close 56608 (time 1726854329.82 total 332.66 last 187.44) - open/close 58451 (time 1726854339.83 total 342.66 last 184.24) - open/close 60000 (time 1726854346.56 total 349.40 last 230.06) - open/close 62686 (time 1726854356.56 total 359.40 last 268.58) total: 64768 open/close in 368.62 seconds: 175.70 ops/second - /unlink 5013 (time 1726854377.14 total 10.00 last 501.11) - /unlink 10000 (time 1726854387.06 total 19.92 last 502.85) - /unlink 14545 (time 1726854397.06 total 29.92 last 454.42) - /unlink 19781 (time 1726854407.06 total 39.92 last 523.57) - /unlink 24896 (time 1726854417.06 total 49.92 last 511.48) - /unlink 29480 (time 1726854427.06 total 59.93 last 458.25) - /unlink 33561 (time 1726854437.07 total 69.93 last 408.02) - /unlink 37146 (time 1726854447.07 total 79.93 last 358.47) - /unlink 40000 (time 1726854456.74 total 89.60 last 295.16) - /unlink 43255 (time 1726854466.74 total 99.60 last 325.38) - /unlink 45991 (time 1726854476.74 total 109.61 last 273.54) - /unlink 49153 (time 1726854486.74 total 119.61 last 316.14) - /unlink 50000 (time 1726854489.06 total 121.92 last 365.47) - /unlink 54505 (time 1726854499.06 total 131.93 last 450.45) - /unlink 59491 (time 1726854509.06 total 141.93 last 498.57) - /unlink 64481 (time 1726854519.06 total 151.93 last 498.99) total: 64768 /unlink in 152.52 seconds: 424.65 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 2335 (time 1726854547.71 total 10.00 last 233.42) - open/close 4403 (time 1726854557.71 total 20.01 last 206.69) - open/close 6920 (time 1726854567.71 total 30.01 last 251.65) - open/close 9320 (time 1726854577.72 total 40.01 last 239.90) - open/close 10000 (time 1726854580.51 total 42.81 last 243.58) - open/close 12944 (time 1726854590.51 total 52.81 last 294.38) - open/close 15573 (time 1726854600.51 total 62.81 last 262.82) - open/close 17578 (time 1726854610.52 total 72.82 last 200.40) - open/close 20000 (time 1726854619.29 total 81.59 last 276.05) - open/close 22361 (time 1726854629.29 total 91.59 last 236.06) - open/close 25192 (time 1726854639.29 total 101.59 last 283.07) - open/close 28242 (time 1726854649.30 total 111.60 last 304.89) - open/close 30000 (time 1726854655.52 total 117.81 last 282.68) - open/close 33378 (time 1726854665.52 total 127.82 last 337.74) - open/close 37283 (time 1726854675.52 total 137.82 last 390.50) - open/close 40000 (time 1726854682.63 total 144.92 last 382.25) - open/close 43899 (time 1726854692.63 total 154.93 last 389.86) - open/close 47634 (time 1726854702.63 total 164.93 last 373.47) - open/close 50000 (time 1726854707.71 total 170.01 last 465.11) - open/close 54253 (time 1726854717.72 total 180.01 last 425.24) - open/close 58426 (time 1726854727.72 total 190.01 last 417.29) - open/close 60000 (time 1726854730.72 total 193.02 last 523.51) - open/close 63556 (time 1726854740.73 total 203.02 last 355.49) total: 64768 open/close in 206.86 seconds: 313.10 ops/second - /unlink 5739 (time 1726854755.53 total 10.00 last 573.86) - /unlink 10000 (time 1726854761.93 total 16.40 last 666.12) - /unlink 16331 (time 1726854771.93 total 26.40 last 633.08) - /unlink 20000 (time 1726854778.35 total 32.82 last 571.37) - /unlink 27709 (time 1726854788.35 total 42.82 last 770.81) - /unlink 30000 (time 1726854791.18 total 45.65 last 810.37) - /unlink 36633 (time 1726854801.18 total 55.65 last 663.26) - /unlink 40000 (time 1726854806.42 total 60.89 last 642.48) - /unlink 46427 (time 1726854816.42 total 70.89 last 642.58) - /unlink 50000 (time 1726854822.03 total 76.49 last 637.62) - /unlink 57310 (time 1726854832.03 total 86.50 last 730.84) - /unlink 60000 (time 1726854836.31 total 90.78 last 628.11) total: 64768 /unlink in 96.68 seconds: 669.93 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3667 (time 1726854873.99 total 10.00 last 366.67) - open/close 7779 (time 1726854883.99 total 20.00 last 411.16) - open/close 10000 (time 1726854888.71 total 24.72 last 470.38) - open/close 15228 (time 1726854898.71 total 34.72 last 522.77) - open/close 18635 (time 1726854908.72 total 44.73 last 340.57) - open/close 20000 (time 1726854912.35 total 48.36 last 375.43) - open/close 23482 (time 1726854922.35 total 58.37 last 348.15) - open/close 27915 (time 1726854932.35 total 68.37 last 443.28) - open/close 30000 (time 1726854936.35 total 72.36 last 521.58) - open/close 34759 (time 1726854946.35 total 82.36 last 475.86) - open/close 39097 (time 1726854956.35 total 92.37 last 433.72) - open/close 44576 (time 1726854966.35 total 102.37 last 547.89) - open/close 48622 (time 1726854976.36 total 112.37 last 404.51) - open/close 50000 (time 1726854980.11 total 116.13 last 366.77) - open/close 54072 (time 1726854990.11 total 126.13 last 407.17) - open/close 57736 (time 1726855000.12 total 136.13 last 366.33) - open/close 60000 (time 1726855007.09 total 143.10 last 324.75) - open/close 64427 (time 1726855017.09 total 153.10 last 442.70) total: 64768 open/close in 153.70 seconds: 421.39 ops/second - /unlink 8738 (time 1726855028.48 total 10.00 last 873.76) - /unlink 17519 (time 1726855038.48 total 20.00 last 878.08) - /unlink 20000 (time 1726855041.06 total 22.58 last 962.82) - /unlink 29248 (time 1726855051.06 total 32.58 last 924.76) - /unlink 37152 (time 1726855061.06 total 42.58 last 790.37) - /unlink 40000 (time 1726855065.07 total 46.58 last 710.84) - /unlink 47731 (time 1726855075.07 total 56.59 last 773.06) - /unlink 50000 (time 1726855077.94 total 59.46 last 789.53) - /unlink 58100 (time 1726855087.94 total 69.46 last 809.95) - /unlink 60000 (time 1726855090.45 total 71.96 last 758.49) total: 64768 /unlink in 76.88 seconds: 842.43 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg436-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg436-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg436-server stop mds service on oleg436-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg436-server unloading modules on: 'oleg436-server' oleg436-server: oleg436-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-server: ssh exited with exit code 2 pdsh@oleg436-client: oleg436-client: ssh exited with exit code 2