== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:18:54 (1734257934) Stopping clients: oleg207-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg207-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 oleg207-server: oleg207-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing load_modules_local oleg207-server: Loading modules from /home/green/git/lustre-release/lustre oleg207-server: detected 4 online CPUs by sysfs oleg207-server: Force libcfs to create 2 CPU partitions oleg207-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg207-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg207-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg207-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16200 392 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1419 (time 1734258002.70 total 10.00 last 141.88) - open/close 2536 (time 1734258012.71 total 20.00 last 111.69) - open/close 3598 (time 1734258022.71 total 30.01 last 106.17) - open/close 4822 (time 1734258032.71 total 40.01 last 122.35) - open/close 5907 (time 1734258042.72 total 50.01 last 108.45) - open/close 7003 (time 1734258052.72 total 60.02 last 109.56) - open/close 8204 (time 1734258062.73 total 70.02 last 120.04) - open/close 9358 (time 1734258072.73 total 80.03 last 115.34) - open/close 10000 (time 1734258078.89 total 86.19 last 104.26) - open/close 11144 (time 1734258088.90 total 96.19 last 114.32) - open/close 12232 (time 1734258098.90 total 106.20 last 108.76) - open/close 13315 (time 1734258108.91 total 116.20 last 108.22) - open/close 14407 (time 1734258118.91 total 126.20 last 109.19) - open/close 15532 (time 1734258128.91 total 136.21 last 112.48) - open/close 16628 (time 1734258138.91 total 146.21 last 109.57) - open/close 17854 (time 1734258148.92 total 156.22 last 122.48) - open/close 19163 (time 1734258158.92 total 166.22 last 130.90) - open/close 20000 (time 1734258166.54 total 173.84 last 109.90) - open/close 21162 (time 1734258176.54 total 183.84 last 116.16) - open/close 22525 (time 1734258186.55 total 193.84 last 136.25) - open/close 23829 (time 1734258196.55 total 203.84 last 130.36) - open/close 25078 (time 1734258206.55 total 213.85 last 124.84) - open/close 26270 (time 1734258216.56 total 223.86 last 119.13) - open/close 27543 (time 1734258226.56 total 233.86 last 127.25) - open/close 28920 (time 1734258236.57 total 243.86 last 137.66) - open/close 30000 (time 1734258244.68 total 251.97 last 133.17) - open/close 31161 (time 1734258254.68 total 261.97 last 116.07) - open/close 32489 (time 1734258264.68 total 271.98 last 132.75) - open/close 33678 (time 1734258274.69 total 281.98 last 118.83) - open/close 34927 (time 1734258284.69 total 291.99 last 124.87) - open/close 36276 (time 1734258294.69 total 301.99 last 134.90) - open/close 37612 (time 1734258304.69 total 311.99 last 133.57) - open/close 38918 (time 1734258314.69 total 321.99 last 130.58) - open/close 40000 (time 1734258323.12 total 330.42 last 128.42) - open/close 41172 (time 1734258333.12 total 340.42 last 117.19) - open/close 42382 (time 1734258343.12 total 350.42 last 120.98) - open/close 43632 (time 1734258353.12 total 360.42 last 124.99) - open/close 44937 (time 1734258363.12 total 370.42 last 130.50) - open/close 46202 (time 1734258373.13 total 380.42 last 126.44) - open/close 47552 (time 1734258383.13 total 390.43 last 134.97) - open/close 48926 (time 1734258393.13 total 400.43 last 137.33) - open/close 50000 (time 1734258399.69 total 406.98 last 163.88) - open/close 51518 (time 1734258409.69 total 416.99 last 151.76) - open/close 53155 (time 1734258419.69 total 426.99 last 163.67) - open/close 54640 (time 1734258429.70 total 437.00 last 148.42) - open/close 56122 (time 1734258439.70 total 447.00 last 148.17) - open/close 57570 (time 1734258449.70 total 457.00 last 144.77) - open/close 58953 (time 1734258459.71 total 467.01 last 138.19) - open/close 60000 (time 1734258466.70 total 474.00 last 149.70) - open/close 61266 (time 1734258476.71 total 484.01 last 126.51) - open/close 62388 (time 1734258486.71 total 494.01 last 112.18) - open/close 63720 (time 1734258496.72 total 504.01 last 133.15) total: 64768 open/close in 512.17 seconds: 126.46 ops/second - /unlink 2153 (time 1734258516.58 total 10.00 last 215.29) - /unlink 4449 (time 1734258526.58 total 20.00 last 229.55) - /unlink 6214 (time 1734258536.58 total 30.01 last 176.44) - /unlink 8236 (time 1734258546.58 total 40.01 last 202.18) - /unlink 10000 (time 1734258553.76 total 47.18 last 245.95) - /unlink 12038 (time 1734258563.76 total 57.18 last 203.78) - /unlink 14053 (time 1734258573.76 total 67.18 last 201.44) - /unlink 16390 (time 1734258583.76 total 77.19 last 233.62) - /unlink 18456 (time 1734258593.77 total 87.19 last 206.55) - /unlink 20000 (time 1734258600.48 total 93.91 last 229.78) - /unlink 22331 (time 1734258610.49 total 103.92 last 232.93) - /unlink 24640 (time 1734258620.49 total 113.92 last 230.87) - /unlink 26904 (time 1734258630.50 total 123.92 last 226.32) - /unlink 28819 (time 1734258640.50 total 133.92 last 191.48) - /unlink 30000 (time 1734258646.28 total 139.71 last 204.11) - /unlink 32267 (time 1734258656.28 total 149.71 last 226.68) - /unlink 34234 (time 1734258666.29 total 159.71 last 196.66) - /unlink 36256 (time 1734258676.29 total 169.71 last 202.16) - /unlink 38989 (time 1734258686.29 total 179.72 last 273.24) - /unlink 40000 (time 1734258690.24 total 183.66 last 256.35) - /unlink 42451 (time 1734258700.24 total 193.66 last 245.05) - /unlink 45442 (time 1734258710.24 total 203.66 last 299.09) - /unlink 47904 (time 1734258720.24 total 213.66 last 246.14) - /unlink 50000 (time 1734258729.56 total 222.98 last 224.97) - /unlink 52408 (time 1734258739.56 total 232.98 last 240.71) - /unlink 54643 (time 1734258749.57 total 242.99 last 223.31) - /unlink 57596 (time 1734258759.57 total 253.00 last 295.25) - /unlink 60000 (time 1734258767.30 total 260.72 last 311.01) - /unlink 62923 (time 1734258777.30 total 270.73 last 292.20) total: 64768 /unlink in 277.07 seconds: 233.76 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1301 (time 1734258812.72 total 10.01 last 130.02) - open/close 2680 (time 1734258822.72 total 20.01 last 137.84) - open/close 4273 (time 1734258832.72 total 30.01 last 159.28) - open/close 5644 (time 1734258842.73 total 40.01 last 137.06) - open/close 7398 (time 1734258852.73 total 50.02 last 175.37) - open/close 9328 (time 1734258862.73 total 60.02 last 192.92) - open/close 10000 (time 1734258866.13 total 63.41 last 198.04) - open/close 12334 (time 1734258876.13 total 73.42 last 233.34) - open/close 14080 (time 1734258886.13 total 83.42 last 174.57) - open/close 15966 (time 1734258896.13 total 93.42 last 188.58) - open/close 17497 (time 1734258906.13 total 103.42 last 153.05) - open/close 19035 (time 1734258916.14 total 113.43 last 153.74) - open/close 20000 (time 1734258922.99 total 120.28 last 140.83) - open/close 21689 (time 1734258932.99 total 130.28 last 168.85) - open/close 23471 (time 1734258942.99 total 140.28 last 178.17) - open/close 25330 (time 1734258952.99 total 150.28 last 185.90) - open/close 27416 (time 1734258963.00 total 160.28 last 208.60) - open/close 29163 (time 1734258973.00 total 170.29 last 174.66) - open/close 30000 (time 1734258977.75 total 175.04 last 176.20) - open/close 31844 (time 1734258987.75 total 185.04 last 184.34) - open/close 33810 (time 1734258997.75 total 195.04 last 196.54) - open/close 35879 (time 1734259007.75 total 205.04 last 206.88) - open/close 37832 (time 1734259017.76 total 215.04 last 195.27) - open/close 39393 (time 1734259027.76 total 225.05 last 156.03) - open/close 40000 (time 1734259031.62 total 228.91 last 157.13) - open/close 41651 (time 1734259041.63 total 238.92 last 164.96) - open/close 43430 (time 1734259051.64 total 248.93 last 177.80) - open/close 45056 (time 1734259061.64 total 258.93 last 162.56) - open/close 46872 (time 1734259071.64 total 268.93 last 181.56) - open/close 48644 (time 1734259081.65 total 278.94 last 177.07) - open/close 50000 (time 1734259090.21 total 287.50 last 158.33) - open/close 51688 (time 1734259100.22 total 297.51 last 168.72) - open/close 53240 (time 1734259110.23 total 307.51 last 155.08) - open/close 55238 (time 1734259120.36 total 317.65 last 197.18) - open/close 56987 (time 1734259130.36 total 327.65 last 174.87) - open/close 58573 (time 1734259140.37 total 337.65 last 158.52) - open/close 60000 (time 1734259148.32 total 345.61 last 179.35) - open/close 61663 (time 1734259158.33 total 355.61 last 166.23) - open/close 63528 (time 1734259168.33 total 365.62 last 186.41) total: 64768 open/close in 372.86 seconds: 173.71 ops/second - /unlink 2856 (time 1734259186.90 total 10.00 last 285.55) - /unlink 6313 (time 1734259196.90 total 20.01 last 345.59) - /unlink 9059 (time 1734259206.90 total 30.01 last 274.55) - /unlink 10000 (time 1734259209.67 total 32.78 last 339.67) - /unlink 13495 (time 1734259219.67 total 42.78 last 349.44) - /unlink 16057 (time 1734259229.68 total 52.78 last 256.14) - /unlink 18820 (time 1734259239.68 total 62.78 last 276.23) - /unlink 20000 (time 1734259243.05 total 66.15 last 350.42) - /unlink 23212 (time 1734259253.05 total 76.15 last 321.15) - /unlink 26361 (time 1734259263.05 total 86.15 last 314.86) - /unlink 29685 (time 1734259273.05 total 96.16 last 332.33) - /unlink 33750 (time 1734259283.05 total 106.16 last 406.43) - /unlink 37708 (time 1734259293.06 total 116.16 last 395.70) - /unlink 40000 (time 1734259300.19 total 123.30 last 321.16) - /unlink 43080 (time 1734259310.19 total 133.30 last 307.94) - /unlink 45789 (time 1734259320.20 total 143.30 last 270.84) - /unlink 48906 (time 1734259330.20 total 153.30 last 311.61) - /unlink 50000 (time 1734259333.66 total 156.76 last 316.40) - /unlink 53408 (time 1734259343.66 total 166.76 last 340.69) - /unlink 56961 (time 1734259353.66 total 176.77 last 355.26) - /unlink 60000 (time 1734259361.25 total 184.35 last 400.50) - /unlink 63288 (time 1734259371.25 total 194.36 last 328.73) total: 64768 /unlink in 198.77 seconds: 325.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1932 (time 1734259404.16 total 10.00 last 193.19) - open/close 4012 (time 1734259414.17 total 20.01 last 207.90) - open/close 5850 (time 1734259424.17 total 30.01 last 183.72) - open/close 7726 (time 1734259434.18 total 40.01 last 187.51) - open/close 9386 (time 1734259444.18 total 50.02 last 165.96) - open/close 10000 (time 1734259447.85 total 53.69 last 167.22) - open/close 11605 (time 1734259457.86 total 63.70 last 160.40) - open/close 13496 (time 1734259467.86 total 73.70 last 189.06) - open/close 15555 (time 1734259477.86 total 83.70 last 205.82) - open/close 17312 (time 1734259487.87 total 93.70 last 175.63) - open/close 19216 (time 1734259497.87 total 103.71 last 190.30) - open/close 20000 (time 1734259502.23 total 108.06 last 180.16) - open/close 21815 (time 1734259512.23 total 118.06 last 181.45) - open/close 23767 (time 1734259522.23 total 128.07 last 195.15) - open/close 25545 (time 1734259532.24 total 138.07 last 177.72) - open/close 27292 (time 1734259542.24 total 148.07 last 174.66) - open/close 29120 (time 1734259552.24 total 158.08 last 182.74) - open/close 30000 (time 1734259557.11 total 162.95 last 180.76) - open/close 31984 (time 1734259567.11 total 172.95 last 198.40) - open/close 33914 (time 1734259577.11 total 182.95 last 192.95) - open/close 35888 (time 1734259587.11 total 192.95 last 197.36) - open/close 38058 (time 1734259597.12 total 202.96 last 216.89) - open/close 40000 (time 1734259606.30 total 212.14 last 211.49) - open/close 42140 (time 1734259616.31 total 222.14 last 213.90) - open/close 44252 (time 1734259626.31 total 232.14 last 211.19) - open/close 46640 (time 1734259636.31 total 242.15 last 238.71) - open/close 49068 (time 1734259646.31 total 252.15 last 242.78) - open/close 50000 (time 1734259650.53 total 256.36 last 221.14) - open/close 52233 (time 1734259660.53 total 266.36 last 223.28) - open/close 54025 (time 1734259670.53 total 276.37 last 179.10) - open/close 56118 (time 1734259680.53 total 286.37 last 209.25) - open/close 58345 (time 1734259690.54 total 296.37 last 222.66) - open/close 60000 (time 1734259697.41 total 303.24 last 240.92) - open/close 62803 (time 1734259707.41 total 313.24 last 280.27) total: 64768 open/close in 322.08 seconds: 201.09 ops/second - /unlink 4056 (time 1734259727.60 total 10.00 last 405.51) - /unlink 7804 (time 1734259737.60 total 20.00 last 374.74) - /unlink 10000 (time 1734259744.66 total 27.06 last 311.31) - /unlink 13897 (time 1734259754.66 total 37.06 last 389.58) - /unlink 17085 (time 1734259764.66 total 47.06 last 318.74) - /unlink 20000 (time 1734259774.19 total 56.59 last 306.05) - /unlink 23235 (time 1734259784.19 total 66.59 last 323.47) - /unlink 26683 (time 1734259794.19 total 76.59 last 344.75) - /unlink 30000 (time 1734259802.32 total 84.72 last 408.01) - /unlink 34405 (time 1734259812.32 total 94.72 last 440.45) - /unlink 38781 (time 1734259822.32 total 104.72 last 437.59) - /unlink 40000 (time 1734259824.82 total 107.22 last 487.85) - /unlink 44122 (time 1734259834.82 total 117.22 last 412.12) - /unlink 48331 (time 1734259844.82 total 127.22 last 420.86) - /unlink 50000 (time 1734259848.51 total 130.91 last 452.45) - /unlink 53210 (time 1734259858.51 total 140.91 last 320.97) - /unlink 57025 (time 1734259868.51 total 150.91 last 381.46) - /unlink 60000 (time 1734259876.34 total 158.74 last 380.23) - /unlink 63721 (time 1734259886.34 total 168.74 last 372.07) total: 64768 /unlink in 171.24 seconds: 378.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2316 (time 1734259920.08 total 10.01 last 231.38) - open/close 5145 (time 1734259930.08 total 20.01 last 282.82) - open/close 8086 (time 1734259940.09 total 30.02 last 294.00) - open/close 10000 (time 1734259946.93 total 36.86 last 279.79) - open/close 13042 (time 1734259956.93 total 46.86 last 304.07) - open/close 15935 (time 1734259966.93 total 56.86 last 289.25) - open/close 18960 (time 1734259976.94 total 66.87 last 302.34) - open/close 20000 (time 1734259980.15 total 70.08 last 323.62) - open/close 23101 (time 1734259990.15 total 80.08 last 310.06) - open/close 25880 (time 1734260000.16 total 90.09 last 277.85) - open/close 29056 (time 1734260010.16 total 100.09 last 317.55) - open/close 30000 (time 1734260013.72 total 103.65 last 265.18) - open/close 32989 (time 1734260023.72 total 113.65 last 298.83) - open/close 36078 (time 1734260033.72 total 123.65 last 308.81) - open/close 39052 (time 1734260043.73 total 133.66 last 297.28) - open/close 40000 (time 1734260047.74 total 137.67 last 235.95) - open/close 42731 (time 1734260057.75 total 147.67 last 273.07) - open/close 46030 (time 1734260067.75 total 157.68 last 329.84) - open/close 49196 (time 1734260077.75 total 167.68 last 316.52) - open/close 50000 (time 1734260080.74 total 170.67 last 268.69) - open/close 52782 (time 1734260090.74 total 180.67 last 278.13) - open/close 55634 (time 1734260100.75 total 190.68 last 285.15) - open/close 58398 (time 1734260110.75 total 200.68 last 276.37) - open/close 60000 (time 1734260116.04 total 205.97 last 302.53) - open/close 62715 (time 1734260126.05 total 215.97 last 271.43) total: 64768 open/close in 223.53 seconds: 289.75 ops/second - /unlink 5238 (time 1734260144.66 total 10.00 last 523.77) - /unlink 10000 (time 1734260153.86 total 19.20 last 517.54) - /unlink 15819 (time 1734260163.86 total 29.20 last 581.83) - /unlink 20000 (time 1734260171.42 total 36.76 last 552.93) - /unlink 25206 (time 1734260181.43 total 46.77 last 520.57) - /unlink 29716 (time 1734260191.43 total 56.77 last 450.93) - /unlink 34873 (time 1734260201.43 total 66.77 last 515.69) - /unlink 40000 (time 1734260211.16 total 76.50 last 526.81) - /unlink 44935 (time 1734260221.16 total 86.50 last 493.50) - /unlink 49220 (time 1734260231.16 total 96.50 last 428.43) - /unlink 54362 (time 1734260241.16 total 106.50 last 514.15) - /unlink 59703 (time 1734260251.16 total 116.50 last 534.09) total: 64768 /unlink in 126.21 seconds: 513.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2993 (time 1734260290.45 total 10.00 last 299.23) - open/close 5901 (time 1734260300.45 total 20.00 last 290.79) - open/close 8325 (time 1734260310.45 total 30.00 last 242.40) - open/close 10000 (time 1734260317.09 total 36.64 last 252.33) - open/close 12549 (time 1734260327.09 total 46.64 last 254.86) - open/close 15167 (time 1734260337.09 total 56.64 last 261.79) - open/close 17531 (time 1734260347.09 total 66.65 last 236.35) - open/close 19930 (time 1734260357.09 total 76.65 last 239.89) - open/close 22477 (time 1734260367.10 total 86.65 last 254.66) - open/close 25113 (time 1734260377.10 total 96.65 last 263.52) - open/close 27613 (time 1734260387.10 total 106.66 last 249.85) - open/close 30000 (time 1734260395.92 total 115.47 last 270.89) - open/close 32970 (time 1734260405.92 total 125.47 last 296.96) - open/close 35726 (time 1734260415.92 total 135.47 last 275.56) - open/close 38490 (time 1734260425.92 total 145.47 last 276.32) - open/close 40000 (time 1734260431.29 total 150.84 last 281.32) - open/close 42470 (time 1734260441.29 total 160.84 last 246.98) - open/close 45722 (time 1734260451.29 total 170.84 last 325.20) - open/close 49064 (time 1734260461.29 total 180.84 last 334.11) - open/close 50000 (time 1734260464.35 total 183.90 last 305.93) - open/close 52822 (time 1734260474.36 total 193.91 last 282.14) - open/close 55423 (time 1734260484.36 total 203.91 last 259.99) - open/close 57886 (time 1734260494.36 total 213.91 last 246.28) - open/close 60000 (time 1734260501.58 total 221.13 last 292.83) - open/close 63096 (time 1734260511.58 total 231.13 last 309.57) total: 64768 open/close in 237.55 seconds: 272.65 ops/second - /unlink 4718 (time 1734260529.05 total 10.00 last 471.75) - /unlink 9607 (time 1734260539.05 total 20.00 last 488.89) - /unlink 14541 (time 1734260549.05 total 30.00 last 493.38) - /unlink 19390 (time 1734260559.06 total 40.00 last 484.80) - /unlink 24434 (time 1734260569.06 total 50.01 last 504.29) - /unlink 29895 (time 1734260579.06 total 60.01 last 546.05) - /unlink 35472 (time 1734260589.06 total 70.01 last 557.69) - /unlink 40000 (time 1734260595.71 total 76.65 last 681.34) - /unlink 46329 (time 1734260605.71 total 86.65 last 632.80) - /unlink 50000 (time 1734260612.52 total 93.47 last 539.00) - /unlink 55652 (time 1734260622.52 total 103.47 last 565.20) - /unlink 60000 (time 1734260629.27 total 110.21 last 644.37) total: 64768 /unlink in 117.29 seconds: 552.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2