== conf-sanity-1 test 106: check osp llog processing when catalog is wrapped ========================================================== 05:17:58 (1734257878) Stopping clients: oleg245-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg245-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 oleg245-server: oleg245-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing load_modules_local oleg245-server: Loading modules from /home/green/git/lustre-release/lustre oleg245-server: detected 4 online CPUs by sysfs oleg245-server: Force libcfs to create 2 CPU partitions oleg245-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg245-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg245-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg245-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg245-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg245-server: oleg245-server.virtnet: executing set_default_debug -1 all pdsh@oleg245-client: oleg245-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg245-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg245-client.virtnet: -o user_xattr,flock 192.168.202.145@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16200 392 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1743 (time 1734257949.42 total 10.00 last 174.29) - open/close 3440 (time 1734257959.42 total 20.00 last 169.69) - open/close 5193 (time 1734257969.43 total 30.01 last 175.19) - open/close 7056 (time 1734257979.44 total 40.01 last 186.19) - open/close 9003 (time 1734257989.44 total 50.02 last 194.65) - open/close 10000 (time 1734257995.52 total 56.10 last 163.99) - open/close 11546 (time 1734258005.52 total 66.10 last 154.54) - open/close 12750 (time 1734258015.52 total 76.10 last 120.40) - open/close 14037 (time 1734258025.53 total 86.10 last 128.64) - open/close 15404 (time 1734258035.53 total 96.11 last 136.60) - open/close 16678 (time 1734258045.54 total 106.12 last 127.32) - open/close 18016 (time 1734258055.55 total 116.12 last 133.72) - open/close 19383 (time 1734258065.55 total 126.13 last 136.63) - open/close 20000 (time 1734258069.88 total 130.46 last 142.40) - open/close 21296 (time 1734258079.89 total 140.47 last 129.49) - open/close 22685 (time 1734258089.90 total 150.48 last 138.82) - open/close 24035 (time 1734258099.90 total 160.48 last 134.98) - open/close 25311 (time 1734258109.91 total 170.48 last 127.52) - open/close 26597 (time 1734258119.91 total 180.49 last 128.52) - open/close 27979 (time 1734258129.91 total 190.49 last 138.20) - open/close 29451 (time 1734258139.91 total 200.49 last 147.16) - open/close 30000 (time 1734258143.58 total 204.15 last 149.91) - open/close 31522 (time 1734258153.59 total 214.16 last 152.06) - open/close 33090 (time 1734258163.59 total 224.17 last 156.76) - open/close 34531 (time 1734258173.59 total 234.17 last 144.03) - open/close 35989 (time 1734258183.59 total 244.17 last 145.80) - open/close 37760 (time 1734258193.59 total 254.17 last 177.09) - open/close 39319 (time 1734258203.59 total 264.17 last 155.90) - open/close 40000 (time 1734258207.69 total 268.27 last 166.24) - open/close 41455 (time 1734258217.69 total 278.27 last 145.46) - open/close 43170 (time 1734258227.70 total 288.27 last 171.46) - open/close 44861 (time 1734258237.70 total 298.28 last 169.01) - open/close 46480 (time 1734258247.70 total 308.28 last 161.87) - open/close 48053 (time 1734258257.71 total 318.28 last 157.24) - open/close 49673 (time 1734258267.72 total 328.30 last 161.82) - open/close 50000 (time 1734258269.90 total 330.48 last 149.78) - open/close 51575 (time 1734258279.90 total 340.48 last 157.50) - open/close 53205 (time 1734258289.91 total 350.49 last 162.89) - open/close 54890 (time 1734258299.91 total 360.49 last 168.46) - open/close 56608 (time 1734258309.91 total 370.49 last 171.74) - open/close 58164 (time 1734258319.92 total 380.50 last 155.53) - open/close 59698 (time 1734258329.92 total 390.50 last 153.34) - open/close 60000 (time 1734258332.08 total 392.66 last 139.82) - open/close 61517 (time 1734258342.09 total 402.66 last 151.66) - open/close 62998 (time 1734258352.09 total 412.67 last 148.06) - open/close 64639 (time 1734258362.09 total 422.67 last 164.08) total: 64768 open/close in 423.46 seconds: 152.95 ops/second - /unlink 2701 (time 1734258374.33 total 10.00 last 270.05) - /unlink 5833 (time 1734258384.33 total 20.00 last 313.11) - /unlink 9147 (time 1734258394.33 total 30.01 last 331.37) - /unlink 10000 (time 1734258396.65 total 32.33 last 367.34) - /unlink 13665 (time 1734258406.65 total 42.33 last 366.50) - /unlink 17168 (time 1734258416.66 total 52.33 last 350.21) - /unlink 20000 (time 1734258423.77 total 59.45 last 398.01) - /unlink 23223 (time 1734258433.77 total 69.45 last 322.25) - /unlink 26747 (time 1734258443.77 total 79.45 last 352.33) - /unlink 29668 (time 1734258453.78 total 89.45 last 292.05) - /unlink 32919 (time 1734258463.78 total 99.45 last 325.07) - /unlink 35677 (time 1734258473.78 total 109.45 last 275.74) - /unlink 38202 (time 1734258483.78 total 119.45 last 252.48) - /unlink 40000 (time 1734258490.53 total 126.21 last 266.25) - /unlink 42942 (time 1734258500.54 total 136.21 last 294.06) - /unlink 45412 (time 1734258510.54 total 146.21 last 246.99) - /unlink 48064 (time 1734258520.54 total 156.22 last 265.11) - /unlink 50000 (time 1734258527.76 total 163.43 last 268.39) - /unlink 52167 (time 1734258537.76 total 173.43 last 216.66) - /unlink 54745 (time 1734258547.76 total 183.44 last 257.66) - /unlink 57432 (time 1734258557.76 total 193.44 last 268.66) - /unlink 60000 (time 1734258567.70 total 203.37 last 258.45) - /unlink 62707 (time 1734258577.70 total 213.38 last 270.61) total: 64768 /unlink in 219.88 seconds: 294.55 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1572 (time 1734258613.26 total 10.00 last 157.18) - open/close 3155 (time 1734258623.27 total 20.01 last 158.22) - open/close 4524 (time 1734258633.27 total 30.01 last 136.85) - open/close 5850 (time 1734258643.27 total 40.01 last 132.60) - open/close 7266 (time 1734258653.28 total 50.02 last 141.52) - open/close 8751 (time 1734258663.28 total 60.02 last 148.43) - open/close 10000 (time 1734258672.16 total 68.90 last 140.71) - open/close 11774 (time 1734258682.16 total 78.90 last 177.37) - open/close 13525 (time 1734258692.16 total 88.90 last 175.02) - open/close 15374 (time 1734258702.17 total 98.91 last 184.85) - open/close 17339 (time 1734258712.17 total 108.91 last 196.49) - open/close 19065 (time 1734258722.17 total 118.91 last 172.49) - open/close 20000 (time 1734258728.15 total 124.88 last 156.58) - open/close 21695 (time 1734258738.15 total 134.89 last 169.42) - open/close 23378 (time 1734258748.15 total 144.89 last 168.23) - open/close 25176 (time 1734258758.16 total 154.89 last 179.78) - open/close 27399 (time 1734258768.16 total 164.90 last 222.26) - open/close 29324 (time 1734258778.16 total 174.90 last 192.42) - open/close 30000 (time 1734258782.11 total 178.85 last 171.25) - open/close 32263 (time 1734258792.11 total 188.85 last 226.29) - open/close 34352 (time 1734258802.11 total 198.85 last 208.88) - open/close 35973 (time 1734258812.11 total 208.85 last 162.05) - open/close 37652 (time 1734258822.12 total 218.86 last 167.75) - open/close 39563 (time 1734258832.12 total 228.86 last 191.09) - open/close 40000 (time 1734258834.57 total 231.31 last 178.46) - open/close 41731 (time 1734258844.57 total 241.31 last 173.06) - open/close 43830 (time 1734258854.58 total 251.32 last 209.80) - open/close 46302 (time 1734258864.58 total 261.32 last 247.20) - open/close 49131 (time 1734258874.58 total 271.32 last 282.84) - open/close 50000 (time 1734258878.26 total 275.00 last 236.30) - open/close 52064 (time 1734258888.26 total 285.00 last 206.32) - open/close 54395 (time 1734258898.26 total 295.00 last 233.06) - open/close 56279 (time 1734258908.27 total 305.00 last 188.38) - open/close 58008 (time 1734258918.27 total 315.00 last 172.89) - open/close 59984 (time 1734258928.27 total 325.01 last 197.54) - open/close 62143 (time 1734258938.27 total 335.01 last 215.86) - open/close 64624 (time 1734258948.27 total 345.01 last 248.05) total: 64768 open/close in 345.71 seconds: 187.35 ops/second - /unlink 4553 (time 1734258960.10 total 10.00 last 455.29) - /unlink 8630 (time 1734258970.10 total 20.00 last 407.69) - /unlink 10000 (time 1734258973.50 total 23.40 last 402.73) - /unlink 13937 (time 1734258983.51 total 33.41 last 393.58) - /unlink 18222 (time 1734258993.51 total 43.41 last 428.49) - /unlink 20000 (time 1734258997.21 total 47.11 last 479.43) - /unlink 24828 (time 1734259007.21 total 57.11 last 482.80) - /unlink 29353 (time 1734259017.22 total 67.12 last 452.43) - /unlink 32891 (time 1734259027.22 total 77.12 last 353.73) - /unlink 36387 (time 1734259037.22 total 87.12 last 349.59) - /unlink 40000 (time 1734259046.57 total 96.47 last 386.27) - /unlink 43770 (time 1734259056.57 total 106.47 last 376.96) - /unlink 48001 (time 1734259066.57 total 116.47 last 423.07) - /unlink 50000 (time 1734259071.39 total 121.29 last 414.83) - /unlink 54156 (time 1734259081.40 total 131.29 last 415.50) - /unlink 58100 (time 1734259091.40 total 141.29 last 394.39) - /unlink 60000 (time 1734259096.16 total 146.06 last 398.51) - /unlink 63832 (time 1734259106.16 total 156.06 last 383.18) total: 64768 /unlink in 158.35 seconds: 409.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2314 (time 1734259134.11 total 10.00 last 231.34) - open/close 4603 (time 1734259144.11 total 20.01 last 228.83) - open/close 6783 (time 1734259154.12 total 30.01 last 217.93) - open/close 9142 (time 1734259164.12 total 40.01 last 235.86) - open/close 10000 (time 1734259167.85 total 43.74 last 229.89) - open/close 12197 (time 1734259177.85 total 53.75 last 219.63) - open/close 14327 (time 1734259187.86 total 63.75 last 212.94) - open/close 16718 (time 1734259197.86 total 73.75 last 239.07) - open/close 18815 (time 1734259207.86 total 83.75 last 209.64) - open/close 20000 (time 1734259212.57 total 88.46 last 251.50) - open/close 22317 (time 1734259222.57 total 98.47 last 231.67) - open/close 24272 (time 1734259232.58 total 108.47 last 195.46) - open/close 26290 (time 1734259242.58 total 118.47 last 201.79) - open/close 28547 (time 1734259252.58 total 128.47 last 225.67) - open/close 30000 (time 1734259259.30 total 135.19 last 216.19) - open/close 32141 (time 1734259269.30 total 145.19 last 214.09) - open/close 34764 (time 1734259279.30 total 155.19 last 262.26) - open/close 37458 (time 1734259289.30 total 165.19 last 269.37) - open/close 39898 (time 1734259299.30 total 175.20 last 243.94) - open/close 42020 (time 1734259309.31 total 185.20 last 212.18) - open/close 43852 (time 1734259319.31 total 195.20 last 183.16) - open/close 45944 (time 1734259329.31 total 205.20 last 209.13) - open/close 48118 (time 1734259339.31 total 215.20 last 217.37) - open/close 50000 (time 1734259346.65 total 222.54 last 256.65) - open/close 52462 (time 1734259356.65 total 232.54 last 246.18) - open/close 54874 (time 1734259366.65 total 242.54 last 241.12) - open/close 57044 (time 1734259376.65 total 252.54 last 216.93) - open/close 59445 (time 1734259386.66 total 262.55 last 240.01) - open/close 60000 (time 1734259388.98 total 264.88 last 238.48) - open/close 62326 (time 1734259398.99 total 274.88 last 232.48) - open/close 64587 (time 1734259408.99 total 284.88 last 226.00) total: 64768 open/close in 285.63 seconds: 226.75 ops/second - /unlink 4119 (time 1734259420.94 total 10.00 last 411.84) - /unlink 7749 (time 1734259430.95 total 20.00 last 362.96) - /unlink 10000 (time 1734259436.55 total 25.60 last 401.89) - /unlink 13586 (time 1734259446.55 total 35.61 last 358.54) - /unlink 17328 (time 1734259456.55 total 45.61 last 374.14) - /unlink 20000 (time 1734259463.31 total 52.37 last 395.06) - /unlink 24546 (time 1734259473.31 total 62.37 last 454.59) - /unlink 28633 (time 1734259483.31 total 72.37 last 408.69) - /unlink 30000 (time 1734259486.83 total 75.89 last 388.64) - /unlink 34275 (time 1734259496.83 total 85.89 last 427.47) - /unlink 37941 (time 1734259506.83 total 95.89 last 366.57) - /unlink 40000 (time 1734259511.41 total 100.47 last 449.79) - /unlink 43949 (time 1734259521.41 total 110.47 last 394.83) - /unlink 47677 (time 1734259531.41 total 120.47 last 372.80) - /unlink 50000 (time 1734259537.35 total 126.41 last 391.01) - /unlink 53597 (time 1734259547.35 total 136.41 last 359.69) - /unlink 57514 (time 1734259557.35 total 146.41 last 391.65) - /unlink 60000 (time 1734259563.31 total 152.37 last 417.43) - /unlink 64204 (time 1734259573.31 total 162.37 last 420.35) total: 64768 /unlink in 164.10 seconds: 394.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2643 (time 1734259604.90 total 10.00 last 264.24) - open/close 5129 (time 1734259614.90 total 20.01 last 248.51) - open/close 7779 (time 1734259624.90 total 30.01 last 264.98) - open/close 10000 (time 1734259632.06 total 37.17 last 310.24) - open/close 12915 (time 1734259642.06 total 47.17 last 291.41) - open/close 15805 (time 1734259652.06 total 57.17 last 288.98) - open/close 18441 (time 1734259662.07 total 67.17 last 263.49) - open/close 20000 (time 1734259668.69 total 73.79 last 235.55) - open/close 22585 (time 1734259678.69 total 83.79 last 258.48) - open/close 25234 (time 1734259688.69 total 93.79 last 264.84) - open/close 28281 (time 1734259698.69 total 103.79 last 304.70) - open/close 30000 (time 1734259704.24 total 109.34 last 309.76) - open/close 32683 (time 1734259714.24 total 119.35 last 268.22) - open/close 35143 (time 1734259724.25 total 129.35 last 245.93) - open/close 37662 (time 1734259734.25 total 139.35 last 251.89) - open/close 40000 (time 1734259744.18 total 149.29 last 235.28) - open/close 42549 (time 1734259754.18 total 159.29 last 254.90) - open/close 44961 (time 1734259764.18 total 169.29 last 241.17) - open/close 46991 (time 1734259774.19 total 179.29 last 202.93) - open/close 49209 (time 1734259784.19 total 189.30 last 221.73) - open/close 50000 (time 1734259787.54 total 192.64 last 236.51) - open/close 52355 (time 1734259797.54 total 202.64 last 235.49) - open/close 55347 (time 1734259807.54 total 212.64 last 299.15) - open/close 58220 (time 1734259817.54 total 222.64 last 287.23) - open/close 60000 (time 1734259824.24 total 229.34 last 265.75) - open/close 62843 (time 1734259834.24 total 239.34 last 284.28) total: 64768 open/close in 246.92 seconds: 262.30 ops/second - /unlink 4747 (time 1734259852.85 total 10.00 last 474.66) - /unlink 9042 (time 1734259862.85 total 20.00 last 429.44) - /unlink 13580 (time 1734259872.85 total 30.00 last 453.76) - /unlink 17925 (time 1734259882.85 total 40.00 last 434.49) - /unlink 20000 (time 1734259886.81 total 43.96 last 524.22) - /unlink 24965 (time 1734259896.81 total 53.96 last 496.43) - /unlink 29731 (time 1734259906.81 total 63.96 last 476.54) - /unlink 34716 (time 1734259916.81 total 73.96 last 498.49) - /unlink 40000 (time 1734259926.69 total 83.84 last 535.09) - /unlink 45599 (time 1734259936.69 total 93.84 last 559.86) - /unlink 50000 (time 1734259944.83 total 101.98 last 540.65) - /unlink 55824 (time 1734259954.83 total 111.98 last 582.31) - /unlink 60000 (time 1734259962.70 total 119.85 last 530.75) total: 64768 /unlink in 127.73 seconds: 507.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2898 (time 1734260000.00 total 10.00 last 289.70) - open/close 6167 (time 1734260010.00 total 20.00 last 326.90) - open/close 9074 (time 1734260020.00 total 30.01 last 290.64) - open/close 10000 (time 1734260023.11 total 33.11 last 298.18) - open/close 13196 (time 1734260033.11 total 43.11 last 319.56) - open/close 16185 (time 1734260043.11 total 53.11 last 298.89) - open/close 18683 (time 1734260053.11 total 63.11 last 249.79) - open/close 20000 (time 1734260057.64 total 67.65 last 290.60) - open/close 23275 (time 1734260067.64 total 77.65 last 327.48) - open/close 26366 (time 1734260077.64 total 87.65 last 309.06) - open/close 29119 (time 1734260087.65 total 97.65 last 275.24) - open/close 30000 (time 1734260090.73 total 100.73 last 285.65) - open/close 32746 (time 1734260100.73 total 110.73 last 274.59) - open/close 35620 (time 1734260110.73 total 120.74 last 287.37) - open/close 38599 (time 1734260120.74 total 130.74 last 297.83) - open/close 40000 (time 1734260125.74 total 135.74 last 280.20) - open/close 42739 (time 1734260135.74 total 145.74 last 273.82) - open/close 45704 (time 1734260145.74 total 155.74 last 296.48) - open/close 48508 (time 1734260155.74 total 165.74 last 280.37) - open/close 50000 (time 1734260160.37 total 170.37 last 322.34) - open/close 52940 (time 1734260170.37 total 180.37 last 293.91) - open/close 55691 (time 1734260180.37 total 190.37 last 275.10) - open/close 58044 (time 1734260190.37 total 200.38 last 235.27) - open/close 60000 (time 1734260198.16 total 208.17 last 251.03) - open/close 62916 (time 1734260208.17 total 218.17 last 291.57) total: 64768 open/close in 225.42 seconds: 287.33 ops/second - /unlink 4473 (time 1734260226.43 total 10.00 last 447.23) - /unlink 8939 (time 1734260236.43 total 20.00 last 446.57) - /unlink 14057 (time 1734260246.43 total 30.00 last 511.75) - /unlink 19090 (time 1734260256.43 total 40.00 last 503.26) - /unlink 24325 (time 1734260266.43 total 50.00 last 523.46) - /unlink 29850 (time 1734260276.43 total 60.01 last 552.44) - /unlink 35313 (time 1734260286.43 total 70.01 last 546.30) - /unlink 40000 (time 1734260294.25 total 77.82 last 599.67) - /unlink 44772 (time 1734260304.25 total 87.82 last 477.15) - /unlink 49142 (time 1734260314.25 total 97.82 last 436.97) - /unlink 53991 (time 1734260324.25 total 107.83 last 484.79) - /unlink 59043 (time 1734260334.25 total 117.83 last 505.17) - /unlink 60000 (time 1734260336.44 total 120.02 last 436.54) - /unlink 64478 (time 1734260346.44 total 130.02 last 447.77) total: 64768 /unlink in 130.54 seconds: 496.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg245-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg245-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg245-server stop mds service on oleg245-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg245-server unloading modules via unload_modules_local on: 'oleg245-server' oleg245-server: oleg245-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-client: ssh exited with exit code 2 pdsh@oleg245-client: oleg245-server: ssh exited with exit code 2