== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 14:27:22 (1773685642) Stopping clients: oleg347-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg347-client.virtnet /mnt/lustre2 (opts:-f) oleg347-server: oleg347-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing load_modules_local oleg347-server: Loading modules from /home/green/git/lustre-release/lustre oleg347-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg347-server: detected 4 online CPUs by sysfs oleg347-server: MODOPTS_LIBCFS= oleg347-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg347-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg347-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg347-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg347-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg347-server: oleg347-server.virtnet: executing set_default_debug -1 all pdsh@oleg347-client: oleg347-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg347-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg347-client.virtnet: -o user_xattr,flock 192.168.203.147@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 32714 458 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32899 643 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 960 (time 1773685745.99 total 10.00 last 95.97) - open/close 1897 (time 1773685756.00 total 20.01 last 93.63) - open/close 2918 (time 1773685766.01 total 30.02 last 101.96) - open/close 3935 (time 1773685776.02 total 40.03 last 101.65) - open/close 4922 (time 1773685786.03 total 50.04 last 98.56) - open/close 5942 (time 1773685796.04 total 60.05 last 101.93) - open/close 7031 (time 1773685806.05 total 70.06 last 108.76) - open/close 8100 (time 1773685816.06 total 80.07 last 106.87) - open/close 9179 (time 1773685826.06 total 90.07 last 107.83) - open/close 10000 (time 1773685832.99 total 97.00 last 118.51) - open/close 11112 (time 1773685843.00 total 107.01 last 111.13) - open/close 12324 (time 1773685853.00 total 117.01 last 121.16) - open/close 13503 (time 1773685863.01 total 127.02 last 117.80) - open/close 14767 (time 1773685873.02 total 137.03 last 126.28) - open/close 16002 (time 1773685883.02 total 147.03 last 123.48) - open/close 17167 (time 1773685893.02 total 157.03 last 116.45) - open/close 18330 (time 1773685903.03 total 167.05 last 116.17) - open/close 19484 (time 1773685913.06 total 177.07 last 115.14) - open/close 20000 (time 1773685917.71 total 181.72 last 110.81) - open/close 21161 (time 1773685927.73 total 191.74 last 115.95) - open/close 22259 (time 1773685937.74 total 201.75 last 109.68) - open/close 23439 (time 1773685947.74 total 211.75 last 117.96) - open/close 24657 (time 1773685957.74 total 221.75 last 121.78) - open/close 25916 (time 1773685967.75 total 231.76 last 125.82) - open/close 27060 (time 1773685977.75 total 241.77 last 114.32) - open/close 28228 (time 1773685987.76 total 251.77 last 116.77) - open/close 29447 (time 1773685997.76 total 261.77 last 121.88) - open/close 30000 (time 1773686002.04 total 266.06 last 129.03) - open/close 31225 (time 1773686012.05 total 276.06 last 122.45) - open/close 32519 (time 1773686022.05 total 286.06 last 129.34) - open/close 33707 (time 1773686032.06 total 296.07 last 118.74) - open/close 34904 (time 1773686042.07 total 306.08 last 119.58) - open/close 36062 (time 1773686052.07 total 316.08 last 115.78) - open/close 37322 (time 1773686062.07 total 326.08 last 125.96) - open/close 38602 (time 1773686072.08 total 336.09 last 127.89) - open/close 39801 (time 1773686082.08 total 346.09 last 119.87) - open/close 40915 (time 1773686092.09 total 356.10 last 111.33) - open/close 42036 (time 1773686102.09 total 366.10 last 112.08) - open/close 43240 (time 1773686112.10 total 376.11 last 120.33) - open/close 44497 (time 1773686122.10 total 386.11 last 125.64) - open/close 45720 (time 1773686132.11 total 396.12 last 122.22) - open/close 47055 (time 1773686142.11 total 406.12 last 133.46) - open/close 48511 (time 1773686152.12 total 416.13 last 145.56) - open/close 49883 (time 1773686162.12 total 426.13 last 137.12) - open/close 51120 (time 1773686172.12 total 436.13 last 123.70) - open/close 52361 (time 1773686182.12 total 446.13 last 124.07) - open/close 53558 (time 1773686192.14 total 456.15 last 119.55) - open/close 54742 (time 1773686202.14 total 466.15 last 118.36) - open/close 56056 (time 1773686212.14 total 476.15 last 131.39) - open/close 57338 (time 1773686222.15 total 486.16 last 128.09) - open/close 58531 (time 1773686232.15 total 496.16 last 119.30) - open/close 59828 (time 1773686242.15 total 506.16 last 129.68) - open/close 61214 (time 1773686252.15 total 516.16 last 138.59) - open/close 62477 (time 1773686262.15 total 526.16 last 126.28) - open/close 63708 (time 1773686272.16 total 536.17 last 122.99) total: 64768 open/close in 544.01 seconds: 119.06 ops/second - /unlink 2295 (time 1773686292.22 total 10.00 last 229.50) - /unlink 4417 (time 1773686302.23 total 20.01 last 212.09) - /unlink 6503 (time 1773686312.24 total 30.01 last 208.42) - /unlink 8778 (time 1773686322.24 total 40.01 last 227.48) - /unlink 10000 (time 1773686327.74 total 45.52 last 221.86) - /unlink 11994 (time 1773686337.75 total 55.52 last 199.38) - /unlink 14219 (time 1773686347.75 total 65.53 last 222.33) - /unlink 16451 (time 1773686357.76 total 75.54 last 223.09) - /unlink 18811 (time 1773686367.76 total 85.54 last 235.98) - /unlink 20000 (time 1773686372.48 total 90.26 last 251.63) - /unlink 22554 (time 1773686382.49 total 100.27 last 255.28) - /unlink 25132 (time 1773686392.49 total 110.27 last 257.78) - /unlink 27168 (time 1773686402.49 total 120.27 last 203.56) - /unlink 29321 (time 1773686412.49 total 130.27 last 215.27) - /unlink 30000 (time 1773686415.53 total 133.30 last 223.92) - /unlink 32084 (time 1773686425.53 total 143.30 last 208.38) - /unlink 34408 (time 1773686435.53 total 153.31 last 232.35) - /unlink 37033 (time 1773686445.53 total 163.31 last 262.41) - /unlink 39218 (time 1773686455.54 total 173.31 last 218.43) - /unlink 40000 (time 1773686459.27 total 177.05 last 209.42) - /unlink 42081 (time 1773686469.27 total 187.05 last 208.09) - /unlink 44292 (time 1773686479.27 total 197.05 last 221.03) - /unlink 46668 (time 1773686489.27 total 207.05 last 237.59) - /unlink 48846 (time 1773686499.27 total 217.05 last 217.77) - /unlink 50000 (time 1773686504.61 total 222.39 last 216.33) - /unlink 52093 (time 1773686514.61 total 232.39 last 209.24) - /unlink 54239 (time 1773686524.61 total 242.39 last 214.57) - /unlink 56966 (time 1773686534.61 total 252.39 last 272.67) - /unlink 59153 (time 1773686544.62 total 262.40 last 218.63) - /unlink 60000 (time 1773686549.20 total 266.98 last 184.82) - /unlink 62266 (time 1773686559.20 total 276.98 last 226.50) - /unlink 64358 (time 1773686569.20 total 286.98 last 209.20) total: 64768 /unlink in 288.87 seconds: 224.21 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1267 (time 1773686604.72 total 10.00 last 126.64) - open/close 2673 (time 1773686614.73 total 20.01 last 140.58) - open/close 4529 (time 1773686624.73 total 30.01 last 185.52) - open/close 6393 (time 1773686634.73 total 40.01 last 186.36) - open/close 8169 (time 1773686644.74 total 50.02 last 177.54) - open/close 10000 (time 1773686653.22 total 58.49 last 215.95) - open/close 11661 (time 1773686663.22 total 68.50 last 166.01) - open/close 13220 (time 1773686673.22 total 78.50 last 155.90) - open/close 14786 (time 1773686683.22 total 88.50 last 156.57) - open/close 16423 (time 1773686693.23 total 98.51 last 163.57) - open/close 18011 (time 1773686703.24 total 108.52 last 158.71) - open/close 19850 (time 1773686713.24 total 118.52 last 183.87) - open/close 21545 (time 1773686723.24 total 128.52 last 169.48) - open/close 23265 (time 1773686733.24 total 138.52 last 171.99) - open/close 25278 (time 1773686743.24 total 148.52 last 201.28) - open/close 27348 (time 1773686753.25 total 158.53 last 206.86) - open/close 29147 (time 1773686763.25 total 168.53 last 179.86) - open/close 30000 (time 1773686767.56 total 172.84 last 197.74) - open/close 32213 (time 1773686777.57 total 182.85 last 221.26) - open/close 34061 (time 1773686787.57 total 192.85 last 184.68) - open/close 36478 (time 1773686797.57 total 202.85 last 241.70) - open/close 38894 (time 1773686807.57 total 212.85 last 241.58) - open/close 40000 (time 1773686813.00 total 218.28 last 203.73) - open/close 42751 (time 1773686823.00 total 228.28 last 275.04) - open/close 45231 (time 1773686833.01 total 238.29 last 247.90) - open/close 47523 (time 1773686843.01 total 248.29 last 229.19) - open/close 49495 (time 1773686853.01 total 258.29 last 197.15) - open/close 50000 (time 1773686855.68 total 260.96 last 188.93) - open/close 51953 (time 1773686865.70 total 270.97 last 195.08) - open/close 53833 (time 1773686875.70 total 280.98 last 187.97) - open/close 55751 (time 1773686885.70 total 290.98 last 191.80) - open/close 58069 (time 1773686895.70 total 300.98 last 231.77) - open/close 60000 (time 1773686904.12 total 309.40 last 229.33) - open/close 62163 (time 1773686914.12 total 319.40 last 216.29) - open/close 64083 (time 1773686924.12 total 329.40 last 191.97) total: 64768 open/close in 332.02 seconds: 195.07 ops/second - /unlink 3712 (time 1773686938.27 total 10.00 last 371.12) - /unlink 6798 (time 1773686948.28 total 20.01 last 308.49) - /unlink 10000 (time 1773686957.60 total 29.33 last 343.58) - /unlink 14057 (time 1773686967.64 total 39.37 last 404.05) - /unlink 18237 (time 1773686977.64 total 49.37 last 417.99) - /unlink 20000 (time 1773686982.11 total 53.83 last 394.59) - /unlink 23791 (time 1773686992.11 total 63.84 last 379.05) - /unlink 27557 (time 1773687002.11 total 73.84 last 376.44) - /unlink 30000 (time 1773687007.99 total 79.72 last 415.36) - /unlink 34239 (time 1773687017.99 total 89.72 last 423.90) - /unlink 37861 (time 1773687028.00 total 99.73 last 362.06) - /unlink 40000 (time 1773687033.79 total 105.52 last 369.35) - /unlink 43919 (time 1773687043.79 total 115.52 last 391.86) - /unlink 47551 (time 1773687053.79 total 125.52 last 363.14) - /unlink 50000 (time 1773687060.28 total 132.00 last 377.62) - /unlink 54441 (time 1773687070.28 total 142.01 last 443.93) - /unlink 59272 (time 1773687080.31 total 152.04 last 481.55) - /unlink 63544 (time 1773687090.31 total 162.04 last 427.18) total: 64768 /unlink in 165.31 seconds: 391.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2551 (time 1773687124.78 total 10.00 last 255.02) - open/close 5835 (time 1773687134.78 total 20.01 last 328.32) - open/close 8873 (time 1773687144.78 total 30.01 last 303.74) - open/close 10000 (time 1773687149.41 total 34.64 last 243.51) - open/close 12667 (time 1773687159.41 total 44.64 last 266.69) - open/close 15173 (time 1773687169.42 total 54.64 last 250.54) - open/close 17597 (time 1773687179.42 total 64.64 last 242.34) - open/close 20000 (time 1773687187.87 total 73.09 last 284.42) - open/close 22852 (time 1773687197.87 total 83.09 last 285.18) - open/close 26205 (time 1773687207.87 total 93.09 last 335.26) - open/close 30000 (time 1773687217.29 total 102.52 last 402.67) - open/close 33704 (time 1773687227.29 total 112.52 last 370.39) - open/close 37095 (time 1773687237.29 total 122.52 last 339.09) - open/close 39930 (time 1773687247.30 total 132.52 last 283.45) - open/close 43118 (time 1773687257.30 total 142.52 last 318.73) - open/close 46099 (time 1773687267.30 total 152.52 last 298.04) - open/close 48693 (time 1773687277.31 total 162.53 last 259.26) - open/close 50000 (time 1773687282.40 total 167.63 last 256.30) - open/close 53527 (time 1773687292.41 total 177.63 last 352.59) - open/close 57207 (time 1773687302.41 total 187.63 last 367.92) - open/close 60000 (time 1773687310.70 total 195.92 last 336.85) - open/close 63541 (time 1773687320.70 total 205.93 last 354.07) total: 64768 open/close in 209.75 seconds: 308.78 ops/second - /unlink 5489 (time 1773687336.18 total 10.00 last 548.87) - /unlink 10000 (time 1773687343.13 total 16.95 last 649.01) - /unlink 16121 (time 1773687353.13 total 26.95 last 612.04) - /unlink 20000 (time 1773687359.99 total 33.80 last 566.16) - /unlink 26505 (time 1773687369.99 total 43.80 last 650.49) - /unlink 30000 (time 1773687375.84 total 49.66 last 597.12) - /unlink 36596 (time 1773687385.84 total 59.66 last 659.30) - /unlink 40000 (time 1773687391.68 total 65.49 last 583.60) - /unlink 46358 (time 1773687401.68 total 75.49 last 635.75) - /unlink 50000 (time 1773687406.70 total 80.52 last 725.02) - /unlink 57182 (time 1773687416.70 total 90.52 last 718.11) - /unlink 60000 (time 1773687420.22 total 94.04 last 801.46) total: 64768 /unlink in 101.61 seconds: 637.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4330 (time 1773687459.56 total 10.00 last 432.98) - open/close 9068 (time 1773687469.56 total 20.00 last 473.73) - open/close 13972 (time 1773687479.56 total 30.00 last 490.38) - open/close 18804 (time 1773687489.56 total 40.00 last 483.17) - open/close 20000 (time 1773687492.11 total 42.55 last 469.59) - open/close 24737 (time 1773687502.11 total 52.55 last 473.70) - open/close 29787 (time 1773687512.11 total 62.55 last 504.94) - open/close 34687 (time 1773687522.11 total 72.55 last 489.97) - open/close 39971 (time 1773687532.11 total 82.55 last 528.37) - open/close 45000 (time 1773687542.11 total 92.55 last 502.82) - open/close 49731 (time 1773687552.11 total 102.55 last 473.10) - open/close 54495 (time 1773687562.19 total 112.63 last 472.63) - open/close 59169 (time 1773687572.19 total 122.63 last 467.38) - open/close 64082 (time 1773687582.19 total 132.64 last 491.26) total: 64768 open/close in 134.06 seconds: 483.12 ops/second - /unlink 7621 (time 1773687594.87 total 10.00 last 762.08) - /unlink 10000 (time 1773687597.74 total 12.86 last 831.28) - /unlink 17574 (time 1773687607.74 total 22.86 last 757.31) - /unlink 20000 (time 1773687611.21 total 26.34 last 697.80) - /unlink 26162 (time 1773687621.21 total 36.34 last 616.18) - /unlink 30000 (time 1773687626.27 total 41.40 last 758.58) - /unlink 37561 (time 1773687636.27 total 51.40 last 756.06) - /unlink 40000 (time 1773687639.25 total 54.38 last 819.72) - /unlink 47435 (time 1773687649.25 total 64.38 last 743.46) - /unlink 50000 (time 1773687652.80 total 67.93 last 722.71) - /unlink 57578 (time 1773687662.80 total 77.93 last 757.77) - /unlink 60000 (time 1773687666.02 total 81.15 last 750.79) total: 64768 /unlink in 87.63 seconds: 739.10 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4449 (time 1773687703.50 total 10.00 last 444.85) - open/close 8670 (time 1773687713.50 total 20.00 last 422.05) - open/close 10000 (time 1773687716.72 total 23.22 last 413.67) - open/close 14391 (time 1773687726.72 total 33.22 last 439.06) - open/close 18402 (time 1773687736.72 total 43.22 last 401.05) - open/close 20000 (time 1773687740.88 total 47.39 last 383.64) - open/close 24371 (time 1773687750.89 total 57.39 last 437.04) - open/close 28677 (time 1773687760.89 total 67.39 last 430.60) - open/close 30000 (time 1773687763.96 total 70.46 last 430.81) - open/close 34449 (time 1773687773.96 total 80.46 last 444.84) - open/close 38858 (time 1773687783.96 total 90.46 last 440.89) - open/close 40000 (time 1773687786.21 total 92.71 last 507.29) - open/close 44649 (time 1773687796.21 total 102.71 last 464.85) - open/close 49265 (time 1773687806.21 total 112.71 last 461.54) - open/close 53183 (time 1773687816.21 total 122.71 last 391.78) - open/close 56926 (time 1773687826.21 total 132.71 last 374.25) - open/close 60000 (time 1773687833.49 total 139.99 last 422.59) - open/close 64379 (time 1773687843.49 total 149.99 last 437.88) total: 64768 open/close in 150.87 seconds: 429.30 ops/second - /unlink 6641 (time 1773687855.73 total 10.00 last 664.08) - /unlink 10000 (time 1773687860.87 total 15.14 last 653.58) - /unlink 17419 (time 1773687870.87 total 25.14 last 741.86) - /unlink 20000 (time 1773687873.95 total 28.22 last 837.81) - /unlink 27679 (time 1773687883.95 total 38.22 last 767.81) - /unlink 30000 (time 1773687887.57 total 41.84 last 641.38) - /unlink 36738 (time 1773687897.57 total 51.84 last 673.70) - /unlink 40000 (time 1773687902.27 total 56.54 last 693.75) - /unlink 47581 (time 1773687912.27 total 66.54 last 758.09) - /unlink 50000 (time 1773687915.34 total 69.61 last 788.94) - /unlink 57209 (time 1773687925.34 total 79.61 last 720.83) - /unlink 60000 (time 1773687929.01 total 83.28 last 759.97) total: 64768 /unlink in 89.55 seconds: 723.26 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg347-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg347-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg347-server stop mds service on oleg347-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg347-server unloading modules via unload_modules_local on: 'oleg347-server' oleg347-server: oleg347-server.virtnet: executing unload_modules_local oleg347-server: modules unloaded. pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-client: ssh exited with exit code 2 pdsh@oleg347-client: oleg347-server: ssh exited with exit code 2