== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 01:49:30 (1770446970) Stopping clients: oleg411-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg411-client.virtnet /mnt/lustre2 (opts:-f) oleg411-server: oleg411-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg411-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1324 (time 1770447037.14 total 10.00 last 132.39) - open/close 2769 (time 1770447047.14 total 20.00 last 144.48) - open/close 4345 (time 1770447057.15 total 30.01 last 157.49) - open/close 5839 (time 1770447067.15 total 40.01 last 149.38) - open/close 7525 (time 1770447077.15 total 50.01 last 168.53) - open/close 9026 (time 1770447087.15 total 60.02 last 150.09) - open/close 10000 (time 1770447095.07 total 67.94 last 122.98) - open/close 11221 (time 1770447105.07 total 77.94 last 122.09) - open/close 12513 (time 1770447115.08 total 87.94 last 129.18) - open/close 13813 (time 1770447125.08 total 97.94 last 129.97) - open/close 15064 (time 1770447135.08 total 107.94 last 125.07) - open/close 16274 (time 1770447145.09 total 117.95 last 120.91) - open/close 17542 (time 1770447155.09 total 127.96 last 126.72) - open/close 18879 (time 1770447165.11 total 137.97 last 133.54) - open/close 20000 (time 1770447173.78 total 146.64 last 129.22) - open/close 21223 (time 1770447183.79 total 156.66 last 122.16) - open/close 22328 (time 1770447193.80 total 166.67 last 110.40) - open/close 23272 (time 1770447203.81 total 176.67 last 94.35) - open/close 24168 (time 1770447214.00 total 186.86 last 87.94) - open/close 24819 (time 1770447224.00 total 196.86 last 65.10) - open/close 25470 (time 1770447234.00 total 206.87 last 65.06) - open/close 26223 (time 1770447244.01 total 216.88 last 75.22) - open/close 26991 (time 1770447254.02 total 226.89 last 76.71) - open/close 27966 (time 1770447264.03 total 236.89 last 97.49) - open/close 29101 (time 1770447274.03 total 246.89 last 113.42) - open/close 30000 (time 1770447283.68 total 256.54 last 93.21) - open/close 30925 (time 1770447293.69 total 266.55 last 92.40) - open/close 31972 (time 1770447303.69 total 276.55 last 104.67) - open/close 32919 (time 1770447313.70 total 286.56 last 94.61) - open/close 33969 (time 1770447323.70 total 296.56 last 104.99) - open/close 34978 (time 1770447333.72 total 306.58 last 100.73) - open/close 35999 (time 1770447343.72 total 316.59 last 102.05) - open/close 37224 (time 1770447353.73 total 326.59 last 122.45) - open/close 38459 (time 1770447363.73 total 336.59 last 123.46) - open/close 39636 (time 1770447373.73 total 346.59 last 117.69) - open/close 40000 (time 1770447376.58 total 349.45 last 127.56) - open/close 41216 (time 1770447386.59 total 359.46 last 121.48) - open/close 42202 (time 1770447396.60 total 369.46 last 98.56) - open/close 43292 (time 1770447406.60 total 379.46 last 108.98) - open/close 44274 (time 1770447416.61 total 389.47 last 98.08) - open/close 45043 (time 1770447426.61 total 399.48 last 76.88) - open/close 46109 (time 1770447436.62 total 409.48 last 106.59) - open/close 47140 (time 1770447446.62 total 419.48 last 103.06) - open/close 48185 (time 1770447456.63 total 429.49 last 104.39) - open/close 49236 (time 1770447466.63 total 439.49 last 105.07) - open/close 50000 (time 1770447474.39 total 447.25 last 98.51) - open/close 50905 (time 1770447484.39 total 457.25 last 90.48) - open/close 51815 (time 1770447494.40 total 467.26 last 90.94) - open/close 52902 (time 1770447504.40 total 477.26 last 108.68) - open/close 53905 (time 1770447514.40 total 487.27 last 100.25) - open/close 54822 (time 1770447524.41 total 497.27 last 91.65) - open/close 55797 (time 1770447534.42 total 507.28 last 97.44) - open/close 56797 (time 1770447544.42 total 517.28 last 99.99) - open/close 57809 (time 1770447554.42 total 527.28 last 101.19) - open/close 58919 (time 1770447564.42 total 537.28 last 110.99) - open/close 59942 (time 1770447574.43 total 547.30 last 102.17) - open/close 61170 (time 1770447584.44 total 557.30 last 122.71) - open/close 62200 (time 1770447594.44 total 567.31 last 102.97) - open/close 63514 (time 1770447604.45 total 577.31 last 131.35) total: 64768 open/close in 586.64 seconds: 110.41 ops/second - /unlink 2147 (time 1770447626.22 total 10.00 last 214.67) - /unlink 3895 (time 1770447636.23 total 20.01 last 174.68) - /unlink 5602 (time 1770447646.23 total 30.01 last 170.70) - /unlink 7602 (time 1770447656.23 total 40.01 last 199.96) - /unlink 9449 (time 1770447666.23 total 50.02 last 184.60) - /unlink 10000 (time 1770447669.16 total 52.94 last 188.50) - /unlink 12166 (time 1770447679.16 total 62.94 last 216.53) - /unlink 14015 (time 1770447689.16 total 72.94 last 184.85) - /unlink 16036 (time 1770447699.17 total 82.95 last 202.05) - /unlink 18262 (time 1770447709.17 total 92.95 last 222.51) - /unlink 20000 (time 1770447717.64 total 101.43 last 205.07) - /unlink 22338 (time 1770447727.65 total 111.43 last 233.73) - /unlink 24530 (time 1770447737.65 total 121.43 last 219.17) - /unlink 26600 (time 1770447747.65 total 131.43 last 206.97) - /unlink 29085 (time 1770447757.65 total 141.43 last 248.48) - /unlink 30000 (time 1770447761.24 total 145.02 last 255.30) - /unlink 32781 (time 1770447771.24 total 155.02 last 278.09) - /unlink 36108 (time 1770447781.24 total 165.02 last 332.67) - /unlink 39183 (time 1770447791.24 total 175.02 last 307.45) - /unlink 40000 (time 1770447794.78 total 178.56 last 230.70) - /unlink 42555 (time 1770447804.78 total 188.56 last 255.50) - /unlink 45544 (time 1770447814.78 total 198.56 last 298.87) - /unlink 48039 (time 1770447824.78 total 208.56 last 249.44) - /unlink 50000 (time 1770447830.46 total 214.25 last 345.19) - /unlink 53254 (time 1770447840.46 total 224.25 last 325.38) - /unlink 56054 (time 1770447850.47 total 234.25 last 279.92) - /unlink 58566 (time 1770447860.47 total 244.25 last 251.07) - /unlink 60000 (time 1770447866.43 total 250.21 last 240.70) - /unlink 62891 (time 1770447876.43 total 260.21 last 289.09) total: 64768 /unlink in 266.28 seconds: 243.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1650 (time 1770447917.40 total 10.00 last 164.92) - open/close 3440 (time 1770447927.40 total 20.01 last 178.99) - open/close 4997 (time 1770447937.40 total 30.01 last 155.69) - open/close 6477 (time 1770447947.41 total 40.01 last 147.94) - open/close 7878 (time 1770447957.41 total 50.01 last 140.08) - open/close 9411 (time 1770447967.41 total 60.02 last 153.21) - open/close 10000 (time 1770447972.33 total 64.93 last 119.84) - open/close 11440 (time 1770447982.34 total 74.94 last 143.90) - open/close 12882 (time 1770447992.34 total 84.94 last 144.15) - open/close 14264 (time 1770448002.34 total 94.95 last 138.16) - open/close 15606 (time 1770448012.35 total 104.95 last 134.16) - open/close 17059 (time 1770448022.35 total 114.95 last 145.29) - open/close 19081 (time 1770448032.35 total 124.95 last 202.18) - open/close 20000 (time 1770448036.88 total 129.48 last 202.93) - open/close 22046 (time 1770448046.88 total 139.48 last 204.58) - open/close 24117 (time 1770448056.88 total 149.48 last 207.05) - open/close 26089 (time 1770448066.89 total 159.49 last 197.10) - open/close 28172 (time 1770448076.89 total 169.49 last 208.24) - open/close 30000 (time 1770448086.22 total 178.82 last 195.95) - open/close 31619 (time 1770448096.22 total 188.83 last 161.81) - open/close 33467 (time 1770448106.23 total 198.83 last 184.75) - open/close 36315 (time 1770448116.23 total 208.83 last 284.78) - open/close 38483 (time 1770448126.24 total 218.84 last 216.46) - open/close 40000 (time 1770448135.04 total 227.64 last 172.40) - open/close 42182 (time 1770448145.04 total 237.65 last 218.14) - open/close 44551 (time 1770448155.05 total 247.65 last 236.88) - open/close 47685 (time 1770448165.05 total 257.65 last 313.35) - open/close 49960 (time 1770448175.05 total 267.66 last 227.37) - open/close 52690 (time 1770448185.05 total 277.66 last 272.97) - open/close 55826 (time 1770448195.05 total 287.66 last 313.57) - open/close 58993 (time 1770448205.06 total 297.66 last 316.66) - open/close 60000 (time 1770448208.38 total 300.99 last 302.52) - open/close 63094 (time 1770448218.39 total 310.99 last 309.37) total: 64768 open/close in 315.40 seconds: 205.35 ops/second - /unlink 5971 (time 1770448234.21 total 10.00 last 597.04) - /unlink 10000 (time 1770448239.45 total 15.24 last 768.56) - /unlink 17897 (time 1770448249.45 total 25.24 last 789.64) - /unlink 20000 (time 1770448252.10 total 27.90 last 793.23) - /unlink 26706 (time 1770448262.10 total 37.90 last 670.59) - /unlink 30000 (time 1770448267.12 total 42.91 last 656.67) - /unlink 35411 (time 1770448277.12 total 52.91 last 540.93) - /unlink 40000 (time 1770448286.01 total 61.80 last 516.44) - /unlink 46024 (time 1770448296.01 total 71.80 last 602.39) - /unlink 50000 (time 1770448301.15 total 76.94 last 773.52) - /unlink 56803 (time 1770448311.15 total 86.94 last 680.26) - /unlink 60000 (time 1770448317.03 total 92.82 last 543.77) total: 64768 /unlink in 102.74 seconds: 630.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4014 (time 1770448354.39 total 10.00 last 401.31) - open/close 7798 (time 1770448364.39 total 20.00 last 378.32) - open/close 10000 (time 1770448371.68 total 27.30 last 301.93) - open/close 13019 (time 1770448381.69 total 37.30 last 301.88) - open/close 15984 (time 1770448391.69 total 47.30 last 296.50) - open/close 18867 (time 1770448401.69 total 57.30 last 288.22) - open/close 20000 (time 1770448406.38 total 62.00 last 241.33) - open/close 22919 (time 1770448416.38 total 72.00 last 291.85) - open/close 25789 (time 1770448426.39 total 82.00 last 286.97) - open/close 28968 (time 1770448436.39 total 92.00 last 317.87) - open/close 30000 (time 1770448439.55 total 95.16 last 326.03) - open/close 33443 (time 1770448449.55 total 105.16 last 344.29) - open/close 36710 (time 1770448459.55 total 115.17 last 326.69) - open/close 40000 (time 1770448468.68 total 124.29 last 360.37) - open/close 43253 (time 1770448478.68 total 134.30 last 325.27) - open/close 46914 (time 1770448488.69 total 144.30 last 366.02) - open/close 49992 (time 1770448498.69 total 154.30 last 307.71) - open/close 52905 (time 1770448508.69 total 164.30 last 291.26) - open/close 55868 (time 1770448518.69 total 174.30 last 296.25) - open/close 58967 (time 1770448528.69 total 184.31 last 309.84) - open/close 60000 (time 1770448531.21 total 186.82 last 410.83) - open/close 64129 (time 1770448541.21 total 196.82 last 412.87) total: 64768 open/close in 198.33 seconds: 326.56 ops/second - /unlink 7607 (time 1770448554.09 total 10.00 last 760.63) - /unlink 10000 (time 1770448557.67 total 13.59 last 667.42) - /unlink 16171 (time 1770448567.68 total 23.59 last 616.71) - /unlink 17919 (time 1770448577.70 total 33.61 last 174.47) - /unlink 19363 (time 1770448587.71 total 43.62 last 144.23) - /unlink 20000 (time 1770448592.19 total 48.11 last 142.03) - /unlink 21732 (time 1770448602.20 total 58.11 last 173.11) - /unlink 23783 (time 1770448612.20 total 68.11 last 205.08) - /unlink 27382 (time 1770448622.20 total 78.11 last 359.90) - /unlink 30000 (time 1770448628.97 total 84.89 last 386.51) - /unlink 34581 (time 1770448638.98 total 94.89 last 458.08) - /unlink 37957 (time 1770448648.98 total 104.89 last 337.44) - /unlink 40000 (time 1770448656.48 total 112.40 last 272.34) - /unlink 42643 (time 1770448666.49 total 122.40 last 264.20) - /unlink 45454 (time 1770448676.49 total 132.40 last 281.08) - /unlink 49087 (time 1770448686.49 total 142.40 last 363.29) - /unlink 53311 (time 1770448696.49 total 152.40 last 422.34) - /unlink 56528 (time 1770448706.49 total 162.40 last 321.61) - /unlink 59617 (time 1770448716.49 total 172.40 last 308.88) - /unlink 62690 (time 1770448726.49 total 182.40 last 307.29) total: 64768 /unlink in 187.58 seconds: 345.28 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1696 (time 1770448762.67 total 10.01 last 169.42) - open/close 3189 (time 1770448772.67 total 20.01 last 149.25) - open/close 4707 (time 1770448782.67 total 30.01 last 151.79) - open/close 6178 (time 1770448792.68 total 40.02 last 147.04) - open/close 7526 (time 1770448802.68 total 50.02 last 134.74) - open/close 8824 (time 1770448812.69 total 60.03 last 129.73) - open/close 10000 (time 1770448820.74 total 68.08 last 146.03) - open/close 12528 (time 1770448830.74 total 78.08 last 252.77) - open/close 15584 (time 1770448840.74 total 88.08 last 305.54) - open/close 18261 (time 1770448850.75 total 98.09 last 267.64) - open/close 20000 (time 1770448858.06 total 105.40 last 237.80) - open/close 22460 (time 1770448868.06 total 115.40 last 245.95) - open/close 24589 (time 1770448878.07 total 125.41 last 212.74) - open/close 27055 (time 1770448888.07 total 135.41 last 246.58) - open/close 28673 (time 1770448898.08 total 145.42 last 161.66) - open/close 30000 (time 1770448907.11 total 154.45 last 147.00) - open/close 31819 (time 1770448917.11 total 164.45 last 181.82) - open/close 33892 (time 1770448927.11 total 174.45 last 207.30) - open/close 35723 (time 1770448937.11 total 184.45 last 183.05) - open/close 37650 (time 1770448947.11 total 194.45 last 192.69) - open/close 39318 (time 1770448957.11 total 204.45 last 166.79) - open/close 40000 (time 1770448961.69 total 209.03 last 148.99) - open/close 42015 (time 1770448971.70 total 219.04 last 201.40) - open/close 43827 (time 1770448981.70 total 229.04 last 181.08) - open/close 45656 (time 1770448991.71 total 239.05 last 182.75) - open/close 47753 (time 1770449001.71 total 249.05 last 209.65) - open/close 50000 (time 1770449011.37 total 258.71 last 232.68) - open/close 52263 (time 1770449021.37 total 268.71 last 226.26) - open/close 54583 (time 1770449031.39 total 278.73 last 231.64) - open/close 56565 (time 1770449041.39 total 288.73 last 198.17) - open/close 58740 (time 1770449051.39 total 298.73 last 217.39) - open/close 60000 (time 1770449056.82 total 304.16 last 232.24) - open/close 62078 (time 1770449066.82 total 314.16 last 207.74) - open/close 63993 (time 1770449076.82 total 324.16 last 191.47) total: 64768 open/close in 327.84 seconds: 197.56 ops/second - /unlink 4864 (time 1770449092.26 total 10.00 last 486.23) - /unlink 9430 (time 1770449102.26 total 20.00 last 456.55) - /unlink 13939 (time 1770449112.27 total 30.01 last 450.87) - /unlink 19188 (time 1770449122.27 total 40.01 last 524.81) - /unlink 20000 (time 1770449124.42 total 42.16 last 376.56) - /unlink 23868 (time 1770449134.42 total 52.16 last 386.79) - /unlink 28057 (time 1770449144.42 total 62.16 last 418.90) - /unlink 30000 (time 1770449149.11 total 66.85 last 414.54) - /unlink 34167 (time 1770449159.11 total 76.85 last 416.66) - /unlink 38321 (time 1770449169.11 total 86.85 last 415.34) - /unlink 40000 (time 1770449173.04 total 90.78 last 427.17) - /unlink 43818 (time 1770449183.04 total 100.78 last 381.77) - /unlink 47727 (time 1770449193.05 total 110.79 last 390.72) - /unlink 50000 (time 1770449198.87 total 116.61 last 390.65) - /unlink 53902 (time 1770449208.87 total 126.61 last 390.14) - /unlink 57399 (time 1770449218.87 total 136.61 last 349.61) - /unlink 60000 (time 1770449224.89 total 142.63 last 432.49) - /unlink 64636 (time 1770449234.89 total 152.63 last 463.56) total: 64768 /unlink in 152.99 seconds: 423.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2402 (time 1770449266.63 total 10.01 last 240.04) - open/close 5430 (time 1770449276.63 total 20.01 last 302.74) - open/close 8276 (time 1770449286.63 total 30.01 last 284.59) - open/close 10000 (time 1770449292.78 total 36.16 last 280.50) - open/close 13663 (time 1770449302.78 total 46.16 last 366.21) - open/close 17497 (time 1770449312.78 total 56.16 last 383.38) - open/close 20000 (time 1770449321.67 total 65.05 last 281.50) - open/close 23148 (time 1770449331.67 total 75.05 last 314.74) - open/close 26413 (time 1770449341.67 total 85.05 last 326.47) - open/close 29124 (time 1770449351.68 total 95.06 last 270.92) - open/close 30000 (time 1770449355.92 total 99.29 last 206.84) - open/close 32279 (time 1770449365.92 total 109.30 last 227.86) - open/close 35043 (time 1770449375.92 total 119.30 last 276.29) - open/close 38498 (time 1770449385.92 total 129.30 last 345.44) - open/close 40000 (time 1770449390.05 total 133.43 last 364.01) - open/close 43842 (time 1770449400.14 total 143.52 last 380.58) - open/close 47459 (time 1770449410.15 total 153.52 last 361.66) - open/close 50000 (time 1770449416.35 total 159.73 last 409.37) - open/close 53284 (time 1770449426.36 total 169.74 last 328.27) - open/close 56548 (time 1770449436.36 total 179.74 last 326.26) - open/close 59899 (time 1770449446.36 total 189.74 last 335.08) - open/close 63943 (time 1770449456.38 total 199.76 last 403.48) total: 64768 open/close in 201.96 seconds: 320.69 ops/second - /unlink 5796 (time 1770449469.99 total 10.00 last 579.59) - /unlink 10000 (time 1770449477.04 total 17.04 last 596.90) - /unlink 16009 (time 1770449487.04 total 27.04 last 600.87) - /unlink 20000 (time 1770449493.67 total 33.68 last 601.29) - /unlink 26244 (time 1770449503.67 total 43.68 last 624.40) - /unlink 30000 (time 1770449509.32 total 49.33 last 665.24) - /unlink 36140 (time 1770449519.32 total 59.33 last 613.96) - /unlink 40000 (time 1770449525.40 total 65.41 last 634.64) - /unlink 46116 (time 1770449535.40 total 75.41 last 611.55) - /unlink 50000 (time 1770449540.44 total 80.45 last 770.49) - /unlink 56725 (time 1770449550.45 total 90.45 last 672.43) - /unlink 60000 (time 1770449555.32 total 95.32 last 672.40) total: 64768 /unlink in 101.50 seconds: 638.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local oleg411-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg411-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2