== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:44:53 (1752687893) Stopping clients: oleg323-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg323-client.virtnet /mnt/lustre2 (opts:-f) oleg323-server: oleg323-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing load_modules_local oleg323-server: Loading modules from /home/green/git/lustre-release/lustre oleg323-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg323-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg323-server: detected 4 online CPUs by sysfs oleg323-server: MODOPTS_LIBCFS= oleg323-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg323-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg323-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg323-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg323-server: oleg323-server.virtnet: executing set_default_debug -1 all pdsh@oleg323-client: oleg323-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg323-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg323-client.virtnet: -o user_xattr,flock 192.168.203.123@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3608 (time 1752687939.22 total 10.00 last 360.68) - open/close 7268 (time 1752687949.22 total 20.00 last 365.96) - open/close 10000 (time 1752687956.90 total 27.68 last 355.75) - open/close 13424 (time 1752687966.90 total 37.69 last 342.24) - open/close 16229 (time 1752687976.90 total 47.69 last 280.48) - open/close 19232 (time 1752687986.91 total 57.69 last 300.17) - open/close 20000 (time 1752687989.25 total 60.04 last 326.96) - open/close 23600 (time 1752687999.26 total 70.04 last 359.92) - open/close 27334 (time 1752688009.26 total 80.05 last 373.34) - open/close 30000 (time 1752688016.95 total 87.73 last 346.74) - open/close 33753 (time 1752688026.95 total 97.74 last 375.24) - open/close 37044 (time 1752688036.95 total 107.74 last 329.02) - open/close 39772 (time 1752688046.95 total 117.74 last 272.76) - open/close 42471 (time 1752688056.95 total 127.74 last 269.89) - open/close 46348 (time 1752688066.96 total 137.74 last 387.57) - open/close 48042 (time 1752688076.96 total 147.75 last 169.38) - open/close 49102 (time 1752688086.96 total 157.75 last 105.94) - open/close 50000 (time 1752688096.85 total 167.64 last 90.80) - open/close 50855 (time 1752688106.86 total 177.65 last 85.45) - open/close 51683 (time 1752688116.87 total 187.66 last 82.70) - open/close 52479 (time 1752688126.88 total 197.67 last 79.53) - open/close 53521 (time 1752688136.89 total 207.68 last 104.12) - open/close 54758 (time 1752688146.89 total 217.68 last 123.63) - open/close 56235 (time 1752688156.89 total 227.68 last 147.69) - open/close 57442 (time 1752688166.90 total 237.68 last 120.66) - open/close 58610 (time 1752688176.91 total 247.69 last 116.68) - open/close 59903 (time 1752688186.91 total 257.70 last 129.26) - open/close 61136 (time 1752688196.92 total 267.71 last 123.20) - open/close 62647 (time 1752688206.92 total 277.71 last 151.04) - open/close 64645 (time 1752688216.92 total 287.71 last 199.80) total: 64768 open/close in 288.16 seconds: 224.76 ops/second - /unlink 3272 (time 1752688228.92 total 10.00 last 327.15) - /unlink 6169 (time 1752688238.92 total 20.00 last 289.66) - /unlink 9347 (time 1752688248.92 total 30.00 last 317.78) - /unlink 10000 (time 1752688251.23 total 32.31 last 282.48) - /unlink 12545 (time 1752688261.23 total 42.32 last 254.45) - /unlink 14423 (time 1752688271.23 total 52.32 last 187.79) - /unlink 16431 (time 1752688281.24 total 62.32 last 200.78) - /unlink 18297 (time 1752688291.24 total 72.32 last 186.59) - /unlink 20000 (time 1752688298.18 total 79.26 last 245.39) - /unlink 22608 (time 1752688308.18 total 89.26 last 260.72) - /unlink 25135 (time 1752688318.18 total 99.26 last 252.66) - /unlink 27627 (time 1752688328.18 total 109.27 last 249.17) - /unlink 30000 (time 1752688336.13 total 117.22 last 298.43) - /unlink 32686 (time 1752688346.14 total 127.22 last 268.56) - /unlink 35204 (time 1752688356.14 total 137.22 last 251.74) - /unlink 37776 (time 1752688366.14 total 147.22 last 257.16) - /unlink 40000 (time 1752688373.20 total 154.28 last 314.93) - /unlink 42641 (time 1752688383.20 total 164.29 last 264.03) - /unlink 45384 (time 1752688393.21 total 174.29 last 274.25) - /unlink 47272 (time 1752688403.21 total 184.29 last 188.71) - /unlink 49328 (time 1752688413.21 total 194.30 last 205.56) - /unlink 50000 (time 1752688416.60 total 197.69 last 198.29) - /unlink 52519 (time 1752688426.60 total 207.69 last 251.88) - /unlink 55024 (time 1752688436.60 total 217.69 last 250.47) - /unlink 57626 (time 1752688446.61 total 227.69 last 260.11) - /unlink 60000 (time 1752688455.99 total 237.07 last 253.01) - /unlink 62487 (time 1752688465.99 total 247.08 last 248.66) total: 64768 /unlink in 256.65 seconds: 252.36 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1465 (time 1752688506.96 total 10.00 last 146.45) - open/close 2731 (time 1752688516.96 total 20.01 last 126.55) - open/close 4210 (time 1752688526.97 total 30.01 last 147.80) - open/close 5946 (time 1752688536.97 total 40.02 last 173.52) - open/close 7634 (time 1752688546.97 total 50.02 last 168.74) - open/close 9473 (time 1752688556.97 total 60.02 last 183.87) - open/close 10000 (time 1752688559.35 total 62.40 last 221.49) - open/close 12346 (time 1752688569.36 total 72.40 last 234.56) - open/close 14827 (time 1752688579.36 total 82.41 last 248.01) - open/close 17247 (time 1752688589.36 total 92.41 last 241.92) - open/close 19208 (time 1752688599.37 total 102.41 last 196.05) - open/close 20000 (time 1752688602.61 total 105.66 last 244.02) - open/close 22608 (time 1752688612.61 total 115.66 last 260.73) - open/close 24706 (time 1752688622.62 total 125.67 last 209.70) - open/close 26857 (time 1752688632.62 total 135.67 last 215.09) - open/close 29267 (time 1752688642.62 total 145.67 last 240.94) - open/close 30000 (time 1752688645.41 total 148.46 last 262.60) - open/close 32387 (time 1752688655.41 total 158.46 last 238.66) - open/close 34711 (time 1752688665.42 total 168.47 last 232.32) - open/close 37189 (time 1752688675.42 total 178.47 last 247.80) - open/close 39384 (time 1752688685.42 total 188.47 last 219.44) - open/close 40000 (time 1752688688.08 total 191.13 last 231.68) - open/close 42133 (time 1752688698.08 total 201.13 last 213.22) - open/close 44298 (time 1752688708.09 total 211.14 last 216.41) - open/close 46647 (time 1752688718.09 total 221.14 last 234.83) - open/close 49058 (time 1752688728.09 total 231.14 last 241.10) - open/close 50000 (time 1752688732.39 total 235.44 last 218.85) - open/close 51816 (time 1752688742.40 total 245.45 last 181.53) - open/close 54140 (time 1752688752.40 total 255.45 last 232.38) - open/close 56616 (time 1752688762.40 total 265.45 last 247.56) - open/close 58960 (time 1752688772.40 total 275.45 last 234.37) - open/close 60000 (time 1752688777.07 total 280.11 last 222.95) - open/close 62154 (time 1752688787.07 total 290.12 last 215.38) - open/close 64281 (time 1752688797.07 total 300.12 last 212.63) total: 64768 open/close in 302.18 seconds: 214.34 ops/second - /unlink 3645 (time 1752688810.98 total 10.00 last 364.48) - /unlink 8189 (time 1752688820.98 total 20.00 last 454.38) - /unlink 10000 (time 1752688825.64 total 24.66 last 388.89) - /unlink 14306 (time 1752688835.64 total 34.66 last 430.58) - /unlink 18799 (time 1752688845.64 total 44.66 last 449.26) - /unlink 20000 (time 1752688848.75 total 47.77 last 386.31) - /unlink 24323 (time 1752688858.75 total 57.77 last 432.27) - /unlink 27841 (time 1752688868.75 total 67.77 last 351.72) - /unlink 30000 (time 1752688874.05 total 73.07 last 407.54) - /unlink 35007 (time 1752688884.05 total 83.07 last 500.65) - /unlink 39678 (time 1752688894.05 total 93.07 last 467.05) - /unlink 44626 (time 1752688904.05 total 103.07 last 494.71) - /unlink 50000 (time 1752688913.96 total 112.98 last 542.58) - /unlink 55101 (time 1752688923.96 total 122.98 last 510.02) - /unlink 59989 (time 1752688933.97 total 132.99 last 488.47) total: 64768 /unlink in 142.36 seconds: 454.97 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3149 (time 1752688974.26 total 10.00 last 314.87) - open/close 6373 (time 1752688984.26 total 20.00 last 322.33) - open/close 9482 (time 1752688994.26 total 30.00 last 310.87) - open/close 12784 (time 1752689004.26 total 40.01 last 330.15) - open/close 15805 (time 1752689014.26 total 50.01 last 302.09) - open/close 18883 (time 1752689024.26 total 60.01 last 307.74) - open/close 20000 (time 1752689027.92 total 63.66 last 305.70) - open/close 23036 (time 1752689037.92 total 73.66 last 303.55) - open/close 26456 (time 1752689047.93 total 83.67 last 341.76) - open/close 29205 (time 1752689057.93 total 93.67 last 274.90) - open/close 30000 (time 1752689060.48 total 96.22 last 311.58) - open/close 33701 (time 1752689070.48 total 106.22 last 370.09) - open/close 36525 (time 1752689080.48 total 116.22 last 282.36) - open/close 39181 (time 1752689090.48 total 126.23 last 265.53) - open/close 40000 (time 1752689093.72 total 129.46 last 252.89) - open/close 42574 (time 1752689103.72 total 139.47 last 257.34) - open/close 45085 (time 1752689113.73 total 149.47 last 251.04) - open/close 48121 (time 1752689123.73 total 159.47 last 303.59) - open/close 50000 (time 1752689129.87 total 165.61 last 305.87) - open/close 52596 (time 1752689139.87 total 175.61 last 259.55) - open/close 55956 (time 1752689149.87 total 185.62 last 335.98) - open/close 59357 (time 1752689159.87 total 195.62 last 340.07) - open/close 62712 (time 1752689169.88 total 205.62 last 335.40) total: 64768 open/close in 211.02 seconds: 306.93 ops/second - /unlink 6585 (time 1752689186.61 total 10.00 last 658.43) - /unlink 10000 (time 1752689191.81 total 15.20 last 656.63) - /unlink 15867 (time 1752689201.81 total 25.20 last 586.63) - /unlink 20000 (time 1752689207.80 total 31.19 last 689.80) - /unlink 26492 (time 1752689217.81 total 41.20 last 649.12) - /unlink 30000 (time 1752689222.98 total 46.38 last 677.31) - /unlink 37534 (time 1752689232.99 total 56.38 last 753.36) - /unlink 40000 (time 1752689236.46 total 59.85 last 709.50) - /unlink 47177 (time 1752689246.46 total 69.85 last 717.65) - /unlink 50000 (time 1752689250.01 total 73.40 last 795.92) - /unlink 57395 (time 1752689260.01 total 83.40 last 739.50) - /unlink 60000 (time 1752689263.92 total 87.31 last 666.30) total: 64768 /unlink in 93.68 seconds: 691.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4882 (time 1752689299.93 total 10.00 last 488.17) - open/close 10000 (time 1752689309.68 total 19.75 last 524.97) - open/close 15200 (time 1752689319.68 total 29.75 last 519.97) - open/close 20000 (time 1752689328.88 total 38.95 last 521.68) - open/close 25169 (time 1752689338.88 total 48.95 last 516.86) - open/close 30000 (time 1752689348.30 total 58.37 last 512.85) - open/close 35387 (time 1752689358.30 total 68.37 last 538.67) - open/close 40000 (time 1752689367.78 total 77.85 last 486.98) - open/close 44518 (time 1752689377.78 total 87.85 last 451.75) - open/close 48987 (time 1752689387.78 total 97.85 last 446.87) - open/close 50000 (time 1752689390.02 total 100.09 last 452.56) - open/close 54661 (time 1752689400.02 total 110.09 last 466.06) - open/close 59406 (time 1752689410.02 total 120.09 last 474.45) - open/close 63875 (time 1752689420.02 total 130.09 last 446.83) total: 64768 open/close in 132.25 seconds: 489.75 ops/second - /unlink 7794 (time 1752689433.55 total 10.00 last 779.33) - /unlink 10000 (time 1752689436.30 total 12.75 last 802.18) - /unlink 17801 (time 1752689446.30 total 22.75 last 780.02) - /unlink 20000 (time 1752689448.88 total 25.33 last 851.41) - /unlink 27662 (time 1752689458.88 total 35.33 last 766.18) - /unlink 30000 (time 1752689461.89 total 38.34 last 777.89) - /unlink 36692 (time 1752689471.89 total 48.34 last 669.18) - /unlink 40000 (time 1752689476.90 total 53.35 last 660.54) - /unlink 48098 (time 1752689486.90 total 63.35 last 809.80) - /unlink 50000 (time 1752689489.11 total 65.56 last 858.31) - /unlink 58407 (time 1752689499.11 total 75.56 last 840.69) - /unlink 60000 (time 1752689501.22 total 77.67 last 756.10) total: 64768 /unlink in 83.09 seconds: 779.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4522 (time 1752689605.03 total 10.00 last 452.19) - open/close 9079 (time 1752689615.03 total 20.00 last 455.61) - open/close 13761 (time 1752689625.03 total 30.00 last 468.15) - open/close 18393 (time 1752689635.03 total 40.01 last 463.09) - open/close 20000 (time 1752689638.47 total 43.44 last 468.22) - open/close 24543 (time 1752689648.47 total 53.44 last 454.21) - open/close 29158 (time 1752689658.47 total 63.44 last 461.45) - open/close 33581 (time 1752689668.47 total 73.44 last 442.29) - open/close 37957 (time 1752689678.47 total 83.44 last 437.52) - open/close 40000 (time 1752689683.06 total 88.04 last 444.84) - open/close 44357 (time 1752689693.06 total 98.04 last 435.69) - open/close 48588 (time 1752689703.06 total 108.04 last 423.04) - open/close 50000 (time 1752689706.50 total 111.47 last 411.16) - open/close 54167 (time 1752689716.50 total 121.47 last 416.70) - open/close 59159 (time 1752689726.50 total 131.47 last 499.15) - open/close 63538 (time 1752689736.50 total 141.47 last 437.88) total: 64768 open/close in 144.22 seconds: 449.10 ops/second - /unlink 7467 (time 1752689750.60 total 10.00 last 746.67) - /unlink 10000 (time 1752689753.72 total 13.12 last 811.50) - /unlink 17131 (time 1752689763.72 total 23.12 last 713.07) - /unlink 20000 (time 1752689768.03 total 27.43 last 666.12) - /unlink 27005 (time 1752689778.03 total 37.43 last 700.43) - /unlink 30000 (time 1752689782.32 total 41.72 last 697.94) - /unlink 37292 (time 1752689792.32 total 51.72 last 729.15) - /unlink 40000 (time 1752689796.30 total 55.70 last 681.58) - /unlink 47431 (time 1752689806.30 total 65.70 last 743.04) - /unlink 50000 (time 1752689809.60 total 69.00 last 777.56) - /unlink 56780 (time 1752689819.60 total 79.00 last 677.96) - /unlink 60000 (time 1752689824.09 total 83.49 last 717.73) total: 64768 /unlink in 90.67 seconds: 714.29 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg323-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg323-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg323-server stop mds service on oleg323-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg323-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg323-server' oleg323-server: oleg323-server.virtnet: executing unload_modules_local oleg323-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg323-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-client: ssh exited with exit code 2 pdsh@oleg323-client: oleg323-server: ssh exited with exit code 2