== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:59:08 (1755039548) Stopping clients: oleg459-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg459-client.virtnet /mnt/lustre2 (opts:-f) oleg459-server: oleg459-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing load_modules_local oleg459-server: Loading modules from /home/green/git/lustre-release/lustre oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg459-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg459-server: detected 4 online CPUs by sysfs oleg459-server: MODOPTS_LIBCFS= oleg459-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg459-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg459-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg459-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg459-server: oleg459-server.virtnet: executing set_default_debug -1 all pdsh@oleg459-client: oleg459-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg459-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg459-client.virtnet: -o user_xattr,flock 192.168.204.159@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4071 (time 1755039596.90 total 10.00 last 407.09) - open/close 7240 (time 1755039606.90 total 20.00 last 316.87) - open/close 10000 (time 1755039614.49 total 27.59 last 363.77) - open/close 14127 (time 1755039624.49 total 37.59 last 412.62) - open/close 18732 (time 1755039634.49 total 47.59 last 460.47) - open/close 20000 (time 1755039637.87 total 50.97 last 375.21) - open/close 23657 (time 1755039647.87 total 60.97 last 365.59) - open/close 28016 (time 1755039657.87 total 70.98 last 435.81) - open/close 30000 (time 1755039663.08 total 76.18 last 380.97) - open/close 33512 (time 1755039673.08 total 86.18 last 351.15) - open/close 37128 (time 1755039683.08 total 96.19 last 361.56) - open/close 40000 (time 1755039691.05 total 104.15 last 360.43) - open/close 43471 (time 1755039701.05 total 114.16 last 347.05) - open/close 47521 (time 1755039711.05 total 124.16 last 404.96) - open/close 50000 (time 1755039717.00 total 130.10 last 417.17) - open/close 54193 (time 1755039727.00 total 140.10 last 419.28) - open/close 58092 (time 1755039737.00 total 150.10 last 389.82) - open/close 60000 (time 1755039741.36 total 154.46 last 438.00) - open/close 64340 (time 1755039751.36 total 164.46 last 433.97) total: 64768 open/close in 165.59 seconds: 391.14 ops/second - /unlink 7229 (time 1755039763.87 total 10.00 last 722.89) - /unlink 10000 (time 1755039767.37 total 13.50 last 791.08) - /unlink 16952 (time 1755039777.37 total 23.50 last 695.18) - /unlink 20000 (time 1755039781.82 total 27.95 last 685.36) - /unlink 26755 (time 1755039791.82 total 37.95 last 675.44) - /unlink 30000 (time 1755039796.10 total 42.23 last 758.36) - /unlink 36426 (time 1755039806.10 total 52.23 last 642.55) - /unlink 40000 (time 1755039812.17 total 58.30 last 589.29) - /unlink 46292 (time 1755039822.17 total 68.30 last 629.14) - /unlink 50000 (time 1755039827.88 total 74.01 last 648.98) - /unlink 57062 (time 1755039837.88 total 84.01 last 706.14) - /unlink 60000 (time 1755039842.07 total 88.20 last 701.84) total: 64768 /unlink in 94.46 seconds: 685.67 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4628 (time 1755039882.85 total 10.00 last 462.77) - open/close 8444 (time 1755039892.86 total 20.00 last 381.55) - open/close 10000 (time 1755039896.77 total 23.92 last 397.40) - open/close 14556 (time 1755039906.77 total 33.92 last 455.55) - open/close 19281 (time 1755039916.77 total 43.92 last 472.42) - open/close 23705 (time 1755039926.78 total 53.92 last 442.34) - open/close 28223 (time 1755039936.78 total 63.92 last 451.73) - open/close 30000 (time 1755039941.22 total 68.37 last 399.88) - open/close 34282 (time 1755039951.22 total 78.37 last 428.15) - open/close 38830 (time 1755039961.22 total 88.37 last 454.75) - open/close 40000 (time 1755039963.60 total 90.75 last 491.11) - open/close 44498 (time 1755039973.61 total 100.75 last 449.72) - open/close 48699 (time 1755039983.61 total 110.75 last 420.02) - open/close 50000 (time 1755039986.64 total 113.79 last 428.64) - open/close 54184 (time 1755039996.64 total 123.79 last 418.39) - open/close 58057 (time 1755040006.64 total 133.79 last 387.30) - open/close 60000 (time 1755040011.63 total 138.78 last 389.32) - open/close 64440 (time 1755040021.64 total 148.78 last 443.98) total: 64768 open/close in 149.71 seconds: 432.63 ops/second - /unlink 7327 (time 1755040033.84 total 10.00 last 732.36) - /unlink 10000 (time 1755040037.82 total 13.99 last 671.30) - /unlink 17183 (time 1755040047.82 total 23.99 last 718.25) - /unlink 20000 (time 1755040051.36 total 27.52 last 797.36) - /unlink 27441 (time 1755040061.36 total 37.52 last 744.08) - /unlink 30000 (time 1755040064.95 total 41.11 last 712.83) - /unlink 36863 (time 1755040074.95 total 51.11 last 686.29) - /unlink 40000 (time 1755040078.73 total 54.90 last 828.42) - /unlink 47597 (time 1755040088.73 total 64.90 last 759.67) - /unlink 50000 (time 1755040091.77 total 67.94 last 790.87) - /unlink 57595 (time 1755040101.77 total 77.94 last 759.48) - /unlink 60000 (time 1755040105.29 total 81.45 last 683.55) total: 64768 /unlink in 87.15 seconds: 743.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3713 (time 1755040193.26 total 10.00 last 371.23) - open/close 7067 (time 1755040203.26 total 20.00 last 335.36) - open/close 10000 (time 1755040212.11 total 28.85 last 331.58) - open/close 13547 (time 1755040222.11 total 38.85 last 354.67) - open/close 16576 (time 1755040232.11 total 48.85 last 302.80) - open/close 20000 (time 1755040241.91 total 58.65 last 349.38) - open/close 24272 (time 1755040251.91 total 68.65 last 427.19) - open/close 28317 (time 1755040261.91 total 78.65 last 404.50) - open/close 30000 (time 1755040266.90 total 83.64 last 337.69) - open/close 34043 (time 1755040276.90 total 93.64 last 404.24) - open/close 37764 (time 1755040286.90 total 103.64 last 372.05) - open/close 40000 (time 1755040292.74 total 109.48 last 382.72) - open/close 44168 (time 1755040302.74 total 119.48 last 416.80) - open/close 47507 (time 1755040312.74 total 129.48 last 333.88) - open/close 50000 (time 1755040318.58 total 135.32 last 427.27) - open/close 54192 (time 1755040328.58 total 145.32 last 419.19) - open/close 57449 (time 1755040338.58 total 155.32 last 325.65) - open/close 60000 (time 1755040347.85 total 164.59 last 275.25) - open/close 63939 (time 1755040357.85 total 174.59 last 393.83) total: 64768 open/close in 176.55 seconds: 366.86 ops/second - /unlink 5928 (time 1755040371.33 total 10.00 last 592.77) - /unlink 10000 (time 1755040377.11 total 15.78 last 704.32) - /unlink 15575 (time 1755040387.11 total 25.78 last 557.49) - /unlink 20000 (time 1755040394.39 total 33.06 last 607.72) - /unlink 26248 (time 1755040404.40 total 43.07 last 624.70) - /unlink 30000 (time 1755040409.59 total 48.26 last 721.95) - /unlink 36185 (time 1755040419.59 total 58.26 last 618.38) - /unlink 40000 (time 1755040426.83 total 65.50 last 527.39) - /unlink 46695 (time 1755040436.83 total 75.50 last 669.47) - /unlink 50000 (time 1755040443.06 total 81.73 last 529.98) - /unlink 55555 (time 1755040453.07 total 91.74 last 555.46) - /unlink 60000 (time 1755040461.48 total 100.15 last 528.11) total: 64768 /unlink in 107.46 seconds: 602.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3845 (time 1755040556.61 total 10.00 last 384.45) - open/close 8604 (time 1755040566.61 total 20.00 last 475.87) - open/close 10000 (time 1755040569.56 total 22.95 last 473.70) - open/close 13926 (time 1755040579.56 total 32.95 last 392.59) - open/close 17612 (time 1755040589.56 total 42.95 last 368.46) - open/close 20000 (time 1755040597.14 total 50.53 last 315.12) - open/close 24104 (time 1755040607.14 total 60.53 last 410.26) - open/close 27645 (time 1755040617.14 total 70.53 last 354.09) - open/close 30000 (time 1755040622.19 total 75.58 last 466.94) - open/close 34678 (time 1755040632.19 total 85.58 last 467.74) - open/close 39406 (time 1755040642.19 total 95.58 last 472.72) - open/close 44074 (time 1755040652.19 total 105.58 last 466.70) - open/close 48690 (time 1755040662.19 total 115.59 last 461.51) - open/close 50000 (time 1755040665.04 total 118.43 last 460.56) - open/close 54379 (time 1755040675.04 total 128.43 last 437.85) - open/close 56899 (time 1755040685.17 total 138.56 last 248.79) - open/close 60000 (time 1755040694.60 total 147.99 last 328.67) - open/close 62462 (time 1755040704.61 total 158.00 last 246.16) total: 64768 open/close in 164.42 seconds: 393.92 ops/second - /unlink 5143 (time 1755040722.34 total 10.00 last 514.27) - /unlink 10000 (time 1755040729.49 total 17.15 last 679.21) - /unlink 16571 (time 1755040739.49 total 27.15 last 657.07) - /unlink 20000 (time 1755040744.37 total 32.03 last 703.37) - /unlink 24904 (time 1755040754.37 total 42.03 last 490.40) - /unlink 30000 (time 1755040763.42 total 51.07 last 563.26) - /unlink 36583 (time 1755040773.42 total 61.07 last 658.26) - /unlink 40000 (time 1755040778.20 total 65.86 last 713.87) - /unlink 46335 (time 1755040788.20 total 75.86 last 633.43) - /unlink 50000 (time 1755040793.44 total 81.10 last 699.55) - /unlink 57490 (time 1755040803.44 total 91.10 last 748.94) - /unlink 60000 (time 1755040806.32 total 93.98 last 873.47) total: 64768 /unlink in 99.85 seconds: 648.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4577 (time 1755040858.43 total 10.00 last 457.65) - open/close 9434 (time 1755040868.44 total 20.00 last 485.61) - open/close 14203 (time 1755040878.44 total 30.00 last 476.88) - open/close 18909 (time 1755040888.44 total 40.00 last 470.56) - open/close 20000 (time 1755040890.73 total 42.30 last 475.21) - open/close 24694 (time 1755040900.73 total 52.30 last 469.32) - open/close 29417 (time 1755040910.73 total 62.30 last 472.28) - open/close 34206 (time 1755040920.74 total 72.30 last 478.90) - open/close 39021 (time 1755040930.74 total 82.30 last 481.48) - open/close 40000 (time 1755040932.74 total 84.31 last 487.32) - open/close 44629 (time 1755040942.74 total 94.31 last 462.87) - open/close 48946 (time 1755040952.75 total 104.31 last 431.60) - open/close 50000 (time 1755040955.79 total 107.35 last 346.90) - open/close 54232 (time 1755040965.79 total 117.35 last 423.16) - open/close 58690 (time 1755040975.79 total 127.36 last 445.73) - open/close 60000 (time 1755040978.68 total 130.25 last 452.69) - open/close 64540 (time 1755040988.68 total 140.25 last 453.91) total: 64768 open/close in 140.85 seconds: 459.84 ops/second - /unlink 6278 (time 1755041000.56 total 10.00 last 627.57) - /unlink 10000 (time 1755041006.25 total 15.69 last 654.77) - /unlink 15729 (time 1755041016.25 total 25.69 last 572.84) - /unlink 20000 (time 1755041021.66 total 31.10 last 788.69) - /unlink 27128 (time 1755041031.66 total 41.10 last 712.78) - /unlink 30000 (time 1755041035.35 total 44.79 last 780.09) - /unlink 37439 (time 1755041045.35 total 54.79 last 743.88) - /unlink 40000 (time 1755041049.04 total 58.48 last 693.22) - /unlink 47555 (time 1755041059.04 total 68.48 last 755.49) - /unlink 50000 (time 1755041062.19 total 71.63 last 775.67) - /unlink 57458 (time 1755041072.19 total 81.63 last 745.77) - /unlink 60000 (time 1755041075.73 total 85.17 last 719.55) total: 64768 /unlink in 91.76 seconds: 705.85 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg459-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg459-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg459-server stop mds service on oleg459-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg459-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg459-server' oleg459-server: oleg459-server.virtnet: executing unload_modules_local oleg459-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg459-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-client: ssh exited with exit code 2 pdsh@oleg459-client: oleg459-server: ssh exited with exit code 2