== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:08:51 (1761962931) Stopping clients: oleg408-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg408-client.virtnet /mnt/lustre2 (opts:-f) oleg408-server: oleg408-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing load_modules_local oleg408-server: Loading modules from /home/green/git/lustre-release/lustre oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1030: echo: write error: Device or resource busy oleg408-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1031: echo: write error: Device or resource busy oleg408-server: detected 4 online CPUs by sysfs oleg408-server: MODOPTS_LIBCFS= oleg408-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg408-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg408-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg408-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg408-server: oleg408-server.virtnet: executing set_default_debug -1 all pdsh@oleg408-client: oleg408-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg408-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg408-client.virtnet: -o user_xattr,flock 192.168.204.108@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2690 (time 1761962978.47 total 10.00 last 268.94) - open/close 5276 (time 1761962988.48 total 20.00 last 258.57) - open/close 7807 (time 1761962998.48 total 30.01 last 253.04) - open/close 10000 (time 1761963006.94 total 38.47 last 259.05) - open/close 11979 (time 1761963016.94 total 48.47 last 197.90) - open/close 14469 (time 1761963026.95 total 58.47 last 248.97) - open/close 16688 (time 1761963036.95 total 68.48 last 221.83) - open/close 18985 (time 1761963046.95 total 78.48 last 229.67) - open/close 20000 (time 1761963050.80 total 82.33 last 263.37) - open/close 23122 (time 1761963060.81 total 92.33 last 312.15) - open/close 26198 (time 1761963070.81 total 102.33 last 307.60) - open/close 30000 (time 1761963080.61 total 112.14 last 387.79) - open/close 33644 (time 1761963090.61 total 122.14 last 364.37) - open/close 37607 (time 1761963100.61 total 132.14 last 396.23) - open/close 40000 (time 1761963107.23 total 138.76 last 361.49) - open/close 43210 (time 1761963117.23 total 148.76 last 320.97) - open/close 46565 (time 1761963127.23 total 158.76 last 335.47) - open/close 49933 (time 1761963137.24 total 168.76 last 336.72) - open/close 53608 (time 1761963147.24 total 178.76 last 367.46) - open/close 57789 (time 1761963157.24 total 188.77 last 418.03) - open/close 60000 (time 1761963162.91 total 194.44 last 389.89) - open/close 63572 (time 1761963172.91 total 204.44 last 357.15) total: 64768 open/close in 207.32 seconds: 312.40 ops/second - /unlink 7148 (time 1761963187.08 total 10.00 last 714.71) - /unlink 10000 (time 1761963190.62 total 13.54 last 806.47) - /unlink 17015 (time 1761963200.62 total 23.54 last 701.41) - /unlink 20000 (time 1761963205.31 total 28.23 last 636.50) - /unlink 27041 (time 1761963215.31 total 38.23 last 704.06) - /unlink 30000 (time 1761963219.42 total 42.34 last 719.97) - /unlink 37238 (time 1761963229.42 total 52.34 last 723.78) - /unlink 40000 (time 1761963233.09 total 56.02 last 751.03) - /unlink 46877 (time 1761963243.09 total 66.02 last 687.70) - /unlink 50000 (time 1761963247.60 total 70.52 last 692.79) - /unlink 56332 (time 1761963257.60 total 80.53 last 633.18) - /unlink 60000 (time 1761963263.18 total 86.10 last 658.22) total: 64768 /unlink in 92.52 seconds: 700.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4365 (time 1761963299.01 total 10.00 last 436.41) - open/close 9233 (time 1761963309.01 total 20.00 last 486.71) - open/close 13800 (time 1761963319.02 total 30.00 last 456.67) - open/close 18850 (time 1761963329.02 total 40.01 last 504.97) - open/close 20000 (time 1761963331.38 total 42.37 last 486.92) - open/close 24916 (time 1761963341.38 total 52.37 last 491.50) - open/close 29764 (time 1761963351.38 total 62.37 last 484.74) - open/close 34136 (time 1761963361.38 total 72.37 last 437.14) - open/close 38690 (time 1761963371.38 total 82.37 last 455.39) - open/close 40000 (time 1761963374.39 total 85.38 last 435.86) - open/close 44410 (time 1761963384.39 total 95.38 last 440.89) - open/close 48716 (time 1761963394.39 total 105.38 last 430.56) - open/close 50000 (time 1761963397.21 total 108.20 last 455.23) - open/close 54666 (time 1761963407.28 total 118.27 last 463.41) - open/close 59488 (time 1761963417.28 total 128.27 last 482.13) - open/close 64236 (time 1761963427.28 total 138.27 last 474.73) total: 64768 open/close in 139.47 seconds: 464.39 ops/second - /unlink 8493 (time 1761963439.76 total 10.00 last 849.28) - /unlink 16300 (time 1761963449.76 total 20.00 last 780.40) - /unlink 20000 (time 1761963453.90 total 24.15 last 893.36) - /unlink 26953 (time 1761963463.90 total 34.15 last 695.27) - /unlink 30000 (time 1761963468.14 total 38.39 last 718.35) - /unlink 37796 (time 1761963478.14 total 48.39 last 779.59) - /unlink 40000 (time 1761963481.17 total 51.42 last 727.44) - /unlink 48169 (time 1761963491.17 total 61.42 last 816.85) - /unlink 56936 (time 1761963501.18 total 71.42 last 876.64) - /unlink 60000 (time 1761963504.32 total 74.56 last 975.71) total: 64768 /unlink in 80.21 seconds: 807.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4329 (time 1761963544.45 total 10.00 last 432.82) - open/close 8866 (time 1761963554.45 total 20.00 last 453.67) - open/close 10000 (time 1761963557.20 total 22.75 last 412.70) - open/close 14871 (time 1761963567.20 total 32.75 last 487.09) - open/close 19789 (time 1761963577.20 total 42.75 last 491.78) - open/close 24946 (time 1761963587.20 total 52.75 last 515.63) - open/close 30000 (time 1761963597.15 total 62.70 last 508.10) - open/close 34717 (time 1761963607.15 total 72.70 last 471.53) - open/close 39579 (time 1761963617.15 total 82.70 last 486.16) - open/close 44456 (time 1761963627.15 total 92.70 last 487.68) - open/close 49109 (time 1761963637.16 total 102.71 last 465.21) - open/close 53901 (time 1761963647.16 total 112.71 last 479.16) - open/close 58791 (time 1761963657.23 total 122.78 last 485.60) - open/close 60000 (time 1761963659.66 total 125.21 last 496.53) total: 64768 open/close in 134.86 seconds: 480.25 ops/second - /unlink 8478 (time 1761963680.55 total 10.00 last 847.78) - /unlink 16866 (time 1761963690.56 total 20.00 last 838.75) - /unlink 20000 (time 1761963694.10 total 23.55 last 884.06) - /unlink 28776 (time 1761963704.10 total 33.55 last 877.55) - /unlink 36414 (time 1761963714.10 total 43.55 last 763.56) - /unlink 40000 (time 1761963718.89 total 48.33 last 749.69) - /unlink 47204 (time 1761963728.89 total 58.33 last 720.39) - /unlink 50000 (time 1761963732.93 total 62.37 last 692.38) - /unlink 57371 (time 1761963742.93 total 72.37 last 737.09) - /unlink 60000 (time 1761963746.64 total 76.08 last 708.55) total: 64768 /unlink in 82.31 seconds: 786.88 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4678 (time 1761963786.30 total 10.00 last 467.75) - open/close 9252 (time 1761963796.30 total 20.00 last 457.36) - open/close 13952 (time 1761963806.30 total 30.00 last 469.95) - open/close 18681 (time 1761963816.30 total 40.00 last 472.82) - open/close 20000 (time 1761963819.14 total 42.84 last 465.68) - open/close 24663 (time 1761963829.14 total 52.84 last 466.27) - open/close 29259 (time 1761963839.14 total 62.84 last 459.58) - open/close 33825 (time 1761963849.14 total 72.84 last 456.54) - open/close 38182 (time 1761963859.14 total 82.84 last 435.63) - open/close 40000 (time 1761963863.46 total 87.17 last 420.31) - open/close 44498 (time 1761963873.46 total 97.17 last 449.80) - open/close 48950 (time 1761963883.47 total 107.17 last 445.19) - open/close 50000 (time 1761963885.89 total 109.59 last 432.39) - open/close 54439 (time 1761963895.89 total 119.60 last 443.84) - open/close 58573 (time 1761963905.90 total 129.60 last 413.36) - open/close 60000 (time 1761963909.24 total 132.94 last 426.48) - open/close 64398 (time 1761963919.24 total 142.95 last 439.70) total: 64768 open/close in 143.93 seconds: 450.01 ops/second - /unlink 7134 (time 1761963931.61 total 10.00 last 713.37) - /unlink 10000 (time 1761963936.05 total 14.44 last 645.88) - /unlink 17126 (time 1761963946.16 total 24.55 last 704.71) - /unlink 20000 (time 1761963950.55 total 28.94 last 654.56) - /unlink 27328 (time 1761963960.56 total 38.94 last 732.73) - /unlink 30000 (time 1761963964.70 total 43.09 last 644.12) - /unlink 36629 (time 1761963974.70 total 53.09 last 662.87) - /unlink 40000 (time 1761963979.54 total 57.93 last 696.87) - /unlink 46145 (time 1761963989.54 total 67.93 last 614.48) - /unlink 50000 (time 1761963995.01 total 73.40 last 704.92) - /unlink 57383 (time 1761964005.01 total 83.40 last 738.26) - /unlink 60000 (time 1761964008.73 total 87.11 last 704.33) total: 64768 /unlink in 93.48 seconds: 692.87 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4791 (time 1761964046.00 total 10.00 last 479.05) - open/close 9429 (time 1761964056.00 total 20.00 last 463.72) - open/close 14006 (time 1761964066.01 total 30.00 last 457.67) - open/close 18685 (time 1761964076.01 total 40.00 last 467.85) - open/close 20000 (time 1761964078.88 total 42.88 last 457.48) - open/close 24424 (time 1761964088.88 total 52.88 last 442.39) - open/close 29065 (time 1761964098.88 total 62.88 last 464.01) - open/close 33576 (time 1761964108.88 total 72.88 last 451.08) - open/close 38011 (time 1761964118.88 total 82.88 last 443.50) - open/close 40000 (time 1761964123.53 total 87.53 last 427.61) - open/close 44491 (time 1761964133.54 total 97.53 last 449.05) - open/close 49128 (time 1761964143.54 total 107.54 last 463.60) - open/close 53725 (time 1761964153.54 total 117.54 last 459.65) - open/close 58149 (time 1761964163.54 total 127.54 last 442.36) - open/close 60000 (time 1761964168.02 total 132.02 last 413.30) - open/close 64364 (time 1761964178.02 total 142.02 last 436.34) total: 64768 open/close in 142.89 seconds: 453.26 ops/second - /unlink 7683 (time 1761964190.18 total 10.00 last 768.26) - /unlink 10000 (time 1761964193.51 total 13.32 last 697.04) - /unlink 17386 (time 1761964203.51 total 23.32 last 738.58) - /unlink 20000 (time 1761964206.41 total 26.22 last 901.37) - /unlink 27617 (time 1761964216.41 total 36.23 last 761.63) - /unlink 30000 (time 1761964219.43 total 39.25 last 788.55) - /unlink 37293 (time 1761964229.43 total 49.25 last 729.27) - /unlink 40000 (time 1761964234.17 total 53.99 last 571.10) - /unlink 46781 (time 1761964244.17 total 63.99 last 678.03) - /unlink 50000 (time 1761964248.03 total 67.85 last 834.67) - /unlink 57513 (time 1761964258.03 total 77.85 last 751.30) - /unlink 60000 (time 1761964261.79 total 81.61 last 660.87) total: 64768 /unlink in 88.38 seconds: 732.84 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg408-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg408-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg408-server stop mds service on oleg408-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg408-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg408-server' oleg408-server: oleg408-server.virtnet: executing unload_modules_local oleg408-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg408-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-client: ssh exited with exit code 2 pdsh@oleg408-client: oleg408-server: ssh exited with exit code 2