== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:54:40 (1761141280) Stopping clients: oleg106-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg106-client.virtnet /mnt/lustre2 (opts:-f) oleg106-server: oleg106-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing load_modules_local oleg106-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg106-server: Loading modules from /home/green/git/lustre-release/lustre oleg106-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg106-server: detected 4 online CPUs by sysfs oleg106-server: MODOPTS_LIBCFS= oleg106-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg106-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg106-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg106-server: oleg106-server.virtnet: executing set_default_debug -1 all pdsh@oleg106-client: oleg106-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg106-client.virtnet: -o user_xattr,flock 192.168.201.106@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3483 (time 1761141334.93 total 10.00 last 348.28) - open/close 5537 (time 1761141344.93 total 20.00 last 205.35) - open/close 7534 (time 1761141354.94 total 30.01 last 199.66) - open/close 9620 (time 1761141364.94 total 40.01 last 208.51) - open/close 11602 (time 1761141374.94 total 50.01 last 198.13) - open/close 13898 (time 1761141384.95 total 60.02 last 229.51) - open/close 15910 (time 1761141394.96 total 70.03 last 200.95) - open/close 17630 (time 1761141404.96 total 80.03 last 171.91) - open/close 19680 (time 1761141414.97 total 90.04 last 204.90) - open/close 21642 (time 1761141424.97 total 100.04 last 196.18) - open/close 23870 (time 1761141434.97 total 110.04 last 222.76) - open/close 26976 (time 1761141444.97 total 120.04 last 310.56) - open/close 30000 (time 1761141454.01 total 129.08 last 334.66) - open/close 32987 (time 1761141464.01 total 139.08 last 298.66) - open/close 35338 (time 1761141474.01 total 149.08 last 235.03) - open/close 38324 (time 1761141484.02 total 159.09 last 298.54) - open/close 40000 (time 1761141489.02 total 164.09 last 335.13) - open/close 43713 (time 1761141499.02 total 174.09 last 371.30) - open/close 47255 (time 1761141509.02 total 184.09 last 354.18) - open/close 50000 (time 1761141516.23 total 191.30 last 380.79) - open/close 53968 (time 1761141526.23 total 201.30 last 396.74) - open/close 57286 (time 1761141536.23 total 211.30 last 331.73) - open/close 60000 (time 1761141545.55 total 220.62 last 291.10) - open/close 63627 (time 1761141555.55 total 230.62 last 362.69) total: 64768 open/close in 233.81 seconds: 277.01 ops/second - /unlink 5177 (time 1761141570.63 total 10.00 last 517.67) - /unlink 9993 (time 1761141580.63 total 20.00 last 481.48) - /unlink 14398 (time 1761141590.63 total 30.00 last 440.50) - /unlink 18831 (time 1761141600.63 total 40.00 last 443.30) - /unlink 20000 (time 1761141603.06 total 42.43 last 481.66) - /unlink 24775 (time 1761141613.06 total 52.43 last 477.49) - /unlink 28958 (time 1761141623.06 total 62.43 last 418.18) - /unlink 30000 (time 1761141625.78 total 65.15 last 383.51) - /unlink 34202 (time 1761141635.78 total 75.15 last 420.11) - /unlink 38873 (time 1761141645.78 total 85.15 last 467.00) - /unlink 40000 (time 1761141648.56 total 87.93 last 405.48) - /unlink 44113 (time 1761141658.56 total 97.94 last 411.21) - /unlink 48943 (time 1761141668.57 total 107.94 last 482.94) - /unlink 54641 (time 1761141678.57 total 117.94 last 569.78) - /unlink 59783 (time 1761141688.57 total 127.94 last 514.20) - /unlink 64071 (time 1761141698.57 total 137.94 last 428.75) total: 64768 /unlink in 140.09 seconds: 462.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2518 (time 1761141732.13 total 10.00 last 251.73) - open/close 4876 (time 1761141742.13 total 20.00 last 235.78) - open/close 7239 (time 1761141752.13 total 30.01 last 236.20) - open/close 9634 (time 1761141762.13 total 40.01 last 239.48) - open/close 12492 (time 1761141772.13 total 50.01 last 285.75) - open/close 14647 (time 1761141782.14 total 60.01 last 215.43) - open/close 16647 (time 1761141792.14 total 70.02 last 199.96) - open/close 19034 (time 1761141802.14 total 80.02 last 238.65) - open/close 20000 (time 1761141805.14 total 83.02 last 321.89) - open/close 22958 (time 1761141815.14 total 93.02 last 295.76) - open/close 25903 (time 1761141825.15 total 103.03 last 294.30) - open/close 29014 (time 1761141835.15 total 113.03 last 311.03) - open/close 30000 (time 1761141837.97 total 115.85 last 350.04) - open/close 33649 (time 1761141847.97 total 125.85 last 364.87) - open/close 37169 (time 1761141857.97 total 135.85 last 351.99) - open/close 40000 (time 1761141865.42 total 143.30 last 380.00) - open/close 43202 (time 1761141875.42 total 153.30 last 320.09) - open/close 46524 (time 1761141885.43 total 163.30 last 332.15) - open/close 49530 (time 1761141895.43 total 173.31 last 300.49) - open/close 52796 (time 1761141905.43 total 183.31 last 326.55) - open/close 56204 (time 1761141915.43 total 193.31 last 340.78) - open/close 58452 (time 1761141925.43 total 203.31 last 224.74) - open/close 60000 (time 1761141931.03 total 208.91 last 276.58) - open/close 63496 (time 1761141941.03 total 218.91 last 349.59) total: 64768 open/close in 223.14 seconds: 290.25 ops/second - /unlink 4604 (time 1761141957.05 total 10.00 last 460.38) - /unlink 8991 (time 1761141967.05 total 20.00 last 438.65) - /unlink 13632 (time 1761141977.05 total 30.00 last 464.06) - /unlink 18311 (time 1761141987.05 total 40.00 last 467.85) - /unlink 20000 (time 1761141989.97 total 42.92 last 578.56) - /unlink 24068 (time 1761141999.97 total 52.92 last 406.74) - /unlink 28082 (time 1761142009.97 total 62.93 last 401.36) - /unlink 30000 (time 1761142014.54 total 67.49 last 420.02) - /unlink 34386 (time 1761142024.54 total 77.49 last 438.57) - /unlink 38977 (time 1761142034.54 total 87.49 last 459.04) - /unlink 40000 (time 1761142036.55 total 89.50 last 510.07) - /unlink 45068 (time 1761142046.55 total 99.50 last 506.79) - /unlink 50000 (time 1761142055.81 total 108.76 last 532.79) - /unlink 54788 (time 1761142065.81 total 118.76 last 478.78) - /unlink 60000 (time 1761142074.35 total 127.31 last 609.73) total: 64768 /unlink in 135.87 seconds: 476.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3545 (time 1761142113.73 total 10.00 last 354.41) - open/close 7564 (time 1761142123.73 total 20.00 last 401.84) - open/close 10000 (time 1761142129.03 total 25.30 last 459.69) - open/close 14639 (time 1761142139.03 total 35.30 last 463.86) - open/close 18892 (time 1761142149.03 total 45.30 last 425.27) - open/close 20000 (time 1761142151.17 total 47.44 last 518.52) - open/close 25206 (time 1761142161.17 total 57.44 last 520.51) - open/close 30000 (time 1761142170.44 total 66.72 last 517.01) - open/close 35058 (time 1761142180.44 total 76.72 last 505.73) - open/close 39987 (time 1761142190.44 total 86.72 last 492.84) - open/close 44182 (time 1761142200.44 total 96.72 last 419.47) - open/close 48597 (time 1761142210.44 total 106.72 last 441.40) - open/close 50000 (time 1761142213.57 total 109.84 last 449.31) - open/close 54673 (time 1761142223.57 total 119.84 last 467.30) - open/close 59210 (time 1761142233.59 total 129.87 last 452.54) - open/close 60000 (time 1761142235.71 total 131.99 last 373.08) - open/close 63921 (time 1761142245.71 total 141.99 last 392.02) total: 64768 open/close in 144.17 seconds: 449.24 ops/second - /unlink 7540 (time 1761142259.36 total 10.00 last 753.91) - /unlink 10000 (time 1761142262.21 total 12.84 last 865.40) - /unlink 17767 (time 1761142272.21 total 22.84 last 776.70) - /unlink 20000 (time 1761142275.52 total 26.16 last 672.96) - /unlink 26872 (time 1761142285.52 total 36.16 last 687.15) - /unlink 30000 (time 1761142290.34 total 40.98 last 649.97) - /unlink 36601 (time 1761142300.34 total 50.98 last 660.05) - /unlink 40000 (time 1761142305.00 total 55.63 last 729.69) - /unlink 47517 (time 1761142315.00 total 65.63 last 751.67) - /unlink 50000 (time 1761142317.92 total 68.56 last 849.96) - /unlink 56935 (time 1761142327.92 total 78.56 last 693.43) - /unlink 60000 (time 1761142332.15 total 82.79 last 723.78) total: 64768 /unlink in 90.39 seconds: 716.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4321 (time 1761142366.72 total 10.00 last 432.02) - open/close 8759 (time 1761142376.72 total 20.00 last 443.78) - open/close 10000 (time 1761142379.52 total 22.80 last 443.57) - open/close 14337 (time 1761142389.52 total 32.80 last 433.64) - open/close 19192 (time 1761142399.52 total 42.80 last 485.49) - open/close 23194 (time 1761142409.52 total 52.80 last 400.16) - open/close 27433 (time 1761142419.52 total 62.80 last 423.88) - open/close 30000 (time 1761142425.42 total 68.70 last 435.34) - open/close 34190 (time 1761142435.42 total 78.70 last 419.00) - open/close 38159 (time 1761142445.42 total 88.70 last 396.86) - open/close 40000 (time 1761142449.83 total 93.12 last 417.00) - open/close 44390 (time 1761142459.83 total 103.12 last 438.94) - open/close 49112 (time 1761142469.84 total 113.12 last 472.13) - open/close 50000 (time 1761142471.84 total 115.12 last 444.00) - open/close 54346 (time 1761142481.84 total 125.12 last 434.53) - open/close 58331 (time 1761142491.84 total 135.12 last 398.50) - open/close 60000 (time 1761142495.77 total 139.05 last 424.30) - open/close 63693 (time 1761142505.77 total 149.05 last 369.29) total: 64768 open/close in 151.84 seconds: 426.55 ops/second - /unlink 7265 (time 1761142519.99 total 10.00 last 726.42) - /unlink 10000 (time 1761142523.64 total 13.65 last 750.22) - /unlink 16436 (time 1761142533.64 total 23.65 last 643.57) - /unlink 20000 (time 1761142538.84 total 28.84 last 685.79) - /unlink 26005 (time 1761142548.84 total 38.84 last 600.46) - /unlink 30000 (time 1761142555.33 total 45.34 last 615.01) - /unlink 36423 (time 1761142565.34 total 55.35 last 641.95) - /unlink 40000 (time 1761142570.52 total 60.53 last 690.04) - /unlink 47196 (time 1761142580.52 total 70.53 last 719.54) - /unlink 50000 (time 1761142584.21 total 74.22 last 760.20) - /unlink 57595 (time 1761142594.21 total 84.22 last 759.50) - /unlink 60000 (time 1761142597.50 total 87.51 last 731.29) total: 64768 /unlink in 93.95 seconds: 689.37 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5190 (time 1761142640.85 total 10.00 last 518.91) - open/close 9804 (time 1761142650.85 total 20.00 last 461.31) - open/close 14429 (time 1761142660.86 total 30.01 last 462.41) - open/close 19038 (time 1761142670.88 total 40.03 last 459.82) - open/close 20000 (time 1761142672.94 total 42.09 last 467.26) - open/close 24605 (time 1761142682.94 total 52.09 last 460.42) - open/close 29216 (time 1761142692.94 total 62.09 last 461.04) - open/close 33668 (time 1761142702.94 total 72.09 last 445.12) - open/close 38297 (time 1761142712.94 total 82.09 last 462.86) - open/close 40000 (time 1761142716.56 total 85.71 last 470.92) - open/close 44530 (time 1761142726.56 total 95.71 last 452.95) - open/close 48888 (time 1761142736.56 total 105.71 last 435.72) - open/close 50000 (time 1761142739.06 total 108.21 last 445.52) - open/close 54302 (time 1761142749.06 total 118.21 last 430.13) - open/close 58644 (time 1761142759.06 total 128.21 last 434.14) - open/close 60000 (time 1761142762.28 total 131.43 last 422.00) - open/close 64435 (time 1761142772.28 total 141.43 last 443.49) total: 64768 open/close in 142.20 seconds: 455.46 ops/second - /unlink 7424 (time 1761142784.44 total 10.00 last 742.37) - /unlink 10000 (time 1761142787.99 total 13.54 last 727.33) - /unlink 16967 (time 1761142797.99 total 23.54 last 696.64) - /unlink 20000 (time 1761142802.41 total 27.97 last 685.73) - /unlink 27278 (time 1761142812.41 total 37.97 last 727.77) - /unlink 30000 (time 1761142816.64 total 42.19 last 643.85) - /unlink 37139 (time 1761142826.64 total 52.19 last 713.84) - /unlink 40000 (time 1761142830.45 total 56.01 last 749.78) - /unlink 46854 (time 1761142840.45 total 66.01 last 685.40) - /unlink 50000 (time 1761142845.12 total 70.68 last 674.33) - /unlink 56898 (time 1761142855.12 total 80.68 last 689.76) - /unlink 60000 (time 1761142859.55 total 85.11 last 699.87) total: 64768 /unlink in 91.80 seconds: 705.53 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg106-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg106-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg106-server stop mds service on oleg106-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg106-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg106-server' oleg106-server: oleg106-server.virtnet: executing unload_modules_local oleg106-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg106-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-client: ssh exited with exit code 2 pdsh@oleg106-client: oleg106-server: ssh exited with exit code 2