== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:40:49 (1755096049) Stopping clients: oleg351-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg351-client.virtnet /mnt/lustre2 (opts:-f) oleg351-server: oleg351-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing load_modules_local oleg351-server: Loading modules from /home/green/git/lustre-release/lustre oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1024: echo: write error: Device or resource busy oleg351-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1025: echo: write error: Device or resource busy oleg351-server: detected 4 online CPUs by sysfs oleg351-server: MODOPTS_LIBCFS= oleg351-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg351-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg351-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4515 (time 1755096095.09 total 10.00 last 451.44) - open/close 7729 (time 1755096105.09 total 20.00 last 321.38) - open/close 10000 (time 1755096112.15 total 27.06 last 321.59) - open/close 13759 (time 1755096122.16 total 37.06 last 375.89) - open/close 17615 (time 1755096132.16 total 47.07 last 385.55) - open/close 20000 (time 1755096140.02 total 54.93 last 303.40) - open/close 23400 (time 1755096150.02 total 64.93 last 339.94) - open/close 26938 (time 1755096160.02 total 74.93 last 353.71) - open/close 30000 (time 1755096169.97 total 84.88 last 307.73) - open/close 33526 (time 1755096179.97 total 94.88 last 352.52) - open/close 37013 (time 1755096189.97 total 104.88 last 348.70) - open/close 40000 (time 1755096198.56 total 113.47 last 347.96) - open/close 43550 (time 1755096208.56 total 123.47 last 354.91) - open/close 46729 (time 1755096218.56 total 133.47 last 317.85) - open/close 49828 (time 1755096228.56 total 143.47 last 309.85) - open/close 53710 (time 1755096238.57 total 153.48 last 388.12) - open/close 56754 (time 1755096248.57 total 163.48 last 304.35) - open/close 59733 (time 1755096258.57 total 173.48 last 297.90) - open/close 63176 (time 1755096268.57 total 183.48 last 344.27) total: 64768 open/close in 188.32 seconds: 343.93 ops/second - /unlink 4441 (time 1755096284.96 total 10.00 last 444.04) - /unlink 9366 (time 1755096294.97 total 20.00 last 492.40) - /unlink 13953 (time 1755096304.97 total 30.01 last 458.62) - /unlink 17770 (time 1755096314.97 total 40.01 last 381.70) - /unlink 20000 (time 1755096320.49 total 45.52 last 404.02) - /unlink 24216 (time 1755096330.49 total 55.53 last 421.51) - /unlink 29209 (time 1755096340.49 total 65.53 last 499.15) - /unlink 34210 (time 1755096350.49 total 75.53 last 499.98) - /unlink 38759 (time 1755096360.50 total 85.53 last 454.82) - /unlink 40000 (time 1755096362.69 total 87.73 last 565.94) - /unlink 44119 (time 1755096372.69 total 97.73 last 411.83) - /unlink 48328 (time 1755096382.69 total 107.73 last 420.85) - /unlink 50000 (time 1755096386.93 total 111.96 last 394.81) - /unlink 54360 (time 1755096396.93 total 121.97 last 435.97) - /unlink 58714 (time 1755096406.93 total 131.97 last 435.34) - /unlink 60000 (time 1755096410.24 total 135.28 last 388.29) total: 64768 /unlink in 144.76 seconds: 447.43 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3428 (time 1755096448.11 total 10.00 last 342.75) - open/close 6969 (time 1755096458.11 total 20.00 last 354.02) - open/close 10000 (time 1755096466.03 total 27.93 last 382.50) - open/close 13553 (time 1755096476.03 total 37.93 last 355.23) - open/close 17685 (time 1755096486.04 total 47.93 last 413.08) - open/close 20000 (time 1755096492.08 total 53.98 last 382.81) - open/close 23537 (time 1755096502.09 total 63.98 last 353.68) - open/close 27190 (time 1755096512.09 total 73.98 last 365.29) - open/close 30000 (time 1755096520.25 total 82.14 last 344.31) - open/close 32803 (time 1755096530.25 total 92.14 last 280.23) - open/close 36298 (time 1755096540.25 total 102.15 last 349.42) - open/close 39426 (time 1755096550.28 total 112.18 last 311.85) - open/close 40000 (time 1755096552.51 total 114.40 last 257.85) - open/close 43362 (time 1755096562.51 total 124.40 last 336.19) - open/close 46861 (time 1755096572.51 total 134.41 last 349.80) - open/close 49927 (time 1755096582.51 total 144.41 last 306.50) - open/close 53481 (time 1755096592.52 total 154.41 last 355.33) - open/close 56718 (time 1755096602.52 total 164.41 last 323.67) - open/close 60000 (time 1755096612.12 total 174.02 last 341.76) - open/close 63889 (time 1755096622.12 total 184.02 last 388.85) total: 64768 open/close in 186.48 seconds: 347.31 ops/second - /unlink 5905 (time 1755096636.11 total 10.00 last 590.49) - /unlink 10000 (time 1755096642.51 total 16.40 last 640.19) - /unlink 16405 (time 1755096652.51 total 26.40 last 640.47) - /unlink 20000 (time 1755096657.39 total 31.27 last 737.06) - /unlink 26706 (time 1755096667.39 total 41.27 last 670.59) - /unlink 30000 (time 1755096672.37 total 46.26 last 661.25) - /unlink 36628 (time 1755096682.37 total 56.26 last 662.75) - /unlink 40000 (time 1755096686.93 total 60.82 last 739.52) - /unlink 47255 (time 1755096696.93 total 70.82 last 725.41) - /unlink 50000 (time 1755096701.42 total 75.30 last 611.88) - /unlink 56916 (time 1755096711.42 total 85.30 last 691.59) - /unlink 60000 (time 1755096715.81 total 89.70 last 701.38) total: 64768 /unlink in 96.67 seconds: 670.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4717 (time 1755096773.14 total 10.00 last 471.65) - open/close 9560 (time 1755096783.14 total 20.00 last 484.27) - open/close 14372 (time 1755096793.14 total 30.00 last 481.13) - open/close 19291 (time 1755096803.14 total 40.00 last 491.84) - open/close 24166 (time 1755096813.14 total 50.01 last 487.46) - open/close 28780 (time 1755096823.14 total 60.01 last 461.34) - open/close 30000 (time 1755096825.68 total 62.54 last 480.76) - open/close 34784 (time 1755096835.68 total 72.55 last 478.34) - open/close 39544 (time 1755096845.68 total 82.55 last 475.97) - open/close 44194 (time 1755096855.69 total 92.55 last 464.94) - open/close 48697 (time 1755096865.69 total 102.55 last 450.26) - open/close 50000 (time 1755096868.26 total 105.12 last 506.37) - open/close 54982 (time 1755096878.26 total 115.12 last 498.16) - open/close 59549 (time 1755096888.26 total 125.12 last 456.66) - open/close 64567 (time 1755096898.26 total 135.12 last 501.76) total: 64768 open/close in 135.52 seconds: 477.92 ops/second - /unlink 8327 (time 1755096910.01 total 10.00 last 832.69) - /unlink 10000 (time 1755096912.22 total 12.21 last 757.36) - /unlink 18167 (time 1755096922.22 total 22.21 last 816.60) - /unlink 20000 (time 1755096924.63 total 24.62 last 760.47) - /unlink 27953 (time 1755096934.63 total 34.62 last 795.23) - /unlink 30000 (time 1755096936.88 total 36.87 last 909.63) - /unlink 37002 (time 1755096946.88 total 46.87 last 700.15) - /unlink 40000 (time 1755096951.69 total 51.68 last 623.53) - /unlink 46203 (time 1755096961.69 total 61.68 last 620.26) - /unlink 50000 (time 1755096966.89 total 66.87 last 731.07) - /unlink 57825 (time 1755096976.89 total 76.88 last 782.44) - /unlink 60000 (time 1755096979.84 total 79.82 last 737.48) total: 64768 /unlink in 85.53 seconds: 757.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4440 (time 1755097015.37 total 10.00 last 443.93) - open/close 9028 (time 1755097025.37 total 20.00 last 458.72) - open/close 13763 (time 1755097035.37 total 30.00 last 473.46) - open/close 18818 (time 1755097045.37 total 40.00 last 505.48) - open/close 20000 (time 1755097047.69 total 42.33 last 508.22) - open/close 25096 (time 1755097057.70 total 52.33 last 509.52) - open/close 30000 (time 1755097067.15 total 61.79 last 518.72) - open/close 34436 (time 1755097077.15 total 71.79 last 443.55) - open/close 38845 (time 1755097087.15 total 81.79 last 440.85) - open/close 40000 (time 1755097089.56 total 84.19 last 479.97) - open/close 44965 (time 1755097099.56 total 94.20 last 496.47) - open/close 50000 (time 1755097109.39 total 104.02 last 512.28) - open/close 54503 (time 1755097119.39 total 114.02 last 450.28) - open/close 58615 (time 1755097129.39 total 124.03 last 411.13) - open/close 60000 (time 1755097132.47 total 127.10 last 450.27) total: 64768 open/close in 136.63 seconds: 474.05 ops/second - /unlink 8150 (time 1755097153.22 total 10.00 last 814.91) - /unlink 10000 (time 1755097155.45 total 12.23 last 828.59) - /unlink 18013 (time 1755097165.45 total 22.23 last 801.30) - /unlink 20000 (time 1755097167.97 total 24.75 last 789.69) - /unlink 28161 (time 1755097177.97 total 34.75 last 816.04) - /unlink 36322 (time 1755097187.97 total 44.75 last 816.02) - /unlink 40000 (time 1755097193.50 total 50.28 last 665.54) - /unlink 48036 (time 1755097203.50 total 60.28 last 803.57) - /unlink 50000 (time 1755097205.66 total 62.44 last 906.55) - /unlink 58204 (time 1755097215.66 total 72.45 last 820.37) - /unlink 60000 (time 1755097217.73 total 74.51 last 868.27) total: 64768 /unlink in 80.43 seconds: 805.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5077 (time 1755097294.29 total 10.00 last 507.69) - open/close 10000 (time 1755097303.90 total 19.61 last 512.11) - open/close 15231 (time 1755097313.90 total 29.61 last 523.03) - open/close 19907 (time 1755097323.91 total 39.62 last 467.51) - open/close 24930 (time 1755097333.91 total 49.62 last 502.26) - open/close 29423 (time 1755097343.91 total 59.62 last 449.29) - open/close 33198 (time 1755097353.91 total 69.62 last 377.49) - open/close 37353 (time 1755097363.91 total 79.62 last 415.43) - open/close 40000 (time 1755097369.54 total 85.25 last 469.88) - open/close 44484 (time 1755097379.54 total 95.25 last 448.38) - open/close 48609 (time 1755097389.54 total 105.26 last 412.41) - open/close 50000 (time 1755097393.55 total 109.27 last 346.89) - open/close 53460 (time 1755097403.56 total 119.27 last 345.98) - open/close 57316 (time 1755097413.56 total 129.27 last 385.59) - open/close 60000 (time 1755097419.89 total 135.60 last 423.61) - open/close 63907 (time 1755097429.89 total 145.60 last 390.69) total: 64768 open/close in 147.66 seconds: 438.62 ops/second - /unlink 7017 (time 1755097443.26 total 10.00 last 701.44) - /unlink 10000 (time 1755097447.65 total 14.40 last 679.22) - /unlink 17185 (time 1755097457.69 total 24.44 last 715.48) - /unlink 20000 (time 1755097461.59 total 28.33 last 722.86) - /unlink 27576 (time 1755097471.59 total 38.33 last 757.57) - /unlink 30000 (time 1755097475.09 total 41.84 last 691.67) - /unlink 37515 (time 1755097485.09 total 51.84 last 751.46) - /unlink 40000 (time 1755097488.14 total 54.88 last 815.47) - /unlink 47147 (time 1755097498.14 total 64.89 last 714.66) - /unlink 50000 (time 1755097503.03 total 69.77 last 583.53) - /unlink 56567 (time 1755097513.03 total 79.78 last 656.63) - /unlink 60000 (time 1755097518.84 total 85.59 last 590.60) total: 64768 /unlink in 92.77 seconds: 698.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local oleg351-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg351-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2