== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:36:14 (1767839774) Stopping clients: oleg304-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg304-client.virtnet /mnt/lustre2 (opts:-f) oleg304-server: oleg304-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing load_modules_local oleg304-server: Loading modules from /home/green/git/lustre-release/lustre oleg304-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg304-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg304-server: detected 4 online CPUs by sysfs oleg304-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg304-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg304-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg304-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg304-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg304-server: oleg304-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg304-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg304-client.virtnet: -o user_xattr,flock oleg304-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16232 424 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16451 643 15808 4% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1121 (time 1767839874.57 total 10.00 last 112.07) - open/close 2181 (time 1767839884.57 total 20.01 last 105.97) - open/close 3373 (time 1767839894.58 total 30.02 last 119.08) - open/close 4323 (time 1767839904.58 total 40.02 last 94.96) - open/close 5379 (time 1767839914.59 total 50.02 last 105.57) - open/close 6515 (time 1767839924.59 total 60.03 last 113.53) - open/close 7612 (time 1767839934.59 total 70.03 last 109.69) - open/close 8712 (time 1767839944.59 total 80.03 last 109.99) - open/close 9946 (time 1767839954.60 total 90.03 last 123.38) - open/close 11219 (time 1767839964.60 total 100.04 last 127.21) - open/close 12533 (time 1767839974.61 total 110.05 last 131.30) - open/close 13902 (time 1767839984.61 total 120.05 last 136.84) - open/close 15207 (time 1767839994.62 total 130.06 last 130.41) - open/close 16505 (time 1767840004.63 total 140.06 last 129.74) - open/close 17803 (time 1767840014.63 total 150.07 last 129.74) - open/close 19074 (time 1767840024.63 total 160.07 last 127.05) - open/close 20000 (time 1767840030.37 total 165.81 last 161.46) - open/close 21767 (time 1767840040.37 total 175.81 last 176.67) - open/close 23498 (time 1767840050.38 total 185.81 last 173.02) - open/close 24913 (time 1767840060.38 total 195.82 last 141.45) - open/close 26468 (time 1767840070.38 total 205.82 last 155.49) - open/close 28139 (time 1767840080.39 total 215.82 last 167.00) - open/close 30000 (time 1767840089.99 total 225.42 last 193.85) - open/close 31905 (time 1767840099.99 total 235.43 last 190.45) - open/close 33762 (time 1767840109.99 total 245.43 last 185.70) - open/close 36134 (time 1767840119.99 total 255.43 last 237.11) - open/close 37860 (time 1767840130.00 total 265.44 last 172.50) - open/close 39283 (time 1767840140.00 total 275.44 last 142.25) - open/close 40000 (time 1767840145.74 total 281.17 last 125.06) - open/close 41282 (time 1767840155.74 total 291.18 last 128.13) - open/close 42509 (time 1767840165.74 total 301.18 last 122.66) - open/close 43809 (time 1767840175.75 total 311.18 last 129.96) - open/close 45199 (time 1767840185.75 total 321.19 last 138.96) - open/close 46834 (time 1767840195.75 total 331.19 last 163.44) - open/close 48779 (time 1767840205.76 total 341.20 last 194.40) - open/close 50000 (time 1767840212.53 total 347.97 last 180.33) - open/close 52082 (time 1767840222.53 total 357.97 last 208.18) - open/close 53838 (time 1767840232.54 total 367.97 last 175.52) - open/close 55387 (time 1767840242.54 total 377.97 last 154.86) - open/close 56970 (time 1767840252.54 total 387.98 last 158.25) - open/close 58707 (time 1767840262.54 total 397.98 last 173.65) - open/close 60000 (time 1767840268.97 total 404.40 last 201.30) - open/close 62675 (time 1767840278.97 total 414.41 last 267.43) total: 64768 open/close in 423.74 seconds: 152.85 ops/second - /unlink 4170 (time 1767840299.87 total 10.00 last 416.97) - /unlink 8436 (time 1767840309.87 total 20.00 last 426.57) - /unlink 10000 (time 1767840313.68 total 23.81 last 410.74) - /unlink 13926 (time 1767840323.68 total 33.81 last 392.53) - /unlink 17207 (time 1767840333.68 total 43.81 last 328.10) - /unlink 20000 (time 1767840342.74 total 52.87 last 308.15) - /unlink 23530 (time 1767840352.74 total 62.88 last 352.97) - /unlink 27142 (time 1767840362.74 total 72.88 last 361.13) - /unlink 30000 (time 1767840371.90 total 82.03 last 312.14) - /unlink 33737 (time 1767840381.90 total 92.03 last 373.69) - /unlink 39194 (time 1767840391.90 total 102.03 last 545.70) - /unlink 46382 (time 1767840401.91 total 112.04 last 718.46) - /unlink 50000 (time 1767840407.37 total 117.50 last 662.75) - /unlink 56705 (time 1767840417.37 total 127.50 last 670.48) - /unlink 60000 (time 1767840422.06 total 132.19 last 702.54) total: 64768 /unlink in 139.05 seconds: 465.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4264 (time 1767840464.08 total 10.00 last 426.31) - open/close 8898 (time 1767840474.09 total 20.01 last 462.92) - open/close 10000 (time 1767840476.28 total 22.21 last 501.53) - open/close 15152 (time 1767840486.28 total 32.21 last 515.18) - open/close 20000 (time 1767840495.56 total 41.49 last 522.32) - open/close 25353 (time 1767840505.57 total 51.49 last 535.25) - open/close 30000 (time 1767840514.40 total 60.33 last 525.99) - open/close 34520 (time 1767840524.40 total 70.33 last 451.93) - open/close 39756 (time 1767840534.40 total 80.33 last 523.58) - open/close 44365 (time 1767840544.40 total 90.33 last 460.84) - open/close 49040 (time 1767840554.40 total 100.33 last 467.45) - open/close 50000 (time 1767840556.48 total 102.41 last 462.10) - open/close 54462 (time 1767840566.48 total 112.41 last 446.14) - open/close 59133 (time 1767840576.48 total 122.41 last 467.05) - open/close 63895 (time 1767840586.49 total 132.41 last 476.13) total: 64768 open/close in 134.20 seconds: 482.63 ops/second - /unlink 7742 (time 1767840599.53 total 10.00 last 774.16) - /unlink 10000 (time 1767840602.94 total 13.41 last 661.97) - /unlink 16790 (time 1767840612.95 total 23.41 last 678.89) - /unlink 20000 (time 1767840617.28 total 27.74 last 741.42) - /unlink 26812 (time 1767840627.28 total 37.74 last 681.20) - /unlink 30000 (time 1767840631.79 total 42.26 last 706.48) - /unlink 36804 (time 1767840641.79 total 52.26 last 680.37) - /unlink 40000 (time 1767840646.34 total 56.81 last 702.47) - /unlink 46140 (time 1767840656.34 total 66.81 last 613.98) - /unlink 50000 (time 1767840661.85 total 72.32 last 700.13) - /unlink 57543 (time 1767840671.85 total 82.32 last 754.27) - /unlink 60000 (time 1767840675.59 total 86.06 last 656.82) total: 64768 /unlink in 92.58 seconds: 699.58 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4745 (time 1767840712.32 total 10.00 last 474.49) - open/close 9722 (time 1767840722.32 total 20.00 last 497.65) - open/close 14349 (time 1767840732.32 total 30.00 last 462.63) - open/close 19557 (time 1767840742.33 total 40.00 last 520.72) - open/close 24416 (time 1767840752.33 total 50.01 last 485.84) - open/close 29478 (time 1767840762.33 total 60.01 last 506.19) - open/close 34801 (time 1767840772.33 total 70.01 last 532.30) - open/close 39795 (time 1767840782.33 total 80.01 last 499.35) - open/close 44416 (time 1767840792.33 total 90.01 last 462.08) - open/close 49190 (time 1767840802.33 total 100.01 last 477.31) - open/close 54013 (time 1767840812.33 total 110.01 last 482.26) - open/close 58992 (time 1767840822.33 total 120.01 last 497.88) - open/close 60000 (time 1767840824.57 total 122.24 last 451.50) - open/close 64025 (time 1767840834.57 total 132.25 last 402.38) total: 64768 open/close in 134.21 seconds: 482.59 ops/second - /unlink 6469 (time 1767840848.14 total 10.00 last 646.88) - /unlink 10000 (time 1767840852.96 total 14.82 last 732.61) - /unlink 18311 (time 1767840862.96 total 24.82 last 831.02) - /unlink 26905 (time 1767840872.98 total 34.84 last 857.75) - /unlink 30000 (time 1767840876.46 total 38.33 last 888.00) - /unlink 36980 (time 1767840886.46 total 48.33 last 697.97) - /unlink 40000 (time 1767840890.93 total 52.79 last 676.64) - /unlink 47186 (time 1767840900.93 total 62.79 last 718.53) - /unlink 50000 (time 1767840905.02 total 66.88 last 687.68) - /unlink 56863 (time 1767840915.02 total 76.88 last 686.23) - /unlink 60000 (time 1767840919.41 total 81.27 last 715.21) total: 64768 /unlink in 87.68 seconds: 738.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5298 (time 1767840961.57 total 10.00 last 529.70) - open/close 8101 (time 1767840971.58 total 20.00 last 280.22) - open/close 9720 (time 1767840981.58 total 30.01 last 161.86) - open/close 11189 (time 1767840991.58 total 40.01 last 146.88) - open/close 12722 (time 1767841001.58 total 50.01 last 153.28) - open/close 14769 (time 1767841011.58 total 60.01 last 204.68) - open/close 17459 (time 1767841021.58 total 70.01 last 268.98) - open/close 20000 (time 1767841031.57 total 80.00 last 254.35) - open/close 22145 (time 1767841041.58 total 90.00 last 214.44) - open/close 24206 (time 1767841051.58 total 100.01 last 205.95) - open/close 26066 (time 1767841061.59 total 110.01 last 185.96) - open/close 28316 (time 1767841071.59 total 120.01 last 224.99) - open/close 30000 (time 1767841078.15 total 126.58 last 256.53) - open/close 33284 (time 1767841088.15 total 136.58 last 328.37) - open/close 35894 (time 1767841098.16 total 146.59 last 260.86) - open/close 39536 (time 1767841108.16 total 156.59 last 364.20) - open/close 40000 (time 1767841110.51 total 158.93 last 197.53) - open/close 42237 (time 1767841120.51 total 168.94 last 223.66) - open/close 44330 (time 1767841130.52 total 178.94 last 209.12) - open/close 46277 (time 1767841140.52 total 188.95 last 194.62) - open/close 48816 (time 1767841150.52 total 198.95 last 253.84) - open/close 50000 (time 1767841153.39 total 201.82 last 412.81) - open/close 53294 (time 1767841163.39 total 211.82 last 329.37) - open/close 57271 (time 1767841173.39 total 221.82 last 397.62) - open/close 60000 (time 1767841180.37 total 228.79 last 391.51) - open/close 64246 (time 1767841190.37 total 238.80 last 424.44) total: 64768 open/close in 240.16 seconds: 269.69 ops/second - /unlink 7909 (time 1767841203.07 total 10.00 last 790.55) - /unlink 10000 (time 1767841205.56 total 12.49 last 840.67) - /unlink 15225 (time 1767841215.56 total 22.49 last 522.46) - /unlink 19698 (time 1767841225.56 total 32.50 last 447.18) - /unlink 26502 (time 1767841235.56 total 42.50 last 680.39) - /unlink 30000 (time 1767841240.06 total 46.99 last 777.66) - /unlink 37443 (time 1767841250.06 total 56.99 last 744.21) - /unlink 40000 (time 1767841254.04 total 60.97 last 643.02) - /unlink 47519 (time 1767841264.04 total 70.97 last 751.84) - /unlink 50000 (time 1767841266.52 total 73.46 last 998.26) - /unlink 58398 (time 1767841276.52 total 83.46 last 839.76) total: 64768 /unlink in 90.21 seconds: 718.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4683 (time 1767841316.50 total 10.00 last 468.24) - open/close 9528 (time 1767841326.50 total 20.00 last 484.43) - open/close 14201 (time 1767841336.51 total 30.00 last 467.21) - open/close 18978 (time 1767841346.51 total 40.01 last 477.62) - open/close 20000 (time 1767841348.58 total 42.08 last 493.61) - open/close 24667 (time 1767841358.58 total 52.08 last 466.67) - open/close 29431 (time 1767841368.58 total 62.08 last 476.31) - open/close 34137 (time 1767841378.58 total 72.08 last 470.51) - open/close 38745 (time 1767841388.58 total 82.08 last 460.80) - open/close 40000 (time 1767841391.39 total 84.89 last 446.69) - open/close 44463 (time 1767841401.39 total 94.89 last 446.20) - open/close 49033 (time 1767841411.40 total 104.89 last 456.91) - open/close 50000 (time 1767841413.55 total 107.05 last 449.23) - open/close 54396 (time 1767841423.55 total 117.05 last 439.58) - open/close 58806 (time 1767841433.55 total 127.05 last 440.94) - open/close 60000 (time 1767841436.34 total 129.84 last 428.34) - open/close 64424 (time 1767841446.34 total 139.84 last 442.36) total: 64768 open/close in 140.63 seconds: 460.55 ops/second - /unlink 8579 (time 1767841458.35 total 10.00 last 857.82) - /unlink 17828 (time 1767841468.35 total 20.00 last 924.82) - /unlink 20000 (time 1767841470.61 total 22.26 last 961.22) - /unlink 27651 (time 1767841480.62 total 32.26 last 765.04) - /unlink 30000 (time 1767841484.37 total 36.02 last 625.40) - /unlink 37900 (time 1767841494.37 total 46.02 last 789.94) - /unlink 40000 (time 1767841497.02 total 48.67 last 793.01) - /unlink 46122 (time 1767841507.04 total 58.68 last 611.27) - /unlink 48448 (time 1767841517.06 total 68.71 last 232.00) - /unlink 50000 (time 1767841523.64 total 75.29 last 235.93) - /unlink 52505 (time 1767841533.64 total 85.29 last 250.44) - /unlink 54800 (time 1767841543.65 total 95.30 last 229.34) - /unlink 56670 (time 1767841553.65 total 105.30 last 186.90) - /unlink 57903 (time 1767841563.66 total 115.31 last 123.23) - /unlink 59623 (time 1767841573.66 total 125.31 last 171.94) - /unlink 61326 (time 1767841583.66 total 135.31 last 170.28) - /unlink 63952 (time 1767841593.67 total 145.31 last 262.57) total: 64768 /unlink in 148.39 seconds: 436.48 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg304-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg304-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg304-server stop mds service on oleg304-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg304-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg304-server' oleg304-server: oleg304-server.virtnet: executing unload_modules_local oleg304-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg304-server: LNET ready to unload oleg304-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-client: ssh exited with exit code 2 pdsh@oleg304-client: oleg304-server: ssh exited with exit code 2