== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:43:16 (1734115396) Stopping clients: oleg227-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg227-client.virtnet /mnt/lustre2 (opts:-f) oleg227-server: oleg227-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing load_modules_local oleg227-server: Loading modules from /home/green/git/lustre-release/lustre oleg227-server: detected 4 online CPUs by sysfs oleg227-server: MODOPTS_LIBCFS= oleg227-server: Force libcfs to create 2 CPU partitions oleg227-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg227-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg227-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg227-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg227-server: oleg227-server.virtnet: executing set_default_debug -1 all pdsh@oleg227-client: oleg227-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg227-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg227-client.virtnet: -o user_xattr,flock 192.168.202.127@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2335 (time 1734115437.40 total 10.00 last 233.49) - open/close 4791 (time 1734115447.41 total 20.00 last 245.54) - open/close 7499 (time 1734115457.41 total 30.01 last 270.71) - open/close 10000 (time 1734115465.73 total 38.32 last 300.68) - open/close 12738 (time 1734115475.73 total 48.33 last 273.72) - open/close 15184 (time 1734115485.73 total 58.33 last 244.56) - open/close 17727 (time 1734115495.74 total 68.33 last 254.22) - open/close 20000 (time 1734115505.48 total 78.07 last 233.32) - open/close 22327 (time 1734115515.48 total 88.08 last 232.64) - open/close 24485 (time 1734115525.48 total 98.08 last 215.76) - open/close 26585 (time 1734115535.48 total 108.08 last 209.99) - open/close 28664 (time 1734115545.48 total 118.08 last 207.88) - open/close 30000 (time 1734115550.86 total 123.45 last 248.53) - open/close 32603 (time 1734115560.86 total 133.46 last 260.15) - open/close 34755 (time 1734115570.87 total 143.46 last 215.14) - open/close 37171 (time 1734115580.87 total 153.46 last 241.58) - open/close 39559 (time 1734115590.87 total 163.47 last 238.72) - open/close 41752 (time 1734115600.88 total 173.47 last 219.21) - open/close 43643 (time 1734115610.88 total 183.48 last 188.99) - open/close 45503 (time 1734115620.88 total 193.48 last 185.97) - open/close 47444 (time 1734115630.89 total 203.48 last 194.00) - open/close 49782 (time 1734115640.89 total 213.49 last 233.70) - open/close 52056 (time 1734115650.89 total 223.49 last 227.39) - open/close 54717 (time 1734115660.89 total 233.49 last 266.06) - open/close 57887 (time 1734115670.89 total 243.49 last 316.97) - open/close 60000 (time 1734115678.57 total 251.17 last 275.25) - open/close 62806 (time 1734115688.57 total 261.17 last 280.57) total: 64768 open/close in 267.58 seconds: 242.05 ops/second - /unlink 7578 (time 1734115705.90 total 10.00 last 757.71) - /unlink 10000 (time 1734115708.61 total 12.71 last 893.70) - /unlink 19128 (time 1734115718.61 total 22.71 last 912.71) - /unlink 27857 (time 1734115728.61 total 32.71 last 872.86) - /unlink 30000 (time 1734115731.00 total 35.10 last 898.41) - /unlink 39197 (time 1734115741.00 total 45.10 last 919.68) - /unlink 47946 (time 1734115751.00 total 55.10 last 874.84) - /unlink 50000 (time 1734115753.31 total 57.41 last 887.55) - /unlink 58966 (time 1734115763.31 total 67.41 last 896.52) total: 64768 /unlink in 74.10 seconds: 874.05 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4720 (time 1734115805.46 total 10.00 last 471.89) - open/close 9757 (time 1734115815.46 total 20.00 last 503.70) - open/close 14559 (time 1734115825.46 total 30.00 last 480.13) - open/close 18956 (time 1734115835.46 total 40.01 last 439.63) - open/close 20000 (time 1734115837.70 total 42.24 last 466.71) - open/close 25457 (time 1734115847.70 total 52.24 last 545.57) - open/close 30000 (time 1734115857.36 total 61.90 last 470.46) - open/close 34916 (time 1734115867.36 total 71.90 last 491.50) - open/close 39304 (time 1734115877.36 total 81.90 last 438.76) - open/close 43737 (time 1734115887.36 total 91.91 last 443.21) - open/close 48390 (time 1734115897.36 total 101.91 last 465.29) - open/close 50000 (time 1734115901.35 total 105.89 last 403.69) - open/close 53770 (time 1734115911.35 total 115.89 last 376.99) - open/close 57734 (time 1734115921.35 total 125.90 last 396.31) - open/close 60000 (time 1734115926.98 total 131.52 last 402.66) - open/close 63468 (time 1734115936.98 total 141.53 last 346.73) total: 64768 open/close in 145.51 seconds: 445.11 ops/second - /unlink 7436 (time 1734115951.90 total 10.00 last 743.53) - /unlink 10000 (time 1734115955.06 total 13.16 last 811.10) - /unlink 18060 (time 1734115965.06 total 23.16 last 805.93) - /unlink 20000 (time 1734115967.45 total 25.55 last 811.79) - /unlink 28651 (time 1734115977.45 total 35.55 last 865.05) - /unlink 37597 (time 1734115987.45 total 45.55 last 894.59) - /unlink 40000 (time 1734115990.22 total 48.32 last 869.38) - /unlink 49452 (time 1734116000.22 total 58.32 last 945.12) - /unlink 58738 (time 1734116010.22 total 68.32 last 928.53) total: 64768 /unlink in 74.44 seconds: 870.03 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5378 (time 1734116050.92 total 10.00 last 537.73) - open/close 10000 (time 1734116059.46 total 18.54 last 541.37) - open/close 14487 (time 1734116069.46 total 28.54 last 448.65) - open/close 20000 (time 1734116078.47 total 37.55 last 611.64) - open/close 25294 (time 1734116088.47 total 47.55 last 529.35) - open/close 29977 (time 1734116098.47 total 57.56 last 468.23) - open/close 35723 (time 1734116108.48 total 67.56 last 574.58) - open/close 40000 (time 1734116115.93 total 75.01 last 573.86) - open/close 45917 (time 1734116125.93 total 85.01 last 591.66) - open/close 50000 (time 1734116133.50 total 92.58 last 539.06) - open/close 55051 (time 1734116143.51 total 102.59 last 505.01) - open/close 59883 (time 1734116153.51 total 112.59 last 483.16) total: 64768 open/close in 122.45 seconds: 528.94 ops/second - /unlink 8574 (time 1734116174.23 total 10.00 last 857.35) - /unlink 17301 (time 1734116184.23 total 20.00 last 872.66) - /unlink 20000 (time 1734116187.37 total 23.14 last 860.81) - /unlink 28658 (time 1734116197.37 total 33.14 last 865.79) - /unlink 36613 (time 1734116207.37 total 43.14 last 795.41) - /unlink 40000 (time 1734116211.53 total 47.30 last 813.29) - /unlink 48018 (time 1734116221.53 total 57.30 last 801.79) - /unlink 50000 (time 1734116223.97 total 59.74 last 813.53) - /unlink 58621 (time 1734116233.97 total 69.74 last 862.05) total: 64768 /unlink in 76.70 seconds: 844.41 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4904 (time 1734116270.55 total 10.00 last 490.34) - open/close 9767 (time 1734116280.55 total 20.00 last 486.30) - open/close 14804 (time 1734116290.55 total 30.00 last 503.67) - open/close 19871 (time 1734116300.55 total 40.00 last 506.69) - open/close 24966 (time 1734116310.55 total 50.00 last 509.47) - open/close 29942 (time 1734116320.55 total 60.00 last 497.52) - open/close 34923 (time 1734116330.55 total 70.00 last 498.09) - open/close 39991 (time 1734116340.55 total 80.01 last 506.73) - open/close 45233 (time 1734116350.55 total 90.01 last 524.18) - open/close 50000 (time 1734116359.88 total 99.33 last 511.09) - open/close 54834 (time 1734116369.88 total 109.33 last 483.36) - open/close 59935 (time 1734116379.88 total 119.33 last 510.09) total: 64768 open/close in 129.13 seconds: 501.56 ops/second - /unlink 8915 (time 1734116400.51 total 10.00 last 891.42) - /unlink 17963 (time 1734116410.51 total 20.00 last 904.76) - /unlink 20000 (time 1734116412.76 total 22.24 last 908.75) - /unlink 27589 (time 1734116422.76 total 32.24 last 758.87) - /unlink 30000 (time 1734116425.54 total 35.03 last 866.55) - /unlink 38192 (time 1734116435.54 total 45.03 last 819.10) - /unlink 40000 (time 1734116438.57 total 48.06 last 596.75) - /unlink 47712 (time 1734116448.57 total 58.06 last 771.19) - /unlink 50000 (time 1734116451.70 total 61.19 last 730.78) - /unlink 59057 (time 1734116461.70 total 71.19 last 905.65) total: 64768 /unlink in 77.33 seconds: 837.54 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5306 (time 1734116501.37 total 10.00 last 530.56) - open/close 10000 (time 1734116510.36 total 18.99 last 522.25) - open/close 14574 (time 1734116520.36 total 28.99 last 457.35) - open/close 19837 (time 1734116530.36 total 38.99 last 526.22) - open/close 24919 (time 1734116540.36 total 48.99 last 508.14) - open/close 30000 (time 1734116549.98 total 58.61 last 528.49) - open/close 35305 (time 1734116559.98 total 68.61 last 530.50) - open/close 40000 (time 1734116569.11 total 77.74 last 514.11) - open/close 45125 (time 1734116579.11 total 87.74 last 512.44) - open/close 50000 (time 1734116588.61 total 97.24 last 513.06) - open/close 54946 (time 1734116598.61 total 107.25 last 494.45) - open/close 59931 (time 1734116608.62 total 117.25 last 498.41) total: 64768 open/close in 126.66 seconds: 511.34 ops/second - /unlink 9067 (time 1734116628.89 total 10.00 last 906.66) - /unlink 18083 (time 1734116638.89 total 20.00 last 901.57) - /unlink 20000 (time 1734116641.06 total 22.16 last 886.68) - /unlink 28804 (time 1734116651.06 total 32.16 last 880.38) - /unlink 37883 (time 1734116661.06 total 42.16 last 907.89) - /unlink 40000 (time 1734116663.36 total 44.47 last 918.91) - /unlink 49281 (time 1734116673.36 total 54.47 last 928.05) - /unlink 58177 (time 1734116683.36 total 64.47 last 889.57) - /unlink 60000 (time 1734116685.43 total 66.53 last 882.97) total: 64768 /unlink in 71.55 seconds: 905.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg227-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg227-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg227-server stop mds service on oleg227-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg227-server unloading modules via unload_modules_local on: 'oleg227-server' oleg227-server: oleg227-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-client: ssh exited with exit code 2 pdsh@oleg227-client: oleg227-server: ssh exited with exit code 2