== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:56:10 (1721879770) Stopping clients: oleg253-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg253-client.virtnet /mnt/lustre2 (opts:-f) oleg253-server: oleg253-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing load_modules_local oleg253-server: Loading modules from /home/green/git/lustre-release/lustre oleg253-server: detected 4 online CPUs by sysfs oleg253-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg253-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg253-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg253-client.virtnet: -o user_xattr,flock oleg253-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5322 (time 1721879803.93 total 10.00 last 532.11) - open/close 10000 (time 1721879812.71 total 18.78 last 532.88) - open/close 15667 (time 1721879822.71 total 28.78 last 566.69) - open/close 20000 (time 1721879831.26 total 37.33 last 506.91) - open/close 25020 (time 1721879841.26 total 47.33 last 501.96) - open/close 30000 (time 1721879851.23 total 57.30 last 499.48) - open/close 35238 (time 1721879861.23 total 67.30 last 523.78) - open/close 40000 (time 1721879870.61 total 76.68 last 507.57) - open/close 44619 (time 1721879880.61 total 86.68 last 461.82) - open/close 49252 (time 1721879890.61 total 96.68 last 463.24) - open/close 53852 (time 1721879900.61 total 106.69 last 459.95) - open/close 58384 (time 1721879910.62 total 116.69 last 453.18) - open/close 60000 (time 1721879913.91 total 119.98 last 490.17) total: 64768 open/close in 129.77 seconds: 499.10 ops/second - /unlink 8141 (time 1721879934.59 total 10.00 last 814.04) - /unlink 10000 (time 1721879936.84 total 12.25 last 825.74) - /unlink 18022 (time 1721879946.84 total 22.25 last 802.19) - /unlink 20000 (time 1721879949.28 total 24.69 last 810.15) - /unlink 27758 (time 1721879959.28 total 34.69 last 775.79) - /unlink 30000 (time 1721879962.15 total 37.56 last 781.21) - /unlink 37713 (time 1721879972.15 total 47.56 last 771.24) - /unlink 40000 (time 1721879974.86 total 50.27 last 844.01) - /unlink 47704 (time 1721879984.86 total 60.28 last 770.28) - /unlink 50000 (time 1721879988.02 total 63.43 last 726.95) - /unlink 57963 (time 1721879998.02 total 73.44 last 796.22) - /unlink 60000 (time 1721880000.43 total 75.84 last 846.41) total: 64768 /unlink in 81.75 seconds: 792.29 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4037 (time 1721880038.11 total 10.00 last 403.59) - open/close 8490 (time 1721880048.12 total 20.00 last 445.23) - open/close 10000 (time 1721880051.60 total 23.49 last 433.62) - open/close 13618 (time 1721880061.60 total 33.49 last 361.77) - open/close 18555 (time 1721880071.60 total 43.49 last 493.63) - open/close 20000 (time 1721880074.55 total 46.44 last 489.40) - open/close 24950 (time 1721880084.55 total 56.44 last 494.99) - open/close 29688 (time 1721880094.56 total 66.44 last 473.71) - open/close 34488 (time 1721880104.56 total 76.44 last 479.98) - open/close 39337 (time 1721880114.56 total 86.45 last 484.83) - open/close 44199 (time 1721880124.56 total 96.45 last 486.10) - open/close 49018 (time 1721880134.56 total 106.45 last 481.84) - open/close 50000 (time 1721880136.60 total 108.49 last 480.98) - open/close 54608 (time 1721880146.60 total 118.49 last 460.71) - open/close 59551 (time 1721880156.61 total 128.49 last 494.24) - open/close 64434 (time 1721880166.61 total 138.49 last 488.23) total: 64768 open/close in 139.23 seconds: 465.18 ops/second - /unlink 7802 (time 1721880178.23 total 10.00 last 780.15) - /unlink 10000 (time 1721880180.88 total 12.65 last 829.53) - /unlink 17706 (time 1721880190.88 total 22.65 last 770.56) - /unlink 20000 (time 1721880193.62 total 25.39 last 837.72) - /unlink 28185 (time 1721880203.62 total 35.39 last 818.46) - /unlink 30000 (time 1721880205.84 total 37.61 last 816.70) - /unlink 38047 (time 1721880215.85 total 47.61 last 804.62) - /unlink 40000 (time 1721880218.26 total 50.03 last 808.18) - /unlink 48181 (time 1721880228.26 total 60.03 last 818.04) - /unlink 50000 (time 1721880230.38 total 62.15 last 858.58) - /unlink 58052 (time 1721880240.38 total 72.15 last 805.15) - /unlink 60000 (time 1721880243.08 total 74.85 last 721.06) total: 64768 /unlink in 80.63 seconds: 803.26 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4742 (time 1721880278.93 total 10.00 last 474.15) - open/close 10000 (time 1721880288.38 total 19.46 last 556.13) - open/close 15643 (time 1721880298.38 total 29.46 last 564.23) - open/close 20000 (time 1721880305.81 total 36.88 last 587.09) - open/close 25104 (time 1721880315.81 total 46.88 last 510.33) - open/close 29773 (time 1721880325.81 total 56.88 last 466.87) - open/close 34555 (time 1721880335.81 total 66.88 last 478.18) - open/close 39176 (time 1721880345.81 total 76.88 last 462.06) - open/close 44043 (time 1721880355.81 total 86.88 last 486.69) - open/close 48961 (time 1721880365.81 total 96.88 last 491.73) - open/close 50000 (time 1721880367.97 total 99.04 last 481.36) - open/close 54664 (time 1721880377.97 total 109.04 last 466.38) - open/close 59404 (time 1721880387.97 total 119.04 last 473.95) - open/close 64271 (time 1721880397.97 total 129.04 last 486.68) total: 64768 open/close in 130.10 seconds: 497.84 ops/second - /unlink 8192 (time 1721880409.88 total 10.00 last 819.13) - /unlink 10000 (time 1721880411.99 total 12.11 last 856.97) - /unlink 18347 (time 1721880421.99 total 22.11 last 834.67) - /unlink 20000 (time 1721880424.01 total 24.13 last 818.16) - /unlink 28320 (time 1721880434.01 total 34.13 last 832.00) - /unlink 30000 (time 1721880436.05 total 36.17 last 823.46) - /unlink 38434 (time 1721880446.06 total 46.17 last 843.34) - /unlink 46922 (time 1721880456.06 total 56.17 last 848.71) - /unlink 50000 (time 1721880459.80 total 59.92 last 821.35) - /unlink 58475 (time 1721880469.80 total 69.92 last 847.49) total: 64768 /unlink in 77.35 seconds: 837.38 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4871 (time 1721880509.15 total 10.00 last 487.04) - open/close 9897 (time 1721880519.15 total 20.00 last 502.60) - open/close 14620 (time 1721880529.15 total 30.00 last 472.22) - open/close 19148 (time 1721880539.15 total 40.00 last 452.78) - open/close 24048 (time 1721880549.15 total 50.00 last 489.92) - open/close 28969 (time 1721880559.15 total 60.01 last 492.04) - open/close 30000 (time 1721880561.25 total 62.10 last 492.46) - open/close 34808 (time 1721880571.25 total 72.10 last 480.79) - open/close 39849 (time 1721880581.25 total 82.10 last 504.04) - open/close 44773 (time 1721880591.25 total 92.10 last 492.40) - open/close 49689 (time 1721880601.25 total 102.10 last 491.56) - open/close 54566 (time 1721880611.25 total 112.10 last 487.67) - open/close 59456 (time 1721880621.25 total 122.10 last 488.93) - open/close 64454 (time 1721880631.25 total 132.11 last 499.72) total: 64768 open/close in 132.79 seconds: 487.76 ops/second - /unlink 8374 (time 1721880642.80 total 10.00 last 837.31) - /unlink 16833 (time 1721880652.80 total 20.00 last 845.85) - /unlink 20000 (time 1721880656.43 total 23.62 last 874.27) - /unlink 28369 (time 1721880666.43 total 33.62 last 836.87) - /unlink 36727 (time 1721880676.43 total 43.63 last 835.75) - /unlink 40000 (time 1721880680.32 total 47.52 last 840.79) - /unlink 48536 (time 1721880690.32 total 57.52 last 853.55) - /unlink 57098 (time 1721880700.32 total 67.52 last 856.15) - /unlink 60000 (time 1721880703.65 total 70.85 last 870.58) total: 64768 /unlink in 76.29 seconds: 849.00 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5060 (time 1721880744.13 total 10.00 last 505.93) - open/close 10000 (time 1721880753.70 total 19.57 last 516.34) - open/close 15097 (time 1721880763.70 total 29.57 last 509.61) - open/close 20000 (time 1721880773.57 total 39.44 last 496.99) - open/close 24955 (time 1721880783.57 total 49.44 last 495.42) - open/close 29936 (time 1721880793.57 total 59.44 last 498.08) - open/close 34918 (time 1721880803.57 total 69.44 last 498.16) - open/close 39848 (time 1721880813.57 total 79.44 last 492.92) - open/close 44807 (time 1721880823.57 total 89.44 last 495.86) - open/close 49584 (time 1721880833.57 total 99.44 last 477.62) - open/close 54577 (time 1721880843.57 total 109.44 last 499.24) - open/close 59565 (time 1721880853.57 total 119.44 last 498.78) - open/close 64436 (time 1721880863.59 total 129.46 last 486.23) total: 64768 open/close in 130.09 seconds: 497.86 ops/second - /unlink 8153 (time 1721880875.11 total 10.00 last 815.25) - /unlink 10000 (time 1721880877.28 total 12.18 last 848.62) - /unlink 18394 (time 1721880887.28 total 22.18 last 839.36) - /unlink 26611 (time 1721880897.28 total 32.18 last 821.67) - /unlink 30000 (time 1721880901.56 total 36.45 last 793.62) - /unlink 36312 (time 1721880911.56 total 46.45 last 631.14) - /unlink 40000 (time 1721880918.67 total 53.56 last 518.35) - /unlink 47449 (time 1721880928.67 total 63.56 last 744.84) - /unlink 50000 (time 1721880931.56 total 66.46 last 881.80) - /unlink 58408 (time 1721880941.56 total 76.46 last 840.78) - /unlink 60000 (time 1721880943.85 total 78.74 last 696.63) total: 64768 /unlink in 84.37 seconds: 767.64 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg253-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg253-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg253-server stop mds service on oleg253-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg253-server unloading modules on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2