== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:03:21 (1715205801) Stopping clients: oleg456-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg456-client.virtnet /mnt/lustre2 (opts:-f) oleg456-server: oleg456-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing load_modules_local oleg456-server: Loading modules from /home/green/git/lustre-release/lustre oleg456-server: detected 4 online CPUs by sysfs oleg456-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg456-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg456-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg456-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg456-server: oleg456-server.virtnet: executing set_default_debug -1 all pdsh@oleg456-client: oleg456-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg456-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg456-client.virtnet: -o user_xattr,flock oleg456-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5757 (time 1715205835.91 total 10.00 last 575.65) - open/close 10000 (time 1715205842.63 total 16.73 last 630.92) - open/close 16048 (time 1715205852.63 total 26.73 last 604.74) - open/close 20000 (time 1715205858.82 total 32.91 last 639.14) - open/close 25417 (time 1715205868.82 total 42.91 last 541.68) - open/close 30000 (time 1715205877.55 total 51.64 last 524.81) - open/close 34793 (time 1715205887.55 total 61.65 last 479.22) - open/close 40000 (time 1715205897.52 total 71.61 last 522.33) - open/close 46164 (time 1715205907.52 total 81.61 last 616.38) - open/close 50000 (time 1715205913.79 total 87.88 last 611.77) - open/close 55129 (time 1715205923.79 total 97.89 last 512.83) - open/close 60000 (time 1715205932.92 total 107.01 last 533.66) total: 64768 open/close in 115.85 seconds: 559.08 ops/second - /unlink 8850 (time 1715205952.57 total 10.00 last 884.96) - /unlink 17575 (time 1715205962.57 total 20.00 last 872.50) - /unlink 20000 (time 1715205965.24 total 22.67 last 909.47) - /unlink 28724 (time 1715205975.24 total 32.67 last 872.38) - /unlink 37180 (time 1715205985.24 total 42.67 last 845.54) - /unlink 40000 (time 1715205988.56 total 45.98 last 850.37) - /unlink 48560 (time 1715205998.56 total 55.98 last 855.99) - /unlink 57470 (time 1715206008.56 total 65.98 last 890.97) - /unlink 60000 (time 1715206011.32 total 68.75 last 914.02) total: 64768 /unlink in 73.92 seconds: 876.20 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5274 (time 1715206049.69 total 10.00 last 527.30) - open/close 8876 (time 1715206059.70 total 20.00 last 360.18) - open/close 10000 (time 1715206061.87 total 22.18 last 516.19) - open/close 15159 (time 1715206071.87 total 32.18 last 515.89) - open/close 20000 (time 1715206081.53 total 41.84 last 501.04) - open/close 24656 (time 1715206091.54 total 51.84 last 465.59) - open/close 29087 (time 1715206101.54 total 61.84 last 443.02) - open/close 33594 (time 1715206111.54 total 71.85 last 450.63) - open/close 37773 (time 1715206121.54 total 81.85 last 417.83) - open/close 40000 (time 1715206125.82 total 86.13 last 520.19) - open/close 44555 (time 1715206135.82 total 96.13 last 455.50) - open/close 49621 (time 1715206145.82 total 106.13 last 506.58) - open/close 54833 (time 1715206155.82 total 116.13 last 521.12) - open/close 60000 (time 1715206165.19 total 125.49 last 551.77) total: 64768 open/close in 134.68 seconds: 480.89 ops/second - /unlink 8837 (time 1715206185.20 total 10.00 last 883.69) - /unlink 16695 (time 1715206195.20 total 20.00 last 785.73) - /unlink 20000 (time 1715206200.79 total 25.59 last 591.48) - /unlink 26955 (time 1715206210.79 total 35.59 last 695.43) - /unlink 30000 (time 1715206214.92 total 39.72 last 737.45) - /unlink 37042 (time 1715206224.92 total 49.72 last 704.14) - /unlink 40000 (time 1715206228.20 total 52.99 last 903.90) - /unlink 48966 (time 1715206238.20 total 62.99 last 896.50) - /unlink 57841 (time 1715206248.20 total 72.99 last 887.43) - /unlink 60000 (time 1715206250.77 total 75.56 last 840.13) total: 64768 /unlink in 80.98 seconds: 799.77 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5108 (time 1715206290.27 total 10.00 last 510.79) - open/close 10000 (time 1715206300.15 total 19.88 last 494.96) - open/close 14951 (time 1715206310.15 total 29.89 last 495.04) - open/close 19746 (time 1715206320.16 total 39.89 last 479.49) - open/close 25036 (time 1715206330.16 total 49.89 last 528.94) - open/close 30000 (time 1715206340.04 total 59.77 last 502.14) - open/close 35245 (time 1715206350.04 total 69.77 last 524.47) - open/close 40000 (time 1715206359.32 total 79.05 last 512.64) - open/close 44900 (time 1715206369.32 total 89.05 last 489.99) - open/close 49877 (time 1715206379.32 total 99.05 last 497.66) - open/close 55090 (time 1715206389.32 total 109.05 last 521.24) - open/close 60000 (time 1715206398.96 total 118.69 last 509.46) - open/close 64715 (time 1715206408.96 total 128.69 last 471.42) total: 64768 open/close in 128.79 seconds: 502.89 ops/second - /unlink 8725 (time 1715206419.91 total 10.00 last 872.38) - /unlink 15576 (time 1715206429.91 total 20.00 last 685.08) - /unlink 20000 (time 1715206436.32 total 26.41 last 689.82) - /unlink 28666 (time 1715206446.32 total 36.42 last 866.51) - /unlink 36049 (time 1715206456.32 total 46.42 last 738.23) - /unlink 40000 (time 1715206461.15 total 51.24 last 819.00) - /unlink 47809 (time 1715206471.15 total 61.24 last 780.84) - /unlink 50000 (time 1715206473.72 total 63.81 last 852.00) - /unlink 57122 (time 1715206483.72 total 73.81 last 712.13) - /unlink 60000 (time 1715206489.00 total 79.09 last 545.11) total: 64768 /unlink in 86.34 seconds: 750.12 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4305 (time 1715206525.98 total 10.00 last 430.47) - open/close 7479 (time 1715206535.98 total 20.00 last 317.31) - open/close 10000 (time 1715206543.83 total 27.85 last 321.14) - open/close 14516 (time 1715206553.83 total 37.85 last 451.56) - open/close 18449 (time 1715206563.83 total 47.86 last 393.26) - open/close 20000 (time 1715206567.85 total 51.88 last 385.72) - open/close 24543 (time 1715206577.85 total 61.88 last 454.22) - open/close 29008 (time 1715206587.86 total 71.88 last 446.41) - open/close 34236 (time 1715206597.86 total 81.88 last 522.68) - open/close 39482 (time 1715206607.86 total 91.88 last 524.49) - open/close 43958 (time 1715206617.86 total 101.89 last 447.54) - open/close 49229 (time 1715206627.86 total 111.89 last 526.94) - open/close 53439 (time 1715206637.87 total 121.89 last 420.75) - open/close 56806 (time 1715206647.93 total 131.95 last 334.71) - open/close 60000 (time 1715206654.36 total 138.39 last 496.30) total: 64768 open/close in 147.76 seconds: 438.35 ops/second - /unlink 7081 (time 1715206674.65 total 10.00 last 708.08) - /unlink 10000 (time 1715206679.32 total 14.67 last 625.35) - /unlink 17641 (time 1715206689.32 total 24.67 last 764.05) - /unlink 20000 (time 1715206692.19 total 27.54 last 822.19) - /unlink 26890 (time 1715206702.19 total 37.54 last 688.94) - /unlink 30000 (time 1715206706.75 total 42.10 last 682.56) - /unlink 37620 (time 1715206716.75 total 52.10 last 761.94) - /unlink 40000 (time 1715206719.51 total 54.86 last 861.73) - /unlink 46744 (time 1715206729.51 total 64.86 last 674.36) - /unlink 50000 (time 1715206734.81 total 70.16 last 614.36) - /unlink 57247 (time 1715206744.81 total 80.16 last 724.66) - /unlink 60000 (time 1715206749.10 total 84.44 last 642.40) total: 64768 /unlink in 92.81 seconds: 697.84 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3993 (time 1715206786.41 total 10.00 last 399.20) - open/close 9262 (time 1715206796.41 total 20.00 last 526.87) - open/close 10000 (time 1715206799.08 total 22.67 last 276.58) - open/close 12996 (time 1715206809.08 total 32.67 last 299.58) - open/close 16324 (time 1715206819.09 total 42.68 last 332.69) - open/close 19525 (time 1715206829.09 total 52.68 last 320.07) - open/close 22903 (time 1715206839.09 total 62.68 last 337.79) - open/close 26158 (time 1715206849.09 total 72.68 last 325.45) - open/close 28912 (time 1715206859.09 total 82.68 last 275.34) - open/close 30000 (time 1715206862.96 total 86.55 last 281.29) - open/close 33048 (time 1715206872.96 total 96.55 last 304.79) - open/close 36575 (time 1715206882.96 total 106.55 last 352.68) - open/close 39974 (time 1715206892.96 total 116.55 last 339.82) - open/close 42901 (time 1715206902.96 total 126.55 last 292.62) - open/close 45630 (time 1715206912.97 total 136.56 last 272.84) - open/close 48362 (time 1715206922.97 total 146.56 last 273.16) - open/close 50000 (time 1715206929.16 total 152.75 last 264.45) - open/close 52541 (time 1715206939.16 total 162.75 last 254.07) - open/close 54953 (time 1715206949.16 total 172.75 last 241.17) - open/close 57492 (time 1715206959.17 total 182.76 last 253.85) - open/close 60000 (time 1715206969.08 total 192.67 last 252.87) - open/close 62781 (time 1715206979.09 total 202.68 last 278.01) total: 64768 open/close in 210.08 seconds: 308.31 ops/second - /unlink 5832 (time 1715206997.64 total 10.00 last 583.14) - /unlink 10000 (time 1715207002.18 total 14.54 last 919.11) - /unlink 15463 (time 1715207012.18 total 24.54 last 546.25) - /unlink 20000 (time 1715207020.45 total 32.81 last 548.09) - /unlink 25443 (time 1715207030.46 total 42.82 last 544.24) - /unlink 30000 (time 1715207039.09 total 51.45 last 527.49) - /unlink 35714 (time 1715207049.10 total 61.46 last 571.29) - /unlink 40000 (time 1715207057.04 total 69.40 last 539.88) - /unlink 45172 (time 1715207067.04 total 79.40 last 517.17) - /unlink 50000 (time 1715207076.82 total 89.18 last 493.40) - /unlink 55211 (time 1715207086.82 total 99.18 last 521.01) - /unlink 60000 (time 1715207095.62 total 107.98 last 544.27) total: 64768 /unlink in 116.48 seconds: 556.05 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg456-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg456-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg456-server stop mds service on oleg456-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg456-server unloading modules on: 'oleg456-server' oleg456-server: oleg456-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-client: ssh exited with exit code 2 pdsh@oleg456-client: oleg456-server: ssh exited with exit code 2