== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:28:48 (1731346128) Stopping clients: oleg449-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg449-client.virtnet /mnt/lustre2 (opts:-f) oleg449-server: oleg449-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: detected 4 online CPUs by sysfs oleg449-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg449-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg449-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16134 326 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5662 (time 1731346161.72 total 10.00 last 566.14) - open/close 10000 (time 1731346169.66 total 17.94 last 546.40) - open/close 15526 (time 1731346179.66 total 27.94 last 552.52) - open/close 20000 (time 1731346187.67 total 35.95 last 558.81) - open/close 25483 (time 1731346197.67 total 45.95 last 548.21) - open/close 30000 (time 1731346206.14 total 54.42 last 533.06) - open/close 35356 (time 1731346216.14 total 64.42 last 535.54) - open/close 40000 (time 1731346224.56 total 72.84 last 551.87) - open/close 45359 (time 1731346234.56 total 82.84 last 535.82) - open/close 50000 (time 1731346243.41 total 91.69 last 524.28) - open/close 55226 (time 1731346253.41 total 101.69 last 522.51) - open/close 60000 (time 1731346262.78 total 111.07 last 509.43) total: 64768 open/close in 120.47 seconds: 537.64 ops/second - /unlink 8909 (time 1731346283.03 total 10.00 last 890.81) - /unlink 17723 (time 1731346293.03 total 20.00 last 881.34) - /unlink 20000 (time 1731346295.51 total 22.48 last 917.14) - /unlink 28968 (time 1731346305.52 total 32.49 last 896.71) - /unlink 37639 (time 1731346315.52 total 42.49 last 867.03) - /unlink 40000 (time 1731346318.26 total 45.23 last 859.62) - /unlink 48872 (time 1731346328.26 total 55.23 last 887.17) - /unlink 57914 (time 1731346338.26 total 65.23 last 904.11) - /unlink 60000 (time 1731346340.55 total 67.52 last 912.55) total: 64768 /unlink in 72.59 seconds: 892.19 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5244 (time 1731346379.76 total 10.00 last 524.37) - open/close 10000 (time 1731346388.89 total 19.14 last 520.59) - open/close 15403 (time 1731346398.89 total 29.14 last 540.23) - open/close 20000 (time 1731346407.50 total 37.75 last 533.99) - open/close 25270 (time 1731346417.51 total 47.75 last 526.91) - open/close 30000 (time 1731346426.37 total 56.61 last 533.63) - open/close 35343 (time 1731346436.37 total 66.61 last 534.25) - open/close 40000 (time 1731346445.34 total 75.59 last 518.95) - open/close 45253 (time 1731346455.34 total 85.59 last 525.25) - open/close 50000 (time 1731346465.11 total 95.35 last 486.09) - open/close 53359 (time 1731346475.11 total 105.36 last 335.80) - open/close 57730 (time 1731346485.17 total 115.42 last 434.54) - open/close 60000 (time 1731346489.74 total 119.99 last 496.48) total: 64768 open/close in 127.79 seconds: 506.83 ops/second - /unlink 7801 (time 1731346508.32 total 10.00 last 779.98) - /unlink 10000 (time 1731346511.13 total 12.82 last 781.57) - /unlink 18669 (time 1731346521.13 total 22.82 last 866.82) - /unlink 28634 (time 1731346531.13 total 32.82 last 996.38) - /unlink 37724 (time 1731346541.13 total 42.82 last 908.91) - /unlink 40000 (time 1731346543.99 total 45.68 last 795.93) - /unlink 49894 (time 1731346553.99 total 55.68 last 989.34) - /unlink 59278 (time 1731346563.99 total 65.68 last 938.29) total: 64768 /unlink in 71.70 seconds: 903.28 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 6029 (time 1731346605.08 total 10.00 last 602.83) - open/close 10000 (time 1731346611.68 total 16.60 last 601.42) - open/close 16239 (time 1731346621.68 total 26.61 last 623.82) - open/close 20000 (time 1731346627.98 total 32.90 last 597.12) - open/close 25909 (time 1731346637.98 total 42.90 last 590.84) - open/close 30000 (time 1731346644.45 total 49.37 last 632.66) - open/close 35764 (time 1731346654.45 total 59.37 last 576.40) - open/close 40000 (time 1731346661.94 total 66.87 last 565.19) - open/close 45400 (time 1731346671.94 total 76.87 last 539.89) - open/close 50000 (time 1731346680.62 total 85.54 last 530.19) - open/close 55558 (time 1731346690.62 total 95.54 last 555.77) - open/close 60000 (time 1731346698.40 total 103.32 last 571.37) total: 64768 open/close in 111.27 seconds: 582.06 ops/second - /unlink 8949 (time 1731346717.10 total 10.00 last 894.88) - /unlink 17549 (time 1731346727.11 total 20.00 last 859.91) - /unlink 20000 (time 1731346730.12 total 23.02 last 813.16) - /unlink 27926 (time 1731346740.12 total 33.02 last 792.60) - /unlink 30000 (time 1731346742.19 total 35.08 last 1002.31) - /unlink 39919 (time 1731346752.19 total 45.09 last 991.84) - /unlink 49538 (time 1731346762.19 total 55.09 last 961.83) - /unlink 59518 (time 1731346772.19 total 65.09 last 997.96) total: 64768 /unlink in 70.10 seconds: 923.93 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5893 (time 1731346815.51 total 10.00 last 589.22) - open/close 10000 (time 1731346822.83 total 17.32 last 561.54) - open/close 15932 (time 1731346832.83 total 27.32 last 593.14) - open/close 20000 (time 1731346840.03 total 34.51 last 565.24) - open/close 25659 (time 1731346850.03 total 44.51 last 565.83) - open/close 30000 (time 1731346857.84 total 52.32 last 555.91) - open/close 35750 (time 1731346867.84 total 62.32 last 574.90) - open/close 40000 (time 1731346875.97 total 70.46 last 522.59) - open/close 45597 (time 1731346885.97 total 80.46 last 559.64) - open/close 50000 (time 1731346894.25 total 88.74 last 531.79) - open/close 55635 (time 1731346904.25 total 98.74 last 563.45) - open/close 60000 (time 1731346911.64 total 106.13 last 590.97) total: 64768 open/close in 114.41 seconds: 566.10 ops/second - /unlink 9722 (time 1731346930.72 total 10.00 last 972.12) - /unlink 19207 (time 1731346940.72 total 20.00 last 948.48) - /unlink 28548 (time 1731346950.72 total 30.00 last 934.02) - /unlink 37691 (time 1731346960.72 total 40.00 last 914.24) - /unlink 40000 (time 1731346963.42 total 42.70 last 857.23) - /unlink 48652 (time 1731346973.42 total 52.70 last 865.13) - /unlink 57085 (time 1731346983.42 total 62.70 last 843.22) - /unlink 60000 (time 1731346986.49 total 65.77 last 948.32) total: 64768 /unlink in 70.47 seconds: 919.04 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4955 (time 1731347030.80 total 10.00 last 495.41) - open/close 10000 (time 1731347040.45 total 19.65 last 522.89) - open/close 15017 (time 1731347050.45 total 29.65 last 501.65) - open/close 19922 (time 1731347060.45 total 39.65 last 490.41) - open/close 24845 (time 1731347070.45 total 49.65 last 492.25) - open/close 29723 (time 1731347080.46 total 59.66 last 487.75) - open/close 34398 (time 1731347090.46 total 69.66 last 467.46) - open/close 39097 (time 1731347100.46 total 79.66 last 469.83) - open/close 43751 (time 1731347110.46 total 89.66 last 465.33) - open/close 48928 (time 1731347120.46 total 99.66 last 517.60) - open/close 50000 (time 1731347122.59 total 101.79 last 504.42) - open/close 55217 (time 1731347132.59 total 111.79 last 521.69) - open/close 60000 (time 1731347142.02 total 121.22 last 507.15) total: 64768 open/close in 130.67 seconds: 495.68 ops/second - /unlink 9006 (time 1731347162.30 total 10.00 last 900.54) - /unlink 17969 (time 1731347172.30 total 20.00 last 896.25) - /unlink 20000 (time 1731347174.59 total 22.30 last 884.21) - /unlink 29133 (time 1731347184.60 total 32.30 last 913.02) - /unlink 38198 (time 1731347194.60 total 42.30 last 906.33) - /unlink 40000 (time 1731347196.67 total 44.38 last 868.93) - /unlink 48742 (time 1731347206.67 total 54.38 last 874.17) - /unlink 57844 (time 1731347216.67 total 64.38 last 910.20) - /unlink 60000 (time 1731347218.96 total 66.67 last 940.26) total: 64768 /unlink in 71.90 seconds: 900.75 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg449-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server unloading modules via unload_modules_local on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2