== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:47:24 (1718927244) Stopping clients: oleg146-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg146-client.virtnet /mnt/lustre2 (opts:-f) oleg146-server: oleg146-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing load_modules_local oleg146-server: Loading modules from /home/green/git/lustre-release/lustre oleg146-server: detected 4 online CPUs by sysfs oleg146-server: Force libcfs to create 2 CPU partitions oleg146-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg146-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg146-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg146-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg146-server: oleg146-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg146-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg146-client.virtnet: -o user_xattr,flock oleg146-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5436 (time 1718927278.77 total 10.00 last 543.54) - open/close 10000 (time 1718927287.52 total 18.75 last 521.77) - open/close 15090 (time 1718927297.52 total 28.75 last 508.97) - open/close 20000 (time 1718927307.15 total 38.38 last 509.76) - open/close 25333 (time 1718927317.15 total 48.38 last 533.27) - open/close 30000 (time 1718927325.80 total 57.03 last 539.75) - open/close 35404 (time 1718927335.80 total 67.03 last 540.31) - open/close 38780 (time 1718927345.81 total 77.04 last 337.33) - open/close 40000 (time 1718927350.34 total 81.57 last 269.40) - open/close 43532 (time 1718927360.34 total 91.57 last 353.17) - open/close 47812 (time 1718927370.34 total 101.57 last 427.95) - open/close 50000 (time 1718927374.40 total 105.63 last 538.14) - open/close 54435 (time 1718927384.40 total 115.63 last 443.49) - open/close 57561 (time 1718927394.41 total 125.64 last 312.44) - open/close 60000 (time 1718927401.27 total 132.50 last 355.32) - open/close 64651 (time 1718927411.27 total 142.50 last 465.07) total: 64768 open/close in 142.72 seconds: 453.80 ops/second - /unlink 9921 (time 1718927422.33 total 10.00 last 992.01) - /unlink 18200 (time 1718927432.33 total 20.00 last 827.82) - /unlink 20000 (time 1718927434.56 total 22.23 last 807.42) - /unlink 26760 (time 1718927444.56 total 32.23 last 675.94) - /unlink 30000 (time 1718927449.59 total 37.26 last 643.87) - /unlink 38246 (time 1718927459.59 total 47.26 last 824.55) - /unlink 48267 (time 1718927469.59 total 57.27 last 1002.06) - /unlink 58316 (time 1718927479.59 total 67.27 last 1004.81) total: 64768 /unlink in 73.67 seconds: 879.12 ops/second - open/close 5466 (time 1718927496.81 total 10.00 last 546.55) - open/close 10000 (time 1718927504.93 total 18.12 last 558.41) - open/close 15799 (time 1718927514.93 total 28.12 last 579.84) - open/close 20000 (time 1718927521.97 total 35.16 last 596.65) - open/close 25790 (time 1718927531.97 total 45.16 last 578.99) - open/close 30000 (time 1718927539.40 total 52.59 last 566.67) - open/close 35821 (time 1718927549.40 total 62.59 last 582.04) - open/close 40000 (time 1718927556.67 total 69.86 last 574.86) - open/close 45547 (time 1718927566.67 total 79.86 last 554.66) - open/close 50000 (time 1718927574.66 total 87.86 last 557.19) - open/close 55748 (time 1718927584.67 total 97.86 last 574.71) - open/close 60000 (time 1718927591.94 total 105.13 last 584.86) total: 64768 open/close in 113.73 seconds: 569.46 ops/second - /unlink 8550 (time 1718927611.39 total 10.00 last 854.95) - /unlink 17194 (time 1718927621.40 total 20.00 last 864.36) - /unlink 20000 (time 1718927624.54 total 23.15 last 891.47) - /unlink 28739 (time 1718927634.54 total 33.15 last 873.87) - /unlink 37331 (time 1718927644.54 total 43.15 last 859.11) - /unlink 40000 (time 1718927647.73 total 46.34 last 836.48) - /unlink 48611 (time 1718927657.73 total 56.34 last 861.09) - /unlink 57133 (time 1718927667.74 total 66.34 last 852.15) - /unlink 60000 (time 1718927670.99 total 69.60 last 880.57) total: 64768 /unlink in 74.77 seconds: 866.21 ops/second - open/close 5041 (time 1718927687.02 total 10.00 last 504.08) - open/close 10000 (time 1718927696.84 total 19.82 last 505.18) - open/close 15072 (time 1718927706.84 total 29.82 last 507.15) - open/close 20000 (time 1718927716.63 total 39.61 last 503.34) - open/close 24958 (time 1718927726.63 total 49.61 last 495.77) - open/close 29830 (time 1718927736.63 total 59.61 last 487.14) - open/close 34944 (time 1718927746.63 total 69.61 last 511.39) - open/close 39975 (time 1718927756.63 total 79.61 last 503.06) - open/close 44944 (time 1718927766.63 total 89.61 last 496.82) - open/close 49596 (time 1718927776.63 total 99.61 last 465.19) - open/close 54526 (time 1718927786.63 total 109.61 last 493.00) - open/close 59556 (time 1718927796.63 total 119.61 last 502.97) - open/close 64516 (time 1718927806.63 total 129.61 last 496.00) total: 64768 open/close in 130.19 seconds: 497.49 ops/second - /unlink 8494 (time 1718927818.12 total 10.00 last 849.30) - /unlink 17026 (time 1718927828.12 total 20.00 last 853.19) - /unlink 20000 (time 1718927831.39 total 23.26 last 911.38) - /unlink 29019 (time 1718927841.39 total 33.26 last 901.86) - /unlink 37841 (time 1718927851.39 total 43.27 last 882.13) - /unlink 40000 (time 1718927853.84 total 45.72 last 880.22) - /unlink 48852 (time 1718927863.84 total 55.72 last 885.17) - /unlink 57877 (time 1718927873.84 total 65.72 last 902.43) - /unlink 60000 (time 1718927876.10 total 67.97 last 941.44) total: 64768 /unlink in 73.01 seconds: 887.12 ops/second - open/close 5313 (time 1718927892.18 total 10.00 last 531.26) - open/close 10000 (time 1718927901.53 total 19.35 last 501.43) - open/close 15113 (time 1718927911.53 total 29.35 last 511.23) - open/close 20000 (time 1718927921.23 total 39.05 last 503.69) - open/close 25116 (time 1718927931.23 total 49.05 last 511.53) - open/close 30000 (time 1718927940.98 total 58.80 last 501.31) - open/close 35019 (time 1718927950.98 total 68.80 last 501.82) - open/close 39909 (time 1718927960.98 total 78.80 last 488.92) - open/close 44863 (time 1718927970.98 total 88.80 last 495.38) - open/close 49997 (time 1718927980.98 total 98.80 last 513.34) - open/close 55152 (time 1718927990.98 total 108.80 last 515.50) - open/close 60000 (time 1718928000.66 total 118.48 last 500.87) total: 64768 open/close in 127.64 seconds: 507.44 ops/second - /unlink 8711 (time 1718928020.66 total 10.00 last 870.95) - /unlink 17481 (time 1718928030.66 total 20.00 last 876.89) - /unlink 20000 (time 1718928033.88 total 23.22 last 782.01) - /unlink 28625 (time 1718928043.88 total 33.23 last 862.43) - /unlink 36786 (time 1718928053.88 total 43.23 last 816.08) - /unlink 40000 (time 1718928057.83 total 47.18 last 813.14) - /unlink 48977 (time 1718928067.83 total 57.18 last 897.67) - /unlink 57164 (time 1718928077.84 total 67.18 last 818.64) - /unlink 60000 (time 1718928082.22 total 71.56 last 647.31) total: 64768 /unlink in 77.27 seconds: 838.21 ops/second - open/close 4541 (time 1718928098.77 total 10.00 last 454.03) - open/close 9057 (time 1718928108.78 total 20.00 last 451.52) - open/close 13623 (time 1718928118.78 total 30.01 last 456.52) - open/close 18814 (time 1718928128.78 total 40.01 last 519.04) - open/close 20000 (time 1718928131.00 total 42.23 last 533.87) - open/close 24856 (time 1718928141.00 total 52.23 last 485.50) - open/close 29640 (time 1718928151.00 total 62.23 last 478.36) - open/close 34553 (time 1718928161.00 total 72.23 last 491.25) - open/close 39427 (time 1718928171.00 total 82.23 last 487.36) - open/close 44181 (time 1718928181.01 total 92.23 last 475.35) - open/close 49152 (time 1718928191.01 total 102.24 last 496.74) - open/close 54123 (time 1718928201.01 total 112.24 last 497.08) - open/close 58255 (time 1718928211.01 total 122.24 last 413.15) - open/close 60000 (time 1718928214.64 total 125.87 last 480.92) total: 64768 open/close in 135.17 seconds: 479.16 ops/second - /unlink 8986 (time 1718928234.77 total 10.00 last 898.54) - /unlink 18057 (time 1718928244.77 total 20.00 last 907.10) - /unlink 20000 (time 1718928246.91 total 22.14 last 906.91) - /unlink 28989 (time 1718928256.91 total 32.14 last 898.85) - /unlink 38172 (time 1718928266.91 total 42.14 last 918.26) - /unlink 40000 (time 1718928268.97 total 44.20 last 887.05) - /unlink 49207 (time 1718928278.97 total 54.21 last 920.69) - /unlink 58258 (time 1718928288.97 total 64.21 last 905.07) total: 64768 /unlink in 71.09 seconds: 911.13 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg146-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg146-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg146-server stop mds service on oleg146-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg146-server LNET ready to unload unloading modules on: 'oleg146-server' oleg146-server: oleg146-server.virtnet: executing unload_modules_local oleg146-server: LNET ready to unload modules unloaded. pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-client: ssh exited with exit code 2 pdsh@oleg146-client: oleg146-server: ssh exited with exit code 2