== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:23:12 (1736263392) Stopping clients: oleg151-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg151-client.virtnet /mnt/lustre2 (opts:-f) oleg151-server: oleg151-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing load_modules_local oleg151-server: Loading modules from /home/green/git/lustre-release/lustre oleg151-server: detected 4 online CPUs by sysfs oleg151-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg151-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg151-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg151-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg151-server: oleg151-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg151-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg151-client.virtnet: -o user_xattr,flock oleg151-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 4068 (time 1736263429.35 total 10.00 last 406.67) - open/close 7550 (time 1736263439.36 total 20.01 last 348.13) - open/close 10000 (time 1736263445.70 total 26.35 last 386.20) - open/close 13986 (time 1736263455.70 total 36.35 last 398.55) - open/close 18668 (time 1736263465.70 total 46.35 last 468.13) - open/close 20000 (time 1736263468.71 total 49.36 last 442.26) - open/close 23973 (time 1736263478.72 total 59.37 last 397.24) - open/close 28459 (time 1736263488.72 total 69.37 last 448.58) - open/close 30000 (time 1736263492.12 total 72.77 last 453.00) - open/close 34510 (time 1736263502.12 total 82.77 last 451.00) - open/close 38789 (time 1736263512.12 total 92.77 last 427.87) - open/close 40000 (time 1736263514.83 total 95.48 last 446.39) - open/close 44363 (time 1736263524.83 total 105.48 last 436.16) - open/close 49380 (time 1736263534.84 total 115.48 last 501.68) - open/close 54003 (time 1736263544.84 total 125.49 last 462.26) - open/close 57659 (time 1736263554.84 total 135.49 last 365.58) - open/close 60000 (time 1736263560.45 total 141.10 last 416.72) - open/close 64384 (time 1736263570.46 total 151.10 last 438.36) total: 64768 open/close in 151.93 seconds: 426.31 ops/second - /unlink 7945 (time 1736263582.19 total 10.00 last 794.49) - /unlink 10000 (time 1736263584.59 total 12.40 last 854.62) - /unlink 17907 (time 1736263594.59 total 22.40 last 790.69) - /unlink 20000 (time 1736263597.49 total 25.30 last 722.11) - /unlink 28474 (time 1736263607.49 total 35.30 last 847.36) - /unlink 36244 (time 1736263617.49 total 45.30 last 776.99) - /unlink 40000 (time 1736263622.21 total 50.02 last 796.43) - /unlink 48915 (time 1736263632.21 total 60.02 last 891.44) - /unlink 57080 (time 1736263642.21 total 70.02 last 816.50) - /unlink 60000 (time 1736263646.62 total 74.43 last 661.96) total: 64768 /unlink in 81.74 seconds: 792.36 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5023 (time 1736263683.96 total 10.00 last 502.26) - open/close 9561 (time 1736263693.96 total 20.00 last 453.75) - open/close 14661 (time 1736263703.96 total 30.00 last 509.88) - open/close 20000 (time 1736263713.93 total 39.97 last 535.69) - open/close 25434 (time 1736263723.93 total 49.97 last 543.35) - open/close 29671 (time 1736263733.93 total 59.97 last 423.62) - open/close 34492 (time 1736263743.93 total 69.98 last 482.03) - open/close 40000 (time 1736263753.74 total 79.78 last 561.53) - open/close 44686 (time 1736263763.74 total 89.78 last 468.57) - open/close 48994 (time 1736263773.74 total 99.79 last 430.77) - open/close 54245 (time 1736263783.74 total 109.79 last 525.04) - open/close 59462 (time 1736263793.74 total 119.79 last 521.68) - open/close 63910 (time 1736263803.74 total 129.79 last 444.76) total: 64768 open/close in 131.43 seconds: 492.80 ops/second - /unlink 8883 (time 1736263816.20 total 10.00 last 888.27) - /unlink 17641 (time 1736263826.20 total 20.00 last 875.80) - /unlink 20000 (time 1736263828.68 total 22.48 last 951.98) - /unlink 29964 (time 1736263838.68 total 32.48 last 996.39) - /unlink 38277 (time 1736263848.68 total 42.48 last 831.22) - /unlink 47447 (time 1736263858.68 total 52.48 last 916.94) - /unlink 50000 (time 1736263861.24 total 55.04 last 997.22) - /unlink 60000 (time 1736263870.62 total 64.41 last 1067.09) total: 64768 /unlink in 68.73 seconds: 942.30 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5303 (time 1736263919.07 total 10.00 last 530.24) - open/close 10000 (time 1736263927.66 total 18.60 last 546.18) - open/close 15333 (time 1736263937.67 total 28.60 last 533.21) - open/close 20000 (time 1736263945.46 total 36.39 last 599.20) - open/close 25458 (time 1736263955.46 total 46.39 last 545.77) - open/close 30000 (time 1736263964.38 total 55.32 last 508.95) - open/close 36006 (time 1736263974.38 total 65.32 last 600.60) - open/close 40000 (time 1736263981.63 total 72.57 last 550.76) - open/close 44929 (time 1736263991.65 total 82.58 last 492.12) - open/close 49573 (time 1736264001.65 total 92.58 last 464.36) - open/close 54481 (time 1736264011.65 total 102.59 last 490.73) - open/close 60000 (time 1736264021.11 total 112.04 last 583.60) total: 64768 open/close in 120.31 seconds: 538.32 ops/second - /unlink 9239 (time 1736264040.21 total 10.00 last 923.89) - /unlink 17946 (time 1736264050.21 total 20.00 last 870.65) - /unlink 20000 (time 1736264052.60 total 22.39 last 861.10) - /unlink 28972 (time 1736264062.60 total 32.39 last 897.16) - /unlink 37705 (time 1736264072.60 total 42.39 last 873.25) - /unlink 40000 (time 1736264075.23 total 45.02 last 872.74) - /unlink 49865 (time 1736264085.23 total 55.02 last 986.46) - /unlink 60000 (time 1736264094.38 total 64.17 last 1107.02) total: 64768 /unlink in 68.34 seconds: 947.71 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5103 (time 1736264139.64 total 10.00 last 510.22) - open/close 10000 (time 1736264149.08 total 19.44 last 518.86) - open/close 15069 (time 1736264159.08 total 29.44 last 506.83) - open/close 19831 (time 1736264169.08 total 39.44 last 476.10) - open/close 24538 (time 1736264179.09 total 49.44 last 470.64) - open/close 30000 (time 1736264187.97 total 58.33 last 614.59) - open/close 35565 (time 1736264197.97 total 68.33 last 556.46) - open/close 40000 (time 1736264206.86 total 77.22 last 498.97) - open/close 45323 (time 1736264216.86 total 87.22 last 532.25) - open/close 50000 (time 1736264225.86 total 96.22 last 519.87) - open/close 55079 (time 1736264235.86 total 106.22 last 507.82) - open/close 60000 (time 1736264245.24 total 115.60 last 524.48) total: 64768 open/close in 124.57 seconds: 519.94 ops/second - /unlink 9219 (time 1736264265.05 total 10.00 last 921.85) - /unlink 18545 (time 1736264275.05 total 20.00 last 932.55) - /unlink 28073 (time 1736264285.05 total 30.00 last 952.73) - /unlink 30000 (time 1736264287.06 total 32.01 last 960.45) - /unlink 39204 (time 1736264297.06 total 42.01 last 920.33) - /unlink 48415 (time 1736264307.06 total 52.01 last 921.04) - /unlink 57838 (time 1736264317.06 total 62.01 last 942.24) - /unlink 60000 (time 1736264319.31 total 64.26 last 959.79) total: 64768 /unlink in 68.98 seconds: 938.94 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4339 (time 1736264360.27 total 10.00 last 433.79) - open/close 9323 (time 1736264370.27 total 20.00 last 498.31) - open/close 15042 (time 1736264380.28 total 30.01 last 571.87) - open/close 20000 (time 1736264389.14 total 38.87 last 559.53) - open/close 25565 (time 1736264399.14 total 48.87 last 556.42) - open/close 29203 (time 1736264409.14 total 58.87 last 363.67) - open/close 34512 (time 1736264419.14 total 68.87 last 530.82) - open/close 40000 (time 1736264428.95 total 78.68 last 559.84) - open/close 45456 (time 1736264438.95 total 88.68 last 545.57) - open/close 50000 (time 1736264447.34 total 97.07 last 541.29) - open/close 55456 (time 1736264457.34 total 107.07 last 545.50) - open/close 60000 (time 1736264465.64 total 115.37 last 547.86) total: 64768 open/close in 123.89 seconds: 522.80 ops/second - /unlink 9564 (time 1736264484.99 total 10.00 last 956.31) - /unlink 19158 (time 1736264494.99 total 20.00 last 959.35) - /unlink 28664 (time 1736264504.99 total 30.00 last 950.52) - /unlink 38230 (time 1736264514.99 total 40.00 last 956.52) - /unlink 47747 (time 1736264524.99 total 50.00 last 951.63) - /unlink 50000 (time 1736264527.30 total 52.32 last 974.66) - /unlink 59339 (time 1736264537.30 total 62.32 last 933.81) total: 64768 /unlink in 67.71 seconds: 956.50 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg151-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg151-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg151-server stop mds service on oleg151-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg151-server LNET ready to unload unloading modules on: 'oleg151-server' oleg151-server: oleg151-server.virtnet: executing unload_modules_local oleg151-server: LNET ready to unload modules unloaded. pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-client: ssh exited with exit code 2 pdsh@oleg151-client: oleg151-server: ssh exited with exit code 2