== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 05:38:49 (1721381929) Stopping clients: oleg349-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg349-client.virtnet /mnt/lustre2 (opts:-f) oleg349-server: oleg349-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: detected 4 online CPUs by sysfs oleg349-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg349-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg349-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock oleg349-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3540 (time 1721381966.10 total 10.00 last 353.95) - open/close 6890 (time 1721381976.10 total 20.00 last 334.93) - open/close 10000 (time 1721381985.57 total 29.47 last 328.36) - open/close 13409 (time 1721381995.57 total 39.48 last 340.77) - open/close 16549 (time 1721382005.57 total 49.48 last 313.94) - open/close 19542 (time 1721382015.58 total 59.48 last 299.27) - open/close 23234 (time 1721382025.58 total 69.48 last 369.14) - open/close 27179 (time 1721382035.58 total 79.48 last 394.47) - open/close 30000 (time 1721382042.24 total 86.15 last 423.20) - open/close 34707 (time 1721382052.25 total 96.15 last 470.62) - open/close 39658 (time 1721382062.25 total 106.15 last 494.98) - open/close 44553 (time 1721382072.25 total 116.15 last 489.46) - open/close 49547 (time 1721382082.25 total 126.16 last 499.32) - open/close 54159 (time 1721382092.25 total 136.16 last 461.12) - open/close 58831 (time 1721382102.25 total 146.16 last 467.15) - open/close 60000 (time 1721382105.25 total 149.16 last 389.41) - open/close 64421 (time 1721382115.26 total 159.16 last 441.99) total: 64768 open/close in 160.02 seconds: 404.75 ops/second - /unlink 7125 (time 1721382127.00 total 10.00 last 712.48) - /unlink 10000 (time 1721382131.23 total 14.23 last 679.14) - /unlink 17339 (time 1721382141.24 total 24.23 last 733.83) - /unlink 20000 (time 1721382144.99 total 27.99 last 708.14) - /unlink 28231 (time 1721382154.99 total 37.99 last 823.06) - /unlink 30000 (time 1721382157.31 total 40.31 last 763.57) - /unlink 38594 (time 1721382167.31 total 50.31 last 859.33) - /unlink 46871 (time 1721382177.31 total 60.31 last 827.62) - /unlink 50000 (time 1721382180.60 total 63.60 last 952.31) - /unlink 60000 (time 1721382190.56 total 73.56 last 1003.40) total: 64768 /unlink in 78.66 seconds: 823.38 ops/second - open/close 4831 (time 1721382206.50 total 10.00 last 483.07) - open/close 9239 (time 1721382216.50 total 20.00 last 440.73) - open/close 13786 (time 1721382226.50 total 30.00 last 454.59) - open/close 18861 (time 1721382236.50 total 40.00 last 507.48) - open/close 20000 (time 1721382238.53 total 42.04 last 560.28) - open/close 25886 (time 1721382248.53 total 52.04 last 588.58) - open/close 30000 (time 1721382255.74 total 59.25 last 570.53) - open/close 35520 (time 1721382265.75 total 69.25 last 551.96) - open/close 40000 (time 1721382273.18 total 76.69 last 602.34) - open/close 46253 (time 1721382283.18 total 86.69 last 625.28) - open/close 50000 (time 1721382289.16 total 92.66 last 627.28) - open/close 56238 (time 1721382299.16 total 102.66 last 623.71) - open/close 60000 (time 1721382305.23 total 108.73 last 619.88) total: 64768 open/close in 117.34 seconds: 551.98 ops/second - /unlink 8532 (time 1721382324.68 total 10.00 last 853.15) - /unlink 16784 (time 1721382334.68 total 20.00 last 825.18) - /unlink 20000 (time 1721382338.02 total 23.34 last 963.15) - /unlink 30000 (time 1721382347.82 total 33.14 last 1020.88) - /unlink 40000 (time 1721382357.31 total 42.62 last 1053.81) - /unlink 50000 (time 1721382366.51 total 51.82 last 1087.14) - /unlink 60000 (time 1721382376.04 total 61.36 last 1048.49) total: 64768 /unlink in 65.78 seconds: 984.68 ops/second - open/close 6130 (time 1721382391.24 total 10.00 last 613.00) - open/close 10000 (time 1721382397.75 total 16.51 last 594.31) - open/close 16017 (time 1721382407.75 total 26.51 last 601.70) - open/close 20000 (time 1721382414.50 total 33.26 last 590.36) - open/close 26151 (time 1721382424.50 total 43.26 last 615.09) - open/close 30000 (time 1721382430.87 total 49.62 last 604.67) - open/close 36330 (time 1721382440.87 total 59.62 last 632.97) - open/close 40000 (time 1721382446.84 total 65.60 last 613.92) - open/close 45093 (time 1721382456.85 total 75.60 last 509.23) - open/close 49928 (time 1721382466.85 total 85.60 last 483.49) - open/close 54812 (time 1721382476.85 total 95.61 last 488.36) - open/close 59874 (time 1721382486.85 total 105.61 last 506.14) total: 64768 open/close in 115.02 seconds: 563.11 ops/second - /unlink 8621 (time 1721382507.11 total 10.00 last 862.04) - /unlink 17686 (time 1721382517.11 total 20.00 last 906.45) - /unlink 20000 (time 1721382519.63 total 22.52 last 919.03) - /unlink 29185 (time 1721382529.63 total 32.52 last 918.45) - /unlink 38339 (time 1721382539.63 total 42.52 last 915.32) - /unlink 47353 (time 1721382549.63 total 52.52 last 901.33) - /unlink 50000 (time 1721382552.54 total 55.43 last 909.98) - /unlink 59132 (time 1721382562.54 total 65.43 last 913.17) total: 64768 /unlink in 71.48 seconds: 906.09 ops/second - open/close 5540 (time 1721382579.46 total 10.00 last 553.97) - open/close 10000 (time 1721382587.93 total 18.47 last 526.88) - open/close 15164 (time 1721382597.93 total 28.47 last 516.35) - open/close 20000 (time 1721382607.10 total 37.64 last 527.24) - open/close 25448 (time 1721382617.10 total 47.64 last 544.72) - open/close 30000 (time 1721382625.57 total 56.11 last 537.66) - open/close 35446 (time 1721382635.57 total 66.11 last 544.58) - open/close 40000 (time 1721382644.17 total 74.70 last 529.70) - open/close 45456 (time 1721382654.17 total 84.71 last 545.51) - open/close 50000 (time 1721382662.78 total 93.32 last 527.41) - open/close 55340 (time 1721382672.78 total 103.32 last 533.90) - open/close 60000 (time 1721382681.36 total 111.90 last 543.32) total: 64768 open/close in 120.96 seconds: 535.44 ops/second - /unlink 8987 (time 1721382701.26 total 10.00 last 898.64) - /unlink 18270 (time 1721382711.26 total 20.00 last 928.19) - /unlink 27532 (time 1721382721.27 total 30.00 last 926.15) - /unlink 30000 (time 1721382723.87 total 32.61 last 946.20) - /unlink 39401 (time 1721382733.87 total 42.61 last 940.09) - /unlink 47833 (time 1721382743.87 total 52.61 last 843.19) - /unlink 50000 (time 1721382746.27 total 55.00 last 905.94) - /unlink 58483 (time 1721382756.27 total 65.00 last 848.25) total: 64768 /unlink in 72.87 seconds: 888.82 ops/second - open/close 5505 (time 1721382775.18 total 10.00 last 550.48) - open/close 10000 (time 1721382784.67 total 19.49 last 473.79) - open/close 15137 (time 1721382794.67 total 29.49 last 513.62) - open/close 20000 (time 1721382803.48 total 38.30 last 552.23) - open/close 24668 (time 1721382813.48 total 48.30 last 466.79) - open/close 30000 (time 1721382823.32 total 58.14 last 541.47) - open/close 35484 (time 1721382833.33 total 68.14 last 548.34) - open/close 40000 (time 1721382841.64 total 76.46 last 543.28) - open/close 45345 (time 1721382851.64 total 86.46 last 534.47) - open/close 50000 (time 1721382860.30 total 95.12 last 537.59) - open/close 54970 (time 1721382870.30 total 105.12 last 496.97) - open/close 59796 (time 1721382880.30 total 115.12 last 482.59) total: 64768 open/close in 124.54 seconds: 520.06 ops/second - /unlink 8867 (time 1721382900.74 total 10.00 last 886.64) - /unlink 16616 (time 1721382910.74 total 20.00 last 774.88) - /unlink 20000 (time 1721382914.95 total 24.21 last 803.37) - /unlink 28109 (time 1721382924.95 total 34.21 last 810.87) - /unlink 30000 (time 1721382927.13 total 36.39 last 870.49) - /unlink 37895 (time 1721382937.13 total 46.39 last 789.43) - /unlink 40000 (time 1721382939.60 total 48.86 last 849.91) - /unlink 48108 (time 1721382949.61 total 58.86 last 810.79) - /unlink 50000 (time 1721382952.17 total 61.43 last 736.50) - /unlink 58194 (time 1721382962.17 total 71.43 last 819.39) - /unlink 60000 (time 1721382964.79 total 74.04 last 691.59) total: 64768 /unlink in 79.82 seconds: 811.41 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server LNET ready to unload unloading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local oleg349-server: LNET ready to unload modules unloaded. pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2