== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:36:38 (1736357798) Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) oleg251-server: oleg251-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5118 (time 1736357835.31 total 10.00 last 511.78) - open/close 9850 (time 1736357845.31 total 20.00 last 473.15) - open/close 14054 (time 1736357855.31 total 30.00 last 420.35) - open/close 17134 (time 1736357865.31 total 40.00 last 307.94) - open/close 20000 (time 1736357874.62 total 49.31 last 307.86) - open/close 23389 (time 1736357884.62 total 59.32 last 338.85) - open/close 26740 (time 1736357894.62 total 69.32 last 335.05) - open/close 30000 (time 1736357904.05 total 78.74 last 346.01) - open/close 33924 (time 1736357914.05 total 88.74 last 392.39) - open/close 38253 (time 1736357924.05 total 98.74 last 432.82) - open/close 40000 (time 1736357927.62 total 102.32 last 488.51) - open/close 44671 (time 1736357937.62 total 112.32 last 467.10) - open/close 48984 (time 1736357947.63 total 122.32 last 431.21) - open/close 50000 (time 1736357949.91 total 124.61 last 444.14) - open/close 54799 (time 1736357959.92 total 134.61 last 479.82) - open/close 59162 (time 1736357969.92 total 144.61 last 436.29) - open/close 60000 (time 1736357971.94 total 146.63 last 414.33) - open/close 63603 (time 1736357981.94 total 156.63 last 360.30) total: 64768 open/close in 159.81 seconds: 405.28 ops/second - /unlink 7200 (time 1736357996.03 total 10.00 last 719.92) - /unlink 10000 (time 1736357999.53 total 13.50 last 799.85) - /unlink 17334 (time 1736358009.54 total 23.50 last 733.34) - /unlink 20000 (time 1736358013.20 total 27.17 last 727.69) - /unlink 28579 (time 1736358023.20 total 37.17 last 857.87) - /unlink 36408 (time 1736358033.20 total 47.17 last 782.82) - /unlink 40000 (time 1736358038.50 total 52.47 last 677.42) - /unlink 49004 (time 1736358048.50 total 62.47 last 900.37) - /unlink 59153 (time 1736358058.50 total 72.47 last 1014.88) total: 64768 /unlink in 78.27 seconds: 827.51 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4996 (time 1736358100.11 total 10.00 last 499.58) - open/close 10000 (time 1736358109.31 total 19.21 last 543.56) - open/close 15193 (time 1736358119.31 total 29.21 last 519.27) - open/close 19958 (time 1736358129.31 total 39.21 last 476.40) - open/close 24531 (time 1736358139.31 total 49.21 last 457.26) - open/close 28976 (time 1736358149.32 total 59.21 last 444.46) - open/close 30000 (time 1736358151.69 total 61.58 last 431.43) - open/close 34958 (time 1736358161.69 total 71.58 last 495.80) - open/close 39557 (time 1736358171.69 total 81.58 last 459.87) - open/close 44398 (time 1736358181.69 total 91.59 last 484.02) - open/close 49588 (time 1736358191.69 total 101.59 last 518.96) - open/close 54777 (time 1736358201.69 total 111.59 last 518.82) - open/close 60000 (time 1736358211.23 total 121.12 last 547.72) total: 64768 open/close in 129.85 seconds: 498.78 ops/second - /unlink 10000 (time 1736358230.64 total 9.89 last 1011.27) - /unlink 20000 (time 1736358240.59 total 19.84 last 1005.23) - /unlink 30000 (time 1736358249.87 total 29.12 last 1077.04) - /unlink 39348 (time 1736358259.87 total 39.12 last 934.79) - /unlink 48259 (time 1736358269.87 total 49.12 last 891.02) - /unlink 57207 (time 1736358279.87 total 59.12 last 894.74) - /unlink 60000 (time 1736358282.77 total 62.02 last 963.97) total: 64768 /unlink in 67.50 seconds: 959.57 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5505 (time 1736358330.42 total 10.00 last 550.42) - open/close 10000 (time 1736358338.77 total 18.35 last 538.30) - open/close 15462 (time 1736358348.77 total 28.35 last 546.10) - open/close 20000 (time 1736358357.02 total 36.60 last 550.17) - open/close 25492 (time 1736358367.02 total 46.60 last 549.13) - open/close 30000 (time 1736358375.51 total 55.09 last 531.22) - open/close 35368 (time 1736358385.51 total 65.09 last 536.79) - open/close 40000 (time 1736358394.17 total 73.75 last 534.80) - open/close 45382 (time 1736358404.17 total 83.75 last 538.16) - open/close 50000 (time 1736358413.03 total 92.62 last 520.89) - open/close 55271 (time 1736358423.04 total 102.62 last 527.01) - open/close 60000 (time 1736358432.02 total 111.60 last 526.55) total: 64768 open/close in 120.52 seconds: 537.39 ops/second - /unlink 9479 (time 1736358451.78 total 10.00 last 947.85) - /unlink 18865 (time 1736358461.78 total 20.00 last 938.54) - /unlink 28281 (time 1736358471.78 total 30.00 last 941.57) - /unlink 37705 (time 1736358481.78 total 40.00 last 942.39) - /unlink 40000 (time 1736358484.28 total 42.51 last 916.19) - /unlink 49140 (time 1736358494.28 total 52.51 last 913.95) - /unlink 58746 (time 1736358504.28 total 62.51 last 960.52) total: 64768 /unlink in 68.69 seconds: 942.95 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5439 (time 1736358555.58 total 10.00 last 543.83) - open/close 10000 (time 1736358564.09 total 18.51 last 535.95) - open/close 15336 (time 1736358574.09 total 28.51 last 533.56) - open/close 20000 (time 1736358582.85 total 37.27 last 532.64) - open/close 25263 (time 1736358592.85 total 47.27 last 526.22) - open/close 30000 (time 1736358602.57 total 57.00 last 487.09) - open/close 35109 (time 1736358612.57 total 67.00 last 510.88) - open/close 40000 (time 1736358622.30 total 76.73 last 502.64) - open/close 45075 (time 1736358632.30 total 86.73 last 507.48) - open/close 50000 (time 1736358641.80 total 96.23 last 518.37) - open/close 54389 (time 1736358651.80 total 106.23 last 438.88) - open/close 58145 (time 1736358661.81 total 116.23 last 375.56) - open/close 60000 (time 1736358666.98 total 121.41 last 358.27) - open/close 64472 (time 1736358676.98 total 131.41 last 447.12) total: 64768 open/close in 132.15 seconds: 490.10 ops/second - /unlink 8721 (time 1736358688.77 total 10.00 last 872.06) - /unlink 17848 (time 1736358698.77 total 20.00 last 912.67) - /unlink 20000 (time 1736358701.03 total 22.26 last 951.58) - /unlink 29334 (time 1736358711.03 total 32.26 last 933.35) - /unlink 38350 (time 1736358721.03 total 42.26 last 901.56) - /unlink 47729 (time 1736358731.03 total 52.26 last 937.86) - /unlink 50000 (time 1736358733.40 total 54.63 last 959.31) - /unlink 58891 (time 1736358743.40 total 64.63 last 889.02) total: 64768 /unlink in 70.84 seconds: 914.24 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5460 (time 1736358791.22 total 10.00 last 545.96) - open/close 10000 (time 1736358799.48 total 18.27 last 549.19) - open/close 15516 (time 1736358809.48 total 28.27 last 551.60) - open/close 20000 (time 1736358817.74 total 36.52 last 543.03) - open/close 25256 (time 1736358827.74 total 46.53 last 525.56) - open/close 30000 (time 1736358837.43 total 56.22 last 489.50) - open/close 35137 (time 1736358847.44 total 66.22 last 513.60) - open/close 39639 (time 1736358857.44 total 76.22 last 450.14) - open/close 44938 (time 1736358867.44 total 86.22 last 529.86) - open/close 50000 (time 1736358877.27 total 96.05 last 514.99) - open/close 55217 (time 1736358887.27 total 106.05 last 521.63) - open/close 60000 (time 1736358896.85 total 115.63 last 499.15) total: 64768 open/close in 124.75 seconds: 519.20 ops/second - /unlink 8052 (time 1736358916.83 total 10.00 last 805.12) - /unlink 10000 (time 1736358919.09 total 12.26 last 860.51) - /unlink 18685 (time 1736358929.09 total 22.27 last 868.41) - /unlink 26785 (time 1736358939.09 total 32.27 last 809.95) - /unlink 30000 (time 1736358944.84 total 38.02 last 558.72) - /unlink 37723 (time 1736358954.84 total 48.02 last 772.29) - /unlink 40000 (time 1736358957.28 total 50.45 last 935.58) - /unlink 49412 (time 1736358967.28 total 60.45 last 941.18) - /unlink 58686 (time 1736358977.28 total 70.46 last 927.34) total: 64768 /unlink in 76.50 seconds: 846.68 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server LNET ready to unload unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local oleg251-server: LNET ready to unload modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2