== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:11:53 (1713208313) Stopping clients: oleg110-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg110-client.virtnet /mnt/lustre2 (opts:-f) oleg110-server: oleg110-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing load_modules_local oleg110-server: Loading modules from /home/green/git/lustre-release/lustre oleg110-server: detected 4 online CPUs by sysfs oleg110-server: Force libcfs to create 2 CPU partitions oleg110-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format mds2: lustre-mdt2/mdt2 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg110-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 start mds service on oleg110-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Commit the device label on lustre-mdt2/mdt2 Started lustre-MDT0001 oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg110-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg110-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg110-server: oleg110-server.virtnet: executing set_default_debug -1 all pdsh@oleg110-client: oleg110-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg110-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg110-client.virtnet: -o user_xattr,flock oleg110-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86890 331 86559 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 87086 312 86774 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16483 643 15840 4% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5113 (time 1713208354.57 total 10.00 last 511.23) - open/close 10000 (time 1713208364.21 total 19.65 last 506.49) - open/close 15149 (time 1713208374.21 total 29.65 last 514.86) - open/close 20000 (time 1713208383.55 total 38.99 last 519.68) - open/close 25247 (time 1713208393.55 total 48.99 last 524.70) - open/close 30000 (time 1713208402.67 total 58.11 last 520.96) - open/close 35162 (time 1713208412.67 total 68.11 last 516.17) - open/close 40000 (time 1713208422.38 total 77.81 last 498.53) - open/close 44878 (time 1713208432.38 total 87.82 last 487.71) - open/close 49845 (time 1713208442.38 total 97.82 last 496.65) - open/close 54775 (time 1713208452.38 total 107.82 last 492.91) - open/close 59759 (time 1713208462.38 total 117.82 last 498.34) - open/close 64441 (time 1713208472.38 total 127.82 last 468.16) total: 64768 open/close in 128.61 seconds: 503.62 ops/second - /unlink 7676 (time 1713208484.04 total 10.00 last 767.55) - /unlink 10000 (time 1713208486.92 total 12.88 last 808.07) - /unlink 17990 (time 1713208496.92 total 22.88 last 798.99) - /unlink 20000 (time 1713208499.47 total 25.43 last 787.66) - /unlink 27930 (time 1713208509.47 total 35.43 last 792.95) - /unlink 30000 (time 1713208512.05 total 38.01 last 801.41) - /unlink 37907 (time 1713208522.06 total 48.01 last 790.63) - /unlink 40000 (time 1713208524.66 total 50.61 last 804.96) - /unlink 47694 (time 1713208534.66 total 60.61 last 769.37) - /unlink 50000 (time 1713208537.62 total 63.58 last 777.93) - /unlink 58141 (time 1713208547.62 total 73.58 last 814.00) - /unlink 60000 (time 1713208549.96 total 75.92 last 793.47) total: 64768 /unlink in 81.76 seconds: 792.17 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5104 (time 1713208582.77 total 10.00 last 510.34) - open/close 9988 (time 1713208592.77 total 20.00 last 488.39) - open/close 14790 (time 1713208602.78 total 30.00 last 480.12) - open/close 19180 (time 1713208612.78 total 40.00 last 438.94) - open/close 24171 (time 1713208622.78 total 50.00 last 499.10) - open/close 29936 (time 1713208632.78 total 60.00 last 576.50) - open/close 35013 (time 1713208642.78 total 70.00 last 507.68) - open/close 39862 (time 1713208652.78 total 80.01 last 484.87) - open/close 44734 (time 1713208662.78 total 90.01 last 487.17) - open/close 49581 (time 1713208672.78 total 100.01 last 484.65) - open/close 54309 (time 1713208682.78 total 110.01 last 472.75) - open/close 58961 (time 1713208692.78 total 120.01 last 465.20) - open/close 60000 (time 1713208695.02 total 122.25 last 464.28) - open/close 64749 (time 1713208705.02 total 132.25 last 474.87) total: 64768 open/close in 132.28 seconds: 489.61 ops/second - /unlink 8141 (time 1713208715.97 total 10.00 last 814.07) - /unlink 10000 (time 1713208718.25 total 12.28 last 813.99) - /unlink 18211 (time 1713208728.25 total 22.28 last 821.07) - /unlink 20000 (time 1713208730.41 total 24.45 last 827.89) - /unlink 28033 (time 1713208740.41 total 34.45 last 803.29) - /unlink 30000 (time 1713208742.96 total 36.99 last 772.35) - /unlink 38262 (time 1713208752.96 total 46.99 last 826.15) - /unlink 40000 (time 1713208754.96 total 48.99 last 868.31) - /unlink 48323 (time 1713208764.96 total 59.00 last 832.23) - /unlink 56304 (time 1713208774.97 total 69.00 last 798.00) - /unlink 60000 (time 1713208779.61 total 73.64 last 795.60) total: 64768 /unlink in 79.31 seconds: 816.66 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5181 (time 1713208813.03 total 10.00 last 518.01) - open/close 10000 (time 1713208822.34 total 19.31 last 517.78) - open/close 15211 (time 1713208832.34 total 29.31 last 521.00) - open/close 20000 (time 1713208841.58 total 38.55 last 518.34) - open/close 25114 (time 1713208851.58 total 48.55 last 511.33) - open/close 30000 (time 1713208861.28 total 58.25 last 503.78) - open/close 35029 (time 1713208871.28 total 68.25 last 502.83) - open/close 40000 (time 1713208881.01 total 77.97 last 511.28) - open/close 45055 (time 1713208891.01 total 87.98 last 505.41) - open/close 49903 (time 1713208901.01 total 97.98 last 484.71) - open/close 54846 (time 1713208911.01 total 107.98 last 494.23) - open/close 59825 (time 1713208921.01 total 117.98 last 497.85) - open/close 64735 (time 1713208931.01 total 127.98 last 490.95) total: 64768 open/close in 128.04 seconds: 505.83 ops/second - /unlink 8158 (time 1713208941.91 total 10.00 last 815.75) - /unlink 10000 (time 1713208944.12 total 12.21 last 832.91) - /unlink 18229 (time 1713208954.12 total 22.21 last 822.85) - /unlink 20000 (time 1713208956.31 total 24.40 last 809.75) - /unlink 28324 (time 1713208966.31 total 34.40 last 832.32) - /unlink 30000 (time 1713208968.32 total 36.41 last 832.16) - /unlink 37608 (time 1713208978.32 total 46.42 last 760.72) - /unlink 40000 (time 1713208981.15 total 49.25 last 844.39) - /unlink 48468 (time 1713208991.16 total 59.25 last 846.74) - /unlink 56761 (time 1713209001.16 total 69.25 last 829.26) - /unlink 60000 (time 1713209005.00 total 73.09 last 842.47) total: 64768 /unlink in 79.04 seconds: 819.46 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4976 (time 1713209038.27 total 10.00 last 497.52) - open/close 7806 (time 1713209048.27 total 20.00 last 282.96) - open/close 10000 (time 1713209055.21 total 26.93 last 316.57) - open/close 14798 (time 1713209065.21 total 36.93 last 479.75) - open/close 19703 (time 1713209075.21 total 46.94 last 490.46) - open/close 23645 (time 1713209085.21 total 56.94 last 394.18) - open/close 28421 (time 1713209095.21 total 66.94 last 477.55) - open/close 30000 (time 1713209098.08 total 69.81 last 549.42) - open/close 34736 (time 1713209108.08 total 79.81 last 473.58) - open/close 39359 (time 1713209118.09 total 89.81 last 462.20) - open/close 44136 (time 1713209128.09 total 99.81 last 477.65) - open/close 48737 (time 1713209138.09 total 109.82 last 460.04) - open/close 50000 (time 1713209141.02 total 112.75 last 429.99) - open/close 54548 (time 1713209151.03 total 122.75 last 454.75) - open/close 59699 (time 1713209161.07 total 132.79 last 513.08) total: 64768 open/close in 142.51 seconds: 454.49 ops/second - /unlink 7318 (time 1713209181.57 total 10.00 last 731.66) - /unlink 10000 (time 1713209185.45 total 13.88 last 691.22) - /unlink 18731 (time 1713209195.45 total 23.88 last 873.03) - /unlink 26519 (time 1713209205.45 total 33.88 last 778.75) - /unlink 30000 (time 1713209209.38 total 37.82 last 884.72) - /unlink 38419 (time 1713209219.38 total 47.82 last 841.85) - /unlink 46457 (time 1713209229.38 total 57.82 last 803.74) - /unlink 50000 (time 1713209234.53 total 62.97 last 688.25) - /unlink 58370 (time 1713209244.53 total 72.97 last 836.99) total: 64768 /unlink in 80.70 seconds: 802.62 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 4619 (time 1713209284.21 total 10.00 last 461.83) - open/close 9640 (time 1713209294.21 total 20.00 last 502.04) - open/close 14069 (time 1713209304.21 total 30.00 last 442.86) - open/close 19097 (time 1713209314.22 total 40.01 last 502.70) - open/close 24104 (time 1713209324.22 total 50.01 last 500.66) - open/close 29156 (time 1713209334.22 total 60.01 last 505.19) - open/close 34355 (time 1713209344.22 total 70.01 last 519.81) - open/close 39361 (time 1713209354.22 total 80.01 last 500.53) - open/close 44570 (time 1713209364.22 total 90.01 last 520.82) - open/close 49092 (time 1713209374.22 total 100.01 last 452.12) - open/close 54281 (time 1713209384.22 total 110.01 last 518.87) - open/close 59671 (time 1713209394.22 total 120.01 last 538.95) total: 64768 open/close in 129.92 seconds: 498.53 ops/second - /unlink 8654 (time 1713209414.93 total 10.00 last 865.33) - /unlink 17020 (time 1713209424.93 total 20.00 last 836.55) - /unlink 20000 (time 1713209428.30 total 23.37 last 885.23) - /unlink 28105 (time 1713209438.30 total 33.37 last 810.46) - /unlink 30000 (time 1713209440.63 total 35.70 last 811.46) - /unlink 38245 (time 1713209450.63 total 45.70 last 824.41) - /unlink 40000 (time 1713209452.74 total 47.81 last 832.68) - /unlink 48747 (time 1713209462.74 total 57.81 last 874.69) - /unlink 57830 (time 1713209472.74 total 67.81 last 908.27) - /unlink 60000 (time 1713209475.12 total 70.20 last 910.38) total: 64768 /unlink in 75.22 seconds: 861.08 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg110-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg110-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg110-server stop mds service on oleg110-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg110-server unloading modules on: 'oleg110-server' oleg110-server: oleg110-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-client: ssh exited with exit code 2 pdsh@oleg110-client: oleg110-server: ssh exited with exit code 2