== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 20:23:37 (1736990617) Stopping clients: oleg307-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg307-client.virtnet /mnt/lustre2 (opts:-f) oleg307-server: oleg307-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing load_modules_local oleg307-server: Loading modules from /home/green/git/lustre-release/lustre oleg307-server: detected 4 online CPUs by sysfs oleg307-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg307-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg307-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg307-server: oleg307-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg307-client.virtnet: -o user_xattr,flock oleg307-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5622 (time 1736990651.53 total 10.00 last 562.11) - open/close 10000 (time 1736990659.25 total 17.72 last 567.08) - open/close 15501 (time 1736990669.25 total 27.72 last 550.09) - open/close 20000 (time 1736990677.35 total 35.82 last 555.40) - open/close 25325 (time 1736990687.35 total 45.82 last 532.46) - open/close 30000 (time 1736990696.04 total 54.51 last 537.97) - open/close 35331 (time 1736990706.05 total 64.51 last 533.03) - open/close 40000 (time 1736990715.25 total 73.72 last 506.99) - open/close 45147 (time 1736990725.25 total 83.72 last 514.69) - open/close 50000 (time 1736990734.38 total 92.85 last 531.83) - open/close 55381 (time 1736990744.38 total 102.85 last 538.08) - open/close 60000 (time 1736990752.97 total 111.44 last 537.53) total: 64768 open/close in 120.31 seconds: 538.34 ops/second - /unlink 9221 (time 1736990772.70 total 10.00 last 922.01) - /unlink 18535 (time 1736990782.70 total 20.00 last 931.37) - /unlink 27509 (time 1736990792.70 total 30.00 last 897.28) - /unlink 30000 (time 1736990795.36 total 32.66 last 936.72) - /unlink 40000 (time 1736990805.24 total 42.54 last 1012.04) - /unlink 48767 (time 1736990815.24 total 52.54 last 876.68) - /unlink 58733 (time 1736990825.24 total 62.54 last 996.54) total: 64768 /unlink in 67.73 seconds: 956.31 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 5408 (time 1736990871.16 total 10.00 last 540.79) - open/close 10000 (time 1736990879.85 total 18.69 last 528.67) - open/close 15093 (time 1736990889.85 total 28.69 last 509.30) - open/close 20000 (time 1736990898.81 total 37.65 last 547.68) - open/close 25282 (time 1736990908.81 total 47.65 last 528.12) - open/close 30000 (time 1736990917.64 total 56.48 last 533.92) - open/close 35427 (time 1736990927.65 total 66.49 last 542.61) - open/close 40000 (time 1736990936.15 total 74.99 last 537.87) - open/close 45445 (time 1736990946.15 total 84.99 last 544.40) - open/close 50000 (time 1736990954.72 total 93.56 last 531.26) - open/close 55496 (time 1736990964.72 total 103.56 last 549.57) - open/close 60000 (time 1736990973.11 total 111.95 last 536.86) total: 64768 open/close in 120.96 seconds: 535.43 ops/second - /unlink 9798 (time 1736990992.94 total 10.00 last 979.77) - /unlink 19083 (time 1736991002.94 total 20.00 last 928.47) - /unlink 27526 (time 1736991012.94 total 30.00 last 844.29) - /unlink 30000 (time 1736991015.50 total 32.56 last 968.00) - /unlink 39851 (time 1736991025.50 total 42.56 last 985.06) - /unlink 49316 (time 1736991035.50 total 52.56 last 946.44) - /unlink 57875 (time 1736991045.50 total 62.56 last 855.89) - /unlink 60000 (time 1736991047.58 total 64.63 last 1024.47) total: 64768 /unlink in 70.45 seconds: 919.40 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3752 (time 1736991096.48 total 10.00 last 375.15) - open/close 8642 (time 1736991106.48 total 20.00 last 488.95) - open/close 10000 (time 1736991109.31 total 22.82 last 481.13) - open/close 15387 (time 1736991119.31 total 32.83 last 538.67) - open/close 19944 (time 1736991129.31 total 42.83 last 455.67) - open/close 23219 (time 1736991139.31 total 52.83 last 327.45) - open/close 26075 (time 1736991149.31 total 62.83 last 285.51) - open/close 28917 (time 1736991159.32 total 72.83 last 284.09) - open/close 30000 (time 1736991163.19 total 76.70 last 279.98) - open/close 33441 (time 1736991173.19 total 86.70 last 344.03) - open/close 36606 (time 1736991183.19 total 96.71 last 316.40) - open/close 39843 (time 1736991193.19 total 106.71 last 323.65) - open/close 43019 (time 1736991203.19 total 116.71 last 317.59) - open/close 46179 (time 1736991213.20 total 126.71 last 315.90) - open/close 48840 (time 1736991223.20 total 136.71 last 266.10) - open/close 50000 (time 1736991227.50 total 141.02 last 269.51) - open/close 52742 (time 1736991237.50 total 151.02 last 274.19) - open/close 56113 (time 1736991247.50 total 161.02 last 337.08) - open/close 59628 (time 1736991257.50 total 171.02 last 351.48) - open/close 64387 (time 1736991267.50 total 181.02 last 475.86) total: 64768 open/close in 182.01 seconds: 355.85 ops/second - /unlink 8922 (time 1736991279.36 total 10.00 last 892.10) - /unlink 16043 (time 1736991289.36 total 20.00 last 712.06) - /unlink 20000 (time 1736991293.72 total 24.37 last 905.94) - /unlink 26920 (time 1736991303.73 total 34.37 last 691.95) - /unlink 30000 (time 1736991308.08 total 38.73 last 707.05) - /unlink 36587 (time 1736991318.08 total 48.73 last 658.63) - /unlink 40000 (time 1736991321.71 total 52.35 last 941.40) - /unlink 46979 (time 1736991331.71 total 62.35 last 697.89) - /unlink 50000 (time 1736991336.71 total 67.36 last 603.73) - /unlink 56588 (time 1736991346.71 total 77.36 last 658.78) - /unlink 60000 (time 1736991353.39 total 84.04 last 510.84) total: 64768 /unlink in 92.72 seconds: 698.51 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3274 (time 1736991391.49 total 10.00 last 327.29) - open/close 7865 (time 1736991401.49 total 20.00 last 459.04) - open/close 10000 (time 1736991405.85 total 24.37 last 489.05) - open/close 13968 (time 1736991415.85 total 34.37 last 396.76) - open/close 17765 (time 1736991425.86 total 44.37 last 379.58) - open/close 20000 (time 1736991432.19 total 50.71 last 352.89) - open/close 24161 (time 1736991442.19 total 60.71 last 416.06) - open/close 27520 (time 1736991452.19 total 70.71 last 335.79) - open/close 30000 (time 1736991461.63 total 80.15 last 262.90) - open/close 32957 (time 1736991471.63 total 90.15 last 295.64) - open/close 37584 (time 1736991481.63 total 100.15 last 462.64) - open/close 40000 (time 1736991486.69 total 105.21 last 477.24) - open/close 44596 (time 1736991496.70 total 115.21 last 459.54) - open/close 48572 (time 1736991506.70 total 125.21 last 397.55) - open/close 50000 (time 1736991511.17 total 129.69 last 319.17) - open/close 53079 (time 1736991521.17 total 139.69 last 307.86) - open/close 56723 (time 1736991531.18 total 149.69 last 364.29) - open/close 60000 (time 1736991539.47 total 157.98 last 395.19) - open/close 63301 (time 1736991549.47 total 167.98 last 330.10) total: 64768 open/close in 170.88 seconds: 379.03 ops/second - /unlink 9105 (time 1736991563.24 total 10.00 last 910.46) - /unlink 16253 (time 1736991573.24 total 20.00 last 714.64) - /unlink 20000 (time 1736991580.60 total 27.37 last 508.86) - /unlink 27169 (time 1736991590.60 total 37.37 last 716.85) - /unlink 30000 (time 1736991594.23 total 41.00 last 779.57) - /unlink 37334 (time 1736991604.24 total 51.00 last 733.35) - /unlink 40000 (time 1736991608.11 total 54.88 last 687.56) - /unlink 48392 (time 1736991618.11 total 64.88 last 839.11) - /unlink 50000 (time 1736991620.36 total 67.12 last 715.72) - /unlink 56869 (time 1736991630.36 total 77.13 last 686.84) - /unlink 60000 (time 1736991635.40 total 82.16 last 621.60) total: 64768 /unlink in 87.41 seconds: 741.00 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete - open/close 3360 (time 1736991672.90 total 10.00 last 335.96) - open/close 6598 (time 1736991682.90 total 20.00 last 323.80) - open/close 9898 (time 1736991692.90 total 30.00 last 329.99) - open/close 15287 (time 1736991702.90 total 40.00 last 538.87) - open/close 20000 (time 1736991711.65 total 48.74 last 539.07) - open/close 24256 (time 1736991721.65 total 58.75 last 425.58) - open/close 28866 (time 1736991731.65 total 68.75 last 460.95) - open/close 30000 (time 1736991733.97 total 71.07 last 488.32) - open/close 33075 (time 1736991743.97 total 81.07 last 307.50) - open/close 36810 (time 1736991753.97 total 91.07 last 373.47) - open/close 40000 (time 1736991762.26 total 99.35 last 385.06) - open/close 44881 (time 1736991772.26 total 109.35 last 488.10) - open/close 50000 (time 1736991782.17 total 119.26 last 516.51) - open/close 54340 (time 1736991792.17 total 129.27 last 433.95) - open/close 58096 (time 1736991802.17 total 139.27 last 375.55) - open/close 60000 (time 1736991806.07 total 143.17 last 487.93) total: 64768 open/close in 151.74 seconds: 426.84 ops/second - /unlink 9292 (time 1736991825.45 total 10.00 last 929.16) - /unlink 19060 (time 1736991835.45 total 20.00 last 976.79) - /unlink 28736 (time 1736991845.45 total 30.00 last 967.59) - /unlink 38484 (time 1736991855.45 total 40.00 last 974.73) - /unlink 47789 (time 1736991865.45 total 50.00 last 930.43) - /unlink 50000 (time 1736991867.73 total 52.29 last 968.44) - /unlink 59733 (time 1736991877.73 total 62.29 last 973.19) total: 64768 /unlink in 67.35 seconds: 961.62 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg307-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg307-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg307-server stop mds service on oleg307-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg307-server LNET ready to unload unloading modules on: 'oleg307-server' oleg307-server: oleg307-server.virtnet: executing unload_modules_local oleg307-server: LNET ready to unload modules unloaded. pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-client: ssh exited with exit code 2 pdsh@oleg307-client: oleg307-server: ssh exited with exit code 2