== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:41:42 (1743194502) Stopping clients: oleg118-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg118-client.virtnet /mnt/lustre2 (opts:-f) oleg118-server: oleg118-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing load_modules_local oleg118-server: Loading modules from /home/green/git/lustre-release/lustre oleg118-server: detected 4 online CPUs by sysfs oleg118-server: MODOPTS_LIBCFS= oleg118-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg118-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg118-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg118-server: oleg118-server.virtnet: executing set_default_debug -1 all pdsh@oleg118-client: oleg118-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg118-client.virtnet: -o user_xattr,flock 192.168.201.118@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4972 (time 1743194536.55 total 10.00 last 497.11) - open/close 9894 (time 1743194546.55 total 20.00 last 492.19) - open/close 15421 (time 1743194556.55 total 30.00 last 552.63) - open/close 20000 (time 1743194564.22 total 37.68 last 596.83) - open/close 25848 (time 1743194574.22 total 47.68 last 584.71) - open/close 30000 (time 1743194581.09 total 54.55 last 604.31) - open/close 36032 (time 1743194591.09 total 64.55 last 603.16) - open/close 40000 (time 1743194597.32 total 70.78 last 636.80) - open/close 45715 (time 1743194607.32 total 80.78 last 571.44) - open/close 50000 (time 1743194614.60 total 88.06 last 588.76) - open/close 55699 (time 1743194624.60 total 98.06 last 569.81) - open/close 60000 (time 1743194632.67 total 106.13 last 533.17) total: 64768 open/close in 115.17 seconds: 562.35 ops/second - /unlink 10000 (time 1743194652.39 total 9.89 last 1011.19) - /unlink 20000 (time 1743194662.03 total 19.53 last 1037.23) - /unlink 29548 (time 1743194672.03 total 29.53 last 954.72) - /unlink 38621 (time 1743194682.03 total 39.53 last 907.24) - /unlink 47818 (time 1743194692.03 total 49.53 last 919.70) - /unlink 50000 (time 1743194694.43 total 51.93 last 909.52) - /unlink 59338 (time 1743194704.43 total 61.93 last 933.80) total: 64768 /unlink in 68.03 seconds: 952.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4526 (time 1743194749.39 total 10.00 last 452.58) - open/close 9000 (time 1743194759.41 total 20.02 last 446.59) - open/close 10000 (time 1743194761.61 total 22.21 last 455.32) - open/close 14397 (time 1743194771.61 total 32.22 last 439.63) - open/close 18882 (time 1743194781.61 total 42.22 last 448.45) - open/close 20000 (time 1743194784.10 total 44.71 last 448.81) - open/close 24739 (time 1743194794.10 total 54.71 last 473.84) - open/close 29537 (time 1743194804.10 total 64.71 last 479.76) - open/close 35453 (time 1743194814.10 total 74.71 last 591.60) - open/close 40000 (time 1743194821.95 total 82.55 last 579.70) - open/close 46043 (time 1743194831.95 total 92.55 last 604.25) - open/close 50000 (time 1743194839.68 total 100.28 last 512.04) - open/close 55153 (time 1743194849.68 total 110.28 last 515.28) - open/close 60000 (time 1743194859.57 total 120.18 last 489.73) total: 64768 open/close in 129.84 seconds: 498.82 ops/second - /unlink 8678 (time 1743194880.10 total 10.00 last 867.80) - /unlink 17491 (time 1743194890.10 total 20.00 last 881.27) - /unlink 20000 (time 1743194892.94 total 22.84 last 884.20) - /unlink 28721 (time 1743194902.94 total 32.84 last 872.01) - /unlink 37477 (time 1743194912.94 total 42.84 last 875.54) - /unlink 40000 (time 1743194915.87 total 45.77 last 859.84) - /unlink 48383 (time 1743194925.87 total 55.77 last 838.24) - /unlink 57593 (time 1743194935.87 total 65.78 last 920.95) - /unlink 60000 (time 1743194938.53 total 68.43 last 907.52) total: 64768 /unlink in 73.55 seconds: 880.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5343 (time 1743194980.00 total 10.00 last 534.22) - open/close 10000 (time 1743194988.75 total 18.75 last 532.22) - open/close 15138 (time 1743194998.75 total 28.75 last 513.73) - open/close 20000 (time 1743195007.86 total 37.86 last 533.60) - open/close 25124 (time 1743195017.87 total 47.87 last 512.34) - open/close 30000 (time 1743195027.29 total 57.29 last 517.51) - open/close 35118 (time 1743195037.29 total 67.29 last 511.76) - open/close 40000 (time 1743195047.16 total 77.16 last 494.69) - open/close 45074 (time 1743195057.16 total 87.16 last 507.32) - open/close 50000 (time 1743195066.81 total 96.81 last 510.68) - open/close 54987 (time 1743195076.81 total 106.81 last 498.61) - open/close 60000 (time 1743195086.66 total 116.66 last 508.82) total: 64768 open/close in 126.00 seconds: 514.05 ops/second - /unlink 9121 (time 1743195106.83 total 10.00 last 912.04) - /unlink 18174 (time 1743195116.83 total 20.00 last 905.20) - /unlink 20000 (time 1743195118.97 total 22.14 last 852.67) - /unlink 28591 (time 1743195128.97 total 32.14 last 859.05) - /unlink 37411 (time 1743195138.97 total 42.14 last 881.97) - /unlink 40000 (time 1743195141.87 total 45.04 last 893.08) - /unlink 48995 (time 1743195151.87 total 55.04 last 899.42) - /unlink 58188 (time 1743195161.87 total 65.04 last 919.23) - /unlink 60000 (time 1743195163.97 total 67.14 last 865.29) total: 64768 /unlink in 72.25 seconds: 896.50 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5368 (time 1743195205.46 total 10.00 last 536.72) - open/close 10000 (time 1743195214.24 total 18.77 last 528.04) - open/close 15248 (time 1743195224.24 total 28.77 last 524.79) - open/close 20000 (time 1743195233.46 total 37.99 last 515.46) - open/close 25184 (time 1743195243.46 total 47.99 last 518.38) - open/close 30000 (time 1743195252.70 total 57.24 last 520.90) - open/close 34982 (time 1743195262.70 total 67.24 last 498.10) - open/close 39840 (time 1743195272.70 total 77.24 last 485.79) - open/close 44216 (time 1743195282.70 total 87.24 last 437.59) - open/close 49218 (time 1743195292.71 total 97.24 last 500.13) - open/close 54090 (time 1743195302.71 total 107.24 last 487.11) - open/close 59186 (time 1743195312.71 total 117.25 last 509.52) - open/close 63779 (time 1743195322.71 total 127.25 last 459.20) total: 64768 open/close in 129.67 seconds: 499.49 ops/second - /unlink 6927 (time 1743195336.02 total 10.00 last 692.64) - /unlink 10000 (time 1743195340.75 total 14.73 last 650.40) - /unlink 15889 (time 1743195350.75 total 24.73 last 588.83) - /unlink 20000 (time 1743195357.36 total 31.34 last 621.92) - /unlink 28379 (time 1743195367.36 total 41.34 last 837.90) - /unlink 36065 (time 1743195377.36 total 51.34 last 768.49) - /unlink 40000 (time 1743195383.02 total 57.00 last 695.12) - /unlink 48778 (time 1743195393.02 total 67.00 last 877.74) - /unlink 55300 (time 1743195403.02 total 77.00 last 652.10) - /unlink 60000 (time 1743195408.96 total 82.95 last 790.77) total: 64768 /unlink in 87.97 seconds: 736.25 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5237 (time 1743195445.94 total 10.00 last 523.57) - open/close 9487 (time 1743195455.94 total 20.01 last 424.88) - open/close 13837 (time 1743195465.94 total 30.01 last 434.99) - open/close 17399 (time 1743195475.95 total 40.01 last 356.14) - open/close 20000 (time 1743195481.31 total 45.38 last 484.56) - open/close 24218 (time 1743195491.31 total 55.38 last 421.76) - open/close 28679 (time 1743195501.32 total 65.38 last 446.05) - open/close 30000 (time 1743195504.23 total 68.29 last 453.82) - open/close 34380 (time 1743195514.23 total 78.29 last 437.99) - open/close 38759 (time 1743195524.23 total 88.29 last 437.82) - open/close 40000 (time 1743195526.81 total 90.87 last 480.86) - open/close 44504 (time 1743195536.81 total 100.87 last 450.37) - open/close 48914 (time 1743195546.81 total 110.87 last 440.99) - open/close 50000 (time 1743195549.19 total 113.25 last 456.01) - open/close 54360 (time 1743195559.19 total 123.26 last 435.91) - open/close 58645 (time 1743195569.20 total 133.26 last 428.42) - open/close 60000 (time 1743195572.33 total 136.39 last 432.32) - open/close 64269 (time 1743195582.33 total 146.39 last 426.88) total: 64768 open/close in 147.60 seconds: 438.79 ops/second - /unlink 6011 (time 1743195594.42 total 10.00 last 601.09) - /unlink 10000 (time 1743195601.72 total 17.30 last 546.72) - /unlink 15443 (time 1743195611.72 total 27.30 last 544.21) - /unlink 20000 (time 1743195620.04 total 35.62 last 547.83) - /unlink 25602 (time 1743195630.04 total 45.62 last 560.16) - /unlink 30000 (time 1743195637.21 total 52.79 last 612.93) - /unlink 35979 (time 1743195647.21 total 62.79 last 597.82) - /unlink 40000 (time 1743195652.54 total 68.13 last 754.21) - /unlink 46312 (time 1743195662.55 total 78.13 last 631.07) - /unlink 50000 (time 1743195668.49 total 84.07 last 620.79) - /unlink 55357 (time 1743195678.49 total 94.07 last 535.62) - /unlink 60000 (time 1743195686.21 total 101.79 last 601.24) total: 64768 /unlink in 109.12 seconds: 593.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg118-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg118-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg118-server stop mds service on oleg118-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg118-server unloading modules via unload_modules_local on: 'oleg118-server' oleg118-server: oleg118-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-client: ssh exited with exit code 2 pdsh@oleg118-client: oleg118-server: ssh exited with exit code 2