== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:52:10 (1743454330) Stopping clients: oleg141-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg141-client.virtnet /mnt/lustre2 (opts:-f) oleg141-server: oleg141-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: MODOPTS_LIBCFS= oleg141-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg141-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg141-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4289 (time 1743454366.06 total 10.00 last 428.85) - open/close 9764 (time 1743454376.06 total 20.00 last 547.42) - open/close 14705 (time 1743454386.06 total 30.00 last 494.06) - open/close 18800 (time 1743454396.06 total 40.00 last 409.50) - open/close 20000 (time 1743454398.53 total 42.47 last 485.76) - open/close 23711 (time 1743454408.53 total 52.48 last 371.03) - open/close 28754 (time 1743454418.53 total 62.48 last 504.27) - open/close 30000 (time 1743454420.56 total 64.51 last 613.42) - open/close 35889 (time 1743454430.56 total 74.51 last 588.90) - open/close 40000 (time 1743454438.03 total 81.98 last 550.45) - open/close 45719 (time 1743454448.03 total 91.98 last 571.84) - open/close 50000 (time 1743454455.04 total 98.98 last 611.01) - open/close 55293 (time 1743454465.04 total 108.98 last 529.29) - open/close 60000 (time 1743454474.95 total 118.90 last 474.78) - open/close 64521 (time 1743454484.95 total 128.90 last 452.05) total: 64768 open/close in 129.45 seconds: 500.33 ops/second - /unlink 9625 (time 1743454496.31 total 10.00 last 962.46) - /unlink 19015 (time 1743454506.31 total 20.00 last 938.89) - /unlink 26783 (time 1743454516.31 total 30.00 last 776.76) - /unlink 30000 (time 1743454521.03 total 34.73 last 681.07) - /unlink 37163 (time 1743454531.03 total 44.73 last 716.28) - /unlink 40000 (time 1743454535.19 total 48.88 last 683.21) - /unlink 46605 (time 1743454545.19 total 58.88 last 660.47) - /unlink 50000 (time 1743454550.30 total 64.00 last 663.40) - /unlink 56468 (time 1743454560.31 total 74.00 last 646.64) - /unlink 60000 (time 1743454565.54 total 79.23 last 675.54) total: 64768 /unlink in 86.07 seconds: 752.50 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4704 (time 1743454598.82 total 10.00 last 470.36) - open/close 9292 (time 1743454608.82 total 20.00 last 458.77) - open/close 14196 (time 1743454618.82 total 30.00 last 490.34) - open/close 18825 (time 1743454628.82 total 40.00 last 462.82) - open/close 20000 (time 1743454631.41 total 42.60 last 453.12) - open/close 24412 (time 1743454641.42 total 52.60 last 441.13) - open/close 28697 (time 1743454651.42 total 62.60 last 428.43) - open/close 30000 (time 1743454654.53 total 65.71 last 418.73) - open/close 34387 (time 1743454664.53 total 75.71 last 438.65) - open/close 38523 (time 1743454674.53 total 85.71 last 413.55) - open/close 40000 (time 1743454677.97 total 89.16 last 429.18) - open/close 44407 (time 1743454687.98 total 99.16 last 440.58) - open/close 48531 (time 1743454697.98 total 109.16 last 412.27) - open/close 50000 (time 1743454701.53 total 112.72 last 413.34) - open/close 54415 (time 1743454711.53 total 122.72 last 441.44) - open/close 59091 (time 1743454721.54 total 132.72 last 467.50) - open/close 60000 (time 1743454723.55 total 134.73 last 452.25) - open/close 64509 (time 1743454733.55 total 144.73 last 450.89) total: 64768 open/close in 145.38 seconds: 445.50 ops/second - /unlink 7669 (time 1743454745.12 total 10.00 last 766.85) - /unlink 10000 (time 1743454748.34 total 13.22 last 723.50) - /unlink 17304 (time 1743454758.34 total 23.22 last 730.32) - /unlink 20000 (time 1743454761.98 total 26.86 last 740.77) - /unlink 27157 (time 1743454771.98 total 36.86 last 715.61) - /unlink 30000 (time 1743454775.81 total 40.69 last 743.30) - /unlink 37177 (time 1743454785.81 total 50.69 last 717.66) - /unlink 40000 (time 1743454789.63 total 54.51 last 739.03) - /unlink 47371 (time 1743454799.63 total 64.51 last 736.91) - /unlink 50000 (time 1743454803.35 total 68.24 last 705.90) - /unlink 57253 (time 1743454813.35 total 78.24 last 725.19) - /unlink 60000 (time 1743454817.19 total 82.08 last 715.47) total: 64768 /unlink in 88.53 seconds: 731.58 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4545 (time 1743454854.82 total 10.00 last 454.43) - open/close 9301 (time 1743454864.82 total 20.00 last 475.56) - open/close 13975 (time 1743454874.82 total 30.00 last 467.38) - open/close 19063 (time 1743454884.82 total 40.00 last 508.73) - open/close 24340 (time 1743454894.82 total 50.00 last 527.65) - open/close 29526 (time 1743454904.82 total 60.01 last 518.54) - open/close 34800 (time 1743454914.82 total 70.01 last 527.31) - open/close 39636 (time 1743454924.82 total 80.01 last 483.59) - open/close 44654 (time 1743454934.82 total 90.01 last 501.75) - open/close 49658 (time 1743454944.83 total 100.01 last 500.37) - open/close 54681 (time 1743454954.83 total 110.01 last 502.25) - open/close 59662 (time 1743454964.83 total 120.01 last 498.05) - open/close 64497 (time 1743454974.83 total 130.01 last 483.48) total: 64768 open/close in 130.62 seconds: 495.84 ops/second - /unlink 8427 (time 1743454986.30 total 10.00 last 842.61) - /unlink 16778 (time 1743454996.31 total 20.00 last 834.98) - /unlink 20000 (time 1743455000.26 total 23.96 last 815.07) - /unlink 28481 (time 1743455010.26 total 33.96 last 848.03) - /unlink 37426 (time 1743455020.26 total 43.96 last 894.43) - /unlink 40000 (time 1743455023.05 total 46.75 last 922.69) - /unlink 49111 (time 1743455033.05 total 56.75 last 911.07) - /unlink 57971 (time 1743455043.05 total 66.75 last 885.93) - /unlink 60000 (time 1743455045.28 total 68.98 last 910.65) total: 64768 /unlink in 74.08 seconds: 874.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5435 (time 1743455085.50 total 10.00 last 543.43) - open/close 10000 (time 1743455094.03 total 18.53 last 535.46) - open/close 14749 (time 1743455104.03 total 28.53 last 474.86) - open/close 19606 (time 1743455114.03 total 38.53 last 485.62) - open/close 24684 (time 1743455124.05 total 48.55 last 506.69) - open/close 29822 (time 1743455134.05 total 58.55 last 513.78) - open/close 34853 (time 1743455144.05 total 68.55 last 503.02) - open/close 38764 (time 1743455154.06 total 78.56 last 390.99) - open/close 40000 (time 1743455156.62 total 81.12 last 482.80) - open/close 43670 (time 1743455166.62 total 91.12 last 366.88) - open/close 48579 (time 1743455176.62 total 101.12 last 490.85) - open/close 50000 (time 1743455179.55 total 104.05 last 484.81) - open/close 54994 (time 1743455189.55 total 114.05 last 499.31) - open/close 60000 (time 1743455199.34 total 123.84 last 511.52) total: 64768 open/close in 132.91 seconds: 487.32 ops/second - /unlink 7199 (time 1743455219.28 total 10.00 last 719.88) - /unlink 10000 (time 1743455222.54 total 13.26 last 858.95) - /unlink 19111 (time 1743455232.54 total 23.26 last 911.03) - /unlink 27042 (time 1743455242.54 total 33.26 last 793.04) - /unlink 30000 (time 1743455246.93 total 37.64 last 675.03) - /unlink 38018 (time 1743455256.93 total 47.65 last 801.70) - /unlink 40000 (time 1743455259.58 total 50.29 last 748.60) - /unlink 48670 (time 1743455269.58 total 60.29 last 866.93) - /unlink 57045 (time 1743455279.58 total 70.30 last 837.41) - /unlink 60000 (time 1743455283.38 total 74.09 last 777.85) total: 64768 /unlink in 79.28 seconds: 816.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4832 (time 1743455319.72 total 10.00 last 483.15) - open/close 9609 (time 1743455329.72 total 20.00 last 477.63) - open/close 14291 (time 1743455339.72 total 30.00 last 468.16) - open/close 19091 (time 1743455349.72 total 40.00 last 479.93) - open/close 24011 (time 1743455359.73 total 50.01 last 491.97) - open/close 28601 (time 1743455369.73 total 60.01 last 458.98) - open/close 30000 (time 1743455372.84 total 63.12 last 448.54) - open/close 34832 (time 1743455382.85 total 73.13 last 483.11) - open/close 38242 (time 1743455392.85 total 83.13 last 340.94) - open/close 40000 (time 1743455396.67 total 86.95 last 459.55) - open/close 43672 (time 1743455406.68 total 96.96 last 367.15) - open/close 48130 (time 1743455416.68 total 106.96 last 445.64) - open/close 50000 (time 1743455422.42 total 112.70 last 325.87) - open/close 54848 (time 1743455432.42 total 122.70 last 484.74) - open/close 59787 (time 1743455442.42 total 132.70 last 493.84) - open/close 64530 (time 1743455452.42 total 142.70 last 474.25) total: 64768 open/close in 143.16 seconds: 452.42 ops/second - /unlink 9356 (time 1743455463.69 total 10.00 last 935.52) - /unlink 17738 (time 1743455473.69 total 20.00 last 838.12) - /unlink 20000 (time 1743455476.38 total 22.69 last 840.15) - /unlink 27318 (time 1743455486.38 total 32.69 last 731.76) - /unlink 30000 (time 1743455489.65 total 35.97 last 818.90) - /unlink 38907 (time 1743455499.66 total 45.97 last 890.66) - /unlink 47650 (time 1743455509.66 total 55.97 last 874.26) - /unlink 50000 (time 1743455512.77 total 59.08 last 755.40) - /unlink 56424 (time 1743455522.77 total 69.08 last 642.33) - /unlink 60000 (time 1743455526.31 total 72.62 last 1010.85) total: 64768 /unlink in 77.37 seconds: 837.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server unloading modules via unload_modules_local on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2