== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:08:22 (1746112102) Stopping clients: oleg437-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg437-client.virtnet /mnt/lustre2 (opts:-f) oleg437-server: oleg437-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing load_modules_local oleg437-server: Loading modules from /home/green/git/lustre-release/lustre oleg437-server: detected 4 online CPUs by sysfs oleg437-server: MODOPTS_LIBCFS= oleg437-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg437-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg437-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg437-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg437-server: oleg437-server.virtnet: executing set_default_debug -1 all pdsh@oleg437-client: oleg437-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg437-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg437-client.virtnet: -o user_xattr,flock 192.168.204.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4341 (time 1746112136.48 total 10.00 last 434.07) - open/close 9544 (time 1746112146.48 total 20.00 last 520.27) - open/close 14933 (time 1746112156.48 total 30.00 last 538.85) - open/close 20000 (time 1746112166.37 total 39.90 last 512.18) - open/close 25193 (time 1746112176.37 total 49.90 last 519.29) - open/close 30000 (time 1746112185.54 total 59.07 last 524.12) - open/close 34949 (time 1746112195.55 total 69.07 last 494.82) - open/close 40000 (time 1746112205.54 total 79.06 last 505.63) - open/close 44775 (time 1746112215.54 total 89.06 last 477.47) - open/close 48900 (time 1746112225.54 total 99.06 last 412.50) - open/close 50000 (time 1746112227.87 total 101.39 last 471.41) - open/close 53879 (time 1746112237.87 total 111.39 last 387.86) - open/close 58585 (time 1746112247.87 total 121.39 last 470.52) - open/close 60000 (time 1746112250.83 total 124.35 last 478.65) - open/close 64419 (time 1746112260.83 total 134.35 last 441.89) total: 64768 open/close in 135.11 seconds: 479.38 ops/second - /unlink 7288 (time 1746112272.42 total 10.00 last 728.79) - /unlink 10000 (time 1746112276.58 total 14.17 last 651.12) - /unlink 18051 (time 1746112286.58 total 24.17 last 805.01) - /unlink 20000 (time 1746112289.19 total 26.78 last 746.34) - /unlink 26616 (time 1746112299.20 total 36.78 last 661.55) - /unlink 30000 (time 1746112302.94 total 40.52 last 903.28) - /unlink 38890 (time 1746112312.94 total 50.53 last 888.96) - /unlink 47411 (time 1746112322.94 total 60.53 last 852.09) - /unlink 50000 (time 1746112325.90 total 63.48 last 876.60) - /unlink 59221 (time 1746112335.90 total 73.48 last 922.01) total: 64768 /unlink in 79.38 seconds: 815.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5332 (time 1746112375.32 total 10.00 last 533.11) - open/close 10000 (time 1746112384.22 total 18.90 last 524.49) - open/close 14596 (time 1746112394.22 total 28.90 last 459.55) - open/close 19326 (time 1746112404.22 total 38.90 last 472.99) - open/close 23450 (time 1746112414.22 total 48.90 last 412.33) - open/close 27495 (time 1746112424.22 total 58.90 last 404.48) - open/close 30000 (time 1746112430.16 total 64.85 last 421.60) - open/close 34897 (time 1746112440.17 total 74.85 last 489.63) - open/close 39507 (time 1746112450.17 total 84.85 last 460.98) - open/close 43375 (time 1746112460.17 total 94.85 last 386.78) - open/close 48470 (time 1746112470.17 total 104.85 last 509.49) - open/close 50000 (time 1746112473.17 total 107.85 last 510.07) - open/close 55226 (time 1746112483.17 total 117.85 last 522.57) - open/close 60000 (time 1746112492.34 total 127.02 last 520.60) total: 64768 open/close in 136.14 seconds: 475.76 ops/second - /unlink 9381 (time 1746112512.28 total 10.00 last 938.09) - /unlink 18626 (time 1746112522.28 total 20.00 last 924.49) - /unlink 27544 (time 1746112532.28 total 30.00 last 891.71) - /unlink 30000 (time 1746112535.04 total 32.77 last 887.55) - /unlink 37591 (time 1746112545.04 total 42.77 last 759.08) - /unlink 40000 (time 1746112547.56 total 45.29 last 955.88) - /unlink 49148 (time 1746112557.56 total 55.29 last 914.76) - /unlink 55391 (time 1746112567.57 total 65.29 last 624.27) - /unlink 60000 (time 1746112576.24 total 73.96 last 531.57) total: 64768 /unlink in 80.06 seconds: 808.98 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2882 (time 1746112609.70 total 10.00 last 288.19) - open/close 6388 (time 1746112619.70 total 20.00 last 350.51) - open/close 9318 (time 1746112629.70 total 30.00 last 293.00) - open/close 10000 (time 1746112632.28 total 32.59 last 264.11) - open/close 12818 (time 1746112642.28 total 42.59 last 281.75) - open/close 15847 (time 1746112652.28 total 52.59 last 302.90) - open/close 19010 (time 1746112662.28 total 62.59 last 316.27) - open/close 20000 (time 1746112664.43 total 64.73 last 462.24) - open/close 24343 (time 1746112674.43 total 74.73 last 434.29) - open/close 27939 (time 1746112684.43 total 84.73 last 359.59) - open/close 30000 (time 1746112690.81 total 91.11 last 322.86) - open/close 34005 (time 1746112700.81 total 101.11 last 400.48) - open/close 38960 (time 1746112710.81 total 111.12 last 495.48) - open/close 40000 (time 1746112713.15 total 113.45 last 445.02) - open/close 43177 (time 1746112723.15 total 123.46 last 317.59) - open/close 45815 (time 1746112733.15 total 133.46 last 263.74) - open/close 49106 (time 1746112743.15 total 143.46 last 329.09) - open/close 50000 (time 1746112745.48 total 145.78 last 384.24) - open/close 54321 (time 1746112755.48 total 155.79 last 432.04) - open/close 57933 (time 1746112765.48 total 165.79 last 361.15) - open/close 60000 (time 1746112772.00 total 172.31 last 317.12) total: 64768 open/close in 182.05 seconds: 355.78 ops/second - /unlink 7759 (time 1746112792.54 total 10.00 last 775.87) - /unlink 10000 (time 1746112796.17 total 13.63 last 616.89) - /unlink 15798 (time 1746112806.17 total 23.63 last 579.75) - /unlink 20000 (time 1746112813.24 total 30.71 last 594.21) - /unlink 25354 (time 1746112823.25 total 40.71 last 535.37) - /unlink 30000 (time 1746112832.47 total 49.93 last 503.77) - /unlink 35479 (time 1746112842.47 total 59.93 last 547.82) - /unlink 40000 (time 1746112850.48 total 67.94 last 564.37) - /unlink 45523 (time 1746112860.48 total 77.94 last 552.25) - /unlink 50000 (time 1746112868.51 total 85.97 last 557.87) - /unlink 55665 (time 1746112878.51 total 95.97 last 566.42) - /unlink 60000 (time 1746112886.61 total 104.07 last 535.01) total: 64768 /unlink in 111.97 seconds: 578.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2765 (time 1746112921.46 total 10.00 last 276.46) - open/close 5349 (time 1746112931.46 total 20.01 last 258.30) - open/close 7915 (time 1746112941.46 total 30.01 last 256.60) - open/close 10000 (time 1746112949.54 total 38.08 last 258.10) - open/close 12562 (time 1746112959.54 total 48.08 last 256.18) - open/close 15067 (time 1746112969.54 total 58.08 last 250.50) - open/close 17665 (time 1746112979.55 total 68.09 last 259.77) - open/close 20000 (time 1746112987.94 total 76.48 last 278.07) - open/close 22865 (time 1746112997.95 total 86.49 last 286.41) - open/close 25559 (time 1746113007.95 total 96.49 last 269.35) - open/close 28632 (time 1746113017.95 total 106.49 last 307.20) - open/close 30000 (time 1746113023.24 total 111.78 last 258.82) - open/close 32536 (time 1746113033.24 total 121.78 last 253.58) - open/close 35027 (time 1746113043.24 total 131.78 last 249.07) - open/close 37550 (time 1746113053.24 total 141.78 last 252.30) - open/close 40000 (time 1746113062.53 total 151.07 last 263.70) - open/close 42500 (time 1746113072.53 total 161.07 last 249.93) - open/close 44992 (time 1746113082.53 total 171.07 last 249.15) - open/close 47448 (time 1746113092.54 total 181.08 last 245.54) - open/close 49956 (time 1746113102.54 total 191.08 last 250.78) - open/close 52472 (time 1746113112.54 total 201.08 last 251.59) - open/close 55137 (time 1746113122.54 total 211.08 last 266.49) - open/close 57657 (time 1746113132.54 total 221.08 last 251.98) - open/close 60000 (time 1746113140.93 total 229.47 last 279.12) - open/close 62693 (time 1746113150.94 total 239.48 last 269.19) total: 64768 open/close in 247.83 seconds: 261.34 ops/second - /unlink 4841 (time 1746113170.55 total 10.00 last 484.05) - /unlink 9874 (time 1746113180.55 total 20.00 last 503.23) - /unlink 14752 (time 1746113190.55 total 30.00 last 487.78) - /unlink 19945 (time 1746113200.56 total 40.00 last 519.27) - /unlink 24833 (time 1746113210.56 total 50.00 last 488.79) - /unlink 29748 (time 1746113220.56 total 60.00 last 491.49) - /unlink 34969 (time 1746113230.56 total 70.00 last 522.08) - /unlink 40000 (time 1746113239.53 total 78.98 last 560.41) - /unlink 45056 (time 1746113249.53 total 88.98 last 505.56) - /unlink 49834 (time 1746113259.53 total 98.98 last 477.78) - /unlink 54697 (time 1746113269.54 total 108.98 last 486.26) - /unlink 59737 (time 1746113279.54 total 118.98 last 503.95) - /unlink 64744 (time 1746113289.54 total 128.99 last 500.59) total: 64768 /unlink in 129.04 seconds: 501.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2510 (time 1746113317.10 total 10.00 last 250.92) - open/close 5111 (time 1746113327.10 total 20.01 last 260.05) - open/close 7681 (time 1746113337.11 total 30.01 last 256.96) - open/close 10000 (time 1746113346.22 total 39.12 last 254.42) - open/close 12558 (time 1746113356.22 total 49.12 last 255.75) - open/close 15206 (time 1746113366.22 total 59.13 last 264.75) - open/close 17901 (time 1746113376.23 total 69.13 last 269.48) - open/close 20000 (time 1746113384.07 total 76.97 last 267.43) - open/close 22758 (time 1746113394.07 total 86.98 last 275.79) - open/close 25518 (time 1746113404.08 total 96.98 last 276.00) - open/close 28339 (time 1746113414.08 total 106.98 last 282.08) - open/close 30000 (time 1746113419.44 total 112.34 last 309.77) - open/close 33303 (time 1746113429.44 total 122.34 last 330.29) - open/close 36490 (time 1746113439.44 total 132.34 last 318.68) - open/close 39354 (time 1746113449.44 total 142.34 last 286.31) - open/close 40000 (time 1746113451.84 total 144.74 last 269.68) - open/close 42744 (time 1746113461.84 total 154.74 last 274.37) - open/close 45425 (time 1746113471.84 total 164.74 last 268.08) - open/close 47997 (time 1746113481.84 total 174.74 last 257.10) - open/close 50000 (time 1746113489.97 total 182.87 last 246.46) - open/close 52476 (time 1746113499.97 total 192.87 last 247.55) - open/close 54966 (time 1746113509.97 total 202.87 last 248.98) - open/close 57484 (time 1746113519.98 total 212.88 last 251.70) - open/close 59917 (time 1746113529.98 total 222.88 last 243.23) - open/close 62420 (time 1746113539.98 total 232.88 last 250.24) total: 64768 open/close in 241.96 seconds: 267.69 ops/second - /unlink 5246 (time 1746113560.35 total 10.00 last 524.56) - /unlink 10000 (time 1746113570.24 total 19.90 last 480.47) - /unlink 15020 (time 1746113580.24 total 29.90 last 501.92) - /unlink 20000 (time 1746113587.05 total 36.71 last 731.25) - /unlink 28531 (time 1746113597.05 total 46.71 last 853.02) - /unlink 38113 (time 1746113607.06 total 56.71 last 958.14) - /unlink 40000 (time 1746113609.12 total 58.77 last 915.48) - /unlink 49102 (time 1746113619.12 total 68.77 last 910.15) - /unlink 58046 (time 1746113629.12 total 78.77 last 894.38) - /unlink 60000 (time 1746113631.27 total 80.93 last 906.00) total: 64768 /unlink in 86.00 seconds: 753.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg437-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg437-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg437-server stop mds service on oleg437-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg437-server unloading modules via unload_modules_local on: 'oleg437-server' oleg437-server: oleg437-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-client: ssh exited with exit code 2 pdsh@oleg437-client: oleg437-server: ssh exited with exit code 2