== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:49:49 (1746791389) Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) oleg420-server: oleg420-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg420-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg420-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 279 79713 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 257 79735 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65976 536 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5800 (time 1746791429.06 total 10.00 last 579.94) - open/close 10000 (time 1746791436.36 total 17.30 last 575.17) - open/close 15779 (time 1746791446.36 total 27.30 last 577.81) - open/close 20000 (time 1746791453.72 total 34.66 last 573.50) - open/close 25646 (time 1746791463.73 total 44.67 last 564.53) - open/close 30000 (time 1746791471.42 total 52.36 last 566.24) - open/close 35538 (time 1746791481.42 total 62.36 last 553.69) - open/close 40000 (time 1746791489.19 total 70.13 last 574.16) - open/close 45619 (time 1746791499.19 total 80.13 last 561.88) - open/close 50000 (time 1746791507.28 total 88.22 last 541.21) - open/close 55646 (time 1746791517.29 total 98.23 last 564.51) - open/close 60000 (time 1746791526.84 total 107.78 last 455.52) total: 64768 open/close in 116.98 seconds: 553.68 ops/second - /unlink 9276 (time 1746791546.91 total 10.00 last 927.49) - /unlink 17371 (time 1746791556.91 total 20.00 last 809.46) - /unlink 20000 (time 1746791560.75 total 23.84 last 684.33) - /unlink 29897 (time 1746791570.75 total 33.84 last 989.63) - /unlink 39773 (time 1746791580.75 total 43.84 last 987.58) - /unlink 49664 (time 1746791590.75 total 53.84 last 989.04) - /unlink 59908 (time 1746791600.75 total 63.85 last 1024.30) total: 64768 /unlink in 68.50 seconds: 945.57 ops/second Waiting for MDT destroys to complete - open/close 5822 (time 1746791712.73 total 10.00 last 582.09) - open/close 10000 (time 1746791720.29 total 17.57 last 552.13) - open/close 15553 (time 1746791730.29 total 27.57 last 555.29) - open/close 20000 (time 1746791738.84 total 36.12 last 520.30) - open/close 25583 (time 1746791748.84 total 46.12 last 558.28) - open/close 30000 (time 1746791756.82 total 54.09 last 553.64) - open/close 35541 (time 1746791766.82 total 64.09 last 554.09) - open/close 40000 (time 1746791774.68 total 71.95 last 567.40) - open/close 45142 (time 1746791784.68 total 81.95 last 514.13) - open/close 49484 (time 1746791794.68 total 91.96 last 434.07) - open/close 53777 (time 1746791804.68 total 101.96 last 429.22) - open/close 57554 (time 1746791814.68 total 111.96 last 377.68) - open/close 60000 (time 1746791821.15 total 118.42 last 378.39) total: 64768 open/close in 127.25 seconds: 508.99 ops/second - /unlink 9123 (time 1746791840.83 total 10.00 last 912.24) - /unlink 18219 (time 1746791850.83 total 20.00 last 909.44) - /unlink 20000 (time 1746791853.09 total 22.26 last 788.38) - /unlink 29569 (time 1746791863.09 total 32.26 last 956.88) - /unlink 39528 (time 1746791873.09 total 42.26 last 995.87) - /unlink 48982 (time 1746791883.09 total 52.26 last 945.34) - /unlink 57567 (time 1746791893.09 total 62.26 last 858.42) - /unlink 60000 (time 1746791896.83 total 66.00 last 650.33) total: 64768 /unlink in 73.71 seconds: 878.71 ops/second Waiting for MDT destroys to complete - open/close 5776 (time 1746792003.10 total 10.00 last 577.59) - open/close 10000 (time 1746792010.40 total 17.30 last 578.39) - open/close 15116 (time 1746792020.40 total 27.30 last 511.54) - open/close 20000 (time 1746792029.18 total 36.08 last 556.84) - open/close 24755 (time 1746792039.18 total 46.08 last 475.44) - open/close 28227 (time 1746792049.18 total 56.08 last 347.18) - open/close 30000 (time 1746792053.61 total 60.51 last 400.19) - open/close 33903 (time 1746792063.61 total 70.51 last 390.30) - open/close 38706 (time 1746792073.61 total 80.51 last 480.27) - open/close 40000 (time 1746792076.38 total 83.28 last 466.66) - open/close 44981 (time 1746792086.38 total 93.28 last 498.06) - open/close 49487 (time 1746792096.38 total 103.28 last 450.55) - open/close 54725 (time 1746792106.38 total 113.28 last 523.73) - open/close 59772 (time 1746792116.38 total 123.29 last 504.66) total: 64768 open/close in 132.34 seconds: 489.42 ops/second - /unlink 9504 (time 1746792136.31 total 10.00 last 950.35) - /unlink 19099 (time 1746792146.31 total 20.00 last 959.46) - /unlink 29253 (time 1746792156.31 total 30.00 last 1015.33) - /unlink 36333 (time 1746792166.31 total 40.00 last 707.99) - /unlink 40000 (time 1746792170.42 total 44.11 last 892.59) - /unlink 49747 (time 1746792180.42 total 54.11 last 974.68) - /unlink 58776 (time 1746792190.42 total 64.11 last 902.89) total: 64768 /unlink in 70.35 seconds: 920.64 ops/second Waiting for MDT destroys to complete - open/close 3741 (time 1746792297.82 total 10.00 last 374.04) - open/close 7852 (time 1746792307.82 total 20.00 last 411.07) - open/close 10000 (time 1746792311.69 total 23.87 last 555.65) - open/close 15007 (time 1746792321.69 total 33.87 last 500.68) - open/close 20000 (time 1746792330.94 total 43.11 last 539.99) - open/close 24229 (time 1746792340.94 total 53.12 last 422.82) - open/close 29089 (time 1746792350.94 total 63.12 last 485.99) - open/close 30000 (time 1746792353.06 total 65.24 last 429.95) - open/close 34286 (time 1746792363.06 total 75.24 last 428.54) - open/close 38236 (time 1746792373.06 total 85.24 last 394.96) - open/close 40000 (time 1746792378.01 total 90.18 last 356.61) - open/close 44659 (time 1746792388.01 total 100.19 last 465.87) - open/close 48399 (time 1746792398.01 total 110.19 last 373.97) - open/close 50000 (time 1746792401.57 total 113.75 last 449.11) - open/close 53283 (time 1746792411.57 total 123.75 last 328.25) - open/close 56301 (time 1746792421.58 total 133.76 last 301.69) - open/close 59454 (time 1746792431.58 total 143.76 last 315.28) - open/close 62253 (time 1746792441.58 total 153.76 last 279.83) total: 64768 open/close in 161.69 seconds: 400.58 ops/second - /unlink 5816 (time 1746792460.64 total 10.00 last 581.50) - /unlink 10000 (time 1746792466.93 total 16.30 last 664.29) - /unlink 16482 (time 1746792476.93 total 26.30 last 648.15) - /unlink 20000 (time 1746792482.41 total 31.77 last 642.84) - /unlink 26633 (time 1746792492.41 total 41.77 last 663.26) - /unlink 30000 (time 1746792497.97 total 47.34 last 604.86) - /unlink 37693 (time 1746792507.98 total 57.34 last 769.24) - /unlink 40000 (time 1746792511.42 total 60.79 last 669.00) - /unlink 48800 (time 1746792521.42 total 70.79 last 879.98) - /unlink 57554 (time 1746792531.42 total 80.79 last 875.35) - /unlink 60000 (time 1746792533.83 total 83.20 last 1016.79) total: 64768 /unlink in 88.19 seconds: 734.44 ops/second Waiting for MDT destroys to complete - open/close 4048 (time 1746792594.33 total 10.00 last 404.77) - open/close 8365 (time 1746792604.33 total 20.00 last 431.63) - open/close 10000 (time 1746792607.57 total 23.23 last 505.94) - open/close 14152 (time 1746792617.57 total 33.24 last 415.13) - open/close 19536 (time 1746792627.57 total 43.24 last 538.33) - open/close 24746 (time 1746792637.57 total 53.24 last 520.98) - open/close 30000 (time 1746792647.02 total 62.69 last 555.84) - open/close 35455 (time 1746792657.02 total 72.69 last 545.44) - open/close 40000 (time 1746792665.36 total 81.03 last 544.86) - open/close 45538 (time 1746792675.37 total 91.03 last 553.72) - open/close 50000 (time 1746792683.67 total 99.33 last 537.58) - open/close 55451 (time 1746792693.67 total 109.33 last 545.05) - open/close 60000 (time 1746792702.89 total 118.56 last 493.26) total: 64768 open/close in 128.53 seconds: 503.92 ops/second - /unlink 7580 (time 1746792724.02 total 10.00 last 758.00) - /unlink 10000 (time 1746792727.13 total 13.12 last 776.49) - /unlink 18079 (time 1746792737.13 total 23.12 last 807.76) - /unlink 20000 (time 1746792740.24 total 26.22 last 618.70) - /unlink 26041 (time 1746792750.24 total 36.22 last 604.05) - /unlink 30000 (time 1746792754.69 total 40.67 last 890.13) - /unlink 35700 (time 1746792764.69 total 50.67 last 569.99) - /unlink 40000 (time 1746792771.40 total 57.38 last 640.56) - /unlink 48417 (time 1746792781.40 total 67.39 last 841.65) - /unlink 50000 (time 1746792784.22 total 70.21 last 561.26) - /unlink 57708 (time 1746792794.22 total 80.21 last 770.65) - /unlink 60000 (time 1746792796.84 total 82.83 last 874.84) total: 64768 /unlink in 87.69 seconds: 738.62 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server unloading modules via unload_modules_local on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2