== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:40:29 (1742902829) Stopping clients: oleg335-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg335-client.virtnet /mnt/lustre2 (opts:-f) oleg335-server: oleg335-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing load_modules_local oleg335-server: Loading modules from /home/green/git/lustre-release/lustre oleg335-server: detected 4 online CPUs by sysfs oleg335-server: MODOPTS_LIBCFS= oleg335-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg335-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg335-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg335-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg335-server: oleg335-server.virtnet: executing set_default_debug -1 all pdsh@oleg335-client: oleg335-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg335-client.virtnet: -o user_xattr,flock 192.168.203.135@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3361 (time 1742902868.30 total 10.00 last 336.08) - open/close 6359 (time 1742902878.30 total 20.00 last 299.77) - open/close 8182 (time 1742902888.30 total 30.00 last 182.30) - open/close 10000 (time 1742902897.92 total 39.63 last 188.92) - open/close 12283 (time 1742902907.93 total 49.63 last 228.23) - open/close 15027 (time 1742902917.93 total 59.63 last 274.34) - open/close 17763 (time 1742902927.93 total 69.63 last 273.59) - open/close 20000 (time 1742902935.57 total 77.27 last 292.77) - open/close 23037 (time 1742902945.57 total 87.27 last 303.65) - open/close 26552 (time 1742902955.57 total 97.27 last 351.48) - open/close 29393 (time 1742902965.58 total 107.28 last 283.99) - open/close 30000 (time 1742902968.04 total 109.74 last 246.80) - open/close 32702 (time 1742902978.04 total 119.74 last 270.13) - open/close 34614 (time 1742902988.04 total 129.74 last 191.16) - open/close 36446 (time 1742902998.04 total 139.75 last 183.14) - open/close 38711 (time 1742903008.04 total 149.75 last 226.48) - open/close 40000 (time 1742903013.72 total 155.42 last 227.23) - open/close 42649 (time 1742903023.72 total 165.42 last 264.81) - open/close 45064 (time 1742903033.72 total 175.43 last 241.40) - open/close 47362 (time 1742903043.73 total 185.43 last 229.76) - open/close 49103 (time 1742903053.73 total 195.44 last 173.97) - open/close 50000 (time 1742903058.40 total 200.10 last 192.38) - open/close 52722 (time 1742903068.40 total 210.10 last 272.11) - open/close 55344 (time 1742903078.40 total 220.11 last 262.10) - open/close 57448 (time 1742903088.41 total 230.11 last 210.38) - open/close 59425 (time 1742903098.41 total 240.11 last 197.66) - open/close 62016 (time 1742903108.41 total 250.11 last 259.09) - open/close 63890 (time 1742903118.41 total 260.11 last 187.40) total: 64768 open/close in 265.34 seconds: 244.09 ops/second - /unlink 3067 (time 1742903134.98 total 10.00 last 306.63) - /unlink 6335 (time 1742903144.98 total 20.00 last 326.76) - /unlink 10000 (time 1742903154.38 total 29.40 last 389.98) - /unlink 14301 (time 1742903164.38 total 39.40 last 430.07) - /unlink 18115 (time 1742903174.38 total 49.40 last 381.35) - /unlink 20000 (time 1742903179.35 total 54.38 last 379.06) - /unlink 24403 (time 1742903189.35 total 64.38 last 440.21) - /unlink 27923 (time 1742903199.36 total 74.38 last 351.89) - /unlink 30000 (time 1742903204.81 total 79.83 last 381.02) - /unlink 34474 (time 1742903214.81 total 89.83 last 447.33) - /unlink 38099 (time 1742903224.81 total 99.84 last 362.34) - /unlink 40000 (time 1742903230.26 total 105.28 last 349.23) - /unlink 43585 (time 1742903240.26 total 115.28 last 358.50) - /unlink 46694 (time 1742903250.26 total 125.28 last 310.88) - /unlink 50000 (time 1742903259.02 total 134.05 last 377.10) - /unlink 53359 (time 1742903269.02 total 144.05 last 335.87) - /unlink 57661 (time 1742903279.03 total 154.05 last 430.10) - /unlink 60000 (time 1742903283.05 total 158.08 last 581.12) total: 64768 /unlink in 167.16 seconds: 387.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3137 (time 1742903317.57 total 10.00 last 313.59) - open/close 6073 (time 1742903327.57 total 20.00 last 293.56) - open/close 9793 (time 1742903337.57 total 30.01 last 371.95) - open/close 13608 (time 1742903347.57 total 40.01 last 381.45) - open/close 18105 (time 1742903357.57 total 50.01 last 449.60) - open/close 20000 (time 1742903361.75 total 54.19 last 453.16) - open/close 24045 (time 1742903371.76 total 64.19 last 404.44) - open/close 28093 (time 1742903381.76 total 74.19 last 404.75) - open/close 30000 (time 1742903386.40 total 78.83 last 410.89) - open/close 34899 (time 1742903396.40 total 88.83 last 489.90) - open/close 40000 (time 1742903406.03 total 98.47 last 529.46) - open/close 45115 (time 1742903416.03 total 108.47 last 511.45) - open/close 49569 (time 1742903426.03 total 118.47 last 445.39) - open/close 53567 (time 1742903436.04 total 128.47 last 399.76) - open/close 57079 (time 1742903446.04 total 138.47 last 351.14) - open/close 60000 (time 1742903454.45 total 146.88 last 347.36) - open/close 63745 (time 1742903464.45 total 156.88 last 374.41) total: 64768 open/close in 160.02 seconds: 404.74 ops/second - /unlink 7744 (time 1742903478.43 total 10.00 last 774.32) - /unlink 10000 (time 1742903481.69 total 13.26 last 691.64) - /unlink 16974 (time 1742903491.69 total 23.26 last 697.36) - /unlink 20000 (time 1742903495.49 total 27.06 last 796.35) - /unlink 26738 (time 1742903505.49 total 37.06 last 673.71) - /unlink 30000 (time 1742903510.70 total 42.27 last 627.03) - /unlink 36246 (time 1742903520.70 total 52.27 last 624.55) - /unlink 40000 (time 1742903526.36 total 57.94 last 662.25) - /unlink 46258 (time 1742903536.36 total 67.94 last 625.79) - /unlink 50000 (time 1742903542.23 total 73.80 last 637.91) - /unlink 56904 (time 1742903552.23 total 83.80 last 690.31) - /unlink 60000 (time 1742903557.04 total 88.61 last 644.13) total: 64768 /unlink in 95.85 seconds: 675.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4202 (time 1742903592.57 total 10.00 last 420.12) - open/close 8379 (time 1742903602.57 total 20.00 last 417.62) - open/close 10000 (time 1742903606.71 total 24.15 last 391.30) - open/close 14674 (time 1742903616.72 total 34.15 last 467.36) - open/close 19564 (time 1742903626.72 total 44.15 last 488.97) - open/close 24144 (time 1742903636.72 total 54.15 last 457.99) - open/close 29865 (time 1742903646.72 total 64.15 last 572.04) - open/close 35396 (time 1742903656.72 total 74.15 last 553.08) - open/close 40000 (time 1742903664.98 total 82.41 last 557.01) - open/close 44807 (time 1742903674.99 total 92.42 last 480.53) - open/close 49948 (time 1742903684.99 total 102.42 last 514.07) - open/close 55248 (time 1742903694.99 total 112.42 last 529.96) - open/close 60000 (time 1742903704.75 total 122.19 last 486.61) - open/close 64630 (time 1742903714.78 total 132.22 last 461.59) total: 64768 open/close in 132.50 seconds: 488.81 ops/second - /unlink 9895 (time 1742903725.93 total 10.00 last 989.42) - /unlink 19528 (time 1742903735.93 total 20.00 last 963.25) - /unlink 28368 (time 1742903745.93 total 30.00 last 883.93) - /unlink 37761 (time 1742903755.93 total 40.00 last 939.22) - /unlink 40000 (time 1742903758.11 total 42.18 last 1027.34) - /unlink 50000 (time 1742903767.53 total 51.61 last 1060.91) - /unlink 58131 (time 1742903777.53 total 61.61 last 813.06) - /unlink 60000 (time 1742903780.05 total 64.12 last 742.83) total: 64768 /unlink in 69.00 seconds: 938.67 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5170 (time 1742903827.78 total 10.00 last 516.99) - open/close 9802 (time 1742903837.78 total 20.00 last 463.17) - open/close 15053 (time 1742903847.78 total 30.00 last 525.04) - open/close 20000 (time 1742903857.64 total 39.86 last 501.89) - open/close 26068 (time 1742903867.64 total 49.86 last 606.72) - open/close 30000 (time 1742903875.56 total 57.77 last 496.85) - open/close 35567 (time 1742903885.56 total 67.78 last 556.63) - open/close 40000 (time 1742903892.89 total 75.11 last 604.25) - open/close 45389 (time 1742903902.89 total 85.11 last 538.86) - open/close 50000 (time 1742903912.20 total 94.42 last 495.49) - open/close 55165 (time 1742903922.20 total 104.42 last 516.46) - open/close 60000 (time 1742903931.67 total 113.89 last 510.48) total: 64768 open/close in 123.34 seconds: 525.12 ops/second - /unlink 9190 (time 1742903951.96 total 10.00 last 918.97) - /unlink 18329 (time 1742903961.96 total 20.00 last 913.84) - /unlink 27650 (time 1742903971.96 total 30.00 last 932.02) - /unlink 30000 (time 1742903974.41 total 32.46 last 957.72) - /unlink 38433 (time 1742903984.41 total 42.46 last 843.21) - /unlink 46318 (time 1742903994.42 total 52.46 last 788.41) - /unlink 50000 (time 1742903998.92 total 56.96 last 818.11) - /unlink 58685 (time 1742904008.92 total 66.96 last 868.49) total: 64768 /unlink in 73.47 seconds: 881.55 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4940 (time 1742904053.21 total 10.00 last 493.94) - open/close 10000 (time 1742904063.00 total 19.79 last 516.85) - open/close 15004 (time 1742904073.00 total 29.79 last 500.33) - open/close 19811 (time 1742904083.00 total 39.79 last 480.66) - open/close 24460 (time 1742904093.01 total 49.80 last 464.81) - open/close 29283 (time 1742904103.01 total 59.80 last 482.20) - open/close 34570 (time 1742904113.01 total 69.80 last 528.64) - open/close 39909 (time 1742904123.01 total 79.80 last 533.90) - open/close 45175 (time 1742904133.01 total 89.80 last 526.57) - open/close 50000 (time 1742904142.42 total 99.21 last 512.64) - open/close 54826 (time 1742904152.42 total 109.21 last 482.57) - open/close 59974 (time 1742904162.42 total 119.21 last 514.78) total: 64768 open/close in 129.07 seconds: 501.80 ops/second - /unlink 8784 (time 1742904183.14 total 10.00 last 878.34) - /unlink 17687 (time 1742904193.14 total 20.00 last 890.26) - /unlink 20000 (time 1742904195.81 total 22.67 last 868.06) - /unlink 28264 (time 1742904205.81 total 32.67 last 826.21) - /unlink 30000 (time 1742904208.02 total 34.88 last 784.91) - /unlink 38546 (time 1742904218.02 total 44.88 last 854.54) - /unlink 47210 (time 1742904228.03 total 54.88 last 866.31) - /unlink 50000 (time 1742904231.05 total 57.91 last 922.46) - /unlink 59293 (time 1742904241.05 total 67.91 last 929.29) total: 64768 /unlink in 73.79 seconds: 877.72 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg335-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg335-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg335-server stop mds service on oleg335-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg335-server unloading modules via unload_modules_local on: 'oleg335-server' oleg335-server: oleg335-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-client: ssh exited with exit code 2 pdsh@oleg335-client: oleg335-server: ssh exited with exit code 2