== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:10:27 (1741626627) Stopping clients: oleg255-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg255-client.virtnet /mnt/lustre2 (opts:-f) oleg255-server: oleg255-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing load_modules_local oleg255-server: Loading modules from /home/green/git/lustre-release/lustre oleg255-server: detected 4 online CPUs by sysfs oleg255-server: MODOPTS_LIBCFS= oleg255-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg255-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg255-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg255-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg255-server: oleg255-server.virtnet: executing set_default_debug -1 all pdsh@oleg255-client: oleg255-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg255-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg255-client.virtnet: -o user_xattr,flock 192.168.202.155@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2241 (time 1741626667.17 total 10.00 last 224.09) - open/close 4257 (time 1741626677.17 total 20.00 last 201.57) - open/close 6170 (time 1741626687.17 total 30.00 last 191.27) - open/close 7970 (time 1741626697.17 total 40.00 last 179.99) - open/close 10000 (time 1741626705.89 total 48.72 last 232.83) - open/close 12442 (time 1741626715.89 total 58.72 last 244.16) - open/close 15375 (time 1741626725.89 total 68.72 last 293.29) - open/close 18223 (time 1741626735.89 total 78.73 last 284.75) - open/close 20000 (time 1741626742.74 total 85.57 last 259.69) - open/close 22451 (time 1741626752.74 total 95.57 last 245.03) - open/close 25286 (time 1741626762.75 total 105.58 last 283.26) - open/close 28244 (time 1741626772.75 total 115.58 last 295.74) - open/close 30000 (time 1741626777.96 total 120.79 last 337.22) - open/close 33280 (time 1741626787.96 total 130.79 last 327.98) - open/close 36770 (time 1741626797.96 total 140.79 last 348.97) - open/close 40000 (time 1741626807.73 total 150.56 last 330.67) - open/close 43056 (time 1741626817.73 total 160.56 last 305.54) - open/close 46211 (time 1741626827.73 total 170.56 last 315.49) - open/close 49305 (time 1741626837.73 total 180.56 last 309.31) - open/close 50000 (time 1741626839.92 total 182.75 last 318.27) - open/close 53336 (time 1741626849.92 total 192.75 last 333.56) - open/close 56829 (time 1741626859.92 total 202.75 last 349.26) - open/close 59918 (time 1741626869.92 total 212.75 last 308.85) - open/close 63501 (time 1741626879.92 total 222.75 last 358.28) total: 64768 open/close in 226.34 seconds: 286.15 ops/second - /unlink 5939 (time 1741626894.41 total 10.00 last 593.85) - /unlink 10000 (time 1741626899.87 total 15.46 last 743.68) - /unlink 16078 (time 1741626909.87 total 25.46 last 607.77) - /unlink 20000 (time 1741626916.66 total 32.25 last 578.06) - /unlink 25611 (time 1741626926.66 total 42.25 last 561.09) - /unlink 30000 (time 1741626933.83 total 49.42 last 611.67) - /unlink 35774 (time 1741626943.84 total 59.42 last 577.34) - /unlink 40000 (time 1741626951.23 total 66.82 last 571.72) - /unlink 45166 (time 1741626961.23 total 76.82 last 516.57) - /unlink 49502 (time 1741626971.23 total 86.82 last 433.57) - /unlink 53433 (time 1741626981.23 total 96.82 last 393.10) - /unlink 57522 (time 1741626991.23 total 106.82 last 408.83) - /unlink 60000 (time 1741626996.69 total 112.27 last 454.15) total: 64768 /unlink in 121.16 seconds: 534.58 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2700 (time 1741627031.14 total 10.04 last 268.88) - open/close 5965 (time 1741627041.14 total 20.04 last 326.47) - open/close 8867 (time 1741627051.14 total 30.04 last 290.17) - open/close 10000 (time 1741627054.71 total 33.61 last 317.88) - open/close 12903 (time 1741627064.75 total 43.65 last 289.06) - open/close 16096 (time 1741627074.75 total 53.65 last 319.24) - open/close 19107 (time 1741627084.75 total 63.65 last 301.04) - open/close 20000 (time 1741627087.80 total 66.70 last 293.25) - open/close 22855 (time 1741627097.80 total 76.70 last 285.40) - open/close 25655 (time 1741627107.81 total 86.71 last 279.90) - open/close 28583 (time 1741627117.81 total 96.71 last 292.80) - open/close 30000 (time 1741627122.43 total 101.33 last 306.71) - open/close 32980 (time 1741627132.43 total 111.33 last 297.98) - open/close 35621 (time 1741627142.43 total 121.33 last 263.98) - open/close 38385 (time 1741627152.43 total 131.33 last 276.35) - open/close 40000 (time 1741627157.81 total 136.71 last 300.14) - open/close 43195 (time 1741627167.81 total 146.72 last 319.45) - open/close 46679 (time 1741627177.84 total 156.74 last 347.42) - open/close 50000 (time 1741627187.45 total 166.35 last 345.82) - open/close 53736 (time 1741627197.45 total 176.35 last 373.56) - open/close 57891 (time 1741627207.45 total 186.35 last 415.48) - open/close 60000 (time 1741627213.12 total 192.02 last 371.96) - open/close 64558 (time 1741627223.12 total 202.02 last 455.68) total: 64768 open/close in 202.58 seconds: 319.72 ops/second - /unlink 8172 (time 1741627234.53 total 10.00 last 817.18) - /unlink 10000 (time 1741627236.66 total 12.12 last 860.61) - /unlink 18427 (time 1741627246.66 total 22.13 last 842.63) - /unlink 20000 (time 1741627248.67 total 24.14 last 781.98) - /unlink 28806 (time 1741627258.67 total 34.14 last 880.58) - /unlink 37374 (time 1741627268.67 total 44.14 last 856.76) - /unlink 40000 (time 1741627271.99 total 47.45 last 792.14) - /unlink 48665 (time 1741627281.99 total 57.45 last 866.45) - /unlink 57378 (time 1741627291.99 total 67.45 last 871.24) - /unlink 60000 (time 1741627294.67 total 70.14 last 976.76) total: 64768 /unlink in 75.00 seconds: 863.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5106 (time 1741627335.48 total 10.00 last 510.55) - open/close 10000 (time 1741627344.14 total 18.66 last 565.14) - open/close 15088 (time 1741627354.14 total 28.66 last 508.73) - open/close 20000 (time 1741627362.85 total 37.37 last 564.20) - open/close 25396 (time 1741627372.85 total 47.37 last 539.59) - open/close 30000 (time 1741627381.08 total 55.60 last 559.45) - open/close 35442 (time 1741627391.08 total 65.60 last 544.11) - open/close 40000 (time 1741627400.69 total 75.21 last 474.12) - open/close 44810 (time 1741627410.69 total 85.21 last 480.99) - open/close 49307 (time 1741627420.69 total 95.22 last 449.61) - open/close 54317 (time 1741627430.69 total 105.22 last 500.99) - open/close 59351 (time 1741627440.70 total 115.22 last 503.30) - open/close 64459 (time 1741627450.70 total 125.22 last 510.79) total: 64768 open/close in 125.86 seconds: 514.61 ops/second - /unlink 8949 (time 1741627462.16 total 10.00 last 894.83) - /unlink 17423 (time 1741627472.16 total 20.00 last 847.33) - /unlink 20000 (time 1741627475.29 total 23.13 last 823.42) - /unlink 28637 (time 1741627485.29 total 33.13 last 863.64) - /unlink 37421 (time 1741627495.29 total 43.13 last 878.33) - /unlink 40000 (time 1741627498.19 total 46.04 last 887.94) - /unlink 48163 (time 1741627508.19 total 56.04 last 816.26) - /unlink 50000 (time 1741627510.30 total 58.14 last 873.23) - /unlink 58880 (time 1741627520.30 total 68.14 last 887.90) total: 64768 /unlink in 74.68 seconds: 867.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4539 (time 1741627561.11 total 10.00 last 453.84) - open/close 7658 (time 1741627571.11 total 20.00 last 311.85) - open/close 10000 (time 1741627579.89 total 28.79 last 266.68) - open/close 14383 (time 1741627589.89 total 38.79 last 438.30) - open/close 19758 (time 1741627599.89 total 48.79 last 537.50) - open/close 23895 (time 1741627609.89 total 58.79 last 413.67) - open/close 29327 (time 1741627619.90 total 68.79 last 543.13) - open/close 34756 (time 1741627629.90 total 78.79 last 542.89) - open/close 39273 (time 1741627639.90 total 88.79 last 451.67) - open/close 43748 (time 1741627649.90 total 98.79 last 447.47) - open/close 48150 (time 1741627659.90 total 108.79 last 440.19) - open/close 50000 (time 1741627664.27 total 113.16 last 423.52) - open/close 54366 (time 1741627674.27 total 123.16 last 436.59) - open/close 59338 (time 1741627684.27 total 133.16 last 497.13) - open/close 64610 (time 1741627694.27 total 143.16 last 527.18) total: 64768 open/close in 143.51 seconds: 451.31 ops/second - /unlink 9400 (time 1741627705.42 total 10.00 last 939.99) - /unlink 18640 (time 1741627715.42 total 20.00 last 923.96) - /unlink 27630 (time 1741627725.42 total 30.00 last 898.79) - /unlink 30000 (time 1741627728.17 total 32.75 last 862.02) - /unlink 37880 (time 1741627738.17 total 42.75 last 787.92) - /unlink 40000 (time 1741627740.97 total 45.55 last 757.49) - /unlink 48429 (time 1741627750.97 total 55.55 last 842.87) - /unlink 57143 (time 1741627760.97 total 65.55 last 871.31) - /unlink 60000 (time 1741627764.33 total 68.91 last 850.37) total: 64768 /unlink in 74.24 seconds: 872.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5042 (time 1741627806.34 total 10.00 last 504.14) - open/close 9855 (time 1741627816.34 total 20.00 last 481.25) - open/close 14994 (time 1741627826.34 total 30.00 last 513.82) - open/close 20000 (time 1741627836.03 total 39.69 last 516.85) - open/close 25043 (time 1741627846.03 total 49.69 last 504.26) - open/close 30000 (time 1741627855.95 total 59.61 last 499.54) - open/close 35104 (time 1741627865.95 total 69.61 last 510.39) - open/close 40000 (time 1741627875.37 total 79.03 last 520.13) - open/close 45088 (time 1741627885.37 total 89.03 last 508.78) - open/close 50000 (time 1741627895.01 total 98.67 last 509.27) - open/close 55046 (time 1741627905.01 total 108.67 last 504.52) - open/close 60000 (time 1741627914.96 total 118.62 last 497.85) - open/close 63784 (time 1741627924.97 total 128.63 last 378.35) total: 64768 open/close in 130.94 seconds: 494.63 ops/second - /unlink 9054 (time 1741627938.12 total 10.00 last 905.32) - /unlink 18177 (time 1741627948.12 total 20.00 last 912.13) - /unlink 27391 (time 1741627958.12 total 30.00 last 921.40) - /unlink 30000 (time 1741627961.01 total 32.90 last 901.00) - /unlink 39100 (time 1741627971.01 total 42.90 last 909.96) - /unlink 48111 (time 1741627981.01 total 52.90 last 901.07) - /unlink 50000 (time 1741627983.04 total 54.93 last 930.99) - /unlink 59392 (time 1741627993.04 total 64.93 last 939.19) total: 64768 /unlink in 70.68 seconds: 916.40 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg255-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg255-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg255-server stop mds service on oleg255-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg255-server unloading modules via unload_modules_local on: 'oleg255-server' oleg255-server: oleg255-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-client: ssh exited with exit code 2 pdsh@oleg255-client: oleg255-server: ssh exited with exit code 2