== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:11:54 (1744045914) Stopping clients: oleg344-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg344-client.virtnet /mnt/lustre2 (opts:-f) oleg344-server: oleg344-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing load_modules_local oleg344-server: Loading modules from /home/green/git/lustre-release/lustre oleg344-server: detected 4 online CPUs by sysfs oleg344-server: MODOPTS_LIBCFS= oleg344-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg344-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg344-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg344-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg344-server: oleg344-server.virtnet: executing set_default_debug -1 all pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg344-client.virtnet: -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4610 (time 1744045953.70 total 10.00 last 460.95) - open/close 8625 (time 1744045963.70 total 20.00 last 401.47) - open/close 10000 (time 1744045966.77 total 23.08 last 446.98) - open/close 14275 (time 1744045976.77 total 33.08 last 427.48) - open/close 18701 (time 1744045986.77 total 43.08 last 442.56) - open/close 20000 (time 1744045989.38 total 45.69 last 498.15) - open/close 24750 (time 1744045999.38 total 55.69 last 475.00) - open/close 29660 (time 1744046009.38 total 65.69 last 490.98) - open/close 34708 (time 1744046019.38 total 75.69 last 504.72) - open/close 40000 (time 1744046028.92 total 85.23 last 554.84) - open/close 44668 (time 1744046038.92 total 95.23 last 466.66) - open/close 49386 (time 1744046048.92 total 105.23 last 471.80) - open/close 54293 (time 1744046058.93 total 115.23 last 490.66) - open/close 59678 (time 1744046068.93 total 125.23 last 538.42) total: 64768 open/close in 133.47 seconds: 485.26 ops/second - /unlink 9764 (time 1744046088.00 total 10.00 last 976.36) - /unlink 19040 (time 1744046098.00 total 20.00 last 927.54) - /unlink 27008 (time 1744046108.00 total 30.00 last 796.69) - /unlink 30000 (time 1744046112.08 total 34.08 last 734.63) - /unlink 37973 (time 1744046122.08 total 44.08 last 797.22) - /unlink 40000 (time 1744046124.39 total 46.39 last 877.26) - /unlink 47868 (time 1744046134.39 total 56.39 last 786.77) - /unlink 50000 (time 1744046136.85 total 58.85 last 867.23) - /unlink 58131 (time 1744046146.85 total 68.85 last 812.92) - /unlink 60000 (time 1744046149.79 total 71.78 last 636.40) total: 64768 /unlink in 77.96 seconds: 830.82 ops/second Waiting for MDT destroys to complete - open/close 4685 (time 1744046247.25 total 10.00 last 468.44) - open/close 10000 (time 1744046256.88 total 19.63 last 552.11) - open/close 14630 (time 1744046266.89 total 29.64 last 462.56) - open/close 19202 (time 1744046276.89 total 39.64 last 457.18) - open/close 20000 (time 1744046278.98 total 41.73 last 381.72) - open/close 24734 (time 1744046288.98 total 51.73 last 473.35) - open/close 28867 (time 1744046298.98 total 61.73 last 413.19) - open/close 30000 (time 1744046301.52 total 64.27 last 446.43) - open/close 34719 (time 1744046311.52 total 74.27 last 471.83) - open/close 38933 (time 1744046321.52 total 84.27 last 421.38) - open/close 44030 (time 1744046331.52 total 94.27 last 509.70) - open/close 48954 (time 1744046341.52 total 104.27 last 492.35) - open/close 52956 (time 1744046351.52 total 114.27 last 400.18) - open/close 56952 (time 1744046361.52 total 124.27 last 399.58) - open/close 60000 (time 1744046368.58 total 131.33 last 431.86) - open/close 63853 (time 1744046378.58 total 141.33 last 385.25) total: 64768 open/close in 143.41 seconds: 451.62 ops/second - /unlink 8576 (time 1744046391.63 total 10.00 last 857.50) - /unlink 16406 (time 1744046401.63 total 20.00 last 782.92) - /unlink 20000 (time 1744046405.17 total 23.55 last 1013.88) - /unlink 30000 (time 1744046415.15 total 33.52 last 1002.22) - /unlink 38944 (time 1744046425.15 total 43.53 last 894.28) - /unlink 48380 (time 1744046435.15 total 53.53 last 943.57) - /unlink 55693 (time 1744046445.15 total 63.53 last 731.21) - /unlink 60000 (time 1744046450.13 total 68.51 last 864.77) total: 64768 /unlink in 74.39 seconds: 870.60 ops/second Waiting for MDT destroys to complete - open/close 5069 (time 1744046553.94 total 10.00 last 506.87) - open/close 9477 (time 1744046563.94 total 20.00 last 440.79) - open/close 14018 (time 1744046573.94 total 30.00 last 454.08) - open/close 18879 (time 1744046583.94 total 40.00 last 486.07) - open/close 20000 (time 1744046586.92 total 42.98 last 376.22) - open/close 23683 (time 1744046596.92 total 52.98 last 368.30) - open/close 27610 (time 1744046606.92 total 62.98 last 392.67) - open/close 30000 (time 1744046612.58 total 68.65 last 421.72) - open/close 34749 (time 1744046622.59 total 78.65 last 474.84) - open/close 39585 (time 1744046632.59 total 88.65 last 483.50) - open/close 43844 (time 1744046642.59 total 98.65 last 425.88) - open/close 48228 (time 1744046652.59 total 108.66 last 438.22) - open/close 50000 (time 1744046656.79 total 112.86 last 421.74) - open/close 54362 (time 1744046666.80 total 122.86 last 436.16) - open/close 58979 (time 1744046676.80 total 132.86 last 461.66) - open/close 60000 (time 1744046678.86 total 134.92 last 494.86) - open/close 64675 (time 1744046688.86 total 144.93 last 467.41) total: 64768 open/close in 145.12 seconds: 446.31 ops/second - /unlink 7828 (time 1744046700.00 total 10.00 last 782.72) - /unlink 10000 (time 1744046702.86 total 12.86 last 759.74) - /unlink 18479 (time 1744046712.86 total 22.86 last 847.89) - /unlink 27150 (time 1744046722.86 total 32.86 last 867.08) - /unlink 30000 (time 1744046726.08 total 36.08 last 885.34) - /unlink 38426 (time 1744046736.08 total 46.08 last 842.52) - /unlink 47146 (time 1744046746.08 total 56.08 last 871.92) - /unlink 50000 (time 1744046749.14 total 59.14 last 933.62) - /unlink 59545 (time 1744046759.14 total 69.14 last 954.44) total: 64768 /unlink in 74.78 seconds: 866.14 ops/second Waiting for MDT destroys to complete - open/close 4624 (time 1744046858.23 total 10.00 last 462.32) - open/close 9037 (time 1744046868.23 total 20.00 last 441.22) - open/close 10000 (time 1744046870.29 total 22.06 last 469.17) - open/close 14626 (time 1744046880.29 total 32.06 last 462.51) - open/close 19294 (time 1744046890.29 total 42.06 last 466.76) - open/close 23940 (time 1744046900.29 total 52.06 last 464.58) - open/close 28650 (time 1744046910.29 total 62.06 last 470.85) - open/close 30000 (time 1744046913.71 total 65.48 last 394.59) - open/close 34819 (time 1744046923.71 total 75.48 last 481.86) - open/close 39611 (time 1744046933.72 total 85.49 last 479.17) - open/close 43628 (time 1744046943.72 total 95.49 last 401.61) - open/close 47581 (time 1744046953.72 total 105.49 last 395.25) - open/close 50000 (time 1744046959.25 total 111.02 last 436.98) - open/close 53884 (time 1744046969.26 total 121.03 last 388.37) - open/close 58188 (time 1744046979.26 total 131.03 last 430.35) - open/close 60000 (time 1744046983.07 total 134.84 last 475.31) - open/close 63730 (time 1744046993.07 total 144.84 last 372.98) total: 64768 open/close in 147.28 seconds: 439.75 ops/second - /unlink 7695 (time 1744047006.40 total 10.00 last 769.45) - /unlink 10000 (time 1744047009.01 total 12.61 last 883.33) - /unlink 18502 (time 1744047019.01 total 22.61 last 850.19) - /unlink 26595 (time 1744047029.01 total 32.61 last 809.23) - /unlink 30000 (time 1744047033.06 total 36.66 last 840.03) - /unlink 38684 (time 1744047043.06 total 46.67 last 868.32) - /unlink 47207 (time 1744047053.06 total 56.67 last 852.26) - /unlink 50000 (time 1744047056.41 total 60.01 last 835.35) - /unlink 59109 (time 1744047066.41 total 70.01 last 910.86) total: 64768 /unlink in 76.41 seconds: 847.69 ops/second Waiting for MDT destroys to complete - open/close 4882 (time 1744047164.85 total 10.00 last 488.15) - open/close 9677 (time 1744047174.85 total 20.00 last 479.49) - open/close 14711 (time 1744047184.85 total 30.00 last 503.34) - open/close 20000 (time 1744047194.78 total 39.94 last 532.35) - open/close 25127 (time 1744047204.78 total 49.94 last 512.70) - open/close 30000 (time 1744047214.21 total 59.36 last 517.04) - open/close 35014 (time 1744047224.21 total 69.36 last 501.39) - open/close 40000 (time 1744047233.41 total 78.56 last 541.98) - open/close 45167 (time 1744047243.41 total 88.56 last 516.61) - open/close 50000 (time 1744047252.58 total 97.73 last 527.34) - open/close 55346 (time 1744047262.58 total 107.73 last 534.50) - open/close 60000 (time 1744047271.22 total 116.38 last 538.31) total: 64768 open/close in 124.99 seconds: 518.19 ops/second - /unlink 9680 (time 1744047290.71 total 10.00 last 967.91) - /unlink 19663 (time 1744047300.71 total 20.00 last 998.24) - /unlink 29736 (time 1744047310.71 total 30.00 last 1007.25) - /unlink 39512 (time 1744047320.71 total 40.00 last 977.53) - /unlink 49266 (time 1744047330.71 total 50.00 last 975.30) - /unlink 59094 (time 1744047340.71 total 60.00 last 982.76) total: 64768 /unlink in 65.91 seconds: 982.73 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg344-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg344-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg344-server stop mds service on oleg344-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg344-server unloading modules via unload_modules_local on: 'oleg344-server' oleg344-server: oleg344-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-client: ssh exited with exit code 2 pdsh@oleg344-client: oleg344-server: ssh exited with exit code 2