== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:17:04 (1748989024) Stopping clients: oleg302-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg302-client.virtnet /mnt/lustre2 (opts:-f) oleg302-server: oleg302-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing load_modules_local oleg302-server: Loading modules from /home/green/git/lustre-release/lustre oleg302-server: detected 4 online CPUs by sysfs oleg302-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg302-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg302-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg302-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg302-server: oleg302-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg302-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg302-client.virtnet: -o user_xattr,flock oleg302-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16131 291 15840 2% /mnt/lustre[OST:0] filesystem_summary: 16169 329 15840 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 5652 (time 1748989057.97 total 10.00 last 565.19) - open/close 10000 (time 1748989066.01 total 18.04 last 540.72) - open/close 15450 (time 1748989076.01 total 28.04 last 544.98) - open/close 20000 (time 1748989084.25 total 36.27 last 552.77) - open/close 25620 (time 1748989094.25 total 46.27 last 561.94) - open/close 30000 (time 1748989102.03 total 54.06 last 562.78) - open/close 35476 (time 1748989112.03 total 64.06 last 547.53) - open/close 40000 (time 1748989120.17 total 72.20 last 555.87) - open/close 43846 (time 1748989130.17 total 82.20 last 384.48) - open/close 47606 (time 1748989140.17 total 92.20 last 375.96) - open/close 50000 (time 1748989146.10 total 98.12 last 404.25) - open/close 54004 (time 1748989156.10 total 108.12 last 400.37) - open/close 57820 (time 1748989166.10 total 118.13 last 381.54) - open/close 60000 (time 1748989170.57 total 122.59 last 487.92) - open/close 64435 (time 1748989180.57 total 132.59 last 443.47) total: 64768 open/close in 133.29 seconds: 485.93 ops/second - /unlink 9082 (time 1748989192.20 total 10.00 last 908.15) - /unlink 17450 (time 1748989202.20 total 20.00 last 836.73) - /unlink 20000 (time 1748989205.23 total 23.04 last 840.31) - /unlink 27716 (time 1748989215.23 total 33.04 last 771.56) - /unlink 30000 (time 1748989218.56 total 36.37 last 685.13) - /unlink 38265 (time 1748989228.56 total 46.37 last 826.50) - /unlink 40000 (time 1748989230.73 total 48.54 last 800.65) - /unlink 47569 (time 1748989240.73 total 58.54 last 756.83) - /unlink 50000 (time 1748989243.57 total 61.38 last 855.91) - /unlink 57090 (time 1748989253.57 total 71.38 last 708.98) - /unlink 60000 (time 1748989256.73 total 74.53 last 922.32) total: 64768 /unlink in 80.53 seconds: 804.32 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2562 (time 1748989293.54 total 10.00 last 256.17) - open/close 5333 (time 1748989303.54 total 20.00 last 277.05) - open/close 9008 (time 1748989313.54 total 30.00 last 367.49) - open/close 10000 (time 1748989316.12 total 32.58 last 385.24) - open/close 12873 (time 1748989326.12 total 42.58 last 287.28) - open/close 15469 (time 1748989336.12 total 52.58 last 259.60) - open/close 18705 (time 1748989346.12 total 62.58 last 323.58) - open/close 20000 (time 1748989348.93 total 65.40 last 459.80) - open/close 23854 (time 1748989358.93 total 75.40 last 385.39) - open/close 26639 (time 1748989368.94 total 85.40 last 278.42) - open/close 29971 (time 1748989378.94 total 95.40 last 333.17) - open/close 32514 (time 1748989388.94 total 105.40 last 254.26) - open/close 35500 (time 1748989398.94 total 115.40 last 298.53) - open/close 38055 (time 1748989408.94 total 125.41 last 255.46) - open/close 40000 (time 1748989416.94 total 133.40 last 243.33) - open/close 42703 (time 1748989426.94 total 143.40 last 270.23) - open/close 46039 (time 1748989436.94 total 153.40 last 333.57) - open/close 49549 (time 1748989446.94 total 163.40 last 350.96) - open/close 53895 (time 1748989456.94 total 173.40 last 434.58) - open/close 58067 (time 1748989466.94 total 183.41 last 417.16) - open/close 60000 (time 1748989470.91 total 187.37 last 487.19) - open/close 63667 (time 1748989480.91 total 197.37 last 366.69) total: 64768 open/close in 199.75 seconds: 324.25 ops/second - /unlink 9009 (time 1748989494.10 total 10.00 last 900.86) - /unlink 18434 (time 1748989504.10 total 20.00 last 942.40) - /unlink 26757 (time 1748989514.10 total 30.00 last 832.29) - /unlink 30000 (time 1748989517.71 total 33.61 last 899.90) - /unlink 38873 (time 1748989527.71 total 43.61 last 887.30) - /unlink 47976 (time 1748989537.71 total 53.61 last 910.30) - /unlink 50000 (time 1748989539.90 total 55.80 last 922.14) - /unlink 56980 (time 1748989549.90 total 65.80 last 697.92) - /unlink 60000 (time 1748989553.57 total 69.47 last 824.12) total: 64768 /unlink in 76.91 seconds: 842.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4052 (time 1748989594.19 total 10.00 last 405.13) - open/close 8576 (time 1748989604.19 total 20.00 last 452.34) - open/close 10000 (time 1748989608.65 total 24.46 last 319.55) - open/close 13978 (time 1748989618.65 total 34.46 last 397.65) - open/close 16856 (time 1748989628.66 total 44.46 last 287.77) - open/close 20000 (time 1748989638.05 total 53.86 last 334.59) - open/close 23871 (time 1748989648.05 total 63.86 last 387.06) - open/close 27215 (time 1748989658.06 total 73.86 last 334.29) - open/close 29978 (time 1748989668.06 total 83.87 last 276.24) - open/close 32455 (time 1748989678.06 total 93.87 last 247.66) - open/close 35004 (time 1748989688.06 total 103.87 last 254.81) - open/close 37608 (time 1748989698.06 total 113.87 last 260.39) - open/close 40000 (time 1748989707.52 total 123.33 last 252.95) - open/close 42510 (time 1748989717.52 total 133.33 last 250.91) - open/close 45138 (time 1748989727.53 total 143.34 last 262.73) - open/close 47580 (time 1748989737.53 total 153.34 last 244.14) - open/close 50000 (time 1748989746.89 total 162.70 last 258.40) - open/close 52544 (time 1748989756.89 total 172.70 last 254.40) - open/close 55111 (time 1748989766.90 total 182.70 last 256.67) - open/close 57634 (time 1748989776.97 total 192.78 last 250.51) - open/close 60000 (time 1748989786.04 total 201.85 last 260.73) - open/close 62589 (time 1748989796.04 total 211.85 last 258.86) total: 64768 open/close in 220.63 seconds: 293.56 ops/second - /unlink 4727 (time 1748989816.11 total 10.00 last 472.65) - /unlink 9422 (time 1748989826.11 total 20.00 last 469.42) - /unlink 14155 (time 1748989836.11 total 30.00 last 473.22) - /unlink 18984 (time 1748989846.11 total 40.01 last 482.86) - /unlink 20000 (time 1748989848.45 total 42.34 last 435.63) - /unlink 24722 (time 1748989858.45 total 52.34 last 472.19) - /unlink 29812 (time 1748989868.45 total 62.34 last 508.97) - /unlink 34598 (time 1748989878.45 total 72.34 last 478.46) - /unlink 39392 (time 1748989888.45 total 82.34 last 479.28) - /unlink 44127 (time 1748989898.45 total 92.34 last 473.48) - /unlink 49229 (time 1748989908.45 total 102.34 last 510.17) - /unlink 53996 (time 1748989918.46 total 112.35 last 476.48) - /unlink 60000 (time 1748989928.26 total 122.15 last 612.38) total: 64768 /unlink in 131.72 seconds: 491.69 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5014 (time 1748989970.11 total 10.00 last 501.40) - open/close 9333 (time 1748989980.12 total 20.00 last 431.79) - open/close 12579 (time 1748989990.12 total 30.00 last 324.56) - open/close 15027 (time 1748990000.12 total 40.01 last 244.77) - open/close 17469 (time 1748990010.12 total 50.01 last 244.18) - open/close 19899 (time 1748990020.12 total 60.01 last 242.93) - open/close 22390 (time 1748990030.12 total 70.01 last 249.07) - open/close 24908 (time 1748990040.12 total 80.01 last 251.77) - open/close 27355 (time 1748990050.13 total 90.01 last 244.64) - open/close 29776 (time 1748990060.13 total 100.01 last 242.08) - open/close 32217 (time 1748990070.13 total 110.02 last 244.08) - open/close 34772 (time 1748990080.13 total 120.02 last 255.49) - open/close 37267 (time 1748990090.13 total 130.02 last 249.49) - open/close 39767 (time 1748990100.13 total 140.02 last 249.95) - open/close 42587 (time 1748990110.13 total 150.02 last 281.95) - open/close 45807 (time 1748990120.13 total 160.02 last 321.99) - open/close 50000 (time 1748990129.66 total 169.55 last 440.22) - open/close 54223 (time 1748990139.66 total 179.55 last 422.17) - open/close 58605 (time 1748990149.66 total 189.55 last 438.18) - open/close 60000 (time 1748990153.86 total 193.74 last 332.47) - open/close 63709 (time 1748990163.86 total 203.75 last 370.83) total: 64768 open/close in 206.42 seconds: 313.76 ops/second - /unlink 5817 (time 1748990177.51 total 10.00 last 581.65) - /unlink 10000 (time 1748990184.34 total 16.83 last 612.79) - /unlink 14912 (time 1748990194.34 total 26.83 last 491.10) - /unlink 19989 (time 1748990204.34 total 36.83 last 507.66) - /unlink 24745 (time 1748990214.34 total 46.83 last 475.57) - /unlink 29323 (time 1748990224.34 total 56.83 last 457.80) - /unlink 33908 (time 1748990234.34 total 66.83 last 458.42) - /unlink 38572 (time 1748990244.34 total 76.83 last 466.36) - /unlink 40000 (time 1748990247.32 total 79.81 last 480.15) - /unlink 44502 (time 1748990257.32 total 89.81 last 450.18) - /unlink 49088 (time 1748990267.32 total 99.81 last 458.53) - /unlink 50000 (time 1748990269.38 total 101.87 last 442.53) - /unlink 56995 (time 1748990279.38 total 111.87 last 699.42) - /unlink 60000 (time 1748990284.77 total 117.26 last 557.38) total: 64768 /unlink in 127.16 seconds: 509.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3332 (time 1748990325.15 total 10.00 last 333.07) - open/close 6686 (time 1748990335.15 total 20.00 last 335.38) - open/close 9557 (time 1748990345.15 total 30.01 last 287.03) - open/close 12360 (time 1748990355.16 total 40.01 last 280.21) - open/close 15539 (time 1748990365.16 total 50.01 last 317.81) - open/close 18322 (time 1748990375.16 total 60.02 last 278.21) - open/close 20000 (time 1748990381.26 total 66.12 last 275.08) - open/close 22577 (time 1748990391.27 total 76.12 last 257.62) - open/close 25074 (time 1748990401.27 total 86.12 last 249.62) - open/close 27651 (time 1748990411.27 total 96.12 last 257.63) - open/close 30000 (time 1748990418.03 total 102.88 last 347.80) - open/close 33474 (time 1748990428.03 total 112.88 last 347.35) - open/close 36421 (time 1748990438.03 total 122.88 last 294.67) - open/close 40000 (time 1748990447.95 total 132.80 last 360.87) - open/close 42897 (time 1748990457.95 total 142.80 last 289.69) - open/close 46316 (time 1748990467.95 total 152.80 last 341.87) - open/close 48736 (time 1748990477.95 total 162.80 last 241.93) - open/close 50000 (time 1748990483.07 total 167.92 last 246.88) - open/close 52435 (time 1748990493.07 total 177.93 last 243.40) - open/close 55435 (time 1748990503.08 total 187.93 last 299.95) - open/close 58482 (time 1748990513.08 total 197.93 last 304.64) - open/close 60000 (time 1748990518.25 total 203.10 last 293.68) - open/close 62869 (time 1748990528.25 total 213.10 last 286.85) total: 64768 open/close in 221.07 seconds: 292.97 ops/second - /unlink 4493 (time 1748990547.43 total 10.00 last 449.28) - /unlink 9098 (time 1748990557.43 total 20.00 last 460.48) - /unlink 10000 (time 1748990559.43 total 22.00 last 450.79) - /unlink 14747 (time 1748990569.43 total 32.00 last 474.69) - /unlink 19469 (time 1748990579.43 total 42.00 last 472.17) - /unlink 24253 (time 1748990589.43 total 52.00 last 478.39) - /unlink 28983 (time 1748990599.43 total 62.00 last 472.94) - /unlink 33730 (time 1748990609.43 total 72.01 last 474.63) - /unlink 38396 (time 1748990619.44 total 82.01 last 466.53) - /unlink 40000 (time 1748990622.16 total 84.73 last 589.48) - /unlink 44719 (time 1748990632.16 total 94.73 last 471.85) - /unlink 49731 (time 1748990642.16 total 104.73 last 501.12) - /unlink 54286 (time 1748990652.16 total 114.73 last 455.48) - /unlink 59034 (time 1748990662.16 total 124.73 last 474.78) - /unlink 60000 (time 1748990664.27 total 126.84 last 458.23) total: 64768 /unlink in 136.48 seconds: 474.56 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg302-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg302-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg302-server stop mds service on oleg302-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg302-server LNET ready to unload unloading modules on: 'oleg302-server' oleg302-server: oleg302-server.virtnet: executing unload_modules_local oleg302-server: LNET ready to unload modules unloaded. pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-client: ssh exited with exit code 2 pdsh@oleg302-client: oleg302-server: ssh exited with exit code 2