== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:21:38 (1740673298) Stopping clients: oleg131-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg131-client.virtnet /mnt/lustre2 (opts:-f) oleg131-server: oleg131-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing load_modules_local oleg131-server: Loading modules from /home/green/git/lustre-release/lustre oleg131-server: detected 4 online CPUs by sysfs oleg131-server: MODOPTS_LIBCFS= oleg131-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg131-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg131-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg131-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg131-server: oleg131-server.virtnet: executing set_default_debug -1 all pdsh@oleg131-client: oleg131-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg131-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg131-client.virtnet: -o user_xattr,flock 192.168.201.131@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5307 (time 1740673333.19 total 10.00 last 530.67) - open/close 10000 (time 1740673341.94 total 18.74 last 536.81) - open/close 15272 (time 1740673351.94 total 28.74 last 527.16) - open/close 20000 (time 1740673360.95 total 37.76 last 524.35) - open/close 25016 (time 1740673370.96 total 47.76 last 501.55) - open/close 30000 (time 1740673380.56 total 57.37 last 518.64) - open/close 34998 (time 1740673390.57 total 67.37 last 499.79) - open/close 40000 (time 1740673400.01 total 76.82 last 529.34) - open/close 45062 (time 1740673410.02 total 86.82 last 506.11) - open/close 50000 (time 1740673419.69 total 96.50 last 510.49) - open/close 55242 (time 1740673429.69 total 106.50 last 524.12) - open/close 60000 (time 1740673438.95 total 115.76 last 513.63) total: 64768 open/close in 124.89 seconds: 518.61 ops/second - /unlink 8351 (time 1740673458.92 total 10.00 last 835.03) - /unlink 17133 (time 1740673468.92 total 20.00 last 878.17) - /unlink 20000 (time 1740673472.11 total 23.19 last 898.93) - /unlink 28988 (time 1740673482.11 total 33.19 last 898.75) - /unlink 37962 (time 1740673492.11 total 43.19 last 897.32) - /unlink 40000 (time 1740673494.53 total 45.61 last 844.04) - /unlink 48422 (time 1740673504.53 total 55.61 last 842.12) - /unlink 57226 (time 1740673514.53 total 65.61 last 880.33) - /unlink 60000 (time 1740673517.90 total 68.98 last 822.53) total: 64768 /unlink in 74.43 seconds: 870.23 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5227 (time 1740673556.88 total 10.00 last 522.64) - open/close 10000 (time 1740673566.16 total 19.28 last 514.28) - open/close 15150 (time 1740673576.16 total 29.28 last 514.97) - open/close 20000 (time 1740673585.40 total 38.53 last 524.61) - open/close 25240 (time 1740673595.40 total 48.53 last 523.97) - open/close 30000 (time 1740673604.30 total 57.42 last 535.14) - open/close 35215 (time 1740673614.30 total 67.42 last 521.47) - open/close 40000 (time 1740673623.40 total 76.52 last 525.78) - open/close 45163 (time 1740673633.40 total 86.53 last 516.23) - open/close 50000 (time 1740673642.73 total 95.85 last 518.66) - open/close 55247 (time 1740673652.73 total 105.85 last 524.69) - open/close 60000 (time 1740673662.03 total 115.16 last 510.86) total: 64768 open/close in 124.81 seconds: 518.94 ops/second - /unlink 8942 (time 1740673682.50 total 10.00 last 894.18) - /unlink 17538 (time 1740673692.50 total 20.00 last 859.54) - /unlink 20000 (time 1740673695.20 total 22.70 last 913.43) - /unlink 28922 (time 1740673705.20 total 32.70 last 892.14) - /unlink 37815 (time 1740673715.20 total 42.70 last 889.25) - /unlink 40000 (time 1740673717.68 total 45.18 last 880.88) - /unlink 48805 (time 1740673727.68 total 55.18 last 880.49) - /unlink 50000 (time 1740673730.06 total 57.56 last 501.58) - /unlink 57847 (time 1740673740.06 total 67.56 last 784.64) - /unlink 60000 (time 1740673742.64 total 70.14 last 834.72) total: 64768 /unlink in 75.61 seconds: 856.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5617 (time 1740673782.13 total 10.00 last 561.64) - open/close 10000 (time 1740673790.09 total 17.96 last 550.97) - open/close 15536 (time 1740673800.09 total 27.96 last 553.55) - open/close 20000 (time 1740673808.10 total 35.96 last 557.45) - open/close 25380 (time 1740673818.10 total 45.97 last 537.92) - open/close 30000 (time 1740673826.49 total 54.36 last 550.51) - open/close 35196 (time 1740673836.49 total 64.36 last 519.60) - open/close 40000 (time 1740673845.39 total 73.26 last 539.54) - open/close 45010 (time 1740673855.39 total 83.26 last 500.92) - open/close 49345 (time 1740673865.40 total 93.26 last 433.47) - open/close 54355 (time 1740673875.40 total 103.27 last 500.98) - open/close 59018 (time 1740673885.40 total 113.27 last 466.26) - open/close 64253 (time 1740673895.40 total 123.27 last 523.48) total: 64768 open/close in 124.38 seconds: 520.74 ops/second - /unlink 8568 (time 1740673907.37 total 10.00 last 856.72) - /unlink 16776 (time 1740673917.37 total 20.00 last 820.72) - /unlink 20000 (time 1740673921.02 total 23.65 last 883.41) - /unlink 30000 (time 1740673930.87 total 33.51 last 1014.75) - /unlink 39028 (time 1740673940.87 total 43.51 last 902.51) - /unlink 48026 (time 1740673950.88 total 53.52 last 899.23) - /unlink 50000 (time 1740673953.08 total 55.72 last 897.42) - /unlink 56968 (time 1740673963.08 total 65.72 last 696.72) - /unlink 60000 (time 1740673971.82 total 74.45 last 347.03) - /unlink 64484 (time 1740673981.82 total 84.45 last 448.39) total: 64768 /unlink in 84.76 seconds: 764.11 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4803 (time 1740674007.15 total 10.00 last 480.29) - open/close 9168 (time 1740674017.15 total 20.00 last 436.48) - open/close 14798 (time 1740674027.15 total 30.00 last 562.96) - open/close 20000 (time 1740674036.83 total 39.67 last 537.83) - open/close 24851 (time 1740674046.83 total 49.67 last 485.09) - open/close 29575 (time 1740674056.83 total 59.67 last 472.38) - open/close 34595 (time 1740674066.83 total 69.67 last 502.00) - open/close 39408 (time 1740674076.83 total 79.67 last 481.28) - open/close 44355 (time 1740674086.83 total 89.68 last 494.62) - open/close 49274 (time 1740674096.83 total 99.68 last 491.88) - open/close 52706 (time 1740674106.83 total 109.68 last 343.16) - open/close 56386 (time 1740674116.83 total 119.68 last 367.94) - open/close 60000 (time 1740674125.32 total 128.17 last 425.69) - open/close 63068 (time 1740674135.32 total 138.17 last 306.71) total: 64768 open/close in 143.91 seconds: 450.05 ops/second - /unlink 6377 (time 1740674152.30 total 10.00 last 637.58) - /unlink 10000 (time 1740674157.48 total 15.18 last 699.11) - /unlink 14848 (time 1740674167.48 total 25.19 last 484.72) - /unlink 19909 (time 1740674177.48 total 35.19 last 506.03) - /unlink 25877 (time 1740674187.48 total 45.19 last 596.75) - /unlink 30000 (time 1740674194.76 total 52.47 last 566.33) - /unlink 35328 (time 1740674204.76 total 62.47 last 532.71) - /unlink 40000 (time 1740674213.89 total 71.60 last 511.80) - /unlink 45993 (time 1740674223.89 total 81.60 last 599.26) - /unlink 50000 (time 1740674230.46 total 88.17 last 610.20) - /unlink 55660 (time 1740674240.46 total 98.17 last 565.98) - /unlink 60000 (time 1740674249.05 total 106.75 last 505.50) total: 64768 /unlink in 114.51 seconds: 565.59 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3168 (time 1740674288.41 total 10.00 last 316.71) - open/close 6745 (time 1740674298.41 total 20.00 last 357.66) - open/close 10000 (time 1740674307.63 total 29.22 last 353.12) - open/close 13652 (time 1740674317.63 total 39.22 last 365.17) - open/close 17084 (time 1740674327.63 total 49.22 last 343.17) - open/close 19996 (time 1740674337.63 total 59.22 last 291.19) - open/close 22971 (time 1740674347.63 total 69.23 last 297.42) - open/close 26678 (time 1740674357.63 total 79.23 last 370.58) - open/close 30000 (time 1740674366.35 total 87.95 last 380.98) - open/close 32617 (time 1740674376.36 total 97.95 last 261.64) - open/close 35310 (time 1740674386.36 total 107.96 last 269.20) - open/close 38814 (time 1740674396.36 total 117.96 last 350.27) - open/close 40000 (time 1740674399.92 total 121.52 last 333.23) - open/close 42924 (time 1740674409.92 total 131.52 last 292.34) - open/close 45639 (time 1740674419.93 total 141.52 last 271.41) - open/close 49265 (time 1740674429.93 total 151.53 last 362.51) - open/close 52096 (time 1740674439.93 total 161.53 last 283.02) - open/close 55271 (time 1740674449.93 total 171.53 last 317.45) - open/close 58596 (time 1740674459.94 total 181.53 last 332.46) - open/close 60000 (time 1740674464.01 total 185.60 last 345.00) - open/close 63234 (time 1740674474.01 total 195.60 last 323.37) total: 64768 open/close in 199.97 seconds: 323.88 ops/second - /unlink 5483 (time 1740674489.54 total 10.00 last 548.29) - /unlink 10000 (time 1740674497.12 total 17.57 last 596.59) - /unlink 14921 (time 1740674507.12 total 27.57 last 492.08) - /unlink 20000 (time 1740674515.47 total 35.93 last 607.90) - /unlink 25641 (time 1740674525.47 total 45.93 last 564.05) - /unlink 30000 (time 1740674530.96 total 51.42 last 793.71) - /unlink 37829 (time 1740674540.97 total 61.42 last 782.83) - /unlink 40000 (time 1740674544.41 total 64.87 last 629.40) - /unlink 49401 (time 1740674554.41 total 74.87 last 940.06) - /unlink 58257 (time 1740674564.42 total 84.87 last 885.54) total: 64768 /unlink in 91.98 seconds: 704.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg131-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg131-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg131-server stop mds service on oleg131-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg131-server unloading modules via unload_modules_local on: 'oleg131-server' oleg131-server: oleg131-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-client: ssh exited with exit code 2 pdsh@oleg131-client: oleg131-server: ssh exited with exit code 2