== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:13:32 (1749478412) Stopping clients: oleg310-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg310-client.virtnet /mnt/lustre2 (opts:-f) oleg310-server: oleg310-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing load_modules_local oleg310-server: Loading modules from /home/green/git/lustre-release/lustre oleg310-server: detected 4 online CPUs by sysfs oleg310-server: MODOPTS_LIBCFS= oleg310-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg310-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg310-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg310-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg310-server: oleg310-server.virtnet: executing set_default_debug -1 all pdsh@oleg310-client: oleg310-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg310-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg310-client.virtnet: -o user_xattr,flock 192.168.203.110@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5359 (time 1749478447.51 total 10.00 last 535.87) - open/close 10000 (time 1749478456.47 total 18.97 last 517.62) - open/close 16286 (time 1749478466.47 total 28.97 last 628.54) - open/close 20000 (time 1749478472.74 total 35.23 last 593.00) - open/close 25608 (time 1749478482.74 total 45.23 last 560.78) - open/close 30000 (time 1749478489.92 total 52.41 last 611.53) - open/close 35432 (time 1749478499.92 total 62.41 last 543.10) - open/close 40000 (time 1749478508.56 total 71.05 last 528.83) - open/close 45321 (time 1749478518.56 total 81.05 last 532.09) - open/close 50000 (time 1749478527.46 total 89.95 last 525.89) - open/close 55237 (time 1749478537.46 total 99.95 last 523.69) - open/close 60000 (time 1749478546.67 total 109.16 last 517.16) total: 64768 open/close in 118.44 seconds: 546.84 ops/second - /unlink 8696 (time 1749478566.80 total 10.00 last 869.50) - /unlink 17129 (time 1749478576.80 total 20.00 last 843.24) - /unlink 20000 (time 1749478581.45 total 24.65 last 617.90) - /unlink 27894 (time 1749478591.45 total 34.65 last 789.33) - /unlink 30000 (time 1749478593.86 total 37.07 last 871.62) - /unlink 38789 (time 1749478603.86 total 47.07 last 878.80) - /unlink 47696 (time 1749478613.86 total 57.07 last 890.70) - /unlink 50000 (time 1749478616.47 total 59.67 last 884.03) - /unlink 59087 (time 1749478626.47 total 69.67 last 908.69) total: 64768 /unlink in 75.76 seconds: 854.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5420 (time 1749478666.24 total 10.04 last 539.77) - open/close 10000 (time 1749478674.75 total 18.56 last 537.94) - open/close 15129 (time 1749478684.75 total 28.56 last 512.87) - open/close 20000 (time 1749478694.26 total 38.06 last 512.42) - open/close 25271 (time 1749478704.26 total 48.06 last 527.08) - open/close 30000 (time 1749478713.21 total 57.01 last 528.46) - open/close 35125 (time 1749478723.21 total 67.01 last 512.45) - open/close 40000 (time 1749478732.66 total 76.47 last 515.67) - open/close 45174 (time 1749478742.67 total 86.47 last 517.31) - open/close 50000 (time 1749478752.64 total 96.44 last 483.71) - open/close 54270 (time 1749478762.65 total 106.45 last 426.89) - open/close 59085 (time 1749478772.65 total 116.45 last 481.42) - open/close 64228 (time 1749478782.65 total 126.45 last 514.25) total: 64768 open/close in 127.57 seconds: 507.72 ops/second - /unlink 8846 (time 1749478794.60 total 10.00 last 884.52) - /unlink 17726 (time 1749478804.61 total 20.00 last 887.88) - /unlink 20000 (time 1749478807.03 total 22.43 last 937.83) - /unlink 28873 (time 1749478817.03 total 32.43 last 887.25) - /unlink 38007 (time 1749478827.03 total 42.43 last 913.37) - /unlink 40000 (time 1749478829.26 total 44.66 last 893.00) - /unlink 49072 (time 1749478839.26 total 54.66 last 907.17) - /unlink 58302 (time 1749478849.26 total 64.66 last 922.92) total: 64768 /unlink in 71.57 seconds: 905.00 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5275 (time 1749478891.40 total 10.00 last 527.45) - open/close 9168 (time 1749478901.40 total 20.00 last 389.29) - open/close 10000 (time 1749478903.66 total 22.26 last 368.78) - open/close 14807 (time 1749478913.66 total 32.26 last 480.61) - open/close 18810 (time 1749478923.66 total 42.26 last 400.25) - open/close 20000 (time 1749478927.16 total 45.75 last 340.61) - open/close 23170 (time 1749478937.16 total 55.76 last 316.93) - open/close 27012 (time 1749478947.16 total 65.76 last 384.12) - open/close 30000 (time 1749478953.35 total 71.95 last 482.77) - open/close 34891 (time 1749478963.35 total 81.95 last 489.05) - open/close 39380 (time 1749478973.35 total 91.95 last 448.85) - open/close 44525 (time 1749478983.35 total 101.95 last 514.45) - open/close 49155 (time 1749478993.35 total 111.95 last 462.97) - open/close 53650 (time 1749479003.36 total 121.95 last 449.36) - open/close 58710 (time 1749479013.36 total 131.96 last 505.91) - open/close 60000 (time 1749479016.77 total 135.37 last 377.74) - open/close 63753 (time 1749479026.78 total 145.37 last 375.23) total: 64768 open/close in 147.49 seconds: 439.14 ops/second - /unlink 7994 (time 1749479039.68 total 10.00 last 799.32) - /unlink 10000 (time 1749479042.06 total 12.39 last 840.86) - /unlink 18244 (time 1749479052.06 total 22.39 last 824.36) - /unlink 20000 (time 1749479054.44 total 24.76 last 740.29) - /unlink 27020 (time 1749479064.44 total 34.76 last 701.98) - /unlink 30000 (time 1749479068.33 total 38.66 last 764.62) - /unlink 39841 (time 1749479078.33 total 48.66 last 984.06) - /unlink 48750 (time 1749479088.33 total 58.66 last 890.89) - /unlink 57187 (time 1749479098.33 total 68.66 last 843.69) - /unlink 60000 (time 1749479101.72 total 72.04 last 830.76) total: 64768 /unlink in 77.41 seconds: 836.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5193 (time 1749479141.61 total 10.00 last 519.19) - open/close 10000 (time 1749479151.23 total 19.63 last 499.47) - open/close 14780 (time 1749479161.24 total 29.63 last 477.93) - open/close 19391 (time 1749479171.24 total 39.63 last 461.04) - open/close 24005 (time 1749479181.24 total 49.63 last 461.39) - open/close 28589 (time 1749479191.24 total 59.63 last 458.36) - open/close 30000 (time 1749479194.17 total 62.56 last 481.09) - open/close 34665 (time 1749479204.17 total 72.56 last 466.45) - open/close 39020 (time 1749479214.19 total 82.58 last 434.68) - open/close 40000 (time 1749479216.28 total 84.67 last 469.99) - open/close 44782 (time 1749479226.28 total 94.67 last 478.13) - open/close 49413 (time 1749479236.28 total 104.67 last 463.03) - open/close 54198 (time 1749479246.28 total 114.67 last 478.45) - open/close 58996 (time 1749479256.28 total 124.67 last 479.74) - open/close 60000 (time 1749479258.57 total 126.96 last 439.59) total: 64768 open/close in 136.46 seconds: 474.63 ops/second - /unlink 8665 (time 1749479278.97 total 10.00 last 866.42) - /unlink 17222 (time 1749479288.97 total 20.00 last 855.65) - /unlink 20000 (time 1749479292.20 total 23.23 last 859.38) - /unlink 28333 (time 1749479302.20 total 33.23 last 833.26) - /unlink 36937 (time 1749479312.20 total 43.23 last 860.38) - /unlink 40000 (time 1749479315.84 total 46.87 last 842.56) - /unlink 48718 (time 1749479325.84 total 56.87 last 871.77) - /unlink 57256 (time 1749479335.84 total 66.87 last 853.70) - /unlink 60000 (time 1749479338.84 total 69.87 last 915.77) total: 64768 /unlink in 75.11 seconds: 862.33 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5279 (time 1749479377.40 total 10.00 last 527.80) - open/close 10000 (time 1749479386.31 total 18.91 last 530.03) - open/close 15266 (time 1749479396.31 total 28.91 last 526.58) - open/close 20000 (time 1749479405.32 total 37.92 last 525.25) - open/close 24903 (time 1749479415.32 total 47.92 last 490.25) - open/close 30000 (time 1749479425.15 total 57.75 last 518.75) - open/close 35355 (time 1749479435.15 total 67.75 last 535.48) - open/close 39821 (time 1749479445.15 total 77.75 last 446.53) - open/close 43876 (time 1749479455.15 total 87.75 last 405.46) - open/close 48094 (time 1749479465.15 total 97.75 last 421.79) - open/close 50000 (time 1749479468.67 total 101.28 last 540.75) - open/close 55022 (time 1749479478.68 total 111.28 last 502.13) - open/close 59861 (time 1749479488.68 total 121.28 last 483.80) total: 64768 open/close in 130.55 seconds: 496.11 ops/second - /unlink 8836 (time 1749479508.79 total 10.00 last 883.59) - /unlink 17902 (time 1749479518.79 total 20.00 last 906.58) - /unlink 20000 (time 1749479521.31 total 22.52 last 832.20) - /unlink 28872 (time 1749479531.31 total 32.52 last 887.18) - /unlink 37238 (time 1749479541.31 total 42.52 last 836.52) - /unlink 40000 (time 1749479544.30 total 45.51 last 925.19) - /unlink 48840 (time 1749479554.30 total 55.51 last 883.96) - /unlink 57052 (time 1749479564.30 total 65.51 last 821.09) - /unlink 60000 (time 1749479567.45 total 68.66 last 936.66) total: 64768 /unlink in 75.24 seconds: 860.81 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg310-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg310-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg310-server stop mds service on oleg310-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg310-server unloading modules via unload_modules_local on: 'oleg310-server' oleg310-server: oleg310-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-client: ssh exited with exit code 2 pdsh@oleg310-client: oleg310-server: ssh exited with exit code 2