== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:32:09 (1749605529) Stopping clients: oleg141-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg141-client.virtnet /mnt/lustre2 (opts:-f) oleg141-server: oleg141-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing load_modules_local oleg141-server: Loading modules from /home/green/git/lustre-release/lustre oleg141-server: detected 4 online CPUs by sysfs oleg141-server: MODOPTS_LIBCFS= oleg141-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg141-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg141-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg141-server: oleg141-server.virtnet: executing set_default_debug -1 all pdsh@oleg141-client: oleg141-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg141-client.virtnet: -o user_xattr,flock 192.168.201.141@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3976 (time 1749605566.70 total 10.00 last 397.49) - open/close 7776 (time 1749605576.70 total 20.01 last 379.89) - open/close 10000 (time 1749605582.16 total 25.46 last 407.40) - open/close 14162 (time 1749605592.16 total 35.47 last 416.17) - open/close 18329 (time 1749605602.16 total 45.47 last 416.67) - open/close 20000 (time 1749605606.93 total 50.23 last 350.85) - open/close 23846 (time 1749605616.93 total 60.23 last 384.46) - open/close 28744 (time 1749605626.93 total 70.23 last 489.72) - open/close 30000 (time 1749605629.73 total 73.03 last 448.51) - open/close 33756 (time 1749605639.73 total 83.04 last 375.58) - open/close 39027 (time 1749605649.73 total 93.04 last 527.07) - open/close 43585 (time 1749605659.73 total 103.04 last 455.71) - open/close 48387 (time 1749605669.74 total 113.04 last 480.17) - open/close 50000 (time 1749605673.20 total 116.50 last 465.96) - open/close 54917 (time 1749605683.20 total 126.50 last 491.62) - open/close 59103 (time 1749605693.20 total 136.50 last 418.58) - open/close 63620 (time 1749605703.20 total 146.50 last 451.62) total: 64768 open/close in 150.96 seconds: 429.05 ops/second - /unlink 5099 (time 1749605718.89 total 10.00 last 509.83) - /unlink 10000 (time 1749605728.14 total 19.24 last 530.30) - /unlink 15919 (time 1749605738.14 total 29.24 last 591.88) - /unlink 20000 (time 1749605746.86 total 37.97 last 467.66) - /unlink 26359 (time 1749605756.86 total 47.97 last 635.77) - /unlink 30000 (time 1749605762.50 total 53.61 last 646.10) - /unlink 37373 (time 1749605772.50 total 63.61 last 737.25) - /unlink 40000 (time 1749605776.40 total 67.51 last 673.48) - /unlink 48070 (time 1749605786.40 total 77.51 last 806.92) - /unlink 50000 (time 1749605788.60 total 79.71 last 878.92) - /unlink 59197 (time 1749605798.60 total 89.71 last 919.63) total: 64768 /unlink in 95.77 seconds: 676.31 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5547 (time 1749605834.30 total 10.00 last 554.68) - open/close 10000 (time 1749605843.42 total 19.11 last 488.58) - open/close 14651 (time 1749605853.42 total 29.12 last 465.07) - open/close 19255 (time 1749605863.42 total 39.12 last 460.32) - open/close 24199 (time 1749605873.42 total 49.12 last 494.37) - open/close 28946 (time 1749605883.42 total 59.12 last 474.67) - open/close 30000 (time 1749605885.50 total 61.20 last 505.56) - open/close 34522 (time 1749605895.51 total 71.20 last 452.12) - open/close 39183 (time 1749605905.51 total 81.20 last 466.09) - open/close 43613 (time 1749605915.51 total 91.20 last 443.00) - open/close 47797 (time 1749605925.51 total 101.20 last 418.40) - open/close 50000 (time 1749605930.93 total 106.63 last 406.24) - open/close 53950 (time 1749605940.93 total 116.63 last 394.99) - open/close 57761 (time 1749605950.93 total 126.63 last 381.08) - open/close 60000 (time 1749605956.71 total 132.41 last 387.17) - open/close 63314 (time 1749605966.71 total 142.41 last 331.34) total: 64768 open/close in 146.20 seconds: 443.01 ops/second - /unlink 6857 (time 1749605981.54 total 10.00 last 685.65) - /unlink 10000 (time 1749605986.16 total 14.61 last 681.66) - /unlink 17380 (time 1749605996.16 total 24.61 last 737.99) - /unlink 20000 (time 1749606000.78 total 29.23 last 566.88) - /unlink 25566 (time 1749606010.78 total 39.23 last 556.51) - /unlink 30000 (time 1749606018.15 total 46.61 last 601.53) - /unlink 35576 (time 1749606028.15 total 56.61 last 557.59) - /unlink 40000 (time 1749606037.34 total 65.80 last 481.45) - /unlink 44843 (time 1749606047.34 total 75.80 last 484.21) - /unlink 50000 (time 1749606056.93 total 85.39 last 537.62) - /unlink 55182 (time 1749606066.93 total 95.39 last 518.14) - /unlink 60000 (time 1749606073.57 total 102.03 last 725.65) total: 64768 /unlink in 110.16 seconds: 587.94 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3220 (time 1749606109.60 total 10.00 last 321.96) - open/close 6770 (time 1749606119.60 total 20.00 last 354.96) - open/close 10000 (time 1749606127.33 total 27.73 last 417.90) - open/close 13334 (time 1749606137.33 total 37.73 last 333.37) - open/close 17718 (time 1749606147.33 total 47.73 last 438.35) - open/close 20000 (time 1749606156.10 total 56.50 last 260.16) - open/close 24250 (time 1749606166.10 total 66.51 last 424.96) - open/close 28015 (time 1749606176.11 total 76.51 last 376.41) - open/close 30000 (time 1749606180.21 total 80.61 last 483.76) - open/close 33858 (time 1749606190.21 total 90.61 last 385.73) - open/close 37309 (time 1749606200.21 total 100.62 last 344.99) - open/close 40000 (time 1749606207.75 total 108.15 last 357.23) - open/close 44146 (time 1749606217.75 total 118.15 last 414.60) - open/close 48620 (time 1749606227.75 total 128.15 last 447.29) - open/close 50000 (time 1749606231.80 total 132.21 last 340.36) - open/close 52779 (time 1749606241.81 total 142.21 last 277.85) - open/close 55520 (time 1749606251.81 total 152.21 last 274.07) - open/close 58359 (time 1749606261.81 total 162.21 last 283.88) - open/close 60000 (time 1749606267.81 total 168.21 last 273.58) - open/close 62809 (time 1749606277.81 total 178.21 last 280.90) total: 64768 open/close in 185.37 seconds: 349.40 ops/second - /unlink 4712 (time 1749606295.78 total 10.00 last 471.11) - /unlink 9353 (time 1749606305.78 total 20.00 last 464.00) - /unlink 14031 (time 1749606315.78 total 30.00 last 467.75) - /unlink 19046 (time 1749606325.78 total 40.01 last 501.45) - /unlink 20000 (time 1749606327.93 total 42.15 last 445.74) - /unlink 26301 (time 1749606337.93 total 52.15 last 629.98) - /unlink 30000 (time 1749606343.88 total 58.10 last 621.33) - /unlink 34971 (time 1749606353.88 total 68.10 last 497.05) - /unlink 40000 (time 1749606362.51 total 76.73 last 583.02) - /unlink 45198 (time 1749606372.51 total 86.73 last 519.73) - /unlink 50000 (time 1749606379.81 total 94.03 last 657.46) - /unlink 56044 (time 1749606389.81 total 104.03 last 604.35) - /unlink 60000 (time 1749606397.94 total 112.16 last 486.75) total: 64768 /unlink in 119.48 seconds: 542.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3771 (time 1749606435.13 total 10.00 last 377.09) - open/close 6833 (time 1749606445.13 total 20.00 last 306.11) - open/close 9713 (time 1749606455.13 total 30.00 last 287.97) - open/close 12394 (time 1749606465.14 total 40.01 last 268.03) - open/close 14956 (time 1749606475.14 total 50.01 last 256.19) - open/close 17516 (time 1749606485.14 total 60.01 last 255.92) - open/close 20000 (time 1749606494.45 total 69.32 last 266.84) - open/close 22875 (time 1749606504.45 total 79.32 last 287.40) - open/close 26171 (time 1749606514.45 total 89.32 last 329.57) - open/close 29212 (time 1749606524.46 total 99.33 last 304.05) - open/close 30000 (time 1749606527.51 total 102.38 last 257.75) - open/close 33020 (time 1749606537.51 total 112.38 last 301.97) - open/close 35927 (time 1749606547.52 total 122.39 last 290.58) - open/close 38500 (time 1749606557.52 total 132.39 last 257.25) - open/close 40000 (time 1749606563.24 total 138.11 last 262.35) - open/close 42934 (time 1749606573.24 total 148.11 last 293.34) - open/close 45799 (time 1749606583.24 total 158.11 last 286.41) - open/close 48633 (time 1749606593.25 total 168.11 last 283.34) - open/close 50000 (time 1749606598.51 total 173.38 last 259.53) - open/close 52913 (time 1749606608.52 total 183.39 last 291.20) - open/close 55413 (time 1749606618.52 total 193.39 last 249.93) - open/close 58282 (time 1749606628.52 total 203.39 last 286.80) - open/close 60000 (time 1749606634.48 total 209.35 last 288.24) - open/close 63061 (time 1749606644.48 total 219.35 last 306.06) total: 64768 open/close in 224.81 seconds: 288.10 ops/second - /unlink 5276 (time 1749606661.17 total 10.00 last 527.59) - /unlink 9901 (time 1749606671.18 total 20.00 last 462.43) - /unlink 14498 (time 1749606681.18 total 30.00 last 459.65) - /unlink 19032 (time 1749606691.18 total 40.00 last 453.37) - /unlink 20000 (time 1749606693.39 total 42.21 last 438.32) - /unlink 24679 (time 1749606703.39 total 52.21 last 467.81) - /unlink 29262 (time 1749606713.39 total 62.21 last 458.26) - /unlink 33899 (time 1749606723.39 total 72.22 last 463.63) - /unlink 38463 (time 1749606733.39 total 82.22 last 456.36) - /unlink 40000 (time 1749606736.67 total 85.50 last 468.57) - /unlink 44551 (time 1749606746.67 total 95.50 last 455.00) - /unlink 48990 (time 1749606756.68 total 105.50 last 443.86) - /unlink 50000 (time 1749606759.04 total 107.87 last 426.81) - /unlink 54561 (time 1749606769.04 total 117.87 last 456.04) - /unlink 59277 (time 1749606779.04 total 127.87 last 471.51) - /unlink 64092 (time 1749606789.04 total 137.87 last 481.49) total: 64768 /unlink in 139.07 seconds: 465.73 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3371 (time 1749606815.69 total 10.00 last 336.98) - open/close 8040 (time 1749606825.69 total 20.00 last 466.84) - open/close 10000 (time 1749606829.74 total 24.06 last 483.46) - open/close 15005 (time 1749606839.74 total 34.06 last 500.41) - open/close 20000 (time 1749606849.30 total 43.62 last 522.79) - open/close 25574 (time 1749606859.30 total 53.62 last 557.33) - open/close 30000 (time 1749606867.33 total 61.65 last 551.12) - open/close 35455 (time 1749606877.33 total 71.65 last 545.41) - open/close 40000 (time 1749606885.40 total 79.72 last 563.24) - open/close 45376 (time 1749606895.40 total 89.72 last 537.58) - open/close 50000 (time 1749606903.94 total 98.26 last 541.37) - open/close 54346 (time 1749606913.94 total 108.26 last 434.55) - open/close 59921 (time 1749606923.94 total 118.26 last 557.49) total: 64768 open/close in 126.93 seconds: 510.28 ops/second - /unlink 8012 (time 1749606943.35 total 10.00 last 801.19) - /unlink 10000 (time 1749606946.12 total 12.77 last 718.22) - /unlink 18551 (time 1749606956.12 total 22.77 last 855.08) - /unlink 27813 (time 1749606966.12 total 32.77 last 926.18) - /unlink 30000 (time 1749606969.08 total 35.73 last 738.92) - /unlink 37928 (time 1749606979.08 total 45.73 last 792.74) - /unlink 40000 (time 1749606981.39 total 48.04 last 897.80) - /unlink 48065 (time 1749606991.39 total 58.04 last 806.42) - /unlink 50000 (time 1749606993.39 total 60.04 last 964.63) - /unlink 58695 (time 1749607003.40 total 70.04 last 869.41) total: 64768 /unlink in 76.30 seconds: 848.86 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg141-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg141-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg141-server stop mds service on oleg141-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg141-server unloading modules via unload_modules_local on: 'oleg141-server' oleg141-server: oleg141-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-client: ssh exited with exit code 2 pdsh@oleg141-client: oleg141-server: ssh exited with exit code 2