== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:16:23 (1738966583) Stopping clients: oleg133-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg133-client.virtnet /mnt/lustre2 (opts:-f) oleg133-server: oleg133-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing load_modules_local oleg133-server: Loading modules from /home/green/git/lustre-release/lustre oleg133-server: detected 4 online CPUs by sysfs oleg133-server: MODOPTS_LIBCFS= oleg133-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg133-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg133-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg133-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg133-server: oleg133-server.virtnet: executing set_default_debug -1 all pdsh@oleg133-client: oleg133-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg133-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg133-client.virtnet: -o user_xattr,flock 192.168.201.133@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3422 (time 1738966618.29 total 10.00 last 342.17) - open/close 6818 (time 1738966628.29 total 20.00 last 339.56) - open/close 10000 (time 1738966635.12 total 26.83 last 465.89) - open/close 15376 (time 1738966645.12 total 36.83 last 537.58) - open/close 20000 (time 1738966653.47 total 45.18 last 554.22) - open/close 25534 (time 1738966663.47 total 55.18 last 553.36) - open/close 30000 (time 1738966671.61 total 63.32 last 548.17) - open/close 35398 (time 1738966681.62 total 73.32 last 539.72) - open/close 40000 (time 1738966689.88 total 81.59 last 556.67) - open/close 45463 (time 1738966699.88 total 91.59 last 546.26) - open/close 50000 (time 1738966708.25 total 99.96 last 542.20) - open/close 55391 (time 1738966718.25 total 109.96 last 539.06) - open/close 60000 (time 1738966726.78 total 118.49 last 540.40) total: 64768 open/close in 127.66 seconds: 507.34 ops/second - /unlink 9382 (time 1738966746.80 total 10.00 last 938.13) - /unlink 18628 (time 1738966756.80 total 20.00 last 924.54) - /unlink 27923 (time 1738966766.81 total 30.00 last 929.45) - /unlink 30000 (time 1738966768.96 total 32.15 last 964.78) - /unlink 39209 (time 1738966778.96 total 42.16 last 920.82) - /unlink 48331 (time 1738966788.96 total 52.16 last 912.13) - /unlink 57660 (time 1738966798.96 total 62.16 last 932.84) - /unlink 60000 (time 1738966801.43 total 64.63 last 946.89) total: 64768 /unlink in 69.44 seconds: 932.66 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4024 (time 1738966846.08 total 10.00 last 402.32) - open/close 8427 (time 1738966856.08 total 20.00 last 440.26) - open/close 10000 (time 1738966862.53 total 26.45 last 244.16) - open/close 13222 (time 1738966872.53 total 36.45 last 322.18) - open/close 17876 (time 1738966882.53 total 46.45 last 465.39) - open/close 20000 (time 1738966886.80 total 50.72 last 497.11) - open/close 24833 (time 1738966896.80 total 60.72 last 483.22) - open/close 28696 (time 1738966906.80 total 70.72 last 386.28) - open/close 30000 (time 1738966910.08 total 74.00 last 398.23) - open/close 33736 (time 1738966920.08 total 84.00 last 373.58) - open/close 37293 (time 1738966930.08 total 94.00 last 355.65) - open/close 40000 (time 1738966936.59 total 100.51 last 415.58) - open/close 45090 (time 1738966946.59 total 110.51 last 508.96) - open/close 49376 (time 1738966956.60 total 120.51 last 428.50) - open/close 50000 (time 1738966958.86 total 122.78 last 275.30) - open/close 53219 (time 1738966968.91 total 132.83 last 320.47) - open/close 56498 (time 1738966978.91 total 142.83 last 327.78) - open/close 59047 (time 1738966988.91 total 152.83 last 254.84) - open/close 60000 (time 1738966991.57 total 155.48 last 359.13) - open/close 62942 (time 1738967001.57 total 165.49 last 294.18) total: 64768 open/close in 171.82 seconds: 376.96 ops/second - /unlink 6301 (time 1738967019.08 total 10.00 last 630.05) - /unlink 10000 (time 1738967024.69 total 15.61 last 659.64) - /unlink 15494 (time 1738967034.69 total 25.61 last 549.38) - /unlink 20000 (time 1738967041.98 total 32.90 last 618.00) - /unlink 25253 (time 1738967051.98 total 42.90 last 525.28) - /unlink 30000 (time 1738967061.03 total 51.95 last 524.52) - /unlink 35207 (time 1738967071.04 total 61.95 last 520.59) - /unlink 40000 (time 1738967080.34 total 71.26 last 515.00) - /unlink 44889 (time 1738967090.34 total 81.26 last 488.86) - /unlink 49916 (time 1738967100.34 total 91.26 last 502.69) - /unlink 54724 (time 1738967110.34 total 101.26 last 480.78) - /unlink 60000 (time 1738967120.22 total 111.14 last 534.01) total: 64768 /unlink in 118.93 seconds: 544.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3060 (time 1738967156.98 total 10.00 last 305.91) - open/close 5908 (time 1738967166.98 total 20.01 last 284.71) - open/close 9582 (time 1738967176.98 total 30.01 last 367.36) - open/close 12500 (time 1738967186.98 total 40.01 last 291.70) - open/close 15389 (time 1738967196.99 total 50.01 last 288.81) - open/close 18194 (time 1738967206.99 total 60.02 last 280.47) - open/close 20000 (time 1738967213.49 total 66.51 last 277.93) - open/close 22717 (time 1738967223.49 total 76.51 last 271.69) - open/close 25557 (time 1738967233.49 total 86.52 last 283.88) - open/close 28193 (time 1738967243.49 total 96.52 last 263.54) - open/close 30000 (time 1738967250.39 total 103.42 last 261.92) - open/close 32661 (time 1738967260.40 total 113.42 last 266.03) - open/close 35194 (time 1738967270.40 total 123.42 last 253.29) - open/close 37727 (time 1738967280.40 total 133.42 last 253.25) - open/close 40000 (time 1738967289.25 total 142.28 last 256.71) - open/close 42577 (time 1738967299.25 total 152.28 last 257.69) - open/close 45256 (time 1738967309.25 total 162.28 last 267.84) - open/close 48466 (time 1738967319.26 total 172.28 last 320.92) - open/close 50000 (time 1738967325.25 total 178.27 last 256.10) - open/close 52840 (time 1738967335.25 total 188.27 last 283.99) - open/close 55478 (time 1738967345.25 total 198.28 last 263.71) - open/close 58032 (time 1738967355.25 total 208.28 last 255.36) - open/close 60000 (time 1738967362.86 total 215.89 last 258.68) - open/close 62650 (time 1738967372.86 total 225.89 last 264.98) total: 64768 open/close in 233.82 seconds: 277.00 ops/second - /unlink 4786 (time 1738967391.95 total 10.00 last 478.53) - /unlink 9681 (time 1738967401.95 total 20.00 last 489.44) - /unlink 14958 (time 1738967411.95 total 30.00 last 527.69) - /unlink 19943 (time 1738967421.96 total 40.00 last 498.43) - /unlink 25101 (time 1738967431.96 total 50.01 last 515.69) - /unlink 30000 (time 1738967441.53 total 59.58 last 511.93) - /unlink 34926 (time 1738967451.53 total 69.58 last 492.51) - /unlink 39700 (time 1738967461.53 total 79.58 last 477.36) - /unlink 44454 (time 1738967471.53 total 89.58 last 475.36) - /unlink 49146 (time 1738967481.53 total 99.58 last 469.10) - /unlink 54452 (time 1738967491.53 total 109.58 last 530.58) - /unlink 59361 (time 1738967501.53 total 119.58 last 490.88) - /unlink 64421 (time 1738967511.53 total 129.58 last 505.99) total: 64768 /unlink in 130.38 seconds: 496.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2795 (time 1738967537.78 total 10.00 last 279.49) - open/close 5431 (time 1738967547.78 total 20.00 last 263.59) - open/close 7971 (time 1738967557.78 total 30.00 last 253.91) - open/close 10000 (time 1738967565.55 total 37.77 last 261.21) - open/close 12957 (time 1738967575.55 total 47.77 last 295.61) - open/close 15917 (time 1738967585.55 total 57.78 last 295.99) - open/close 18572 (time 1738967595.56 total 67.78 last 265.45) - open/close 20000 (time 1738967601.31 total 73.53 last 248.07) - open/close 22648 (time 1738967611.31 total 83.54 last 264.73) - open/close 25334 (time 1738967621.32 total 93.54 last 268.51) - open/close 27797 (time 1738967631.32 total 103.54 last 246.22) - open/close 30000 (time 1738967640.05 total 112.27 last 252.44) - open/close 32490 (time 1738967650.05 total 122.27 last 248.93) - open/close 35013 (time 1738967660.05 total 132.28 last 252.23) - open/close 37469 (time 1738967670.06 total 142.28 last 245.54) - open/close 40000 (time 1738967678.72 total 150.94 last 292.10) - open/close 43162 (time 1738967688.72 total 160.94 last 316.18) - open/close 45801 (time 1738967698.72 total 170.95 last 263.83) - open/close 48387 (time 1738967708.72 total 180.95 last 258.58) - open/close 50000 (time 1738967715.49 total 187.71 last 238.45) - open/close 52709 (time 1738967725.49 total 197.71 last 270.80) - open/close 55245 (time 1738967735.50 total 207.72 last 253.52) - open/close 57837 (time 1738967745.50 total 217.72 last 259.17) - open/close 60000 (time 1738967752.92 total 225.14 last 291.31) - open/close 62638 (time 1738967762.92 total 235.15 last 263.74) total: 64768 open/close in 243.17 seconds: 266.35 ops/second - /unlink 4637 (time 1738967782.23 total 10.00 last 463.66) - /unlink 9716 (time 1738967792.24 total 20.00 last 507.87) - /unlink 14613 (time 1738967802.24 total 30.00 last 489.67) - /unlink 19475 (time 1738967812.24 total 40.00 last 486.09) - /unlink 24303 (time 1738967822.24 total 50.01 last 482.77) - /unlink 29331 (time 1738967832.24 total 60.01 last 502.75) - /unlink 35292 (time 1738967842.24 total 70.01 last 596.02) - /unlink 40000 (time 1738967850.63 total 78.40 last 561.14) - /unlink 45681 (time 1738967860.63 total 88.40 last 568.08) - /unlink 50000 (time 1738967869.12 total 96.89 last 508.67) - /unlink 57677 (time 1738967879.12 total 106.89 last 767.64) - /unlink 60000 (time 1738967881.97 total 109.74 last 815.09) total: 64768 /unlink in 115.29 seconds: 561.78 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4516 (time 1738967917.96 total 10.00 last 451.59) - open/close 9119 (time 1738967927.96 total 20.00 last 460.29) - open/close 14707 (time 1738967937.96 total 30.00 last 558.75) - open/close 20000 (time 1738967947.88 total 39.92 last 533.43) - open/close 25269 (time 1738967957.88 total 49.92 last 526.84) - open/close 30000 (time 1738967967.19 total 59.23 last 508.52) - open/close 34784 (time 1738967977.19 total 69.23 last 478.35) - open/close 37359 (time 1738967987.19 total 79.23 last 257.44) - open/close 40000 (time 1738967996.17 total 88.21 last 294.21) - open/close 42650 (time 1738968006.17 total 98.21 last 264.99) - open/close 45102 (time 1738968016.17 total 108.21 last 245.15) - open/close 47927 (time 1738968026.17 total 118.21 last 282.45) - open/close 50000 (time 1738968034.03 total 126.07 last 263.75) - open/close 52637 (time 1738968044.03 total 136.07 last 263.64) - open/close 55174 (time 1738968054.03 total 146.08 last 253.68) - open/close 57794 (time 1738968064.03 total 156.08 last 261.98) - open/close 60000 (time 1738968068.71 total 160.75 last 471.65) - open/close 64355 (time 1738968078.71 total 170.76 last 435.34) total: 64768 open/close in 171.86 seconds: 376.87 ops/second - /unlink 4997 (time 1738968090.92 total 10.00 last 499.63) - /unlink 10000 (time 1738968098.57 total 17.65 last 654.21) - /unlink 19307 (time 1738968108.57 total 27.65 last 930.62) - /unlink 27802 (time 1738968118.57 total 37.65 last 849.48) - /unlink 30000 (time 1738968120.88 total 39.96 last 951.32) - /unlink 38653 (time 1738968130.88 total 49.96 last 865.27) - /unlink 46788 (time 1738968140.88 total 59.96 last 813.41) - /unlink 50000 (time 1738968144.74 total 63.82 last 832.15) - /unlink 58930 (time 1738968154.74 total 73.82 last 892.94) total: 64768 /unlink in 83.39 seconds: 776.71 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg133-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg133-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg133-server stop mds service on oleg133-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg133-server unloading modules via unload_modules_local on: 'oleg133-server' oleg133-server: oleg133-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-client: ssh exited with exit code 2 pdsh@oleg133-client: oleg133-server: ssh exited with exit code 2