== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:29:35 (1748867375) Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f) oleg417-server: oleg417-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/pcc_cpufreq/refcnt': 'No such device' oleg417-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg417-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1318 (time 1748867426.95 total 10.00 last 131.75) - open/close 2588 (time 1748867436.95 total 20.00 last 126.98) - open/close 3926 (time 1748867446.96 total 30.02 last 133.65) - open/close 5257 (time 1748867456.97 total 40.02 last 133.05) - open/close 6567 (time 1748867466.97 total 50.03 last 130.93) - open/close 8094 (time 1748867476.98 total 60.03 last 152.62) - open/close 9387 (time 1748867486.98 total 70.03 last 129.28) - open/close 10000 (time 1748867491.95 total 75.00 last 123.37) - open/close 11128 (time 1748867501.96 total 85.01 last 112.71) - open/close 12370 (time 1748867511.96 total 95.01 last 124.18) - open/close 13676 (time 1748867521.96 total 105.01 last 130.56) - open/close 14956 (time 1748867531.96 total 115.02 last 127.96) - open/close 16274 (time 1748867541.97 total 125.02 last 131.74) - open/close 17581 (time 1748867551.97 total 135.02 last 130.65) - open/close 18919 (time 1748867561.98 total 145.03 last 133.67) - open/close 20000 (time 1748867568.72 total 151.77 last 160.49) - open/close 21829 (time 1748867578.72 total 161.77 last 182.84) - open/close 23519 (time 1748867588.72 total 171.77 last 168.98) - open/close 25069 (time 1748867598.72 total 181.77 last 154.98) - open/close 26542 (time 1748867608.72 total 191.78 last 147.30) - open/close 27944 (time 1748867618.73 total 201.78 last 140.16) - open/close 29398 (time 1748867628.73 total 211.78 last 145.33) - open/close 30000 (time 1748867632.78 total 215.83 last 148.66) - open/close 31681 (time 1748867642.79 total 225.84 last 167.93) - open/close 33373 (time 1748867652.80 total 235.85 last 169.04) - open/close 34857 (time 1748867662.80 total 245.85 last 148.39) - open/close 36440 (time 1748867672.80 total 255.85 last 158.27) - open/close 38089 (time 1748867682.81 total 265.86 last 164.80) - open/close 39638 (time 1748867692.81 total 275.86 last 154.87) - open/close 40000 (time 1748867695.52 total 278.57 last 133.85) - open/close 41408 (time 1748867705.52 total 288.57 last 140.76) - open/close 42819 (time 1748867715.53 total 298.58 last 140.95) - open/close 44165 (time 1748867725.53 total 308.58 last 134.59) - open/close 45284 (time 1748867735.53 total 318.58 last 111.88) - open/close 46444 (time 1748867745.53 total 328.59 last 115.96) - open/close 47668 (time 1748867755.54 total 338.60 last 122.29) - open/close 48754 (time 1748867765.55 total 348.60 last 108.56) - open/close 49785 (time 1748867775.57 total 358.62 last 102.88) - open/close 50000 (time 1748867777.57 total 360.62 last 107.42) - open/close 51141 (time 1748867787.57 total 370.62 last 114.07) - open/close 52313 (time 1748867797.57 total 380.63 last 117.18) - open/close 53625 (time 1748867807.58 total 390.63 last 131.12) - open/close 54941 (time 1748867817.58 total 400.63 last 131.59) - open/close 56364 (time 1748867827.58 total 410.64 last 142.25) - open/close 57837 (time 1748867837.60 total 420.65 last 147.13) - open/close 59126 (time 1748867847.60 total 430.65 last 128.88) - open/close 60000 (time 1748867853.43 total 436.48 last 149.94) - open/close 61709 (time 1748867863.43 total 446.48 last 170.81) - open/close 63271 (time 1748867873.43 total 456.49 last 156.15) total: 64768 open/close in 466.19 seconds: 138.93 ops/second - /unlink 2795 (time 1748867894.58 total 10.00 last 279.44) - /unlink 5906 (time 1748867904.58 total 20.00 last 311.07) - /unlink 9115 (time 1748867914.58 total 30.00 last 320.86) - /unlink 10000 (time 1748867917.34 total 32.77 last 320.41) - /unlink 13184 (time 1748867927.34 total 42.77 last 318.34) - /unlink 15461 (time 1748867937.35 total 52.77 last 227.70) - /unlink 18256 (time 1748867947.35 total 62.77 last 279.43) - /unlink 20000 (time 1748867952.87 total 68.29 last 315.85) - /unlink 23238 (time 1748867962.87 total 78.29 last 323.74) - /unlink 26230 (time 1748867972.87 total 88.30 last 299.11) - /unlink 28994 (time 1748867982.87 total 98.30 last 276.39) - /unlink 30000 (time 1748867986.25 total 101.68 last 297.70) - /unlink 34096 (time 1748867996.26 total 111.68 last 409.54) - /unlink 38149 (time 1748868006.26 total 121.68 last 405.22) - /unlink 40000 (time 1748868011.05 total 126.47 last 386.50) - /unlink 44631 (time 1748868021.05 total 136.47 last 463.09) - /unlink 50000 (time 1748868030.83 total 146.26 last 548.59) - /unlink 54858 (time 1748868040.83 total 156.26 last 485.75) - /unlink 59330 (time 1748868050.84 total 166.26 last 447.06) - /unlink 64064 (time 1748868060.84 total 176.26 last 473.39) total: 64768 /unlink in 177.91 seconds: 364.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2132 (time 1748868089.42 total 10.01 last 213.09) - open/close 4013 (time 1748868099.43 total 20.01 last 188.04) - open/close 6439 (time 1748868109.43 total 30.01 last 242.55) - open/close 9206 (time 1748868119.43 total 40.01 last 276.69) - open/close 10000 (time 1748868121.90 total 42.48 last 321.11) - open/close 12641 (time 1748868131.90 total 52.48 last 264.07) - open/close 15243 (time 1748868141.91 total 62.49 last 260.11) - open/close 18016 (time 1748868151.91 total 72.49 last 277.23) - open/close 20000 (time 1748868157.03 total 77.61 last 387.17) - open/close 23583 (time 1748868167.03 total 87.62 last 358.20) - open/close 26253 (time 1748868177.04 total 97.62 last 266.95) - open/close 29097 (time 1748868187.04 total 107.62 last 284.36) - open/close 30000 (time 1748868190.77 total 111.35 last 242.16) - open/close 32980 (time 1748868200.77 total 121.35 last 297.96) - open/close 36315 (time 1748868210.77 total 131.35 last 333.42) - open/close 39309 (time 1748868220.77 total 141.36 last 299.34) - open/close 40000 (time 1748868222.89 total 143.48 last 325.68) - open/close 43309 (time 1748868232.90 total 153.48 last 330.84) - open/close 46439 (time 1748868242.90 total 163.48 last 312.97) - open/close 50000 (time 1748868251.26 total 171.84 last 425.79) - open/close 54482 (time 1748868261.26 total 181.85 last 448.04) - open/close 58038 (time 1748868271.26 total 191.85 last 355.59) - open/close 60000 (time 1748868275.16 total 195.74 last 503.62) - open/close 64711 (time 1748868285.16 total 205.74 last 471.03) total: 64768 open/close in 205.84 seconds: 314.65 ops/second - /unlink 8554 (time 1748868296.12 total 10.00 last 855.38) - /unlink 15958 (time 1748868306.12 total 20.00 last 740.32) - /unlink 20000 (time 1748868312.24 total 26.13 last 659.72) - /unlink 27204 (time 1748868322.24 total 36.13 last 720.38) - /unlink 30000 (time 1748868326.23 total 40.11 last 701.59) - /unlink 36496 (time 1748868336.23 total 50.11 last 649.57) - /unlink 40000 (time 1748868340.27 total 54.16 last 866.43) - /unlink 48706 (time 1748868350.27 total 64.16 last 870.56) - /unlink 57908 (time 1748868360.28 total 74.16 last 920.11) - /unlink 60000 (time 1748868362.58 total 76.47 last 905.95) total: 64768 /unlink in 81.80 seconds: 791.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3093 (time 1748868399.08 total 10.00 last 309.25) - open/close 6934 (time 1748868409.08 total 20.00 last 384.01) - open/close 10000 (time 1748868415.47 total 26.38 last 480.51) - open/close 14454 (time 1748868425.47 total 36.39 last 445.32) - open/close 19292 (time 1748868435.47 total 46.39 last 483.75) - open/close 24005 (time 1748868445.47 total 56.39 last 471.29) - open/close 28594 (time 1748868455.47 total 66.39 last 458.82) - open/close 30000 (time 1748868458.53 total 69.45 last 459.82) - open/close 34791 (time 1748868468.53 total 79.45 last 479.08) - open/close 39263 (time 1748868478.53 total 89.45 last 447.14) - open/close 43693 (time 1748868488.53 total 99.45 last 442.95) - open/close 47323 (time 1748868498.53 total 109.45 last 362.96) - open/close 50000 (time 1748868505.63 total 116.55 last 376.98) - open/close 54454 (time 1748868515.63 total 126.55 last 445.35) - open/close 58808 (time 1748868525.64 total 136.55 last 435.32) - open/close 60000 (time 1748868528.57 total 139.49 last 405.97) - open/close 64139 (time 1748868538.57 total 149.49 last 413.83) total: 64768 open/close in 151.30 seconds: 428.06 ops/second - /unlink 7216 (time 1748868551.35 total 10.00 last 721.56) - /unlink 10000 (time 1748868554.91 total 13.56 last 782.02) - /unlink 17780 (time 1748868564.92 total 23.56 last 777.92) - /unlink 20000 (time 1748868567.62 total 26.27 last 820.42) - /unlink 28138 (time 1748868577.62 total 36.27 last 813.75) - /unlink 30000 (time 1748868579.86 total 38.50 last 832.58) - /unlink 38668 (time 1748868589.86 total 48.51 last 866.75) - /unlink 46895 (time 1748868599.86 total 58.51 last 822.61) - /unlink 50000 (time 1748868603.75 total 62.40 last 798.12) - /unlink 58453 (time 1748868613.75 total 72.40 last 845.30) total: 64768 /unlink in 79.91 seconds: 810.55 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4820 (time 1748868654.73 total 10.00 last 481.95) - open/close 9194 (time 1748868664.74 total 20.00 last 437.31) - open/close 13484 (time 1748868674.78 total 30.05 last 427.10) - open/close 17788 (time 1748868684.78 total 40.05 last 430.39) - open/close 20000 (time 1748868689.66 total 44.93 last 453.41) - open/close 24233 (time 1748868699.66 total 54.93 last 423.29) - open/close 28639 (time 1748868709.66 total 64.93 last 440.59) - open/close 30000 (time 1748868712.60 total 67.87 last 462.79) - open/close 34820 (time 1748868722.60 total 77.87 last 481.94) - open/close 39760 (time 1748868732.60 total 87.87 last 493.94) - open/close 44796 (time 1748868742.60 total 97.87 last 503.59) - open/close 49774 (time 1748868752.60 total 107.87 last 497.79) - open/close 54685 (time 1748868762.61 total 117.87 last 491.05) - open/close 59435 (time 1748868772.61 total 127.87 last 474.92) - open/close 64247 (time 1748868782.61 total 137.87 last 481.15) total: 64768 open/close in 139.03 seconds: 465.86 ops/second - /unlink 8298 (time 1748868794.62 total 10.00 last 829.75) - /unlink 16780 (time 1748868804.62 total 20.00 last 848.19) - /unlink 20000 (time 1748868808.45 total 23.83 last 840.81) - /unlink 28498 (time 1748868818.45 total 33.83 last 849.78) - /unlink 36444 (time 1748868828.45 total 43.83 last 794.53) - /unlink 40000 (time 1748868832.64 total 48.02 last 849.74) - /unlink 48524 (time 1748868842.64 total 58.02 last 852.40) - /unlink 56616 (time 1748868852.64 total 68.02 last 809.19) - /unlink 60000 (time 1748868856.77 total 72.15 last 819.39) total: 64768 /unlink in 77.71 seconds: 833.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4828 (time 1748868895.32 total 10.00 last 482.73) - open/close 9562 (time 1748868905.32 total 20.00 last 473.37) - open/close 14173 (time 1748868915.32 total 30.00 last 461.10) - open/close 18885 (time 1748868925.32 total 40.00 last 471.15) - open/close 24277 (time 1748868935.32 total 50.01 last 539.08) - open/close 29292 (time 1748868945.32 total 60.01 last 501.47) - open/close 34151 (time 1748868955.32 total 70.01 last 485.81) - open/close 39155 (time 1748868965.32 total 80.01 last 500.32) - open/close 44225 (time 1748868975.33 total 90.01 last 506.86) - open/close 49094 (time 1748868985.33 total 100.01 last 486.90) - open/close 53699 (time 1748868995.33 total 110.01 last 460.49) - open/close 57663 (time 1748869005.33 total 120.01 last 396.35) - open/close 60000 (time 1748869011.48 total 126.16 last 380.20) total: 64768 open/close in 135.83 seconds: 476.82 ops/second - /unlink 8553 (time 1748869031.95 total 10.00 last 855.22) - /unlink 17413 (time 1748869041.95 total 20.00 last 885.94) - /unlink 20000 (time 1748869044.71 total 22.76 last 937.40) - /unlink 28876 (time 1748869054.71 total 32.76 last 887.54) - /unlink 37865 (time 1748869064.71 total 42.76 last 898.86) - /unlink 40000 (time 1748869067.09 total 45.15 last 895.45) - /unlink 47995 (time 1748869077.10 total 55.15 last 799.42) - /unlink 50000 (time 1748869079.37 total 57.43 last 879.93) - /unlink 59044 (time 1748869089.37 total 67.43 last 904.31) total: 64768 /unlink in 73.70 seconds: 878.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2