== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:20:33 (1721406033) Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f) oleg402-server: oleg402-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: detected 4 online CPUs by sysfs oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg402-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg402-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock oleg402-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3479 (time 1721406072.02 total 10.00 last 347.88) - open/close 6856 (time 1721406082.03 total 20.00 last 337.61) - open/close 10000 (time 1721406089.43 total 27.40 last 424.78) - open/close 14500 (time 1721406099.43 total 37.41 last 449.99) - open/close 19060 (time 1721406109.43 total 47.41 last 455.97) - open/close 23284 (time 1721406119.43 total 57.41 last 422.35) - open/close 26317 (time 1721406129.43 total 67.41 last 303.22) - open/close 29420 (time 1721406139.44 total 77.41 last 310.22) - open/close 30000 (time 1721406141.51 total 79.49 last 279.51) - open/close 32592 (time 1721406151.51 total 89.49 last 259.18) - open/close 35468 (time 1721406161.54 total 99.52 last 286.68) - open/close 37915 (time 1721406171.55 total 109.52 last 244.63) - open/close 40000 (time 1721406179.71 total 117.69 last 255.28) - open/close 43196 (time 1721406189.72 total 127.69 last 319.48) - open/close 45690 (time 1721406199.72 total 137.70 last 249.34) - open/close 48581 (time 1721406209.72 total 147.70 last 289.06) - open/close 50000 (time 1721406214.21 total 152.18 last 316.38) - open/close 52833 (time 1721406224.21 total 162.19 last 283.19) - open/close 56115 (time 1721406234.21 total 172.19 last 328.12) - open/close 59261 (time 1721406244.21 total 182.19 last 314.58) - open/close 60000 (time 1721406247.08 total 185.06 last 257.57) - open/close 62825 (time 1721406257.09 total 195.06 last 282.40) total: 64768 open/close in 200.70 seconds: 322.71 ops/second - /unlink 7103 (time 1721406273.59 total 10.00 last 710.30) - /unlink 10000 (time 1721406278.30 total 14.71 last 614.48) - /unlink 15191 (time 1721406288.30 total 24.71 last 519.08) - /unlink 20000 (time 1721406296.33 total 32.74 last 599.23) - /unlink 28064 (time 1721406306.33 total 42.74 last 806.40) - /unlink 30000 (time 1721406309.50 total 45.91 last 610.65) - /unlink 35749 (time 1721406319.50 total 55.91 last 574.89) - /unlink 40000 (time 1721406327.07 total 63.49 last 561.10) - /unlink 45321 (time 1721406337.07 total 73.49 last 532.05) - /unlink 50000 (time 1721406345.62 total 82.03 last 547.74) - /unlink 55608 (time 1721406355.62 total 92.03 last 560.73) - /unlink 60000 (time 1721406363.54 total 99.95 last 554.64) total: 64768 /unlink in 107.60 seconds: 601.92 ops/second - open/close 3341 (time 1721406382.41 total 10.00 last 334.08) - open/close 6862 (time 1721406392.41 total 20.00 last 352.01) - open/close 10000 (time 1721406401.82 total 29.42 last 333.23) - open/close 14503 (time 1721406411.83 total 39.42 last 450.21) - open/close 17214 (time 1721406421.83 total 49.43 last 271.00) - open/close 19930 (time 1721406431.83 total 59.43 last 271.57) - open/close 22592 (time 1721406441.83 total 69.43 last 266.11) - open/close 25711 (time 1721406451.84 total 79.43 last 311.88) - open/close 28632 (time 1721406461.84 total 89.43 last 292.02) - open/close 30000 (time 1721406467.32 total 94.92 last 249.35) - open/close 32770 (time 1721406477.33 total 104.92 last 276.91) - open/close 35767 (time 1721406487.33 total 114.92 last 299.67) - open/close 38787 (time 1721406497.33 total 124.92 last 301.99) - open/close 40000 (time 1721406499.63 total 127.22 last 527.92) - open/close 44185 (time 1721406509.63 total 137.22 last 418.40) - open/close 48414 (time 1721406519.63 total 147.22 last 422.89) - open/close 50000 (time 1721406522.66 total 150.26 last 523.14) - open/close 54703 (time 1721406532.66 total 160.26 last 470.28) - open/close 59982 (time 1721406542.66 total 170.26 last 527.86) - open/close 64521 (time 1721406552.66 total 180.26 last 453.89) total: 64768 open/close in 180.89 seconds: 358.05 ops/second - /unlink 8586 (time 1721406564.28 total 10.00 last 858.54) - /unlink 10000 (time 1721406566.31 total 12.04 last 693.45) - /unlink 16329 (time 1721406576.32 total 22.04 last 632.82) - /unlink 20000 (time 1721406580.56 total 26.29 last 864.28) - /unlink 26555 (time 1721406590.56 total 36.29 last 655.43) - /unlink 30000 (time 1721406595.12 total 40.85 last 755.45) - /unlink 35172 (time 1721406605.12 total 50.85 last 517.18) - /unlink 40000 (time 1721406614.66 total 60.38 last 506.55) - /unlink 45156 (time 1721406624.66 total 70.38 last 515.55) - /unlink 50000 (time 1721406632.82 total 78.54 last 593.61) - /unlink 56216 (time 1721406642.82 total 88.54 last 621.59) - /unlink 60000 (time 1721406649.13 total 94.86 last 599.36) total: 64768 /unlink in 103.03 seconds: 628.65 ops/second - open/close 3168 (time 1721406668.60 total 10.00 last 316.78) - open/close 7773 (time 1721406678.60 total 20.00 last 460.42) - open/close 10000 (time 1721406683.68 total 25.09 last 437.86) - open/close 14028 (time 1721406693.69 total 35.09 last 402.68) - open/close 16862 (time 1721406703.69 total 45.10 last 283.29) - open/close 19720 (time 1721406713.69 total 55.10 last 285.72) - open/close 22626 (time 1721406723.69 total 65.10 last 290.58) - open/close 25869 (time 1721406733.70 total 75.10 last 324.23) - open/close 28609 (time 1721406743.70 total 85.10 last 273.97) - open/close 30000 (time 1721406748.59 total 89.99 last 284.52) - open/close 32666 (time 1721406758.59 total 99.99 last 266.53) - open/close 35461 (time 1721406768.59 total 110.00 last 279.47) - open/close 38105 (time 1721406778.59 total 120.00 last 264.32) - open/close 40000 (time 1721406785.51 total 126.92 last 273.82) - open/close 42597 (time 1721406795.52 total 136.92 last 259.69) - open/close 46274 (time 1721406805.52 total 146.92 last 367.67) - open/close 49529 (time 1721406815.52 total 156.92 last 325.48) - open/close 52376 (time 1721406825.52 total 166.92 last 284.66) - open/close 55555 (time 1721406835.52 total 176.92 last 317.84) - open/close 58508 (time 1721406845.52 total 186.93 last 295.22) - open/close 60000 (time 1721406851.11 total 192.52 last 266.87) - open/close 62938 (time 1721406861.11 total 202.52 last 293.77) total: 64768 open/close in 208.32 seconds: 310.91 ops/second - /unlink 4748 (time 1721406878.20 total 10.00 last 474.76) - /unlink 10000 (time 1721406887.86 total 19.66 last 543.68) - /unlink 15911 (time 1721406897.86 total 29.66 last 591.08) - /unlink 20000 (time 1721406905.46 total 37.27 last 537.72) - /unlink 25754 (time 1721406915.46 total 47.27 last 575.34) - /unlink 30000 (time 1721406923.45 total 55.25 last 531.76) - /unlink 35139 (time 1721406933.45 total 65.25 last 513.84) - /unlink 40000 (time 1721406942.52 total 74.32 last 535.94) - /unlink 47129 (time 1721406952.52 total 84.32 last 712.83) - /unlink 50000 (time 1721406955.86 total 87.66 last 859.75) - /unlink 58670 (time 1721406965.86 total 97.66 last 866.98) total: 64768 /unlink in 103.92 seconds: 623.24 ops/second - open/close 5286 (time 1721406982.94 total 10.00 last 528.58) - open/close 10000 (time 1721406992.13 total 19.19 last 512.73) - open/close 14850 (time 1721407002.13 total 29.20 last 484.93) - open/close 19689 (time 1721407012.13 total 39.20 last 483.90) - open/close 23911 (time 1721407022.13 total 49.20 last 422.15) - open/close 28850 (time 1721407032.13 total 59.20 last 493.88) - open/close 30000 (time 1721407035.54 total 62.60 last 337.79) - open/close 34993 (time 1721407045.54 total 72.60 last 499.27) - open/close 40000 (time 1721407055.21 total 82.27 last 518.00) - open/close 45242 (time 1721407065.21 total 92.27 last 524.12) - open/close 50000 (time 1721407074.36 total 101.42 last 520.04) - open/close 55251 (time 1721407084.36 total 111.42 last 525.05) - open/close 60000 (time 1721407093.41 total 120.47 last 524.61) total: 64768 open/close in 129.63 seconds: 499.62 ops/second - /unlink 9155 (time 1721407113.42 total 10.00 last 915.43) - /unlink 18222 (time 1721407123.42 total 20.00 last 906.67) - /unlink 27380 (time 1721407133.42 total 30.00 last 915.74) - /unlink 30000 (time 1721407136.22 total 32.81 last 934.13) - /unlink 39340 (time 1721407146.22 total 42.81 last 933.99) - /unlink 48715 (time 1721407156.22 total 52.81 last 937.48) - /unlink 57982 (time 1721407166.22 total 62.81 last 926.65) - /unlink 60000 (time 1721407168.43 total 65.02 last 913.09) total: 64768 /unlink in 69.96 seconds: 925.84 ops/second - open/close 5347 (time 1721407184.21 total 10.00 last 534.65) - open/close 9611 (time 1721407194.21 total 20.00 last 426.33) - open/close 14247 (time 1721407204.21 total 30.00 last 463.57) - open/close 18679 (time 1721407214.22 total 40.00 last 443.14) - open/close 20000 (time 1721407216.99 total 42.78 last 476.36) - open/close 24585 (time 1721407226.99 total 52.78 last 458.50) - open/close 29463 (time 1721407236.99 total 62.78 last 487.74) - open/close 34143 (time 1721407246.99 total 72.78 last 467.92) - open/close 37724 (time 1721407256.99 total 82.78 last 358.01) - open/close 40000 (time 1721407261.68 total 87.47 last 485.42) - open/close 44263 (time 1721407271.68 total 97.47 last 426.23) - open/close 48758 (time 1721407281.69 total 107.47 last 449.45) - open/close 50000 (time 1721407283.97 total 109.76 last 544.02) - open/close 55332 (time 1721407293.97 total 119.76 last 533.17) - open/close 60000 (time 1721407303.13 total 128.92 last 509.34) total: 64768 open/close in 138.09 seconds: 469.02 ops/second - /unlink 9231 (time 1721407323.14 total 10.00 last 923.01) - /unlink 18398 (time 1721407333.14 total 20.00 last 916.68) - /unlink 27446 (time 1721407343.14 total 30.00 last 904.74) - /unlink 30000 (time 1721407345.93 total 32.79 last 915.88) - /unlink 39193 (time 1721407355.93 total 42.79 last 919.25) - /unlink 47963 (time 1721407365.93 total 52.79 last 876.94) - /unlink 50000 (time 1721407368.17 total 55.03 last 910.07) - /unlink 59426 (time 1721407378.17 total 65.03 last 942.55) total: 64768 /unlink in 70.47 seconds: 919.14 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server LNET ready to unload unloading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing unload_modules_local oleg402-server: LNET ready to unload modules unloaded. pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2