== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 12:47:45 (1772473665) Stopping clients: oleg350-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg350-client.virtnet /mnt/lustre2 (opts:-f) oleg350-server: oleg350-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing load_modules_local oleg350-server: Loading modules from /home/green/git/lustre-release/lustre oleg350-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg350-server: detected 4 online CPUs by sysfs oleg350-server: MODOPTS_LIBCFS= oleg350-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg350-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg350-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg350-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg350-server: oleg350-server.virtnet: executing set_default_debug -1 all pdsh@oleg350-client: oleg350-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg350-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg350-client.virtnet: -o user_xattr,flock 192.168.203.150@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 810 (time 1772473750.96 total 10.00 last 80.96) - open/close 1637 (time 1772473760.97 total 20.01 last 82.65) - open/close 2430 (time 1772473770.97 total 30.01 last 79.29) - open/close 3227 (time 1772473780.97 total 40.01 last 79.69) - open/close 4185 (time 1772473790.98 total 50.02 last 95.74) - open/close 5048 (time 1772473800.98 total 60.02 last 86.25) - open/close 5970 (time 1772473810.99 total 70.03 last 92.16) - open/close 6927 (time 1772473820.99 total 80.03 last 95.69) - open/close 7766 (time 1772473830.99 total 90.03 last 83.90) - open/close 8500 (time 1772473840.99 total 100.04 last 73.37) - open/close 9393 (time 1772473850.99 total 110.04 last 89.28) - open/close 10000 (time 1772473857.04 total 116.09 last 100.34) - open/close 10800 (time 1772473867.05 total 126.09 last 79.96) - open/close 11657 (time 1772473877.05 total 136.09 last 85.70) - open/close 12656 (time 1772473887.06 total 146.10 last 99.77) - open/close 13583 (time 1772473897.07 total 156.11 last 92.65) - open/close 14456 (time 1772473907.07 total 166.11 last 87.30) - open/close 15520 (time 1772473917.08 total 176.12 last 106.28) - open/close 16430 (time 1772473927.10 total 186.14 last 90.83) - open/close 17417 (time 1772473937.10 total 196.15 last 98.63) - open/close 18243 (time 1772473947.11 total 206.15 last 82.57) - open/close 19090 (time 1772473957.11 total 216.16 last 84.65) - open/close 20000 (time 1772473966.99 total 226.04 last 92.12) - open/close 20944 (time 1772473976.99 total 236.04 last 94.38) - open/close 21840 (time 1772473987.01 total 246.06 last 89.44) - open/close 22911 (time 1772473997.02 total 256.06 last 107.05) - open/close 23872 (time 1772474007.02 total 266.06 last 96.08) - open/close 24774 (time 1772474017.03 total 276.07 last 90.11) - open/close 25697 (time 1772474027.03 total 286.08 last 92.28) - open/close 26547 (time 1772474037.03 total 296.08 last 84.99) - open/close 27458 (time 1772474047.03 total 306.08 last 91.10) - open/close 28413 (time 1772474057.03 total 316.08 last 95.50) - open/close 29347 (time 1772474067.04 total 326.08 last 93.36) - open/close 30000 (time 1772474073.77 total 332.82 last 96.94) - open/close 30987 (time 1772474083.78 total 342.82 last 98.68) - open/close 31963 (time 1772474093.78 total 352.82 last 97.58) - open/close 32906 (time 1772474103.78 total 362.83 last 94.24) - open/close 33840 (time 1772474113.79 total 372.84 last 93.32) - open/close 34817 (time 1772474123.80 total 382.84 last 97.66) - open/close 35890 (time 1772474133.80 total 392.85 last 107.24) - open/close 36997 (time 1772474143.81 total 402.86 last 110.59) - open/close 37994 (time 1772474153.82 total 412.87 last 99.59) - open/close 38910 (time 1772474163.83 total 422.88 last 91.51) - open/close 39844 (time 1772474173.85 total 432.89 last 93.25) - open/close 40832 (time 1772474183.86 total 442.90 last 98.71) - open/close 41820 (time 1772474193.86 total 452.90 last 98.76) - open/close 42765 (time 1772474203.87 total 462.91 last 94.47) - open/close 43846 (time 1772474213.87 total 472.91 last 108.04) - open/close 44887 (time 1772474223.87 total 482.92 last 104.07) - open/close 45833 (time 1772474233.88 total 492.93 last 94.50) - open/close 46813 (time 1772474243.89 total 502.93 last 97.93) - open/close 47870 (time 1772474253.89 total 512.94 last 105.69) - open/close 48950 (time 1772474263.89 total 522.94 last 107.99) - open/close 49950 (time 1772474273.90 total 532.95 last 99.90) - open/close 51042 (time 1772474283.91 total 542.95 last 109.11) - open/close 52102 (time 1772474293.92 total 552.96 last 105.93) - open/close 53225 (time 1772474303.92 total 562.96 last 112.29) - open/close 54237 (time 1772474313.93 total 572.97 last 101.12) - open/close 55214 (time 1772474323.93 total 582.97 last 97.67) - open/close 56199 (time 1772474333.94 total 592.98 last 98.39) - open/close 57241 (time 1772474343.95 total 602.99 last 104.12) - open/close 58239 (time 1772474353.95 total 613.00 last 99.74) - open/close 59295 (time 1772474363.96 total 623.00 last 105.52) - open/close 60000 (time 1772474370.99 total 630.04 last 100.27) - open/close 61026 (time 1772474381.00 total 640.04 last 102.51) - open/close 62051 (time 1772474391.01 total 650.05 last 102.46) - open/close 63136 (time 1772474401.01 total 660.05 last 108.47) - open/close 64147 (time 1772474411.01 total 670.05 last 101.08) total: 64768 open/close in 675.60 seconds: 95.87 ops/second - /unlink 1205 (time 1772474429.41 total 10.00 last 120.47) - /unlink 2389 (time 1772474439.41 total 20.01 last 118.36) - /unlink 3503 (time 1772474449.41 total 30.01 last 111.39) - /unlink 4660 (time 1772474459.41 total 40.01 last 115.65) - /unlink 5842 (time 1772474469.42 total 50.02 last 118.15) - /unlink 7026 (time 1772474479.43 total 60.02 last 118.33) - /unlink 8188 (time 1772474489.46 total 70.06 last 115.75) - /unlink 9517 (time 1772474499.47 total 80.06 last 132.88) - /unlink 10000 (time 1772474503.15 total 83.74 last 131.13) - /unlink 11128 (time 1772474513.15 total 93.75 last 112.78) - /unlink 12214 (time 1772474523.15 total 103.75 last 108.60) - /unlink 13451 (time 1772474533.15 total 113.75 last 123.70) - /unlink 14591 (time 1772474543.16 total 123.76 last 113.90) - /unlink 15696 (time 1772474553.16 total 133.76 last 110.46) - /unlink 16980 (time 1772474563.17 total 143.76 last 128.38) - /unlink 18350 (time 1772474573.17 total 153.77 last 136.91) - /unlink 19758 (time 1772474583.18 total 163.77 last 140.73) - /unlink 21193 (time 1772474593.18 total 173.78 last 143.47) - /unlink 22606 (time 1772474603.19 total 183.78 last 141.19) - /unlink 23734 (time 1772474613.20 total 193.79 last 112.70) - /unlink 24970 (time 1772474623.20 total 203.80 last 123.53) - /unlink 26877 (time 1772474633.20 total 213.80 last 190.67) - /unlink 29792 (time 1772474643.20 total 223.80 last 291.49) - /unlink 32987 (time 1772474653.20 total 233.80 last 319.50) - /unlink 35711 (time 1772474663.21 total 243.80 last 272.34) - /unlink 37863 (time 1772474673.21 total 253.80 last 215.20) - /unlink 39977 (time 1772474683.21 total 263.80 last 211.36) - /unlink 42181 (time 1772474693.21 total 273.81 last 220.35) - /unlink 44668 (time 1772474703.21 total 283.81 last 248.67) - /unlink 47239 (time 1772474713.21 total 293.81 last 257.10) - /unlink 49690 (time 1772474723.22 total 303.81 last 244.97) - /unlink 52310 (time 1772474733.22 total 313.81 last 261.96) - /unlink 55067 (time 1772474743.22 total 323.82 last 275.67) - /unlink 57645 (time 1772474753.22 total 333.82 last 257.75) - /unlink 60000 (time 1772474763.14 total 343.74 last 237.43) - /unlink 62283 (time 1772474773.14 total 353.74 last 228.30) - /unlink 64530 (time 1772474783.14 total 363.74 last 224.67) total: 64768 /unlink in 364.80 seconds: 177.54 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1233 (time 1772474818.58 total 10.00 last 123.25) - open/close 2413 (time 1772474828.59 total 20.01 last 117.97) - open/close 3723 (time 1772474838.59 total 30.01 last 130.97) - open/close 4994 (time 1772474848.59 total 40.01 last 127.06) - open/close 6409 (time 1772474858.59 total 50.01 last 141.47) - open/close 7832 (time 1772474868.60 total 60.02 last 142.18) - open/close 9139 (time 1772474878.60 total 70.02 last 130.67) - open/close 10000 (time 1772474885.25 total 76.67 last 129.62) - open/close 11465 (time 1772474895.25 total 86.67 last 146.46) - open/close 12789 (time 1772474905.25 total 96.67 last 132.33) - open/close 14222 (time 1772474915.26 total 106.68 last 143.27) - open/close 15699 (time 1772474925.26 total 116.68 last 147.65) - open/close 17391 (time 1772474935.26 total 126.68 last 169.17) - open/close 18985 (time 1772474945.26 total 136.68 last 159.37) - open/close 20000 (time 1772474951.64 total 143.07 last 159.06) - open/close 21802 (time 1772474961.65 total 153.07 last 180.16) - open/close 23278 (time 1772474971.65 total 163.07 last 147.55) - open/close 24703 (time 1772474981.65 total 173.08 last 142.44) - open/close 25989 (time 1772474991.66 total 183.08 last 128.54) - open/close 27287 (time 1772475001.66 total 193.08 last 129.75) - open/close 28534 (time 1772475011.66 total 203.08 last 124.69) - open/close 30000 (time 1772475021.27 total 212.69 last 152.58) - open/close 31620 (time 1772475031.28 total 222.70 last 161.94) - open/close 33382 (time 1772475041.28 total 232.70 last 176.06) - open/close 35230 (time 1772475051.28 total 242.70 last 184.78) - open/close 37049 (time 1772475061.28 total 252.71 last 181.89) - open/close 38987 (time 1772475071.29 total 262.71 last 193.71) - open/close 40000 (time 1772475076.57 total 267.99 last 191.73) - open/close 41754 (time 1772475086.57 total 277.99 last 175.38) - open/close 43340 (time 1772475096.58 total 288.00 last 158.54) - open/close 44859 (time 1772475106.58 total 298.00 last 151.87) - open/close 46578 (time 1772475116.58 total 308.00 last 171.83) - open/close 48619 (time 1772475126.59 total 318.01 last 203.96) - open/close 50000 (time 1772475133.42 total 324.84 last 202.25) - open/close 51876 (time 1772475143.42 total 334.84 last 187.56) - open/close 53409 (time 1772475153.43 total 344.85 last 153.21) - open/close 55300 (time 1772475163.43 total 354.85 last 189.08) - open/close 57337 (time 1772475173.43 total 364.85 last 203.68) - open/close 59889 (time 1772475183.43 total 374.85 last 255.15) - open/close 62569 (time 1772475193.43 total 384.85 last 267.93) - open/close 64460 (time 1772475203.44 total 394.86 last 189.05) total: 64768 open/close in 396.28 seconds: 163.44 ops/second - /unlink 2919 (time 1772475216.63 total 10.00 last 291.83) - /unlink 6222 (time 1772475226.63 total 20.00 last 330.29) - /unlink 9150 (time 1772475236.63 total 30.00 last 292.79) - /unlink 10000 (time 1772475239.78 total 33.15 last 270.00) - /unlink 13804 (time 1772475249.78 total 43.15 last 380.40) - /unlink 17804 (time 1772475259.78 total 53.15 last 399.85) - /unlink 20000 (time 1772475265.88 total 59.25 last 360.17) - /unlink 23321 (time 1772475275.88 total 69.25 last 332.05) - /unlink 26234 (time 1772475285.88 total 79.25 last 291.30) - /unlink 29580 (time 1772475295.89 total 89.26 last 334.49) - /unlink 32219 (time 1772475305.89 total 99.26 last 263.89) - /unlink 35158 (time 1772475315.89 total 109.26 last 293.88) - /unlink 38468 (time 1772475325.89 total 119.26 last 330.92) - /unlink 40000 (time 1772475329.90 total 123.27 last 381.97) - /unlink 43623 (time 1772475339.90 total 133.27 last 362.22) - /unlink 47195 (time 1772475349.90 total 143.27 last 357.14) - /unlink 50000 (time 1772475357.40 total 150.77 last 374.14) - /unlink 53810 (time 1772475367.40 total 160.77 last 380.89) - /unlink 57189 (time 1772475377.41 total 170.78 last 337.81) - /unlink 60000 (time 1772475384.94 total 178.31 last 373.38) - /unlink 64022 (time 1772475394.94 total 188.31 last 402.19) total: 64768 /unlink in 189.74 seconds: 341.35 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1994 (time 1772475425.04 total 10.00 last 199.34) - open/close 4350 (time 1772475435.05 total 20.01 last 235.55) - open/close 6593 (time 1772475445.05 total 30.01 last 224.22) - open/close 8738 (time 1772475455.05 total 40.01 last 214.38) - open/close 10000 (time 1772475462.86 total 47.82 last 161.63) - open/close 11740 (time 1772475472.86 total 57.82 last 173.96) - open/close 13903 (time 1772475482.86 total 67.82 last 216.30) - open/close 15957 (time 1772475492.87 total 77.83 last 205.28) - open/close 18343 (time 1772475502.87 total 87.83 last 238.55) - open/close 20000 (time 1772475509.14 total 94.10 last 264.47) - open/close 21859 (time 1772475519.14 total 104.10 last 185.86) - open/close 24211 (time 1772475529.14 total 114.10 last 235.19) - open/close 27180 (time 1772475539.14 total 124.10 last 296.88) - open/close 30000 (time 1772475548.23 total 133.19 last 310.40) - open/close 32555 (time 1772475558.23 total 143.19 last 255.49) - open/close 35253 (time 1772475568.23 total 153.19 last 269.76) - open/close 38463 (time 1772475578.23 total 163.19 last 320.96) - open/close 40000 (time 1772475582.69 total 167.65 last 344.48) - open/close 43390 (time 1772475592.69 total 177.65 last 338.91) - open/close 46581 (time 1772475602.70 total 187.66 last 319.05) - open/close 50000 (time 1772475612.55 total 197.51 last 346.76) - open/close 53157 (time 1772475622.56 total 207.52 last 315.62) - open/close 56258 (time 1772475632.56 total 217.52 last 310.10) - open/close 59614 (time 1772475642.56 total 227.52 last 335.56) - open/close 62980 (time 1772475652.56 total 237.52 last 336.57) total: 64768 open/close in 242.71 seconds: 266.85 ops/second - /unlink 5854 (time 1772475669.17 total 10.00 last 585.13) - /unlink 10000 (time 1772475676.28 total 17.12 last 582.62) - /unlink 16445 (time 1772475686.28 total 27.12 last 644.46) - /unlink 20000 (time 1772475692.22 total 33.06 last 598.38) - /unlink 25984 (time 1772475702.22 total 43.06 last 598.34) - /unlink 30000 (time 1772475707.90 total 48.74 last 707.46) - /unlink 37193 (time 1772475717.90 total 58.74 last 719.28) - /unlink 40000 (time 1772475721.04 total 61.88 last 893.65) - /unlink 47675 (time 1772475731.04 total 71.88 last 767.44) - /unlink 50000 (time 1772475734.08 total 74.92 last 764.54) - /unlink 56570 (time 1772475744.08 total 84.92 last 656.93) - /unlink 60000 (time 1772475748.74 total 89.57 last 737.58) total: 64768 /unlink in 96.09 seconds: 674.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3948 (time 1772475790.20 total 10.00 last 394.73) - open/close 7648 (time 1772475800.20 total 20.00 last 369.89) - open/close 10000 (time 1772475806.99 total 26.80 last 346.28) - open/close 14190 (time 1772475816.99 total 36.80 last 418.98) - open/close 18510 (time 1772475826.99 total 46.80 last 431.93) - open/close 20000 (time 1772475830.80 total 50.61 last 391.23) - open/close 23221 (time 1772475840.80 total 60.61 last 322.06) - open/close 27280 (time 1772475850.80 total 70.61 last 405.90) - open/close 30000 (time 1772475857.10 total 76.90 last 432.13) - open/close 33847 (time 1772475867.10 total 86.91 last 384.62) - open/close 36857 (time 1772475877.10 total 96.91 last 300.97) - open/close 40000 (time 1772475886.28 total 106.09 last 342.36) - open/close 42684 (time 1772475896.28 total 116.09 last 268.38) - open/close 45962 (time 1772475906.28 total 126.09 last 327.80) - open/close 49611 (time 1772475916.28 total 136.09 last 364.82) - open/close 53613 (time 1772475926.29 total 146.09 last 400.09) - open/close 56673 (time 1772475936.29 total 156.09 last 305.92) - open/close 59038 (time 1772475946.29 total 166.10 last 236.44) - open/close 60000 (time 1772475949.14 total 168.95 last 337.58) - open/close 62938 (time 1772475959.15 total 178.95 last 293.66) total: 64768 open/close in 184.36 seconds: 351.32 ops/second - /unlink 6395 (time 1772475976.13 total 10.00 last 639.47) - /unlink 10000 (time 1772475981.32 total 15.19 last 694.73) - /unlink 16766 (time 1772475991.32 total 25.19 last 676.58) - /unlink 20000 (time 1772475995.90 total 29.77 last 705.94) - /unlink 26792 (time 1772476005.90 total 39.77 last 679.18) - /unlink 30000 (time 1772476012.21 total 46.08 last 508.66) - /unlink 35879 (time 1772476022.21 total 56.08 last 587.82) - /unlink 40000 (time 1772476028.88 total 62.76 last 617.29) - /unlink 46264 (time 1772476038.89 total 72.76 last 626.32) - /unlink 50000 (time 1772476044.09 total 77.96 last 717.60) - /unlink 56530 (time 1772476054.09 total 87.96 last 652.97) - /unlink 60000 (time 1772476059.73 total 93.60 last 615.81) total: 64768 /unlink in 101.96 seconds: 635.26 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3324 (time 1772476094.98 total 10.00 last 332.35) - open/close 6865 (time 1772476104.98 total 20.00 last 354.04) - open/close 10000 (time 1772476113.74 total 28.77 last 357.65) - open/close 13841 (time 1772476123.75 total 38.77 last 384.03) - open/close 17808 (time 1772476133.75 total 48.77 last 396.58) - open/close 20000 (time 1772476140.69 total 55.72 last 315.66) - open/close 22675 (time 1772476150.70 total 65.72 last 267.36) - open/close 25346 (time 1772476160.70 total 75.73 last 267.05) - open/close 28647 (time 1772476170.70 total 85.73 last 330.05) - open/close 30000 (time 1772476174.99 total 90.02 last 315.40) - open/close 32894 (time 1772476184.99 total 100.02 last 289.35) - open/close 35816 (time 1772476195.00 total 110.02 last 292.13) - open/close 38873 (time 1772476205.00 total 120.02 last 305.63) - open/close 40000 (time 1772476208.21 total 123.23 last 351.01) - open/close 43500 (time 1772476218.21 total 133.23 last 349.96) - open/close 47106 (time 1772476228.21 total 143.24 last 360.49) - open/close 50000 (time 1772476236.50 total 151.52 last 349.35) - open/close 53726 (time 1772476246.50 total 161.52 last 372.51) - open/close 56856 (time 1772476256.51 total 171.53 last 312.78) - open/close 60000 (time 1772476266.13 total 181.15 last 326.68) - open/close 63218 (time 1772476276.13 total 191.16 last 321.75) total: 64768 open/close in 195.65 seconds: 331.03 ops/second - /unlink 5712 (time 1772476292.14 total 10.00 last 571.10) - /unlink 10000 (time 1772476300.02 total 17.88 last 543.98) - /unlink 15528 (time 1772476310.02 total 27.88 last 552.77) - /unlink 20000 (time 1772476318.32 total 36.18 last 539.05) - /unlink 25123 (time 1772476328.32 total 46.18 last 512.18) - /unlink 30000 (time 1772476337.59 total 55.45 last 526.09) - /unlink 35487 (time 1772476347.59 total 65.45 last 548.65) - /unlink 40000 (time 1772476355.01 total 72.88 last 607.82) - /unlink 46451 (time 1772476365.01 total 82.88 last 645.05) - /unlink 50000 (time 1772476369.70 total 87.57 last 756.85) - /unlink 56655 (time 1772476379.70 total 97.57 last 665.47) - /unlink 60000 (time 1772476384.72 total 102.58 last 667.42) total: 64768 /unlink in 109.03 seconds: 594.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg350-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg350-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg350-server stop mds service on oleg350-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg350-server unloading modules via unload_modules_local on: 'oleg350-server' oleg350-server: oleg350-server.virtnet: executing unload_modules_local oleg350-server: modules unloaded. pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-server: ssh exited with exit code 2 pdsh@oleg350-client: oleg350-client: ssh exited with exit code 2