== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:10:12 (1767384612) Stopping clients: oleg238-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg238-client.virtnet /mnt/lustre2 (opts:-f) oleg238-server: oleg238-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing load_modules_local oleg238-server: Loading modules from /home/green/git/lustre-release/lustre oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg238-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg238-server: detected 4 online CPUs by sysfs oleg238-server: MODOPTS_LIBCFS= oleg238-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg238-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg238-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg238-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg238-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg238-server: oleg238-server.virtnet: executing set_default_debug -1 all pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg238-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg238-client.virtnet: -o user_xattr,flock 192.168.202.138@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4745 (time 1767384663.07 total 10.00 last 474.44) - open/close 9430 (time 1767384673.07 total 20.00 last 468.41) - open/close 14078 (time 1767384683.07 total 30.01 last 464.71) - open/close 18547 (time 1767384693.08 total 40.01 last 446.80) - open/close 20000 (time 1767384696.72 total 43.65 last 399.14) - open/close 25566 (time 1767384706.72 total 53.65 last 556.59) - open/close 30000 (time 1767384715.74 total 62.68 last 491.17) - open/close 34739 (time 1767384725.75 total 72.68 last 473.81) - open/close 39523 (time 1767384735.75 total 82.68 last 478.37) - open/close 44031 (time 1767384745.75 total 92.68 last 450.80) - open/close 48558 (time 1767384755.75 total 102.68 last 452.61) - open/close 50000 (time 1767384758.89 total 105.82 last 458.71) - open/close 54657 (time 1767384768.89 total 115.83 last 465.61) - open/close 59335 (time 1767384778.89 total 125.83 last 467.77) - open/close 64031 (time 1767384788.90 total 135.83 last 469.56) total: 64768 open/close in 137.42 seconds: 471.31 ops/second - /unlink 7482 (time 1767384801.93 total 10.00 last 748.04) - /unlink 10000 (time 1767384806.09 total 14.17 last 604.79) - /unlink 17514 (time 1767384816.09 total 24.17 last 751.37) - /unlink 20000 (time 1767384819.36 total 27.44 last 759.66) - /unlink 27281 (time 1767384829.37 total 37.44 last 728.03) - /unlink 30000 (time 1767384833.40 total 41.47 last 674.57) - /unlink 35813 (time 1767384843.40 total 51.47 last 581.28) - /unlink 40000 (time 1767384850.65 total 58.73 last 576.86) - /unlink 46148 (time 1767384860.66 total 68.73 last 614.74) - /unlink 50000 (time 1767384865.12 total 73.19 last 863.15) - /unlink 56967 (time 1767384875.12 total 83.19 last 696.66) - /unlink 60000 (time 1767384881.14 total 89.21 last 503.86) total: 64768 /unlink in 97.09 seconds: 667.08 ops/second Waiting for MDT destroys to complete oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 - open/close 3441 (time 1767384924.83 total 10.00 last 344.09) - open/close 7004 (time 1767384934.83 total 20.00 last 356.29) - open/close 10000 (time 1767384943.21 total 28.39 last 357.27) - open/close 13937 (time 1767384953.21 total 38.39 last 393.67) - open/close 18206 (time 1767384963.22 total 48.39 last 426.86) - open/close 20000 (time 1767384967.70 total 52.88 last 399.65) - open/close 23155 (time 1767384977.71 total 62.88 last 315.47) - open/close 26974 (time 1767384987.71 total 72.88 last 381.80) - open/close 30000 (time 1767384995.08 total 80.25 last 410.35) - open/close 34313 (time 1767385005.08 total 90.26 last 431.20) - open/close 38003 (time 1767385015.09 total 100.26 last 368.96) - open/close 40000 (time 1767385019.90 total 105.07 last 414.87) - open/close 42965 (time 1767385029.90 total 115.07 last 296.46) - open/close 47187 (time 1767385039.90 total 125.07 last 422.16) - open/close 50000 (time 1767385048.77 total 133.95 last 317.03) - open/close 53106 (time 1767385058.78 total 143.95 last 310.55) - open/close 55875 (time 1767385068.78 total 153.95 last 276.84) - open/close 58763 (time 1767385078.78 total 163.95 last 288.68) - open/close 60000 (time 1767385083.18 total 168.36 last 281.09) - open/close 63713 (time 1767385093.18 total 178.36 last 371.28) total: 64768 open/close in 181.81 seconds: 356.24 ops/second - /unlink 5630 (time 1767385108.60 total 10.00 last 562.94) - /unlink 10000 (time 1767385116.85 total 18.25 last 529.55) - /unlink 15296 (time 1767385126.85 total 28.25 last 529.51) - /unlink 20000 (time 1767385133.69 total 35.09 last 688.07) - /unlink 26582 (time 1767385143.69 total 45.09 last 658.20) - /unlink 30000 (time 1767385148.86 total 50.26 last 661.47) - /unlink 35284 (time 1767385158.86 total 60.26 last 528.36) - /unlink 40000 (time 1767385167.82 total 69.22 last 526.26) - /unlink 45695 (time 1767385177.82 total 79.22 last 569.48) - /unlink 50000 (time 1767385186.91 total 88.31 last 473.62) - /unlink 54274 (time 1767385196.91 total 98.31 last 427.35) - /unlink 58797 (time 1767385206.91 total 108.31 last 452.24) - /unlink 60000 (time 1767385209.97 total 111.37 last 394.05) - /unlink 64343 (time 1767385219.97 total 121.37 last 434.23) total: 64768 /unlink in 122.48 seconds: 528.80 ops/second Waiting for MDT destroys to complete oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 - open/close 2205 (time 1767385260.62 total 10.00 last 220.48) - open/close 4474 (time 1767385270.63 total 20.00 last 226.84) - open/close 6599 (time 1767385280.63 total 30.01 last 212.45) - open/close 8737 (time 1767385290.63 total 40.01 last 213.73) - open/close 10000 (time 1767385296.39 total 45.77 last 219.40) - open/close 12482 (time 1767385306.39 total 55.77 last 248.18) - open/close 15302 (time 1767385316.39 total 65.77 last 281.98) - open/close 18100 (time 1767385326.39 total 75.77 last 279.74) - open/close 20000 (time 1767385334.80 total 84.18 last 225.90) - open/close 22172 (time 1767385344.81 total 94.18 last 217.13) - open/close 24338 (time 1767385354.81 total 104.19 last 216.53) - open/close 26509 (time 1767385364.81 total 114.19 last 217.04) - open/close 28601 (time 1767385374.81 total 124.19 last 209.19) - open/close 30000 (time 1767385381.49 total 130.86 last 209.62) - open/close 32183 (time 1767385391.49 total 140.87 last 218.24) - open/close 34333 (time 1767385401.49 total 150.87 last 214.98) - open/close 36604 (time 1767385411.49 total 160.87 last 226.99) - open/close 38717 (time 1767385421.50 total 170.87 last 211.29) - open/close 40000 (time 1767385427.63 total 177.01 last 209.18) - open/close 42249 (time 1767385437.63 total 187.01 last 224.83) - open/close 44556 (time 1767385447.64 total 197.01 last 230.60) - open/close 46778 (time 1767385457.64 total 207.02 last 222.11) - open/close 49003 (time 1767385467.64 total 217.02 last 222.46) - open/close 50000 (time 1767385472.04 total 221.42 last 226.80) - open/close 52166 (time 1767385482.04 total 231.42 last 216.56) - open/close 54294 (time 1767385492.04 total 241.42 last 212.79) - open/close 56438 (time 1767385502.04 total 251.42 last 214.38) - open/close 58581 (time 1767385512.04 total 261.42 last 214.29) - open/close 60000 (time 1767385518.39 total 267.77 last 223.61) - open/close 62214 (time 1767385528.39 total 277.77 last 221.33) - open/close 64628 (time 1767385538.39 total 287.77 last 241.37) total: 64768 open/close in 288.34 seconds: 224.62 ops/second - /unlink 4688 (time 1767385550.81 total 10.00 last 468.80) - /unlink 8945 (time 1767385560.81 total 20.00 last 425.67) - /unlink 10000 (time 1767385563.29 total 22.48 last 424.95) - /unlink 14394 (time 1767385573.29 total 32.48 last 439.40) - /unlink 18616 (time 1767385583.29 total 42.48 last 422.16) - /unlink 20000 (time 1767385586.69 total 45.88 last 407.88) - /unlink 24588 (time 1767385596.69 total 55.88 last 458.77) - /unlink 28936 (time 1767385606.69 total 65.88 last 434.74) - /unlink 30000 (time 1767385609.15 total 68.34 last 432.62) - /unlink 34211 (time 1767385619.15 total 78.34 last 421.04) - /unlink 38414 (time 1767385629.15 total 88.34 last 420.27) - /unlink 40000 (time 1767385632.69 total 91.88 last 448.37) - /unlink 44287 (time 1767385642.69 total 101.88 last 428.70) - /unlink 49286 (time 1767385652.69 total 111.88 last 499.83) - /unlink 53793 (time 1767385662.69 total 121.88 last 450.64) - /unlink 58595 (time 1767385672.69 total 131.88 last 480.18) - /unlink 60000 (time 1767385675.94 total 135.13 last 432.33) - /unlink 64613 (time 1767385685.94 total 145.13 last 461.22) total: 64768 /unlink in 145.47 seconds: 445.22 ops/second Waiting for MDT destroys to complete oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 - open/close 2221 (time 1767385705.57 total 10.00 last 222.01) - open/close 4518 (time 1767385715.57 total 20.01 last 229.67) - open/close 6711 (time 1767385725.57 total 30.01 last 219.24) - open/close 8920 (time 1767385735.57 total 40.01 last 220.90) - open/close 10000 (time 1767385740.38 total 44.82 last 224.65) - open/close 12278 (time 1767385750.38 total 54.82 last 227.75) - open/close 14859 (time 1767385760.39 total 64.82 last 258.00) - open/close 17284 (time 1767385770.39 total 74.82 last 242.43) - open/close 19927 (time 1767385780.39 total 84.83 last 264.27) - open/close 22229 (time 1767385790.39 total 94.83 last 230.18) - open/close 24575 (time 1767385800.39 total 104.83 last 234.55) - open/close 26782 (time 1767385810.39 total 114.83 last 220.70) - open/close 28978 (time 1767385820.40 total 124.83 last 219.52) - open/close 30000 (time 1767385825.21 total 129.64 last 212.56) - open/close 32276 (time 1767385835.21 total 139.64 last 227.51) - open/close 34810 (time 1767385845.21 total 149.65 last 253.30) - open/close 37172 (time 1767385855.22 total 159.65 last 236.18) - open/close 39392 (time 1767385865.22 total 169.65 last 221.90) - open/close 40000 (time 1767385867.74 total 172.17 last 241.52) - open/close 42283 (time 1767385877.74 total 182.17 last 228.21) - open/close 44697 (time 1767385887.74 total 192.18 last 241.38) - open/close 46895 (time 1767385897.74 total 202.18 last 219.76) - open/close 49486 (time 1767385907.75 total 212.18 last 258.99) - open/close 50000 (time 1767385910.02 total 214.46 last 225.77) - open/close 52242 (time 1767385920.03 total 224.46 last 224.17) - open/close 54634 (time 1767385930.03 total 234.46 last 239.19) - open/close 56919 (time 1767385940.03 total 244.46 last 228.43) - open/close 59149 (time 1767385950.03 total 254.47 last 222.91) - open/close 60000 (time 1767385953.56 total 258.00 last 240.97) - open/close 62295 (time 1767385963.57 total 268.00 last 229.48) - open/close 64470 (time 1767385973.57 total 278.00 last 217.49) total: 64768 open/close in 279.32 seconds: 231.88 ops/second - /unlink 4084 (time 1767385986.90 total 10.00 last 408.38) - /unlink 8483 (time 1767385996.90 total 20.00 last 439.80) - /unlink 10000 (time 1767386000.75 total 23.85 last 394.10) - /unlink 14311 (time 1767386010.75 total 33.85 last 431.09) - /unlink 18643 (time 1767386020.75 total 43.85 last 433.17) - /unlink 20000 (time 1767386023.88 total 46.97 last 434.73) - /unlink 24300 (time 1767386033.88 total 56.98 last 429.96) - /unlink 28541 (time 1767386043.88 total 66.98 last 424.08) - /unlink 30000 (time 1767386047.18 total 70.28 last 442.00) - /unlink 34494 (time 1767386057.18 total 80.28 last 449.33) - /unlink 38598 (time 1767386067.18 total 90.28 last 410.40) - /unlink 40000 (time 1767386070.67 total 93.77 last 401.05) - /unlink 44660 (time 1767386080.67 total 103.77 last 465.99) - /unlink 49137 (time 1767386090.68 total 113.78 last 447.66) - /unlink 53974 (time 1767386100.68 total 123.78 last 483.63) - /unlink 58549 (time 1767386110.68 total 133.78 last 457.48) - /unlink 60000 (time 1767386113.49 total 136.59 last 516.62) total: 64768 /unlink in 145.17 seconds: 446.15 ops/second Waiting for MDT destroys to complete oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 - open/close 2192 (time 1767386141.83 total 10.00 last 219.13) - open/close 4313 (time 1767386151.83 total 20.01 last 212.06) - open/close 6626 (time 1767386161.84 total 30.01 last 231.23) - open/close 8978 (time 1767386171.84 total 40.01 last 235.12) - open/close 10000 (time 1767386175.86 total 44.04 last 253.97) - open/close 12168 (time 1767386185.87 total 54.04 last 216.72) - open/close 14316 (time 1767386195.87 total 64.04 last 214.74) - open/close 16501 (time 1767386205.87 total 74.04 last 218.49) - open/close 18674 (time 1767386215.88 total 84.05 last 217.22) - open/close 20000 (time 1767386221.87 total 90.04 last 221.34) - open/close 22276 (time 1767386231.87 total 100.04 last 227.52) - open/close 24494 (time 1767386241.87 total 110.04 last 221.78) - open/close 26681 (time 1767386251.87 total 120.04 last 218.69) - open/close 28958 (time 1767386261.87 total 130.05 last 227.60) - open/close 30000 (time 1767386265.94 total 134.11 last 256.35) - open/close 32352 (time 1767386275.94 total 144.11 last 235.17) - open/close 34536 (time 1767386285.94 total 154.11 last 218.37) - open/close 36764 (time 1767386295.95 total 164.12 last 222.71) - open/close 39005 (time 1767386305.95 total 174.12 last 224.02) - open/close 40000 (time 1767386310.52 total 178.69 last 217.95) - open/close 42188 (time 1767386320.52 total 188.69 last 218.70) - open/close 44447 (time 1767386330.52 total 198.69 last 225.89) - open/close 46953 (time 1767386340.52 total 208.69 last 250.59) - open/close 49506 (time 1767386350.52 total 218.69 last 255.27) - open/close 51703 (time 1767386360.52 total 228.70 last 219.65) - open/close 54001 (time 1767386370.53 total 238.70 last 229.77) - open/close 56097 (time 1767386380.53 total 248.70 last 209.51) - open/close 58249 (time 1767386390.53 total 258.70 last 215.16) - open/close 60000 (time 1767386398.56 total 266.73 last 218.09) - open/close 62330 (time 1767386408.56 total 276.73 last 232.99) total: 64768 open/close in 285.88 seconds: 226.55 ops/second - /unlink 5130 (time 1767386429.16 total 10.00 last 512.95) - /unlink 10000 (time 1767386439.15 total 19.99 last 487.52) - /unlink 14402 (time 1767386449.15 total 29.99 last 440.10) - /unlink 18691 (time 1767386459.15 total 39.99 last 428.89) - /unlink 20000 (time 1767386461.79 total 42.64 last 495.06) - /unlink 24553 (time 1767386471.79 total 52.64 last 455.26) - /unlink 28857 (time 1767386481.80 total 62.64 last 430.37) - /unlink 30000 (time 1767386484.61 total 65.46 last 405.50) - /unlink 34296 (time 1767386494.61 total 75.46 last 429.59) - /unlink 38634 (time 1767386504.61 total 85.46 last 433.79) - /unlink 40000 (time 1767386507.48 total 88.33 last 476.08) - /unlink 44727 (time 1767386517.49 total 98.33 last 472.66) - /unlink 49693 (time 1767386527.49 total 108.33 last 496.59) - /unlink 54380 (time 1767386537.49 total 118.33 last 468.68) - /unlink 59386 (time 1767386547.49 total 128.33 last 500.51) - /unlink 63844 (time 1767386557.49 total 138.33 last 445.73) total: 64768 /unlink in 140.43 seconds: 461.22 ops/second Waiting for MDT destroys to complete oleg238-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg238-client: oleg238-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg238-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg238-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg238-server stop mds service on oleg238-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg238-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg238-server' oleg238-server: oleg238-server.virtnet: executing unload_modules_local oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg238-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-client: ssh exited with exit code 2 pdsh@oleg238-client: oleg238-server: ssh exited with exit code 2