== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 06:38:25 (1756809505) Stopping clients: oleg423-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg423-client.virtnet /mnt/lustre2 (opts:-f) oleg423-server: oleg423-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing load_modules_local oleg423-server: Loading modules from /home/green/git/lustre-release/lustre oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg423-server: detected 4 online CPUs by sysfs oleg423-server: MODOPTS_LIBCFS= oleg423-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg423-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg423-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg423-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg423-server: oleg423-server.virtnet: executing set_default_debug -1 all pdsh@oleg423-client: oleg423-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg423-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg423-client.virtnet: -o user_xattr,flock 192.168.204.123@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 969 (time 1756809571.48 total 10.00 last 96.88) - open/close 1907 (time 1756809581.48 total 20.00 last 93.79) - open/close 3030 (time 1756809591.49 total 30.01 last 112.24) - open/close 4111 (time 1756809601.49 total 40.02 last 108.02) - open/close 5226 (time 1756809611.50 total 50.02 last 111.45) - open/close 6281 (time 1756809621.50 total 60.03 last 105.43) - open/close 7291 (time 1756809631.51 total 70.03 last 100.96) - open/close 8472 (time 1756809641.51 total 80.03 last 118.09) - open/close 9621 (time 1756809651.52 total 90.04 last 114.79) - open/close 10000 (time 1756809655.77 total 94.29 last 89.27) - open/close 10903 (time 1756809665.77 total 104.29 last 90.26) - open/close 11856 (time 1756809675.78 total 114.30 last 95.25) - open/close 12793 (time 1756809685.78 total 124.31 last 93.61) - open/close 13921 (time 1756809695.79 total 134.31 last 112.78) - open/close 14927 (time 1756809705.79 total 144.31 last 100.54) - open/close 15895 (time 1756809715.80 total 154.32 last 96.75) - open/close 16865 (time 1756809725.80 total 164.33 last 96.94) - open/close 17764 (time 1756809735.81 total 174.33 last 89.85) - open/close 18751 (time 1756809745.82 total 184.34 last 98.62) - open/close 19723 (time 1756809755.83 total 194.35 last 97.12) - open/close 20000 (time 1756809758.55 total 197.07 last 101.61) - open/close 20860 (time 1756809768.55 total 207.08 last 85.99) - open/close 21762 (time 1756809778.56 total 217.08 last 90.15) - open/close 22711 (time 1756809788.57 total 227.09 last 94.82) - open/close 24115 (time 1756809798.57 total 237.09 last 140.34) - open/close 25589 (time 1756809808.58 total 247.11 last 147.22) - open/close 26884 (time 1756809818.59 total 257.11 last 129.41) - open/close 28280 (time 1756809828.60 total 267.12 last 139.53) - open/close 29515 (time 1756809838.60 total 277.12 last 123.50) - open/close 30000 (time 1756809842.16 total 280.68 last 136.20) - open/close 31309 (time 1756809852.16 total 290.68 last 130.90) - open/close 32879 (time 1756809862.16 total 300.68 last 157.00) - open/close 34463 (time 1756809872.16 total 310.68 last 158.39) - open/close 36131 (time 1756809882.16 total 320.68 last 166.76) - open/close 37895 (time 1756809892.16 total 330.68 last 176.36) - open/close 39644 (time 1756809902.16 total 340.69 last 174.87) - open/close 40000 (time 1756809904.54 total 343.07 last 149.63) - open/close 41606 (time 1756809914.55 total 353.07 last 160.56) - open/close 43584 (time 1756809924.55 total 363.07 last 197.70) - open/close 45473 (time 1756809934.55 total 373.07 last 188.88) - open/close 47452 (time 1756809944.56 total 383.08 last 197.81) - open/close 48871 (time 1756809954.56 total 393.08 last 141.88) - open/close 50000 (time 1756809960.79 total 399.32 last 181.06) - open/close 52013 (time 1756809970.80 total 409.32 last 201.24) - open/close 53685 (time 1756809980.80 total 419.32 last 167.14) - open/close 55512 (time 1756809990.80 total 429.32 last 182.68) - open/close 57579 (time 1756810000.81 total 439.33 last 206.47) - open/close 59370 (time 1756810010.82 total 449.34 last 179.06) - open/close 60000 (time 1756810013.79 total 452.32 last 211.48) - open/close 62386 (time 1756810023.79 total 462.32 last 238.59) - open/close 64712 (time 1756810033.80 total 472.32 last 232.58) total: 64768 open/close in 472.57 seconds: 137.05 ops/second - /unlink 2640 (time 1756810045.89 total 10.00 last 263.91) - /unlink 4390 (time 1756810055.89 total 20.00 last 174.99) - /unlink 6223 (time 1756810065.90 total 30.01 last 183.27) - /unlink 9066 (time 1756810075.90 total 40.01 last 284.26) - /unlink 10000 (time 1756810078.84 total 42.95 last 317.56) - /unlink 13273 (time 1756810088.84 total 52.95 last 327.29) - /unlink 16124 (time 1756810098.84 total 62.95 last 285.00) - /unlink 19045 (time 1756810108.85 total 72.96 last 291.93) - /unlink 20000 (time 1756810112.55 total 76.66 last 257.65) - /unlink 22033 (time 1756810122.56 total 86.67 last 203.25) - /unlink 24310 (time 1756810132.56 total 96.67 last 227.69) - /unlink 26636 (time 1756810142.56 total 106.67 last 232.54) - /unlink 29681 (time 1756810152.56 total 116.67 last 304.44) - /unlink 32836 (time 1756810162.56 total 126.67 last 315.48) - /unlink 36045 (time 1756810172.57 total 136.68 last 320.82) - /unlink 39679 (time 1756810182.57 total 146.68 last 363.37) - /unlink 42520 (time 1756810192.57 total 156.68 last 284.00) - /unlink 44694 (time 1756810202.57 total 166.68 last 217.38) - /unlink 46681 (time 1756810212.57 total 176.68 last 198.68) - /unlink 48705 (time 1756810222.58 total 186.69 last 202.32) - /unlink 50000 (time 1756810228.72 total 192.83 last 210.84) - /unlink 52050 (time 1756810238.72 total 202.83 last 204.94) - /unlink 54343 (time 1756810248.72 total 212.83 last 229.21) - /unlink 56611 (time 1756810258.73 total 222.84 last 226.72) - /unlink 58784 (time 1756810268.73 total 232.84 last 217.24) - /unlink 60000 (time 1756810274.39 total 238.50 last 214.95) - /unlink 62341 (time 1756810284.39 total 248.50 last 234.07) total: 64768 /unlink in 257.40 seconds: 251.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2407 (time 1756810325.73 total 10.00 last 240.67) - open/close 4494 (time 1756810335.73 total 20.00 last 208.66) - open/close 7073 (time 1756810345.73 total 30.01 last 257.85) - open/close 8947 (time 1756810355.73 total 40.01 last 187.38) - open/close 10000 (time 1756810361.88 total 46.16 last 171.24) - open/close 11582 (time 1756810371.88 total 56.16 last 158.19) - open/close 13161 (time 1756810381.89 total 66.16 last 157.83) - open/close 14845 (time 1756810391.89 total 76.16 last 168.40) - open/close 16821 (time 1756810401.89 total 86.16 last 197.58) - open/close 18626 (time 1756810411.89 total 96.16 last 180.46) - open/close 20000 (time 1756810418.15 total 102.43 last 219.33) - open/close 22433 (time 1756810428.16 total 112.43 last 243.19) - open/close 24478 (time 1756810438.16 total 122.44 last 204.45) - open/close 26491 (time 1756810448.16 total 132.44 last 201.29) - open/close 28452 (time 1756810458.17 total 142.44 last 195.95) - open/close 30000 (time 1756810467.83 total 152.10 last 160.33) - open/close 31923 (time 1756810477.83 total 162.10 last 192.30) - open/close 33869 (time 1756810487.83 total 172.10 last 194.57) - open/close 35666 (time 1756810497.83 total 182.10 last 179.66) - open/close 37777 (time 1756810507.83 total 192.10 last 211.06) - open/close 39609 (time 1756810517.84 total 202.11 last 183.12) - open/close 40000 (time 1756810520.34 total 204.62 last 156.03) - open/close 41730 (time 1756810530.35 total 214.62 last 172.91) - open/close 43383 (time 1756810540.35 total 224.62 last 165.25) - open/close 45183 (time 1756810550.35 total 234.63 last 179.92) - open/close 47071 (time 1756810560.36 total 244.63 last 188.77) - open/close 48998 (time 1756810570.36 total 254.63 last 192.65) - open/close 50000 (time 1756810574.93 total 259.20 last 219.21) - open/close 51748 (time 1756810584.94 total 269.22 last 174.56) - open/close 53657 (time 1756810594.95 total 279.22 last 190.84) - open/close 55437 (time 1756810604.96 total 289.23 last 177.82) - open/close 57381 (time 1756810614.96 total 299.23 last 194.37) - open/close 58865 (time 1756810624.96 total 309.24 last 148.31) - open/close 60000 (time 1756810632.83 total 317.11 last 144.27) - open/close 61451 (time 1756810642.84 total 327.11 last 145.03) - open/close 62823 (time 1756810652.84 total 337.12 last 137.10) - open/close 64137 (time 1756810662.85 total 347.12 last 131.32) total: 64768 open/close in 351.87 seconds: 184.07 ops/second - /unlink 2604 (time 1756810679.43 total 10.00 last 260.34) - /unlink 5639 (time 1756810689.43 total 20.00 last 303.49) - /unlink 8242 (time 1756810699.44 total 30.00 last 260.24) - /unlink 10000 (time 1756810705.32 total 35.89 last 298.77) - /unlink 12724 (time 1756810715.32 total 45.89 last 272.37) - /unlink 14983 (time 1756810725.32 total 55.89 last 225.87) - /unlink 18600 (time 1756810735.32 total 65.89 last 361.68) - /unlink 20000 (time 1756810740.30 total 70.87 last 281.08) - /unlink 23842 (time 1756810750.31 total 80.87 last 384.13) - /unlink 27909 (time 1756810760.31 total 90.88 last 406.62) - /unlink 30000 (time 1756810766.08 total 96.65 last 362.00) - /unlink 33953 (time 1756810776.09 total 106.65 last 395.23) - /unlink 38528 (time 1756810786.09 total 116.66 last 457.33) - /unlink 40000 (time 1756810789.92 total 120.49 last 384.04) - /unlink 45993 (time 1756810799.92 total 130.49 last 599.28) - /unlink 50000 (time 1756810807.08 total 137.65 last 559.51) - /unlink 55426 (time 1756810817.09 total 147.66 last 542.50) - /unlink 60000 (time 1756810826.18 total 156.75 last 502.99) total: 64768 /unlink in 164.12 seconds: 394.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3324 (time 1756810866.93 total 10.00 last 332.33) - open/close 6062 (time 1756810876.93 total 20.00 last 273.79) - open/close 8429 (time 1756810886.93 total 30.00 last 236.69) - open/close 10000 (time 1756810892.35 total 35.42 last 289.81) - open/close 13191 (time 1756810902.35 total 45.42 last 319.08) - open/close 15601 (time 1756810912.35 total 55.43 last 240.95) - open/close 18174 (time 1756810922.35 total 65.43 last 257.28) - open/close 20000 (time 1756810929.12 total 72.20 last 269.78) - open/close 23096 (time 1756810939.12 total 82.20 last 309.59) - open/close 26218 (time 1756810949.12 total 92.20 last 312.18) - open/close 28532 (time 1756810959.13 total 102.20 last 231.28) - open/close 30000 (time 1756810965.82 total 108.89 last 219.47) - open/close 32944 (time 1756810975.82 total 118.89 last 294.36) - open/close 35677 (time 1756810985.82 total 128.89 last 273.29) - open/close 38235 (time 1756810995.82 total 138.89 last 255.74) - open/close 40000 (time 1756811001.91 total 144.98 last 289.98) - open/close 42688 (time 1756811011.91 total 154.98 last 268.74) - open/close 46002 (time 1756811021.91 total 164.98 last 331.36) - open/close 49745 (time 1756811031.91 total 174.99 last 374.22) - open/close 54083 (time 1756811041.92 total 184.99 last 433.75) - open/close 58065 (time 1756811051.92 total 194.99 last 398.09) - open/close 60000 (time 1756811057.60 total 200.68 last 340.40) - open/close 63832 (time 1756811067.60 total 210.68 last 383.11) total: 64768 open/close in 212.94 seconds: 304.16 ops/second - /unlink 5795 (time 1756811081.35 total 10.00 last 579.47) - /unlink 10000 (time 1756811090.66 total 19.31 last 451.64) - /unlink 14716 (time 1756811100.66 total 29.31 last 471.55) - /unlink 19063 (time 1756811110.66 total 39.31 last 434.65) - /unlink 22825 (time 1756811120.66 total 49.31 last 376.20) - /unlink 26945 (time 1756811130.66 total 59.31 last 411.94) - /unlink 30000 (time 1756811137.15 total 65.80 last 471.26) - /unlink 34851 (time 1756811147.15 total 75.80 last 485.03) - /unlink 39582 (time 1756811157.15 total 85.80 last 473.07) - /unlink 44473 (time 1756811167.15 total 95.80 last 489.03) - /unlink 49835 (time 1756811177.15 total 105.80 last 536.13) - /unlink 55791 (time 1756811187.15 total 115.80 last 595.54) - /unlink 60000 (time 1756811194.41 total 123.06 last 579.95) total: 64768 /unlink in 131.93 seconds: 490.91 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4646 (time 1756811237.79 total 10.00 last 464.52) - open/close 8777 (time 1756811247.79 total 20.00 last 413.05) - open/close 10000 (time 1756811250.73 total 22.94 last 416.58) - open/close 14244 (time 1756811260.73 total 32.94 last 424.32) - open/close 18740 (time 1756811270.73 total 42.94 last 449.54) - open/close 20000 (time 1756811273.36 total 45.57 last 478.76) - open/close 24311 (time 1756811283.37 total 55.58 last 431.01) - open/close 28503 (time 1756811293.37 total 65.58 last 419.15) - open/close 30000 (time 1756811296.28 total 68.49 last 513.99) - open/close 35031 (time 1756811306.28 total 78.49 last 503.09) - open/close 39408 (time 1756811316.28 total 88.49 last 437.64) - open/close 43358 (time 1756811326.28 total 98.49 last 394.96) - open/close 47552 (time 1756811336.39 total 108.60 last 414.96) - open/close 50000 (time 1756811342.61 total 114.82 last 393.18) - open/close 54702 (time 1756811352.62 total 124.83 last 470.12) - open/close 59535 (time 1756811362.62 total 134.83 last 483.22) - open/close 64660 (time 1756811372.62 total 144.83 last 512.43) total: 64768 open/close in 145.14 seconds: 446.24 ops/second - /unlink 7872 (time 1756811384.23 total 10.00 last 787.13) - /unlink 10000 (time 1756811386.47 total 12.24 last 951.76) - /unlink 17926 (time 1756811396.47 total 22.24 last 792.57) - /unlink 20000 (time 1756811399.11 total 24.88 last 784.28) - /unlink 27807 (time 1756811409.11 total 34.88 last 780.64) - /unlink 30000 (time 1756811411.57 total 37.34 last 894.12) - /unlink 38028 (time 1756811421.57 total 47.34 last 802.79) - /unlink 40000 (time 1756811424.21 total 49.98 last 746.62) - /unlink 47718 (time 1756811434.21 total 59.98 last 771.74) - /unlink 50000 (time 1756811437.24 total 63.01 last 752.03) - /unlink 57803 (time 1756811447.24 total 73.01 last 780.25) - /unlink 60000 (time 1756811450.03 total 75.79 last 789.71) total: 64768 /unlink in 81.54 seconds: 794.26 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5133 (time 1756811485.45 total 10.00 last 513.18) - open/close 10000 (time 1756811495.42 total 19.97 last 488.05) - open/close 14851 (time 1756811505.42 total 29.98 last 485.05) - open/close 19308 (time 1756811515.42 total 39.98 last 445.69) - open/close 23194 (time 1756811525.42 total 49.98 last 388.56) - open/close 28322 (time 1756811535.42 total 59.98 last 512.75) - open/close 30000 (time 1756811538.86 total 63.42 last 488.18) - open/close 34500 (time 1756811548.86 total 73.42 last 449.96) - open/close 37999 (time 1756811558.86 total 83.42 last 349.83) - open/close 40000 (time 1756811562.74 total 87.29 last 516.17) - open/close 44188 (time 1756811572.74 total 97.30 last 418.74) - open/close 48458 (time 1756811582.74 total 107.30 last 426.91) - open/close 50000 (time 1756811586.20 total 110.75 last 446.26) - open/close 54589 (time 1756811596.20 total 120.75 last 458.86) - open/close 59402 (time 1756811606.20 total 130.75 last 481.28) - open/close 64275 (time 1756811616.20 total 140.76 last 487.28) total: 64768 open/close in 141.87 seconds: 456.53 ops/second - /unlink 7250 (time 1756811628.66 total 10.00 last 724.99) - /unlink 10000 (time 1756811631.78 total 13.12 last 882.32) - /unlink 17380 (time 1756811641.78 total 23.12 last 737.96) - /unlink 20000 (time 1756811645.23 total 26.57 last 758.65) - /unlink 28440 (time 1756811655.23 total 36.57 last 843.94) - /unlink 30000 (time 1756811657.51 total 38.84 last 686.77) - /unlink 38357 (time 1756811667.51 total 48.84 last 835.62) - /unlink 40000 (time 1756811669.68 total 51.02 last 756.27) - /unlink 47651 (time 1756811679.68 total 61.02 last 765.09) - /unlink 50000 (time 1756811682.46 total 63.80 last 843.85) - /unlink 58483 (time 1756811692.46 total 73.80 last 848.23) - /unlink 60000 (time 1756811694.53 total 75.87 last 733.30) total: 64768 /unlink in 82.87 seconds: 781.58 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg423-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg423-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg423-server stop mds service on oleg423-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg423-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg423-server' oleg423-server: oleg423-server.virtnet: executing unload_modules_local oleg423-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg423-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-server: ssh exited with exit code 2 pdsh@oleg423-client: oleg423-client: ssh exited with exit code 2