== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:14:10 (1767384850) Stopping clients: oleg359-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg359-client.virtnet /mnt/lustre2 (opts:-f) oleg359-server: oleg359-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing load_modules_local oleg359-server: Loading modules from /home/green/git/lustre-release/lustre oleg359-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1033: echo: write error: Device or resource busy oleg359-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg359-server: detected 4 online CPUs by sysfs oleg359-server: MODOPTS_LIBCFS= oleg359-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg359-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg359-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg359-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg359-server: oleg359-server.virtnet: executing set_default_debug -1 all pdsh@oleg359-client: oleg359-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg359-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg359-client.virtnet: -o user_xattr,flock 192.168.203.159@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 3896 (time 1767384896.60 total 10.00 last 389.53) - open/close 8056 (time 1767384906.60 total 20.00 last 415.98) - open/close 10000 (time 1767384911.14 total 24.54 last 428.29) - open/close 13838 (time 1767384921.14 total 34.54 last 383.75) - open/close 17946 (time 1767384931.14 total 44.54 last 410.78) - open/close 20000 (time 1767384935.96 total 49.36 last 426.26) - open/close 24603 (time 1767384945.96 total 59.36 last 460.24) - open/close 29138 (time 1767384955.96 total 69.36 last 453.45) - open/close 33515 (time 1767384965.96 total 79.37 last 437.64) - open/close 37874 (time 1767384975.96 total 89.37 last 435.88) - open/close 40000 (time 1767384980.26 total 93.67 last 494.03) - open/close 44770 (time 1767384990.27 total 103.67 last 476.91) - open/close 49392 (time 1767385000.27 total 113.67 last 462.14) - open/close 53819 (time 1767385010.27 total 123.68 last 442.52) - open/close 57277 (time 1767385020.27 total 133.68 last 345.77) - open/close 60000 (time 1767385027.11 total 140.51 last 398.40) - open/close 63772 (time 1767385037.11 total 150.51 last 377.19) total: 64768 open/close in 152.79 seconds: 423.90 ops/second - /unlink 7852 (time 1767385050.81 total 10.00 last 785.11) - /unlink 10000 (time 1767385053.29 total 12.48 last 867.89) - /unlink 18100 (time 1767385063.29 total 22.48 last 810.00) - /unlink 20000 (time 1767385065.97 total 25.15 last 709.41) - /unlink 27973 (time 1767385075.97 total 35.16 last 797.21) - /unlink 30000 (time 1767385078.31 total 37.50 last 865.32) - /unlink 37011 (time 1767385088.31 total 47.50 last 701.10) - /unlink 40000 (time 1767385092.92 total 52.10 last 649.19) - /unlink 45310 (time 1767385102.92 total 62.10 last 530.95) - /unlink 50000 (time 1767385109.83 total 69.02 last 678.55) - /unlink 55916 (time 1767385119.83 total 79.02 last 591.58) - /unlink 60000 (time 1767385125.94 total 85.13 last 667.88) total: 64768 /unlink in 91.75 seconds: 705.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2462 (time 1767385162.75 total 10.00 last 246.12) - open/close 5374 (time 1767385172.75 total 20.00 last 291.18) - open/close 8572 (time 1767385182.75 total 30.01 last 319.72) - open/close 10000 (time 1767385187.56 total 34.81 last 297.00) - open/close 12930 (time 1767385197.56 total 44.82 last 292.98) - open/close 15899 (time 1767385207.56 total 54.82 last 296.79) - open/close 18418 (time 1767385217.57 total 64.82 last 251.83) - open/close 20000 (time 1767385223.82 total 71.08 last 252.95) - open/close 23105 (time 1767385233.82 total 81.08 last 310.47) - open/close 26004 (time 1767385243.82 total 91.08 last 289.84) - open/close 29130 (time 1767385253.83 total 101.08 last 312.56) - open/close 30000 (time 1767385257.43 total 104.69 last 241.23) - open/close 32830 (time 1767385267.43 total 114.69 last 282.96) - open/close 35471 (time 1767385277.44 total 124.69 last 264.07) - open/close 38218 (time 1767385287.44 total 134.69 last 274.68) - open/close 40000 (time 1767385293.93 total 141.18 last 274.55) - open/close 42892 (time 1767385303.93 total 151.18 last 289.09) - open/close 45942 (time 1767385313.93 total 161.19 last 304.96) - open/close 48424 (time 1767385323.93 total 171.19 last 248.13) - open/close 50000 (time 1767385328.85 total 176.10 last 320.60) - open/close 52871 (time 1767385338.85 total 186.11 last 287.04) - open/close 55483 (time 1767385348.85 total 196.11 last 261.19) - open/close 57999 (time 1767385358.86 total 206.11 last 251.51) - open/close 60000 (time 1767385366.53 total 213.78 last 260.87) - open/close 62344 (time 1767385376.53 total 223.78 last 234.37) total: 64768 open/close in 232.40 seconds: 278.70 ops/second - /unlink 6112 (time 1767385397.15 total 10.00 last 611.15) - /unlink 10000 (time 1767385403.58 total 16.43 last 604.89) - /unlink 14007 (time 1767385413.58 total 26.43 last 400.68) - /unlink 20000 (time 1767385422.32 total 35.18 last 685.13) - /unlink 26492 (time 1767385432.32 total 45.18 last 649.16) - /unlink 30000 (time 1767385438.08 total 50.93 last 609.49) - /unlink 37840 (time 1767385448.08 total 60.93 last 783.99) - /unlink 40000 (time 1767385450.56 total 63.42 last 869.95) - /unlink 46481 (time 1767385460.56 total 73.42 last 648.07) - /unlink 50000 (time 1767385466.76 total 79.62 last 567.46) - /unlink 54921 (time 1767385476.76 total 89.62 last 492.07) - /unlink 60000 (time 1767385486.58 total 99.43 last 517.51) total: 64768 /unlink in 108.42 seconds: 597.38 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2382 (time 1767385525.20 total 10.00 last 238.14) - open/close 4757 (time 1767385535.21 total 20.01 last 237.43) - open/close 7057 (time 1767385545.21 total 30.01 last 229.96) - open/close 9576 (time 1767385555.21 total 40.01 last 251.82) - open/close 12288 (time 1767385565.21 total 50.01 last 271.15) - open/close 15143 (time 1767385575.21 total 60.01 last 285.50) - open/close 17677 (time 1767385585.22 total 70.01 last 253.38) - open/close 20000 (time 1767385594.93 total 79.72 last 239.21) - open/close 22275 (time 1767385604.93 total 89.72 last 227.49) - open/close 24536 (time 1767385614.93 total 99.73 last 226.08) - open/close 27167 (time 1767385624.93 total 109.73 last 263.06) - open/close 30000 (time 1767385634.53 total 119.33 last 295.14) - open/close 33136 (time 1767385644.53 total 129.33 last 313.52) - open/close 35588 (time 1767385654.53 total 139.33 last 245.11) - open/close 37996 (time 1767385664.54 total 149.33 last 240.75) - open/close 40000 (time 1767385672.92 total 157.71 last 239.12) - open/close 42566 (time 1767385682.92 total 167.72 last 256.57) - open/close 44911 (time 1767385692.94 total 177.74 last 234.03) - open/close 47653 (time 1767385702.94 total 187.74 last 274.09) - open/close 50000 (time 1767385710.49 total 195.29 last 311.04) - open/close 52771 (time 1767385720.49 total 205.29 last 277.09) - open/close 55076 (time 1767385730.49 total 215.29 last 230.45) - open/close 57781 (time 1767385740.49 total 225.29 last 270.38) - open/close 60000 (time 1767385748.74 total 233.54 last 269.16) - open/close 62735 (time 1767385758.74 total 243.54 last 273.42) total: 64768 open/close in 252.10 seconds: 256.91 ops/second - /unlink 6276 (time 1767385778.76 total 10.00 last 627.53) - /unlink 10000 (time 1767385783.03 total 14.27 last 873.08) - /unlink 14288 (time 1767385793.03 total 24.27 last 428.78) - /unlink 20000 (time 1767385800.75 total 31.98 last 740.18) - /unlink 26487 (time 1767385810.75 total 41.98 last 648.67) - /unlink 30000 (time 1767385817.96 total 49.20 last 486.79) - /unlink 35664 (time 1767385827.96 total 59.20 last 566.32) - /unlink 40000 (time 1767385833.87 total 65.11 last 734.03) - /unlink 48683 (time 1767385843.89 total 75.12 last 867.06) - /unlink 54167 (time 1767385853.89 total 85.13 last 548.27) - /unlink 58830 (time 1767385863.89 total 95.13 last 466.27) - /unlink 60000 (time 1767385865.98 total 97.22 last 558.20) total: 64768 /unlink in 107.12 seconds: 604.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2442 (time 1767385958.86 total 10.00 last 244.09) - open/close 4998 (time 1767385968.87 total 20.01 last 255.51) - open/close 7343 (time 1767385978.87 total 30.01 last 234.50) - open/close 10000 (time 1767385988.03 total 39.17 last 290.14) - open/close 12645 (time 1767385998.03 total 49.17 last 264.39) - open/close 15485 (time 1767386008.03 total 59.17 last 283.98) - open/close 18411 (time 1767386018.03 total 69.17 last 292.56) - open/close 20000 (time 1767386024.22 total 75.36 last 256.59) - open/close 22591 (time 1767386034.23 total 85.37 last 259.03) - open/close 24946 (time 1767386044.23 total 95.37 last 235.48) - open/close 27349 (time 1767386054.23 total 105.37 last 240.25) - open/close 30000 (time 1767386064.12 total 115.26 last 268.13) - open/close 32635 (time 1767386074.12 total 125.26 last 263.48) - open/close 35445 (time 1767386084.12 total 135.26 last 280.98) - open/close 37966 (time 1767386094.12 total 145.26 last 252.07) - open/close 40000 (time 1767386102.92 total 154.06 last 231.14) - open/close 42580 (time 1767386112.92 total 164.06 last 257.96) - open/close 45204 (time 1767386122.92 total 174.07 last 262.29) - open/close 47830 (time 1767386132.93 total 184.07 last 262.51) - open/close 50000 (time 1767386141.25 total 192.39 last 260.84) - open/close 52435 (time 1767386151.25 total 202.39 last 243.44) - open/close 55254 (time 1767386161.25 total 212.39 last 281.82) - open/close 58237 (time 1767386171.26 total 222.40 last 298.22) - open/close 60000 (time 1767386177.39 total 228.53 last 287.22) - open/close 62664 (time 1767386187.39 total 238.53 last 266.39) total: 64768 open/close in 246.05 seconds: 263.23 ops/second - /unlink 6018 (time 1767386206.25 total 10.00 last 601.77) - /unlink 10000 (time 1767386212.01 total 15.77 last 690.38) - /unlink 17033 (time 1767386222.02 total 25.77 last 703.23) - /unlink 20000 (time 1767386226.36 total 30.11 last 683.46) - /unlink 28649 (time 1767386236.36 total 40.11 last 864.82) - /unlink 34426 (time 1767386246.36 total 50.11 last 577.70) - /unlink 40000 (time 1767386254.27 total 58.03 last 704.31) - /unlink 48426 (time 1767386264.28 total 68.03 last 842.10) - /unlink 56281 (time 1767386274.28 total 78.03 last 785.49) - /unlink 60000 (time 1767386280.21 total 83.96 last 627.27) - /unlink 64468 (time 1767386290.21 total 93.96 last 446.69) total: 64768 /unlink in 94.78 seconds: 683.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2257 (time 1767386378.72 total 10.00 last 225.64) - open/close 4502 (time 1767386388.72 total 20.00 last 224.45) - open/close 6712 (time 1767386398.72 total 30.01 last 220.95) - open/close 9377 (time 1767386408.73 total 40.01 last 266.47) - open/close 10000 (time 1767386411.46 total 42.74 last 228.12) - open/close 12635 (time 1767386421.46 total 52.74 last 263.50) - open/close 15153 (time 1767386431.46 total 62.74 last 251.78) - open/close 17514 (time 1767386441.46 total 72.74 last 236.08) - open/close 19888 (time 1767386451.46 total 82.74 last 237.33) - open/close 22057 (time 1767386461.46 total 92.75 last 216.85) - open/close 24467 (time 1767386471.47 total 102.75 last 240.91) - open/close 26727 (time 1767386481.47 total 112.75 last 225.90) - open/close 28997 (time 1767386491.47 total 122.76 last 226.94) - open/close 30000 (time 1767386495.79 total 127.07 last 232.52) - open/close 32452 (time 1767386505.79 total 137.07 last 245.15) - open/close 34746 (time 1767386515.79 total 147.07 last 229.34) - open/close 37126 (time 1767386525.79 total 157.08 last 237.97) - open/close 39638 (time 1767386535.80 total 167.08 last 251.10) - open/close 42186 (time 1767386545.80 total 177.08 last 254.76) - open/close 44650 (time 1767386555.80 total 187.09 last 246.30) - open/close 47089 (time 1767386565.81 total 197.09 last 243.83) - open/close 49704 (time 1767386575.81 total 207.09 last 261.43) - open/close 52127 (time 1767386585.81 total 217.09 last 242.23) - open/close 54717 (time 1767386595.81 total 227.10 last 258.91) - open/close 57088 (time 1767386605.82 total 237.10 last 237.01) - open/close 59578 (time 1767386615.82 total 247.10 last 248.93) - open/close 62053 (time 1767386625.84 total 257.12 last 247.06) - open/close 64459 (time 1767386635.84 total 267.12 last 240.53) total: 64768 open/close in 268.38 seconds: 241.33 ops/second - /unlink 5778 (time 1767386649.19 total 10.00 last 577.67) - /unlink 10000 (time 1767386655.99 total 16.80 last 621.11) - /unlink 18253 (time 1767386665.99 total 26.80 last 825.30) - /unlink 20000 (time 1767386668.04 total 28.85 last 851.75) - /unlink 26079 (time 1767386678.04 total 38.85 last 607.82) - /unlink 30000 (time 1767386687.02 total 47.83 last 436.59) - /unlink 34522 (time 1767386697.02 total 57.83 last 452.13) - /unlink 40000 (time 1767386706.37 total 67.18 last 586.10) - /unlink 45601 (time 1767386716.37 total 77.18 last 559.98) - /unlink 50000 (time 1767386724.86 total 85.67 last 518.27) - /unlink 55484 (time 1767386734.87 total 95.68 last 547.71) - /unlink 60000 (time 1767386743.89 total 104.71 last 500.58) total: 64768 /unlink in 112.94 seconds: 573.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg359-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg359-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg359-server stop mds service on oleg359-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg359-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg359-server' oleg359-server: oleg359-server.virtnet: executing unload_modules_local oleg359-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg359-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-client: ssh exited with exit code 2 pdsh@oleg359-client: oleg359-server: ssh exited with exit code 2