== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:08:50 (1781046530) Stopping clients: oleg251-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg251-client.virtnet /mnt/lustre2 (opts:-f) oleg251-server: oleg251-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing load_modules_local oleg251-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg251-server: Loading modules from /home/green/git/lustre-release/lustre oleg251-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg251-server: detected 4 online CPUs by sysfs oleg251-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg251-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg251-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg251-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg251-server: oleg251-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg251-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg251-client.virtnet: -o user_xattr,flock oleg251-server@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 16166 358 15808 3% /mnt/lustre[OST:0] filesystem_summary: 16137 329 15808 3% /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 3590 (time 1781046578.25 total 10.00 last 358.96) - open/close 6643 (time 1781046588.25 total 20.01 last 305.17) - open/close 9873 (time 1781046598.25 total 30.01 last 322.98) - open/close 13343 (time 1781046608.25 total 40.01 last 346.97) - open/close 15942 (time 1781046618.26 total 50.01 last 259.79) - open/close 17957 (time 1781046628.26 total 60.02 last 201.42) - open/close 20000 (time 1781046637.72 total 69.47 last 215.96) - open/close 22289 (time 1781046647.72 total 79.48 last 228.85) - open/close 24910 (time 1781046657.73 total 89.48 last 261.98) - open/close 27743 (time 1781046667.73 total 99.49 last 283.12) - open/close 30000 (time 1781046673.75 total 105.50 last 375.42) - open/close 32432 (time 1781046683.75 total 115.50 last 243.14) - open/close 34921 (time 1781046693.75 total 125.50 last 248.88) - open/close 37451 (time 1781046703.75 total 135.50 last 252.98) - open/close 39763 (time 1781046713.75 total 145.50 last 231.19) - open/close 42285 (time 1781046723.75 total 155.50 last 252.19) - open/close 44609 (time 1781046733.75 total 165.51 last 232.37) - open/close 46837 (time 1781046743.76 total 175.51 last 222.68) - open/close 48949 (time 1781046753.76 total 185.52 last 211.10) - open/close 50000 (time 1781046758.85 total 190.61 last 206.51) - open/close 51912 (time 1781046768.85 total 200.61 last 191.16) - open/close 53925 (time 1781046778.85 total 210.61 last 201.28) - open/close 56363 (time 1781046788.86 total 220.61 last 243.72) - open/close 58965 (time 1781046798.86 total 230.61 last 260.15) - open/close 60000 (time 1781046802.35 total 234.10 last 296.62) - open/close 62820 (time 1781046812.35 total 244.10 last 281.96) total: 64768 open/close in 250.79 seconds: 258.25 ops/second - /unlink 4196 (time 1781046830.81 total 10.00 last 419.59) - /unlink 8386 (time 1781046840.81 total 20.00 last 418.95) - /unlink 10000 (time 1781046843.15 total 22.34 last 690.57) - /unlink 16673 (time 1781046853.15 total 32.34 last 667.26) - /unlink 20000 (time 1781046858.02 total 37.21 last 682.95) - /unlink 25407 (time 1781046868.02 total 47.21 last 540.70) - /unlink 29892 (time 1781046878.02 total 57.21 last 448.39) - /unlink 35143 (time 1781046888.02 total 67.22 last 525.01) - /unlink 39637 (time 1781046898.02 total 77.22 last 449.39) - /unlink 44658 (time 1781046908.02 total 87.22 last 502.05) - /unlink 49713 (time 1781046918.02 total 97.22 last 505.46) - /unlink 54980 (time 1781046928.02 total 107.22 last 526.68) - /unlink 60000 (time 1781046936.07 total 115.26 last 623.96) total: 64768 /unlink in 122.59 seconds: 528.32 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3109 (time 1781046975.38 total 10.00 last 310.78) - open/close 6797 (time 1781046985.38 total 20.01 last 368.75) - open/close 10000 (time 1781046995.00 total 29.62 last 333.01) - open/close 13033 (time 1781047005.00 total 39.62 last 303.27) - open/close 16709 (time 1781047015.00 total 49.63 last 367.54) - open/close 20000 (time 1781047023.33 total 57.95 last 395.43) - open/close 23405 (time 1781047033.33 total 67.95 last 340.46) - open/close 26378 (time 1781047043.33 total 77.95 last 297.14) - open/close 29628 (time 1781047053.34 total 87.96 last 324.92) - open/close 32857 (time 1781047063.34 total 97.96 last 322.84) - open/close 35447 (time 1781047073.34 total 107.96 last 258.96) - open/close 38117 (time 1781047083.34 total 117.96 last 266.99) - open/close 40000 (time 1781047091.84 total 126.47 last 221.38) - open/close 42010 (time 1781047101.85 total 136.47 last 200.97) - open/close 44833 (time 1781047111.85 total 146.47 last 282.28) - open/close 48576 (time 1781047121.85 total 156.47 last 374.27) - open/close 50000 (time 1781047125.48 total 160.11 last 391.68) - open/close 53535 (time 1781047135.48 total 170.11 last 353.46) - open/close 56942 (time 1781047145.48 total 180.11 last 340.70) - open/close 60000 (time 1781047154.38 total 189.00 last 343.66) - open/close 62929 (time 1781047164.39 total 199.01 last 292.82) total: 64768 open/close in 205.94 seconds: 314.50 ops/second - /unlink 6501 (time 1781047182.78 total 10.00 last 650.07) - /unlink 10000 (time 1781047187.54 total 14.76 last 734.93) - /unlink 17225 (time 1781047197.55 total 24.76 last 722.28) - /unlink 20000 (time 1781047203.44 total 30.65 last 471.13) - /unlink 25465 (time 1781047213.44 total 40.66 last 546.47) - /unlink 30000 (time 1781047220.38 total 47.59 last 653.59) - /unlink 36054 (time 1781047230.38 total 57.60 last 605.18) - /unlink 40000 (time 1781047238.06 total 65.28 last 513.80) - /unlink 45857 (time 1781047248.06 total 75.28 last 585.62) - /unlink 50000 (time 1781047255.08 total 82.30 last 589.92) - /unlink 56264 (time 1781047265.08 total 92.30 last 626.40) - /unlink 60000 (time 1781047271.68 total 98.89 last 566.72) total: 64768 /unlink in 106.45 seconds: 608.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3693 (time 1781047306.88 total 10.00 last 369.29) - open/close 7999 (time 1781047316.88 total 20.00 last 430.58) - open/close 10000 (time 1781047321.80 total 24.92 last 406.72) - open/close 14118 (time 1781047331.80 total 34.92 last 411.72) - open/close 18385 (time 1781047341.80 total 44.92 last 426.63) - open/close 20000 (time 1781047345.68 total 48.80 last 416.34) - open/close 24733 (time 1781047355.68 total 58.80 last 473.29) - open/close 28344 (time 1781047365.68 total 68.80 last 361.07) - open/close 30000 (time 1781047370.54 total 73.67 last 340.60) - open/close 33627 (time 1781047380.54 total 83.67 last 362.63) - open/close 38145 (time 1781047390.54 total 93.67 last 451.75) - open/close 40000 (time 1781047395.31 total 98.44 last 389.01) - open/close 44069 (time 1781047405.31 total 108.44 last 406.86) - open/close 48120 (time 1781047415.31 total 118.44 last 405.07) - open/close 50000 (time 1781047420.02 total 123.15 last 399.46) - open/close 54398 (time 1781047430.02 total 133.15 last 439.73) - open/close 58122 (time 1781047440.03 total 143.15 last 372.29) - open/close 60000 (time 1781047445.01 total 148.13 last 376.80) - open/close 64074 (time 1781047455.01 total 158.14 last 407.22) total: 64768 open/close in 159.87 seconds: 405.13 ops/second - /unlink 7061 (time 1781047468.17 total 10.00 last 706.05) - /unlink 10000 (time 1781047472.36 total 14.19 last 701.72) - /unlink 17741 (time 1781047482.36 total 24.19 last 774.05) - /unlink 20000 (time 1781047485.48 total 27.31 last 723.90) - /unlink 28105 (time 1781047495.48 total 37.31 last 810.49) - /unlink 36829 (time 1781047505.48 total 47.31 last 872.37) - /unlink 40000 (time 1781047508.79 total 50.62 last 959.13) - /unlink 47696 (time 1781047518.79 total 60.62 last 769.54) - /unlink 50000 (time 1781047521.87 total 63.70 last 747.16) - /unlink 58296 (time 1781047531.87 total 73.70 last 829.59) total: 64768 /unlink in 81.16 seconds: 798.03 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4847 (time 1781047573.65 total 10.00 last 484.65) - open/close 9699 (time 1781047583.65 total 20.00 last 485.18) - open/close 14435 (time 1781047593.65 total 30.00 last 473.59) - open/close 19852 (time 1781047603.65 total 40.00 last 541.62) - open/close 24731 (time 1781047613.65 total 50.01 last 487.78) - open/close 30000 (time 1781047623.23 total 59.58 last 550.09) - open/close 35247 (time 1781047633.23 total 69.59 last 524.61) - open/close 40000 (time 1781047642.42 total 78.78 last 517.20) - open/close 44604 (time 1781047652.42 total 88.78 last 460.34) - open/close 49104 (time 1781047662.42 total 98.78 last 449.93) - open/close 50000 (time 1781047664.51 total 100.86 last 429.91) - open/close 54471 (time 1781047674.51 total 110.86 last 447.05) - open/close 59650 (time 1781047684.51 total 120.87 last 517.78) - open/close 64528 (time 1781047694.51 total 130.87 last 487.73) total: 64768 open/close in 131.40 seconds: 492.91 ops/second - /unlink 8089 (time 1781047706.20 total 10.00 last 808.85) - /unlink 10000 (time 1781047708.22 total 12.02 last 947.25) - /unlink 18596 (time 1781047718.22 total 22.02 last 859.58) - /unlink 27211 (time 1781047728.22 total 32.02 last 861.46) - /unlink 30000 (time 1781047731.82 total 35.62 last 775.09) - /unlink 38477 (time 1781047741.82 total 45.62 last 847.66) - /unlink 46696 (time 1781047751.82 total 55.62 last 821.81) - /unlink 50000 (time 1781047755.27 total 59.07 last 957.37) - /unlink 58777 (time 1781047765.27 total 69.07 last 877.68) total: 64768 /unlink in 76.35 seconds: 848.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4946 (time 1781047803.62 total 10.00 last 494.51) - open/close 9806 (time 1781047813.62 total 20.00 last 485.98) - open/close 14629 (time 1781047823.62 total 30.00 last 482.29) - open/close 19537 (time 1781047833.62 total 40.00 last 490.80) - open/close 24460 (time 1781047843.62 total 50.00 last 492.24) - open/close 29329 (time 1781047853.62 total 60.00 last 486.84) - open/close 34200 (time 1781047863.62 total 70.01 last 487.09) - open/close 38926 (time 1781047873.62 total 80.01 last 472.58) - open/close 40000 (time 1781047875.87 total 82.26 last 476.93) - open/close 44793 (time 1781047885.88 total 92.26 last 479.22) - open/close 49443 (time 1781047895.88 total 102.26 last 464.93) - open/close 53881 (time 1781047905.88 total 112.26 last 443.75) - open/close 58521 (time 1781047915.88 total 122.26 last 463.96) - open/close 60000 (time 1781047919.29 total 125.67 last 434.15) - open/close 64648 (time 1781047929.29 total 135.67 last 464.71) total: 64768 open/close in 135.92 seconds: 476.52 ops/second - /unlink 7368 (time 1781047940.82 total 10.00 last 736.79) - /unlink 10000 (time 1781047944.04 total 13.22 last 818.00) - /unlink 18247 (time 1781047954.04 total 23.22 last 824.67) - /unlink 20000 (time 1781047956.43 total 25.60 last 734.66) - /unlink 28865 (time 1781047966.43 total 35.60 last 886.46) - /unlink 37790 (time 1781047976.43 total 45.61 last 892.38) - /unlink 40000 (time 1781047978.77 total 47.94 last 945.38) - /unlink 48670 (time 1781047988.77 total 57.94 last 866.95) - /unlink 57537 (time 1781047998.77 total 67.94 last 886.67) - /unlink 60000 (time 1781048001.49 total 70.67 last 903.50) total: 64768 /unlink in 76.20 seconds: 850.01 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg251-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg251-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg251-server stop mds service on oleg251-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg251-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg251-server' oleg251-server: oleg251-server.virtnet: executing unload_modules_local oleg251-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg251-server: LNET ready to unload oleg251-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-client: ssh exited with exit code 2 pdsh@oleg251-client: oleg251-server: ssh exited with exit code 2