== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:42:11 (1773232931) Stopping clients: oleg453-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg453-client.virtnet /mnt/lustre2 (opts:-f) oleg453-server: oleg453-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing load_modules_local oleg453-server: Loading modules from /home/green/git/lustre-release/lustre oleg453-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg453-server: detected 4 online CPUs by sysfs oleg453-server: MODOPTS_LIBCFS= oleg453-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg453-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg453-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg453-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg453-server: oleg453-server.virtnet: executing set_default_debug -1 all pdsh@oleg453-client: oleg453-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg453-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg453-client.virtnet: -o user_xattr,flock 192.168.204.153@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2545 (time 1773232983.71 total 10.00 last 254.48) - open/close 5051 (time 1773232993.71 total 20.00 last 250.59) - open/close 7370 (time 1773233003.73 total 30.01 last 231.65) - open/close 9248 (time 1773233013.73 total 40.01 last 187.76) - open/close 10000 (time 1773233017.03 total 43.31 last 227.95) - open/close 11855 (time 1773233027.03 total 53.31 last 185.47) - open/close 13476 (time 1773233037.04 total 63.33 last 161.93) - open/close 15177 (time 1773233047.04 total 73.33 last 170.04) - open/close 17004 (time 1773233057.04 total 83.33 last 182.65) - open/close 19351 (time 1773233067.06 total 93.34 last 234.44) - open/close 20000 (time 1773233070.30 total 96.59 last 199.82) - open/close 23288 (time 1773233080.31 total 106.59 last 328.73) - open/close 26152 (time 1773233090.31 total 116.59 last 286.36) - open/close 28460 (time 1773233100.31 total 126.59 last 230.79) - open/close 30000 (time 1773233106.58 total 132.87 last 245.54) - open/close 32905 (time 1773233116.58 total 142.87 last 290.50) - open/close 36040 (time 1773233126.58 total 152.87 last 313.42) - open/close 39179 (time 1773233136.58 total 162.87 last 313.87) - open/close 40000 (time 1773233138.84 total 165.13 last 363.82) - open/close 42900 (time 1773233148.84 total 175.13 last 289.95) - open/close 45429 (time 1773233158.84 total 185.13 last 252.84) - open/close 47594 (time 1773233168.85 total 195.13 last 216.44) - open/close 50000 (time 1773233176.04 total 202.33 last 334.53) - open/close 53314 (time 1773233186.04 total 212.33 last 331.30) - open/close 56065 (time 1773233196.04 total 222.33 last 275.04) - open/close 58702 (time 1773233206.04 total 232.33 last 263.70) - open/close 60000 (time 1773233210.25 total 236.53 last 308.86) - open/close 64165 (time 1773233220.25 total 246.53 last 416.43) total: 64768 open/close in 248.05 seconds: 261.10 ops/second - /unlink 5427 (time 1773233233.23 total 10.00 last 542.58) - /unlink 10000 (time 1773233242.77 total 19.55 last 478.97) - /unlink 14164 (time 1773233252.78 total 29.55 last 416.24) - /unlink 18877 (time 1773233262.79 total 39.57 last 470.70) - /unlink 20000 (time 1773233264.89 total 41.67 last 534.61) - /unlink 25381 (time 1773233274.89 total 51.67 last 538.05) - /unlink 30000 (time 1773233283.72 total 60.50 last 523.10) - /unlink 35919 (time 1773233293.72 total 70.50 last 591.81) - /unlink 40000 (time 1773233302.00 total 78.78 last 492.85) - /unlink 44569 (time 1773233312.01 total 88.78 last 456.76) - /unlink 49533 (time 1773233322.01 total 98.79 last 496.25) - /unlink 53588 (time 1773233332.01 total 108.79 last 405.47) - /unlink 58401 (time 1773233342.01 total 118.79 last 481.26) - /unlink 60000 (time 1773233345.18 total 121.96 last 504.19) total: 64768 /unlink in 129.23 seconds: 501.19 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3612 (time 1773233381.23 total 10.00 last 361.12) - open/close 6858 (time 1773233391.23 total 20.00 last 324.51) - open/close 10000 (time 1773233399.01 total 27.79 last 403.69) - open/close 14005 (time 1773233409.01 total 37.79 last 400.45) - open/close 17392 (time 1773233419.02 total 47.79 last 338.62) - open/close 20000 (time 1773233425.37 total 54.15 last 410.32) - open/close 23995 (time 1773233435.37 total 64.15 last 399.46) - open/close 27515 (time 1773233445.38 total 74.15 last 351.84) - open/close 30000 (time 1773233451.17 total 79.95 last 428.92) - open/close 33351 (time 1773233461.17 total 89.95 last 335.10) - open/close 37564 (time 1773233471.17 total 99.95 last 421.22) - open/close 40000 (time 1773233476.66 total 105.43 last 444.04) - open/close 44008 (time 1773233486.66 total 115.44 last 400.72) - open/close 47965 (time 1773233496.66 total 125.44 last 395.66) - open/close 50000 (time 1773233501.04 total 129.82 last 464.32) - open/close 54212 (time 1773233511.06 total 139.83 last 420.63) - open/close 58000 (time 1773233521.06 total 149.83 last 378.76) - open/close 60000 (time 1773233526.74 total 155.51 last 352.23) - open/close 64101 (time 1773233536.74 total 165.52 last 410.01) total: 64768 open/close in 167.07 seconds: 387.66 ops/second - /unlink 7449 (time 1773233549.66 total 10.00 last 744.85) - /unlink 10000 (time 1773233553.86 total 14.19 last 608.44) - /unlink 17648 (time 1773233563.86 total 24.19 last 764.77) - /unlink 20000 (time 1773233566.67 total 27.01 last 836.36) - /unlink 27062 (time 1773233576.67 total 37.01 last 706.17) - /unlink 30000 (time 1773233581.34 total 41.68 last 628.88) - /unlink 36995 (time 1773233591.34 total 51.68 last 699.39) - /unlink 40000 (time 1773233596.05 total 56.39 last 638.27) - /unlink 46907 (time 1773233606.05 total 66.39 last 690.68) - /unlink 50000 (time 1773233610.80 total 71.14 last 650.99) - /unlink 57166 (time 1773233620.80 total 81.14 last 716.57) - /unlink 60000 (time 1773233625.30 total 85.64 last 629.74) total: 64768 /unlink in 92.80 seconds: 697.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3938 (time 1773233666.36 total 10.00 last 393.79) - open/close 8135 (time 1773233676.36 total 20.00 last 419.67) - open/close 10000 (time 1773233682.10 total 25.74 last 324.80) - open/close 13601 (time 1773233692.10 total 35.75 last 360.02) - open/close 17550 (time 1773233702.11 total 45.75 last 394.82) - open/close 20000 (time 1773233708.53 total 52.17 last 381.31) - open/close 24528 (time 1773233718.53 total 62.17 last 452.75) - open/close 29399 (time 1773233728.53 total 72.17 last 487.10) - open/close 34061 (time 1773233738.53 total 82.17 last 466.14) - open/close 38100 (time 1773233748.54 total 92.18 last 403.73) - open/close 40000 (time 1773233752.55 total 96.19 last 473.57) - open/close 45027 (time 1773233762.55 total 106.19 last 502.66) - open/close 50000 (time 1773233772.27 total 115.91 last 511.63) - open/close 55063 (time 1773233782.27 total 125.91 last 506.21) - open/close 59944 (time 1773233792.27 total 135.92 last 488.01) total: 64768 open/close in 145.88 seconds: 443.98 ops/second - /unlink 8207 (time 1773233813.56 total 10.00 last 820.69) - /unlink 16029 (time 1773233823.56 total 20.00 last 782.12) - /unlink 20000 (time 1773233828.31 total 24.76 last 835.00) - /unlink 28371 (time 1773233838.32 total 34.76 last 837.06) - /unlink 36504 (time 1773233848.32 total 44.76 last 813.28) - /unlink 40000 (time 1773233852.54 total 48.98 last 827.25) - /unlink 48383 (time 1773233862.54 total 58.98 last 838.28) - /unlink 50000 (time 1773233864.85 total 61.29 last 701.17) - /unlink 58993 (time 1773233874.85 total 71.29 last 899.19) total: 64768 /unlink in 79.48 seconds: 814.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4661 (time 1773233914.32 total 10.00 last 466.04) - open/close 9378 (time 1773233924.33 total 20.00 last 471.69) - open/close 14091 (time 1773233934.33 total 30.00 last 471.28) - open/close 18807 (time 1773233944.33 total 40.00 last 471.51) - open/close 20000 (time 1773233946.94 total 42.62 last 455.80) - open/close 24531 (time 1773233956.95 total 52.62 last 453.06) - open/close 29299 (time 1773233966.95 total 62.62 last 476.76) - open/close 34067 (time 1773233976.95 total 72.62 last 476.75) - open/close 38811 (time 1773233986.95 total 82.62 last 474.39) - open/close 40000 (time 1773233989.50 total 85.18 last 465.38) - open/close 44679 (time 1773233999.50 total 95.18 last 467.85) - open/close 49348 (time 1773234009.50 total 105.18 last 466.84) - open/close 53934 (time 1773234019.51 total 115.18 last 458.52) - open/close 58548 (time 1773234029.51 total 125.18 last 461.34) - open/close 60000 (time 1773234032.64 total 128.31 last 464.26) - open/close 64519 (time 1773234042.64 total 138.31 last 451.89) total: 64768 open/close in 138.82 seconds: 466.57 ops/second - /unlink 7749 (time 1773234054.56 total 10.00 last 774.86) - /unlink 10000 (time 1773234057.38 total 12.82 last 797.43) - /unlink 17674 (time 1773234067.38 total 22.82 last 767.37) - /unlink 20000 (time 1773234070.11 total 25.56 last 851.61) - /unlink 28744 (time 1773234080.12 total 35.56 last 874.35) - /unlink 37758 (time 1773234090.12 total 45.56 last 901.39) - /unlink 40000 (time 1773234092.73 total 48.17 last 856.13) - /unlink 48789 (time 1773234102.73 total 58.17 last 878.88) - /unlink 57462 (time 1773234112.74 total 68.18 last 867.21) - /unlink 60000 (time 1773234115.47 total 70.91 last 926.76) total: 64768 /unlink in 76.60 seconds: 845.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 5117 (time 1773234148.46 total 10.00 last 511.62) - open/close 10000 (time 1773234157.41 total 18.95 last 545.61) - open/close 15301 (time 1773234167.41 total 28.95 last 530.06) - open/close 20000 (time 1773234175.80 total 37.35 last 559.86) - open/close 25382 (time 1773234185.80 total 47.35 last 538.13) - open/close 30000 (time 1773234194.47 total 56.02 last 532.43) - open/close 35489 (time 1773234204.48 total 66.02 last 548.81) - open/close 40000 (time 1773234212.88 total 74.42 last 536.88) - open/close 45140 (time 1773234222.88 total 84.43 last 513.93) - open/close 49660 (time 1773234232.88 total 94.43 last 451.99) - open/close 54235 (time 1773234242.91 total 104.46 last 455.97) - open/close 58718 (time 1773234252.92 total 114.46 last 448.20) - open/close 60000 (time 1773234255.79 total 117.34 last 445.87) - open/close 64642 (time 1773234265.79 total 127.34 last 464.12) total: 64768 open/close in 127.73 seconds: 507.07 ops/second - /unlink 8090 (time 1773234277.60 total 10.00 last 808.95) - /unlink 10000 (time 1773234279.73 total 12.13 last 895.79) - /unlink 18866 (time 1773234289.73 total 22.13 last 886.54) - /unlink 26992 (time 1773234299.73 total 32.13 last 812.55) - /unlink 30000 (time 1773234303.87 total 36.28 last 726.27) - /unlink 37191 (time 1773234313.87 total 46.28 last 719.08) - /unlink 40000 (time 1773234317.60 total 50.00 last 753.86) - /unlink 47362 (time 1773234327.60 total 60.00 last 736.17) - /unlink 50000 (time 1773234331.26 total 63.67 last 719.86) - /unlink 57447 (time 1773234341.26 total 73.67 last 744.65) - /unlink 60000 (time 1773234344.85 total 77.26 last 711.06) total: 64768 /unlink in 83.47 seconds: 775.91 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg453-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg453-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg453-server stop mds service on oleg453-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg453-server unloading modules via unload_modules_local on: 'oleg453-server' oleg453-server: oleg453-server.virtnet: executing unload_modules_local oleg453-server: modules unloaded. pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-server: ssh exited with exit code 2 pdsh@oleg453-client: oleg453-client: ssh exited with exit code 2