== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 21:40:44 (1772678444) Stopping clients: oleg337-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg337-client.virtnet /mnt/lustre2 (opts:-f) oleg337-server: oleg337-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing load_modules_local oleg337-server: Loading modules from /home/green/git/lustre-release/lustre oleg337-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg337-server: detected 4 online CPUs by sysfs oleg337-server: MODOPTS_LIBCFS= oleg337-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg337-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg337-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg337-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg337-server: oleg337-server.virtnet: executing set_default_debug -1 all pdsh@oleg337-client: oleg337-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg337-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg337-client.virtnet: -o user_xattr,flock 192.168.203.137@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2683 (time 1772678492.95 total 10.01 last 268.07) - open/close 5056 (time 1772678502.96 total 20.01 last 237.23) - open/close 7269 (time 1772678512.96 total 30.01 last 221.23) - open/close 9378 (time 1772678522.97 total 40.02 last 210.75) - open/close 10000 (time 1772678526.39 total 43.44 last 181.85) - open/close 12685 (time 1772678536.39 total 53.44 last 268.50) - open/close 15247 (time 1772678546.39 total 63.45 last 256.10) - open/close 17008 (time 1772678556.40 total 73.45 last 176.03) - open/close 20000 (time 1772678565.97 total 83.03 last 312.36) - open/close 22408 (time 1772678575.98 total 93.03 last 240.76) - open/close 24617 (time 1772678585.98 total 103.03 last 220.86) - open/close 26494 (time 1772678595.98 total 113.04 last 187.63) - open/close 28213 (time 1772678605.98 total 123.04 last 171.86) - open/close 30000 (time 1772678615.81 total 132.87 last 181.79) - open/close 32061 (time 1772678625.81 total 142.87 last 206.10) - open/close 33713 (time 1772678635.82 total 152.87 last 165.16) - open/close 36100 (time 1772678645.82 total 162.87 last 238.68) - open/close 38575 (time 1772678655.82 total 172.88 last 247.42) - open/close 40000 (time 1772678661.10 total 178.16 last 269.82) - open/close 42438 (time 1772678671.18 total 188.23 last 241.98) - open/close 45387 (time 1772678681.18 total 198.23 last 294.83) - open/close 47870 (time 1772678691.18 total 208.24 last 248.25) - open/close 49654 (time 1772678701.18 total 218.24 last 178.40) - open/close 50000 (time 1772678703.71 total 220.77 last 136.80) - open/close 51904 (time 1772678713.71 total 230.77 last 190.39) - open/close 54064 (time 1772678723.71 total 240.77 last 215.94) - open/close 56723 (time 1772678733.72 total 250.77 last 265.87) - open/close 58595 (time 1772678743.72 total 260.77 last 187.16) - open/close 60000 (time 1772678751.18 total 268.23 last 188.30) - open/close 62239 (time 1772678761.18 total 278.24 last 223.82) - open/close 64370 (time 1772678771.19 total 288.24 last 213.03) total: 64768 open/close in 289.91 seconds: 223.41 ops/second - /unlink 4588 (time 1772678784.48 total 10.00 last 458.70) - /unlink 8243 (time 1772678794.48 total 20.00 last 365.48) - /unlink 10000 (time 1772678799.61 total 25.13 last 342.46) - /unlink 13357 (time 1772678809.61 total 35.13 last 335.65) - /unlink 17130 (time 1772678819.61 total 45.14 last 377.26) - /unlink 20000 (time 1772678825.55 total 51.08 last 483.09) - /unlink 24945 (time 1772678835.55 total 61.08 last 494.45) - /unlink 29752 (time 1772678845.55 total 71.08 last 480.65) - /unlink 33833 (time 1772678855.56 total 81.08 last 407.94) - /unlink 38745 (time 1772678865.56 total 91.08 last 491.17) - /unlink 40000 (time 1772678868.72 total 94.25 last 396.29) - /unlink 45053 (time 1772678878.73 total 104.25 last 505.26) - /unlink 50000 (time 1772678886.66 total 112.19 last 623.23) - /unlink 54910 (time 1772678896.67 total 122.19 last 490.90) - /unlink 60000 (time 1772678906.45 total 131.98 last 520.12) - /unlink 64411 (time 1772678916.46 total 141.98 last 440.93) total: 64768 /unlink in 142.87 seconds: 453.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2641 (time 1772678948.06 total 10.00 last 264.06) - open/close 5112 (time 1772678958.06 total 20.00 last 247.10) - open/close 7871 (time 1772678968.06 total 30.00 last 275.90) - open/close 10000 (time 1772678974.47 total 36.41 last 332.26) - open/close 12520 (time 1772678984.47 total 46.41 last 251.92) - open/close 15624 (time 1772678994.47 total 56.41 last 310.39) - open/close 18235 (time 1772679004.48 total 66.42 last 260.93) - open/close 20000 (time 1772679010.46 total 72.40 last 295.31) - open/close 23071 (time 1772679020.46 total 82.40 last 306.98) - open/close 25090 (time 1772679030.46 total 92.40 last 201.86) - open/close 27318 (time 1772679040.46 total 102.40 last 222.79) - open/close 29605 (time 1772679050.47 total 112.41 last 228.64) - open/close 32092 (time 1772679060.47 total 122.41 last 248.70) - open/close 34577 (time 1772679070.47 total 132.41 last 248.45) - open/close 37098 (time 1772679080.47 total 142.41 last 252.03) - open/close 40000 (time 1772679089.84 total 151.78 last 309.85) - open/close 42214 (time 1772679099.84 total 161.78 last 221.31) - open/close 44946 (time 1772679109.84 total 171.78 last 273.17) - open/close 47497 (time 1772679119.85 total 181.79 last 254.95) - open/close 49549 (time 1772679129.85 total 191.79 last 205.18) - open/close 50000 (time 1772679132.26 total 194.20 last 186.58) - open/close 52775 (time 1772679142.27 total 204.21 last 277.49) - open/close 55231 (time 1772679152.31 total 214.25 last 244.50) - open/close 57830 (time 1772679162.31 total 224.25 last 259.85) - open/close 60000 (time 1772679169.00 total 230.94 last 324.70) - open/close 62428 (time 1772679179.00 total 240.94 last 242.67) - open/close 64296 (time 1772679189.00 total 250.94 last 186.73) total: 64768 open/close in 252.92 seconds: 256.08 ops/second - /unlink 4629 (time 1772679202.58 total 10.00 last 462.87) - /unlink 9822 (time 1772679212.58 total 20.00 last 519.25) - /unlink 16211 (time 1772679222.58 total 30.00 last 638.78) - /unlink 20000 (time 1772679229.33 total 36.75 last 561.48) - /unlink 25379 (time 1772679239.33 total 46.75 last 537.88) - /unlink 30000 (time 1772679248.94 total 56.37 last 480.70) - /unlink 36135 (time 1772679258.94 total 66.37 last 613.49) - /unlink 40000 (time 1772679268.58 total 76.00 last 401.29) - /unlink 43877 (time 1772679278.58 total 86.00 last 387.60) - /unlink 49994 (time 1772679288.58 total 96.00 last 611.62) - /unlink 54613 (time 1772679298.58 total 106.00 last 461.85) - /unlink 58533 (time 1772679308.58 total 116.00 last 391.96) - /unlink 60000 (time 1772679310.98 total 118.40 last 611.88) - /unlink 64296 (time 1772679320.98 total 128.40 last 429.54) total: 64768 /unlink in 130.14 seconds: 497.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3144 (time 1772679354.37 total 10.00 last 314.33) - open/close 5945 (time 1772679364.37 total 20.00 last 280.09) - open/close 8612 (time 1772679374.38 total 30.01 last 266.56) - open/close 10000 (time 1772679378.87 total 34.50 last 309.11) - open/close 13484 (time 1772679388.87 total 44.50 last 348.34) - open/close 17069 (time 1772679398.87 total 54.50 last 358.47) - open/close 20000 (time 1772679407.18 total 62.81 last 352.87) - open/close 24067 (time 1772679417.18 total 72.81 last 406.62) - open/close 28420 (time 1772679427.18 total 82.81 last 435.24) - open/close 30000 (time 1772679430.74 total 86.37 last 443.28) - open/close 34191 (time 1772679440.75 total 96.37 last 419.09) - open/close 38898 (time 1772679450.75 total 106.37 last 470.69) - open/close 40000 (time 1772679453.28 total 108.91 last 434.90) - open/close 44058 (time 1772679463.28 total 118.91 last 405.77) - open/close 48326 (time 1772679473.28 total 128.91 last 426.76) - open/close 50000 (time 1772679477.52 total 133.15 last 394.48) - open/close 54049 (time 1772679487.53 total 143.15 last 404.88) - open/close 57469 (time 1772679497.53 total 153.16 last 341.67) - open/close 60000 (time 1772679504.56 total 160.19 last 360.49) - open/close 63178 (time 1772679514.56 total 170.19 last 317.79) total: 64768 open/close in 175.31 seconds: 369.46 ops/second - /unlink 6114 (time 1772679531.06 total 10.00 last 611.38) - /unlink 10000 (time 1772679538.35 total 17.29 last 533.36) - /unlink 15536 (time 1772679548.35 total 27.29 last 553.52) - /unlink 20000 (time 1772679555.71 total 34.65 last 606.52) - /unlink 25640 (time 1772679565.71 total 44.65 last 563.97) - /unlink 30000 (time 1772679572.95 total 51.89 last 602.14) - /unlink 35597 (time 1772679582.95 total 61.89 last 559.64) - /unlink 40000 (time 1772679588.71 total 67.64 last 765.09) - /unlink 46487 (time 1772679598.71 total 77.65 last 648.69) - /unlink 50000 (time 1772679603.89 total 82.83 last 677.80) - /unlink 56719 (time 1772679613.89 total 92.83 last 671.84) - /unlink 60000 (time 1772679619.27 total 98.21 last 609.89) total: 64768 /unlink in 107.58 seconds: 602.03 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3725 (time 1772679659.26 total 10.00 last 372.50) - open/close 8345 (time 1772679669.27 total 20.00 last 461.97) - open/close 10000 (time 1772679672.95 total 23.69 last 448.94) - open/close 14332 (time 1772679682.95 total 33.69 last 433.10) - open/close 18797 (time 1772679692.96 total 43.69 last 446.45) - open/close 20000 (time 1772679695.54 total 46.28 last 464.64) - open/close 24283 (time 1772679705.54 total 56.28 last 428.29) - open/close 29062 (time 1772679715.55 total 66.28 last 477.66) - open/close 30000 (time 1772679717.88 total 68.62 last 402.30) - open/close 34371 (time 1772679727.88 total 78.62 last 437.03) - open/close 38816 (time 1772679737.88 total 88.62 last 444.44) - open/close 40000 (time 1772679740.39 total 91.13 last 472.06) - open/close 45070 (time 1772679750.39 total 101.13 last 506.90) - open/close 49331 (time 1772679760.40 total 111.13 last 426.04) - open/close 53514 (time 1772679770.40 total 121.13 last 418.26) - open/close 57662 (time 1772679780.40 total 131.13 last 414.78) - open/close 60000 (time 1772679786.11 total 136.84 last 409.43) - open/close 63891 (time 1772679796.11 total 146.84 last 389.05) total: 64768 open/close in 149.08 seconds: 434.45 ops/second - /unlink 6037 (time 1772679809.82 total 10.00 last 603.65) - /unlink 10000 (time 1772679815.68 total 15.86 last 675.79) - /unlink 17353 (time 1772679825.69 total 25.87 last 735.25) - /unlink 20000 (time 1772679829.10 total 29.28 last 774.21) - /unlink 26442 (time 1772679839.10 total 39.28 last 644.19) - /unlink 30000 (time 1772679844.46 total 44.64 last 664.40) - /unlink 36744 (time 1772679854.46 total 54.64 last 674.36) - /unlink 40000 (time 1772679860.68 total 60.86 last 523.44) - /unlink 46624 (time 1772679870.68 total 70.86 last 662.39) - /unlink 50000 (time 1772679875.42 total 75.60 last 712.69) - /unlink 56131 (time 1772679885.42 total 85.60 last 612.99) - /unlink 60000 (time 1772679892.03 total 92.21 last 585.27) total: 64768 /unlink in 99.45 seconds: 651.26 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3806 (time 1772679932.98 total 10.00 last 380.59) - open/close 7259 (time 1772679942.98 total 20.00 last 345.23) - open/close 10000 (time 1772679950.61 total 27.63 last 359.33) - open/close 14166 (time 1772679960.61 total 37.63 last 416.51) - open/close 18115 (time 1772679970.61 total 47.63 last 394.84) - open/close 20000 (time 1772679975.25 total 52.27 last 406.40) - open/close 24029 (time 1772679985.25 total 62.27 last 402.87) - open/close 28222 (time 1772679995.25 total 72.27 last 419.29) - open/close 30000 (time 1772680001.27 total 78.29 last 295.38) - open/close 33384 (time 1772680011.27 total 88.29 last 338.36) - open/close 37307 (time 1772680021.28 total 98.30 last 392.23) - open/close 40000 (time 1772680027.89 total 104.91 last 407.00) - open/close 43802 (time 1772680037.89 total 114.91 last 380.19) - open/close 48304 (time 1772680047.89 total 124.91 last 450.15) - open/close 50000 (time 1772680051.77 total 128.79 last 438.00) - open/close 54283 (time 1772680061.77 total 138.79 last 428.21) - open/close 58572 (time 1772680071.77 total 148.79 last 428.88) - open/close 60000 (time 1772680075.02 total 152.04 last 439.73) - open/close 62756 (time 1772680085.02 total 162.04 last 275.58) - open/close 64166 (time 1772680095.02 total 172.04 last 140.98) total: 64768 open/close in 176.52 seconds: 366.92 ops/second - /unlink 2325 (time 1772680111.86 total 10.00 last 232.47) - /unlink 6339 (time 1772680121.86 total 20.00 last 401.26) - /unlink 10000 (time 1772680131.47 total 29.62 last 380.82) - /unlink 14228 (time 1772680141.48 total 39.62 last 422.71) - /unlink 19266 (time 1772680151.48 total 49.62 last 503.79) - /unlink 20000 (time 1772680153.66 total 51.80 last 336.82) - /unlink 23201 (time 1772680163.66 total 61.80 last 320.03) - /unlink 26857 (time 1772680173.66 total 71.80 last 365.59) - /unlink 30000 (time 1772680179.43 total 77.57 last 544.67) - /unlink 35650 (time 1772680189.43 total 87.57 last 564.93) - /unlink 40000 (time 1772680197.66 total 95.80 last 528.69) - /unlink 43950 (time 1772680207.66 total 105.80 last 394.95) - /unlink 47407 (time 1772680217.66 total 115.81 last 345.60) - /unlink 50000 (time 1772680225.02 total 123.16 last 352.61) - /unlink 53706 (time 1772680235.02 total 133.16 last 370.52) - /unlink 58745 (time 1772680245.02 total 143.16 last 503.90) - /unlink 60000 (time 1772680247.06 total 145.20 last 614.50) total: 64768 /unlink in 153.07 seconds: 423.13 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg337-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg337-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg337-server stop mds service on oleg337-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg337-server unloading modules via unload_modules_local on: 'oleg337-server' oleg337-server: oleg337-server.virtnet: executing unload_modules_local oleg337-server: modules unloaded. pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-client: ssh exited with exit code 2 pdsh@oleg337-client: oleg337-server: ssh exited with exit code 2