== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:45:24 (1772757924) Stopping clients: oleg209-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg209-client.virtnet /mnt/lustre2 (opts:-f) oleg209-server: oleg209-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing load_modules_local oleg209-server: Loading modules from /home/green/git/lustre-release/lustre oleg209-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg209-server: detected 4 online CPUs by sysfs oleg209-server: MODOPTS_LIBCFS= oleg209-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg209-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg209-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg209-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg209-server: oleg209-server.virtnet: executing set_default_debug -1 all pdsh@oleg209-client: oleg209-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg209-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg209-client.virtnet: -o user_xattr,flock 192.168.202.109@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4423 (time 1772757967.27 total 10.00 last 442.23) - open/close 8420 (time 1772757977.28 total 20.00 last 399.62) - open/close 10000 (time 1772757981.41 total 24.14 last 382.31) - open/close 13596 (time 1772757991.41 total 34.14 last 359.53) - open/close 16978 (time 1772758001.41 total 44.14 last 338.12) - open/close 19953 (time 1772758011.41 total 54.14 last 297.49) - open/close 22462 (time 1772758021.42 total 64.14 last 250.82) - open/close 24531 (time 1772758031.42 total 74.15 last 206.83) - open/close 27581 (time 1772758041.42 total 84.15 last 304.95) - open/close 30000 (time 1772758048.78 total 91.51 last 328.78) - open/close 33753 (time 1772758058.78 total 101.51 last 375.28) - open/close 37020 (time 1772758068.78 total 111.51 last 326.66) - open/close 40000 (time 1772758078.49 total 121.21 last 307.04) - open/close 42323 (time 1772758088.49 total 131.21 last 232.29) - open/close 44823 (time 1772758098.49 total 141.22 last 249.97) - open/close 48090 (time 1772758108.49 total 151.22 last 326.68) - open/close 50000 (time 1772758114.09 total 156.81 last 341.19) - open/close 53355 (time 1772758124.09 total 166.82 last 335.47) - open/close 55842 (time 1772758134.09 total 176.82 last 248.55) - open/close 58250 (time 1772758144.09 total 186.82 last 240.79) - open/close 60000 (time 1772758150.22 total 192.95 last 285.62) - open/close 62637 (time 1772758160.22 total 202.95 last 263.68) total: 64768 open/close in 211.21 seconds: 306.64 ops/second - /unlink 6529 (time 1772758179.88 total 10.00 last 652.85) - /unlink 10000 (time 1772758186.40 total 16.52 last 532.26) - /unlink 16875 (time 1772758196.40 total 26.52 last 687.49) - /unlink 20000 (time 1772758200.93 total 31.05 last 689.77) - /unlink 26194 (time 1772758210.93 total 41.05 last 619.39) - /unlink 30000 (time 1772758217.65 total 47.77 last 566.67) - /unlink 35750 (time 1772758227.65 total 57.77 last 574.99) - /unlink 40000 (time 1772758235.79 total 65.91 last 521.97) - /unlink 45985 (time 1772758245.79 total 75.91 last 598.44) - /unlink 50000 (time 1772758252.36 total 82.48 last 611.81) - /unlink 55999 (time 1772758262.36 total 92.48 last 599.82) - /unlink 60000 (time 1772758269.00 total 99.12 last 602.27) total: 64768 /unlink in 107.94 seconds: 600.02 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3051 (time 1772758309.92 total 10.00 last 305.07) - open/close 6341 (time 1772758319.92 total 20.00 last 328.93) - open/close 9772 (time 1772758329.92 total 30.00 last 343.09) - open/close 12979 (time 1772758339.92 total 40.00 last 320.67) - open/close 16631 (time 1772758349.93 total 50.01 last 365.12) - open/close 19901 (time 1772758359.93 total 60.01 last 326.95) - open/close 22796 (time 1772758369.93 total 70.01 last 289.49) - open/close 26002 (time 1772758379.93 total 80.01 last 320.58) - open/close 29393 (time 1772758389.93 total 90.01 last 339.08) - open/close 32673 (time 1772758399.93 total 100.01 last 327.99) - open/close 35905 (time 1772758409.93 total 110.01 last 323.18) - open/close 38731 (time 1772758419.93 total 120.01 last 282.56) - open/close 40000 (time 1772758423.97 total 124.05 last 314.28) - open/close 43214 (time 1772758433.97 total 134.05 last 321.37) - open/close 46482 (time 1772758443.97 total 144.05 last 326.76) - open/close 49717 (time 1772758453.97 total 154.05 last 323.45) - open/close 52976 (time 1772758463.98 total 164.05 last 325.85) - open/close 56018 (time 1772758473.98 total 174.06 last 304.19) - open/close 59053 (time 1772758483.98 total 184.06 last 303.49) - open/close 60000 (time 1772758486.95 total 187.03 last 318.50) - open/close 63169 (time 1772758496.95 total 197.03 last 316.84) total: 64768 open/close in 202.58 seconds: 319.72 ops/second - /unlink 4945 (time 1772758514.01 total 10.00 last 494.45) - /unlink 10000 (time 1772758523.68 total 19.68 last 522.44) - /unlink 16024 (time 1772758533.68 total 29.68 last 602.29) - /unlink 20000 (time 1772758540.89 total 36.88 last 551.90) - /unlink 25063 (time 1772758550.89 total 46.88 last 506.28) - /unlink 29993 (time 1772758560.89 total 56.88 last 492.96) - /unlink 35345 (time 1772758570.89 total 66.88 last 535.18) - /unlink 40000 (time 1772758578.34 total 74.33 last 624.92) - /unlink 46255 (time 1772758588.34 total 84.33 last 625.48) - /unlink 50000 (time 1772758595.15 total 91.14 last 549.94) - /unlink 55152 (time 1772758605.15 total 101.14 last 515.17) - /unlink 60000 (time 1772758613.17 total 109.17 last 604.37) total: 64768 /unlink in 116.67 seconds: 555.12 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3386 (time 1772758650.25 total 10.00 last 338.52) - open/close 6913 (time 1772758660.25 total 20.00 last 352.66) - open/close 10000 (time 1772758667.92 total 27.67 last 402.48) - open/close 14704 (time 1772758677.92 total 37.67 last 470.40) - open/close 18626 (time 1772758687.92 total 47.68 last 392.11) - open/close 20000 (time 1772758691.58 total 51.33 last 375.67) - open/close 23591 (time 1772758701.58 total 61.34 last 358.99) - open/close 27327 (time 1772758711.58 total 71.34 last 373.57) - open/close 30000 (time 1772758717.66 total 77.41 last 439.96) - open/close 34250 (time 1772758727.66 total 87.41 last 424.96) - open/close 38573 (time 1772758737.66 total 97.41 last 432.29) - open/close 40000 (time 1772758741.38 total 101.14 last 383.20) - open/close 43407 (time 1772758751.38 total 111.14 last 340.65) - open/close 47517 (time 1772758761.38 total 121.14 last 410.99) - open/close 50000 (time 1772758767.42 total 127.18 last 411.11) - open/close 54502 (time 1772758777.42 total 137.18 last 450.17) - open/close 58949 (time 1772758787.43 total 147.18 last 444.54) - open/close 60000 (time 1772758790.01 total 149.77 last 406.96) - open/close 64122 (time 1772758800.01 total 159.77 last 412.15) total: 64768 open/close in 161.14 seconds: 401.93 ops/second - /unlink 8745 (time 1772758812.68 total 10.00 last 874.49) - /unlink 16923 (time 1772758822.68 total 20.00 last 817.74) - /unlink 20000 (time 1772758826.37 total 23.69 last 834.27) - /unlink 28645 (time 1772758836.37 total 33.69 last 864.49) - /unlink 37126 (time 1772758846.37 total 43.69 last 848.08) - /unlink 40000 (time 1772758849.66 total 46.98 last 874.73) - /unlink 47330 (time 1772758859.66 total 56.98 last 732.99) - /unlink 50000 (time 1772758863.32 total 60.64 last 729.00) - /unlink 57564 (time 1772758873.32 total 70.64 last 756.38) - /unlink 60000 (time 1772758876.71 total 74.02 last 719.42) total: 64768 /unlink in 80.39 seconds: 805.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4800 (time 1772758912.75 total 10.00 last 479.96) - open/close 9566 (time 1772758922.75 total 20.00 last 476.53) - open/close 14320 (time 1772758932.75 total 30.00 last 475.32) - open/close 19132 (time 1772758942.75 total 40.00 last 481.18) - open/close 23781 (time 1772758952.75 total 50.00 last 464.87) - open/close 28269 (time 1772758962.75 total 60.01 last 448.77) - open/close 30000 (time 1772758966.46 total 63.72 last 466.43) - open/close 34632 (time 1772758976.48 total 73.74 last 462.34) - open/close 39415 (time 1772758986.48 total 83.74 last 478.23) - open/close 43985 (time 1772758996.48 total 93.74 last 456.98) - open/close 48111 (time 1772759006.49 total 103.74 last 412.50) - open/close 50000 (time 1772759011.25 total 108.51 last 396.09) - open/close 53923 (time 1772759021.25 total 118.51 last 392.29) - open/close 58316 (time 1772759031.26 total 128.51 last 439.23) - open/close 60000 (time 1772759035.05 total 132.31 last 443.41) - open/close 64004 (time 1772759045.05 total 142.31 last 400.35) total: 64768 open/close in 144.14 seconds: 449.35 ops/second - /unlink 7749 (time 1772759058.33 total 10.00 last 774.88) - /unlink 10000 (time 1772759061.41 total 13.08 last 731.75) - /unlink 17591 (time 1772759071.41 total 23.08 last 759.01) - /unlink 20000 (time 1772759074.68 total 26.35 last 736.86) - /unlink 27196 (time 1772759084.68 total 36.35 last 719.54) - /unlink 30000 (time 1772759088.58 total 40.25 last 718.89) - /unlink 37068 (time 1772759098.58 total 50.25 last 706.78) - /unlink 40000 (time 1772759102.29 total 53.96 last 790.17) - /unlink 47735 (time 1772759112.29 total 63.96 last 773.44) - /unlink 50000 (time 1772759115.39 total 67.06 last 730.02) - /unlink 57416 (time 1772759125.40 total 77.06 last 741.54) - /unlink 60000 (time 1772759128.92 total 80.59 last 733.50) total: 64768 /unlink in 86.79 seconds: 746.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4803 (time 1772759166.99 total 10.00 last 480.27) - open/close 9596 (time 1772759176.99 total 20.00 last 479.26) - open/close 14396 (time 1772759186.99 total 30.00 last 479.99) - open/close 19125 (time 1772759196.99 total 40.00 last 472.81) - open/close 23847 (time 1772759206.99 total 50.01 last 472.11) - open/close 28588 (time 1772759216.99 total 60.01 last 474.10) - open/close 30000 (time 1772759219.94 total 62.95 last 478.79) - open/close 34702 (time 1772759229.94 total 72.95 last 470.19) - open/close 39317 (time 1772759239.94 total 82.96 last 461.47) - open/close 44175 (time 1772759249.94 total 92.96 last 485.77) - open/close 48833 (time 1772759259.94 total 102.96 last 465.76) - open/close 50000 (time 1772759262.41 total 105.42 last 473.25) - open/close 54691 (time 1772759272.41 total 115.42 last 469.03) - open/close 59317 (time 1772759282.41 total 125.43 last 462.54) - open/close 63903 (time 1772759292.41 total 135.43 last 458.60) total: 64768 open/close in 137.25 seconds: 471.90 ops/second - /unlink 6873 (time 1772759305.53 total 10.00 last 687.25) - /unlink 10000 (time 1772759310.47 total 14.95 last 632.22) - /unlink 16722 (time 1772759320.47 total 24.95 last 672.12) - /unlink 20000 (time 1772759326.04 total 30.52 last 588.39) - /unlink 26875 (time 1772759336.05 total 40.52 last 687.47) - /unlink 30000 (time 1772759340.79 total 45.27 last 657.92) - /unlink 37037 (time 1772759350.80 total 55.27 last 703.68) - /unlink 40000 (time 1772759355.10 total 59.57 last 688.51) - /unlink 47901 (time 1772759365.10 total 69.57 last 790.01) - /unlink 50000 (time 1772759367.67 total 72.15 last 815.26) - /unlink 57632 (time 1772759377.67 total 82.15 last 763.18) - /unlink 60000 (time 1772759380.83 total 85.31 last 749.86) total: 64768 /unlink in 91.52 seconds: 707.67 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg209-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg209-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg209-server stop mds service on oleg209-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg209-server unloading modules via unload_modules_local on: 'oleg209-server' oleg209-server: oleg209-server.virtnet: executing unload_modules_local oleg209-server: modules unloaded. pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-server: ssh exited with exit code 2 pdsh@oleg209-client: oleg209-client: ssh exited with exit code 2