== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:47:46 (1760125666) Stopping clients: oleg432-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg432-client.virtnet /mnt/lustre2 (opts:-f) oleg432-server: oleg432-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing load_modules_local oleg432-server: Loading modules from /home/green/git/lustre-release/lustre oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg432-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg432-server: detected 4 online CPUs by sysfs oleg432-server: MODOPTS_LIBCFS= oleg432-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg432-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg432-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg432-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg432-server: oleg432-server.virtnet: executing set_default_debug -1 all pdsh@oleg432-client: oleg432-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg432-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg432-client.virtnet: -o user_xattr,flock 192.168.204.132@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1015 (time 1760125730.90 total 10.00 last 101.48) - open/close 1976 (time 1760125740.90 total 20.00 last 96.07) - open/close 3044 (time 1760125750.91 total 30.02 last 106.69) - open/close 4200 (time 1760125760.91 total 40.02 last 115.52) - open/close 5079 (time 1760125770.92 total 50.03 last 87.84) - open/close 5995 (time 1760125780.92 total 60.03 last 91.57) - open/close 6969 (time 1760125790.93 total 70.04 last 97.35) - open/close 7896 (time 1760125800.93 total 80.04 last 92.68) - open/close 8806 (time 1760125810.94 total 90.04 last 90.98) - open/close 9760 (time 1760125820.94 total 100.05 last 95.35) - open/close 10000 (time 1760125823.19 total 102.30 last 106.51) - open/close 10854 (time 1760125833.20 total 112.31 last 85.35) - open/close 11711 (time 1760125843.21 total 122.31 last 85.65) - open/close 12661 (time 1760125853.21 total 132.31 last 95.00) - open/close 13677 (time 1760125863.21 total 142.32 last 101.57) - open/close 14796 (time 1760125873.22 total 152.32 last 111.82) - open/close 15867 (time 1760125883.22 total 162.32 last 107.09) - open/close 16935 (time 1760125893.22 total 172.33 last 106.74) - open/close 18307 (time 1760125903.22 total 182.33 last 137.20) - open/close 19889 (time 1760125913.23 total 192.34 last 158.12) - open/close 21872 (time 1760125923.23 total 202.34 last 198.26) - open/close 23509 (time 1760125933.25 total 212.36 last 163.33) - open/close 25011 (time 1760125943.26 total 222.36 last 150.17) - open/close 26536 (time 1760125953.26 total 232.37 last 152.41) - open/close 28486 (time 1760125963.27 total 242.37 last 194.91) - open/close 30000 (time 1760125972.63 total 251.74 last 161.70) - open/close 31855 (time 1760125982.63 total 261.74 last 185.43) - open/close 34197 (time 1760125992.63 total 271.74 last 234.17) - open/close 36458 (time 1760126002.64 total 281.74 last 226.05) - open/close 38589 (time 1760126012.64 total 291.75 last 213.02) - open/close 40000 (time 1760126018.25 total 297.36 last 251.31) - open/close 42203 (time 1760126028.26 total 307.36 last 220.24) - open/close 44275 (time 1760126038.26 total 317.37 last 207.12) - open/close 46379 (time 1760126048.26 total 327.37 last 210.34) - open/close 48623 (time 1760126058.27 total 337.38 last 224.26) - open/close 50000 (time 1760126064.54 total 343.65 last 219.47) - open/close 51829 (time 1760126074.55 total 353.65 last 182.84) - open/close 53389 (time 1760126084.55 total 363.66 last 155.97) - open/close 54734 (time 1760126094.55 total 373.66 last 134.49) - open/close 56121 (time 1760126104.56 total 383.67 last 138.58) - open/close 57863 (time 1760126114.56 total 393.67 last 174.19) - open/close 59367 (time 1760126124.56 total 403.67 last 150.38) - open/close 60000 (time 1760126128.76 total 407.86 last 150.81) - open/close 61728 (time 1760126138.76 total 417.87 last 172.79) - open/close 63432 (time 1760126148.76 total 427.87 last 170.40) total: 64768 open/close in 436.31 seconds: 148.44 ops/second - /unlink 4036 (time 1760126169.00 total 10.00 last 403.58) - /unlink 7498 (time 1760126179.02 total 20.02 last 345.41) - /unlink 10000 (time 1760126185.84 total 26.85 last 366.77) - /unlink 12763 (time 1760126195.84 total 36.85 last 276.30) - /unlink 15554 (time 1760126205.85 total 46.85 last 278.99) - /unlink 18720 (time 1760126215.85 total 56.85 last 316.54) - /unlink 20000 (time 1760126219.88 total 60.89 last 317.27) - /unlink 23305 (time 1760126229.88 total 70.89 last 330.40) - /unlink 26562 (time 1760126239.89 total 80.89 last 325.67) - /unlink 30000 (time 1760126249.05 total 90.05 last 375.30) - /unlink 33046 (time 1760126259.05 total 100.05 last 304.55) - /unlink 36245 (time 1760126269.05 total 110.05 last 319.86) - /unlink 40000 (time 1760126277.97 total 118.97 last 421.12) - /unlink 44283 (time 1760126287.97 total 128.97 last 428.25) - /unlink 48071 (time 1760126297.97 total 138.97 last 378.74) - /unlink 50000 (time 1760126303.24 total 144.25 last 365.77) - /unlink 53867 (time 1760126313.24 total 154.25 last 386.70) - /unlink 57640 (time 1760126323.24 total 164.25 last 377.24) - /unlink 60000 (time 1760126329.40 total 170.41 last 383.29) - /unlink 63815 (time 1760126339.40 total 180.41 last 381.43) total: 64768 /unlink in 182.69 seconds: 354.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2588 (time 1760126371.07 total 10.00 last 258.71) - open/close 5033 (time 1760126381.08 total 20.00 last 244.47) - open/close 7531 (time 1760126391.08 total 30.00 last 249.79) - open/close 9982 (time 1760126401.08 total 40.01 last 245.00) - open/close 12000 (time 1760126411.08 total 50.01 last 201.76) - open/close 12888 (time 1760126421.09 total 60.02 last 88.73) - open/close 13381 (time 1760126431.09 total 70.02 last 49.30) - open/close 13782 (time 1760126441.11 total 80.04 last 40.02) - open/close 14116 (time 1760126451.11 total 90.04 last 33.38) - open/close 14687 (time 1760126461.13 total 100.06 last 57.03) - open/close 15230 (time 1760126471.14 total 110.07 last 54.24) - open/close 15692 (time 1760126481.15 total 120.08 last 46.15) - open/close 16433 (time 1760126491.17 total 130.10 last 73.93) - open/close 17191 (time 1760126501.18 total 140.11 last 75.73) - open/close 17890 (time 1760126511.18 total 150.11 last 69.88) - open/close 18699 (time 1760126521.19 total 160.11 last 80.89) - open/close 19530 (time 1760126531.19 total 170.12 last 83.10) - open/close 20000 (time 1760126536.45 total 175.38 last 89.31) - open/close 20857 (time 1760126546.45 total 185.38 last 85.66) - open/close 21653 (time 1760126556.46 total 195.39 last 79.56) - open/close 22431 (time 1760126566.46 total 205.39 last 77.78) - open/close 23245 (time 1760126576.47 total 215.39 last 81.36) - open/close 24043 (time 1760126586.47 total 225.40 last 79.73) - open/close 24851 (time 1760126596.48 total 235.40 last 80.80) - open/close 25681 (time 1760126606.49 total 245.42 last 82.89) - open/close 26476 (time 1760126616.49 total 255.42 last 79.49) - open/close 27305 (time 1760126626.49 total 265.42 last 82.86) - open/close 28240 (time 1760126636.50 total 275.43 last 93.41) - open/close 29081 (time 1760126646.51 total 285.44 last 84.08) - open/close 29835 (time 1760126656.51 total 295.44 last 75.38) - open/close 30000 (time 1760126658.62 total 297.55 last 78.07) - open/close 30767 (time 1760126668.63 total 307.55 last 76.68) - open/close 31530 (time 1760126678.64 total 317.57 last 76.18) - open/close 32305 (time 1760126688.64 total 327.57 last 77.48) - open/close 33046 (time 1760126698.66 total 337.58 last 74.01) - open/close 33679 (time 1760126708.68 total 347.61 last 63.17) - open/close 34222 (time 1760126718.70 total 357.63 last 54.15) - open/close 34897 (time 1760126728.71 total 367.64 last 67.46) - open/close 35597 (time 1760126738.73 total 377.66 last 69.87) - open/close 36232 (time 1760126748.74 total 387.67 last 63.44) - open/close 36932 (time 1760126758.74 total 397.67 last 70.00) - open/close 37597 (time 1760126768.74 total 407.67 last 66.47) - open/close 38325 (time 1760126778.75 total 417.68 last 72.72) - open/close 39021 (time 1760126788.77 total 427.69 last 69.52) - open/close 39719 (time 1760126798.79 total 437.72 last 69.62) - open/close 40000 (time 1760126803.18 total 442.11 last 64.04) - open/close 40651 (time 1760126813.18 total 452.11 last 65.07) - open/close 41307 (time 1760126823.20 total 462.13 last 65.50) - open/close 41918 (time 1760126833.21 total 472.14 last 61.05) - open/close 42583 (time 1760126843.21 total 482.14 last 66.49) - open/close 43258 (time 1760126853.21 total 492.14 last 67.48) - open/close 43883 (time 1760126863.22 total 502.15 last 62.43) - open/close 44545 (time 1760126873.24 total 512.17 last 66.06) - open/close 45185 (time 1760126883.25 total 522.18 last 63.98) - open/close 45844 (time 1760126893.25 total 532.18 last 65.89) - open/close 46358 (time 1760126903.26 total 542.19 last 51.35) - open/close 47034 (time 1760126913.28 total 552.21 last 67.45) - open/close 47712 (time 1760126923.29 total 562.22 last 67.78) - open/close 48463 (time 1760126933.29 total 572.22 last 75.09) - open/close 49237 (time 1760126943.29 total 582.22 last 77.37) - open/close 50000 (time 1760126952.84 total 591.77 last 79.92) - open/close 50754 (time 1760126962.85 total 601.78 last 75.33) - open/close 51565 (time 1760126972.85 total 611.78 last 81.07) - open/close 52243 (time 1760126982.86 total 621.78 last 67.77) - open/close 52920 (time 1760126992.86 total 631.79 last 67.69) - open/close 53555 (time 1760127002.87 total 641.80 last 63.39) - open/close 54202 (time 1760127012.88 total 651.81 last 64.69) - open/close 54914 (time 1760127022.88 total 661.81 last 71.18) - open/close 55534 (time 1760127032.89 total 671.81 last 61.96) - open/close 56271 (time 1760127042.91 total 681.84 last 73.55) - open/close 56919 (time 1760127052.92 total 691.85 last 64.70) - open/close 57544 (time 1760127062.94 total 701.87 last 62.41) - open/close 58217 (time 1760127072.97 total 711.90 last 67.06) - open/close 58910 (time 1760127082.98 total 721.91 last 69.28) - open/close 59611 (time 1760127092.98 total 731.91 last 70.10) - open/close 60000 (time 1760127098.36 total 737.29 last 72.25) - open/close 60691 (time 1760127108.37 total 747.30 last 69.05) - open/close 61395 (time 1760127118.37 total 757.30 last 70.37) - open/close 62069 (time 1760127128.37 total 767.30 last 67.39) - open/close 62721 (time 1760127138.40 total 777.32 last 65.06) - open/close 63440 (time 1760127148.41 total 787.33 last 71.82) - open/close 64149 (time 1760127158.42 total 797.35 last 70.79) total: 64768 open/close in 806.40 seconds: 80.32 ops/second - /unlink 675 (time 1760127181.56 total 10.00 last 67.48) - /unlink 1317 (time 1760127191.56 total 20.01 last 64.16) - /unlink 2054 (time 1760127201.57 total 30.02 last 73.63) - /unlink 2789 (time 1760127211.58 total 40.02 last 73.46) - /unlink 3584 (time 1760127221.59 total 50.03 last 79.43) - /unlink 4427 (time 1760127231.59 total 60.04 last 84.23) - /unlink 5260 (time 1760127241.60 total 70.04 last 83.30) - /unlink 6031 (time 1760127251.60 total 80.05 last 77.06) - /unlink 6783 (time 1760127261.60 total 90.05 last 75.20) - /unlink 7561 (time 1760127271.63 total 100.08 last 77.56) - /unlink 8374 (time 1760127281.63 total 110.08 last 81.28) - /unlink 9172 (time 1760127291.64 total 120.09 last 79.76) - /unlink 9965 (time 1760127301.64 total 130.09 last 79.28) - /unlink 10716 (time 1760127311.64 total 140.09 last 75.10) - /unlink 11430 (time 1760127321.64 total 150.09 last 71.38) - /unlink 12123 (time 1760127331.66 total 160.11 last 69.20) - /unlink 12785 (time 1760127341.66 total 170.11 last 66.16) - /unlink 13497 (time 1760127351.67 total 180.12 last 71.15) - /unlink 14268 (time 1760127361.68 total 190.12 last 77.06) - /unlink 15050 (time 1760127371.69 total 200.13 last 78.10) - /unlink 15819 (time 1760127381.70 total 210.14 last 76.84) - /unlink 16581 (time 1760127391.70 total 220.15 last 76.15) - /unlink 17332 (time 1760127401.71 total 230.16 last 75.02) - /unlink 18070 (time 1760127411.72 total 240.17 last 73.77) - /unlink 18868 (time 1760127421.74 total 250.19 last 79.61) - /unlink 19664 (time 1760127431.75 total 260.20 last 79.53) - /unlink 20000 (time 1760127436.18 total 264.63 last 75.89) - /unlink 20811 (time 1760127446.18 total 274.63 last 81.09) - /unlink 21619 (time 1760127456.19 total 284.64 last 80.70) - /unlink 22544 (time 1760127466.20 total 294.65 last 92.44) - /unlink 23351 (time 1760127476.22 total 304.66 last 80.56) - /unlink 24109 (time 1760127486.22 total 314.67 last 75.74) - /unlink 24900 (time 1760127496.23 total 324.67 last 79.08) - /unlink 25677 (time 1760127506.23 total 334.67 last 77.69) - /unlink 26465 (time 1760127516.24 total 344.68 last 78.74) - /unlink 27257 (time 1760127526.24 total 354.69 last 79.16) - /unlink 28072 (time 1760127536.25 total 364.69 last 81.46) - /unlink 28856 (time 1760127546.26 total 374.70 last 78.32) - /unlink 29653 (time 1760127556.26 total 384.71 last 79.66) - /unlink 30000 (time 1760127561.08 total 389.53 last 71.95) - /unlink 30717 (time 1760127571.09 total 399.54 last 71.66) - /unlink 31469 (time 1760127581.10 total 409.55 last 75.09) - /unlink 32219 (time 1760127591.11 total 419.56 last 74.92) - /unlink 32942 (time 1760127601.12 total 429.57 last 72.27) - /unlink 33655 (time 1760127611.13 total 439.58 last 71.21) - /unlink 34424 (time 1760127621.13 total 449.58 last 76.89) - /unlink 35128 (time 1760127631.14 total 459.59 last 70.36) - /unlink 35865 (time 1760127641.15 total 469.59 last 73.64) - /unlink 36575 (time 1760127651.17 total 479.62 last 70.83) - /unlink 37374 (time 1760127661.18 total 489.63 last 79.79) - /unlink 38120 (time 1760127671.19 total 499.64 last 74.56) - /unlink 38886 (time 1760127681.22 total 509.67 last 76.32) - /unlink 39635 (time 1760127691.23 total 519.68 last 74.84) - /unlink 40000 (time 1760127695.87 total 524.32 last 78.62) - /unlink 40742 (time 1760127705.90 total 534.34 last 74.03) - /unlink 41527 (time 1760127715.91 total 544.36 last 78.41) - /unlink 42286 (time 1760127725.91 total 554.36 last 75.89) - /unlink 43039 (time 1760127735.92 total 564.37 last 75.23) - /unlink 43820 (time 1760127745.93 total 574.38 last 78.02) - /unlink 44579 (time 1760127755.94 total 584.39 last 75.79) - /unlink 45331 (time 1760127765.95 total 594.39 last 75.18) - /unlink 46123 (time 1760127775.95 total 604.40 last 79.14) - /unlink 46927 (time 1760127785.96 total 614.40 last 80.38) - /unlink 47708 (time 1760127795.96 total 624.41 last 78.06) - /unlink 48525 (time 1760127805.97 total 634.41 last 81.65) - /unlink 49284 (time 1760127815.98 total 644.42 last 75.83) - /unlink 50000 (time 1760127825.02 total 653.47 last 79.17) - /unlink 50827 (time 1760127835.05 total 663.49 last 82.49) - /unlink 51651 (time 1760127845.06 total 673.50 last 82.31) - /unlink 52581 (time 1760127855.06 total 683.51 last 92.98) - /unlink 53499 (time 1760127865.06 total 693.51 last 91.80) - /unlink 54311 (time 1760127875.07 total 703.51 last 81.15) - /unlink 55193 (time 1760127885.07 total 713.52 last 88.15) - /unlink 55994 (time 1760127895.08 total 723.52 last 80.05) - /unlink 56845 (time 1760127905.08 total 733.53 last 85.10) - /unlink 57823 (time 1760127915.08 total 743.53 last 97.74) - /unlink 58733 (time 1760127925.09 total 753.54 last 90.94) - /unlink 59624 (time 1760127935.09 total 763.54 last 89.07) - /unlink 60000 (time 1760127939.10 total 767.54 last 93.96) - /unlink 61020 (time 1760127949.10 total 777.55 last 101.91) - /unlink 61895 (time 1760127959.11 total 787.56 last 87.43) - /unlink 62838 (time 1760127969.12 total 797.57 last 94.22) - /unlink 63635 (time 1760127979.13 total 807.57 last 79.66) - /unlink 64505 (time 1760127989.13 total 817.58 last 86.94) total: 64768 /unlink in 820.79 seconds: 78.91 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 803 (time 1760128021.60 total 10.01 last 80.22) - open/close 1382 (time 1760128031.62 total 20.03 last 57.78) - open/close 2018 (time 1760128041.63 total 30.04 last 63.56) - open/close 2809 (time 1760128051.63 total 40.04 last 79.09) - open/close 3635 (time 1760128061.64 total 50.05 last 82.48) - open/close 4481 (time 1760128071.64 total 60.05 last 84.59) - open/close 5283 (time 1760128081.66 total 70.06 last 80.12) - open/close 6079 (time 1760128091.66 total 80.07 last 79.54) - open/close 6825 (time 1760128101.67 total 90.08 last 74.52) - open/close 7606 (time 1760128111.68 total 100.09 last 78.05) - open/close 8401 (time 1760128121.69 total 110.10 last 79.44) - open/close 9140 (time 1760128131.69 total 120.10 last 73.88) - open/close 9942 (time 1760128141.69 total 130.10 last 80.17) - open/close 10754 (time 1760128151.70 total 140.11 last 81.16) - open/close 11525 (time 1760128161.70 total 150.11 last 77.07) - open/close 12312 (time 1760128171.71 total 160.12 last 78.65) - open/close 13088 (time 1760128181.71 total 170.12 last 77.59) - open/close 13847 (time 1760128191.72 total 180.13 last 75.82) - open/close 14677 (time 1760128201.73 total 190.14 last 82.93) - open/close 15503 (time 1760128211.74 total 200.15 last 82.49) - open/close 16271 (time 1760128221.75 total 210.16 last 76.71) - open/close 17122 (time 1760128231.77 total 220.17 last 85.01) - open/close 17967 (time 1760128241.77 total 230.18 last 84.45) - open/close 18797 (time 1760128251.77 total 240.18 last 82.98) - open/close 19634 (time 1760128261.78 total 250.19 last 83.68) - open/close 20000 (time 1760128266.07 total 254.48 last 85.23) - open/close 20800 (time 1760128276.08 total 264.48 last 79.96) - open/close 21625 (time 1760128286.08 total 274.49 last 82.45) - open/close 22415 (time 1760128296.10 total 284.50 last 78.90) - open/close 23162 (time 1760128306.10 total 294.50 last 74.69) - open/close 23903 (time 1760128316.12 total 304.53 last 73.94) - open/close 24628 (time 1760128326.12 total 314.53 last 72.49) - open/close 25448 (time 1760128336.12 total 324.53 last 81.97) - open/close 26253 (time 1760128346.12 total 334.53 last 80.50) - open/close 27104 (time 1760128356.13 total 344.54 last 85.05) - open/close 27888 (time 1760128366.13 total 354.54 last 78.39) - open/close 28717 (time 1760128376.14 total 364.54 last 82.86) - open/close 29525 (time 1760128386.14 total 374.55 last 80.74) - open/close 30000 (time 1760128391.81 total 380.21 last 83.87) - open/close 30883 (time 1760128401.81 total 390.22 last 88.29) - open/close 31810 (time 1760128411.82 total 400.23 last 92.57) - open/close 32969 (time 1760128421.83 total 410.23 last 115.87) - open/close 33898 (time 1760128431.83 total 420.24 last 92.87) - open/close 34823 (time 1760128441.83 total 430.24 last 92.47) - open/close 35734 (time 1760128451.84 total 440.25 last 91.03) - open/close 36851 (time 1760128461.85 total 450.26 last 111.59) - open/close 37744 (time 1760128471.86 total 460.27 last 89.19) - open/close 38687 (time 1760128481.86 total 470.27 last 94.30) - open/close 39626 (time 1760128491.87 total 480.28 last 93.84) - open/close 40000 (time 1760128495.63 total 484.04 last 99.42) - open/close 40943 (time 1760128505.63 total 494.04 last 94.27) - open/close 41950 (time 1760128515.64 total 504.04 last 100.67) - open/close 42915 (time 1760128525.64 total 514.05 last 96.45) - open/close 43707 (time 1760128535.64 total 524.05 last 79.19) - open/close 44581 (time 1760128545.64 total 534.05 last 87.39) - open/close 45467 (time 1760128555.65 total 544.06 last 88.53) - open/close 46324 (time 1760128565.66 total 554.07 last 85.59) - open/close 47201 (time 1760128575.68 total 564.08 last 87.60) - open/close 48183 (time 1760128585.68 total 574.09 last 98.18) - open/close 49132 (time 1760128595.68 total 584.09 last 94.86) - open/close 50000 (time 1760128602.34 total 590.75 last 130.34) - open/close 51120 (time 1760128612.35 total 600.76 last 111.92) - open/close 52235 (time 1760128622.36 total 610.76 last 111.41) - open/close 53320 (time 1760128632.36 total 620.77 last 108.45) - open/close 54430 (time 1760128642.37 total 630.78 last 110.91) - open/close 55445 (time 1760128652.37 total 640.78 last 101.48) - open/close 56529 (time 1760128662.38 total 650.79 last 108.31) - open/close 57487 (time 1760128672.38 total 660.79 last 95.79) - open/close 58415 (time 1760128682.38 total 670.79 last 92.79) - open/close 59486 (time 1760128692.42 total 680.83 last 106.69) - open/close 60000 (time 1760128695.74 total 684.15 last 154.81) - open/close 61053 (time 1760128705.75 total 694.16 last 105.20) - open/close 61961 (time 1760128715.76 total 704.16 last 90.75) - open/close 62938 (time 1760128725.76 total 714.17 last 97.62) - open/close 63905 (time 1760128735.77 total 724.18 last 96.66) total: 64768 open/close in 733.22 seconds: 88.33 ops/second - /unlink 1015 (time 1760128757.47 total 10.00 last 101.47) - /unlink 1975 (time 1760128767.48 total 20.01 last 95.94) - /unlink 2983 (time 1760128777.48 total 30.01 last 100.76) - /unlink 4000 (time 1760128787.50 total 40.03 last 101.56) - /unlink 4932 (time 1760128797.50 total 50.03 last 93.14) - /unlink 6159 (time 1760128807.51 total 60.04 last 122.63) - /unlink 7381 (time 1760128817.51 total 70.04 last 122.14) - /unlink 8380 (time 1760128827.52 total 80.05 last 99.84) - /unlink 9577 (time 1760128837.53 total 90.06 last 119.55) - /unlink 10000 (time 1760128841.00 total 93.53 last 122.04) - /unlink 11222 (time 1760128851.01 total 103.53 last 122.13) - /unlink 12235 (time 1760128861.01 total 113.54 last 101.27) - /unlink 13418 (time 1760128871.01 total 123.54 last 118.26) - /unlink 14579 (time 1760128881.01 total 133.54 last 116.07) - /unlink 15759 (time 1760128891.02 total 143.55 last 117.93) - /unlink 17135 (time 1760128901.03 total 153.55 last 137.54) - /unlink 18694 (time 1760128911.03 total 163.55 last 155.89) - /unlink 19999 (time 1760128921.03 total 173.56 last 130.43) - /unlink 21288 (time 1760128931.04 total 183.56 last 128.84) - /unlink 22230 (time 1760128941.05 total 193.57 last 94.11) - /unlink 23076 (time 1760128951.05 total 203.58 last 84.55) - /unlink 23949 (time 1760128961.05 total 213.58 last 87.29) - /unlink 24735 (time 1760128971.06 total 223.59 last 78.51) - /unlink 25775 (time 1760128981.07 total 233.60 last 103.94) - /unlink 26654 (time 1760128991.07 total 243.60 last 87.89) - /unlink 27678 (time 1760129001.08 total 253.61 last 102.28) - /unlink 28480 (time 1760129011.09 total 263.62 last 80.15) - /unlink 29613 (time 1760129021.09 total 273.62 last 113.24) - /unlink 30000 (time 1760129024.98 total 277.51 last 99.61) - /unlink 30790 (time 1760129034.99 total 287.52 last 78.94) - /unlink 31601 (time 1760129044.99 total 297.52 last 81.08) - /unlink 32429 (time 1760129055.00 total 307.52 last 82.76) - /unlink 33324 (time 1760129065.00 total 317.53 last 89.45) - /unlink 34294 (time 1760129075.00 total 327.53 last 96.96) - /unlink 35244 (time 1760129085.01 total 337.54 last 94.94) - /unlink 36295 (time 1760129095.01 total 347.54 last 105.10) - /unlink 37462 (time 1760129105.02 total 357.55 last 116.62) - /unlink 38446 (time 1760129115.03 total 367.55 last 98.33) - /unlink 39485 (time 1760129125.03 total 377.56 last 103.85) - /unlink 40000 (time 1760129130.80 total 383.32 last 89.30) - /unlink 40959 (time 1760129140.80 total 393.32 last 95.90) - /unlink 42072 (time 1760129150.80 total 403.33 last 111.29) - /unlink 43370 (time 1760129160.80 total 413.33 last 129.71) - /unlink 44933 (time 1760129170.81 total 423.34 last 156.25) - /unlink 46455 (time 1760129180.81 total 433.34 last 152.14) - /unlink 47700 (time 1760129190.82 total 443.35 last 124.37) - /unlink 48839 (time 1760129200.83 total 453.35 last 113.85) - /unlink 49897 (time 1760129210.83 total 463.36 last 105.78) - /unlink 50916 (time 1760129220.83 total 473.36 last 101.87) - /unlink 51980 (time 1760129230.83 total 483.36 last 106.40) - /unlink 53196 (time 1760129240.84 total 493.37 last 121.49) - /unlink 54114 (time 1760129250.85 total 503.38 last 91.69) - /unlink 55008 (time 1760129260.87 total 513.39 last 89.28) - /unlink 55970 (time 1760129270.87 total 523.40 last 96.12) - /unlink 56859 (time 1760129280.88 total 533.41 last 88.86) - /unlink 57723 (time 1760129290.88 total 543.41 last 86.38) - /unlink 58520 (time 1760129300.88 total 553.41 last 79.70) - /unlink 59462 (time 1760129310.90 total 563.43 last 93.99) - /unlink 60000 (time 1760129317.33 total 569.86 last 83.71) - /unlink 60806 (time 1760129327.34 total 579.86 last 80.54) - /unlink 61702 (time 1760129337.34 total 589.87 last 89.58) - /unlink 62479 (time 1760129347.34 total 599.87 last 77.69) - /unlink 63285 (time 1760129357.34 total 609.87 last 80.57) - /unlink 64113 (time 1760129367.35 total 619.87 last 82.78) total: 64768 /unlink in 627.15 seconds: 103.27 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1040 (time 1760129402.70 total 10.01 last 103.89) - open/close 2074 (time 1760129412.71 total 20.02 last 103.35) - open/close 3070 (time 1760129422.72 total 30.03 last 99.50) - open/close 4140 (time 1760129432.72 total 40.03 last 106.99) - open/close 5234 (time 1760129442.73 total 50.03 last 109.32) - open/close 6243 (time 1760129452.73 total 60.04 last 100.87) - open/close 7383 (time 1760129462.73 total 70.04 last 113.94) - open/close 8557 (time 1760129472.74 total 80.05 last 117.32) - open/close 9710 (time 1760129482.74 total 90.05 last 115.30) - open/close 10000 (time 1760129485.36 total 92.66 last 110.95) - open/close 11339 (time 1760129495.36 total 102.67 last 133.83) - open/close 12485 (time 1760129505.37 total 112.67 last 114.53) - open/close 13510 (time 1760129515.37 total 122.68 last 102.43) - open/close 14605 (time 1760129525.38 total 132.68 last 109.46) - open/close 15676 (time 1760129535.38 total 142.69 last 107.04) - open/close 16850 (time 1760129545.38 total 152.69 last 117.38) - open/close 18183 (time 1760129555.38 total 162.69 last 133.29) - open/close 19522 (time 1760129565.39 total 172.70 last 133.82) - open/close 20000 (time 1760129569.43 total 176.74 last 118.26) - open/close 21108 (time 1760129579.44 total 186.74 last 110.75) - open/close 22347 (time 1760129589.44 total 196.74 last 123.89) - open/close 23752 (time 1760129599.44 total 206.75 last 140.43) - open/close 24966 (time 1760129609.45 total 216.75 last 121.37) - open/close 26211 (time 1760129619.45 total 226.75 last 124.47) - open/close 27496 (time 1760129629.45 total 236.76 last 128.46) - open/close 28607 (time 1760129639.45 total 246.76 last 111.07) - open/close 29655 (time 1760129649.46 total 256.76 last 104.77) - open/close 30000 (time 1760129652.01 total 259.31 last 135.27) - open/close 31434 (time 1760129662.01 total 269.32 last 143.31) - open/close 32825 (time 1760129672.02 total 279.33 last 139.00) - open/close 34224 (time 1760129682.03 total 289.33 last 139.83) - open/close 35768 (time 1760129692.03 total 299.34 last 154.32) - open/close 37296 (time 1760129702.03 total 309.34 last 152.79) - open/close 38464 (time 1760129712.04 total 319.35 last 116.68) - open/close 39896 (time 1760129722.04 total 329.35 last 143.18) - open/close 41354 (time 1760129732.05 total 339.36 last 145.71) - open/close 42938 (time 1760129742.05 total 349.36 last 158.32) - open/close 44251 (time 1760129752.06 total 359.36 last 131.28) - open/close 45499 (time 1760129762.06 total 369.36 last 124.79) - open/close 47143 (time 1760129772.06 total 379.37 last 164.31) - open/close 48186 (time 1760129782.08 total 389.38 last 104.14) - open/close 49382 (time 1760129792.08 total 399.39 last 119.56) - open/close 50000 (time 1760129797.03 total 404.34 last 124.77) - open/close 51596 (time 1760129807.04 total 414.35 last 159.44) - open/close 53061 (time 1760129817.05 total 424.35 last 146.47) - open/close 54813 (time 1760129827.05 total 434.36 last 175.14) - open/close 56564 (time 1760129837.05 total 444.36 last 175.09) - open/close 58645 (time 1760129847.05 total 454.36 last 208.02) - open/close 60000 (time 1760129855.65 total 462.96 last 157.62) - open/close 61734 (time 1760129865.65 total 472.96 last 173.35) - open/close 63331 (time 1760129875.66 total 482.96 last 159.62) total: 64768 open/close in 489.85 seconds: 132.22 ops/second - /unlink 3144 (time 1760129894.29 total 10.00 last 314.37) - /unlink 6055 (time 1760129904.30 total 20.01 last 290.98) - /unlink 9787 (time 1760129914.30 total 30.01 last 373.19) - /unlink 13366 (time 1760129924.32 total 40.03 last 357.12) - /unlink 15960 (time 1760129934.32 total 50.03 last 259.35) - /unlink 19207 (time 1760129944.32 total 60.03 last 324.57) - /unlink 20000 (time 1760129946.78 total 62.49 last 322.34) - /unlink 24250 (time 1760129956.78 total 72.49 last 424.96) - /unlink 28220 (time 1760129966.79 total 82.49 last 396.98) - /unlink 30000 (time 1760129971.90 total 87.61 last 348.26) - /unlink 33819 (time 1760129981.90 total 97.61 last 381.76) - /unlink 38554 (time 1760129991.91 total 107.62 last 473.22) - /unlink 40000 (time 1760129995.79 total 111.50 last 372.06) - /unlink 43963 (time 1760130005.79 total 121.50 last 396.20) - /unlink 46767 (time 1760130015.80 total 131.51 last 280.31) - /unlink 50000 (time 1760130024.87 total 140.58 last 356.48) - /unlink 53687 (time 1760130034.87 total 150.58 last 368.57) - /unlink 56562 (time 1760130044.87 total 160.58 last 287.44) - /unlink 59593 (time 1760130054.88 total 170.59 last 303.01) - /unlink 62511 (time 1760130064.88 total 180.59 last 291.77) total: 64768 /unlink in 186.74 seconds: 346.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1545 (time 1760130102.81 total 10.00 last 154.48) - open/close 3269 (time 1760130112.81 total 20.00 last 172.36) - open/close 4688 (time 1760130122.82 total 30.01 last 141.81) - open/close 6201 (time 1760130132.82 total 40.01 last 151.29) - open/close 7694 (time 1760130142.82 total 50.02 last 149.22) - open/close 8929 (time 1760130152.83 total 60.02 last 123.41) - open/close 10000 (time 1760130160.85 total 68.04 last 133.62) - open/close 11582 (time 1760130170.85 total 78.04 last 158.13) - open/close 13055 (time 1760130180.86 total 88.05 last 147.19) - open/close 14439 (time 1760130190.86 total 98.05 last 138.40) - open/close 16156 (time 1760130200.87 total 108.06 last 171.56) - open/close 18331 (time 1760130210.87 total 118.06 last 217.47) - open/close 20000 (time 1760130220.11 total 127.30 last 180.57) - open/close 21959 (time 1760130230.11 total 137.31 last 195.86) - open/close 23556 (time 1760130240.12 total 147.31 last 159.58) - open/close 25305 (time 1760130250.13 total 157.32 last 174.78) - open/close 27019 (time 1760130260.13 total 167.32 last 171.35) - open/close 29079 (time 1760130270.14 total 177.33 last 205.79) - open/close 30000 (time 1760130275.00 total 182.19 last 189.57) - open/close 31767 (time 1760130285.00 total 192.20 last 176.62) - open/close 34101 (time 1760130295.01 total 202.20 last 233.33) - open/close 36738 (time 1760130305.01 total 212.20 last 263.69) - open/close 39365 (time 1760130315.01 total 222.20 last 262.67) - open/close 40000 (time 1760130317.35 total 224.54 last 271.29) - open/close 42374 (time 1760130327.35 total 234.54 last 237.33) - open/close 44658 (time 1760130337.35 total 244.55 last 228.30) - open/close 47135 (time 1760130347.36 total 254.55 last 247.67) - open/close 50000 (time 1760130356.86 total 264.05 last 301.44) - open/close 53247 (time 1760130366.86 total 274.06 last 324.65) - open/close 55545 (time 1760130376.97 total 284.16 last 227.37) - open/close 58601 (time 1760130386.97 total 294.16 last 305.60) - open/close 60000 (time 1760130391.00 total 298.19 last 347.30) - open/close 62820 (time 1760130401.00 total 308.19 last 281.97) total: 64768 open/close in 315.46 seconds: 205.31 ops/second - /unlink 5016 (time 1760130419.88 total 10.00 last 501.56) - /unlink 10000 (time 1760130429.35 total 19.47 last 526.09) - /unlink 15144 (time 1760130439.35 total 29.47 last 514.39) - /unlink 19730 (time 1760130449.35 total 39.48 last 458.45) - /unlink 24055 (time 1760130459.41 total 49.53 last 430.06) - /unlink 28396 (time 1760130469.41 total 59.53 last 434.09) - /unlink 30000 (time 1760130472.65 total 62.78 last 494.72) - /unlink 35606 (time 1760130482.65 total 72.78 last 560.57) - /unlink 40000 (time 1760130490.76 total 80.88 last 542.23) - /unlink 45278 (time 1760130500.76 total 90.88 last 527.71) - /unlink 50000 (time 1760130508.03 total 98.16 last 649.14) - /unlink 56723 (time 1760130518.03 total 108.16 last 672.28) - /unlink 60000 (time 1760130523.12 total 113.24 last 644.37) total: 64768 /unlink in 120.46 seconds: 537.68 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg432-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg432-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg432-server stop mds service on oleg432-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg432-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg432-server' oleg432-server: oleg432-server.virtnet: executing unload_modules_local oleg432-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg432-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-client: ssh exited with exit code 2 pdsh@oleg432-client: oleg432-server: ssh exited with exit code 2