== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:13:41 (1746155621) Stopping clients: oleg349-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg349-client.virtnet /mnt/lustre2 (opts:-f) oleg349-server: oleg349-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing load_modules_local oleg349-server: Loading modules from /home/green/git/lustre-release/lustre oleg349-server: detected 4 online CPUs by sysfs oleg349-server: MODOPTS_LIBCFS= oleg349-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg349-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg349-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg349-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg349-server: oleg349-server.virtnet: executing set_default_debug -1 all pdsh@oleg349-client: oleg349-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg349-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg349-client.virtnet: -o user_xattr,flock 192.168.203.149@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1147 (time 1746155677.61 total 10.00 last 114.65) - open/close 2295 (time 1746155687.62 total 20.01 last 114.73) - open/close 3508 (time 1746155697.62 total 30.01 last 121.29) - open/close 4675 (time 1746155707.62 total 40.02 last 116.63) - open/close 5958 (time 1746155717.62 total 50.02 last 128.28) - open/close 7175 (time 1746155727.63 total 60.02 last 121.65) - open/close 8324 (time 1746155737.63 total 70.03 last 114.83) - open/close 9467 (time 1746155747.65 total 80.04 last 114.15) - open/close 10000 (time 1746155752.93 total 85.33 last 100.81) - open/close 11024 (time 1746155762.94 total 95.34 last 102.33) - open/close 12120 (time 1746155772.94 total 105.34 last 109.59) - open/close 13388 (time 1746155782.95 total 115.34 last 126.75) - open/close 14510 (time 1746155792.96 total 125.35 last 112.10) - open/close 15597 (time 1746155802.96 total 135.35 last 108.65) - open/close 16683 (time 1746155812.97 total 145.36 last 108.50) - open/close 17585 (time 1746155822.97 total 155.37 last 90.17) - open/close 18607 (time 1746155832.97 total 165.37 last 102.19) - open/close 19645 (time 1746155842.98 total 175.37 last 103.73) - open/close 20000 (time 1746155846.78 total 179.18 last 93.33) - open/close 20930 (time 1746155856.79 total 189.19 last 92.93) - open/close 22012 (time 1746155866.80 total 199.19 last 108.14) - open/close 23128 (time 1746155876.80 total 209.19 last 111.57) - open/close 24230 (time 1746155886.81 total 219.20 last 110.11) - open/close 25308 (time 1746155896.81 total 229.21 last 107.77) - open/close 26417 (time 1746155906.82 total 239.21 last 110.82) - open/close 27463 (time 1746155916.82 total 249.22 last 104.54) - open/close 28515 (time 1746155926.84 total 259.23 last 105.05) - open/close 29766 (time 1746155936.84 total 269.24 last 125.02) - open/close 30887 (time 1746155946.85 total 279.24 last 112.04) - open/close 32097 (time 1746155956.86 total 289.25 last 120.87) - open/close 33156 (time 1746155966.87 total 299.26 last 105.83) - open/close 34218 (time 1746155976.88 total 309.28 last 106.04) - open/close 35349 (time 1746155986.89 total 319.28 last 113.05) - open/close 36550 (time 1746155996.89 total 329.29 last 120.04) - open/close 37734 (time 1746156006.90 total 339.29 last 118.31) - open/close 39035 (time 1746156016.90 total 349.30 last 130.05) - open/close 40000 (time 1746156024.06 total 356.45 last 134.82) - open/close 41120 (time 1746156034.07 total 366.46 last 111.91) - open/close 42114 (time 1746156044.08 total 376.48 last 99.25) - open/close 43127 (time 1746156054.09 total 386.49 last 101.18) - open/close 44188 (time 1746156064.10 total 396.50 last 106.03) - open/close 45409 (time 1746156074.10 total 406.50 last 122.09) - open/close 46502 (time 1746156084.11 total 416.50 last 109.24) - open/close 47563 (time 1746156094.12 total 426.51 last 105.99) - open/close 48527 (time 1746156104.12 total 436.52 last 96.36) - open/close 49608 (time 1746156114.13 total 446.52 last 108.06) - open/close 50000 (time 1746156117.93 total 450.33 last 103.03) - open/close 51058 (time 1746156127.94 total 460.33 last 105.74) - open/close 52074 (time 1746156137.95 total 470.35 last 101.46) - open/close 53021 (time 1746156147.96 total 480.35 last 94.64) - open/close 54235 (time 1746156157.96 total 490.35 last 121.37) - open/close 55344 (time 1746156167.96 total 500.35 last 110.90) - open/close 56674 (time 1746156177.96 total 510.36 last 132.97) - open/close 57876 (time 1746156187.97 total 520.36 last 120.13) - open/close 58937 (time 1746156197.97 total 530.36 last 106.10) - open/close 60000 (time 1746156205.83 total 538.23 last 135.15) - open/close 60978 (time 1746156215.84 total 548.23 last 97.76) - open/close 62141 (time 1746156225.84 total 558.23 last 116.28) - open/close 63179 (time 1746156235.84 total 568.24 last 103.77) - open/close 64287 (time 1746156245.85 total 578.24 last 110.75) total: 64768 open/close in 582.13 seconds: 111.26 ops/second - /unlink 1756 (time 1746156261.52 total 10.00 last 175.55) - /unlink 3443 (time 1746156271.52 total 20.00 last 168.67) - /unlink 5088 (time 1746156281.53 total 30.01 last 164.43) - /unlink 7107 (time 1746156291.53 total 40.01 last 201.84) - /unlink 9275 (time 1746156301.53 total 50.01 last 216.73) - /unlink 10000 (time 1746156304.66 total 53.14 last 231.70) - /unlink 11612 (time 1746156314.67 total 63.15 last 161.17) - /unlink 13216 (time 1746156324.67 total 73.15 last 160.36) - /unlink 15096 (time 1746156334.67 total 83.15 last 187.95) - /unlink 16926 (time 1746156344.67 total 93.15 last 182.95) - /unlink 19152 (time 1746156354.68 total 103.16 last 222.57) - /unlink 20000 (time 1746156358.04 total 106.52 last 252.38) - /unlink 22010 (time 1746156368.04 total 116.52 last 200.93) - /unlink 24094 (time 1746156378.04 total 126.52 last 208.35) - /unlink 26223 (time 1746156388.04 total 136.52 last 212.82) - /unlink 28357 (time 1746156398.06 total 146.54 last 213.17) - /unlink 30000 (time 1746156406.99 total 155.47 last 183.82) - /unlink 31956 (time 1746156417.00 total 165.48 last 195.50) - /unlink 33851 (time 1746156427.00 total 175.48 last 189.44) - /unlink 35814 (time 1746156437.00 total 185.48 last 196.27) - /unlink 37981 (time 1746156447.01 total 195.49 last 216.63) - /unlink 40000 (time 1746156455.90 total 204.38 last 226.93) - /unlink 42322 (time 1746156465.90 total 214.38 last 232.19) - /unlink 44173 (time 1746156475.90 total 224.38 last 185.08) - /unlink 46272 (time 1746156485.91 total 234.39 last 209.88) - /unlink 48182 (time 1746156495.91 total 244.39 last 190.96) - /unlink 50000 (time 1746156504.08 total 252.56 last 222.60) - /unlink 53259 (time 1746156514.08 total 262.56 last 325.89) - /unlink 55948 (time 1746156524.08 total 272.56 last 268.83) - /unlink 58475 (time 1746156534.08 total 282.56 last 252.66) - /unlink 60000 (time 1746156538.91 total 287.39 last 315.60) - /unlink 63605 (time 1746156548.91 total 297.39 last 360.48) total: 64768 /unlink in 301.98 seconds: 214.48 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1824 (time 1746156583.94 total 10.01 last 182.28) - open/close 3414 (time 1746156593.94 total 20.01 last 159.00) - open/close 5100 (time 1746156603.95 total 30.01 last 168.55) - open/close 6401 (time 1746156613.95 total 40.01 last 130.06) - open/close 7787 (time 1746156623.95 total 50.02 last 138.58) - open/close 9278 (time 1746156633.95 total 60.02 last 149.10) - open/close 10000 (time 1746156638.96 total 65.02 last 144.18) - open/close 11401 (time 1746156648.96 total 75.03 last 140.04) - open/close 12609 (time 1746156658.97 total 85.03 last 120.78) - open/close 13773 (time 1746156668.97 total 95.03 last 116.36) - open/close 14916 (time 1746156678.97 total 105.03 last 114.29) - open/close 16085 (time 1746156688.98 total 115.04 last 116.85) - open/close 17219 (time 1746156698.98 total 125.04 last 113.35) - open/close 18494 (time 1746156708.99 total 135.05 last 127.42) - open/close 19742 (time 1746156718.99 total 145.05 last 124.77) - open/close 20000 (time 1746156721.25 total 147.32 last 113.85) - open/close 21184 (time 1746156731.26 total 157.32 last 118.39) - open/close 22413 (time 1746156741.26 total 167.32 last 122.87) - open/close 23655 (time 1746156751.26 total 177.32 last 124.17) - open/close 24841 (time 1746156761.26 total 187.33 last 118.57) - open/close 26186 (time 1746156771.27 total 197.33 last 134.47) - open/close 27425 (time 1746156781.27 total 207.33 last 123.87) - open/close 28819 (time 1746156791.28 total 217.34 last 139.29) - open/close 30000 (time 1746156800.26 total 226.32 last 131.51) - open/close 31336 (time 1746156810.26 total 236.32 last 133.56) - open/close 32704 (time 1746156820.26 total 246.33 last 136.76) - open/close 33902 (time 1746156830.27 total 256.33 last 119.70) - open/close 34971 (time 1746156840.27 total 266.34 last 106.87) - open/close 36051 (time 1746156850.28 total 276.34 last 107.98) - open/close 37191 (time 1746156860.28 total 286.34 last 113.94) - open/close 38304 (time 1746156870.29 total 296.36 last 111.14) - open/close 39382 (time 1746156880.30 total 306.36 last 107.79) - open/close 40000 (time 1746156885.99 total 312.05 last 108.61) - open/close 41174 (time 1746156896.00 total 322.06 last 117.28) - open/close 42353 (time 1746156906.00 total 332.06 last 117.88) - open/close 43338 (time 1746156916.00 total 342.06 last 98.48) - open/close 44428 (time 1746156926.00 total 352.07 last 108.96) - open/close 45468 (time 1746156936.01 total 362.07 last 103.98) - open/close 46484 (time 1746156946.01 total 372.07 last 101.55) - open/close 47814 (time 1746156956.01 total 382.07 last 132.99) - open/close 48948 (time 1746156966.02 total 392.09 last 113.27) - open/close 49921 (time 1746156976.02 total 402.09 last 97.29) - open/close 51171 (time 1746156986.03 total 412.10 last 124.90) - open/close 52379 (time 1746156996.03 total 422.10 last 120.78) - open/close 53470 (time 1746157006.04 total 432.10 last 109.04) - open/close 54466 (time 1746157016.04 total 442.10 last 99.60) - open/close 55504 (time 1746157026.04 total 452.10 last 103.79) - open/close 56459 (time 1746157036.05 total 462.11 last 95.45) - open/close 57443 (time 1746157046.05 total 472.11 last 98.37) - open/close 58610 (time 1746157056.05 total 482.11 last 116.66) - open/close 59698 (time 1746157066.06 total 492.12 last 108.75) - open/close 60000 (time 1746157068.98 total 495.04 last 103.38) - open/close 61197 (time 1746157078.98 total 505.04 last 119.67) - open/close 62187 (time 1746157088.99 total 515.05 last 98.92) - open/close 63139 (time 1746157099.00 total 525.06 last 95.11) - open/close 64099 (time 1746157109.00 total 535.06 last 95.99) total: 64768 open/close in 542.23 seconds: 119.45 ops/second - /unlink 1630 (time 1746157128.52 total 10.00 last 162.98) - /unlink 3218 (time 1746157138.52 total 20.00 last 158.76) - /unlink 5238 (time 1746157148.52 total 30.01 last 201.96) - /unlink 6989 (time 1746157158.52 total 40.01 last 175.10) - /unlink 8695 (time 1746157168.53 total 50.01 last 170.49) - /unlink 10000 (time 1746157176.91 total 58.40 last 155.63) - /unlink 11745 (time 1746157186.92 total 68.40 last 174.44) - /unlink 13831 (time 1746157196.92 total 78.40 last 208.59) - /unlink 16131 (time 1746157206.92 total 88.40 last 230.00) - /unlink 18548 (time 1746157216.92 total 98.40 last 241.65) - /unlink 20000 (time 1746157223.21 total 104.70 last 230.80) - /unlink 22394 (time 1746157233.21 total 114.70 last 239.39) - /unlink 24983 (time 1746157243.21 total 124.70 last 258.86) - /unlink 27175 (time 1746157253.22 total 134.70 last 219.06) - /unlink 29290 (time 1746157263.22 total 144.71 last 211.47) - /unlink 30000 (time 1746157266.80 total 148.28 last 198.40) - /unlink 32047 (time 1746157276.81 total 158.29 last 204.60) - /unlink 34605 (time 1746157286.81 total 168.29 last 255.79) - /unlink 37250 (time 1746157296.81 total 178.29 last 264.38) - /unlink 39565 (time 1746157306.81 total 188.30 last 231.47) - /unlink 40000 (time 1746157308.89 total 190.37 last 209.33) - /unlink 41884 (time 1746157318.89 total 200.38 last 188.35) - /unlink 43850 (time 1746157328.89 total 210.38 last 196.56) - /unlink 46220 (time 1746157338.90 total 220.38 last 236.92) - /unlink 48553 (time 1746157348.90 total 230.38 last 233.23) - /unlink 50000 (time 1746157356.27 total 237.76 last 196.30) - /unlink 52341 (time 1746157366.27 total 247.76 last 234.10) - /unlink 54319 (time 1746157376.27 total 257.76 last 197.78) - /unlink 56559 (time 1746157386.28 total 267.76 last 223.87) - /unlink 58974 (time 1746157396.28 total 277.77 last 241.43) - /unlink 60000 (time 1746157399.75 total 281.23 last 296.28) - /unlink 62822 (time 1746157409.75 total 291.23 last 282.18) total: 64768 /unlink in 299.57 seconds: 216.20 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1538 (time 1746157446.02 total 10.00 last 153.74) - open/close 3584 (time 1746157456.02 total 20.01 last 204.53) - open/close 5373 (time 1746157466.03 total 30.01 last 178.80) - open/close 7077 (time 1746157476.03 total 40.01 last 170.37) - open/close 8600 (time 1746157486.04 total 50.02 last 152.17) - open/close 10000 (time 1746157493.71 total 57.70 last 182.42) - open/close 11838 (time 1746157503.71 total 67.70 last 183.79) - open/close 13720 (time 1746157513.71 total 77.70 last 188.20) - open/close 15594 (time 1746157523.71 total 87.70 last 187.39) - open/close 17223 (time 1746157533.72 total 97.70 last 162.82) - open/close 18600 (time 1746157543.72 total 107.70 last 137.69) - open/close 20000 (time 1746157553.08 total 117.06 last 149.57) - open/close 21491 (time 1746157563.08 total 127.07 last 149.06) - open/close 23156 (time 1746157573.08 total 137.07 last 166.48) - open/close 24805 (time 1746157583.09 total 147.08 last 164.74) - open/close 26512 (time 1746157593.10 total 157.08 last 170.66) - open/close 27963 (time 1746157603.10 total 167.08 last 145.08) - open/close 29420 (time 1746157613.10 total 177.08 last 145.68) - open/close 30000 (time 1746157617.49 total 181.47 last 132.14) - open/close 31175 (time 1746157627.50 total 191.48 last 117.39) - open/close 32460 (time 1746157637.50 total 201.48 last 128.48) - open/close 33876 (time 1746157647.50 total 211.49 last 141.54) - open/close 35455 (time 1746157657.51 total 221.49 last 157.83) - open/close 36881 (time 1746157667.52 total 231.50 last 142.48) - open/close 38364 (time 1746157677.52 total 241.50 last 148.25) - open/close 39905 (time 1746157687.52 total 251.51 last 154.04) - open/close 41305 (time 1746157697.53 total 261.51 last 139.94) - open/close 42762 (time 1746157707.53 total 271.52 last 145.62) - open/close 44296 (time 1746157717.53 total 281.52 last 153.36) - open/close 45628 (time 1746157727.54 total 291.52 last 133.18) - open/close 46921 (time 1746157737.54 total 301.52 last 129.27) - open/close 48246 (time 1746157747.54 total 311.53 last 132.44) - open/close 49421 (time 1746157757.55 total 321.53 last 117.45) - open/close 50000 (time 1746157762.83 total 326.82 last 109.57) - open/close 51322 (time 1746157772.83 total 336.82 last 132.16) - open/close 52649 (time 1746157782.84 total 346.82 last 132.68) - open/close 53839 (time 1746157792.84 total 356.82 last 119.00) - open/close 55217 (time 1746157802.84 total 366.82 last 137.75) - open/close 56362 (time 1746157812.85 total 376.83 last 114.42) - open/close 57527 (time 1746157822.85 total 386.84 last 116.44) - open/close 58646 (time 1746157832.85 total 396.84 last 111.89) - open/close 59732 (time 1746157842.86 total 406.85 last 108.49) - open/close 60000 (time 1746157845.10 total 409.09 last 119.54) - open/close 61025 (time 1746157855.11 total 419.09 last 102.49) - open/close 62071 (time 1746157865.11 total 429.09 last 104.55) - open/close 63299 (time 1746157875.12 total 439.10 last 122.74) - open/close 64554 (time 1746157885.12 total 449.11 last 125.42) total: 64768 open/close in 450.75 seconds: 143.69 ops/second - /unlink 1907 (time 1746157898.26 total 10.00 last 190.65) - /unlink 3791 (time 1746157908.26 total 20.00 last 188.40) - /unlink 5881 (time 1746157918.27 total 30.01 last 208.92) - /unlink 8416 (time 1746157928.27 total 40.01 last 253.45) - /unlink 10000 (time 1746157935.59 total 47.33 last 216.30) - /unlink 12249 (time 1746157945.59 total 57.33 last 224.85) - /unlink 14379 (time 1746157955.60 total 67.34 last 212.94) - /unlink 16334 (time 1746157965.60 total 77.34 last 195.44) - /unlink 18255 (time 1746157975.60 total 87.34 last 192.09) - /unlink 20000 (time 1746157983.93 total 95.67 last 209.54) - /unlink 22076 (time 1746157993.93 total 105.67 last 207.57) - /unlink 24291 (time 1746158003.93 total 115.67 last 221.45) - /unlink 26621 (time 1746158013.93 total 125.68 last 232.93) - /unlink 28894 (time 1746158023.94 total 135.68 last 227.20) - /unlink 30000 (time 1746158029.78 total 141.52 last 189.38) - /unlink 32108 (time 1746158039.79 total 151.53 last 210.62) - /unlink 34288 (time 1746158049.79 total 161.53 last 217.97) - /unlink 36426 (time 1746158059.79 total 171.53 last 213.78) - /unlink 38311 (time 1746158069.79 total 181.53 last 188.45) - /unlink 40000 (time 1746158079.56 total 191.31 last 172.83) - /unlink 42050 (time 1746158089.57 total 201.31 last 204.99) - /unlink 44968 (time 1746158099.57 total 211.31 last 291.72) - /unlink 48858 (time 1746158109.57 total 221.31 last 388.99) - /unlink 50000 (time 1746158112.01 total 223.75 last 468.31) - /unlink 52929 (time 1746158122.01 total 233.75 last 292.89) - /unlink 55241 (time 1746158132.01 total 243.75 last 231.15) - /unlink 57491 (time 1746158142.01 total 253.75 last 224.94) - /unlink 59402 (time 1746158152.01 total 263.76 last 191.05) - /unlink 60000 (time 1746158154.86 total 266.60 last 210.17) - /unlink 61951 (time 1746158164.86 total 276.60 last 195.04) - /unlink 64036 (time 1746158174.87 total 286.61 last 208.44) total: 64768 /unlink in 289.62 seconds: 223.63 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1762 (time 1746158207.30 total 10.01 last 176.09) - open/close 3173 (time 1746158217.30 total 20.01 last 141.06) - open/close 4637 (time 1746158227.31 total 30.02 last 146.31) - open/close 6165 (time 1746158237.31 total 40.02 last 152.72) - open/close 7981 (time 1746158247.32 total 50.03 last 181.52) - open/close 9375 (time 1746158257.32 total 60.03 last 139.34) - open/close 10000 (time 1746158261.73 total 64.44 last 141.71) - open/close 11811 (time 1746158271.74 total 74.45 last 181.01) - open/close 13095 (time 1746158281.74 total 84.45 last 128.38) - open/close 14634 (time 1746158291.75 total 94.45 last 153.79) - open/close 16537 (time 1746158301.75 total 104.46 last 190.23) - open/close 19077 (time 1746158311.75 total 114.46 last 253.98) - open/close 20000 (time 1746158315.61 total 118.31 last 239.39) - open/close 22134 (time 1746158325.61 total 128.32 last 213.37) - open/close 23805 (time 1746158335.61 total 138.32 last 167.07) - open/close 25357 (time 1746158345.61 total 148.32 last 155.15) - open/close 26989 (time 1746158355.62 total 158.33 last 163.12) - open/close 28561 (time 1746158365.62 total 168.33 last 157.16) - open/close 30000 (time 1746158370.47 total 173.17 last 296.90) - open/close 32055 (time 1746158380.47 total 183.17 last 205.49) - open/close 34278 (time 1746158390.47 total 193.18 last 222.30) - open/close 35829 (time 1746158400.47 total 203.18 last 154.98) - open/close 37334 (time 1746158410.48 total 213.18 last 150.48) - open/close 39295 (time 1746158420.48 total 223.18 last 196.10) - open/close 40000 (time 1746158425.48 total 228.18 last 141.04) - open/close 41389 (time 1746158435.48 total 238.19 last 138.83) - open/close 42896 (time 1746158445.49 total 248.20 last 150.56) - open/close 44644 (time 1746158455.49 total 258.20 last 174.78) - open/close 46219 (time 1746158465.49 total 268.20 last 157.46) - open/close 48052 (time 1746158475.50 total 278.20 last 183.25) - open/close 49434 (time 1746158485.50 total 288.21 last 138.13) - open/close 50000 (time 1746158489.09 total 291.80 last 157.81) - open/close 51688 (time 1746158499.09 total 301.80 last 168.77) - open/close 53723 (time 1746158509.09 total 311.80 last 203.45) - open/close 55955 (time 1746158519.10 total 321.81 last 223.04) - open/close 58002 (time 1746158529.10 total 331.81 last 204.61) - open/close 60000 (time 1746158538.76 total 341.47 last 206.86) - open/close 62201 (time 1746158548.76 total 351.47 last 220.07) - open/close 63607 (time 1746158558.77 total 361.47 last 140.57) total: 64768 open/close in 368.87 seconds: 175.58 ops/second - /unlink 3600 (time 1746158577.28 total 10.00 last 359.90) - /unlink 6052 (time 1746158587.28 total 20.00 last 245.20) - /unlink 8774 (time 1746158597.28 total 30.00 last 272.19) - /unlink 10000 (time 1746158601.66 total 34.39 last 279.63) - /unlink 12767 (time 1746158611.66 total 44.39 last 276.64) - /unlink 15529 (time 1746158621.67 total 54.39 last 276.07) - /unlink 17946 (time 1746158631.67 total 64.40 last 241.64) - /unlink 20000 (time 1746158640.70 total 73.43 last 227.39) - /unlink 22529 (time 1746158650.71 total 83.43 last 252.84) - /unlink 25019 (time 1746158660.71 total 93.43 last 248.99) - /unlink 27472 (time 1746158670.71 total 103.43 last 245.25) - /unlink 29766 (time 1746158680.71 total 113.44 last 229.38) - /unlink 31879 (time 1746158690.71 total 123.44 last 211.28) - /unlink 33745 (time 1746158700.72 total 133.44 last 186.46) - /unlink 35714 (time 1746158710.73 total 143.45 last 196.73) - /unlink 37870 (time 1746158720.73 total 153.45 last 215.56) - /unlink 40000 (time 1746158729.29 total 162.02 last 248.73) - /unlink 42386 (time 1746158739.29 total 172.02 last 238.57) - /unlink 44605 (time 1746158749.29 total 182.02 last 221.88) - /unlink 47887 (time 1746158759.30 total 192.02 last 328.15) - /unlink 50000 (time 1746158765.40 total 198.13 last 346.03) - /unlink 53277 (time 1746158775.40 total 208.13 last 327.65) - /unlink 55561 (time 1746158785.41 total 218.13 last 228.33) - /unlink 58315 (time 1746158795.41 total 228.13 last 275.39) - /unlink 60000 (time 1746158802.41 total 235.14 last 240.60) - /unlink 62852 (time 1746158812.41 total 245.14 last 285.11) total: 64768 /unlink in 249.90 seconds: 259.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1369 (time 1746158843.78 total 10.00 last 136.87) - open/close 3006 (time 1746158853.81 total 20.03 last 163.27) - open/close 4631 (time 1746158863.81 total 30.03 last 162.41) - open/close 5913 (time 1746158873.82 total 40.04 last 128.16) - open/close 7122 (time 1746158883.82 total 50.04 last 120.84) - open/close 8428 (time 1746158893.83 total 60.05 last 130.49) - open/close 9634 (time 1746158903.83 total 70.05 last 120.55) - open/close 10000 (time 1746158906.66 total 72.88 last 129.48) - open/close 11168 (time 1746158916.67 total 82.89 last 116.73) - open/close 12466 (time 1746158926.67 total 92.89 last 129.70) - open/close 13699 (time 1746158936.68 total 102.90 last 123.24) - open/close 14899 (time 1746158946.68 total 112.90 last 119.95) - open/close 16247 (time 1746158956.69 total 122.91 last 134.66) - open/close 17475 (time 1746158966.69 total 132.92 last 122.77) - open/close 18586 (time 1746158976.70 total 142.92 last 111.03) - open/close 19756 (time 1746158986.70 total 152.92 last 116.99) - open/close 20000 (time 1746158989.13 total 155.35 last 100.47) - open/close 21136 (time 1746158999.14 total 165.36 last 113.46) - open/close 22363 (time 1746159009.15 total 175.37 last 122.64) - open/close 23769 (time 1746159019.15 total 185.37 last 140.56) - open/close 25086 (time 1746159029.16 total 195.38 last 131.61) - open/close 26358 (time 1746159039.16 total 205.38 last 127.14) - open/close 27645 (time 1746159049.16 total 215.39 last 128.66) - open/close 28911 (time 1746159059.17 total 225.39 last 126.57) - open/close 30000 (time 1746159067.84 total 234.06 last 125.56) - open/close 31341 (time 1746159077.84 total 244.06 last 134.10) - open/close 32556 (time 1746159087.84 total 254.07 last 121.46) - open/close 33591 (time 1746159097.84 total 264.07 last 103.50) - open/close 34667 (time 1746159107.85 total 274.07 last 107.58) - open/close 35894 (time 1746159117.85 total 284.07 last 122.66) - open/close 37162 (time 1746159127.85 total 294.08 last 126.74) - open/close 38379 (time 1746159137.86 total 304.08 last 121.68) - open/close 39680 (time 1746159147.86 total 314.08 last 130.06) - open/close 40000 (time 1746159150.29 total 316.51 last 131.56) - open/close 41375 (time 1746159160.29 total 326.51 last 137.47) - open/close 42697 (time 1746159170.29 total 336.51 last 132.20) - open/close 44199 (time 1746159180.30 total 346.52 last 150.16) - open/close 45492 (time 1746159190.30 total 356.52 last 129.22) - open/close 46768 (time 1746159200.32 total 366.54 last 127.40) - open/close 48035 (time 1746159210.32 total 376.54 last 126.69) - open/close 49149 (time 1746159220.32 total 386.54 last 111.34) - open/close 50000 (time 1746159227.74 total 393.96 last 114.70) - open/close 51109 (time 1746159237.75 total 403.97 last 110.84) - open/close 52216 (time 1746159247.75 total 413.97 last 110.66) - open/close 53459 (time 1746159257.75 total 423.97 last 124.28) - open/close 54866 (time 1746159267.75 total 433.98 last 140.68) - open/close 56291 (time 1746159277.76 total 443.98 last 142.47) - open/close 57568 (time 1746159287.76 total 453.98 last 127.68) - open/close 58983 (time 1746159297.76 total 463.98 last 141.47) - open/close 60000 (time 1746159305.41 total 471.63 last 132.99) - open/close 61171 (time 1746159315.41 total 481.63 last 117.04) - open/close 62386 (time 1746159325.42 total 491.64 last 121.44) - open/close 63478 (time 1746159335.42 total 501.64 last 109.15) - open/close 64766 (time 1746159345.42 total 511.65 last 128.77) total: 64768 open/close in 511.65 seconds: 126.59 ops/second - /unlink 1895 (time 1746159357.44 total 10.00 last 189.45) - /unlink 3623 (time 1746159367.44 total 20.00 last 172.77) - /unlink 5287 (time 1746159377.45 total 30.01 last 166.25) - /unlink 7057 (time 1746159387.45 total 40.02 last 176.96) - /unlink 9238 (time 1746159397.45 total 50.02 last 218.01) - /unlink 10000 (time 1746159400.26 total 52.83 last 271.52) - /unlink 11975 (time 1746159410.26 total 62.83 last 197.44) - /unlink 13970 (time 1746159420.26 total 72.83 last 199.50) - /unlink 16112 (time 1746159430.26 total 82.83 last 214.18) - /unlink 18118 (time 1746159440.27 total 92.84 last 200.50) - /unlink 19862 (time 1746159450.27 total 102.84 last 174.33) - /unlink 21605 (time 1746159460.28 total 112.85 last 174.11) - /unlink 23491 (time 1746159470.29 total 122.85 last 188.56) - /unlink 25530 (time 1746159480.29 total 132.85 last 203.87) - /unlink 27389 (time 1746159490.29 total 142.86 last 185.86) - /unlink 29308 (time 1746159500.29 total 152.86 last 191.87) - /unlink 30000 (time 1746159503.93 total 156.50 last 190.23) - /unlink 31827 (time 1746159513.93 total 166.50 last 182.70) - /unlink 33538 (time 1746159523.94 total 176.51 last 170.93) - /unlink 35044 (time 1746159533.97 total 186.54 last 150.08) - /unlink 36858 (time 1746159543.98 total 196.55 last 181.30) - /unlink 38860 (time 1746159553.98 total 206.55 last 200.15) - /unlink 40000 (time 1746159559.95 total 212.52 last 191.02) - /unlink 41835 (time 1746159569.95 total 222.52 last 183.45) - /unlink 43763 (time 1746159579.95 total 232.52 last 192.79) - /unlink 45714 (time 1746159589.96 total 242.52 last 194.99) - /unlink 47598 (time 1746159599.96 total 252.53 last 188.32) - /unlink 50000 (time 1746159609.76 total 262.32 last 245.22) - /unlink 52330 (time 1746159619.76 total 272.33 last 232.95) - /unlink 54843 (time 1746159629.76 total 282.33 last 251.28) - /unlink 57919 (time 1746159639.76 total 292.33 last 307.58) - /unlink 60000 (time 1746159648.15 total 300.72 last 248.08) - /unlink 62820 (time 1746159658.15 total 310.72 last 281.95) total: 64768 /unlink in 317.40 seconds: 204.06 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg349-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg349-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg349-server stop mds service on oleg349-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg349-server unloading modules via unload_modules_local on: 'oleg349-server' oleg349-server: oleg349-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-client: ssh exited with exit code 2 pdsh@oleg349-client: oleg349-server: ssh exited with exit code 2