== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:02:27 (1746140547) Stopping clients: oleg224-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg224-client.virtnet /mnt/lustre2 (opts:-f) oleg224-server: oleg224-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing load_modules_local oleg224-server: Loading modules from /home/green/git/lustre-release/lustre oleg224-server: detected 4 online CPUs by sysfs oleg224-server: MODOPTS_LIBCFS= oleg224-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg224-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg224-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg224-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg224-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg224-server: oleg224-server.virtnet: executing set_default_debug -1 all pdsh@oleg224-client: oleg224-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg224-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg224-client.virtnet: -o user_xattr,flock 192.168.202.124@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1178 (time 1746140618.07 total 10.01 last 117.71) - open/close 2267 (time 1746140628.08 total 20.02 last 108.81) - open/close 3430 (time 1746140638.09 total 30.03 last 116.16) - open/close 4514 (time 1746140648.09 total 40.03 last 108.35) - open/close 5624 (time 1746140658.09 total 50.03 last 110.99) - open/close 6715 (time 1746140668.10 total 60.04 last 109.08) - open/close 7752 (time 1746140678.10 total 70.04 last 103.63) - open/close 8735 (time 1746140688.11 total 80.04 last 98.27) - open/close 10000 (time 1746140697.60 total 89.53 last 133.31) - open/close 11084 (time 1746140707.60 total 99.54 last 108.32) - open/close 12097 (time 1746140717.60 total 109.54 last 101.30) - open/close 13117 (time 1746140727.60 total 119.54 last 101.99) - open/close 14496 (time 1746140737.61 total 129.55 last 137.79) - open/close 15593 (time 1746140747.62 total 139.56 last 109.63) - open/close 16679 (time 1746140757.62 total 149.56 last 108.57) - open/close 17720 (time 1746140767.62 total 159.56 last 104.05) - open/close 18919 (time 1746140777.63 total 169.57 last 119.85) - open/close 19980 (time 1746140787.64 total 179.58 last 106.00) - open/close 21089 (time 1746140797.65 total 189.59 last 110.80) - open/close 22180 (time 1746140807.65 total 199.59 last 109.06) - open/close 23449 (time 1746140817.65 total 209.59 last 126.87) - open/close 24559 (time 1746140827.66 total 219.60 last 110.96) - open/close 25616 (time 1746140837.67 total 229.61 last 105.58) - open/close 26653 (time 1746140847.67 total 239.61 last 103.67) - open/close 27798 (time 1746140857.68 total 249.61 last 114.45) - open/close 28939 (time 1746140867.68 total 259.62 last 114.04) - open/close 29932 (time 1746140877.69 total 269.63 last 99.20) - open/close 30996 (time 1746140887.71 total 279.64 last 106.24) - open/close 31991 (time 1746140897.71 total 289.65 last 99.49) - open/close 33036 (time 1746140907.72 total 299.65 last 104.41) - open/close 34041 (time 1746140917.72 total 309.66 last 100.47) - open/close 35028 (time 1746140927.72 total 319.66 last 98.64) - open/close 36183 (time 1746140937.73 total 329.67 last 115.40) - open/close 37322 (time 1746140947.74 total 339.68 last 113.85) - open/close 38475 (time 1746140957.75 total 349.69 last 115.18) - open/close 39594 (time 1746140967.76 total 359.70 last 111.72) - open/close 40000 (time 1746140971.62 total 363.55 last 105.40) - open/close 41137 (time 1746140981.62 total 373.56 last 113.65) - open/close 42330 (time 1746140991.63 total 383.57 last 119.21) - open/close 43241 (time 1746141001.63 total 393.57 last 91.04) - open/close 44229 (time 1746141011.64 total 403.58 last 98.78) - open/close 45321 (time 1746141021.65 total 413.59 last 109.09) - open/close 46460 (time 1746141031.65 total 423.59 last 113.88) - open/close 47388 (time 1746141041.65 total 433.59 last 92.76) - open/close 48364 (time 1746141051.66 total 443.59 last 97.58) - open/close 49415 (time 1746141061.66 total 453.60 last 105.02) - open/close 50000 (time 1746141066.98 total 458.92 last 110.03) - open/close 51146 (time 1746141076.99 total 468.92 last 114.53) - open/close 52091 (time 1746141086.99 total 478.93 last 94.44) - open/close 53091 (time 1746141096.99 total 488.93 last 99.98) - open/close 54172 (time 1746141107.00 total 498.94 last 108.07) - open/close 55397 (time 1746141117.00 total 508.94 last 122.49) - open/close 56468 (time 1746141127.00 total 518.94 last 107.07) - open/close 57579 (time 1746141137.01 total 528.95 last 110.94) - open/close 58634 (time 1746141147.02 total 538.96 last 105.45) - open/close 59751 (time 1746141157.02 total 548.96 last 111.65) - open/close 60000 (time 1746141159.89 total 551.83 last 86.74) - open/close 60928 (time 1746141169.90 total 561.84 last 92.75) - open/close 61916 (time 1746141179.91 total 571.85 last 98.72) - open/close 62909 (time 1746141189.91 total 581.85 last 99.29) - open/close 63975 (time 1746141199.92 total 591.86 last 106.46) total: 64768 open/close in 600.99 seconds: 107.77 ops/second - /unlink 850 (time 1746141221.55 total 10.00 last 84.97) - /unlink 1773 (time 1746141231.56 total 20.01 last 92.27) - /unlink 2843 (time 1746141241.56 total 30.01 last 106.97) - /unlink 3953 (time 1746141251.56 total 40.01 last 110.98) - /unlink 4961 (time 1746141261.56 total 50.01 last 100.79) - /unlink 5964 (time 1746141271.57 total 60.02 last 100.21) - /unlink 7106 (time 1746141281.58 total 70.03 last 114.13) - /unlink 8358 (time 1746141291.58 total 80.03 last 125.19) - /unlink 9524 (time 1746141301.58 total 90.03 last 116.58) - /unlink 10000 (time 1746141305.26 total 93.71 last 129.46) - /unlink 11077 (time 1746141315.26 total 103.71 last 107.68) - /unlink 12383 (time 1746141325.27 total 113.72 last 130.49) - /unlink 13623 (time 1746141335.27 total 123.73 last 123.89) - /unlink 14675 (time 1746141345.28 total 133.73 last 105.14) - /unlink 15928 (time 1746141355.29 total 143.74 last 125.17) - /unlink 17409 (time 1746141365.30 total 153.75 last 148.02) - /unlink 19028 (time 1746141375.30 total 163.75 last 161.83) - /unlink 20000 (time 1746141379.33 total 167.78 last 241.50) - /unlink 22225 (time 1746141389.33 total 177.78 last 222.38) - /unlink 24308 (time 1746141399.34 total 187.79 last 208.07) - /unlink 26113 (time 1746141409.34 total 197.79 last 180.49) - /unlink 28128 (time 1746141419.34 total 207.79 last 201.48) - /unlink 30000 (time 1746141428.29 total 216.74 last 209.30) - /unlink 32119 (time 1746141438.29 total 226.74 last 211.89) - /unlink 35164 (time 1746141448.29 total 236.74 last 304.50) - /unlink 37961 (time 1746141458.29 total 246.74 last 279.59) - /unlink 40000 (time 1746141466.81 total 255.26 last 239.52) - /unlink 42230 (time 1746141476.81 total 265.26 last 222.97) - /unlink 44266 (time 1746141486.81 total 275.26 last 203.56) - /unlink 46519 (time 1746141496.81 total 285.26 last 225.25) - /unlink 48600 (time 1746141506.81 total 295.26 last 208.06) - /unlink 50000 (time 1746141512.75 total 301.20 last 235.79) - /unlink 52022 (time 1746141522.75 total 311.20 last 202.13) - /unlink 54696 (time 1746141532.76 total 321.21 last 267.33) - /unlink 57915 (time 1746141542.76 total 331.21 last 321.81) - /unlink 60000 (time 1746141551.77 total 340.22 last 231.44) - /unlink 62573 (time 1746141561.77 total 350.22 last 257.29) total: 64768 /unlink in 359.83 seconds: 179.99 ops/second Waiting for MDT destroys to complete - open/close 1356 (time 1746141592.80 total 10.00 last 135.58) - open/close 2661 (time 1746141602.80 total 20.00 last 130.48) - open/close 4209 (time 1746141612.80 total 30.01 last 154.77) - open/close 5812 (time 1746141622.81 total 40.01 last 160.25) - open/close 7356 (time 1746141632.81 total 50.02 last 154.27) - open/close 8714 (time 1746141642.82 total 60.02 last 135.69) - open/close 10000 (time 1746141651.60 total 68.81 last 146.45) - open/close 11227 (time 1746141661.60 total 78.81 last 122.69) - open/close 12647 (time 1746141671.62 total 88.82 last 141.82) - open/close 14034 (time 1746141681.62 total 98.82 last 138.70) - open/close 15442 (time 1746141691.62 total 108.82 last 140.77) - open/close 16852 (time 1746141701.63 total 118.83 last 140.89) - open/close 18268 (time 1746141711.63 total 128.83 last 141.53) - open/close 19521 (time 1746141721.64 total 138.84 last 125.22) - open/close 20000 (time 1746141724.79 total 141.99 last 152.00) - open/close 21246 (time 1746141734.80 total 152.00 last 124.51) - open/close 22302 (time 1746141744.80 total 162.00 last 105.55) - open/close 23615 (time 1746141754.81 total 172.01 last 131.23) - open/close 24787 (time 1746141764.81 total 182.01 last 117.16) - open/close 25805 (time 1746141774.81 total 192.02 last 101.75) - open/close 26897 (time 1746141784.81 total 202.02 last 109.20) - open/close 27878 (time 1746141794.82 total 212.03 last 98.02) - open/close 28851 (time 1746141804.83 total 222.03 last 97.24) - open/close 29849 (time 1746141814.84 total 232.04 last 99.74) - open/close 30914 (time 1746141824.84 total 242.05 last 106.41) - open/close 31930 (time 1746141834.85 total 252.05 last 101.55) - open/close 33032 (time 1746141844.85 total 262.06 last 110.14) - open/close 34059 (time 1746141854.86 total 272.06 last 102.67) - open/close 35184 (time 1746141864.87 total 282.07 last 112.36) - open/close 36254 (time 1746141874.87 total 292.08 last 106.98) - open/close 37400 (time 1746141884.88 total 302.08 last 114.52) - open/close 38515 (time 1746141894.89 total 312.09 last 111.37) - open/close 39703 (time 1746141904.89 total 322.09 last 118.79) - open/close 40000 (time 1746141907.34 total 324.55 last 121.13) - open/close 41438 (time 1746141917.34 total 334.55 last 143.80) - open/close 42708 (time 1746141927.35 total 344.55 last 126.93) - open/close 43848 (time 1746141937.35 total 354.56 last 113.94) - open/close 45082 (time 1746141947.36 total 364.56 last 123.32) - open/close 46164 (time 1746141957.37 total 374.57 last 108.13) - open/close 47200 (time 1746141967.38 total 384.58 last 103.48) - open/close 48380 (time 1746141977.38 total 394.58 last 117.97) - open/close 49489 (time 1746141987.40 total 404.60 last 110.72) - open/close 50000 (time 1746141991.43 total 408.63 last 126.76) - open/close 51244 (time 1746142001.43 total 418.64 last 124.32) - open/close 52401 (time 1746142011.44 total 428.64 last 115.65) - open/close 53490 (time 1746142021.44 total 438.64 last 108.90) - open/close 54789 (time 1746142031.44 total 448.65 last 129.84) - open/close 56089 (time 1746142041.45 total 458.65 last 129.95) - open/close 57315 (time 1746142051.45 total 468.65 last 122.55) - open/close 58640 (time 1746142061.46 total 478.67 last 132.34) - open/close 59788 (time 1746142071.46 total 488.67 last 114.80) - open/close 61354 (time 1746142081.47 total 498.67 last 156.49) - open/close 62838 (time 1746142091.47 total 508.68 last 148.34) - open/close 64059 (time 1746142101.48 total 518.68 last 122.05) total: 64768 open/close in 524.95 seconds: 123.38 ops/second - /unlink 1881 (time 1746142119.54 total 10.00 last 188.08) - /unlink 4149 (time 1746142129.55 total 20.00 last 226.77) - /unlink 6751 (time 1746142139.55 total 30.00 last 260.13) - /unlink 9175 (time 1746142149.55 total 40.01 last 242.36) - /unlink 10000 (time 1746142153.17 total 43.63 last 227.62) - /unlink 12129 (time 1746142163.18 total 53.64 last 212.79) - /unlink 14219 (time 1746142173.18 total 63.64 last 208.99) - /unlink 16236 (time 1746142183.18 total 73.64 last 201.67) - /unlink 18081 (time 1746142193.18 total 83.64 last 184.48) - /unlink 20000 (time 1746142202.87 total 93.32 last 198.12) - /unlink 21802 (time 1746142212.87 total 103.33 last 180.14) - /unlink 23713 (time 1746142222.88 total 113.34 last 190.96) - /unlink 25589 (time 1746142232.88 total 123.34 last 187.57) - /unlink 27603 (time 1746142242.88 total 133.34 last 201.34) - /unlink 29463 (time 1746142252.89 total 143.34 last 185.93) - /unlink 30000 (time 1746142255.40 total 145.86 last 213.30) - /unlink 31880 (time 1746142265.40 total 155.86 last 187.99) - /unlink 33749 (time 1746142275.41 total 165.87 last 186.82) - /unlink 35379 (time 1746142285.41 total 175.87 last 162.96) - /unlink 36794 (time 1746142295.41 total 185.87 last 141.49) - /unlink 38459 (time 1746142305.42 total 195.87 last 166.43) - /unlink 40000 (time 1746142314.26 total 204.71 last 174.31) - /unlink 41970 (time 1746142324.26 total 214.71 last 196.97) - /unlink 44082 (time 1746142334.26 total 224.72 last 211.16) - /unlink 46126 (time 1746142344.26 total 234.72 last 204.38) - /unlink 48286 (time 1746142354.26 total 244.72 last 215.99) - /unlink 50000 (time 1746142363.01 total 253.47 last 195.92) - /unlink 52061 (time 1746142373.02 total 263.47 last 205.94) - /unlink 53838 (time 1746142383.02 total 273.48 last 177.65) - /unlink 56054 (time 1746142393.02 total 283.48 last 221.57) - /unlink 58193 (time 1746142403.02 total 293.48 last 213.87) - /unlink 60000 (time 1746142411.22 total 301.67 last 220.51) - /unlink 61996 (time 1746142421.22 total 311.68 last 199.58) - /unlink 64002 (time 1746142431.22 total 321.68 last 200.54) total: 64768 /unlink in 325.00 seconds: 199.29 ops/second Waiting for MDT destroys to complete - open/close 1037 (time 1746142454.71 total 10.00 last 103.70) - open/close 2324 (time 1746142464.71 total 20.01 last 128.59) - open/close 3585 (time 1746142474.72 total 30.01 last 126.07) - open/close 4915 (time 1746142484.72 total 40.02 last 132.94) - open/close 6325 (time 1746142494.72 total 50.02 last 140.99) - open/close 7706 (time 1746142504.72 total 60.02 last 138.07) - open/close 9219 (time 1746142514.73 total 70.02 last 151.23) - open/close 10000 (time 1746142520.71 total 76.01 last 130.47) - open/close 11283 (time 1746142530.72 total 86.01 last 128.24) - open/close 12541 (time 1746142540.73 total 96.02 last 125.71) - open/close 13669 (time 1746142550.73 total 106.03 last 112.73) - open/close 14971 (time 1746142560.74 total 116.03 last 130.15) - open/close 16275 (time 1746142570.75 total 126.04 last 130.28) - open/close 17523 (time 1746142580.75 total 136.04 last 124.78) - open/close 18911 (time 1746142590.75 total 146.05 last 138.75) - open/close 20000 (time 1746142600.12 total 155.42 last 116.21) - open/close 21079 (time 1746142610.13 total 165.42 last 107.85) - open/close 22100 (time 1746142620.13 total 175.42 last 102.08) - open/close 23140 (time 1746142630.14 total 185.43 last 103.91) - open/close 24213 (time 1746142640.14 total 195.44 last 107.25) - open/close 25543 (time 1746142650.14 total 205.44 last 132.98) - open/close 26812 (time 1746142660.15 total 215.45 last 126.78) - open/close 28041 (time 1746142670.16 total 225.45 last 122.86) - open/close 29323 (time 1746142680.16 total 235.45 last 128.16) - open/close 30000 (time 1746142685.64 total 240.93 last 123.51) - open/close 31153 (time 1746142695.64 total 250.94 last 115.27) - open/close 32328 (time 1746142705.65 total 260.95 last 117.39) - open/close 33432 (time 1746142715.65 total 270.95 last 110.38) - open/close 34624 (time 1746142725.66 total 280.95 last 119.19) - open/close 35870 (time 1746142735.66 total 290.96 last 124.54) - open/close 37033 (time 1746142745.67 total 300.97 last 116.15) - open/close 37981 (time 1746142755.68 total 310.97 last 94.74) - open/close 38930 (time 1746142765.69 total 320.98 last 94.85) - open/close 39956 (time 1746142775.69 total 330.98 last 102.58) - open/close 41127 (time 1746142785.70 total 340.99 last 116.99) - open/close 42323 (time 1746142795.70 total 350.99 last 119.59) - open/close 43354 (time 1746142805.70 total 361.00 last 103.05) - open/close 44584 (time 1746142815.71 total 371.00 last 122.96) - open/close 45722 (time 1746142825.71 total 381.00 last 113.77) - open/close 46885 (time 1746142835.71 total 391.01 last 116.23) - open/close 48048 (time 1746142845.72 total 401.01 last 116.27) - open/close 49323 (time 1746142855.72 total 411.01 last 127.46) - open/close 50000 (time 1746142862.43 total 417.72 last 100.90) - open/close 51176 (time 1746142872.43 total 427.73 last 117.57) - open/close 52309 (time 1746142882.44 total 437.73 last 113.21) - open/close 53520 (time 1746142892.45 total 447.75 last 120.95) - open/close 54811 (time 1746142902.46 total 457.75 last 129.02) - open/close 55981 (time 1746142912.46 total 467.76 last 116.93) - open/close 57240 (time 1746142922.47 total 477.76 last 125.88) - open/close 58222 (time 1746142932.48 total 487.77 last 98.11) - open/close 59229 (time 1746142942.48 total 497.78 last 100.64) - open/close 60000 (time 1746142949.48 total 504.78 last 110.09) - open/close 61161 (time 1746142959.49 total 514.78 last 116.06) - open/close 62220 (time 1746142969.49 total 524.79 last 105.85) - open/close 63345 (time 1746142979.50 total 534.79 last 112.46) - open/close 64415 (time 1746142989.50 total 544.79 last 106.95) total: 64768 open/close in 548.74 seconds: 118.03 ops/second - /unlink 1839 (time 1746143005.56 total 10.00 last 183.84) - /unlink 4066 (time 1746143015.56 total 20.01 last 222.56) - /unlink 5740 (time 1746143025.56 total 30.01 last 167.39) - /unlink 7462 (time 1746143035.56 total 40.01 last 172.19) - /unlink 9386 (time 1746143045.59 total 50.04 last 191.84) - /unlink 10000 (time 1746143048.99 total 53.44 last 180.60) - /unlink 11744 (time 1746143059.00 total 63.44 last 174.36) - /unlink 13463 (time 1746143069.00 total 73.44 last 171.90) - /unlink 15096 (time 1746143079.13 total 83.58 last 161.14) - /unlink 16548 (time 1746143089.14 total 93.59 last 145.06) - /unlink 18319 (time 1746143099.14 total 103.59 last 177.03) - /unlink 20000 (time 1746143108.64 total 113.09 last 176.99) - /unlink 21853 (time 1746143118.64 total 123.09 last 185.28) - /unlink 23507 (time 1746143128.65 total 133.09 last 165.35) - /unlink 25386 (time 1746143138.65 total 143.10 last 187.81) - /unlink 27145 (time 1746143148.66 total 153.10 last 175.80) - /unlink 28746 (time 1746143158.70 total 163.14 last 159.42) - /unlink 30000 (time 1746143166.96 total 171.41 last 151.75) - /unlink 31759 (time 1746143176.97 total 181.42 last 175.71) - /unlink 33532 (time 1746143186.97 total 191.42 last 177.29) - /unlink 35713 (time 1746143196.98 total 201.42 last 218.05) - /unlink 37555 (time 1746143206.98 total 211.42 last 184.13) - /unlink 39398 (time 1746143217.00 total 221.45 last 183.85) - /unlink 40000 (time 1746143220.30 total 224.74 last 182.72) - /unlink 41921 (time 1746143230.30 total 234.75 last 192.04) - /unlink 43829 (time 1746143240.30 total 244.75 last 190.75) - /unlink 45652 (time 1746143250.31 total 254.75 last 182.26) - /unlink 47575 (time 1746143260.33 total 264.77 last 191.90) - /unlink 49482 (time 1746143270.33 total 274.77 last 190.69) - /unlink 50000 (time 1746143273.35 total 277.79 last 171.52) - /unlink 51847 (time 1746143283.35 total 287.80 last 184.63) - /unlink 53810 (time 1746143293.35 total 297.80 last 196.29) - /unlink 55753 (time 1746143303.35 total 307.80 last 194.29) - /unlink 57497 (time 1746143313.35 total 317.80 last 174.37) - /unlink 59319 (time 1746143323.36 total 327.80 last 182.20) - /unlink 60000 (time 1746143326.88 total 331.32 last 193.28) - /unlink 61843 (time 1746143336.88 total 341.32 last 184.30) - /unlink 63921 (time 1746143346.89 total 351.33 last 207.61) total: 64768 /unlink in 355.76 seconds: 182.05 ops/second Waiting for MDT destroys to complete - open/close 1154 (time 1746143371.68 total 10.00 last 115.36) - open/close 2279 (time 1746143381.68 total 20.00 last 112.50) - open/close 3461 (time 1746143391.68 total 30.00 last 118.18) - open/close 4560 (time 1746143401.69 total 40.02 last 109.72) - open/close 5728 (time 1746143411.70 total 50.03 last 116.68) - open/close 6704 (time 1746143421.71 total 60.03 last 97.58) - open/close 7839 (time 1746143431.71 total 70.04 last 113.41) - open/close 8976 (time 1746143441.71 total 80.04 last 113.69) - open/close 10000 (time 1746143450.01 total 88.34 last 123.41) - open/close 11325 (time 1746143460.02 total 98.35 last 132.36) - open/close 12600 (time 1746143470.03 total 108.36 last 127.37) - open/close 13889 (time 1746143480.05 total 118.37 last 128.74) - open/close 15079 (time 1746143490.05 total 128.37 last 118.99) - open/close 16343 (time 1746143500.05 total 138.38 last 126.32) - open/close 17570 (time 1746143510.06 total 148.39 last 122.61) - open/close 18656 (time 1746143520.07 total 158.39 last 108.53) - open/close 19899 (time 1746143530.07 total 168.39 last 124.30) - open/close 21203 (time 1746143540.07 total 178.40 last 130.37) - open/close 22422 (time 1746143550.08 total 188.40 last 121.81) - open/close 23545 (time 1746143560.10 total 198.43 last 112.00) - open/close 24613 (time 1746143570.12 total 208.45 last 106.64) - open/close 25764 (time 1746143580.12 total 218.45 last 115.10) - open/close 26956 (time 1746143590.13 total 228.45 last 119.10) - open/close 28103 (time 1746143600.13 total 238.46 last 114.69) - open/close 29357 (time 1746143610.13 total 248.46 last 125.40) - open/close 30000 (time 1746143615.36 total 253.69 last 122.91) - open/close 31206 (time 1746143625.36 total 263.69 last 120.55) - open/close 32477 (time 1746143635.37 total 273.69 last 127.07) - open/close 33767 (time 1746143645.38 total 283.70 last 128.89) - open/close 34923 (time 1746143655.38 total 293.71 last 115.54) - open/close 36083 (time 1746143665.39 total 303.71 last 115.92) - open/close 37250 (time 1746143675.39 total 313.71 last 116.70) - open/close 38400 (time 1746143685.39 total 323.72 last 114.94) - open/close 39537 (time 1746143695.40 total 333.73 last 113.61) - open/close 40000 (time 1746143699.69 total 338.02 last 107.88) - open/close 41196 (time 1746143709.69 total 348.02 last 119.58) - open/close 42480 (time 1746143719.70 total 358.02 last 128.35) - open/close 43639 (time 1746143729.70 total 368.03 last 115.89) - open/close 44911 (time 1746143739.70 total 378.03 last 127.17) - open/close 46172 (time 1746143749.71 total 388.04 last 125.91) - open/close 47379 (time 1746143759.72 total 398.05 last 120.64) - open/close 48792 (time 1746143769.75 total 408.08 last 140.88) - open/close 50000 (time 1746143779.10 total 417.43 last 129.20) - open/close 51246 (time 1746143789.10 total 427.43 last 124.57) - open/close 52601 (time 1746143799.11 total 437.44 last 135.42) - open/close 53980 (time 1746143809.11 total 447.44 last 137.87) - open/close 55378 (time 1746143819.11 total 457.44 last 139.76) - open/close 56778 (time 1746143829.11 total 467.44 last 140.00) - open/close 58078 (time 1746143839.12 total 477.45 last 129.91) - open/close 59330 (time 1746143849.13 total 487.46 last 125.08) - open/close 60000 (time 1746143854.62 total 492.95 last 122.06) - open/close 61275 (time 1746143864.62 total 502.95 last 127.44) - open/close 62570 (time 1746143874.63 total 512.96 last 129.42) - open/close 63932 (time 1746143884.63 total 522.96 last 136.19) total: 64768 open/close in 528.74 seconds: 122.49 ops/second - /unlink 2239 (time 1746143902.05 total 10.01 last 223.76) - /unlink 4657 (time 1746143912.05 total 20.01 last 241.79) - /unlink 6640 (time 1746143922.05 total 30.01 last 198.30) - /unlink 8700 (time 1746143932.05 total 40.01 last 205.96) - /unlink 10000 (time 1746143938.60 total 46.56 last 198.45) - /unlink 12037 (time 1746143948.60 total 56.56 last 203.66) - /unlink 14112 (time 1746143958.60 total 66.56 last 207.48) - /unlink 16358 (time 1746143968.61 total 76.56 last 224.56) - /unlink 18213 (time 1746143978.61 total 86.57 last 185.44) - /unlink 20000 (time 1746143986.58 total 94.53 last 224.28) - /unlink 22457 (time 1746143996.58 total 104.54 last 245.63) - /unlink 24689 (time 1746144006.58 total 114.54 last 223.19) - /unlink 26890 (time 1746144016.58 total 124.54 last 220.01) - /unlink 28755 (time 1746144026.59 total 134.54 last 186.47) - /unlink 30000 (time 1746144032.52 total 140.48 last 209.77) - /unlink 31894 (time 1746144042.52 total 150.48 last 189.33) - /unlink 33834 (time 1746144052.53 total 160.48 last 193.96) - /unlink 35514 (time 1746144062.53 total 170.49 last 167.96) - /unlink 37268 (time 1746144072.53 total 180.49 last 175.34) - /unlink 39280 (time 1746144082.54 total 190.50 last 201.05) - /unlink 40000 (time 1746144086.20 total 194.16 last 196.53) - /unlink 41981 (time 1746144096.21 total 204.17 last 198.00) - /unlink 43922 (time 1746144106.21 total 214.17 last 194.05) - /unlink 45982 (time 1746144116.21 total 224.17 last 205.95) - /unlink 48409 (time 1746144126.21 total 234.17 last 242.69) - /unlink 50000 (time 1746144133.99 total 241.94 last 204.72) - /unlink 52204 (time 1746144143.99 total 251.95 last 220.35) - /unlink 54465 (time 1746144153.99 total 261.95 last 226.09) - /unlink 56624 (time 1746144164.00 total 271.96 last 215.64) - /unlink 58736 (time 1746144174.00 total 281.96 last 211.19) - /unlink 60000 (time 1746144180.16 total 288.12 last 205.18) - /unlink 62220 (time 1746144190.16 total 298.12 last 221.97) - /unlink 64593 (time 1746144200.16 total 308.12 last 237.26) total: 64768 /unlink in 308.71 seconds: 209.80 ops/second Waiting for MDT destroys to complete - open/close 1238 (time 1746144222.38 total 10.01 last 123.65) - open/close 2488 (time 1746144232.38 total 20.02 last 124.95) - open/close 3789 (time 1746144242.39 total 30.02 last 130.04) - open/close 5298 (time 1746144252.40 total 40.03 last 150.77) - open/close 6665 (time 1746144262.40 total 50.03 last 136.66) - open/close 7935 (time 1746144272.40 total 60.03 last 127.00) - open/close 9280 (time 1746144282.40 total 70.04 last 134.44) - open/close 10000 (time 1746144287.50 total 75.13 last 141.41) - open/close 11295 (time 1746144297.50 total 85.14 last 129.40) - open/close 12432 (time 1746144307.51 total 95.14 last 113.65) - open/close 13641 (time 1746144317.51 total 105.14 last 120.90) - open/close 15029 (time 1746144327.51 total 115.14 last 138.77) - open/close 16393 (time 1746144337.51 total 125.14 last 136.40) - open/close 17826 (time 1746144347.51 total 135.15 last 143.25) - open/close 19291 (time 1746144357.52 total 145.15 last 146.48) - open/close 20000 (time 1746144362.24 total 149.87 last 150.24) - open/close 21379 (time 1746144372.24 total 159.87 last 137.83) - open/close 22657 (time 1746144382.25 total 169.88 last 127.69) - open/close 23868 (time 1746144392.25 total 179.89 last 121.03) - open/close 25209 (time 1746144402.26 total 189.89 last 134.01) - open/close 26339 (time 1746144412.26 total 199.90 last 112.96) - open/close 27642 (time 1746144422.27 total 209.90 last 130.29) - open/close 28959 (time 1746144432.27 total 219.91 last 131.58) - open/close 30000 (time 1746144441.76 total 229.39 last 109.79) - open/close 31207 (time 1746144451.76 total 239.39 last 120.68) - open/close 32466 (time 1746144461.76 total 249.40 last 125.83) - open/close 33855 (time 1746144471.77 total 259.40 last 138.81) - open/close 35517 (time 1746144481.77 total 269.40 last 166.16) - open/close 37409 (time 1746144491.78 total 279.41 last 189.14) - open/close 39387 (time 1746144501.78 total 289.41 last 197.75) - open/close 40000 (time 1746144504.80 total 292.43 last 202.74) - open/close 41777 (time 1746144514.80 total 302.43 last 177.70) - open/close 43187 (time 1746144524.81 total 312.44 last 140.94) - open/close 44716 (time 1746144534.81 total 322.45 last 152.79) - open/close 46328 (time 1746144544.82 total 332.45 last 161.06) - open/close 47847 (time 1746144554.82 total 342.46 last 151.85) - open/close 49378 (time 1746144564.83 total 352.46 last 153.07) - open/close 50000 (time 1746144569.08 total 356.72 last 146.12) - open/close 51427 (time 1746144579.08 total 366.72 last 142.68) - open/close 52790 (time 1746144589.09 total 376.72 last 136.26) - open/close 54128 (time 1746144599.09 total 386.73 last 133.73) - open/close 55566 (time 1746144609.10 total 396.73 last 143.74) - open/close 56957 (time 1746144619.10 total 406.73 last 139.04) - open/close 58479 (time 1746144629.11 total 416.74 last 152.13) - open/close 60000 (time 1746144638.51 total 426.14 last 161.75) - open/close 61495 (time 1746144648.51 total 436.14 last 149.47) - open/close 63304 (time 1746144658.52 total 446.15 last 180.82) - open/close 64543 (time 1746144668.52 total 456.15 last 123.88) total: 64768 open/close in 458.09 seconds: 141.39 ops/second - /unlink 2122 (time 1746144682.25 total 10.00 last 212.14) - /unlink 4378 (time 1746144692.26 total 20.01 last 225.43) - /unlink 6687 (time 1746144702.26 total 30.01 last 230.83) - /unlink 8964 (time 1746144712.26 total 40.01 last 227.69) - /unlink 10000 (time 1746144717.07 total 44.83 last 215.12) - /unlink 12402 (time 1746144727.08 total 54.83 last 240.16) - /unlink 14974 (time 1746144737.08 total 64.83 last 257.11) - /unlink 17844 (time 1746144747.08 total 74.84 last 286.93) - /unlink 20000 (time 1746144756.24 total 84.00 last 235.37) - /unlink 21981 (time 1746144766.25 total 94.00 last 198.04) - /unlink 24317 (time 1746144776.25 total 104.00 last 233.60) - /unlink 26745 (time 1746144786.25 total 114.00 last 242.78) - /unlink 29326 (time 1746144796.25 total 124.00 last 258.09) - /unlink 30000 (time 1746144799.07 total 126.82 last 238.81) - /unlink 32120 (time 1746144809.08 total 136.83 last 211.85) - /unlink 34075 (time 1746144819.08 total 146.83 last 195.48) - /unlink 36374 (time 1746144829.08 total 156.83 last 229.90) - /unlink 38813 (time 1746144839.08 total 166.83 last 243.89) - /unlink 40000 (time 1746144843.84 total 171.59 last 249.40) - /unlink 42443 (time 1746144853.84 total 181.59 last 244.30) - /unlink 44716 (time 1746144863.84 total 191.59 last 227.25) - /unlink 46587 (time 1746144873.85 total 201.60 last 186.98) - /unlink 48648 (time 1746144883.85 total 211.60 last 206.07) - /unlink 50000 (time 1746144889.72 total 217.47 last 230.29) - /unlink 51982 (time 1746144899.72 total 227.47 last 198.18) - /unlink 54520 (time 1746144909.72 total 237.48 last 253.68) - /unlink 56713 (time 1746144919.73 total 247.48 last 219.18) - /unlink 58909 (time 1746144929.73 total 257.49 last 219.54) - /unlink 60000 (time 1746144934.70 total 262.46 last 219.38) - /unlink 62234 (time 1746144944.71 total 272.46 last 223.38) total: 64768 /unlink in 282.46 seconds: 229.30 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg224-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg224-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg224-server stop mds service on oleg224-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg224-server unloading modules via unload_modules_local on: 'oleg224-server' oleg224-server: oleg224-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-client: ssh exited with exit code 2 pdsh@oleg224-client: oleg224-server: ssh exited with exit code 2