== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 23:23:50 (1746156230) Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg331-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg331-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg331-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1248 (time 1746156297.85 total 10.00 last 124.76) - open/close 2522 (time 1746156307.86 total 20.01 last 127.35) - open/close 3594 (time 1746156317.86 total 30.01 last 107.15) - open/close 4741 (time 1746156327.87 total 40.02 last 114.64) - open/close 5950 (time 1746156337.87 total 50.02 last 120.88) - open/close 7091 (time 1746156347.87 total 60.02 last 114.05) - open/close 8609 (time 1746156357.88 total 70.03 last 151.74) - open/close 9906 (time 1746156367.88 total 80.03 last 129.66) - open/close 11241 (time 1746156377.88 total 90.04 last 133.42) - open/close 12609 (time 1746156387.89 total 100.04 last 136.79) - open/close 13955 (time 1746156397.89 total 110.04 last 134.53) - open/close 15141 (time 1746156407.90 total 120.05 last 118.56) - open/close 16412 (time 1746156417.90 total 130.05 last 127.08) - open/close 17705 (time 1746156427.90 total 140.05 last 129.28) - open/close 18976 (time 1746156437.91 total 150.06 last 127.00) - open/close 20000 (time 1746156445.72 total 157.87 last 131.01) - open/close 21346 (time 1746156455.72 total 167.88 last 134.59) - open/close 22770 (time 1746156465.73 total 177.88 last 142.33) - open/close 24027 (time 1746156475.74 total 187.89 last 125.62) - open/close 25389 (time 1746156485.75 total 197.90 last 136.03) - open/close 26645 (time 1746156495.75 total 207.90 last 125.56) - open/close 28155 (time 1746156505.76 total 217.91 last 150.93) - open/close 29902 (time 1746156515.76 total 227.92 last 174.56) - open/close 31453 (time 1746156525.77 total 237.92 last 155.05) - open/close 32927 (time 1746156535.77 total 247.92 last 147.39) - open/close 34901 (time 1746156545.77 total 257.92 last 197.39) - open/close 36360 (time 1746156555.78 total 267.93 last 145.79) - open/close 37673 (time 1746156565.78 total 277.93 last 131.27) - open/close 39470 (time 1746156575.78 total 287.93 last 179.61) - open/close 40000 (time 1746156579.14 total 291.29 last 158.08) - open/close 41724 (time 1746156589.14 total 301.29 last 172.27) - open/close 43253 (time 1746156599.14 total 311.30 last 152.89) - open/close 44688 (time 1746156609.15 total 321.31 last 143.35) - open/close 45965 (time 1746156619.16 total 331.31 last 127.65) - open/close 47360 (time 1746156629.16 total 341.31 last 139.44) - open/close 48880 (time 1746156639.16 total 351.31 last 151.98) - open/close 50000 (time 1746156647.60 total 359.75 last 132.70) - open/close 51335 (time 1746156657.62 total 369.77 last 133.30) - open/close 52460 (time 1746156667.63 total 379.78 last 112.38) - open/close 53584 (time 1746156677.63 total 389.79 last 112.34) - open/close 54734 (time 1746156687.64 total 399.79 last 114.98) - open/close 55905 (time 1746156697.64 total 409.79 last 117.02) - open/close 57257 (time 1746156707.65 total 419.80 last 135.14) - open/close 58442 (time 1746156717.65 total 429.80 last 118.44) - open/close 59712 (time 1746156727.65 total 439.80 last 127.00) - open/close 60000 (time 1746156730.10 total 442.26 last 117.37) - open/close 61290 (time 1746156740.11 total 452.26 last 128.95) - open/close 62645 (time 1746156750.11 total 462.26 last 135.48) - open/close 63916 (time 1746156760.12 total 472.27 last 126.98) total: 64768 open/close in 478.79 seconds: 135.28 ops/second - /unlink 2268 (time 1746156778.36 total 10.00 last 226.71) - /unlink 4561 (time 1746156788.36 total 20.01 last 229.27) - /unlink 6922 (time 1746156798.37 total 30.01 last 236.02) - /unlink 9276 (time 1746156808.37 total 40.01 last 235.32) - /unlink 10000 (time 1746156811.06 total 42.71 last 268.71) - /unlink 12391 (time 1746156821.07 total 52.71 last 239.02) - /unlink 14649 (time 1746156831.07 total 62.71 last 225.74) - /unlink 16688 (time 1746156841.07 total 72.72 last 203.82) - /unlink 18705 (time 1746156851.08 total 82.72 last 201.65) - /unlink 20000 (time 1746156856.90 total 88.55 last 222.27) - /unlink 21939 (time 1746156866.91 total 98.55 last 193.76) - /unlink 23871 (time 1746156876.92 total 108.56 last 193.06) - /unlink 25917 (time 1746156886.93 total 118.57 last 204.41) - /unlink 28112 (time 1746156896.93 total 128.57 last 219.40) - /unlink 30000 (time 1746156905.76 total 137.40 last 213.84) - /unlink 31804 (time 1746156915.76 total 147.40 last 180.37) - /unlink 33895 (time 1746156925.77 total 157.41 last 209.06) - /unlink 35949 (time 1746156935.77 total 167.41 last 205.26) - /unlink 37904 (time 1746156945.77 total 177.41 last 195.50) - /unlink 40000 (time 1746156954.73 total 186.37 last 233.93) - /unlink 42177 (time 1746156964.74 total 196.38 last 217.63) - /unlink 44086 (time 1746156974.74 total 206.38 last 190.86) - /unlink 46449 (time 1746156984.74 total 216.38 last 236.26) - /unlink 48692 (time 1746156994.74 total 226.38 last 224.28) - /unlink 50000 (time 1746157000.83 total 232.48 last 214.63) - /unlink 52189 (time 1746157010.84 total 242.48 last 218.73) - /unlink 54148 (time 1746157020.85 total 252.49 last 195.68) - /unlink 56340 (time 1746157030.86 total 262.50 last 219.15) - /unlink 58353 (time 1746157040.86 total 272.50 last 201.28) - /unlink 60000 (time 1746157048.56 total 280.20 last 213.86) - /unlink 62549 (time 1746157058.56 total 290.20 last 254.90) - /unlink 64648 (time 1746157068.56 total 300.20 last 209.89) total: 64768 /unlink in 300.98 seconds: 215.19 ops/second Waiting for MDT destroys to complete - open/close 1226 (time 1746157088.91 total 10.00 last 122.55) - open/close 2455 (time 1746157098.91 total 20.01 last 122.79) - open/close 3786 (time 1746157108.92 total 30.02 last 133.04) - open/close 5030 (time 1746157118.93 total 40.03 last 124.26) - open/close 6240 (time 1746157128.94 total 50.04 last 120.88) - open/close 7380 (time 1746157138.94 total 60.04 last 113.96) - open/close 8690 (time 1746157148.94 total 70.04 last 130.99) - open/close 9885 (time 1746157158.95 total 80.05 last 119.44) - open/close 11159 (time 1746157168.96 total 90.06 last 127.29) - open/close 12413 (time 1746157178.96 total 100.06 last 125.33) - open/close 13639 (time 1746157188.97 total 110.07 last 122.54) - open/close 14962 (time 1746157198.97 total 120.07 last 132.24) - open/close 16499 (time 1746157208.98 total 130.08 last 153.62) - open/close 17921 (time 1746157218.98 total 140.08 last 142.18) - open/close 19382 (time 1746157228.98 total 150.08 last 146.05) - open/close 20000 (time 1746157233.07 total 154.17 last 151.21) - open/close 21547 (time 1746157243.07 total 164.17 last 154.67) - open/close 22960 (time 1746157253.08 total 174.17 last 141.26) - open/close 24220 (time 1746157263.08 total 184.18 last 125.97) - open/close 25410 (time 1746157273.09 total 194.18 last 118.91) - open/close 26874 (time 1746157283.09 total 204.19 last 146.37) - open/close 28369 (time 1746157293.09 total 214.19 last 149.45) - open/close 29934 (time 1746157303.10 total 224.19 last 156.42) - open/close 31326 (time 1746157313.10 total 234.20 last 139.17) - open/close 32594 (time 1746157323.11 total 244.20 last 126.72) - open/close 33881 (time 1746157333.11 total 254.21 last 128.64) - open/close 35345 (time 1746157343.12 total 264.21 last 146.32) - open/close 36750 (time 1746157353.12 total 274.22 last 140.41) - open/close 38133 (time 1746157363.13 total 284.23 last 138.17) - open/close 39557 (time 1746157373.13 total 294.23 last 142.36) - open/close 40000 (time 1746157376.55 total 297.64 last 129.83) - open/close 41554 (time 1746157386.55 total 307.65 last 155.36) - open/close 43059 (time 1746157396.55 total 317.65 last 150.49) - open/close 44750 (time 1746157406.56 total 327.66 last 168.97) - open/close 46261 (time 1746157416.56 total 337.66 last 151.10) - open/close 47812 (time 1746157426.56 total 347.66 last 155.01) - open/close 49557 (time 1746157436.57 total 357.67 last 174.43) - open/close 50000 (time 1746157439.42 total 360.52 last 155.33) - open/close 51739 (time 1746157449.42 total 370.52 last 173.81) - open/close 53701 (time 1746157459.43 total 380.52 last 196.17) - open/close 55345 (time 1746157469.44 total 390.53 last 164.25) - open/close 57141 (time 1746157479.44 total 400.54 last 179.51) - open/close 58622 (time 1746157489.44 total 410.54 last 148.07) - open/close 60000 (time 1746157496.85 total 417.95 last 185.94) - open/close 61847 (time 1746157506.85 total 427.95 last 184.67) - open/close 63649 (time 1746157516.86 total 437.96 last 180.07) total: 64768 open/close in 443.92 seconds: 145.90 ops/second - /unlink 2980 (time 1746157533.99 total 10.00 last 297.91) - /unlink 5487 (time 1746157543.99 total 20.00 last 250.67) - /unlink 8169 (time 1746157554.00 total 30.01 last 268.17) - /unlink 10000 (time 1746157560.69 total 36.70 last 273.33) - /unlink 12832 (time 1746157570.70 total 46.71 last 283.10) - /unlink 15883 (time 1746157580.70 total 56.71 last 305.02) - /unlink 18944 (time 1746157590.70 total 66.71 last 306.09) - /unlink 20000 (time 1746157593.83 total 69.84 last 337.89) - /unlink 22626 (time 1746157603.83 total 79.84 last 262.56) - /unlink 25351 (time 1746157613.83 total 89.84 last 272.49) - /unlink 27627 (time 1746157623.83 total 99.84 last 227.56) - /unlink 29953 (time 1746157633.83 total 109.84 last 232.57) - /unlink 32393 (time 1746157643.83 total 119.84 last 243.97) - /unlink 35301 (time 1746157653.83 total 129.84 last 290.77) - /unlink 38289 (time 1746157663.84 total 139.85 last 298.63) - /unlink 40000 (time 1746157671.06 total 147.07 last 237.11) - /unlink 42920 (time 1746157681.06 total 157.07 last 291.96) - /unlink 45547 (time 1746157691.06 total 167.07 last 262.64) - /unlink 48222 (time 1746157701.06 total 177.07 last 267.38) - /unlink 50000 (time 1746157707.55 total 183.56 last 274.20) - /unlink 52885 (time 1746157717.55 total 193.56 last 288.47) - /unlink 55462 (time 1746157727.55 total 203.56 last 257.57) - /unlink 58043 (time 1746157737.55 total 213.56 last 258.08) - /unlink 60000 (time 1746157745.66 total 221.67 last 241.47) - /unlink 62368 (time 1746157755.66 total 231.67 last 236.72) - /unlink 64517 (time 1746157765.66 total 241.67 last 214.89) total: 64768 /unlink in 242.84 seconds: 266.71 ops/second Waiting for MDT destroys to complete - open/close 1396 (time 1746157785.05 total 10.01 last 139.50) - open/close 2793 (time 1746157795.05 total 20.01 last 139.68) - open/close 4249 (time 1746157805.05 total 30.01 last 145.52) - open/close 5468 (time 1746157815.06 total 40.02 last 121.85) - open/close 6834 (time 1746157825.06 total 50.02 last 136.59) - open/close 8096 (time 1746157835.07 total 60.03 last 126.08) - open/close 9459 (time 1746157845.07 total 70.03 last 136.24) - open/close 10000 (time 1746157849.27 total 74.23 last 128.97) - open/close 11291 (time 1746157859.27 total 84.23 last 129.04) - open/close 12614 (time 1746157869.28 total 94.24 last 132.26) - open/close 14000 (time 1746157879.28 total 104.24 last 138.59) - open/close 15471 (time 1746157889.28 total 114.24 last 147.01) - open/close 16842 (time 1746157899.28 total 124.24 last 137.09) - open/close 18193 (time 1746157909.29 total 134.25 last 135.04) - open/close 19613 (time 1746157919.29 total 144.25 last 141.99) - open/close 20000 (time 1746157921.75 total 146.71 last 157.34) - open/close 21418 (time 1746157931.75 total 156.71 last 141.77) - open/close 22869 (time 1746157941.75 total 166.71 last 145.07) - open/close 24366 (time 1746157951.75 total 176.71 last 149.68) - open/close 25670 (time 1746157961.76 total 186.72 last 130.34) - open/close 26984 (time 1746157971.76 total 196.72 last 131.39) - open/close 28351 (time 1746157981.77 total 206.73 last 136.58) - open/close 29728 (time 1746157991.77 total 216.73 last 137.62) - open/close 31107 (time 1746158001.78 total 226.74 last 137.80) - open/close 32582 (time 1746158011.78 total 236.74 last 147.46) - open/close 34166 (time 1746158021.79 total 246.75 last 158.32) - open/close 35506 (time 1746158031.79 total 256.75 last 134.00) - open/close 36951 (time 1746158041.80 total 266.76 last 144.40) - open/close 38427 (time 1746158051.80 total 276.76 last 147.56) - open/close 39790 (time 1746158061.80 total 286.76 last 136.21) - open/close 41218 (time 1746158071.82 total 296.78 last 142.63) - open/close 42419 (time 1746158081.82 total 306.78 last 120.05) - open/close 43903 (time 1746158091.83 total 316.79 last 148.34) - open/close 45716 (time 1746158101.83 total 326.79 last 181.17) - open/close 48115 (time 1746158111.83 total 336.79 last 239.84) - open/close 49925 (time 1746158121.84 total 346.80 last 180.82) - open/close 51472 (time 1746158131.85 total 356.81 last 154.57) - open/close 52986 (time 1746158141.86 total 366.82 last 151.34) - open/close 54242 (time 1746158151.86 total 376.82 last 125.57) - open/close 55426 (time 1746158161.86 total 386.82 last 118.35) - open/close 56719 (time 1746158171.87 total 396.83 last 129.24) - open/close 58226 (time 1746158181.87 total 406.83 last 150.70) - open/close 59743 (time 1746158191.87 total 416.83 last 151.69) - open/close 61690 (time 1746158201.87 total 426.83 last 194.64) - open/close 63332 (time 1746158211.88 total 436.84 last 164.13) total: 64768 open/close in 446.19 seconds: 145.16 ops/second - /unlink 2986 (time 1746158232.71 total 10.00 last 298.55) - /unlink 6081 (time 1746158242.71 total 20.00 last 309.47) - /unlink 9115 (time 1746158252.72 total 30.00 last 303.37) - /unlink 10000 (time 1746158256.56 total 33.85 last 230.02) - /unlink 13017 (time 1746158266.57 total 43.85 last 301.64) - /unlink 15869 (time 1746158276.57 total 53.85 last 285.20) - /unlink 18203 (time 1746158286.57 total 63.86 last 233.27) - /unlink 20000 (time 1746158292.64 total 69.93 last 296.14) - /unlink 23462 (time 1746158302.64 total 79.93 last 346.17) - /unlink 28256 (time 1746158312.64 total 89.93 last 479.32) - /unlink 30000 (time 1746158316.44 total 93.73 last 458.60) - /unlink 33968 (time 1746158326.45 total 103.73 last 396.75) - /unlink 36927 (time 1746158336.45 total 113.74 last 295.81) - /unlink 39837 (time 1746158346.45 total 123.74 last 291.00) - /unlink 42586 (time 1746158356.45 total 133.74 last 274.79) - /unlink 45517 (time 1746158366.45 total 143.74 last 293.08) - /unlink 50000 (time 1746158376.39 total 153.68 last 451.24) - /unlink 53506 (time 1746158386.39 total 163.68 last 350.56) - /unlink 57341 (time 1746158396.39 total 173.68 last 383.45) - /unlink 60000 (time 1746158406.31 total 183.59 last 268.19) - /unlink 63550 (time 1746158416.31 total 193.60 last 354.88) total: 64768 /unlink in 197.46 seconds: 328.01 ops/second Waiting for MDT destroys to complete - open/close 1367 (time 1746158442.03 total 10.00 last 136.68) - open/close 3145 (time 1746158452.04 total 20.01 last 177.71) - open/close 4771 (time 1746158462.04 total 30.01 last 162.59) - open/close 6484 (time 1746158472.04 total 40.01 last 171.29) - open/close 8119 (time 1746158482.05 total 50.01 last 163.44) - open/close 9742 (time 1746158492.05 total 60.02 last 162.18) - open/close 11433 (time 1746158502.06 total 70.03 last 168.93) - open/close 13723 (time 1746158512.06 total 80.03 last 228.97) - open/close 15690 (time 1746158522.07 total 90.03 last 196.65) - open/close 17675 (time 1746158532.07 total 100.04 last 198.37) - open/close 19822 (time 1746158542.08 total 110.04 last 214.59) - open/close 21878 (time 1746158552.08 total 120.05 last 205.58) - open/close 23433 (time 1746158562.08 total 130.05 last 155.47) - open/close 25384 (time 1746158572.08 total 140.05 last 195.07) - open/close 26902 (time 1746158582.09 total 150.05 last 151.72) - open/close 28339 (time 1746158592.09 total 160.06 last 143.65) - open/close 29850 (time 1746158602.10 total 170.06 last 151.03) - open/close 31362 (time 1746158612.10 total 180.07 last 151.16) - open/close 32884 (time 1746158622.10 total 190.07 last 152.19) - open/close 34292 (time 1746158632.10 total 200.07 last 140.76) - open/close 35552 (time 1746158642.11 total 210.08 last 125.91) - open/close 37021 (time 1746158652.11 total 220.08 last 146.86) - open/close 38550 (time 1746158662.12 total 230.09 last 152.80) - open/close 40000 (time 1746158672.08 total 240.05 last 145.54) - open/close 41407 (time 1746158682.09 total 250.05 last 140.65) - open/close 42645 (time 1746158692.09 total 260.06 last 123.74) - open/close 43880 (time 1746158702.09 total 270.06 last 123.47) - open/close 45157 (time 1746158712.10 total 280.07 last 127.62) - open/close 46513 (time 1746158722.10 total 290.07 last 135.52) - open/close 48009 (time 1746158732.11 total 300.08 last 149.48) - open/close 49441 (time 1746158742.11 total 310.08 last 143.19) - open/close 50000 (time 1746158746.55 total 314.52 last 125.89) - open/close 51535 (time 1746158756.56 total 324.52 last 153.48) - open/close 53479 (time 1746158766.56 total 334.53 last 194.32) - open/close 55172 (time 1746158776.57 total 344.53 last 169.18) - open/close 56544 (time 1746158786.57 total 354.54 last 137.14) - open/close 57976 (time 1746158796.58 total 364.54 last 143.10) - open/close 59360 (time 1746158806.58 total 374.55 last 138.37) - open/close 60000 (time 1746158811.02 total 378.99 last 144.01) - open/close 61910 (time 1746158821.02 total 388.99 last 190.98) - open/close 63545 (time 1746158831.03 total 399.00 last 163.39) total: 64768 open/close in 408.68 seconds: 158.48 ops/second - /unlink 2700 (time 1746158852.11 total 10.00 last 269.91) - /unlink 5686 (time 1746158862.11 total 20.00 last 298.57) - /unlink 8074 (time 1746158872.12 total 30.01 last 238.68) - /unlink 10000 (time 1746158881.81 total 39.71 last 198.64) - /unlink 12286 (time 1746158891.81 total 49.71 last 228.60) - /unlink 14499 (time 1746158901.81 total 59.71 last 221.26) - /unlink 16476 (time 1746158911.82 total 69.71 last 197.62) - /unlink 18354 (time 1746158921.82 total 79.71 last 187.76) - /unlink 20000 (time 1746158927.95 total 85.84 last 268.70) - /unlink 22197 (time 1746158937.95 total 95.84 last 219.67) - /unlink 24169 (time 1746158947.95 total 105.84 last 197.16) - /unlink 26494 (time 1746158957.96 total 115.85 last 232.21) - /unlink 28462 (time 1746158967.96 total 125.86 last 196.74) - /unlink 30000 (time 1746158976.50 total 134.39 last 180.20) - /unlink 31911 (time 1746158986.50 total 144.39 last 191.06) - /unlink 33778 (time 1746158996.50 total 154.40 last 186.68) - /unlink 35592 (time 1746159006.50 total 164.40 last 181.39) - /unlink 37749 (time 1746159016.51 total 174.40 last 215.66) - /unlink 39776 (time 1746159026.51 total 184.40 last 202.66) - /unlink 41667 (time 1746159036.51 total 194.41 last 188.97) - /unlink 43691 (time 1746159046.52 total 204.41 last 202.30) - /unlink 45630 (time 1746159056.53 total 214.42 last 193.76) - /unlink 47537 (time 1746159066.53 total 224.42 last 190.69) - /unlink 49493 (time 1746159076.53 total 234.42 last 195.56) - /unlink 50000 (time 1746159078.83 total 236.73 last 219.89) - /unlink 51916 (time 1746159088.84 total 246.73 last 191.56) - /unlink 53902 (time 1746159098.84 total 256.74 last 198.47) - /unlink 55940 (time 1746159108.85 total 266.74 last 203.75) - /unlink 57991 (time 1746159118.85 total 276.74 last 205.06) - /unlink 60000 (time 1746159128.82 total 286.71 last 201.53) - /unlink 62260 (time 1746159138.82 total 296.71 last 225.96) - /unlink 64323 (time 1746159148.82 total 306.71 last 206.23) total: 64768 /unlink in 308.56 seconds: 209.91 ops/second Waiting for MDT destroys to complete - open/close 1332 (time 1746159173.04 total 10.00 last 133.17) - open/close 2707 (time 1746159183.05 total 20.01 last 137.45) - open/close 3980 (time 1746159193.05 total 30.01 last 127.28) - open/close 5247 (time 1746159203.05 total 40.01 last 126.69) - open/close 6454 (time 1746159213.06 total 50.01 last 120.61) - open/close 7742 (time 1746159223.06 total 60.02 last 128.77) - open/close 8838 (time 1746159233.06 total 70.02 last 109.57) - open/close 10000 (time 1746159242.75 total 79.71 last 119.96) - open/close 11028 (time 1746159252.75 total 89.71 last 102.78) - open/close 12323 (time 1746159262.75 total 99.71 last 129.48) - open/close 13683 (time 1746159272.76 total 109.71 last 135.95) - open/close 15073 (time 1746159282.79 total 119.75 last 138.56) - open/close 16272 (time 1746159292.79 total 129.75 last 119.87) - open/close 17633 (time 1746159302.80 total 139.76 last 136.01) - open/close 18963 (time 1746159312.80 total 149.76 last 132.92) - open/close 20000 (time 1746159320.63 total 157.59 last 132.43) - open/close 21321 (time 1746159330.64 total 167.59 last 132.06) - open/close 22607 (time 1746159340.64 total 177.60 last 128.52) - open/close 23811 (time 1746159350.65 total 187.61 last 120.30) - open/close 25098 (time 1746159360.66 total 197.61 last 128.64) - open/close 26146 (time 1746159370.66 total 207.62 last 104.72) - open/close 27331 (time 1746159380.67 total 217.62 last 118.46) - open/close 28525 (time 1746159390.67 total 227.63 last 119.33) - open/close 29775 (time 1746159400.67 total 237.63 last 124.97) - open/close 30000 (time 1746159402.72 total 239.67 last 110.20) - open/close 31168 (time 1746159412.72 total 249.68 last 116.79) - open/close 32281 (time 1746159422.72 total 259.68 last 111.28) - open/close 33493 (time 1746159432.72 total 269.68 last 121.17) - open/close 34658 (time 1746159442.73 total 279.68 last 116.45) - open/close 35757 (time 1746159452.73 total 289.69 last 109.83) - open/close 36921 (time 1746159462.74 total 299.69 last 116.35) - open/close 38160 (time 1746159472.74 total 309.70 last 123.83) - open/close 39324 (time 1746159482.75 total 319.70 last 116.37) - open/close 40000 (time 1746159488.74 total 325.70 last 112.78) - open/close 41091 (time 1746159498.74 total 335.70 last 109.05) - open/close 42207 (time 1746159508.75 total 345.71 last 111.56) - open/close 43269 (time 1746159518.75 total 355.71 last 106.18) - open/close 44358 (time 1746159528.76 total 365.72 last 108.78) - open/close 45458 (time 1746159538.76 total 375.72 last 109.96) - open/close 46740 (time 1746159548.78 total 385.73 last 128.05) - open/close 48004 (time 1746159558.78 total 395.74 last 126.36) - open/close 49240 (time 1746159568.78 total 405.74 last 123.54) - open/close 50000 (time 1746159574.67 total 411.63 last 129.06) - open/close 51270 (time 1746159584.68 total 421.64 last 126.92) - open/close 52626 (time 1746159594.69 total 431.65 last 135.47) - open/close 53924 (time 1746159604.69 total 441.65 last 129.78) - open/close 55334 (time 1746159614.69 total 451.65 last 140.93) - open/close 56688 (time 1746159624.70 total 461.65 last 135.39) - open/close 58324 (time 1746159634.70 total 471.66 last 163.57) - open/close 59812 (time 1746159644.70 total 481.66 last 148.77) - open/close 61193 (time 1746159654.70 total 491.66 last 138.05) - open/close 62742 (time 1746159664.70 total 501.66 last 154.87) - open/close 64227 (time 1746159674.71 total 511.66 last 148.46) total: 64768 open/close in 515.46 seconds: 125.65 ops/second - /unlink 2312 (time 1746159690.09 total 10.00 last 231.20) - /unlink 4637 (time 1746159700.09 total 20.00 last 232.49) - /unlink 6787 (time 1746159710.10 total 30.00 last 214.97) - /unlink 8873 (time 1746159720.10 total 40.01 last 208.52) - /unlink 10000 (time 1746159725.58 total 45.49 last 205.51) - /unlink 12216 (time 1746159735.58 total 55.49 last 221.59) - /unlink 14278 (time 1746159745.58 total 65.49 last 206.19) - /unlink 16406 (time 1746159755.58 total 75.49 last 212.79) - /unlink 18255 (time 1746159765.58 total 85.49 last 184.89) - /unlink 20000 (time 1746159773.26 total 93.17 last 227.37) - /unlink 22016 (time 1746159783.26 total 103.17 last 201.59) - /unlink 23983 (time 1746159793.26 total 113.17 last 196.69) - /unlink 25999 (time 1746159803.26 total 123.17 last 201.56) - /unlink 28226 (time 1746159813.26 total 133.17 last 222.70) - /unlink 30000 (time 1746159821.85 total 141.75 last 206.65) - /unlink 31954 (time 1746159831.85 total 151.75 last 195.39) - /unlink 34007 (time 1746159841.85 total 161.76 last 205.26) - /unlink 36010 (time 1746159851.89 total 171.80 last 199.52) - /unlink 38082 (time 1746159861.89 total 181.80 last 207.17) - /unlink 40000 (time 1746159871.20 total 191.11 last 206.01) - /unlink 41934 (time 1746159881.23 total 201.14 last 192.78) - /unlink 43970 (time 1746159891.23 total 211.14 last 203.58) - /unlink 46060 (time 1746159901.23 total 221.14 last 209.00) - /unlink 48490 (time 1746159911.24 total 231.14 last 242.93) - /unlink 50000 (time 1746159917.18 total 237.08 last 254.15) - /unlink 52779 (time 1746159927.18 total 247.09 last 277.85) - /unlink 55267 (time 1746159937.18 total 257.09 last 248.80) - /unlink 57793 (time 1746159947.18 total 267.09 last 252.57) - /unlink 60000 (time 1746159956.08 total 275.98 last 248.04) - /unlink 62481 (time 1746159966.08 total 285.99 last 248.06) total: 64768 /unlink in 295.34 seconds: 219.30 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2