-----============= acceptance-small: conf-sanity ============----- Sun Jun 22 07:14:47 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg104-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg104-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg104-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg104-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg104-server oleg104-server: oleg104-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg104-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg104-server: oleg104-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg104-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 07:16:06 (1750590966) Stopping clients: oleg104-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg104-client.virtnet /mnt/lustre2 (opts:-f) oleg104-server: oleg104-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg104-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg104-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg104-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg104-server: oleg104-server.virtnet: executing set_default_debug -1 all pdsh@oleg104-client: oleg104-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg104-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg104-client.virtnet: -o user_xattr,flock 192.168.201.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1226 (time 1750591023.77 total 10.00 last 122.55) - open/close 2472 (time 1750591033.77 total 20.00 last 124.59) - open/close 3775 (time 1750591043.78 total 30.01 last 130.29) - open/close 5073 (time 1750591053.78 total 40.01 last 129.76) - open/close 6294 (time 1750591063.78 total 50.01 last 122.02) - open/close 7360 (time 1750591073.79 total 60.02 last 106.51) - open/close 8514 (time 1750591083.80 total 70.03 last 115.33) - open/close 9896 (time 1750591093.80 total 80.03 last 138.12) - open/close 11117 (time 1750591103.81 total 90.04 last 122.05) - open/close 12379 (time 1750591113.81 total 100.04 last 126.12) - open/close 13538 (time 1750591123.82 total 110.05 last 115.81) - open/close 14836 (time 1750591133.83 total 120.06 last 129.69) - open/close 16163 (time 1750591143.83 total 130.06 last 132.70) - open/close 17347 (time 1750591153.83 total 140.06 last 118.37) - open/close 18540 (time 1750591163.84 total 150.07 last 119.23) - open/close 19682 (time 1750591173.84 total 160.07 last 114.18) - open/close 20000 (time 1750591176.63 total 162.86 last 114.09) - open/close 21151 (time 1750591186.64 total 172.87 last 115.01) - open/close 22248 (time 1750591196.64 total 182.87 last 109.66) - open/close 23372 (time 1750591206.65 total 192.88 last 112.32) - open/close 24607 (time 1750591216.65 total 202.88 last 123.43) - open/close 25971 (time 1750591226.65 total 212.88 last 136.39) - open/close 27290 (time 1750591236.66 total 222.89 last 131.84) - open/close 28402 (time 1750591246.66 total 232.89 last 111.17) - open/close 29509 (time 1750591256.67 total 242.90 last 110.61) - open/close 30000 (time 1750591260.70 total 246.93 last 121.95) - open/close 31346 (time 1750591270.70 total 256.93 last 134.50) - open/close 32534 (time 1750591280.71 total 266.94 last 118.75) - open/close 33979 (time 1750591290.71 total 276.94 last 144.48) - open/close 35199 (time 1750591300.71 total 286.94 last 121.94) - open/close 36372 (time 1750591310.72 total 296.95 last 117.23) - open/close 37598 (time 1750591320.72 total 306.95 last 122.56) - open/close 38939 (time 1750591330.72 total 316.95 last 134.08) - open/close 40000 (time 1750591340.05 total 326.28 last 113.79) - open/close 41121 (time 1750591350.06 total 336.29 last 111.99) - open/close 42382 (time 1750591360.06 total 346.29 last 126.07) - open/close 43651 (time 1750591370.06 total 356.29 last 126.86) - open/close 44937 (time 1750591380.08 total 366.31 last 128.45) - open/close 46066 (time 1750591390.08 total 376.31 last 112.83) - open/close 47477 (time 1750591400.08 total 386.31 last 141.08) - open/close 48838 (time 1750591410.09 total 396.32 last 136.03) - open/close 50000 (time 1750591418.90 total 405.13 last 131.94) - open/close 51231 (time 1750591428.90 total 415.13 last 123.07) - open/close 52503 (time 1750591438.90 total 425.13 last 127.16) - open/close 53910 (time 1750591448.91 total 435.14 last 140.64) - open/close 55181 (time 1750591458.91 total 445.14 last 127.09) - open/close 56456 (time 1750591468.91 total 455.14 last 127.41) - open/close 57748 (time 1750591478.92 total 465.15 last 129.16) - open/close 59162 (time 1750591488.92 total 475.15 last 141.35) - open/close 60000 (time 1750591494.96 total 481.19 last 138.84) - open/close 61324 (time 1750591504.97 total 491.20 last 132.27) - open/close 62626 (time 1750591514.97 total 501.20 last 130.16) - open/close 63937 (time 1750591524.98 total 511.21 last 131.00) total: 64768 open/close in 517.14 seconds: 125.24 ops/second - /unlink 2806 (time 1750591542.77 total 10.00 last 280.58) - /unlink 5769 (time 1750591552.77 total 20.00 last 296.22) - /unlink 7905 (time 1750591562.77 total 30.01 last 213.53) - /unlink 10000 (time 1750591572.30 total 39.54 last 219.84) - /unlink 12475 (time 1750591582.30 total 49.54 last 247.46) - /unlink 15109 (time 1750591592.31 total 59.54 last 263.33) - /unlink 17359 (time 1750591602.31 total 69.54 last 224.95) - /unlink 19765 (time 1750591612.31 total 79.54 last 240.56) - /unlink 22367 (time 1750591622.31 total 89.55 last 260.16) - /unlink 24836 (time 1750591632.31 total 99.55 last 246.84) - /unlink 27395 (time 1750591642.32 total 109.55 last 255.80) - /unlink 29694 (time 1750591652.32 total 119.56 last 229.82) - /unlink 32173 (time 1750591662.32 total 129.56 last 247.83) - /unlink 34601 (time 1750591672.33 total 139.56 last 242.75) - /unlink 37326 (time 1750591682.33 total 149.56 last 272.49) - /unlink 39799 (time 1750591692.33 total 159.56 last 247.25) - /unlink 42736 (time 1750591702.33 total 169.56 last 293.69) - /unlink 45112 (time 1750591712.33 total 179.56 last 237.60) - /unlink 47429 (time 1750591722.33 total 189.57 last 231.60) - /unlink 50000 (time 1750591731.34 total 198.57 last 285.61) - /unlink 52970 (time 1750591741.34 total 208.57 last 296.90) - /unlink 55839 (time 1750591751.34 total 218.58 last 286.81) - /unlink 58222 (time 1750591761.34 total 228.58 last 238.28) - /unlink 60000 (time 1750591768.35 total 235.59 last 253.60) - /unlink 62671 (time 1750591778.36 total 245.59 last 267.06) total: 64768 /unlink in 252.54 seconds: 256.46 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1418 (time 1750591814.49 total 10.00 last 141.75) - open/close 2841 (time 1750591824.49 total 20.00 last 142.29) - open/close 4311 (time 1750591834.49 total 30.01 last 146.98) - open/close 5845 (time 1750591844.50 total 40.01 last 153.32) - open/close 7457 (time 1750591854.50 total 50.02 last 161.12) - open/close 8975 (time 1750591864.50 total 60.02 last 151.74) - open/close 10000 (time 1750591872.64 total 68.16 last 125.94) - open/close 11473 (time 1750591882.64 total 78.16 last 147.29) - open/close 12622 (time 1750591892.66 total 88.18 last 114.71) - open/close 13684 (time 1750591902.67 total 98.19 last 106.11) - open/close 15019 (time 1750591912.67 total 108.19 last 133.49) - open/close 16223 (time 1750591922.67 total 118.19 last 120.40) - open/close 17633 (time 1750591932.67 total 128.19 last 140.94) - open/close 18962 (time 1750591942.68 total 138.19 last 132.85) - open/close 20000 (time 1750591949.99 total 145.51 last 141.96) - open/close 21377 (time 1750591960.00 total 155.51 last 137.63) - open/close 22623 (time 1750591970.00 total 165.51 last 124.59) - open/close 24083 (time 1750591980.01 total 175.52 last 145.86) - open/close 25433 (time 1750591990.01 total 185.52 last 134.96) - open/close 26788 (time 1750592000.02 total 195.53 last 135.40) - open/close 27982 (time 1750592010.02 total 205.53 last 119.37) - open/close 29594 (time 1750592020.02 total 215.54 last 161.12) - open/close 30000 (time 1750592022.70 total 218.22 last 151.44) - open/close 31183 (time 1750592032.71 total 228.22 last 118.27) - open/close 32564 (time 1750592042.71 total 238.22 last 138.09) - open/close 33703 (time 1750592052.71 total 248.23 last 113.86) - open/close 34807 (time 1750592062.72 total 258.24 last 110.30) - open/close 36210 (time 1750592072.72 total 268.24 last 140.25) - open/close 37760 (time 1750592082.73 total 278.24 last 154.96) - open/close 39171 (time 1750592092.73 total 288.24 last 141.08) - open/close 40000 (time 1750592098.34 total 293.86 last 147.62) - open/close 41317 (time 1750592108.36 total 303.87 last 131.52) - open/close 42538 (time 1750592118.37 total 313.88 last 122.01) - open/close 43664 (time 1750592128.37 total 323.89 last 112.54) - open/close 44942 (time 1750592138.38 total 333.89 last 127.73) - open/close 46225 (time 1750592148.39 total 343.91 last 128.11) - open/close 47457 (time 1750592158.39 total 353.91 last 123.18) - open/close 48657 (time 1750592168.40 total 363.91 last 119.96) - open/close 49892 (time 1750592178.40 total 373.91 last 123.48) - open/close 51127 (time 1750592188.40 total 383.92 last 123.47) - open/close 52380 (time 1750592198.41 total 393.92 last 125.23) - open/close 53688 (time 1750592208.41 total 403.92 last 130.79) - open/close 54768 (time 1750592218.42 total 413.93 last 107.88) - open/close 56110 (time 1750592228.42 total 423.94 last 134.14) - open/close 57288 (time 1750592238.42 total 433.94 last 117.80) - open/close 58486 (time 1750592248.43 total 443.94 last 119.73) - open/close 59726 (time 1750592258.43 total 453.94 last 123.99) - open/close 60000 (time 1750592260.66 total 456.17 last 123.04) - open/close 61181 (time 1750592270.66 total 466.18 last 118.06) - open/close 62258 (time 1750592280.67 total 476.18 last 107.62) - open/close 63572 (time 1750592290.67 total 486.19 last 131.36) total: 64768 open/close in 495.84 seconds: 130.62 ops/second - /unlink 2067 (time 1750592311.82 total 10.00 last 206.64) - /unlink 4555 (time 1750592321.82 total 20.01 last 248.75) - /unlink 6807 (time 1750592331.82 total 30.01 last 225.08) - /unlink 8691 (time 1750592341.83 total 40.01 last 188.38) - /unlink 10000 (time 1750592349.08 total 47.27 last 180.45) - /unlink 12069 (time 1750592359.08 total 57.27 last 206.88) - /unlink 13977 (time 1750592369.08 total 67.27 last 190.77) - /unlink 16215 (time 1750592379.09 total 77.27 last 223.72) - /unlink 18389 (time 1750592389.09 total 87.28 last 217.29) - /unlink 20000 (time 1750592397.02 total 95.20 last 203.31) - /unlink 21937 (time 1750592407.02 total 105.21 last 193.57) - /unlink 24037 (time 1750592417.02 total 115.21 last 209.95) - /unlink 26134 (time 1750592427.03 total 125.21 last 209.63) - /unlink 28389 (time 1750592437.03 total 135.21 last 225.47) - /unlink 30000 (time 1750592445.32 total 143.51 last 194.27) - /unlink 32264 (time 1750592455.32 total 153.51 last 226.35) - /unlink 34599 (time 1750592465.33 total 163.51 last 233.46) - /unlink 36855 (time 1750592475.33 total 173.51 last 225.55) - /unlink 39215 (time 1750592485.33 total 183.51 last 235.98) - /unlink 40000 (time 1750592488.67 total 186.86 last 234.77) - /unlink 42282 (time 1750592498.67 total 196.86 last 228.15) - /unlink 45009 (time 1750592508.68 total 206.87 last 272.56) - /unlink 47525 (time 1750592518.68 total 216.87 last 251.58) - /unlink 50000 (time 1750592527.09 total 225.27 last 294.35) - /unlink 52334 (time 1750592537.09 total 235.27 last 233.39) - /unlink 54766 (time 1750592547.10 total 245.28 last 243.04) - /unlink 56607 (time 1750592557.10 total 255.28 last 184.05) - /unlink 58508 (time 1750592567.10 total 265.29 last 190.04) - /unlink 60000 (time 1750592574.83 total 273.02 last 192.92) - /unlink 61761 (time 1750592584.84 total 283.02 last 176.03) - /unlink 63692 (time 1750592594.84 total 293.03 last 193.03) total: 64768 /unlink in 298.25 seconds: 217.16 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1162 (time 1750592629.50 total 10.00 last 116.20) - open/close 2302 (time 1750592639.50 total 20.00 last 113.96) - open/close 3318 (time 1750592649.50 total 30.01 last 101.56) - open/close 4328 (time 1750592659.51 total 40.01 last 100.98) - open/close 5312 (time 1750592669.51 total 50.02 last 98.32) - open/close 6386 (time 1750592679.52 total 60.02 last 107.32) - open/close 7532 (time 1750592689.52 total 70.03 last 114.59) - open/close 8660 (time 1750592699.52 total 80.03 last 112.78) - open/close 9835 (time 1750592709.53 total 90.03 last 117.42) - open/close 10922 (time 1750592719.53 total 100.04 last 108.67) - open/close 12041 (time 1750592729.54 total 110.04 last 111.86) - open/close 13119 (time 1750592739.54 total 120.05 last 107.74) - open/close 14244 (time 1750592749.55 total 130.05 last 112.46) - open/close 15605 (time 1750592759.55 total 140.05 last 136.09) - open/close 16833 (time 1750592769.55 total 150.05 last 122.74) - open/close 18159 (time 1750592779.56 total 160.06 last 132.53) - open/close 19399 (time 1750592789.56 total 170.06 last 123.99) - open/close 20000 (time 1750592794.11 total 174.61 last 132.08) - open/close 21288 (time 1750592804.11 total 184.61 last 128.79) - open/close 22672 (time 1750592814.11 total 194.61 last 138.39) - open/close 24161 (time 1750592824.11 total 204.61 last 148.87) - open/close 25553 (time 1750592834.12 total 214.62 last 139.13) - open/close 26759 (time 1750592844.13 total 224.63 last 120.42) - open/close 27954 (time 1750592854.14 total 234.64 last 119.41) - open/close 29311 (time 1750592864.15 total 244.65 last 135.61) - open/close 30000 (time 1750592870.42 total 250.92 last 109.83) - open/close 31023 (time 1750592880.43 total 260.93 last 102.24) - open/close 32038 (time 1750592890.43 total 270.93 last 101.46) - open/close 33088 (time 1750592900.43 total 280.94 last 104.96) - open/close 34250 (time 1750592910.43 total 290.94 last 116.18) - open/close 35378 (time 1750592920.44 total 300.94 last 112.73) - open/close 36472 (time 1750592930.44 total 310.95 last 109.38) - open/close 37561 (time 1750592940.45 total 320.95 last 108.85) - open/close 38711 (time 1750592950.45 total 330.96 last 114.93) - open/close 39904 (time 1750592960.46 total 340.96 last 119.23) - open/close 41117 (time 1750592970.46 total 350.97 last 121.26) - open/close 42236 (time 1750592980.47 total 360.97 last 111.85) - open/close 43429 (time 1750592990.47 total 370.97 last 119.27) - open/close 44529 (time 1750593000.47 total 380.98 last 109.97) - open/close 45627 (time 1750593010.47 total 390.98 last 109.79) - open/close 46719 (time 1750593020.48 total 400.99 last 109.09) - open/close 47927 (time 1750593030.49 total 410.99 last 120.74) - open/close 49168 (time 1750593040.50 total 421.00 last 124.00) - open/close 50000 (time 1750593047.72 total 428.23 last 115.13) - open/close 51128 (time 1750593057.73 total 438.24 last 112.68) - open/close 52409 (time 1750593067.74 total 448.24 last 128.04) - open/close 53585 (time 1750593077.74 total 458.25 last 117.52) - open/close 54859 (time 1750593087.75 total 468.25 last 127.34) - open/close 56005 (time 1750593097.76 total 478.26 last 114.50) - open/close 57025 (time 1750593107.76 total 488.26 last 102.00) - open/close 58094 (time 1750593117.76 total 498.26 last 106.88) - open/close 59119 (time 1750593127.77 total 508.27 last 102.43) - open/close 60000 (time 1750593135.96 total 516.46 last 107.55) - open/close 61074 (time 1750593145.96 total 526.47 last 107.35) - open/close 62061 (time 1750593155.96 total 536.47 last 98.69) - open/close 62999 (time 1750593165.97 total 546.48 last 93.72) - open/close 64041 (time 1750593175.97 total 556.48 last 104.17) total: 64768 open/close in 563.99 seconds: 114.84 ops/second - /unlink 1707 (time 1750593196.06 total 10.01 last 170.61) - /unlink 3523 (time 1750593206.07 total 20.01 last 181.52) - /unlink 5427 (time 1750593216.07 total 30.02 last 190.28) - /unlink 7261 (time 1750593226.07 total 40.02 last 183.38) - /unlink 8926 (time 1750593236.08 total 50.02 last 166.44) - /unlink 10000 (time 1750593241.29 total 55.23 last 206.07) - /unlink 12166 (time 1750593251.29 total 65.23 last 216.59) - /unlink 14422 (time 1750593261.29 total 75.23 last 225.58) - /unlink 17056 (time 1750593271.29 total 85.24 last 263.32) - /unlink 19699 (time 1750593281.30 total 95.24 last 264.26) - /unlink 22226 (time 1750593291.30 total 105.24 last 252.68) - /unlink 24732 (time 1750593301.30 total 115.24 last 250.59) - /unlink 26891 (time 1750593311.30 total 125.24 last 215.83) - /unlink 29105 (time 1750593321.30 total 135.25 last 221.35) - /unlink 30000 (time 1750593324.79 total 138.74 last 256.35) - /unlink 32414 (time 1750593334.79 total 148.74 last 241.39) - /unlink 34817 (time 1750593344.79 total 158.74 last 240.29) - /unlink 36867 (time 1750593354.79 total 168.74 last 204.98) - /unlink 38852 (time 1750593364.80 total 178.74 last 198.46) - /unlink 40000 (time 1750593371.10 total 185.05 last 181.99) - /unlink 41796 (time 1750593381.11 total 195.05 last 179.57) - /unlink 43895 (time 1750593391.11 total 205.05 last 209.87) - /unlink 45720 (time 1750593401.11 total 215.05 last 182.49) - /unlink 47717 (time 1750593411.11 total 225.05 last 199.67) - /unlink 49705 (time 1750593421.11 total 235.05 last 198.77) - /unlink 51415 (time 1750593431.11 total 245.05 last 171.00) - /unlink 53058 (time 1750593441.11 total 255.05 last 164.30) - /unlink 54862 (time 1750593451.11 total 265.06 last 180.34) - /unlink 56720 (time 1750593461.12 total 275.06 last 185.75) - /unlink 58496 (time 1750593471.12 total 285.06 last 177.55) - /unlink 60000 (time 1750593478.73 total 292.68 last 197.59) - /unlink 62046 (time 1750593488.73 total 302.68 last 204.58) - /unlink 63854 (time 1750593498.74 total 312.69 last 180.64) total: 64768 /unlink in 318.04 seconds: 203.65 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 997 (time 1750593530.79 total 10.01 last 99.62) - open/close 2071 (time 1750593540.79 total 20.01 last 107.36) - open/close 3065 (time 1750593550.80 total 30.02 last 99.31) - open/close 4108 (time 1750593560.81 total 40.03 last 104.25) - open/close 5175 (time 1750593570.81 total 50.03 last 106.68) - open/close 6348 (time 1750593580.81 total 60.03 last 117.28) - open/close 7699 (time 1750593590.81 total 70.03 last 135.03) - open/close 8730 (time 1750593600.82 total 80.04 last 103.02) - open/close 9817 (time 1750593610.82 total 90.04 last 108.67) - open/close 11052 (time 1750593620.83 total 100.05 last 123.42) - open/close 12164 (time 1750593630.83 total 110.05 last 111.16) - open/close 13333 (time 1750593640.84 total 120.05 last 116.89) - open/close 14441 (time 1750593650.84 total 130.06 last 110.72) - open/close 15509 (time 1750593660.84 total 140.06 last 106.79) - open/close 16671 (time 1750593670.85 total 150.07 last 116.11) - open/close 17825 (time 1750593680.85 total 160.07 last 115.37) - open/close 18937 (time 1750593690.86 total 170.08 last 111.16) - open/close 20000 (time 1750593700.35 total 179.57 last 111.97) - open/close 21070 (time 1750593710.36 total 189.58 last 106.92) - open/close 22137 (time 1750593720.36 total 199.58 last 106.68) - open/close 23141 (time 1750593730.36 total 209.58 last 100.38) - open/close 24166 (time 1750593740.37 total 219.59 last 102.43) - open/close 25156 (time 1750593750.37 total 229.59 last 98.97) - open/close 26193 (time 1750593760.38 total 239.60 last 103.65) - open/close 27219 (time 1750593770.38 total 249.60 last 102.55) - open/close 28274 (time 1750593780.38 total 259.60 last 105.48) - open/close 29231 (time 1750593790.39 total 269.61 last 95.66) - open/close 30000 (time 1750593797.54 total 276.75 last 107.60) - open/close 31027 (time 1750593807.55 total 286.77 last 102.56) - open/close 32091 (time 1750593817.56 total 296.78 last 106.31) - open/close 33163 (time 1750593827.57 total 306.79 last 107.08) - open/close 34277 (time 1750593837.57 total 316.79 last 111.34) - open/close 35443 (time 1750593847.58 total 326.80 last 116.58) - open/close 36518 (time 1750593857.58 total 336.80 last 107.47) - open/close 37564 (time 1750593867.59 total 346.81 last 104.49) - open/close 38629 (time 1750593877.60 total 356.82 last 106.39) - open/close 39620 (time 1750593887.61 total 366.83 last 99.01) - open/close 40000 (time 1750593891.15 total 370.36 last 107.44) - open/close 41076 (time 1750593901.15 total 380.37 last 107.52) - open/close 41987 (time 1750593911.16 total 390.38 last 91.07) - open/close 42970 (time 1750593921.16 total 400.38 last 98.26) - open/close 44011 (time 1750593931.16 total 410.38 last 104.08) - open/close 45047 (time 1750593941.17 total 420.39 last 103.54) - open/close 46177 (time 1750593951.17 total 430.39 last 112.94) - open/close 47476 (time 1750593961.18 total 440.40 last 129.85) - open/close 48683 (time 1750593971.18 total 450.40 last 120.65) - open/close 50000 (time 1750593980.99 total 460.21 last 134.24) - open/close 51005 (time 1750593990.99 total 470.21 last 100.47) - open/close 51913 (time 1750594001.01 total 480.23 last 90.69) - open/close 52868 (time 1750594011.01 total 490.23 last 95.48) - open/close 53900 (time 1750594021.01 total 500.23 last 103.16) - open/close 55013 (time 1750594031.02 total 510.23 last 111.26) - open/close 56088 (time 1750594041.02 total 520.24 last 107.41) - open/close 57112 (time 1750594051.03 total 530.25 last 102.32) - open/close 58194 (time 1750594061.04 total 540.26 last 108.13) - open/close 59385 (time 1750594071.04 total 550.26 last 119.04) - open/close 60000 (time 1750594076.07 total 555.29 last 122.38) - open/close 61168 (time 1750594086.07 total 565.29 last 116.75) - open/close 62362 (time 1750594096.08 total 575.30 last 119.36) - open/close 63523 (time 1750594106.08 total 585.30 last 116.07) total: 64768 open/close in 594.94 seconds: 108.87 ops/second - /unlink 2043 (time 1750594127.84 total 10.00 last 204.21) - /unlink 4287 (time 1750594137.84 total 20.00 last 224.40) - /unlink 6232 (time 1750594147.84 total 30.01 last 194.46) - /unlink 8239 (time 1750594157.84 total 40.01 last 200.66) - /unlink 10000 (time 1750594167.29 total 49.46 last 186.36) - /unlink 11938 (time 1750594177.30 total 59.46 last 193.78) - /unlink 13926 (time 1750594187.30 total 69.47 last 198.63) - /unlink 15818 (time 1750594197.30 total 79.47 last 189.20) - /unlink 17845 (time 1750594207.31 total 89.47 last 202.61) - /unlink 20000 (time 1750594216.50 total 98.66 last 234.53) - /unlink 22122 (time 1750594226.50 total 108.66 last 212.13) - /unlink 24199 (time 1750594236.50 total 118.67 last 207.65) - /unlink 26571 (time 1750594246.51 total 128.67 last 237.14) - /unlink 28591 (time 1750594256.51 total 138.67 last 201.93) - /unlink 30000 (time 1750594264.65 total 146.81 last 173.07) - /unlink 32051 (time 1750594274.65 total 156.82 last 205.07) - /unlink 33828 (time 1750594284.66 total 166.82 last 177.56) - /unlink 35993 (time 1750594294.66 total 176.82 last 216.50) - /unlink 38134 (time 1750594304.66 total 186.83 last 214.05) - /unlink 40000 (time 1750594313.43 total 195.59 last 212.94) - /unlink 42186 (time 1750594323.43 total 205.59 last 218.59) - /unlink 44309 (time 1750594333.43 total 215.59 last 212.29) - /unlink 46417 (time 1750594343.43 total 225.59 last 210.79) - /unlink 48303 (time 1750594353.44 total 235.60 last 188.44) - /unlink 50000 (time 1750594362.93 total 245.09 last 178.81) - /unlink 51795 (time 1750594372.93 total 255.09 last 179.41) - /unlink 53610 (time 1750594382.93 total 265.10 last 181.47) - /unlink 55552 (time 1750594392.94 total 275.10 last 194.12) - /unlink 57412 (time 1750594402.94 total 285.10 last 185.95) - /unlink 59265 (time 1750594412.94 total 295.10 last 185.28) - /unlink 60000 (time 1750594416.77 total 298.93 last 191.88) - /unlink 62120 (time 1750594426.77 total 308.93 last 211.99) - /unlink 64438 (time 1750594436.77 total 318.94 last 231.71) total: 64768 /unlink in 320.50 seconds: 202.08 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1265 (time 1750594464.32 total 10.00 last 126.48) - open/close 2498 (time 1750594474.33 total 20.01 last 123.22) - open/close 3805 (time 1750594484.33 total 30.01 last 130.69) - open/close 5011 (time 1750594494.33 total 40.01 last 120.58) - open/close 6149 (time 1750594504.34 total 50.01 last 113.77) - open/close 7223 (time 1750594514.34 total 60.02 last 107.38) - open/close 8274 (time 1750594524.34 total 70.02 last 105.06) - open/close 9408 (time 1750594534.35 total 80.03 last 113.31) - open/close 10000 (time 1750594539.64 total 85.32 last 111.92) - open/close 11071 (time 1750594549.64 total 95.32 last 107.04) - open/close 12280 (time 1750594559.65 total 105.33 last 120.82) - open/close 13357 (time 1750594569.66 total 115.33 last 107.64) - open/close 14543 (time 1750594579.66 total 125.34 last 118.57) - open/close 15746 (time 1750594589.66 total 135.34 last 120.28) - open/close 16866 (time 1750594599.66 total 145.34 last 111.97) - open/close 18049 (time 1750594609.66 total 155.34 last 118.29) - open/close 19141 (time 1750594619.67 total 165.35 last 109.12) - open/close 20000 (time 1750594626.42 total 172.10 last 127.20) - open/close 21448 (time 1750594636.43 total 182.10 last 144.75) - open/close 22708 (time 1750594646.43 total 192.11 last 125.91) - open/close 23890 (time 1750594656.44 total 202.12 last 118.07) - open/close 25069 (time 1750594666.44 total 212.12 last 117.90) - open/close 26255 (time 1750594676.45 total 222.12 last 118.59) - open/close 27282 (time 1750594686.45 total 232.13 last 102.62) - open/close 28349 (time 1750594696.46 total 242.13 last 106.67) - open/close 29559 (time 1750594706.46 total 252.14 last 120.93) - open/close 30000 (time 1750594709.98 total 255.66 last 125.41) - open/close 31354 (time 1750594719.98 total 265.66 last 135.38) - open/close 32598 (time 1750594729.98 total 275.66 last 124.40) - open/close 33749 (time 1750594739.99 total 285.67 last 115.01) - open/close 34949 (time 1750594749.99 total 295.67 last 119.95) - open/close 36254 (time 1750594759.99 total 305.67 last 130.48) - open/close 37584 (time 1750594770.00 total 315.67 last 132.98) - open/close 38742 (time 1750594780.00 total 325.68 last 115.74) - open/close 39935 (time 1750594790.01 total 335.69 last 119.22) - open/close 41056 (time 1750594800.01 total 345.69 last 112.04) - open/close 42196 (time 1750594810.01 total 355.69 last 114.00) - open/close 43315 (time 1750594820.02 total 365.69 last 111.87) - open/close 44548 (time 1750594830.02 total 375.69 last 123.29) - open/close 45671 (time 1750594840.02 total 385.70 last 112.22) - open/close 46936 (time 1750594850.03 total 395.71 last 126.41) - open/close 48003 (time 1750594860.03 total 405.71 last 106.66) - open/close 49331 (time 1750594870.04 total 415.71 last 132.77) - open/close 50000 (time 1750594875.47 total 421.15 last 123.09) - open/close 51209 (time 1750594885.48 total 431.16 last 120.77) - open/close 52354 (time 1750594895.48 total 441.16 last 114.49) - open/close 53404 (time 1750594905.49 total 451.17 last 104.95) - open/close 54523 (time 1750594915.49 total 461.17 last 111.84) - open/close 55635 (time 1750594925.51 total 471.19 last 110.99) - open/close 56825 (time 1750594935.51 total 481.19 last 118.98) - open/close 57802 (time 1750594945.52 total 491.20 last 97.67) - open/close 58832 (time 1750594955.52 total 501.20 last 103.00) - open/close 59926 (time 1750594965.52 total 511.20 last 109.35) - open/close 60903 (time 1750594975.53 total 521.20 last 97.67) - open/close 62003 (time 1750594985.53 total 531.21 last 109.92) - open/close 63197 (time 1750594995.54 total 541.22 last 119.31) - open/close 64229 (time 1750595005.54 total 551.22 last 103.16) total: 64768 open/close in 556.41 seconds: 116.40 ops/second - /unlink 2080 (time 1750595022.66 total 10.00 last 207.98) - /unlink 3995 (time 1750595032.67 total 20.01 last 191.42) - /unlink 5733 (time 1750595042.67 total 30.01 last 173.69) - /unlink 7593 (time 1750595052.68 total 40.02 last 185.93) - /unlink 9564 (time 1750595062.68 total 50.02 last 197.07) - /unlink 10000 (time 1750595064.94 total 52.28 last 193.05) - /unlink 11661 (time 1750595074.94 total 62.28 last 166.08) - /unlink 13485 (time 1750595084.94 total 72.28 last 182.39) - /unlink 15269 (time 1750595094.94 total 82.28 last 178.33) - /unlink 17048 (time 1750595104.94 total 92.28 last 177.90) - /unlink 18971 (time 1750595114.95 total 102.28 last 192.27) - /unlink 20000 (time 1750595120.46 total 107.80 last 186.50) - /unlink 22068 (time 1750595130.47 total 117.80 last 206.76) - /unlink 23932 (time 1750595140.47 total 127.80 last 186.39) - /unlink 25903 (time 1750595150.47 total 137.81 last 197.06) - /unlink 27962 (time 1750595160.47 total 147.81 last 205.85) - /unlink 29879 (time 1750595170.47 total 157.81 last 191.63) - /unlink 31897 (time 1750595180.47 total 167.81 last 201.79) - /unlink 34097 (time 1750595190.47 total 177.81 last 219.98) - /unlink 36193 (time 1750595200.48 total 187.81 last 209.55) - /unlink 38166 (time 1750595210.48 total 197.82 last 197.29) - /unlink 40000 (time 1750595218.06 total 205.40 last 241.86) - /unlink 41840 (time 1750595228.06 total 215.40 last 183.97) - /unlink 43880 (time 1750595238.07 total 225.40 last 203.92) - /unlink 45729 (time 1750595248.07 total 235.41 last 184.88) - /unlink 47560 (time 1750595258.07 total 245.41 last 183.02) - /unlink 49675 (time 1750595268.07 total 255.41 last 211.50) - /unlink 51880 (time 1750595278.07 total 265.41 last 220.47) - /unlink 53881 (time 1750595288.07 total 275.41 last 200.09) - /unlink 55868 (time 1750595298.08 total 285.41 last 198.65) - /unlink 57768 (time 1750595308.08 total 295.42 last 189.88) - /unlink 59710 (time 1750595318.08 total 305.42 last 194.18) - /unlink 61588 (time 1750595328.09 total 315.43 last 187.69) - /unlink 63431 (time 1750595338.09 total 325.43 last 184.22) total: 64768 /unlink in 332.80 seconds: 194.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg104-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg104-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg104-server stop mds service on oleg104-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg104-server unloading modules via unload_modules_local on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-client: ssh exited with exit code 2 pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 PASS 106 (4418s) Stopping clients: oleg104-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg104-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg104-client: oleg104-server: ssh exited with exit code 2 oleg104-server: oleg104-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg104-server' oleg104-server: oleg104-server.virtnet: executing load_modules_local oleg104-server: Loading modules from /home/green/git/lustre-release/lustre oleg104-server: detected 4 online CPUs by sysfs oleg104-server: MODOPTS_LIBCFS= oleg104-server: Force libcfs to create 2 CPU partitions oleg104-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg104-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4526 sec ========== 08:30:13 (1750595413) === conf-sanity: start cleanup 08:30:13 (1750595413) === === conf-sanity: finish cleanup 08:30:14 (1750595414) ===