-----============= acceptance-small: conf-sanity ============----- Thu May 1 18:24:53 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg342-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg342-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg342-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg342-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg342-server oleg342-server: oleg342-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: detected 4 online CPUs by sysfs oleg342-server: MODOPTS_LIBCFS= oleg342-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg342-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg342-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg342-server: oleg342-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg342-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server stop mds service on oleg342-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:26:04 (1746138364) Stopping clients: oleg342-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg342-client.virtnet /mnt/lustre2 (opts:-f) oleg342-server: oleg342-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: detected 4 online CPUs by sysfs oleg342-server: MODOPTS_LIBCFS= oleg342-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg342-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg342-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg342-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg342-server: oleg342-server.virtnet: executing set_default_debug -1 all pdsh@oleg342-client: oleg342-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg342-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg342-client.virtnet: -o user_xattr,flock 192.168.203.142@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1789 (time 1746138408.02 total 10.00 last 178.87) - open/close 2973 (time 1746138418.02 total 20.01 last 118.33) - open/close 4234 (time 1746138428.03 total 30.02 last 125.96) - open/close 5380 (time 1746138438.04 total 40.02 last 114.60) - open/close 6823 (time 1746138448.04 total 50.03 last 144.19) - open/close 7922 (time 1746138458.04 total 60.03 last 109.88) - open/close 9222 (time 1746138468.05 total 70.03 last 129.97) - open/close 10000 (time 1746138474.63 total 76.61 last 118.26) - open/close 11149 (time 1746138484.63 total 86.61 last 114.87) - open/close 12288 (time 1746138494.64 total 96.62 last 113.78) - open/close 13444 (time 1746138504.64 total 106.62 last 115.59) - open/close 14658 (time 1746138514.64 total 116.63 last 121.37) - open/close 16240 (time 1746138524.65 total 126.63 last 158.09) - open/close 17928 (time 1746138534.65 total 136.64 last 168.77) - open/close 19517 (time 1746138544.66 total 146.64 last 158.82) - open/close 20000 (time 1746138547.03 total 149.01 last 203.42) - open/close 22269 (time 1746138557.03 total 159.02 last 226.88) - open/close 24627 (time 1746138567.03 total 169.02 last 235.77) - open/close 26763 (time 1746138577.04 total 179.02 last 213.48) - open/close 28520 (time 1746138587.05 total 189.03 last 175.56) - open/close 30000 (time 1746138595.52 total 197.51 last 174.59) - open/close 31646 (time 1746138605.53 total 207.52 last 164.47) - open/close 32864 (time 1746138615.53 total 217.52 last 121.78) - open/close 34056 (time 1746138625.54 total 227.52 last 119.19) - open/close 35084 (time 1746138635.54 total 237.53 last 102.71) - open/close 36240 (time 1746138645.55 total 247.53 last 115.53) - open/close 37354 (time 1746138655.55 total 257.54 last 111.36) - open/close 38598 (time 1746138665.56 total 267.54 last 124.36) - open/close 39851 (time 1746138675.56 total 277.55 last 125.24) - open/close 40880 (time 1746138685.57 total 287.55 last 102.82) - open/close 42008 (time 1746138695.58 total 297.56 last 112.68) - open/close 43092 (time 1746138705.59 total 307.57 last 108.29) - open/close 44074 (time 1746138715.59 total 317.58 last 98.18) - open/close 45276 (time 1746138725.59 total 327.58 last 120.19) - open/close 46385 (time 1746138735.60 total 337.59 last 110.79) - open/close 47586 (time 1746138745.60 total 347.59 last 120.07) - open/close 48685 (time 1746138755.61 total 357.60 last 109.79) - open/close 49718 (time 1746138765.61 total 367.60 last 103.30) - open/close 50000 (time 1746138768.14 total 370.12 last 111.77) - open/close 51146 (time 1746138778.14 total 380.12 last 114.57) - open/close 52277 (time 1746138788.14 total 390.13 last 113.09) - open/close 53398 (time 1746138798.15 total 400.13 last 112.00) - open/close 54308 (time 1746138808.16 total 410.14 last 90.92) - open/close 55351 (time 1746138818.16 total 420.14 last 104.30) - open/close 56354 (time 1746138828.16 total 430.14 last 100.30) - open/close 57450 (time 1746138838.17 total 440.15 last 109.53) - open/close 58601 (time 1746138848.19 total 450.17 last 114.87) - open/close 59808 (time 1746138858.19 total 460.17 last 120.69) - open/close 61079 (time 1746138868.19 total 470.18 last 127.04) - open/close 62148 (time 1746138878.20 total 480.18 last 106.84) - open/close 63366 (time 1746138888.20 total 490.19 last 121.74) - open/close 64640 (time 1746138898.20 total 500.19 last 127.36) total: 64768 open/close in 501.44 seconds: 129.16 ops/second - /unlink 1928 (time 1746138911.22 total 10.00 last 192.79) - /unlink 3840 (time 1746138921.23 total 20.00 last 191.17) - /unlink 5508 (time 1746138931.23 total 30.00 last 166.79) - /unlink 7543 (time 1746138941.23 total 40.00 last 203.45) - /unlink 9280 (time 1746138951.23 total 50.01 last 173.67) - /unlink 10000 (time 1746138954.87 total 53.65 last 197.56) - /unlink 11730 (time 1746138964.88 total 63.65 last 172.98) - /unlink 13596 (time 1746138974.88 total 73.65 last 186.57) - /unlink 15408 (time 1746138984.89 total 83.66 last 181.06) - /unlink 17143 (time 1746138994.89 total 93.66 last 173.47) - /unlink 19022 (time 1746139004.89 total 103.67 last 187.81) - /unlink 20000 (time 1746139010.79 total 109.56 last 165.87) - /unlink 21625 (time 1746139020.79 total 119.56 last 162.49) - /unlink 23300 (time 1746139030.79 total 129.57 last 167.46) - /unlink 25074 (time 1746139040.80 total 139.58 last 177.26) - /unlink 26815 (time 1746139050.80 total 149.58 last 174.07) - /unlink 28573 (time 1746139060.80 total 159.58 last 175.75) - /unlink 30000 (time 1746139069.19 total 167.96 last 170.20) - /unlink 31638 (time 1746139079.19 total 177.96 last 163.79) - /unlink 33284 (time 1746139089.19 total 187.97 last 164.52) - /unlink 34963 (time 1746139099.19 total 197.97 last 167.89) - /unlink 36369 (time 1746139109.20 total 207.97 last 140.58) - /unlink 37875 (time 1746139119.20 total 217.98 last 150.51) - /unlink 39791 (time 1746139129.21 total 227.98 last 191.46) - /unlink 41826 (time 1746139139.21 total 237.99 last 203.41) - /unlink 43604 (time 1746139149.22 total 247.99 last 177.73) - /unlink 45630 (time 1746139159.22 total 258.00 last 202.52) - /unlink 47724 (time 1746139169.22 total 268.00 last 209.40) - /unlink 49857 (time 1746139179.22 total 278.00 last 213.23) - /unlink 51823 (time 1746139189.22 total 288.00 last 196.59) - /unlink 53792 (time 1746139199.23 total 298.00 last 196.85) - /unlink 55696 (time 1746139209.23 total 308.01 last 190.29) - /unlink 57658 (time 1746139219.23 total 318.01 last 196.17) - /unlink 59529 (time 1746139229.24 total 328.02 last 186.99) - /unlink 60000 (time 1746139231.60 total 330.37 last 199.91) - /unlink 62064 (time 1746139241.60 total 340.37 last 206.35) - /unlink 64224 (time 1746139251.60 total 350.38 last 215.93) total: 64768 /unlink in 352.56 seconds: 183.71 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1206 (time 1746139282.52 total 10.01 last 120.50) - open/close 2482 (time 1746139292.52 total 20.01 last 127.59) - open/close 3724 (time 1746139302.52 total 30.01 last 124.17) - open/close 4904 (time 1746139312.53 total 40.02 last 117.91) - open/close 6045 (time 1746139322.53 total 50.02 last 114.04) - open/close 7301 (time 1746139332.54 total 60.03 last 125.54) - open/close 8692 (time 1746139342.54 total 70.03 last 139.10) - open/close 10000 (time 1746139351.61 total 79.10 last 144.18) - open/close 11309 (time 1746139361.61 total 89.10 last 130.88) - open/close 12474 (time 1746139371.61 total 99.11 last 116.48) - open/close 13571 (time 1746139381.61 total 109.11 last 109.68) - open/close 14846 (time 1746139391.62 total 119.11 last 127.45) - open/close 16326 (time 1746139401.62 total 129.11 last 147.98) - open/close 17774 (time 1746139411.62 total 139.11 last 144.79) - open/close 18999 (time 1746139421.63 total 149.13 last 122.34) - open/close 20000 (time 1746139429.84 total 157.33 last 121.96) - open/close 21343 (time 1746139439.84 total 167.33 last 134.30) - open/close 22713 (time 1746139449.85 total 177.34 last 136.94) - open/close 24106 (time 1746139459.85 total 187.34 last 139.23) - open/close 25360 (time 1746139469.86 total 197.35 last 125.33) - open/close 26554 (time 1746139479.86 total 207.36 last 119.31) - open/close 28004 (time 1746139489.87 total 217.36 last 144.92) - open/close 29262 (time 1746139499.88 total 227.37 last 125.72) - open/close 30000 (time 1746139505.47 total 232.96 last 131.90) - open/close 31557 (time 1746139515.49 total 242.98 last 155.44) - open/close 33024 (time 1746139525.49 total 252.99 last 146.62) - open/close 34532 (time 1746139535.50 total 263.00 last 150.64) - open/close 36070 (time 1746139545.51 total 273.00 last 153.71) - open/close 37561 (time 1746139555.51 total 283.01 last 149.04) - open/close 39261 (time 1746139565.52 total 293.02 last 169.85) - open/close 40000 (time 1746139570.15 total 297.64 last 159.72) - open/close 41708 (time 1746139580.15 total 307.64 last 170.79) - open/close 43168 (time 1746139590.15 total 317.65 last 145.94) - open/close 44584 (time 1746139600.16 total 327.65 last 141.56) - open/close 45841 (time 1746139610.17 total 337.66 last 125.60) - open/close 47068 (time 1746139620.17 total 347.66 last 122.68) - open/close 48257 (time 1746139630.17 total 357.67 last 118.83) - open/close 49537 (time 1746139640.18 total 367.67 last 127.96) - open/close 50000 (time 1746139644.81 total 372.30 last 99.89) - open/close 51016 (time 1746139654.81 total 382.31 last 101.58) - open/close 52176 (time 1746139664.82 total 392.31 last 115.93) - open/close 53352 (time 1746139674.83 total 402.32 last 117.53) - open/close 54515 (time 1746139684.83 total 412.32 last 116.29) - open/close 55621 (time 1746139694.83 total 422.33 last 110.51) - open/close 56662 (time 1746139704.84 total 432.33 last 104.06) - open/close 57712 (time 1746139714.84 total 442.33 last 104.96) - open/close 58689 (time 1746139724.85 total 452.34 last 97.60) - open/close 59766 (time 1746139734.86 total 462.35 last 107.63) - open/close 61239 (time 1746139744.86 total 472.36 last 147.20) - open/close 62424 (time 1746139754.87 total 482.36 last 118.43) - open/close 63587 (time 1746139764.87 total 492.37 last 116.28) - open/close 64669 (time 1746139774.88 total 502.37 last 108.16) total: 64768 open/close in 503.21 seconds: 128.71 ops/second - /unlink 1624 (time 1746139787.46 total 10.01 last 162.25) - /unlink 3238 (time 1746139797.47 total 20.02 last 161.25) - /unlink 5240 (time 1746139807.48 total 30.02 last 200.11) - /unlink 7179 (time 1746139817.48 total 40.03 last 193.77) - /unlink 9216 (time 1746139827.49 total 50.04 last 203.57) - /unlink 10000 (time 1746139831.75 total 54.30 last 183.90) - /unlink 12474 (time 1746139841.75 total 64.30 last 247.39) - /unlink 14848 (time 1746139851.76 total 74.30 last 237.37) - /unlink 17018 (time 1746139861.76 total 84.30 last 216.96) - /unlink 19526 (time 1746139871.76 total 94.30 last 250.75) - /unlink 22708 (time 1746139881.76 total 104.31 last 318.18) - /unlink 25038 (time 1746139891.76 total 114.31 last 232.95) - /unlink 26956 (time 1746139901.77 total 124.31 last 191.74) - /unlink 29379 (time 1746139911.77 total 134.31 last 242.28) - /unlink 30000 (time 1746139914.61 total 137.16 last 218.32) - /unlink 32818 (time 1746139924.61 total 147.16 last 281.75) - /unlink 35046 (time 1746139934.61 total 157.16 last 222.74) - /unlink 37118 (time 1746139944.62 total 167.16 last 207.15) - /unlink 38872 (time 1746139954.62 total 177.16 last 175.38) - /unlink 40000 (time 1746139959.77 total 182.31 last 219.12) - /unlink 42064 (time 1746139969.77 total 192.31 last 206.38) - /unlink 43869 (time 1746139979.77 total 202.32 last 180.46) - /unlink 45776 (time 1746139989.77 total 212.32 last 190.70) - /unlink 47754 (time 1746139999.77 total 222.32 last 197.78) - /unlink 49639 (time 1746140009.77 total 232.32 last 188.44) - /unlink 51590 (time 1746140019.78 total 242.32 last 195.09) - /unlink 53317 (time 1746140029.78 total 252.33 last 172.60) - /unlink 55259 (time 1746140039.79 total 262.33 last 194.07) - /unlink 57201 (time 1746140049.79 total 272.33 last 194.18) - /unlink 58959 (time 1746140059.79 total 282.34 last 175.77) - /unlink 60000 (time 1746140066.14 total 288.68 last 164.00) - /unlink 62494 (time 1746140076.14 total 298.69 last 249.33) - /unlink 64383 (time 1746140086.15 total 308.70 last 188.65) total: 64768 /unlink in 311.13 seconds: 208.17 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1074 (time 1746140118.77 total 10.01 last 107.34) - open/close 2010 (time 1746140128.78 total 20.01 last 93.54) - open/close 3100 (time 1746140138.78 total 30.02 last 108.96) - open/close 4131 (time 1746140148.79 total 40.02 last 103.04) - open/close 5117 (time 1746140158.80 total 50.03 last 98.50) - open/close 6107 (time 1746140168.80 total 60.04 last 98.95) - open/close 7189 (time 1746140178.81 total 70.04 last 108.13) - open/close 8217 (time 1746140188.81 total 80.05 last 102.73) - open/close 9294 (time 1746140198.82 total 90.06 last 107.63) - open/close 10000 (time 1746140205.96 total 97.20 last 98.87) - open/close 11186 (time 1746140215.97 total 107.20 last 118.53) - open/close 12219 (time 1746140225.98 total 117.21 last 103.20) - open/close 13108 (time 1746140235.98 total 127.22 last 88.83) - open/close 14188 (time 1746140245.99 total 137.22 last 107.98) - open/close 15228 (time 1746140255.99 total 147.22 last 104.00) - open/close 16308 (time 1746140265.99 total 157.22 last 108.00) - open/close 17292 (time 1746140275.99 total 167.23 last 98.35) - open/close 18356 (time 1746140286.00 total 177.23 last 106.36) - open/close 19423 (time 1746140296.01 total 187.24 last 106.60) - open/close 20000 (time 1746140301.91 total 193.15 last 97.65) - open/close 21070 (time 1746140311.92 total 203.15 last 106.96) - open/close 22183 (time 1746140321.93 total 213.16 last 111.17) - open/close 23234 (time 1746140331.94 total 223.18 last 104.99) - open/close 24184 (time 1746140341.94 total 233.18 last 94.97) - open/close 25208 (time 1746140351.95 total 243.18 last 102.34) - open/close 26183 (time 1746140361.95 total 253.19 last 97.48) - open/close 27303 (time 1746140371.96 total 263.19 last 111.94) - open/close 28343 (time 1746140381.97 total 273.20 last 103.90) - open/close 29363 (time 1746140391.98 total 283.21 last 101.89) - open/close 30000 (time 1746140397.94 total 289.18 last 106.75) - open/close 31099 (time 1746140407.95 total 299.19 last 109.83) - open/close 32276 (time 1746140417.95 total 309.19 last 117.65) - open/close 33489 (time 1746140427.96 total 319.19 last 121.30) - open/close 34784 (time 1746140437.96 total 329.19 last 129.45) - open/close 35906 (time 1746140447.96 total 339.20 last 112.19) - open/close 36943 (time 1746140457.96 total 349.20 last 103.70) - open/close 38046 (time 1746140467.97 total 359.20 last 110.24) - open/close 39150 (time 1746140477.97 total 369.21 last 110.34) - open/close 40000 (time 1746140485.46 total 376.69 last 113.56) - open/close 41129 (time 1746140495.46 total 386.69 last 112.88) - open/close 42288 (time 1746140505.46 total 396.70 last 115.83) - open/close 43378 (time 1746140515.47 total 406.70 last 108.98) - open/close 44391 (time 1746140525.47 total 416.70 last 101.27) - open/close 45421 (time 1746140535.47 total 426.71 last 102.98) - open/close 46494 (time 1746140545.47 total 436.71 last 107.30) - open/close 47475 (time 1746140555.48 total 446.71 last 98.06) - open/close 48446 (time 1746140565.48 total 456.71 last 97.09) - open/close 49416 (time 1746140575.48 total 466.72 last 96.94) - open/close 50000 (time 1746140580.55 total 471.78 last 115.33) - open/close 51136 (time 1746140590.55 total 481.79 last 113.53) - open/close 52401 (time 1746140600.56 total 491.80 last 126.39) - open/close 53545 (time 1746140610.56 total 501.80 last 114.39) - open/close 54761 (time 1746140620.57 total 511.80 last 121.51) - open/close 55873 (time 1746140630.58 total 521.81 last 111.09) - open/close 56866 (time 1746140640.58 total 531.81 last 99.29) - open/close 58291 (time 1746140650.58 total 541.82 last 142.42) - open/close 59883 (time 1746140660.59 total 551.82 last 159.19) - open/close 61209 (time 1746140670.59 total 561.82 last 132.56) - open/close 62507 (time 1746140680.60 total 571.83 last 129.69) - open/close 63649 (time 1746140690.60 total 581.84 last 114.12) total: 64768 open/close in 590.07 seconds: 109.76 ops/second - /unlink 2369 (time 1746140710.34 total 10.00 last 236.85) - /unlink 5032 (time 1746140720.34 total 20.00 last 266.29) - /unlink 7286 (time 1746140730.35 total 30.01 last 225.27) - /unlink 9318 (time 1746140740.35 total 40.01 last 203.11) - /unlink 10000 (time 1746140744.00 total 43.67 last 186.67) - /unlink 11682 (time 1746140754.01 total 53.67 last 168.10) - /unlink 13317 (time 1746140764.01 total 63.67 last 163.49) - /unlink 15424 (time 1746140774.01 total 73.68 last 210.65) - /unlink 17426 (time 1746140784.02 total 83.68 last 200.13) - /unlink 19084 (time 1746140794.02 total 93.68 last 165.75) - /unlink 20000 (time 1746140798.85 total 98.52 last 189.49) - /unlink 21788 (time 1746140808.86 total 108.52 last 178.74) - /unlink 24007 (time 1746140818.86 total 118.52 last 221.87) - /unlink 26378 (time 1746140828.86 total 128.52 last 237.00) - /unlink 28249 (time 1746140838.87 total 138.53 last 187.00) - /unlink 30000 (time 1746140846.87 total 146.53 last 218.76) - /unlink 32062 (time 1746140856.87 total 156.54 last 206.18) - /unlink 33951 (time 1746140866.87 total 166.54 last 188.87) - /unlink 35689 (time 1746140876.88 total 176.54 last 173.74) - /unlink 37559 (time 1746140886.88 total 186.55 last 186.88) - /unlink 39430 (time 1746140896.89 total 196.55 last 187.02) - /unlink 40000 (time 1746140899.67 total 199.33 last 205.08) - /unlink 42155 (time 1746140909.67 total 209.33 last 215.47) - /unlink 44382 (time 1746140919.67 total 219.33 last 222.65) - /unlink 47027 (time 1746140929.67 total 229.33 last 264.47) - /unlink 49053 (time 1746140939.67 total 239.33 last 202.59) - /unlink 50000 (time 1746140944.10 total 243.76 last 213.89) - /unlink 52572 (time 1746140954.10 total 253.76 last 257.15) - /unlink 54993 (time 1746140964.10 total 263.77 last 242.06) - /unlink 57063 (time 1746140974.11 total 273.77 last 206.98) - /unlink 59260 (time 1746140984.11 total 283.77 last 219.65) - /unlink 60000 (time 1746140988.57 total 288.23 last 166.00) - /unlink 62112 (time 1746140998.57 total 298.23 last 211.19) - /unlink 64054 (time 1746141008.57 total 308.23 last 194.10) total: 64768 /unlink in 311.60 seconds: 207.85 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1133 (time 1746141040.37 total 10.01 last 113.24) - open/close 2272 (time 1746141050.37 total 20.01 last 113.88) - open/close 3300 (time 1746141060.37 total 30.01 last 102.80) - open/close 4283 (time 1746141070.38 total 40.02 last 98.20) - open/close 5170 (time 1746141080.39 total 50.03 last 88.60) - open/close 6073 (time 1746141090.40 total 60.04 last 90.23) - open/close 6977 (time 1746141100.40 total 70.04 last 90.33) - open/close 7910 (time 1746141110.41 total 80.05 last 93.27) - open/close 8848 (time 1746141120.41 total 90.05 last 93.79) - open/close 9769 (time 1746141130.41 total 100.05 last 92.07) - open/close 10000 (time 1746141132.68 total 102.32 last 101.89) - open/close 11005 (time 1746141142.68 total 112.32 last 100.48) - open/close 12167 (time 1746141152.69 total 122.33 last 116.10) - open/close 13240 (time 1746141162.70 total 132.34 last 107.20) - open/close 14381 (time 1746141172.70 total 142.34 last 114.06) - open/close 15517 (time 1746141182.71 total 152.35 last 113.48) - open/close 16546 (time 1746141192.72 total 162.36 last 102.83) - open/close 17609 (time 1746141202.73 total 172.36 last 106.24) - open/close 18665 (time 1746141212.73 total 182.37 last 105.58) - open/close 19650 (time 1746141222.73 total 192.37 last 98.44) - open/close 20000 (time 1746141225.68 total 195.32 last 118.76) - open/close 21142 (time 1746141235.68 total 205.32 last 114.17) - open/close 22342 (time 1746141245.69 total 215.32 last 119.98) - open/close 23462 (time 1746141255.69 total 225.33 last 111.97) - open/close 24585 (time 1746141265.69 total 235.33 last 112.25) - open/close 25490 (time 1746141275.70 total 245.34 last 90.46) - open/close 26499 (time 1746141285.70 total 255.34 last 100.88) - open/close 27573 (time 1746141295.70 total 265.34 last 107.36) - open/close 28652 (time 1746141305.70 total 275.34 last 107.89) - open/close 29853 (time 1746141315.71 total 285.35 last 120.08) - open/close 30886 (time 1746141325.71 total 295.35 last 103.26) - open/close 31981 (time 1746141335.71 total 305.35 last 109.48) - open/close 33222 (time 1746141345.72 total 315.36 last 124.02) - open/close 34256 (time 1746141355.72 total 325.36 last 103.37) - open/close 35335 (time 1746141365.72 total 335.36 last 107.87) - open/close 36587 (time 1746141375.73 total 345.37 last 125.17) - open/close 37690 (time 1746141385.73 total 355.37 last 110.26) - open/close 38801 (time 1746141395.73 total 365.37 last 111.10) - open/close 39985 (time 1746141405.74 total 375.38 last 118.33) - open/close 41238 (time 1746141415.74 total 385.38 last 125.29) - open/close 42303 (time 1746141425.74 total 395.38 last 106.49) - open/close 43729 (time 1746141435.74 total 405.38 last 142.51) - open/close 45042 (time 1746141445.75 total 415.39 last 131.28) - open/close 46183 (time 1746141455.75 total 425.39 last 114.04) - open/close 47315 (time 1746141465.75 total 435.39 last 113.20) - open/close 48650 (time 1746141475.75 total 445.39 last 133.49) - open/close 50000 (time 1746141485.74 total 455.38 last 135.19) - open/close 51378 (time 1746141495.74 total 465.38 last 137.74) - open/close 53049 (time 1746141505.75 total 475.39 last 167.01) - open/close 54728 (time 1746141515.75 total 485.39 last 167.85) - open/close 56122 (time 1746141525.75 total 495.39 last 139.39) - open/close 57656 (time 1746141535.76 total 505.40 last 153.28) - open/close 59329 (time 1746141545.76 total 515.40 last 167.27) - open/close 60000 (time 1746141549.60 total 519.24 last 174.88) - open/close 61453 (time 1746141559.61 total 529.25 last 145.19) - open/close 63058 (time 1746141569.61 total 539.25 last 160.43) - open/close 64492 (time 1746141579.61 total 549.25 last 143.37) total: 64768 open/close in 551.16 seconds: 117.51 ops/second - /unlink 2245 (time 1746141592.88 total 10.01 last 224.35) - /unlink 4909 (time 1746141602.88 total 20.01 last 266.40) - /unlink 7363 (time 1746141612.88 total 30.01 last 245.26) - /unlink 9799 (time 1746141622.89 total 40.02 last 243.48) - /unlink 12212 (time 1746141632.89 total 50.02 last 241.23) - /unlink 15877 (time 1746141642.89 total 60.02 last 366.45) - /unlink 19630 (time 1746141652.89 total 70.02 last 375.28) - /unlink 22435 (time 1746141662.89 total 80.03 last 280.43) - /unlink 25428 (time 1746141672.90 total 90.03 last 299.24) - /unlink 28699 (time 1746141682.90 total 100.03 last 327.08) - /unlink 30000 (time 1746141687.75 total 104.88 last 268.36) - /unlink 32890 (time 1746141697.75 total 114.88 last 288.96) - /unlink 35668 (time 1746141707.75 total 124.88 last 277.80) - /unlink 38002 (time 1746141717.75 total 134.88 last 233.31) - /unlink 40000 (time 1746141726.06 total 143.19 last 240.40) - /unlink 42759 (time 1746141736.06 total 153.19 last 275.85) - /unlink 44974 (time 1746141746.07 total 163.20 last 221.36) - /unlink 47131 (time 1746141756.07 total 173.20 last 215.69) - /unlink 48962 (time 1746141766.07 total 183.20 last 183.09) - /unlink 50000 (time 1746141771.02 total 188.15 last 209.93) - /unlink 52870 (time 1746141781.02 total 198.15 last 286.97) - /unlink 54752 (time 1746141791.02 total 208.15 last 188.15) - /unlink 56818 (time 1746141801.02 total 218.15 last 206.59) - /unlink 58906 (time 1746141811.02 total 228.15 last 208.73) - /unlink 60000 (time 1746141815.77 total 232.90 last 230.36) - /unlink 62805 (time 1746141825.77 total 242.91 last 280.42) total: 64768 /unlink in 250.48 seconds: 258.57 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1366 (time 1746141861.46 total 10.00 last 136.56) - open/close 2669 (time 1746141871.47 total 20.01 last 130.26) - open/close 3949 (time 1746141881.47 total 30.01 last 127.91) - open/close 5245 (time 1746141891.48 total 40.02 last 129.55) - open/close 6677 (time 1746141901.48 total 50.02 last 143.19) - open/close 8355 (time 1746141911.48 total 60.02 last 167.70) - open/close 9811 (time 1746141921.49 total 70.03 last 145.54) - open/close 11258 (time 1746141931.49 total 80.03 last 144.65) - open/close 12675 (time 1746141941.50 total 90.04 last 141.64) - open/close 14066 (time 1746141951.50 total 100.04 last 139.01) - open/close 15387 (time 1746141961.51 total 110.04 last 132.06) - open/close 16805 (time 1746141971.51 total 120.05 last 141.77) - open/close 18173 (time 1746141981.51 total 130.05 last 136.73) - open/close 19788 (time 1746141991.52 total 140.06 last 161.44) - open/close 21230 (time 1746142001.52 total 150.06 last 144.18) - open/close 22369 (time 1746142011.54 total 160.07 last 113.71) - open/close 23609 (time 1746142021.54 total 170.08 last 123.90) - open/close 25264 (time 1746142031.55 total 180.09 last 165.34) - open/close 26851 (time 1746142041.56 total 190.10 last 158.65) - open/close 28231 (time 1746142051.56 total 200.10 last 137.94) - open/close 29743 (time 1746142061.56 total 210.10 last 151.17) - open/close 31196 (time 1746142071.56 total 220.10 last 145.27) - open/close 32756 (time 1746142081.57 total 230.11 last 155.92) - open/close 34666 (time 1746142091.58 total 240.12 last 190.84) - open/close 35996 (time 1746142101.58 total 250.12 last 133.00) - open/close 37329 (time 1746142111.58 total 260.12 last 133.26) - open/close 39061 (time 1746142121.58 total 270.12 last 173.15) - open/close 40000 (time 1746142128.01 total 276.55 last 146.01) - open/close 41688 (time 1746142138.02 total 286.56 last 168.74) - open/close 43703 (time 1746142148.02 total 296.56 last 201.49) - open/close 45353 (time 1746142158.03 total 306.56 last 164.88) - open/close 46896 (time 1746142168.03 total 316.57 last 154.19) - open/close 48335 (time 1746142178.04 total 326.58 last 143.82) - open/close 49666 (time 1746142188.04 total 336.58 last 133.02) - open/close 50000 (time 1746142190.39 total 338.93 last 142.45) - open/close 51607 (time 1746142200.39 total 348.93 last 160.68) - open/close 53114 (time 1746142210.39 total 358.93 last 150.70) - open/close 54600 (time 1746142220.39 total 368.93 last 148.54) - open/close 56505 (time 1746142230.40 total 378.94 last 190.40) - open/close 58431 (time 1746142240.40 total 388.94 last 192.59) - open/close 60000 (time 1746142249.86 total 398.40 last 165.85) - open/close 61317 (time 1746142259.87 total 408.41 last 131.58) - open/close 63115 (time 1746142269.87 total 418.41 last 179.79) - open/close 64691 (time 1746142279.87 total 428.41 last 157.57) total: 64768 open/close in 428.96 seconds: 150.99 ops/second - /unlink 2369 (time 1746142292.14 total 10.00 last 236.89) - /unlink 5208 (time 1746142302.14 total 20.00 last 283.81) - /unlink 7772 (time 1746142312.14 total 30.01 last 256.35) - /unlink 10000 (time 1746142320.68 total 38.54 last 260.92) - /unlink 12532 (time 1746142330.68 total 48.55 last 253.15) - /unlink 15446 (time 1746142340.68 total 58.55 last 291.36) - /unlink 18487 (time 1746142350.68 total 68.55 last 304.09) - /unlink 20000 (time 1746142354.90 total 72.77 last 358.74) - /unlink 22370 (time 1746142364.90 total 82.77 last 236.96) - /unlink 24564 (time 1746142374.90 total 92.77 last 219.39) - /unlink 26701 (time 1746142384.91 total 102.77 last 213.64) - /unlink 29002 (time 1746142394.91 total 112.77 last 230.05) - /unlink 30000 (time 1746142399.09 total 116.95 last 238.86) - /unlink 32261 (time 1746142409.09 total 126.95 last 226.07) - /unlink 34718 (time 1746142419.09 total 136.95 last 245.66) - /unlink 37054 (time 1746142429.09 total 146.96 last 233.53) - /unlink 39444 (time 1746142439.09 total 156.96 last 238.98) - /unlink 40000 (time 1746142441.27 total 159.13 last 255.77) - /unlink 42334 (time 1746142451.27 total 169.13 last 233.36) - /unlink 44579 (time 1746142461.27 total 179.14 last 224.45) - /unlink 46880 (time 1746142471.27 total 189.14 last 230.10) - /unlink 49319 (time 1746142481.27 total 199.14 last 243.87) - /unlink 50000 (time 1746142484.00 total 201.86 last 249.71) - /unlink 52362 (time 1746142494.00 total 211.87 last 236.17) - /unlink 54779 (time 1746142504.00 total 221.87 last 241.67) - /unlink 57453 (time 1746142514.00 total 231.87 last 267.37) - /unlink 59985 (time 1746142524.00 total 241.87 last 253.16) - /unlink 62309 (time 1746142534.01 total 251.87 last 232.34) - /unlink 64538 (time 1746142544.01 total 261.87 last 222.84) total: 64768 /unlink in 262.96 seconds: 246.30 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg342-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg342-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg342-server stop mds service on oleg342-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg342-server unloading modules via unload_modules_local on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-client: ssh exited with exit code 2 pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 PASS 106 (4224s) Stopping clients: oleg342-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg342-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg342-client: oleg342-server: ssh exited with exit code 2 oleg342-server: oleg342-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg342-server' oleg342-server: oleg342-server.virtnet: executing load_modules_local oleg342-server: Loading modules from /home/green/git/lustre-release/lustre oleg342-server: detected 4 online CPUs by sysfs oleg342-server: MODOPTS_LIBCFS= oleg342-server: Force libcfs to create 2 CPU partitions oleg342-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg342-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4325 sec ========== 19:36:58 (1746142618) === conf-sanity: start cleanup 19:36:59 (1746142619) === === conf-sanity: finish cleanup 19:37:00 (1746142620) ===