-----============= acceptance-small: conf-sanity ============----- Thu May 1 18:25:18 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg331-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg331-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg331-server oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg331-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg331-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg331-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg331-server: oleg331-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg331-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg331-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:26:51 (1746138411) Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg331-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg331-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg331-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg331-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg331-server: oleg331-server.virtnet: executing set_default_debug -1 all pdsh@oleg331-client: oleg331-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg331-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg331-client.virtnet: -o user_xattr,flock 192.168.203.131@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1433 (time 1746138473.91 total 10.01 last 143.15) - open/close 2752 (time 1746138483.92 total 20.01 last 131.85) - open/close 4190 (time 1746138493.92 total 30.02 last 143.70) - open/close 5533 (time 1746138503.93 total 40.03 last 134.23) - open/close 6930 (time 1746138513.94 total 50.04 last 139.55) - open/close 8822 (time 1746138523.94 total 60.04 last 189.18) - open/close 10000 (time 1746138529.77 total 65.87 last 202.11) - open/close 11751 (time 1746138539.77 total 75.87 last 175.04) - open/close 14038 (time 1746138549.77 total 85.87 last 228.68) - open/close 16464 (time 1746138559.77 total 95.87 last 242.59) - open/close 18990 (time 1746138569.78 total 105.87 last 252.55) - open/close 20000 (time 1746138573.75 total 109.84 last 254.35) - open/close 22009 (time 1746138583.75 total 119.85 last 200.84) - open/close 23998 (time 1746138593.75 total 129.85 last 198.80) - open/close 25847 (time 1746138603.76 total 139.85 last 184.87) - open/close 27224 (time 1746138613.76 total 149.86 last 137.69) - open/close 28581 (time 1746138623.76 total 159.86 last 135.69) - open/close 29791 (time 1746138633.77 total 169.86 last 120.91) - open/close 31027 (time 1746138643.77 total 179.86 last 123.60) - open/close 32314 (time 1746138653.77 total 189.87 last 128.62) - open/close 33829 (time 1746138663.78 total 199.88 last 151.36) - open/close 35326 (time 1746138673.79 total 209.88 last 149.62) - open/close 36555 (time 1746138683.79 total 219.89 last 122.84) - open/close 37761 (time 1746138693.80 total 229.90 last 120.51) - open/close 38962 (time 1746138703.80 total 239.90 last 120.08) - open/close 40000 (time 1746138713.17 total 249.27 last 110.78) - open/close 41247 (time 1746138723.18 total 259.27 last 124.64) - open/close 42496 (time 1746138733.18 total 269.28 last 124.87) - open/close 43848 (time 1746138743.18 total 279.28 last 135.11) - open/close 44989 (time 1746138753.20 total 289.30 last 113.95) - open/close 46135 (time 1746138763.20 total 299.30 last 114.54) - open/close 47410 (time 1746138773.20 total 309.30 last 127.49) - open/close 48685 (time 1746138783.20 total 319.30 last 127.50) - open/close 49954 (time 1746138793.21 total 329.31 last 126.83) - open/close 51056 (time 1746138803.21 total 339.31 last 110.18) - open/close 52153 (time 1746138813.22 total 349.31 last 109.65) - open/close 53411 (time 1746138823.22 total 359.32 last 125.69) - open/close 54549 (time 1746138833.23 total 369.33 last 113.71) - open/close 55839 (time 1746138843.23 total 379.33 last 128.98) - open/close 57197 (time 1746138853.23 total 389.33 last 135.79) - open/close 58694 (time 1746138863.24 total 399.34 last 149.61) - open/close 60000 (time 1746138873.07 total 409.17 last 132.87) - open/close 61249 (time 1746138883.08 total 419.17 last 124.83) - open/close 62773 (time 1746138893.08 total 429.18 last 152.37) - open/close 64111 (time 1746138903.08 total 439.18 last 133.75) total: 64768 open/close in 444.30 seconds: 145.78 ops/second - /unlink 2302 (time 1746138919.97 total 10.00 last 230.15) - /unlink 4448 (time 1746138929.97 total 20.01 last 214.52) - /unlink 6775 (time 1746138939.97 total 30.01 last 232.67) - /unlink 8995 (time 1746138949.98 total 40.01 last 221.91) - /unlink 10000 (time 1746138954.85 total 44.88 last 206.22) - /unlink 12084 (time 1746138964.85 total 54.89 last 208.35) - /unlink 14303 (time 1746138974.86 total 64.89 last 221.85) - /unlink 16428 (time 1746138984.86 total 74.89 last 212.44) - /unlink 18440 (time 1746138994.86 total 84.90 last 201.12) - /unlink 20000 (time 1746139001.62 total 91.65 last 230.92) - /unlink 21946 (time 1746139011.62 total 101.65 last 194.57) - /unlink 23809 (time 1746139021.62 total 111.66 last 186.24) - /unlink 25879 (time 1746139031.62 total 121.66 last 206.96) - /unlink 27803 (time 1746139041.63 total 131.66 last 192.34) - /unlink 29805 (time 1746139051.63 total 141.66 last 200.14) - /unlink 31870 (time 1746139061.63 total 151.67 last 206.40) - /unlink 33840 (time 1746139071.64 total 161.67 last 196.91) - /unlink 35773 (time 1746139081.65 total 171.68 last 193.15) - /unlink 37788 (time 1746139091.65 total 181.68 last 201.45) - /unlink 39893 (time 1746139101.65 total 191.69 last 210.44) - /unlink 41772 (time 1746139111.65 total 201.69 last 187.85) - /unlink 43769 (time 1746139121.66 total 211.69 last 199.65) - /unlink 46267 (time 1746139131.66 total 221.69 last 249.74) - /unlink 48522 (time 1746139141.66 total 231.70 last 225.42) - /unlink 50000 (time 1746139148.58 total 238.62 last 213.63) - /unlink 52511 (time 1746139158.58 total 248.62 last 251.08) - /unlink 54861 (time 1746139168.59 total 258.62 last 234.90) - /unlink 57254 (time 1746139178.59 total 268.63 last 239.17) - /unlink 59717 (time 1746139188.59 total 278.63 last 246.24) - /unlink 62187 (time 1746139198.60 total 288.63 last 246.98) - /unlink 64649 (time 1746139208.60 total 298.63 last 246.13) total: 64768 /unlink in 299.09 seconds: 216.55 ops/second Waiting for MDT destroys to complete - open/close 1374 (time 1746139229.57 total 10.00 last 137.34) - open/close 2863 (time 1746139239.58 total 20.01 last 148.76) - open/close 4321 (time 1746139249.59 total 30.02 last 145.71) - open/close 5948 (time 1746139259.59 total 40.02 last 162.67) - open/close 7552 (time 1746139269.60 total 50.03 last 160.24) - open/close 8950 (time 1746139279.60 total 60.03 last 139.76) - open/close 10000 (time 1746139286.60 total 67.03 last 150.12) - open/close 11532 (time 1746139296.60 total 77.03 last 153.15) - open/close 12839 (time 1746139306.61 total 87.04 last 130.59) - open/close 14035 (time 1746139316.61 total 97.04 last 119.57) - open/close 15261 (time 1746139326.62 total 107.05 last 122.53) - open/close 16638 (time 1746139336.63 total 117.06 last 137.60) - open/close 18238 (time 1746139346.63 total 127.06 last 159.94) - open/close 19660 (time 1746139356.64 total 137.07 last 142.11) - open/close 20000 (time 1746139358.95 total 139.38 last 146.80) - open/close 21404 (time 1746139368.95 total 149.38 last 140.39) - open/close 22705 (time 1746139378.95 total 159.38 last 130.10) - open/close 24169 (time 1746139388.95 total 169.38 last 146.39) - open/close 25736 (time 1746139398.96 total 179.39 last 156.59) - open/close 27343 (time 1746139408.96 total 189.39 last 160.68) - open/close 28875 (time 1746139418.96 total 199.39 last 153.18) - open/close 30000 (time 1746139427.35 total 207.78 last 134.13) - open/close 31528 (time 1746139437.36 total 217.79 last 152.71) - open/close 33127 (time 1746139447.36 total 227.79 last 159.86) - open/close 34685 (time 1746139457.36 total 237.80 last 155.70) - open/close 36217 (time 1746139467.37 total 247.80 last 153.19) - open/close 37635 (time 1746139477.37 total 257.80 last 141.76) - open/close 39341 (time 1746139487.37 total 267.80 last 170.59) - open/close 40000 (time 1746139491.42 total 271.85 last 162.65) - open/close 41575 (time 1746139501.42 total 281.85 last 157.50) - open/close 43354 (time 1746139511.43 total 291.86 last 177.80) - open/close 44978 (time 1746139521.43 total 301.86 last 162.37) - open/close 46809 (time 1746139531.43 total 311.86 last 183.09) - open/close 48430 (time 1746139541.44 total 321.87 last 162.00) - open/close 50000 (time 1746139550.69 total 331.12 last 169.63) - open/close 51745 (time 1746139560.69 total 341.12 last 174.45) - open/close 53667 (time 1746139570.70 total 351.13 last 192.12) - open/close 55510 (time 1746139580.70 total 361.13 last 184.24) - open/close 57163 (time 1746139590.70 total 371.13 last 165.29) - open/close 58686 (time 1746139600.70 total 381.13 last 152.29) - open/close 60000 (time 1746139610.04 total 390.47 last 140.74) - open/close 61401 (time 1746139620.05 total 400.48 last 139.98) - open/close 62758 (time 1746139630.05 total 410.48 last 135.66) - open/close 64333 (time 1746139640.06 total 420.49 last 157.38) total: 64768 open/close in 423.91 seconds: 152.79 ops/second - /unlink 2089 (time 1746139655.44 total 10.00 last 208.88) - /unlink 4410 (time 1746139665.44 total 20.00 last 232.03) - /unlink 6577 (time 1746139675.44 total 30.01 last 216.58) - /unlink 8814 (time 1746139685.45 total 40.01 last 223.59) - /unlink 10000 (time 1746139690.70 total 45.26 last 225.93) - /unlink 12190 (time 1746139700.70 total 55.27 last 218.96) - /unlink 14257 (time 1746139710.70 total 65.27 last 206.61) - /unlink 16355 (time 1746139720.71 total 75.28 last 209.66) - /unlink 18416 (time 1746139730.71 total 85.28 last 206.09) - /unlink 20000 (time 1746139737.22 total 91.78 last 243.50) - /unlink 22934 (time 1746139747.22 total 101.78 last 293.36) - /unlink 25192 (time 1746139757.22 total 111.79 last 225.72) - /unlink 27651 (time 1746139767.23 total 121.79 last 245.82) - /unlink 29964 (time 1746139777.23 total 131.79 last 231.25) - /unlink 32015 (time 1746139787.23 total 141.79 last 205.10) - /unlink 34159 (time 1746139797.23 total 151.79 last 214.37) - /unlink 36640 (time 1746139807.23 total 161.79 last 248.07) - /unlink 38980 (time 1746139817.23 total 171.80 last 233.93) - /unlink 40000 (time 1746139821.08 total 175.64 last 265.37) - /unlink 42440 (time 1746139831.08 total 185.64 last 243.94) - /unlink 45162 (time 1746139841.08 total 195.65 last 272.16) - /unlink 48058 (time 1746139851.09 total 205.65 last 289.46) - /unlink 50000 (time 1746139859.31 total 213.87 last 236.17) - /unlink 52677 (time 1746139869.31 total 223.87 last 267.70) - /unlink 56127 (time 1746139879.31 total 233.88 last 344.91) - /unlink 59011 (time 1746139889.31 total 243.88 last 288.34) - /unlink 60000 (time 1746139893.70 total 248.26 last 225.54) - /unlink 62631 (time 1746139903.70 total 258.26 last 263.05) total: 64768 /unlink in 265.15 seconds: 244.27 ops/second Waiting for MDT destroys to complete - open/close 1843 (time 1746139930.41 total 10.01 last 184.20) - open/close 3338 (time 1746139940.42 total 20.01 last 149.43) - open/close 4738 (time 1746139950.42 total 30.01 last 140.00) - open/close 6200 (time 1746139960.42 total 40.02 last 146.10) - open/close 7849 (time 1746139970.43 total 50.02 last 164.80) - open/close 9183 (time 1746139980.44 total 60.03 last 133.31) - open/close 10000 (time 1746139986.86 total 66.46 last 127.15) - open/close 11344 (time 1746139996.87 total 76.46 last 134.32) - open/close 12554 (time 1746140006.87 total 86.46 last 120.99) - open/close 13791 (time 1746140016.87 total 96.46 last 123.68) - open/close 15170 (time 1746140026.88 total 106.47 last 137.83) - open/close 16498 (time 1746140036.88 total 116.47 last 132.78) - open/close 17829 (time 1746140046.88 total 126.47 last 133.10) - open/close 19232 (time 1746140056.88 total 136.47 last 140.25) - open/close 20000 (time 1746140063.64 total 143.23 last 113.61) - open/close 21531 (time 1746140073.64 total 153.24 last 153.03) - open/close 23012 (time 1746140083.66 total 163.25 last 147.93) - open/close 24214 (time 1746140093.66 total 173.25 last 120.19) - open/close 25592 (time 1746140103.66 total 183.25 last 137.80) - open/close 26956 (time 1746140113.66 total 193.25 last 136.37) - open/close 28228 (time 1746140123.67 total 203.26 last 127.07) - open/close 29459 (time 1746140133.67 total 213.26 last 123.09) - open/close 30000 (time 1746140137.71 total 217.31 last 133.88) - open/close 31447 (time 1746140147.72 total 227.32 last 144.52) - open/close 32657 (time 1746140157.73 total 237.32 last 120.95) - open/close 33938 (time 1746140167.74 total 247.33 last 127.99) - open/close 35287 (time 1746140177.74 total 257.33 last 134.87) - open/close 36694 (time 1746140187.74 total 267.34 last 140.63) - open/close 38006 (time 1746140197.75 total 277.34 last 131.14) - open/close 39335 (time 1746140207.75 total 287.35 last 132.84) - open/close 40000 (time 1746140212.08 total 291.67 last 153.69) - open/close 41340 (time 1746140222.09 total 301.68 last 133.88) - open/close 42649 (time 1746140232.09 total 311.69 last 130.82) - open/close 43898 (time 1746140242.10 total 321.70 last 124.78) - open/close 45170 (time 1746140252.11 total 331.70 last 127.13) - open/close 46345 (time 1746140262.11 total 341.71 last 117.44) - open/close 47569 (time 1746140272.12 total 351.72 last 122.30) - open/close 48894 (time 1746140282.13 total 361.72 last 132.39) - open/close 50000 (time 1746140290.48 total 370.07 last 132.52) - open/close 51254 (time 1746140300.48 total 380.07 last 125.36) - open/close 52659 (time 1746140310.48 total 390.07 last 140.50) - open/close 54012 (time 1746140320.48 total 400.08 last 135.28) - open/close 55293 (time 1746140330.49 total 410.09 last 127.94) - open/close 56609 (time 1746140340.50 total 420.09 last 131.55) - open/close 57947 (time 1746140350.50 total 430.09 last 133.77) - open/close 59231 (time 1746140360.50 total 440.10 last 128.38) - open/close 60000 (time 1746140366.74 total 446.34 last 123.19) - open/close 61389 (time 1746140376.74 total 456.34 last 138.89) - open/close 62547 (time 1746140386.75 total 466.34 last 115.78) - open/close 63841 (time 1746140396.75 total 476.34 last 129.37) total: 64768 open/close in 483.17 seconds: 134.05 ops/second - /unlink 2159 (time 1746140415.37 total 10.00 last 215.86) - /unlink 4510 (time 1746140425.37 total 20.00 last 235.03) - /unlink 7076 (time 1746140435.38 total 30.01 last 256.51) - /unlink 9238 (time 1746140445.38 total 40.01 last 216.18) - /unlink 10000 (time 1746140448.91 total 43.54 last 215.92) - /unlink 12260 (time 1746140458.93 total 53.56 last 225.46) - /unlink 14477 (time 1746140468.93 total 63.56 last 221.65) - /unlink 16812 (time 1746140478.93 total 73.57 last 233.45) - /unlink 19172 (time 1746140488.94 total 83.57 last 235.89) - /unlink 20000 (time 1746140492.63 total 87.26 last 224.59) - /unlink 22277 (time 1746140502.63 total 97.26 last 227.66) - /unlink 24707 (time 1746140512.63 total 107.26 last 242.95) - /unlink 26717 (time 1746140522.64 total 117.27 last 200.85) - /unlink 28878 (time 1746140532.64 total 127.27 last 216.09) - /unlink 30000 (time 1746140538.13 total 132.76 last 204.22) - /unlink 32269 (time 1746140548.13 total 142.77 last 226.84) - /unlink 34371 (time 1746140558.13 total 152.77 last 210.20) - /unlink 36436 (time 1746140568.14 total 162.77 last 206.47) - /unlink 38662 (time 1746140578.14 total 172.77 last 222.52) - /unlink 40000 (time 1746140583.90 total 178.53 last 232.42) - /unlink 42476 (time 1746140593.90 total 188.53 last 247.59) - /unlink 45090 (time 1746140603.90 total 198.53 last 261.40) - /unlink 47396 (time 1746140613.90 total 208.53 last 230.56) - /unlink 50000 (time 1746140623.89 total 218.52 last 260.73) - /unlink 52272 (time 1746140633.89 total 228.52 last 227.17) - /unlink 54564 (time 1746140643.89 total 238.52 last 229.10) - /unlink 58061 (time 1746140653.89 total 248.53 last 349.62) - /unlink 60000 (time 1746140660.49 total 255.12 last 294.00) - /unlink 62825 (time 1746140670.49 total 265.12 last 282.46) total: 64768 /unlink in 272.06 seconds: 238.07 ops/second Waiting for MDT destroys to complete - open/close 1555 (time 1746140697.92 total 10.00 last 155.45) - open/close 3136 (time 1746140707.93 total 20.01 last 158.05) - open/close 5080 (time 1746140717.93 total 30.01 last 194.31) - open/close 6694 (time 1746140727.93 total 40.01 last 161.34) - open/close 8152 (time 1746140737.94 total 50.02 last 145.79) - open/close 9634 (time 1746140747.94 total 60.02 last 148.09) - open/close 10000 (time 1746140750.60 total 62.68 last 137.48) - open/close 11270 (time 1746140760.61 total 72.69 last 127.00) - open/close 12639 (time 1746140770.61 total 82.69 last 136.85) - open/close 14196 (time 1746140780.61 total 92.69 last 155.66) - open/close 15535 (time 1746140790.62 total 102.70 last 133.82) - open/close 16938 (time 1746140800.62 total 112.70 last 140.25) - open/close 18340 (time 1746140810.63 total 122.71 last 140.07) - open/close 19900 (time 1746140820.63 total 132.71 last 156.00) - open/close 21574 (time 1746140830.63 total 142.71 last 167.34) - open/close 23001 (time 1746140840.63 total 152.71 last 142.68) - open/close 24587 (time 1746140850.64 total 162.72 last 158.52) - open/close 26049 (time 1746140860.64 total 172.72 last 146.18) - open/close 27521 (time 1746140870.64 total 182.72 last 147.16) - open/close 29033 (time 1746140880.65 total 192.73 last 151.17) - open/close 30000 (time 1746140887.75 total 199.83 last 136.20) - open/close 31489 (time 1746140897.75 total 209.83 last 148.80) - open/close 33034 (time 1746140907.76 total 219.84 last 154.41) - open/close 34581 (time 1746140917.76 total 229.84 last 154.60) - open/close 36337 (time 1746140927.77 total 239.85 last 175.48) - open/close 37843 (time 1746140937.77 total 249.85 last 150.57) - open/close 39343 (time 1746140947.78 total 259.86 last 149.92) - open/close 40000 (time 1746140951.29 total 263.37 last 186.98) - open/close 41644 (time 1746140961.30 total 273.38 last 164.30) - open/close 43021 (time 1746140971.30 total 283.38 last 137.69) - open/close 44412 (time 1746140981.30 total 293.38 last 139.05) - open/close 45777 (time 1746140991.31 total 303.39 last 136.45) - open/close 47188 (time 1746141001.31 total 313.39 last 141.02) - open/close 48534 (time 1746141011.32 total 323.40 last 134.51) - open/close 50000 (time 1746141021.32 total 333.40 last 146.64) - open/close 51341 (time 1746141031.33 total 343.41 last 133.95) - open/close 52643 (time 1746141041.33 total 353.41 last 130.12) - open/close 54004 (time 1746141051.34 total 363.42 last 135.97) - open/close 55195 (time 1746141061.35 total 373.43 last 119.06) - open/close 56348 (time 1746141071.35 total 383.43 last 115.30) - open/close 57555 (time 1746141081.35 total 393.43 last 120.67) - open/close 58787 (time 1746141091.35 total 403.43 last 123.17) - open/close 59921 (time 1746141101.35 total 413.43 last 113.38) - open/close 61135 (time 1746141111.36 total 423.44 last 121.32) - open/close 62381 (time 1746141121.36 total 433.44 last 124.55) - open/close 63550 (time 1746141131.37 total 443.45 last 116.81) total: 64768 open/close in 452.94 seconds: 143.00 ops/second - /unlink 2434 (time 1746141152.60 total 10.00 last 243.31) - /unlink 4692 (time 1746141162.61 total 20.01 last 225.72) - /unlink 7279 (time 1746141172.61 total 30.01 last 258.68) - /unlink 9722 (time 1746141182.61 total 40.02 last 244.12) - /unlink 11951 (time 1746141192.62 total 50.02 last 222.81) - /unlink 14288 (time 1746141202.62 total 60.02 last 233.66) - /unlink 16579 (time 1746141212.62 total 70.03 last 228.96) - /unlink 18927 (time 1746141222.63 total 80.04 last 234.59) - /unlink 20000 (time 1746141226.54 total 83.94 last 274.49) - /unlink 22487 (time 1746141236.55 total 93.95 last 248.62) - /unlink 25211 (time 1746141246.55 total 103.95 last 272.30) - /unlink 27832 (time 1746141256.55 total 113.95 last 262.04) - /unlink 30000 (time 1746141265.55 total 122.95 last 241.04) - /unlink 32338 (time 1746141275.55 total 132.95 last 233.72) - /unlink 34862 (time 1746141285.55 total 142.95 last 252.40) - /unlink 37467 (time 1746141295.55 total 152.95 last 260.43) - /unlink 40000 (time 1746141305.09 total 162.50 last 265.47) - /unlink 42780 (time 1746141315.09 total 172.50 last 277.98) - /unlink 45164 (time 1746141325.10 total 182.50 last 238.37) - /unlink 47625 (time 1746141335.10 total 192.50 last 246.07) - /unlink 50000 (time 1746141343.37 total 200.77 last 287.12) - /unlink 52597 (time 1746141353.37 total 210.77 last 259.67) - /unlink 54921 (time 1746141363.37 total 220.77 last 232.39) - /unlink 57765 (time 1746141373.37 total 230.77 last 284.33) - /unlink 60000 (time 1746141382.67 total 240.07 last 240.40) - /unlink 62524 (time 1746141392.67 total 250.07 last 252.36) total: 64768 /unlink in 258.59 seconds: 250.46 ops/second Waiting for MDT destroys to complete - open/close 1798 (time 1746141423.02 total 10.00 last 179.77) - open/close 3708 (time 1746141433.02 total 20.00 last 190.96) - open/close 5534 (time 1746141443.02 total 30.00 last 182.59) - open/close 7152 (time 1746141453.03 total 40.01 last 161.77) - open/close 8795 (time 1746141463.03 total 50.01 last 164.24) - open/close 10000 (time 1746141469.56 total 56.54 last 184.60) - open/close 12022 (time 1746141479.56 total 66.54 last 202.14) - open/close 13896 (time 1746141489.57 total 76.54 last 187.32) - open/close 16058 (time 1746141499.57 total 86.55 last 216.17) - open/close 18514 (time 1746141509.57 total 96.55 last 245.59) - open/close 20000 (time 1746141516.66 total 103.64 last 209.46) - open/close 21951 (time 1746141526.66 total 113.64 last 195.07) - open/close 24028 (time 1746141536.66 total 123.64 last 207.68) - open/close 26475 (time 1746141546.67 total 133.64 last 244.67) - open/close 28789 (time 1746141556.67 total 143.65 last 231.38) - open/close 30000 (time 1746141562.08 total 149.06 last 223.84) - open/close 32327 (time 1746141572.08 total 159.06 last 232.60) - open/close 34241 (time 1746141582.08 total 169.06 last 191.40) - open/close 35997 (time 1746141592.08 total 179.06 last 175.55) - open/close 37970 (time 1746141602.09 total 189.07 last 197.24) - open/close 39901 (time 1746141612.09 total 199.07 last 193.04) - open/close 41882 (time 1746141622.09 total 209.07 last 198.09) - open/close 43757 (time 1746141632.09 total 219.07 last 187.45) - open/close 46261 (time 1746141642.09 total 229.07 last 250.38) - open/close 49041 (time 1746141652.09 total 239.07 last 277.96) - open/close 50000 (time 1746141657.00 total 243.98 last 195.56) - open/close 52080 (time 1746141667.00 total 253.98 last 207.91) - open/close 54336 (time 1746141677.01 total 263.99 last 225.45) - open/close 56315 (time 1746141687.01 total 273.99 last 197.83) - open/close 58369 (time 1746141697.02 total 283.99 last 205.36) - open/close 60000 (time 1746141704.80 total 291.78 last 209.41) - open/close 61682 (time 1746141714.81 total 301.79 last 168.12) - open/close 63328 (time 1746141724.81 total 311.79 last 164.52) total: 64768 open/close in 319.06 seconds: 202.99 ops/second - /unlink 2973 (time 1746141743.70 total 10.00 last 297.29) - /unlink 5646 (time 1746141753.71 total 20.00 last 267.23) - /unlink 8086 (time 1746141763.71 total 30.00 last 243.96) - /unlink 10000 (time 1746141770.91 total 37.21 last 265.71) - /unlink 13820 (time 1746141780.94 total 47.24 last 380.76) - /unlink 16552 (time 1746141790.95 total 57.25 last 273.06) - /unlink 19458 (time 1746141800.96 total 67.25 last 290.33) - /unlink 20000 (time 1746141802.98 total 69.27 last 268.32) - /unlink 22838 (time 1746141812.98 total 79.28 last 283.72) - /unlink 26070 (time 1746141822.98 total 89.28 last 323.20) - /unlink 29237 (time 1746141832.98 total 99.28 last 316.63) - /unlink 30000 (time 1746141835.23 total 101.53 last 339.76) - /unlink 33577 (time 1746141845.23 total 111.53 last 357.68) - /unlink 36945 (time 1746141855.23 total 121.53 last 336.73) - /unlink 39733 (time 1746141865.23 total 131.53 last 278.78) - /unlink 42445 (time 1746141875.24 total 141.53 last 271.10) - /unlink 45298 (time 1746141885.24 total 151.53 last 285.30) - /unlink 48073 (time 1746141895.24 total 161.54 last 277.36) - /unlink 50000 (time 1746141901.42 total 167.72 last 311.68) - /unlink 53599 (time 1746141911.43 total 177.72 last 359.80) - /unlink 56632 (time 1746141921.43 total 187.73 last 303.21) - /unlink 59660 (time 1746141931.43 total 197.73 last 302.73) - /unlink 62720 (time 1746141941.44 total 207.74 last 305.77) total: 64768 /unlink in 214.69 seconds: 301.69 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg331-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg331-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg331-server stop mds service on oleg331-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg331-server unloading modules via unload_modules_local on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-client: ssh exited with exit code 2 pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 PASS 106 (3574s) Stopping clients: oleg331-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg331-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg331-client: oleg331-server: ssh exited with exit code 2 oleg331-server: oleg331-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg331-server' oleg331-server: oleg331-server.virtnet: executing load_modules_local oleg331-server: Loading modules from /home/green/git/lustre-release/lustre oleg331-server: detected 4 online CPUs by sysfs oleg331-server: MODOPTS_LIBCFS= oleg331-server: Force libcfs to create 2 CPU partitions oleg331-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory oleg331-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg331-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3695 sec ========== 19:26:54 (1746142014) === conf-sanity: start cleanup 19:26:55 (1746142015) === === conf-sanity: finish cleanup 19:26:57 (1746142017) ===