-----============= acceptance-small: conf-sanity ============----- Thu May 1 18:00:15 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg430-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg430-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg430-server oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg430-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg430-server: oleg430-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg430-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:01:39 (1746136899) Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg430-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg430-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg430-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg430-server: oleg430-server.virtnet: executing set_default_debug -1 all pdsh@oleg430-client: oleg430-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg430-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg430-client.virtnet: -o user_xattr,flock 192.168.204.130@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1357 (time 1746136954.46 total 10.00 last 135.68) - open/close 2617 (time 1746136964.47 total 20.00 last 125.99) - open/close 3807 (time 1746136974.47 total 30.01 last 118.95) - open/close 5100 (time 1746136984.47 total 40.01 last 129.24) - open/close 6288 (time 1746136994.48 total 50.01 last 118.78) - open/close 7317 (time 1746137004.48 total 60.02 last 102.85) - open/close 8397 (time 1746137014.48 total 70.02 last 107.99) - open/close 9803 (time 1746137024.48 total 80.02 last 140.56) - open/close 11128 (time 1746137034.49 total 90.02 last 132.46) - open/close 12467 (time 1746137044.49 total 100.03 last 133.89) - open/close 13952 (time 1746137054.49 total 110.03 last 148.48) - open/close 15411 (time 1746137064.50 total 120.03 last 145.81) - open/close 16827 (time 1746137074.50 total 130.04 last 141.50) - open/close 18357 (time 1746137084.50 total 140.04 last 152.96) - open/close 19719 (time 1746137094.51 total 150.04 last 136.17) - open/close 20000 (time 1746137096.67 total 152.21 last 129.71) - open/close 21472 (time 1746137106.68 total 162.22 last 147.11) - open/close 22758 (time 1746137116.69 total 172.23 last 128.47) - open/close 24022 (time 1746137126.70 total 182.24 last 126.29) - open/close 25184 (time 1746137136.70 total 192.24 last 116.15) - open/close 26482 (time 1746137146.71 total 202.24 last 129.74) - open/close 27843 (time 1746137156.71 total 212.25 last 136.08) - open/close 29019 (time 1746137166.71 total 222.25 last 117.54) - open/close 30000 (time 1746137173.95 total 229.49 last 135.54) - open/close 31310 (time 1746137183.95 total 239.49 last 130.98) - open/close 32973 (time 1746137193.95 total 249.49 last 166.28) - open/close 34385 (time 1746137203.96 total 259.50 last 141.06) - open/close 35718 (time 1746137213.97 total 269.50 last 133.27) - open/close 37235 (time 1746137223.97 total 279.51 last 151.61) - open/close 38618 (time 1746137233.97 total 289.51 last 138.28) - open/close 40000 (time 1746137243.70 total 299.24 last 142.12) - open/close 41349 (time 1746137253.71 total 309.24 last 134.80) - open/close 42934 (time 1746137263.71 total 319.24 last 158.47) - open/close 44596 (time 1746137273.71 total 329.25 last 166.13) - open/close 46012 (time 1746137283.72 total 339.26 last 141.48) - open/close 47464 (time 1746137293.72 total 349.26 last 145.15) - open/close 49113 (time 1746137303.72 total 359.26 last 164.90) - open/close 50000 (time 1746137309.86 total 365.39 last 144.59) - open/close 51381 (time 1746137319.86 total 375.40 last 138.07) - open/close 52960 (time 1746137329.86 total 385.40 last 157.87) - open/close 54543 (time 1746137339.86 total 395.40 last 158.27) - open/close 56055 (time 1746137349.87 total 405.40 last 151.17) - open/close 57571 (time 1746137359.87 total 415.41 last 151.54) - open/close 59246 (time 1746137369.87 total 425.41 last 167.47) - open/close 60000 (time 1746137374.97 total 430.51 last 147.77) - open/close 61759 (time 1746137384.98 total 440.51 last 175.83) - open/close 63385 (time 1746137394.98 total 450.52 last 162.56) total: 64768 open/close in 459.46 seconds: 140.97 ops/second - /unlink 2772 (time 1746137415.48 total 10.00 last 277.16) - /unlink 5796 (time 1746137425.49 total 20.01 last 302.23) - /unlink 8209 (time 1746137435.50 total 30.01 last 241.17) - /unlink 10000 (time 1746137443.15 total 37.66 last 234.04) - /unlink 12591 (time 1746137453.15 total 47.67 last 259.03) - /unlink 15073 (time 1746137463.15 total 57.67 last 248.20) - /unlink 17593 (time 1746137473.15 total 67.67 last 251.94) - /unlink 20000 (time 1746137482.88 total 77.40 last 247.43) - /unlink 22766 (time 1746137492.89 total 87.40 last 276.43) - /unlink 25423 (time 1746137502.89 total 97.41 last 265.65) - /unlink 28702 (time 1746137512.89 total 107.41 last 327.84) - /unlink 30000 (time 1746137516.86 total 111.38 last 326.72) - /unlink 33636 (time 1746137526.87 total 121.38 last 363.55) - /unlink 37108 (time 1746137536.87 total 131.39 last 347.04) - /unlink 39728 (time 1746137546.87 total 141.39 last 261.97) - /unlink 42196 (time 1746137556.87 total 151.39 last 246.72) - /unlink 44357 (time 1746137566.88 total 161.40 last 216.02) - /unlink 46941 (time 1746137576.88 total 171.40 last 258.39) - /unlink 49482 (time 1746137586.88 total 181.40 last 254.01) - /unlink 52157 (time 1746137596.88 total 191.40 last 267.45) - /unlink 54564 (time 1746137606.89 total 201.40 last 240.64) - /unlink 57418 (time 1746137616.89 total 211.41 last 285.32) - /unlink 59923 (time 1746137626.89 total 221.41 last 250.44) - /unlink 63042 (time 1746137636.89 total 231.41 last 311.82) total: 64768 /unlink in 238.83 seconds: 271.18 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1167 (time 1746137671.17 total 10.00 last 116.68) - open/close 2599 (time 1746137681.17 total 20.01 last 143.13) - open/close 4196 (time 1746137691.18 total 30.01 last 159.57) - open/close 5678 (time 1746137701.18 total 40.02 last 148.14) - open/close 7339 (time 1746137711.19 total 50.02 last 166.05) - open/close 8850 (time 1746137721.19 total 60.02 last 151.08) - open/close 10000 (time 1746137728.46 total 67.30 last 158.11) - open/close 11695 (time 1746137738.47 total 77.30 last 169.39) - open/close 13135 (time 1746137748.47 total 87.30 last 143.97) - open/close 14896 (time 1746137758.47 total 97.31 last 176.06) - open/close 16539 (time 1746137768.47 total 107.31 last 164.26) - open/close 18108 (time 1746137778.48 total 117.32 last 156.77) - open/close 19751 (time 1746137788.48 total 127.32 last 164.26) - open/close 21666 (time 1746137798.49 total 137.33 last 191.33) - open/close 23167 (time 1746137808.50 total 147.34 last 149.98) - open/close 24489 (time 1746137818.50 total 157.34 last 132.15) - open/close 25937 (time 1746137828.51 total 167.35 last 144.69) - open/close 27165 (time 1746137838.52 total 177.35 last 122.76) - open/close 28915 (time 1746137848.52 total 187.36 last 174.90) - open/close 30000 (time 1746137855.53 total 194.37 last 154.73) - open/close 31620 (time 1746137865.54 total 204.38 last 161.85) - open/close 33001 (time 1746137875.54 total 214.38 last 138.07) - open/close 34349 (time 1746137885.55 total 224.39 last 134.72) - open/close 36152 (time 1746137895.55 total 234.39 last 180.26) - open/close 37547 (time 1746137905.56 total 244.40 last 139.39) - open/close 39179 (time 1746137915.56 total 254.40 last 163.16) - open/close 40000 (time 1746137920.44 total 259.27 last 168.35) - open/close 41849 (time 1746137930.44 total 269.28 last 184.83) - open/close 43841 (time 1746137940.45 total 279.28 last 199.16) - open/close 45646 (time 1746137950.45 total 289.28 last 180.44) - open/close 47502 (time 1746137960.45 total 299.29 last 185.58) - open/close 49640 (time 1746137970.45 total 309.29 last 213.73) - open/close 51730 (time 1746137980.45 total 319.29 last 208.99) - open/close 53717 (time 1746137990.46 total 329.30 last 198.57) - open/close 55359 (time 1746138000.47 total 339.30 last 164.12) - open/close 57171 (time 1746138010.47 total 349.30 last 181.13) - open/close 58933 (time 1746138020.47 total 359.30 last 176.20) - open/close 60000 (time 1746138025.91 total 364.74 last 196.22) - open/close 61992 (time 1746138035.91 total 374.74 last 199.19) - open/close 63574 (time 1746138045.92 total 384.75 last 158.06) total: 64768 open/close in 392.40 seconds: 165.06 ops/second - /unlink 2801 (time 1746138064.78 total 10.00 last 280.01) - /unlink 5044 (time 1746138074.79 total 20.01 last 224.14) - /unlink 8289 (time 1746138084.79 total 30.01 last 324.44) - /unlink 10000 (time 1746138090.17 total 35.39 last 317.90) - /unlink 12948 (time 1746138100.18 total 45.40 last 294.66) - /unlink 16240 (time 1746138110.18 total 55.40 last 329.17) - /unlink 19134 (time 1746138120.18 total 65.40 last 289.31) - /unlink 20000 (time 1746138122.44 total 67.66 last 384.13) - /unlink 23570 (time 1746138132.44 total 77.66 last 356.90) - /unlink 27432 (time 1746138142.44 total 87.66 last 386.14) - /unlink 30000 (time 1746138151.72 total 96.94 last 276.69) - /unlink 32800 (time 1746138161.72 total 106.95 last 279.95) - /unlink 35504 (time 1746138171.73 total 116.95 last 270.36) - /unlink 38378 (time 1746138181.73 total 126.95 last 287.36) - /unlink 40000 (time 1746138188.14 total 133.36 last 252.87) - /unlink 42572 (time 1746138198.14 total 143.36 last 257.18) - /unlink 45177 (time 1746138208.14 total 153.37 last 260.45) - /unlink 48303 (time 1746138218.15 total 163.37 last 312.54) - /unlink 50000 (time 1746138223.65 total 168.88 last 308.09) - /unlink 53547 (time 1746138233.66 total 178.88 last 354.66) - /unlink 57841 (time 1746138243.66 total 188.88 last 429.29) - /unlink 60000 (time 1746138249.11 total 194.33 last 395.94) - /unlink 63064 (time 1746138259.11 total 204.33 last 306.39) total: 64768 /unlink in 208.63 seconds: 310.45 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1826 (time 1746138290.98 total 10.01 last 182.50) - open/close 3257 (time 1746138300.99 total 20.01 last 143.04) - open/close 4661 (time 1746138310.99 total 30.01 last 140.35) - open/close 6103 (time 1746138321.00 total 40.02 last 144.11) - open/close 7523 (time 1746138331.00 total 50.02 last 141.96) - open/close 8761 (time 1746138341.00 total 60.03 last 123.75) - open/close 10000 (time 1746138348.85 total 67.88 last 157.81) - open/close 11728 (time 1746138358.86 total 77.88 last 172.71) - open/close 13437 (time 1746138368.86 total 87.88 last 170.88) - open/close 14901 (time 1746138378.86 total 97.89 last 146.35) - open/close 16617 (time 1746138388.87 total 107.89 last 171.54) - open/close 18534 (time 1746138398.87 total 117.89 last 191.66) - open/close 20000 (time 1746138406.29 total 125.31 last 197.59) - open/close 22216 (time 1746138416.29 total 135.32 last 221.51) - open/close 23945 (time 1746138426.30 total 145.32 last 172.85) - open/close 25931 (time 1746138436.30 total 155.32 last 198.59) - open/close 27921 (time 1746138446.30 total 165.32 last 198.98) - open/close 29725 (time 1746138456.31 total 175.33 last 180.27) - open/close 31446 (time 1746138466.31 total 185.33 last 172.08) - open/close 33376 (time 1746138476.31 total 195.33 last 192.93) - open/close 34972 (time 1746138486.31 total 205.34 last 159.56) - open/close 36595 (time 1746138496.32 total 215.34 last 162.23) - open/close 38037 (time 1746138506.32 total 225.34 last 144.15) - open/close 39504 (time 1746138516.32 total 235.35 last 146.66) - open/close 40000 (time 1746138519.72 total 238.74 last 146.05) - open/close 41478 (time 1746138529.72 total 248.74 last 147.77) - open/close 42879 (time 1746138539.73 total 258.76 last 139.92) - open/close 44479 (time 1746138549.74 total 268.76 last 159.88) - open/close 45757 (time 1746138559.74 total 278.77 last 127.78) - open/close 47355 (time 1746138569.75 total 288.77 last 159.75) - open/close 48867 (time 1746138579.75 total 298.77 last 151.15) - open/close 50000 (time 1746138588.25 total 307.27 last 133.36) - open/close 51322 (time 1746138598.25 total 317.27 last 132.17) - open/close 52876 (time 1746138608.25 total 327.27 last 155.35) - open/close 54068 (time 1746138618.26 total 337.28 last 119.15) - open/close 55538 (time 1746138628.26 total 347.28 last 146.98) - open/close 57115 (time 1746138638.26 total 357.29 last 157.59) - open/close 58870 (time 1746138648.27 total 367.29 last 175.49) - open/close 60000 (time 1746138653.61 total 372.63 last 211.51) - open/close 61649 (time 1746138663.61 total 382.64 last 164.79) - open/close 63275 (time 1746138673.62 total 392.64 last 162.54) total: 64768 open/close in 402.25 seconds: 161.01 ops/second - /unlink 2134 (time 1746138694.71 total 10.00 last 213.31) - /unlink 4449 (time 1746138704.71 total 20.01 last 231.47) - /unlink 6396 (time 1746138714.71 total 30.01 last 194.65) - /unlink 8233 (time 1746138724.71 total 40.01 last 183.68) - /unlink 10000 (time 1746138733.35 total 48.65 last 204.58) - /unlink 12449 (time 1746138743.35 total 58.65 last 244.82) - /unlink 14921 (time 1746138753.36 total 68.65 last 247.14) - /unlink 17425 (time 1746138763.36 total 78.65 last 250.36) - /unlink 19485 (time 1746138773.36 total 88.65 last 205.98) - /unlink 20000 (time 1746138775.85 total 91.15 last 206.76) - /unlink 22706 (time 1746138785.85 total 101.15 last 270.50) - /unlink 25075 (time 1746138795.86 total 111.15 last 236.82) - /unlink 27206 (time 1746138805.86 total 121.15 last 213.09) - /unlink 29498 (time 1746138815.86 total 131.15 last 229.15) - /unlink 30000 (time 1746138818.40 total 133.69 last 197.67) - /unlink 32125 (time 1746138828.40 total 143.70 last 212.44) - /unlink 34314 (time 1746138838.40 total 153.70 last 218.90) - /unlink 36685 (time 1746138848.40 total 163.70 last 237.08) - /unlink 38856 (time 1746138858.41 total 173.70 last 217.02) - /unlink 40000 (time 1746138863.32 total 178.62 last 232.58) - /unlink 42099 (time 1746138873.33 total 188.62 last 209.85) - /unlink 44075 (time 1746138883.33 total 198.62 last 197.58) - /unlink 46211 (time 1746138893.33 total 208.62 last 213.58) - /unlink 48090 (time 1746138903.33 total 218.63 last 187.84) - /unlink 50000 (time 1746138912.83 total 228.13 last 201.08) - /unlink 51779 (time 1746138922.83 total 238.13 last 177.86) - /unlink 53557 (time 1746138932.83 total 248.13 last 177.77) - /unlink 55955 (time 1746138942.84 total 258.13 last 239.76) - /unlink 58288 (time 1746138952.84 total 268.14 last 233.19) - /unlink 60000 (time 1746138959.77 total 275.07 last 247.01) - /unlink 62428 (time 1746138969.77 total 285.07 last 242.79) - /unlink 64286 (time 1746138979.78 total 295.07 last 185.66) total: 64768 /unlink in 297.76 seconds: 217.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1293 (time 1746139010.75 total 10.01 last 129.20) - open/close 2534 (time 1746139020.76 total 20.02 last 123.96) - open/close 3957 (time 1746139030.76 total 30.02 last 142.30) - open/close 5179 (time 1746139040.77 total 40.03 last 122.12) - open/close 6476 (time 1746139050.77 total 50.03 last 129.62) - open/close 7702 (time 1746139060.78 total 60.03 last 122.56) - open/close 8715 (time 1746139070.79 total 70.04 last 101.21) - open/close 9877 (time 1746139080.79 total 80.05 last 116.17) - open/close 11195 (time 1746139090.79 total 90.05 last 131.76) - open/close 12646 (time 1746139100.80 total 100.05 last 145.01) - open/close 14068 (time 1746139110.80 total 110.06 last 142.17) - open/close 15821 (time 1746139120.80 total 120.06 last 175.22) - open/close 17764 (time 1746139130.81 total 130.07 last 194.18) - open/close 19049 (time 1746139140.81 total 140.07 last 128.45) - open/close 20000 (time 1746139148.43 total 147.69 last 124.84) - open/close 21349 (time 1746139158.44 total 157.70 last 134.80) - open/close 22911 (time 1746139168.44 total 167.70 last 156.16) - open/close 24243 (time 1746139178.45 total 177.71 last 133.03) - open/close 25326 (time 1746139188.46 total 187.72 last 108.26) - open/close 26678 (time 1746139198.46 total 197.72 last 135.14) - open/close 27993 (time 1746139208.47 total 207.73 last 131.43) - open/close 29183 (time 1746139218.47 total 217.73 last 118.99) - open/close 30000 (time 1746139224.61 total 223.87 last 133.10) - open/close 31336 (time 1746139234.61 total 233.87 last 133.58) - open/close 32572 (time 1746139244.62 total 243.87 last 123.51) - open/close 33948 (time 1746139254.62 total 253.88 last 137.57) - open/close 35154 (time 1746139264.62 total 263.88 last 120.57) - open/close 36696 (time 1746139274.62 total 273.88 last 154.15) - open/close 38362 (time 1746139284.63 total 283.89 last 166.44) - open/close 39610 (time 1746139294.64 total 293.90 last 124.75) - open/close 40000 (time 1746139298.43 total 297.69 last 102.78) - open/close 41063 (time 1746139308.44 total 307.70 last 106.22) - open/close 42160 (time 1746139318.45 total 317.71 last 109.60) - open/close 43536 (time 1746139328.45 total 327.71 last 137.58) - open/close 44985 (time 1746139338.46 total 337.72 last 144.78) - open/close 46245 (time 1746139348.46 total 347.72 last 125.95) - open/close 47473 (time 1746139358.46 total 357.72 last 122.77) - open/close 48563 (time 1746139368.48 total 367.73 last 108.88) - open/close 49671 (time 1746139378.48 total 377.74 last 110.75) - open/close 50000 (time 1746139381.11 total 380.37 last 125.14) - open/close 51142 (time 1746139391.12 total 390.37 last 114.14) - open/close 52559 (time 1746139401.12 total 400.38 last 141.65) - open/close 53841 (time 1746139411.12 total 410.38 last 128.18) - open/close 55314 (time 1746139421.12 total 420.38 last 147.25) - open/close 56525 (time 1746139431.13 total 430.39 last 121.01) - open/close 57697 (time 1746139441.14 total 440.40 last 117.10) - open/close 58760 (time 1746139451.15 total 450.40 last 106.24) - open/close 59889 (time 1746139461.15 total 460.41 last 112.82) - open/close 60945 (time 1746139471.16 total 470.42 last 105.55) - open/close 62093 (time 1746139481.16 total 480.42 last 114.73) - open/close 63055 (time 1746139491.17 total 490.42 last 96.19) - open/close 64082 (time 1746139501.17 total 500.43 last 102.66) total: 64768 open/close in 506.74 seconds: 127.81 ops/second - /unlink 2102 (time 1746139519.34 total 10.00 last 210.19) - /unlink 4196 (time 1746139529.34 total 20.00 last 209.37) - /unlink 6006 (time 1746139539.35 total 30.01 last 180.94) - /unlink 8091 (time 1746139549.35 total 40.01 last 208.49) - /unlink 10000 (time 1746139555.76 total 46.42 last 297.64) - /unlink 12728 (time 1746139565.76 total 56.42 last 272.74) - /unlink 14988 (time 1746139575.76 total 66.42 last 225.98) - /unlink 17993 (time 1746139585.77 total 76.42 last 300.43) - /unlink 20000 (time 1746139592.48 total 83.14 last 298.89) - /unlink 21734 (time 1746139602.48 total 93.14 last 173.34) - /unlink 23436 (time 1746139612.49 total 103.14 last 170.17) - /unlink 25867 (time 1746139622.49 total 113.15 last 242.99) - /unlink 28102 (time 1746139632.49 total 123.15 last 223.48) - /unlink 29827 (time 1746139642.49 total 133.15 last 172.49) - /unlink 31842 (time 1746139652.50 total 143.16 last 201.37) - /unlink 34097 (time 1746139662.50 total 153.16 last 225.46) - /unlink 36084 (time 1746139672.51 total 163.17 last 198.51) - /unlink 38194 (time 1746139682.52 total 173.18 last 210.82) - /unlink 40000 (time 1746139691.17 total 181.83 last 208.73) - /unlink 41915 (time 1746139701.17 total 191.83 last 191.46) - /unlink 43809 (time 1746139711.18 total 201.83 last 189.35) - /unlink 45802 (time 1746139721.18 total 211.84 last 199.26) - /unlink 47667 (time 1746139731.18 total 221.84 last 186.49) - /unlink 49402 (time 1746139741.18 total 231.84 last 173.43) - /unlink 50000 (time 1746139744.41 total 235.06 last 185.55) - /unlink 52194 (time 1746139754.41 total 245.07 last 219.37) - /unlink 54341 (time 1746139764.41 total 255.07 last 214.66) - /unlink 56518 (time 1746139774.41 total 265.07 last 217.64) - /unlink 58551 (time 1746139784.41 total 275.07 last 203.26) - /unlink 60000 (time 1746139789.84 total 280.50 last 266.86) - /unlink 62197 (time 1746139799.84 total 290.50 last 219.68) - /unlink 64107 (time 1746139809.85 total 300.51 last 190.95) total: 64768 /unlink in 303.37 seconds: 213.49 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1293 (time 1746139842.21 total 10.00 last 129.30) - open/close 2713 (time 1746139852.21 total 20.00 last 141.99) - open/close 4338 (time 1746139862.22 total 30.00 last 162.45) - open/close 5859 (time 1746139872.22 total 40.01 last 152.00) - open/close 7305 (time 1746139882.22 total 50.01 last 144.60) - open/close 8926 (time 1746139892.23 total 60.02 last 162.00) - open/close 10000 (time 1746139899.42 total 67.21 last 149.33) - open/close 11526 (time 1746139909.42 total 77.21 last 152.55) - open/close 12796 (time 1746139919.43 total 87.21 last 126.98) - open/close 14047 (time 1746139929.43 total 97.22 last 125.08) - open/close 15436 (time 1746139939.43 total 107.22 last 138.88) - open/close 16665 (time 1746139949.43 total 117.22 last 122.89) - open/close 17836 (time 1746139959.43 total 127.22 last 117.07) - open/close 18958 (time 1746139969.44 total 137.23 last 112.14) - open/close 20000 (time 1746139978.96 total 146.75 last 109.39) - open/close 21176 (time 1746139988.97 total 156.76 last 117.51) - open/close 22454 (time 1746139998.98 total 166.77 last 127.66) - open/close 23673 (time 1746140009.00 total 176.79 last 121.63) - open/close 24872 (time 1746140019.01 total 186.80 last 119.79) - open/close 25912 (time 1746140029.02 total 196.81 last 103.88) - open/close 27029 (time 1746140039.02 total 206.81 last 111.70) - open/close 28138 (time 1746140049.03 total 216.82 last 110.87) - open/close 29345 (time 1746140059.03 total 226.82 last 120.67) - open/close 30000 (time 1746140065.19 total 232.98 last 106.28) - open/close 30996 (time 1746140075.19 total 242.98 last 99.58) - open/close 32100 (time 1746140085.21 total 252.99 last 110.29) - open/close 33282 (time 1746140095.21 total 263.00 last 118.15) - open/close 34477 (time 1746140105.21 total 273.00 last 119.50) - open/close 35650 (time 1746140115.22 total 283.01 last 117.20) - open/close 36760 (time 1746140125.22 total 293.01 last 111.00) - open/close 37797 (time 1746140135.23 total 303.02 last 103.58) - open/close 38728 (time 1746140145.23 total 313.02 last 93.09) - open/close 39801 (time 1746140155.23 total 323.02 last 107.27) - open/close 40000 (time 1746140157.28 total 325.07 last 97.29) - open/close 41055 (time 1746140167.28 total 335.07 last 105.50) - open/close 42108 (time 1746140177.28 total 345.07 last 105.24) - open/close 43197 (time 1746140187.29 total 355.08 last 108.85) - open/close 44408 (time 1746140197.29 total 365.08 last 121.07) - open/close 45428 (time 1746140207.30 total 375.09 last 101.90) - open/close 46570 (time 1746140217.30 total 385.09 last 114.17) - open/close 47699 (time 1746140227.31 total 395.10 last 112.81) - open/close 48817 (time 1746140237.32 total 405.11 last 111.74) - open/close 49884 (time 1746140247.32 total 415.11 last 106.64) - open/close 50960 (time 1746140257.33 total 425.12 last 107.55) - open/close 51967 (time 1746140267.33 total 435.12 last 100.63) - open/close 53048 (time 1746140277.33 total 445.12 last 108.09) - open/close 54162 (time 1746140287.34 total 455.13 last 111.36) - open/close 55273 (time 1746140297.34 total 465.13 last 111.04) - open/close 56307 (time 1746140307.35 total 475.14 last 103.38) - open/close 57394 (time 1746140317.35 total 485.14 last 108.66) - open/close 58408 (time 1746140327.35 total 495.14 last 101.38) - open/close 59440 (time 1746140337.36 total 505.14 last 103.17) - open/close 60000 (time 1746140342.47 total 510.25 last 109.59) - open/close 61100 (time 1746140352.47 total 520.26 last 109.96) - open/close 62194 (time 1746140362.47 total 530.26 last 109.40) - open/close 63297 (time 1746140372.48 total 540.27 last 110.19) - open/close 64413 (time 1746140382.48 total 550.27 last 111.58) total: 64768 open/close in 553.50 seconds: 117.02 ops/second - /unlink 1879 (time 1746140397.79 total 10.00 last 187.85) - /unlink 3635 (time 1746140407.80 total 20.01 last 175.47) - /unlink 5089 (time 1746140417.80 total 30.01 last 145.36) - /unlink 6769 (time 1746140427.81 total 40.02 last 167.90) - /unlink 8380 (time 1746140437.81 total 50.02 last 161.05) - /unlink 10000 (time 1746140447.09 total 59.30 last 174.64) - /unlink 11788 (time 1746140457.09 total 69.30 last 178.78) - /unlink 13742 (time 1746140467.09 total 79.30 last 195.37) - /unlink 15599 (time 1746140477.09 total 89.31 last 185.61) - /unlink 17460 (time 1746140487.10 total 99.31 last 186.08) - /unlink 19245 (time 1746140497.10 total 109.31 last 178.42) - /unlink 20000 (time 1746140501.31 total 113.52 last 179.52) - /unlink 21778 (time 1746140511.31 total 123.52 last 177.78) - /unlink 23454 (time 1746140521.31 total 133.52 last 167.53) - /unlink 25197 (time 1746140531.31 total 143.52 last 174.28) - /unlink 26947 (time 1746140541.31 total 153.52 last 174.99) - /unlink 28877 (time 1746140551.32 total 163.53 last 192.94) - /unlink 30000 (time 1746140557.64 total 169.85 last 177.49) - /unlink 31844 (time 1746140567.64 total 179.85 last 184.40) - /unlink 33614 (time 1746140577.65 total 189.86 last 176.89) - /unlink 35444 (time 1746140587.66 total 199.87 last 182.85) - /unlink 37192 (time 1746140597.66 total 209.87 last 174.73) - /unlink 38864 (time 1746140607.66 total 219.87 last 167.19) - /unlink 40000 (time 1746140614.04 total 226.25 last 178.14) - /unlink 41841 (time 1746140624.04 total 236.25 last 184.05) - /unlink 43489 (time 1746140634.05 total 246.26 last 164.68) - /unlink 45196 (time 1746140644.05 total 256.27 last 170.60) - /unlink 47401 (time 1746140654.06 total 266.27 last 220.48) - /unlink 49523 (time 1746140664.06 total 276.27 last 212.19) - /unlink 51664 (time 1746140674.06 total 286.27 last 214.10) - /unlink 54587 (time 1746140684.06 total 296.27 last 292.25) - /unlink 56967 (time 1746140694.06 total 306.27 last 238.00) - /unlink 59310 (time 1746140704.06 total 316.27 last 234.19) - /unlink 60000 (time 1746140707.73 total 319.94 last 188.34) - /unlink 61736 (time 1746140717.73 total 329.94 last 173.50) - /unlink 63882 (time 1746140727.74 total 339.95 last 214.51) total: 64768 /unlink in 344.64 seconds: 187.93 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg430-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg430-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg430-server stop mds service on oleg430-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg430-server unloading modules via unload_modules_local on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-client: ssh exited with exit code 2 pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 PASS 106 (3867s) Stopping clients: oleg430-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg430-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg430-client: oleg430-server: ssh exited with exit code 2 oleg430-server: oleg430-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg430-server' oleg430-server: oleg430-server.virtnet: executing load_modules_local oleg430-server: Loading modules from /home/green/git/lustre-release/lustre oleg430-server: detected 4 online CPUs by sysfs oleg430-server: MODOPTS_LIBCFS= oleg430-server: Force libcfs to create 2 CPU partitions oleg430-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg430-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 3981 sec ========== 19:06:37 (1746140797) === conf-sanity: start cleanup 19:06:38 (1746140798) === === conf-sanity: finish cleanup 19:06:39 (1746140799) ===