-----============= acceptance-small: conf-sanity ============----- Thu May 1 15:00:46 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg449-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg449-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg449-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg449-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg449-server oleg449-server: oleg449-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: detected 4 online CPUs by sysfs oleg449-server: MODOPTS_LIBCFS= oleg449-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg449-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg449-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg449-server: oleg449-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg449-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:02:37 (1746126157) Stopping clients: oleg449-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg449-client.virtnet /mnt/lustre2 (opts:-f) oleg449-server: oleg449-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: detected 4 online CPUs by sysfs oleg449-server: MODOPTS_LIBCFS= oleg449-server: Force libcfs to create 2 CPU partitions oleg449-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg449-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg449-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock 192.168.204.149@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32648 392 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1085 (time 1746126223.25 total 10.00 last 108.48) - open/close 2210 (time 1746126233.25 total 20.00 last 112.50) - open/close 3334 (time 1746126243.25 total 30.01 last 112.35) - open/close 4556 (time 1746126253.25 total 40.01 last 122.19) - open/close 5701 (time 1746126263.26 total 50.02 last 114.36) - open/close 6793 (time 1746126273.27 total 60.03 last 109.14) - open/close 7895 (time 1746126283.27 total 70.03 last 110.18) - open/close 9009 (time 1746126293.28 total 80.04 last 111.32) - open/close 10000 (time 1746126302.61 total 89.36 last 106.23) - open/close 11028 (time 1746126312.61 total 99.37 last 102.74) - open/close 12070 (time 1746126322.62 total 109.37 last 104.16) - open/close 13100 (time 1746126332.62 total 119.38 last 102.95) - open/close 14080 (time 1746126342.62 total 129.38 last 97.99) - open/close 15110 (time 1746126352.63 total 139.39 last 102.94) - open/close 16087 (time 1746126362.63 total 149.39 last 97.65) - open/close 17095 (time 1746126372.64 total 159.40 last 100.75) - open/close 18099 (time 1746126382.64 total 169.40 last 100.38) - open/close 19097 (time 1746126392.65 total 179.41 last 99.69) - open/close 19926 (time 1746126402.67 total 189.42 last 82.78) - open/close 20912 (time 1746126412.67 total 199.43 last 98.54) - open/close 21882 (time 1746126422.68 total 209.43 last 96.98) - open/close 22880 (time 1746126432.68 total 219.44 last 99.75) - open/close 23994 (time 1746126442.68 total 229.44 last 111.39) - open/close 25169 (time 1746126452.68 total 239.44 last 117.49) - open/close 26256 (time 1746126462.69 total 249.45 last 108.61) - open/close 27432 (time 1746126472.71 total 259.46 last 117.43) - open/close 28606 (time 1746126482.71 total 269.47 last 117.35) - open/close 29752 (time 1746126492.71 total 279.47 last 114.57) - open/close 30000 (time 1746126494.87 total 281.62 last 115.17) - open/close 31135 (time 1746126504.88 total 291.63 last 113.37) - open/close 32261 (time 1746126514.89 total 301.64 last 112.51) - open/close 33427 (time 1746126524.89 total 311.64 last 116.60) - open/close 34523 (time 1746126534.90 total 321.66 last 109.40) - open/close 35622 (time 1746126544.91 total 331.66 last 109.87) - open/close 36731 (time 1746126554.91 total 341.67 last 110.87) - open/close 37803 (time 1746126564.91 total 351.67 last 107.18) - open/close 38813 (time 1746126574.91 total 361.67 last 100.98) - open/close 39843 (time 1746126584.92 total 371.68 last 102.92) - open/close 40929 (time 1746126594.92 total 381.68 last 108.56) - open/close 42030 (time 1746126604.93 total 391.68 last 110.09) - open/close 43097 (time 1746126614.93 total 401.69 last 106.65) - open/close 44224 (time 1746126624.94 total 411.70 last 112.57) - open/close 45383 (time 1746126634.95 total 421.71 last 115.80) - open/close 46504 (time 1746126644.95 total 431.71 last 112.08) - open/close 47806 (time 1746126654.96 total 441.71 last 130.17) - open/close 49153 (time 1746126664.96 total 451.71 last 134.68) - open/close 50000 (time 1746126671.74 total 458.50 last 124.85) - open/close 51167 (time 1746126681.75 total 468.50 last 116.63) - open/close 52499 (time 1746126691.75 total 478.50 last 133.18) - open/close 53822 (time 1746126701.75 total 488.51 last 132.22) - open/close 55170 (time 1746126711.76 total 498.51 last 134.76) - open/close 56828 (time 1746126721.76 total 508.52 last 165.77) - open/close 58240 (time 1746126731.76 total 518.52 last 141.17) - open/close 59722 (time 1746126741.77 total 528.53 last 148.07) - open/close 61169 (time 1746126751.77 total 538.53 last 144.64) - open/close 62615 (time 1746126761.78 total 548.53 last 144.58) - open/close 64460 (time 1746126771.78 total 558.54 last 184.38) total: 64768 open/close in 560.46 seconds: 115.56 ops/second - /unlink 3715 (time 1746126784.94 total 10.00 last 371.44) - /unlink 6707 (time 1746126794.94 total 20.01 last 299.09) - /unlink 8959 (time 1746126804.94 total 30.01 last 225.20) - /unlink 10000 (time 1746126808.74 total 33.80 last 274.55) - /unlink 12968 (time 1746126818.74 total 43.80 last 296.78) - /unlink 16347 (time 1746126828.74 total 53.80 last 337.78) - /unlink 19052 (time 1746126838.74 total 63.80 last 270.49) - /unlink 20000 (time 1746126842.82 total 67.88 last 232.40) - /unlink 22958 (time 1746126852.82 total 77.88 last 295.72) - /unlink 26623 (time 1746126862.82 total 87.89 last 366.43) - /unlink 30000 (time 1746126872.81 total 97.87 last 338.21) - /unlink 32822 (time 1746126882.81 total 107.87 last 282.16) - /unlink 35405 (time 1746126892.81 total 117.87 last 258.26) - /unlink 37818 (time 1746126902.81 total 127.87 last 241.29) - /unlink 40000 (time 1746126911.61 total 136.67 last 248.01) - /unlink 42449 (time 1746126921.61 total 146.67 last 244.85) - /unlink 46010 (time 1746126931.62 total 156.68 last 355.92) - /unlink 50000 (time 1746126941.26 total 166.32 last 413.67) - /unlink 53904 (time 1746126951.26 total 176.33 last 390.37) - /unlink 57166 (time 1746126961.26 total 186.33 last 326.17) - /unlink 60000 (time 1746126970.95 total 196.01 last 292.50) - /unlink 62951 (time 1746126980.95 total 206.02 last 295.07) total: 64768 /unlink in 211.30 seconds: 306.52 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1545 (time 1746127011.26 total 10.00 last 154.49) - open/close 3495 (time 1746127021.26 total 20.00 last 194.95) - open/close 5576 (time 1746127031.27 total 30.01 last 208.02) - open/close 7362 (time 1746127041.27 total 40.01 last 178.56) - open/close 9175 (time 1746127051.27 total 50.01 last 181.29) - open/close 10000 (time 1746127056.08 total 54.82 last 171.36) - open/close 12053 (time 1746127066.08 total 64.83 last 205.26) - open/close 14027 (time 1746127076.09 total 74.83 last 197.33) - open/close 16088 (time 1746127086.09 total 84.83 last 206.10) - open/close 17779 (time 1746127096.09 total 94.83 last 169.08) - open/close 19988 (time 1746127106.09 total 104.83 last 220.81) - open/close 22283 (time 1746127116.10 total 114.84 last 229.41) - open/close 24221 (time 1746127126.10 total 124.84 last 193.78) - open/close 26216 (time 1746127136.10 total 134.84 last 199.45) - open/close 28282 (time 1746127146.10 total 144.84 last 206.57) - open/close 30000 (time 1746127154.54 total 153.28 last 203.68) - open/close 32246 (time 1746127164.54 total 163.28 last 224.51) - open/close 34412 (time 1746127174.55 total 173.29 last 216.52) - open/close 36368 (time 1746127184.55 total 183.29 last 195.48) - open/close 38332 (time 1746127194.57 total 193.31 last 196.10) - open/close 39847 (time 1746127204.57 total 203.31 last 151.40) - open/close 41770 (time 1746127214.58 total 213.32 last 192.28) - open/close 43272 (time 1746127224.58 total 223.32 last 150.16) - open/close 45095 (time 1746127234.58 total 233.32 last 182.27) - open/close 46808 (time 1746127244.59 total 243.33 last 171.16) - open/close 49222 (time 1746127254.59 total 253.33 last 241.37) - open/close 50000 (time 1746127258.12 total 256.86 last 220.42) - open/close 52109 (time 1746127268.12 total 266.86 last 210.90) - open/close 54003 (time 1746127278.12 total 276.86 last 189.34) - open/close 56020 (time 1746127288.13 total 286.87 last 201.59) - open/close 57962 (time 1746127298.13 total 296.87 last 194.15) - open/close 60000 (time 1746127307.88 total 306.63 last 208.91) - open/close 61418 (time 1746127317.89 total 316.63 last 141.72) - open/close 62644 (time 1746127327.89 total 326.64 last 122.55) - open/close 64549 (time 1746127337.90 total 336.64 last 190.38) total: 64768 open/close in 337.59 seconds: 191.85 ops/second - /unlink 2854 (time 1746127349.87 total 10.00 last 285.35) - /unlink 6092 (time 1746127359.87 total 20.00 last 323.75) - /unlink 10000 (time 1746127369.68 total 29.81 last 398.36) - /unlink 13059 (time 1746127379.70 total 39.83 last 305.45) - /unlink 16422 (time 1746127389.71 total 49.84 last 336.03) - /unlink 19152 (time 1746127399.71 total 59.84 last 272.93) - /unlink 20000 (time 1746127403.02 total 63.16 last 255.69) - /unlink 23174 (time 1746127413.03 total 73.16 last 317.21) - /unlink 27100 (time 1746127423.03 total 83.16 last 392.57) - /unlink 29802 (time 1746127433.03 total 93.17 last 270.10) - /unlink 32779 (time 1746127443.04 total 103.17 last 297.63) - /unlink 35721 (time 1746127453.04 total 113.17 last 294.20) - /unlink 38214 (time 1746127463.04 total 123.18 last 249.10) - /unlink 40000 (time 1746127472.04 total 132.17 last 198.55) - /unlink 41891 (time 1746127482.04 total 142.17 last 189.08) - /unlink 43410 (time 1746127492.05 total 152.18 last 151.78) - /unlink 44994 (time 1746127502.05 total 162.18 last 158.38) - /unlink 46846 (time 1746127512.06 total 172.19 last 185.05) - /unlink 48726 (time 1746127522.06 total 182.19 last 187.97) - /unlink 49949 (time 1746127532.07 total 192.20 last 122.18) - /unlink 51179 (time 1746127542.07 total 202.21 last 122.94) - /unlink 52351 (time 1746127552.08 total 212.21 last 117.14) - /unlink 53539 (time 1746127562.09 total 222.22 last 118.72) - /unlink 54687 (time 1746127572.10 total 232.24 last 114.59) - /unlink 55898 (time 1746127582.11 total 242.24 last 121.08) - /unlink 58032 (time 1746127592.11 total 252.24 last 213.33) - /unlink 59969 (time 1746127602.11 total 262.24 last 193.68) - /unlink 61789 (time 1746127612.11 total 272.24 last 181.96) - /unlink 63682 (time 1746127622.11 total 282.24 last 189.26) total: 64768 /unlink in 287.77 seconds: 225.07 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1044 (time 1746127658.53 total 10.01 last 104.34) - open/close 2158 (time 1746127668.53 total 20.01 last 111.40) - open/close 3296 (time 1746127678.53 total 30.01 last 113.79) - open/close 4515 (time 1746127688.53 total 40.01 last 121.87) - open/close 5649 (time 1746127698.54 total 50.01 last 113.35) - open/close 6819 (time 1746127708.54 total 60.01 last 116.99) - open/close 7964 (time 1746127718.54 total 70.02 last 114.46) - open/close 9130 (time 1746127728.55 total 80.03 last 116.51) - open/close 10000 (time 1746127736.34 total 87.81 last 111.71) - open/close 11166 (time 1746127746.34 total 97.82 last 116.56) - open/close 12343 (time 1746127756.34 total 107.82 last 117.64) - open/close 13462 (time 1746127766.35 total 117.83 last 111.86) - open/close 14647 (time 1746127776.35 total 127.83 last 118.44) - open/close 15881 (time 1746127786.37 total 137.85 last 123.21) - open/close 17071 (time 1746127796.37 total 147.85 last 118.95) - open/close 18248 (time 1746127806.39 total 157.86 last 117.55) - open/close 19269 (time 1746127816.39 total 167.87 last 102.07) - open/close 20000 (time 1746127822.91 total 174.39 last 112.10) - open/close 21066 (time 1746127832.91 total 184.39 last 106.58) - open/close 22122 (time 1746127842.92 total 194.39 last 105.55) - open/close 23250 (time 1746127852.92 total 204.40 last 112.79) - open/close 24485 (time 1746127862.92 total 214.40 last 123.41) - open/close 25593 (time 1746127872.93 total 224.41 last 110.76) - open/close 26800 (time 1746127882.93 total 234.41 last 120.67) - open/close 27861 (time 1746127892.93 total 244.41 last 106.09) - open/close 28932 (time 1746127902.94 total 254.41 last 107.05) - open/close 29994 (time 1746127912.94 total 264.42 last 106.13) - open/close 31038 (time 1746127922.95 total 274.43 last 104.34) - open/close 32070 (time 1746127932.96 total 284.44 last 103.07) - open/close 33035 (time 1746127942.97 total 294.45 last 96.42) - open/close 34102 (time 1746127953.00 total 304.48 last 106.37) - open/close 35085 (time 1746127963.01 total 314.49 last 98.17) - open/close 36041 (time 1746127973.02 total 324.50 last 95.50) - open/close 37017 (time 1746127983.03 total 334.51 last 97.54) - open/close 37996 (time 1746127993.04 total 344.52 last 97.82) - open/close 39028 (time 1746128003.04 total 354.52 last 103.19) - open/close 40000 (time 1746128012.76 total 364.24 last 100.01) - open/close 41017 (time 1746128022.76 total 374.24 last 101.68) - open/close 42026 (time 1746128032.77 total 384.24 last 100.85) - open/close 43052 (time 1746128042.77 total 394.25 last 102.57) - open/close 44091 (time 1746128052.77 total 404.25 last 103.83) - open/close 45092 (time 1746128062.79 total 414.27 last 99.96) - open/close 46109 (time 1746128072.81 total 424.29 last 101.52) - open/close 47085 (time 1746128082.82 total 434.30 last 97.50) - open/close 48203 (time 1746128092.84 total 444.32 last 111.57) - open/close 49407 (time 1746128102.85 total 454.33 last 120.26) - open/close 50000 (time 1746128107.78 total 459.26 last 120.29) - open/close 51199 (time 1746128117.78 total 469.26 last 119.83) - open/close 52323 (time 1746128127.79 total 479.26 last 112.39) - open/close 53355 (time 1746128137.80 total 489.27 last 103.10) - open/close 54349 (time 1746128147.80 total 499.28 last 99.32) - open/close 55285 (time 1746128157.80 total 509.28 last 93.60) - open/close 56076 (time 1746128167.81 total 519.29 last 79.06) - open/close 56826 (time 1746128177.82 total 529.30 last 74.93) - open/close 57724 (time 1746128187.83 total 539.31 last 89.71) - open/close 58684 (time 1746128197.83 total 549.31 last 95.99) - open/close 59527 (time 1746128207.83 total 559.31 last 84.30) - open/close 60000 (time 1746128213.29 total 564.77 last 86.56) - open/close 60924 (time 1746128223.29 total 574.77 last 92.40) - open/close 61839 (time 1746128233.30 total 584.78 last 91.47) - open/close 62672 (time 1746128243.32 total 594.80 last 83.12) - open/close 63540 (time 1746128253.32 total 604.80 last 86.78) - open/close 64421 (time 1746128263.33 total 614.80 last 88.05) total: 64768 open/close in 618.75 seconds: 104.68 ops/second - /unlink 980 (time 1746128279.54 total 10.00 last 97.99) - /unlink 1908 (time 1746128289.55 total 20.01 last 92.71) - /unlink 2903 (time 1746128299.55 total 30.01 last 99.49) - /unlink 3962 (time 1746128309.56 total 40.02 last 105.83) - /unlink 5064 (time 1746128319.57 total 50.03 last 110.05) - /unlink 6125 (time 1746128329.57 total 60.03 last 106.07) - /unlink 7256 (time 1746128339.58 total 70.04 last 113.08) - /unlink 8457 (time 1746128349.59 total 80.05 last 119.97) - /unlink 9549 (time 1746128359.59 total 90.05 last 109.17) - /unlink 10000 (time 1746128363.79 total 94.25 last 107.33) - /unlink 11018 (time 1746128373.80 total 104.26 last 101.76) - /unlink 12116 (time 1746128383.81 total 114.27 last 109.69) - /unlink 13065 (time 1746128393.81 total 124.27 last 94.87) - /unlink 14084 (time 1746128403.81 total 134.27 last 101.86) - /unlink 15132 (time 1746128413.82 total 144.28 last 104.71) - /unlink 16231 (time 1746128423.82 total 154.28 last 109.90) - /unlink 17233 (time 1746128433.83 total 164.29 last 100.15) - /unlink 18287 (time 1746128443.84 total 174.30 last 105.30) - /unlink 19388 (time 1746128453.85 total 184.31 last 109.98) - /unlink 20000 (time 1746128459.59 total 190.05 last 106.58) - /unlink 21061 (time 1746128469.60 total 200.06 last 106.01) - /unlink 22051 (time 1746128479.60 total 210.06 last 98.94) - /unlink 22959 (time 1746128489.61 total 220.07 last 90.75) - /unlink 23896 (time 1746128499.62 total 230.08 last 93.56) - /unlink 24920 (time 1746128509.63 total 240.09 last 102.35) - /unlink 26049 (time 1746128519.63 total 250.09 last 112.90) - /unlink 27182 (time 1746128529.63 total 260.09 last 113.27) - /unlink 28252 (time 1746128539.63 total 270.09 last 106.99) - /unlink 29303 (time 1746128549.64 total 280.10 last 105.03) - /unlink 30000 (time 1746128556.30 total 286.76 last 104.64) - /unlink 30945 (time 1746128566.31 total 296.77 last 94.41) - /unlink 32087 (time 1746128576.32 total 306.78 last 114.13) - /unlink 33207 (time 1746128586.33 total 316.79 last 111.90) - /unlink 34236 (time 1746128596.33 total 326.79 last 102.82) - /unlink 35189 (time 1746128606.35 total 336.81 last 95.11) - /unlink 36179 (time 1746128616.37 total 346.83 last 98.81) - /unlink 37228 (time 1746128626.38 total 356.84 last 104.87) - /unlink 38229 (time 1746128636.38 total 366.84 last 100.04) - /unlink 39255 (time 1746128646.40 total 376.86 last 102.43) - /unlink 40000 (time 1746128653.50 total 383.96 last 104.87) - /unlink 40873 (time 1746128663.50 total 393.96 last 87.29) - /unlink 41771 (time 1746128673.52 total 403.98 last 89.68) - /unlink 42759 (time 1746128683.53 total 413.99 last 98.65) - /unlink 43714 (time 1746128693.53 total 423.99 last 95.47) - /unlink 44715 (time 1746128703.54 total 434.00 last 100.04) - /unlink 45712 (time 1746128713.55 total 444.01 last 99.65) - /unlink 46640 (time 1746128723.55 total 454.01 last 92.76) - /unlink 47592 (time 1746128733.55 total 464.01 last 95.16) - /unlink 48500 (time 1746128743.56 total 474.02 last 90.76) - /unlink 49323 (time 1746128753.57 total 484.03 last 82.23) - /unlink 50000 (time 1746128761.58 total 492.04 last 84.48) - /unlink 50881 (time 1746128771.58 total 502.04 last 88.08) - /unlink 51836 (time 1746128781.60 total 512.06 last 95.35) - /unlink 52696 (time 1746128791.61 total 522.07 last 85.93) - /unlink 53535 (time 1746128801.61 total 532.07 last 83.90) - /unlink 54393 (time 1746128811.62 total 542.08 last 85.67) - /unlink 55246 (time 1746128821.63 total 552.09 last 85.25) - /unlink 56054 (time 1746128831.64 total 562.10 last 80.73) - /unlink 56913 (time 1746128841.64 total 572.10 last 85.89) - /unlink 57880 (time 1746128851.64 total 582.10 last 96.66) - /unlink 58753 (time 1746128861.64 total 592.10 last 87.28) - /unlink 59539 (time 1746128871.65 total 602.11 last 78.55) - /unlink 60000 (time 1746128877.65 total 608.11 last 76.85) - /unlink 60899 (time 1746128887.65 total 618.11 last 89.87) - /unlink 61845 (time 1746128897.66 total 628.12 last 94.52) - /unlink 62770 (time 1746128907.66 total 638.12 last 92.49) - /unlink 63797 (time 1746128917.67 total 648.13 last 102.65) - /unlink 64713 (time 1746128927.67 total 658.13 last 91.55) total: 64768 /unlink in 658.61 seconds: 98.34 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1022 (time 1746128961.17 total 10.00 last 102.20) - open/close 1952 (time 1746128971.17 total 20.00 last 92.99) - open/close 2905 (time 1746128981.17 total 30.00 last 95.29) - open/close 3760 (time 1746128991.18 total 40.00 last 85.48) - open/close 4784 (time 1746129001.19 total 50.02 last 102.28) - open/close 5737 (time 1746129011.19 total 60.02 last 95.25) - open/close 6631 (time 1746129021.20 total 70.03 last 89.31) - open/close 7495 (time 1746129031.21 total 80.03 last 86.37) - open/close 8371 (time 1746129041.21 total 90.04 last 87.54) - open/close 9375 (time 1746129051.22 total 100.04 last 100.38) - open/close 10000 (time 1746129058.15 total 106.98 last 90.09) - open/close 11044 (time 1746129068.16 total 116.99 last 104.36) - open/close 12132 (time 1746129078.16 total 126.99 last 108.80) - open/close 13214 (time 1746129088.16 total 136.99 last 108.15) - open/close 14360 (time 1746129098.17 total 147.00 last 114.53) - open/close 15480 (time 1746129108.17 total 157.00 last 111.98) - open/close 16581 (time 1746129118.17 total 167.00 last 110.05) - open/close 17672 (time 1746129128.18 total 177.01 last 109.06) - open/close 18699 (time 1746129138.18 total 187.01 last 102.68) - open/close 19809 (time 1746129148.18 total 197.01 last 110.99) - open/close 20933 (time 1746129158.19 total 207.01 last 112.34) - open/close 22040 (time 1746129168.19 total 217.02 last 110.64) - open/close 23177 (time 1746129178.19 total 227.02 last 113.69) - open/close 24291 (time 1746129188.20 total 237.03 last 111.29) - open/close 25434 (time 1746129198.20 total 247.03 last 114.29) - open/close 26499 (time 1746129208.21 total 257.04 last 106.46) - open/close 27653 (time 1746129218.21 total 267.04 last 115.32) - open/close 28874 (time 1746129228.22 total 277.05 last 122.04) - open/close 30000 (time 1746129238.14 total 286.97 last 113.52) - open/close 31065 (time 1746129248.14 total 296.97 last 106.49) - open/close 32139 (time 1746129258.14 total 306.97 last 107.40) - open/close 33256 (time 1746129268.14 total 316.97 last 111.67) - open/close 34408 (time 1746129278.14 total 326.97 last 115.19) - open/close 35560 (time 1746129288.14 total 336.97 last 115.17) - open/close 36663 (time 1746129298.15 total 346.97 last 110.28) - open/close 37750 (time 1746129308.15 total 356.97 last 108.70) - open/close 38788 (time 1746129318.16 total 366.99 last 103.64) - open/close 39906 (time 1746129328.17 total 376.99 last 111.75) - open/close 41179 (time 1746129338.17 total 387.00 last 127.26) - open/close 42396 (time 1746129348.17 total 397.00 last 121.65) - open/close 43566 (time 1746129358.18 total 407.00 last 116.96) - open/close 44693 (time 1746129368.18 total 417.00 last 112.70) - open/close 45910 (time 1746129378.18 total 427.01 last 121.66) - open/close 47161 (time 1746129388.18 total 437.01 last 125.10) - open/close 48589 (time 1746129398.19 total 447.01 last 142.72) - open/close 49777 (time 1746129408.20 total 457.02 last 118.68) - open/close 50961 (time 1746129418.20 total 467.03 last 118.31) - open/close 52126 (time 1746129428.21 total 477.03 last 116.46) - open/close 53577 (time 1746129438.21 total 487.04 last 145.08) - open/close 55076 (time 1746129448.22 total 497.05 last 149.71) - open/close 56340 (time 1746129458.23 total 507.06 last 126.25) - open/close 57563 (time 1746129468.23 total 517.06 last 122.30) - open/close 58748 (time 1746129478.24 total 527.06 last 118.46) - open/close 60000 (time 1746129488.21 total 537.04 last 125.56) - open/close 61142 (time 1746129498.21 total 547.04 last 114.14) - open/close 62693 (time 1746129508.22 total 557.04 last 155.05) - open/close 64163 (time 1746129518.22 total 567.05 last 146.88) total: 64768 open/close in 571.24 seconds: 113.38 ops/second - /unlink 2528 (time 1746129533.80 total 10.00 last 252.72) - /unlink 5422 (time 1746129543.80 total 20.01 last 289.25) - /unlink 7505 (time 1746129553.80 total 30.01 last 208.23) - /unlink 9420 (time 1746129563.80 total 40.01 last 191.49) - /unlink 10000 (time 1746129566.68 total 42.89 last 201.49) - /unlink 12244 (time 1746129576.68 total 52.89 last 224.40) - /unlink 14214 (time 1746129586.69 total 62.89 last 196.94) - /unlink 16359 (time 1746129596.69 total 72.90 last 214.46) - /unlink 18436 (time 1746129606.69 total 82.90 last 207.61) - /unlink 20000 (time 1746129613.54 total 89.75 last 228.38) - /unlink 22049 (time 1746129623.55 total 99.75 last 204.79) - /unlink 24145 (time 1746129633.55 total 109.76 last 209.57) - /unlink 26118 (time 1746129643.56 total 119.77 last 197.06) - /unlink 28067 (time 1746129653.56 total 129.77 last 194.88) - /unlink 30000 (time 1746129663.46 total 139.67 last 195.24) - /unlink 31868 (time 1746129673.46 total 149.67 last 186.75) - /unlink 33860 (time 1746129683.47 total 159.68 last 199.09) - /unlink 35748 (time 1746129693.47 total 169.68 last 188.78) - /unlink 37570 (time 1746129703.47 total 179.68 last 182.13) - /unlink 39060 (time 1746129713.48 total 189.69 last 148.93) - /unlink 40000 (time 1746129719.79 total 196.00 last 148.96) - /unlink 41699 (time 1746129729.79 total 206.00 last 169.89) - /unlink 43497 (time 1746129739.80 total 216.00 last 179.71) - /unlink 45262 (time 1746129749.80 total 226.01 last 176.45) - /unlink 46850 (time 1746129759.81 total 236.02 last 158.57) - /unlink 48645 (time 1746129769.81 total 246.02 last 179.50) - /unlink 50000 (time 1746129777.51 total 253.71 last 176.10) - /unlink 52108 (time 1746129787.51 total 263.72 last 210.76) - /unlink 54291 (time 1746129797.51 total 273.72 last 218.26) - /unlink 56237 (time 1746129807.51 total 283.72 last 194.59) - /unlink 58203 (time 1746129817.51 total 293.72 last 196.57) - /unlink 60000 (time 1746129825.77 total 301.98 last 217.59) - /unlink 62006 (time 1746129835.77 total 311.98 last 200.55) - /unlink 64466 (time 1746129845.78 total 321.98 last 245.95) total: 64768 /unlink in 323.02 seconds: 200.51 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1927 (time 1746129871.57 total 10.00 last 192.68) - open/close 4333 (time 1746129881.57 total 20.00 last 240.57) - open/close 6388 (time 1746129891.57 total 30.01 last 205.44) - open/close 8327 (time 1746129901.57 total 40.01 last 193.87) - open/close 10000 (time 1746129909.97 total 48.41 last 199.11) - open/close 12206 (time 1746129919.97 total 58.41 last 220.57) - open/close 13890 (time 1746129929.98 total 68.42 last 168.29) - open/close 15434 (time 1746129939.99 total 78.42 last 154.33) - open/close 16755 (time 1746129949.99 total 88.43 last 132.03) - open/close 18421 (time 1746129959.99 total 98.43 last 166.59) - open/close 19971 (time 1746129969.99 total 108.43 last 154.96) - open/close 22482 (time 1746129979.99 total 118.43 last 251.07) - open/close 24867 (time 1746129990.00 total 128.43 last 238.44) - open/close 27224 (time 1746130000.00 total 138.44 last 235.57) - open/close 29747 (time 1746130010.01 total 148.44 last 252.22) - open/close 31687 (time 1746130020.01 total 158.45 last 193.94) - open/close 33829 (time 1746130030.01 total 168.45 last 214.18) - open/close 35522 (time 1746130040.01 total 178.45 last 169.27) - open/close 36950 (time 1746130050.02 total 188.45 last 142.73) - open/close 38702 (time 1746130060.02 total 198.46 last 175.14) - open/close 40000 (time 1746130069.28 total 207.71 last 140.25) - open/close 41417 (time 1746130079.28 total 217.71 last 141.67) - open/close 42733 (time 1746130089.28 total 227.72 last 131.56) - open/close 44464 (time 1746130099.29 total 237.72 last 173.00) - open/close 46071 (time 1746130109.29 total 247.72 last 160.66) - open/close 47721 (time 1746130119.30 total 257.74 last 164.78) - open/close 49579 (time 1746130129.31 total 267.74 last 185.68) - open/close 50000 (time 1746130131.72 total 270.15 last 174.69) - open/close 51629 (time 1746130141.73 total 280.16 last 162.77) - open/close 53365 (time 1746130151.73 total 290.16 last 173.60) - open/close 54589 (time 1746130161.73 total 300.17 last 122.35) - open/close 55864 (time 1746130171.73 total 310.17 last 127.47) - open/close 57521 (time 1746130181.74 total 320.17 last 165.63) - open/close 59113 (time 1746130191.75 total 330.18 last 159.07) - open/close 60000 (time 1746130196.76 total 335.19 last 176.92) - open/close 61541 (time 1746130206.76 total 345.20 last 154.06) - open/close 63214 (time 1746130216.77 total 355.20 last 167.22) total: 64768 open/close in 363.70 seconds: 178.08 ops/second - /unlink 2557 (time 1746130236.51 total 10.00 last 255.64) - /unlink 5504 (time 1746130246.52 total 20.00 last 294.66) - /unlink 9290 (time 1746130256.52 total 30.00 last 378.60) - /unlink 10000 (time 1746130259.37 total 32.86 last 248.43) - /unlink 12796 (time 1746130269.38 total 42.87 last 279.50) - /unlink 15628 (time 1746130279.38 total 52.87 last 283.13) - /unlink 18535 (time 1746130289.38 total 62.87 last 290.69) - /unlink 20000 (time 1746130295.32 total 68.81 last 246.57) - /unlink 22405 (time 1746130305.32 total 78.81 last 240.47) - /unlink 24828 (time 1746130315.33 total 88.82 last 242.09) - /unlink 26990 (time 1746130325.34 total 98.83 last 216.06) - /unlink 29044 (time 1746130335.34 total 108.83 last 205.28) - /unlink 30000 (time 1746130338.98 total 112.47 last 263.11) - /unlink 32631 (time 1746130348.98 total 122.47 last 263.08) - /unlink 35035 (time 1746130358.98 total 132.47 last 240.39) - /unlink 38145 (time 1746130368.98 total 142.47 last 310.92) - /unlink 40000 (time 1746130374.21 total 147.69 last 355.08) - /unlink 43317 (time 1746130384.21 total 157.70 last 331.57) - /unlink 46127 (time 1746130394.21 total 167.70 last 280.98) - /unlink 48473 (time 1746130404.21 total 177.70 last 234.53) - /unlink 50000 (time 1746130410.06 total 183.55 last 261.17) - /unlink 52288 (time 1746130420.07 total 193.55 last 228.67) - /unlink 54574 (time 1746130430.07 total 203.56 last 228.54) - /unlink 56796 (time 1746130440.08 total 213.56 last 222.03) - /unlink 58642 (time 1746130450.08 total 223.56 last 184.58) - /unlink 60000 (time 1746130454.89 total 228.38 last 282.29) - /unlink 63855 (time 1746130464.89 total 238.38 last 385.50) total: 64768 /unlink in 240.20 seconds: 269.64 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg449-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server unloading modules via unload_modules_local on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 PASS 106 (4339s) Stopping clients: oleg449-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg449-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 oleg449-server: oleg449-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing load_modules_local oleg449-server: Loading modules from /home/green/git/lustre-release/lustre oleg449-server: detected 4 online CPUs by sysfs oleg449-server: MODOPTS_LIBCFS= oleg449-server: Force libcfs to create 2 CPU partitions oleg449-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg449-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4477 sec ========== 16:15:23 (1746130523) === conf-sanity: start cleanup 16:15:23 (1746130523) === === conf-sanity: finish cleanup 16:15:24 (1746130524) ===