-----============= acceptance-small: conf-sanity ============----- Wed May 29 17:18:29 EDT 2024 excepting tests: 32newtarball 110 Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg150-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg150-server oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: Force libcfs to create 2 CPU partitions oleg150-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg150-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg150-server: oleg150-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg150-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 17:20:00 (1717017600) Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg150-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg150-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg150-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg150-server: oleg150-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg150-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg150-client.virtnet: -o user_xattr,flock oleg150-server@tcp:/lustre /mnt/lustre fail_loc=0x1312 fail_val=5 - open/close 1126 (time 1717017658.41 total 10.00 last 112.60) - open/close 2210 (time 1717017668.41 total 20.00 last 108.39) - open/close 3502 (time 1717017678.42 total 30.01 last 129.10) - open/close 4737 (time 1717017688.43 total 40.01 last 123.43) - open/close 5969 (time 1717017698.43 total 50.02 last 123.13) - open/close 7277 (time 1717017708.44 total 60.02 last 130.76) - open/close 8555 (time 1717017718.45 total 70.04 last 127.63) - open/close 9871 (time 1717017728.45 total 80.04 last 131.57) - open/close 11155 (time 1717017738.46 total 90.04 last 128.32) - open/close 12482 (time 1717017748.46 total 100.04 last 132.70) - open/close 13859 (time 1717017758.46 total 110.05 last 137.65) - open/close 15181 (time 1717017768.47 total 120.06 last 132.06) - open/close 16469 (time 1717017778.48 total 130.06 last 128.73) - open/close 17825 (time 1717017788.48 total 140.07 last 135.58) - open/close 19034 (time 1717017798.48 total 150.07 last 120.83) - open/close 20000 (time 1717017806.12 total 157.71 last 126.53) - open/close 21225 (time 1717017816.13 total 167.71 last 122.41) - open/close 22562 (time 1717017826.13 total 177.72 last 133.62) - open/close 23930 (time 1717017836.14 total 187.73 last 136.72) - open/close 25341 (time 1717017846.14 total 197.73 last 141.05) - open/close 26748 (time 1717017856.15 total 207.73 last 140.66) - open/close 27980 (time 1717017866.15 total 217.73 last 123.19) - open/close 29170 (time 1717017876.15 total 227.74 last 118.93) - open/close 30000 (time 1717017882.21 total 233.79 last 137.12) - open/close 31263 (time 1717017892.21 total 243.80 last 126.25) - open/close 32531 (time 1717017902.21 total 253.80 last 126.77) - open/close 33957 (time 1717017912.21 total 263.80 last 142.56) - open/close 35159 (time 1717017922.22 total 273.81 last 120.12) - open/close 36294 (time 1717017932.22 total 283.81 last 113.48) - open/close 37518 (time 1717017942.23 total 293.82 last 122.30) - open/close 38825 (time 1717017952.23 total 303.82 last 130.67) - open/close 40000 (time 1717017960.97 total 312.56 last 134.45) - open/close 41315 (time 1717017970.98 total 322.57 last 131.42) - open/close 42959 (time 1717017980.98 total 332.57 last 164.34) - open/close 44222 (time 1717017990.99 total 342.58 last 126.17) - open/close 45482 (time 1717018001.00 total 352.58 last 125.97) - open/close 46827 (time 1717018011.00 total 362.59 last 134.42) - open/close 48169 (time 1717018021.00 total 372.59 last 134.17) - open/close 49459 (time 1717018031.00 total 382.59 last 128.98) - open/close 50000 (time 1717018035.69 total 387.28 last 115.49) - open/close 51072 (time 1717018045.69 total 397.28 last 107.14) - open/close 52257 (time 1717018055.70 total 407.28 last 118.47) - open/close 53478 (time 1717018065.70 total 417.29 last 122.02) - open/close 54667 (time 1717018075.71 total 427.29 last 118.86) - open/close 55899 (time 1717018085.71 total 437.30 last 123.17) - open/close 57058 (time 1717018095.71 total 447.30 last 115.86) - open/close 58254 (time 1717018105.72 total 457.30 last 119.55) - open/close 59626 (time 1717018115.72 total 467.31 last 137.17) - open/close 60000 (time 1717018118.55 total 470.14 last 132.09) - open/close 61341 (time 1717018128.55 total 480.14 last 134.08) - open/close 62686 (time 1717018138.56 total 490.14 last 134.44) - open/close 64149 (time 1717018148.57 total 500.15 last 146.16) total: 64768 open/close in 505.08 seconds: 128.23 ops/second - /unlink 2151 (time 1717018165.36 total 10.00 last 215.02) - /unlink 4408 (time 1717018175.36 total 20.01 last 225.64) - /unlink 6412 (time 1717018185.37 total 30.01 last 200.39) - /unlink 8396 (time 1717018195.37 total 40.01 last 198.33) - /unlink 10000 (time 1717018202.43 total 47.07 last 227.16) - /unlink 12453 (time 1717018212.43 total 57.08 last 245.19) - /unlink 14606 (time 1717018222.44 total 67.08 last 215.24) - /unlink 17652 (time 1717018232.44 total 77.08 last 304.54) - /unlink 20000 (time 1717018242.44 total 87.08 last 234.83) - /unlink 22282 (time 1717018252.44 total 97.08 last 228.11) - /unlink 24742 (time 1717018262.44 total 107.08 last 245.99) - /unlink 27500 (time 1717018272.44 total 117.09 last 275.73) - /unlink 30000 (time 1717018281.00 total 125.64 last 292.20) - /unlink 32442 (time 1717018291.00 total 135.64 last 244.15) - /unlink 34947 (time 1717018301.00 total 145.64 last 250.49) - /unlink 37431 (time 1717018311.01 total 155.65 last 248.28) - /unlink 39858 (time 1717018321.01 total 165.65 last 242.63) - /unlink 42215 (time 1717018331.01 total 175.66 last 235.62) - /unlink 44496 (time 1717018341.02 total 185.66 last 227.92) - /unlink 46867 (time 1717018351.03 total 195.67 last 236.97) - /unlink 49428 (time 1717018361.03 total 205.67 last 256.07) - /unlink 50000 (time 1717018363.06 total 207.70 last 281.97) - /unlink 52426 (time 1717018373.06 total 217.70 last 242.57) - /unlink 54943 (time 1717018383.06 total 227.70 last 251.65) - /unlink 57456 (time 1717018393.06 total 237.70 last 251.25) - /unlink 59878 (time 1717018403.07 total 247.71 last 242.14) - /unlink 62056 (time 1717018413.07 total 257.71 last 217.67) - /unlink 64234 (time 1717018423.07 total 267.72 last 217.73) total: 64768 /unlink in 270.21 seconds: 239.69 ops/second - open/close 1299 (time 1717018437.45 total 10.00 last 129.89) - open/close 2569 (time 1717018447.47 total 20.02 last 126.80) - open/close 3919 (time 1717018457.47 total 30.02 last 134.98) - open/close 5198 (time 1717018467.47 total 40.02 last 127.85) - open/close 6550 (time 1717018477.48 total 50.03 last 135.11) - open/close 8092 (time 1717018487.48 total 60.03 last 154.14) - open/close 9427 (time 1717018497.49 total 70.04 last 133.44) - open/close 10000 (time 1717018501.77 total 74.32 last 133.91) - open/close 11580 (time 1717018511.77 total 84.32 last 157.97) - open/close 12920 (time 1717018521.77 total 94.32 last 133.99) - open/close 14162 (time 1717018531.78 total 104.33 last 124.08) - open/close 15497 (time 1717018541.79 total 114.34 last 133.40) - open/close 16867 (time 1717018551.79 total 124.34 last 136.96) - open/close 18120 (time 1717018561.80 total 134.35 last 125.18) - open/close 19319 (time 1717018571.81 total 144.35 last 119.83) - open/close 20000 (time 1717018577.12 total 149.67 last 128.18) - open/close 21186 (time 1717018587.12 total 159.67 last 118.55) - open/close 22352 (time 1717018597.13 total 169.68 last 116.55) - open/close 23543 (time 1717018607.13 total 179.68 last 119.04) - open/close 24811 (time 1717018617.14 total 189.69 last 126.72) - open/close 26081 (time 1717018627.14 total 199.69 last 126.96) - open/close 27442 (time 1717018637.14 total 209.69 last 136.07) - open/close 28795 (time 1717018647.15 total 219.70 last 135.23) - open/close 30000 (time 1717018656.78 total 229.33 last 125.09) - open/close 31302 (time 1717018666.79 total 239.33 last 130.15) - open/close 32496 (time 1717018676.79 total 249.34 last 119.35) - open/close 33698 (time 1717018686.80 total 259.34 last 120.14) - open/close 34908 (time 1717018696.80 total 269.35 last 120.93) - open/close 36147 (time 1717018706.81 total 279.36 last 123.82) - open/close 37359 (time 1717018716.82 total 289.36 last 121.12) - open/close 38537 (time 1717018726.82 total 299.36 last 117.80) - open/close 39726 (time 1717018736.82 total 309.36 last 118.88) - open/close 41114 (time 1717018746.82 total 319.37 last 138.75) - open/close 42416 (time 1717018756.82 total 329.37 last 130.19) - open/close 43825 (time 1717018766.83 total 339.38 last 140.79) - open/close 45302 (time 1717018776.84 total 349.38 last 147.59) - open/close 46795 (time 1717018786.84 total 359.39 last 149.22) - open/close 48080 (time 1717018796.85 total 369.39 last 128.43) - open/close 49558 (time 1717018806.85 total 379.40 last 147.69) - open/close 50000 (time 1717018810.23 total 382.78 last 130.90) - open/close 51290 (time 1717018820.23 total 392.78 last 128.97) - open/close 52641 (time 1717018830.23 total 402.78 last 135.10) - open/close 54181 (time 1717018840.23 total 412.78 last 153.97) - open/close 55631 (time 1717018850.23 total 422.78 last 145.00) - open/close 57028 (time 1717018860.24 total 432.78 last 139.69) - open/close 58401 (time 1717018870.24 total 442.78 last 137.28) - open/close 59774 (time 1717018880.24 total 452.79 last 137.27) - open/close 61164 (time 1717018890.24 total 462.79 last 138.93) - open/close 62444 (time 1717018900.25 total 472.80 last 127.93) - open/close 63859 (time 1717018910.26 total 482.80 last 141.42) total: 64768 open/close in 489.17 seconds: 132.41 ops/second - /unlink 2419 (time 1717018927.97 total 10.00 last 241.81) - /unlink 4555 (time 1717018937.97 total 20.00 last 213.58) - /unlink 6556 (time 1717018947.97 total 30.01 last 200.06) - /unlink 8670 (time 1717018957.97 total 40.01 last 211.33) - /unlink 10000 (time 1717018963.96 total 45.99 last 222.27) - /unlink 12029 (time 1717018973.96 total 55.99 last 202.90) - /unlink 13990 (time 1717018983.96 total 66.00 last 196.05) - /unlink 16008 (time 1717018993.97 total 76.00 last 201.69) - /unlink 17904 (time 1717019003.97 total 86.01 last 189.50) - /unlink 19970 (time 1717019013.97 total 96.01 last 206.53) - /unlink 21787 (time 1717019023.98 total 106.01 last 181.69) - /unlink 23978 (time 1717019033.98 total 116.01 last 219.09) - /unlink 26224 (time 1717019043.98 total 126.01 last 224.56) - /unlink 28253 (time 1717019053.98 total 136.02 last 202.83) - /unlink 30000 (time 1717019062.17 total 144.20 last 213.42) - /unlink 31929 (time 1717019072.17 total 154.21 last 192.76) - /unlink 34218 (time 1717019082.18 total 164.21 last 228.82) - /unlink 36264 (time 1717019092.19 total 174.22 last 204.42) - /unlink 38440 (time 1717019102.19 total 184.23 last 217.44) - /unlink 40000 (time 1717019109.34 total 191.37 last 218.36) - /unlink 42389 (time 1717019119.34 total 201.38 last 238.83) - /unlink 44625 (time 1717019129.34 total 211.38 last 223.54) - /unlink 46356 (time 1717019139.35 total 221.38 last 173.05) - /unlink 48180 (time 1717019149.35 total 231.38 last 182.38) - /unlink 50000 (time 1717019158.49 total 240.52 last 199.10) - /unlink 52017 (time 1717019168.49 total 250.52 last 201.70) - /unlink 54227 (time 1717019178.49 total 260.53 last 220.93) - /unlink 56744 (time 1717019188.49 total 270.53 last 251.60) - /unlink 59117 (time 1717019198.50 total 280.53 last 237.25) - /unlink 60000 (time 1717019201.88 total 283.91 last 261.36) - /unlink 62391 (time 1717019211.88 total 293.91 last 239.09) - /unlink 64722 (time 1717019221.88 total 303.91 last 233.07) total: 64768 /unlink in 304.12 seconds: 212.97 ops/second - open/close 1189 (time 1717019233.81 total 10.00 last 118.85) - open/close 2413 (time 1717019243.82 total 20.01 last 122.29) - open/close 3604 (time 1717019253.83 total 30.02 last 119.04) - open/close 4693 (time 1717019263.83 total 40.02 last 108.83) - open/close 5803 (time 1717019273.84 total 50.03 last 110.89) - open/close 6956 (time 1717019283.85 total 60.04 last 115.26) - open/close 8126 (time 1717019293.85 total 70.04 last 116.97) - open/close 9240 (time 1717019303.85 total 80.04 last 111.40) - open/close 10000 (time 1717019309.54 total 85.73 last 133.66) - open/close 11233 (time 1717019319.54 total 95.73 last 123.22) - open/close 12492 (time 1717019329.54 total 105.74 last 125.87) - open/close 13879 (time 1717019339.55 total 115.74 last 138.65) - open/close 15352 (time 1717019349.55 total 125.75 last 147.20) - open/close 16636 (time 1717019359.56 total 135.75 last 128.34) - open/close 17915 (time 1717019369.56 total 145.75 last 127.86) - open/close 19087 (time 1717019379.57 total 155.76 last 117.14) - open/close 20000 (time 1717019386.78 total 162.97 last 126.65) - open/close 21271 (time 1717019396.78 total 172.97 last 127.05) - open/close 22638 (time 1717019406.78 total 182.97 last 136.64) - open/close 24013 (time 1717019416.79 total 192.98 last 137.45) - open/close 25280 (time 1717019426.79 total 202.98 last 126.67) - open/close 26410 (time 1717019436.80 total 212.99 last 112.91) - open/close 28219 (time 1717019446.80 total 222.99 last 180.89) - open/close 29622 (time 1717019456.80 total 232.99 last 140.28) - open/close 30000 (time 1717019459.33 total 235.52 last 149.36) - open/close 31532 (time 1717019469.33 total 245.52 last 153.18) - open/close 32958 (time 1717019479.33 total 255.52 last 142.57) - open/close 34452 (time 1717019489.33 total 265.52 last 149.39) - open/close 35917 (time 1717019499.34 total 275.53 last 146.45) - open/close 37131 (time 1717019509.34 total 285.53 last 121.35) - open/close 38490 (time 1717019519.35 total 295.54 last 135.83) - open/close 39782 (time 1717019529.35 total 305.54 last 129.19) - open/close 41070 (time 1717019539.35 total 315.55 last 128.70) - open/close 42533 (time 1717019549.36 total 325.55 last 146.28) - open/close 43852 (time 1717019559.36 total 335.55 last 131.86) - open/close 45030 (time 1717019569.36 total 345.55 last 117.78) - open/close 46310 (time 1717019579.36 total 355.55 last 127.97) - open/close 47704 (time 1717019589.37 total 365.56 last 139.35) - open/close 48897 (time 1717019599.38 total 375.57 last 119.18) - open/close 50000 (time 1717019606.57 total 382.77 last 153.22) - open/close 51545 (time 1717019616.58 total 392.77 last 154.42) - open/close 52814 (time 1717019626.59 total 402.78 last 126.76) - open/close 54156 (time 1717019636.59 total 412.79 last 134.15) - open/close 55661 (time 1717019646.60 total 422.79 last 150.41) - open/close 57130 (time 1717019656.60 total 432.79 last 146.88) - open/close 58506 (time 1717019666.60 total 442.79 last 137.59) - open/close 59724 (time 1717019676.61 total 452.80 last 121.72) - open/close 60000 (time 1717019679.02 total 455.21 last 114.39) - open/close 61287 (time 1717019689.02 total 465.21 last 128.68) - open/close 62598 (time 1717019699.03 total 475.22 last 131.07) - open/close 64135 (time 1717019709.03 total 485.22 last 153.65) total: 64768 open/close in 488.92 seconds: 132.47 ops/second - /unlink 2615 (time 1717019723.91 total 10.00 last 261.44) - /unlink 5035 (time 1717019733.92 total 20.01 last 241.82) - /unlink 6994 (time 1717019743.92 total 30.01 last 195.90) - /unlink 8889 (time 1717019753.93 total 40.02 last 189.39) - /unlink 10000 (time 1717019759.59 total 45.68 last 196.24) - /unlink 11855 (time 1717019769.60 total 55.68 last 185.39) - /unlink 14342 (time 1717019779.60 total 65.68 last 248.67) - /unlink 16424 (time 1717019789.60 total 75.69 last 208.12) - /unlink 18608 (time 1717019799.60 total 85.69 last 218.33) - /unlink 20000 (time 1717019805.19 total 91.27 last 249.31) - /unlink 22157 (time 1717019815.19 total 101.28 last 215.62) - /unlink 24860 (time 1717019825.20 total 111.28 last 270.17) - /unlink 27322 (time 1717019835.20 total 121.28 last 246.17) - /unlink 29823 (time 1717019845.20 total 131.28 last 250.09) - /unlink 32858 (time 1717019855.20 total 141.28 last 303.48) - /unlink 35496 (time 1717019865.20 total 151.29 last 263.74) - /unlink 37910 (time 1717019875.20 total 161.29 last 241.38) - /unlink 40000 (time 1717019881.36 total 167.45 last 339.08) - /unlink 42907 (time 1717019891.37 total 177.45 last 290.67) - /unlink 45426 (time 1717019901.37 total 187.45 last 251.89) - /unlink 48104 (time 1717019911.37 total 197.46 last 267.74) - /unlink 50000 (time 1717019917.75 total 203.83 last 297.23) - /unlink 53346 (time 1717019927.75 total 213.84 last 334.52) - /unlink 56294 (time 1717019937.75 total 223.84 last 294.73) - /unlink 58768 (time 1717019947.75 total 233.84 last 247.38) - /unlink 60000 (time 1717019952.38 total 238.47 last 266.27) - /unlink 62460 (time 1717019962.38 total 248.47 last 245.95) total: 64768 /unlink in 256.24 seconds: 252.76 ops/second - open/close 1349 (time 1717019981.54 total 10.01 last 134.81) - open/close 2861 (time 1717019991.56 total 20.02 last 150.96) - open/close 4002 (time 1717020001.56 total 30.03 last 114.06) - open/close 5231 (time 1717020011.57 total 40.03 last 122.83) - open/close 6410 (time 1717020021.57 total 50.04 last 117.80) - open/close 7654 (time 1717020031.58 total 60.05 last 124.32) - open/close 8896 (time 1717020041.60 total 70.07 last 123.96) - open/close 10000 (time 1717020050.83 total 79.29 last 119.64) - open/close 11279 (time 1717020060.83 total 89.30 last 127.82) - open/close 12893 (time 1717020070.84 total 99.31 last 161.33) - open/close 14324 (time 1717020080.84 total 109.31 last 143.04) - open/close 15695 (time 1717020090.85 total 119.31 last 137.03) - open/close 16961 (time 1717020100.85 total 129.32 last 126.53) - open/close 18315 (time 1717020110.86 total 139.33 last 135.30) - open/close 19737 (time 1717020120.87 total 149.33 last 142.11) - open/close 21072 (time 1717020130.88 total 159.34 last 133.40) - open/close 22289 (time 1717020140.88 total 169.35 last 121.62) - open/close 23427 (time 1717020150.89 total 179.35 last 113.74) - open/close 24457 (time 1717020160.89 total 189.36 last 102.94) - open/close 25608 (time 1717020170.90 total 199.36 last 115.03) - open/close 26754 (time 1717020180.90 total 209.37 last 114.56) - open/close 27858 (time 1717020190.91 total 219.37 last 110.37) - open/close 29138 (time 1717020200.91 total 229.38 last 127.93) - open/close 30000 (time 1717020207.44 total 235.91 last 132.02) - open/close 31274 (time 1717020217.44 total 245.91 last 127.35) - open/close 32342 (time 1717020227.45 total 255.91 last 106.76) - open/close 33459 (time 1717020237.45 total 265.92 last 111.68) - open/close 34601 (time 1717020247.45 total 275.92 last 114.19) - open/close 35732 (time 1717020257.46 total 285.92 last 113.05) - open/close 36957 (time 1717020267.46 total 295.93 last 122.45) - open/close 38144 (time 1717020277.46 total 305.93 last 118.68) - open/close 39287 (time 1717020287.46 total 315.93 last 114.27) - open/close 40000 (time 1717020293.22 total 321.68 last 123.97) - open/close 41170 (time 1717020303.23 total 331.69 last 116.88) - open/close 42356 (time 1717020313.23 total 341.70 last 118.52) - open/close 43618 (time 1717020323.24 total 351.70 last 126.16) - open/close 44973 (time 1717020333.24 total 361.71 last 135.37) - open/close 46154 (time 1717020343.25 total 371.72 last 118.03) - open/close 47306 (time 1717020353.25 total 381.72 last 115.18) - open/close 48426 (time 1717020363.26 total 391.72 last 111.93) - open/close 49613 (time 1717020373.26 total 401.73 last 118.63) - open/close 50000 (time 1717020376.80 total 405.27 last 109.34) - open/close 51138 (time 1717020386.81 total 415.28 last 113.73) - open/close 52367 (time 1717020396.81 total 425.28 last 122.89) - open/close 53626 (time 1717020406.81 total 435.28 last 125.88) - open/close 54717 (time 1717020416.82 total 445.29 last 109.00) - open/close 55884 (time 1717020426.82 total 455.29 last 116.66) - open/close 56986 (time 1717020436.83 total 465.30 last 110.15) - open/close 58055 (time 1717020446.84 total 475.30 last 106.81) - open/close 59079 (time 1717020456.84 total 485.31 last 102.35) - open/close 60000 (time 1717020464.99 total 493.46 last 113.05) - open/close 61079 (time 1717020474.99 total 503.46 last 107.85) - open/close 62142 (time 1717020485.00 total 513.46 last 106.27) - open/close 63254 (time 1717020495.00 total 523.46 last 111.19) - open/close 64358 (time 1717020505.00 total 533.47 last 110.40) total: 64768 open/close in 537.18 seconds: 120.57 ops/second - /unlink 1891 (time 1717020520.67 total 10.00 last 189.09) - /unlink 3863 (time 1717020530.67 total 20.00 last 197.19) - /unlink 5953 (time 1717020540.67 total 30.00 last 209.00) - /unlink 8358 (time 1717020550.67 total 40.00 last 240.45) - /unlink 10000 (time 1717020557.44 total 46.77 last 242.56) - /unlink 12078 (time 1717020567.45 total 56.77 last 207.77) - /unlink 14073 (time 1717020577.45 total 66.78 last 199.42) - /unlink 16277 (time 1717020587.46 total 76.78 last 220.25) - /unlink 18363 (time 1717020597.46 total 86.79 last 208.58) - /unlink 20000 (time 1717020606.41 total 95.74 last 182.78) - /unlink 21714 (time 1717020616.41 total 105.74 last 171.37) - /unlink 23498 (time 1717020626.42 total 115.75 last 178.34) - /unlink 25543 (time 1717020636.42 total 125.75 last 204.49) - /unlink 27479 (time 1717020646.42 total 135.75 last 193.55) - /unlink 29234 (time 1717020656.42 total 145.75 last 175.49) - /unlink 30000 (time 1717020661.00 total 150.33 last 167.14) - /unlink 31787 (time 1717020671.01 total 160.34 last 178.65) - /unlink 33603 (time 1717020681.01 total 170.34 last 181.54) - /unlink 35639 (time 1717020691.01 total 180.34 last 203.55) - /unlink 37674 (time 1717020701.02 total 190.34 last 203.44) - /unlink 39470 (time 1717020711.02 total 200.35 last 179.58) - /unlink 40000 (time 1717020713.93 total 203.26 last 181.66) - /unlink 41980 (time 1717020723.94 total 213.26 last 197.97) - /unlink 44185 (time 1717020733.95 total 223.28 last 220.24) - /unlink 46125 (time 1717020743.95 total 233.28 last 193.95) - /unlink 48126 (time 1717020753.95 total 243.28 last 200.05) - /unlink 49965 (time 1717020763.96 total 253.29 last 183.82) - /unlink 51844 (time 1717020773.96 total 263.29 last 187.87) - /unlink 53899 (time 1717020783.96 total 273.29 last 205.43) - /unlink 55694 (time 1717020793.96 total 283.29 last 179.49) - /unlink 57565 (time 1717020803.96 total 293.29 last 187.05) - /unlink 59495 (time 1717020813.97 total 303.30 last 192.93) - /unlink 60000 (time 1717020816.54 total 305.87 last 196.09) - /unlink 62047 (time 1717020826.55 total 315.88 last 204.63) - /unlink 64369 (time 1717020836.55 total 325.88 last 232.12) total: 64768 /unlink in 328.07 seconds: 197.42 ops/second - open/close 1100 (time 1717020850.83 total 10.01 last 109.92) - open/close 2268 (time 1717020860.83 total 20.01 last 116.78) - open/close 3437 (time 1717020870.84 total 30.02 last 116.81) - open/close 4641 (time 1717020880.85 total 40.02 last 120.31) - open/close 5892 (time 1717020890.85 total 50.03 last 125.08) - open/close 7167 (time 1717020900.85 total 60.03 last 127.43) - open/close 8328 (time 1717020910.86 total 70.03 last 116.05) - open/close 9586 (time 1717020920.86 total 80.04 last 125.79) - open/close 10000 (time 1717020924.18 total 83.35 last 124.73) - open/close 11202 (time 1717020934.19 total 93.36 last 120.11) - open/close 12425 (time 1717020944.19 total 103.36 last 122.28) - open/close 13653 (time 1717020954.20 total 113.37 last 122.69) - open/close 14836 (time 1717020964.21 total 123.38 last 118.19) - open/close 16039 (time 1717020974.21 total 133.39 last 120.23) - open/close 17239 (time 1717020984.22 total 143.39 last 119.93) - open/close 18412 (time 1717020994.22 total 153.39 last 117.29) - open/close 19536 (time 1717021004.23 total 163.40 last 112.32) - open/close 20000 (time 1717021008.04 total 167.22 last 121.54) - open/close 21154 (time 1717021018.05 total 177.23 last 115.32) - open/close 22275 (time 1717021028.05 total 187.23 last 112.08) - open/close 23501 (time 1717021038.05 total 197.23 last 122.57) - open/close 24667 (time 1717021048.07 total 207.24 last 116.47) - open/close 25879 (time 1717021058.07 total 217.25 last 121.15) - open/close 27159 (time 1717021068.08 total 227.25 last 127.91) - open/close 28327 (time 1717021078.08 total 237.26 last 116.72) - open/close 29756 (time 1717021088.09 total 247.26 last 142.86) - open/close 30000 (time 1717021090.13 total 249.31 last 119.26) - open/close 31193 (time 1717021100.14 total 259.31 last 119.22) - open/close 32386 (time 1717021110.14 total 269.32 last 119.29) - open/close 33557 (time 1717021120.14 total 279.32 last 117.09) - open/close 34773 (time 1717021130.15 total 289.32 last 121.52) - open/close 35906 (time 1717021140.15 total 299.33 last 113.24) - open/close 37071 (time 1717021150.15 total 309.33 last 116.49) - open/close 38196 (time 1717021160.16 total 319.33 last 112.45) - open/close 39383 (time 1717021170.16 total 329.34 last 118.64) - open/close 40000 (time 1717021174.73 total 333.91 last 135.08) - open/close 41198 (time 1717021184.74 total 343.91 last 119.72) - open/close 42417 (time 1717021194.74 total 353.92 last 121.87) - open/close 43617 (time 1717021204.74 total 363.92 last 119.96) - open/close 44810 (time 1717021214.74 total 373.92 last 119.29) - open/close 46048 (time 1717021224.75 total 383.92 last 123.79) - open/close 47106 (time 1717021234.75 total 393.93 last 105.71) - open/close 48185 (time 1717021244.76 total 403.94 last 107.80) - open/close 49430 (time 1717021254.76 total 413.94 last 124.48) - open/close 50000 (time 1717021258.78 total 417.96 last 141.89) - open/close 51117 (time 1717021268.79 total 427.97 last 111.57) - open/close 52326 (time 1717021278.79 total 437.97 last 120.88) - open/close 53536 (time 1717021288.80 total 447.97 last 120.95) - open/close 54700 (time 1717021298.80 total 457.98 last 116.38) - open/close 56021 (time 1717021308.80 total 467.98 last 132.06) - open/close 57248 (time 1717021318.82 total 477.99 last 122.55) - open/close 58409 (time 1717021328.82 total 488.00 last 116.00) - open/close 59687 (time 1717021338.83 total 498.00 last 127.75) - open/close 60000 (time 1717021340.98 total 500.15 last 145.56) - open/close 61250 (time 1717021350.98 total 510.16 last 124.98) - open/close 62589 (time 1717021360.98 total 520.16 last 133.88) - open/close 63837 (time 1717021370.98 total 530.16 last 124.79) total: 64768 open/close in 538.55 seconds: 120.26 ops/second - /unlink 1912 (time 1717021391.26 total 10.01 last 191.08) - /unlink 4044 (time 1717021401.27 total 20.01 last 213.12) - /unlink 6008 (time 1717021411.27 total 30.02 last 196.30) - /unlink 7937 (time 1717021421.27 total 40.02 last 192.87) - /unlink 9931 (time 1717021431.28 total 50.02 last 199.34) - /unlink 12049 (time 1717021441.28 total 60.02 last 211.74) - /unlink 14101 (time 1717021451.28 total 70.02 last 205.18) - /unlink 16074 (time 1717021461.28 total 80.03 last 197.26) - /unlink 18117 (time 1717021471.29 total 90.03 last 204.24) - /unlink 20000 (time 1717021480.46 total 99.20 last 205.22) - /unlink 22039 (time 1717021490.47 total 109.21 last 203.83) - /unlink 24451 (time 1717021500.47 total 119.21 last 241.14) - /unlink 26375 (time 1717021510.47 total 129.21 last 192.32) - /unlink 28510 (time 1717021520.47 total 139.22 last 213.48) - /unlink 30000 (time 1717021527.93 total 146.67 last 199.76) - /unlink 32155 (time 1717021537.93 total 156.68 last 215.48) - /unlink 34229 (time 1717021547.94 total 166.68 last 207.34) - /unlink 36244 (time 1717021557.94 total 176.68 last 201.45) - /unlink 38143 (time 1717021567.94 total 186.68 last 189.85) - /unlink 40000 (time 1717021576.39 total 195.13 last 219.81) - /unlink 42262 (time 1717021586.39 total 205.13 last 226.13) - /unlink 44569 (time 1717021596.40 total 215.14 last 230.63) - /unlink 46745 (time 1717021606.40 total 225.14 last 217.53) - /unlink 48873 (time 1717021616.41 total 235.15 last 212.64) - /unlink 50000 (time 1717021622.35 total 241.09 last 189.73) - /unlink 52184 (time 1717021632.35 total 251.09 last 218.36) - /unlink 54256 (time 1717021642.35 total 261.09 last 207.16) - /unlink 56405 (time 1717021652.35 total 271.09 last 214.87) - /unlink 58856 (time 1717021662.35 total 281.10 last 245.05) - /unlink 60000 (time 1717021668.27 total 287.01 last 193.31) - /unlink 62138 (time 1717021678.28 total 297.02 last 213.66) - /unlink 64600 (time 1717021688.28 total 307.03 last 246.08) total: 64768 /unlink in 307.81 seconds: 210.42 ops/second sleep 5 for ZFS zfs sleep 5 for ZFS zfs Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg150-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg150-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg150-server stop mds service on oleg150-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg150-server LNET ready to unload unloading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing unload_modules_local oleg150-server: LNET ready to unload modules unloaded. pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-client: ssh exited with exit code 2 pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 PASS 106 (4134s) Stopping clients: oleg150-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg150-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg150-client: oleg150-server: ssh exited with exit code 2 oleg150-server: oleg150-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg150-server' oleg150-server: oleg150-server.virtnet: executing load_modules_local oleg150-server: Loading modules from /home/green/git/lustre-release/lustre oleg150-server: detected 4 online CPUs by sysfs oleg150-server: Force libcfs to create 2 CPU partitions oleg150-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg150-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg150-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory oleg150-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg150-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4255 sec ========== 18:29:25 (1717021765)