-----============= acceptance-small: conf-sanity ============----- Thu May 1 19:21:51 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg403-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg403-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg403-server oleg403-server: oleg403-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg403-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg403-server: oleg403-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg403-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:23:34 (1746141814) Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) oleg403-server: oleg403-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg403-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg403-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg403-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg403-server: oleg403-server.virtnet: executing set_default_debug -1 all pdsh@oleg403-client: oleg403-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg403-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg403-client.virtnet: -o user_xattr,flock 192.168.204.103@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1118 (time 1746141870.01 total 10.00 last 111.76) - open/close 2223 (time 1746141880.02 total 20.02 last 110.38) - open/close 3303 (time 1746141890.02 total 30.02 last 108.00) - open/close 4748 (time 1746141900.03 total 40.03 last 144.36) - open/close 5917 (time 1746141910.04 total 50.04 last 116.78) - open/close 6977 (time 1746141920.04 total 60.04 last 105.99) - open/close 7970 (time 1746141930.04 total 70.04 last 99.27) - open/close 9028 (time 1746141940.06 total 80.05 last 105.66) - open/close 10000 (time 1746141949.48 total 89.48 last 103.16) - open/close 10986 (time 1746141959.48 total 99.48 last 98.56) - open/close 12092 (time 1746141969.49 total 109.49 last 110.52) - open/close 13201 (time 1746141979.49 total 119.49 last 110.88) - open/close 14385 (time 1746141989.50 total 129.50 last 118.29) - open/close 15782 (time 1746141999.51 total 139.51 last 139.56) - open/close 17039 (time 1746142009.52 total 149.51 last 125.61) - open/close 18339 (time 1746142019.52 total 159.52 last 129.97) - open/close 19458 (time 1746142029.52 total 169.52 last 111.85) - open/close 20000 (time 1746142035.12 total 175.12 last 96.86) - open/close 21132 (time 1746142045.13 total 185.12 last 113.15) - open/close 22417 (time 1746142055.13 total 195.13 last 128.41) - open/close 23795 (time 1746142065.14 total 205.14 last 137.71) - open/close 24984 (time 1746142075.14 total 215.14 last 118.89) - open/close 26136 (time 1746142085.14 total 225.14 last 115.20) - open/close 27322 (time 1746142095.15 total 235.15 last 118.50) - open/close 28360 (time 1746142105.15 total 245.15 last 103.77) - open/close 29643 (time 1746142115.15 total 255.15 last 128.28) - open/close 30000 (time 1746142117.77 total 257.76 last 136.61) - open/close 31280 (time 1746142127.77 total 267.77 last 127.91) - open/close 32592 (time 1746142137.78 total 277.78 last 131.08) - open/close 33878 (time 1746142147.78 total 287.78 last 128.60) - open/close 35043 (time 1746142157.79 total 297.79 last 116.43) - open/close 36020 (time 1746142167.80 total 307.79 last 97.63) - open/close 37184 (time 1746142177.80 total 317.80 last 116.32) - open/close 38582 (time 1746142187.81 total 327.80 last 139.74) - open/close 39639 (time 1746142197.81 total 337.81 last 105.63) - open/close 40000 (time 1746142200.76 total 340.76 last 122.47) - open/close 41127 (time 1746142210.77 total 350.76 last 112.66) - open/close 42766 (time 1746142220.77 total 360.76 last 163.89) - open/close 44540 (time 1746142230.77 total 370.77 last 177.28) - open/close 45811 (time 1746142240.78 total 380.77 last 127.04) - open/close 47037 (time 1746142250.78 total 390.78 last 122.52) - open/close 48187 (time 1746142260.79 total 400.78 last 114.99) - open/close 49411 (time 1746142270.79 total 410.78 last 122.37) - open/close 50000 (time 1746142275.72 total 415.71 last 119.49) - open/close 51301 (time 1746142285.72 total 425.72 last 130.07) - open/close 52670 (time 1746142295.72 total 435.72 last 136.85) - open/close 54176 (time 1746142305.73 total 445.72 last 150.53) - open/close 55638 (time 1746142315.73 total 455.72 last 146.19) - open/close 56883 (time 1746142325.73 total 465.73 last 124.48) - open/close 58052 (time 1746142335.73 total 475.73 last 116.86) - open/close 59458 (time 1746142345.74 total 485.73 last 140.53) - open/close 60000 (time 1746142349.30 total 489.30 last 151.98) - open/close 61270 (time 1746142359.30 total 499.30 last 126.99) - open/close 62492 (time 1746142369.31 total 509.30 last 122.17) - open/close 63890 (time 1746142379.31 total 519.31 last 139.75) total: 64768 open/close in 526.87 seconds: 122.93 ops/second - /unlink 1818 (time 1746142398.51 total 10.00 last 181.74) - /unlink 3989 (time 1746142408.51 total 20.00 last 217.07) - /unlink 5949 (time 1746142418.51 total 30.01 last 195.97) - /unlink 8054 (time 1746142428.52 total 40.01 last 210.35) - /unlink 9749 (time 1746142438.52 total 50.02 last 169.46) - /unlink 11513 (time 1746142448.53 total 60.02 last 176.30) - /unlink 13746 (time 1746142458.53 total 70.03 last 223.21) - /unlink 16628 (time 1746142468.53 total 80.03 last 288.10) - /unlink 19474 (time 1746142478.54 total 90.04 last 284.34) - /unlink 20000 (time 1746142480.94 total 92.43 last 219.84) - /unlink 22324 (time 1746142490.94 total 102.43 last 232.36) - /unlink 24592 (time 1746142500.94 total 112.44 last 226.69) - /unlink 26490 (time 1746142510.94 total 122.44 last 189.78) - /unlink 28603 (time 1746142520.95 total 132.44 last 211.26) - /unlink 30000 (time 1746142527.24 total 138.73 last 222.04) - /unlink 31927 (time 1746142537.24 total 148.73 last 192.67) - /unlink 33655 (time 1746142547.24 total 158.74 last 172.71) - /unlink 35586 (time 1746142557.25 total 168.74 last 193.03) - /unlink 37285 (time 1746142567.25 total 178.74 last 169.89) - /unlink 38934 (time 1746142577.25 total 188.75 last 164.86) - /unlink 40000 (time 1746142583.31 total 194.81 last 175.81) - /unlink 41985 (time 1746142593.31 total 204.81 last 198.49) - /unlink 43842 (time 1746142603.32 total 214.81 last 185.64) - /unlink 45609 (time 1746142613.32 total 224.81 last 176.66) - /unlink 47460 (time 1746142623.32 total 234.82 last 185.09) - /unlink 49552 (time 1746142633.32 total 244.82 last 209.11) - /unlink 51850 (time 1746142643.33 total 254.83 last 229.67) - /unlink 53941 (time 1746142653.33 total 264.83 last 209.08) - /unlink 56348 (time 1746142663.34 total 274.83 last 240.51) - /unlink 59235 (time 1746142673.34 total 284.84 last 288.63) - /unlink 60000 (time 1746142676.36 total 287.86 last 253.44) - /unlink 62730 (time 1746142686.36 total 297.86 last 272.99) total: 64768 /unlink in 306.61 seconds: 211.24 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1309 (time 1746142725.47 total 10.00 last 130.89) - open/close 2448 (time 1746142735.48 total 20.01 last 113.76) - open/close 3481 (time 1746142745.50 total 30.03 last 103.11) - open/close 4566 (time 1746142755.51 total 40.04 last 108.42) - open/close 5708 (time 1746142765.51 total 50.04 last 114.17) - open/close 6921 (time 1746142775.51 total 60.04 last 121.28) - open/close 7984 (time 1746142785.52 total 70.05 last 106.26) - open/close 8935 (time 1746142795.52 total 80.06 last 95.02) - open/close 9956 (time 1746142805.53 total 90.06 last 102.08) - open/close 10918 (time 1746142815.54 total 100.07 last 96.10) - open/close 11920 (time 1746142825.55 total 110.08 last 100.10) - open/close 12965 (time 1746142835.56 total 120.09 last 104.38) - open/close 13987 (time 1746142845.56 total 130.09 last 102.19) - open/close 15069 (time 1746142855.56 total 140.09 last 108.20) - open/close 16208 (time 1746142865.56 total 150.09 last 113.86) - open/close 17296 (time 1746142875.57 total 160.10 last 108.75) - open/close 18459 (time 1746142885.58 total 170.12 last 116.10) - open/close 19799 (time 1746142895.59 total 180.12 last 133.93) - open/close 21006 (time 1746142905.59 total 190.12 last 120.65) - open/close 22151 (time 1746142915.60 total 200.14 last 114.37) - open/close 23158 (time 1746142925.61 total 210.14 last 100.63) - open/close 24225 (time 1746142935.61 total 220.15 last 106.67) - open/close 25336 (time 1746142945.62 total 230.15 last 111.06) - open/close 26530 (time 1746142955.62 total 240.15 last 119.36) - open/close 27947 (time 1746142965.63 total 250.16 last 141.61) - open/close 29327 (time 1746142975.64 total 260.17 last 137.86) - open/close 30000 (time 1746142981.28 total 265.81 last 119.33) - open/close 31516 (time 1746142991.28 total 275.81 last 151.58) - open/close 32758 (time 1746143001.28 total 285.81 last 124.20) - open/close 34069 (time 1746143011.28 total 295.81 last 131.08) - open/close 35432 (time 1746143021.29 total 305.82 last 136.18) - open/close 36654 (time 1746143031.29 total 315.82 last 122.19) - open/close 37791 (time 1746143041.29 total 325.82 last 113.70) - open/close 39050 (time 1746143051.29 total 335.82 last 125.89) - open/close 40000 (time 1746143059.04 total 343.57 last 122.60) - open/close 41284 (time 1746143069.04 total 353.57 last 128.38) - open/close 42426 (time 1746143079.05 total 363.58 last 114.10) - open/close 43573 (time 1746143089.06 total 373.59 last 114.61) - open/close 44889 (time 1746143099.06 total 383.59 last 131.59) - open/close 46428 (time 1746143109.06 total 393.60 last 153.84) - open/close 47905 (time 1746143119.07 total 403.60 last 147.68) - open/close 49333 (time 1746143129.07 total 413.60 last 142.79) - open/close 50000 (time 1746143134.22 total 418.76 last 129.33) - open/close 51491 (time 1746143144.22 total 428.76 last 149.09) - open/close 53411 (time 1746143154.23 total 438.76 last 191.94) - open/close 54604 (time 1746143164.23 total 448.76 last 119.29) - open/close 55730 (time 1746143174.23 total 458.76 last 112.55) - open/close 57057 (time 1746143184.24 total 468.77 last 132.63) - open/close 58469 (time 1746143194.24 total 478.77 last 141.17) - open/close 59787 (time 1746143204.25 total 488.78 last 131.73) - open/close 61118 (time 1746143214.25 total 498.78 last 133.02) - open/close 62484 (time 1746143224.25 total 508.78 last 136.59) - open/close 63698 (time 1746143234.26 total 518.79 last 121.30) total: 64768 open/close in 528.09 seconds: 122.65 ops/second - /unlink 2084 (time 1746143255.34 total 10.00 last 208.35) - /unlink 3803 (time 1746143265.35 total 20.00 last 171.87) - /unlink 5699 (time 1746143275.35 total 30.01 last 189.56) - /unlink 7702 (time 1746143285.35 total 40.01 last 200.19) - /unlink 9758 (time 1746143295.36 total 50.01 last 205.55) - /unlink 11898 (time 1746143305.36 total 60.02 last 213.96) - /unlink 14396 (time 1746143315.36 total 70.02 last 249.78) - /unlink 17395 (time 1746143325.36 total 80.02 last 299.83) - /unlink 20000 (time 1746143334.11 total 88.77 last 297.57) - /unlink 22390 (time 1746143344.12 total 98.78 last 238.89) - /unlink 25488 (time 1746143354.12 total 108.78 last 309.77) - /unlink 28007 (time 1746143364.12 total 118.78 last 251.85) - /unlink 30000 (time 1746143373.26 total 127.92 last 218.11) - /unlink 32015 (time 1746143383.26 total 137.92 last 201.48) - /unlink 34531 (time 1746143393.26 total 147.92 last 251.59) - /unlink 36594 (time 1746143403.26 total 157.92 last 206.25) - /unlink 38879 (time 1746143413.27 total 167.93 last 228.42) - /unlink 40000 (time 1746143417.69 total 172.35 last 253.21) - /unlink 43088 (time 1746143427.70 total 182.36 last 308.73) - /unlink 45905 (time 1746143437.70 total 192.36 last 281.61) - /unlink 49115 (time 1746143447.70 total 202.36 last 320.98) - /unlink 50000 (time 1746143450.00 total 204.66 last 385.31) - /unlink 53423 (time 1746143460.00 total 214.66 last 342.26) - /unlink 56270 (time 1746143470.00 total 224.66 last 284.59) - /unlink 59092 (time 1746143480.00 total 234.66 last 282.17) - /unlink 60000 (time 1746143483.40 total 238.06 last 267.51) - /unlink 62531 (time 1746143493.40 total 248.06 last 253.07) total: 64768 /unlink in 257.22 seconds: 251.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1195 (time 1746143532.12 total 10.01 last 119.44) - open/close 2284 (time 1746143542.12 total 20.01 last 108.88) - open/close 3494 (time 1746143552.14 total 30.03 last 120.74) - open/close 4565 (time 1746143562.15 total 40.04 last 106.98) - open/close 5653 (time 1746143572.16 total 50.05 last 108.70) - open/close 6696 (time 1746143582.17 total 60.06 last 104.23) - open/close 7880 (time 1746143592.17 total 70.06 last 118.36) - open/close 9122 (time 1746143602.17 total 80.06 last 124.17) - open/close 10000 (time 1746143609.96 total 87.85 last 112.79) - open/close 11354 (time 1746143619.97 total 97.85 last 135.28) - open/close 12840 (time 1746143629.97 total 107.86 last 148.53) - open/close 14236 (time 1746143639.98 total 117.87 last 139.46) - open/close 15535 (time 1746143649.99 total 127.87 last 129.84) - open/close 16820 (time 1746143659.99 total 137.88 last 128.47) - open/close 18241 (time 1746143670.00 total 147.88 last 142.01) - open/close 19444 (time 1746143680.00 total 157.89 last 120.26) - open/close 20000 (time 1746143683.81 total 161.70 last 145.92) - open/close 21474 (time 1746143693.82 total 171.70 last 147.31) - open/close 22684 (time 1746143703.82 total 181.71 last 120.96) - open/close 23969 (time 1746143713.82 total 191.71 last 128.47) - open/close 25213 (time 1746143723.82 total 201.71 last 124.36) - open/close 26348 (time 1746143733.84 total 211.73 last 113.31) - open/close 27223 (time 1746143743.85 total 221.73 last 87.44) - open/close 28293 (time 1746143753.85 total 231.74 last 106.97) - open/close 29548 (time 1746143763.86 total 241.74 last 125.41) - open/close 30000 (time 1746143767.67 total 245.56 last 118.51) - open/close 31202 (time 1746143777.67 total 255.56 last 120.18) - open/close 32437 (time 1746143787.67 total 265.56 last 123.49) - open/close 33870 (time 1746143797.67 total 275.56 last 143.28) - open/close 35208 (time 1746143807.68 total 285.56 last 133.76) - open/close 36620 (time 1746143817.68 total 295.57 last 141.11) - open/close 37858 (time 1746143827.69 total 305.58 last 123.75) - open/close 39128 (time 1746143837.69 total 315.58 last 126.94) - open/close 40000 (time 1746143845.12 total 323.01 last 117.36) - open/close 41157 (time 1746143855.13 total 333.01 last 115.66) - open/close 42190 (time 1746143865.14 total 343.03 last 103.15) - open/close 43273 (time 1746143875.14 total 353.03 last 108.27) - open/close 44218 (time 1746143885.14 total 363.03 last 94.49) - open/close 45283 (time 1746143895.15 total 373.04 last 106.40) - open/close 46440 (time 1746143905.17 total 383.06 last 115.53) - open/close 47521 (time 1746143915.17 total 393.06 last 108.09) - open/close 48653 (time 1746143925.18 total 403.07 last 113.05) - open/close 49879 (time 1746143935.19 total 413.08 last 122.51) - open/close 51019 (time 1746143945.19 total 423.08 last 113.99) - open/close 52369 (time 1746143955.19 total 433.08 last 134.99) - open/close 53966 (time 1746143965.20 total 443.08 last 159.63) - open/close 55559 (time 1746143975.20 total 453.08 last 159.28) - open/close 56885 (time 1746143985.20 total 463.09 last 132.54) - open/close 58217 (time 1746143995.21 total 473.09 last 133.13) - open/close 59638 (time 1746144005.21 total 483.10 last 142.08) - open/close 60000 (time 1746144007.83 total 485.72 last 137.93) - open/close 61435 (time 1746144017.84 total 495.72 last 143.46) - open/close 62864 (time 1746144027.84 total 505.72 last 142.87) - open/close 64641 (time 1746144037.84 total 515.73 last 177.64) total: 64768 open/close in 516.54 seconds: 125.39 ops/second - /unlink 2327 (time 1746144050.20 total 10.00 last 232.65) - /unlink 4822 (time 1746144060.20 total 20.00 last 249.45) - /unlink 7830 (time 1746144070.20 total 30.01 last 300.67) - /unlink 10000 (time 1746144078.29 total 38.10 last 268.22) - /unlink 12584 (time 1746144088.30 total 48.10 last 258.33) - /unlink 15036 (time 1746144098.30 total 58.11 last 245.08) - /unlink 17408 (time 1746144108.30 total 68.11 last 237.19) - /unlink 19545 (time 1746144118.30 total 78.11 last 213.67) - /unlink 20000 (time 1746144120.60 total 80.40 last 198.39) - /unlink 21941 (time 1746144130.60 total 90.41 last 194.01) - /unlink 24769 (time 1746144140.60 total 100.41 last 282.75) - /unlink 27595 (time 1746144150.60 total 110.41 last 282.51) - /unlink 29955 (time 1746144160.61 total 120.41 last 235.94) - /unlink 33144 (time 1746144170.61 total 130.42 last 318.81) - /unlink 35606 (time 1746144180.61 total 140.42 last 246.20) - /unlink 37780 (time 1746144190.61 total 150.42 last 217.34) - /unlink 40000 (time 1746144200.05 total 159.85 last 235.31) - /unlink 42228 (time 1746144210.05 total 169.86 last 222.72) - /unlink 44361 (time 1746144220.05 total 179.86 last 213.25) - /unlink 46458 (time 1746144230.05 total 189.86 last 209.67) - /unlink 48909 (time 1746144240.06 total 199.86 last 245.09) - /unlink 50000 (time 1746144244.89 total 204.70 last 225.73) - /unlink 51880 (time 1746144254.90 total 214.70 last 187.85) - /unlink 53753 (time 1746144264.90 total 224.71 last 187.22) - /unlink 55445 (time 1746144274.91 total 234.71 last 169.11) - /unlink 57239 (time 1746144284.91 total 244.72 last 179.30) - /unlink 59111 (time 1746144294.91 total 254.72 last 187.19) - /unlink 60000 (time 1746144299.02 total 258.83 last 216.45) - /unlink 61865 (time 1746144309.02 total 268.83 last 186.45) - /unlink 63888 (time 1746144319.03 total 278.83 last 202.23) total: 64768 /unlink in 282.95 seconds: 228.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1507 (time 1746144353.23 total 10.00 last 150.67) - open/close 2838 (time 1746144363.23 total 20.00 last 133.06) - open/close 4278 (time 1746144373.23 total 30.01 last 143.97) - open/close 5764 (time 1746144383.24 total 40.01 last 148.53) - open/close 7236 (time 1746144393.24 total 50.01 last 147.18) - open/close 8983 (time 1746144403.24 total 60.02 last 174.61) - open/close 10000 (time 1746144408.98 total 65.75 last 177.40) - open/close 11360 (time 1746144418.98 total 75.76 last 135.93) - open/close 12505 (time 1746144428.99 total 85.76 last 114.45) - open/close 13582 (time 1746144438.99 total 95.77 last 107.65) - open/close 14861 (time 1746144449.00 total 105.77 last 127.86) - open/close 15947 (time 1746144459.00 total 115.77 last 108.55) - open/close 16906 (time 1746144469.00 total 125.77 last 95.89) - open/close 18003 (time 1746144479.01 total 135.79 last 109.56) - open/close 19057 (time 1746144489.02 total 145.80 last 105.30) - open/close 20000 (time 1746144498.49 total 155.26 last 99.61) - open/close 21092 (time 1746144508.49 total 165.27 last 109.18) - open/close 22244 (time 1746144518.49 total 175.27 last 115.19) - open/close 23470 (time 1746144528.50 total 185.27 last 122.52) - open/close 24632 (time 1746144538.51 total 195.28 last 116.09) - open/close 25769 (time 1746144548.52 total 205.29 last 113.65) - open/close 26882 (time 1746144558.52 total 215.30 last 111.23) - open/close 27912 (time 1746144568.53 total 225.30 last 102.97) - open/close 29100 (time 1746144578.53 total 235.30 last 118.75) - open/close 30000 (time 1746144586.09 total 242.86 last 119.01) - open/close 31068 (time 1746144596.10 total 252.87 last 106.75) - open/close 32360 (time 1746144606.10 total 262.87 last 129.16) - open/close 33662 (time 1746144616.10 total 272.88 last 130.16) - open/close 35099 (time 1746144626.11 total 282.88 last 143.65) - open/close 36434 (time 1746144636.11 total 292.88 last 133.49) - open/close 37650 (time 1746144646.11 total 302.89 last 121.53) - open/close 38795 (time 1746144656.11 total 312.89 last 114.50) - open/close 39937 (time 1746144666.11 total 322.89 last 114.17) - open/close 41040 (time 1746144676.12 total 332.89 last 110.29) - open/close 42339 (time 1746144686.12 total 342.89 last 129.87) - open/close 43640 (time 1746144696.12 total 352.89 last 130.09) - open/close 44898 (time 1746144706.12 total 362.90 last 125.76) - open/close 46254 (time 1746144716.13 total 372.90 last 135.50) - open/close 47407 (time 1746144726.13 total 382.91 last 115.24) - open/close 48646 (time 1746144736.14 total 392.92 last 123.80) - open/close 49854 (time 1746144746.14 total 402.92 last 120.78) - open/close 51109 (time 1746144756.14 total 412.92 last 125.49) - open/close 52267 (time 1746144766.15 total 422.93 last 115.71) - open/close 53440 (time 1746144776.16 total 432.93 last 117.26) - open/close 54645 (time 1746144786.16 total 442.93 last 120.44) - open/close 55956 (time 1746144796.16 total 452.94 last 131.04) - open/close 57062 (time 1746144806.17 total 462.94 last 110.60) - open/close 58597 (time 1746144816.17 total 472.94 last 153.44) - open/close 60000 (time 1746144825.93 total 482.70 last 143.80) - open/close 61579 (time 1746144835.93 total 492.71 last 157.77) - open/close 62796 (time 1746144845.94 total 502.71 last 121.62) - open/close 63947 (time 1746144855.95 total 512.72 last 115.04) total: 64768 open/close in 519.41 seconds: 124.69 ops/second - /unlink 3029 (time 1746144874.20 total 10.00 last 302.84) - /unlink 5836 (time 1746144884.21 total 20.00 last 280.62) - /unlink 8364 (time 1746144894.21 total 30.01 last 252.72) - /unlink 10000 (time 1746144901.36 total 37.16 last 228.81) - /unlink 12779 (time 1746144911.36 total 47.16 last 277.86) - /unlink 15367 (time 1746144921.36 total 57.16 last 258.80) - /unlink 18229 (time 1746144931.36 total 67.16 last 286.14) - /unlink 20000 (time 1746144937.37 total 73.17 last 294.81) - /unlink 23388 (time 1746144947.37 total 83.17 last 338.74) - /unlink 27011 (time 1746144957.37 total 93.17 last 362.23) - /unlink 30000 (time 1746144965.48 total 101.28 last 368.69) - /unlink 32958 (time 1746144975.48 total 111.28 last 295.76) - /unlink 35846 (time 1746144985.49 total 121.29 last 288.65) - /unlink 37896 (time 1746144995.50 total 131.29 last 204.83) - /unlink 40000 (time 1746145003.77 total 139.57 last 254.28) - /unlink 43133 (time 1746145013.77 total 149.57 last 313.23) - /unlink 45503 (time 1746145023.77 total 159.57 last 236.97) - /unlink 47636 (time 1746145033.77 total 169.57 last 213.27) - /unlink 50000 (time 1746145041.76 total 177.56 last 295.99) - /unlink 52939 (time 1746145051.76 total 187.56 last 293.81) - /unlink 55667 (time 1746145061.77 total 197.57 last 272.73) - /unlink 59159 (time 1746145071.77 total 207.57 last 349.16) - /unlink 60000 (time 1746145074.06 total 209.86 last 366.87) - /unlink 62690 (time 1746145084.06 total 219.86 last 268.98) total: 64768 /unlink in 227.15 seconds: 285.14 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 1868 (time 1746145124.50 total 10.00 last 186.78) - open/close 4254 (time 1746145134.51 total 20.01 last 238.36) - open/close 5790 (time 1746145144.52 total 30.01 last 153.56) - open/close 8337 (time 1746145154.52 total 40.01 last 254.68) - open/close 10000 (time 1746145158.96 total 44.46 last 374.04) - open/close 12568 (time 1746145168.97 total 54.46 last 256.74) - open/close 14432 (time 1746145178.97 total 64.46 last 186.38) - open/close 16527 (time 1746145188.97 total 74.46 last 209.48) - open/close 18438 (time 1746145198.97 total 84.47 last 191.08) - open/close 20000 (time 1746145206.39 total 91.88 last 210.63) - open/close 21947 (time 1746145216.39 total 101.88 last 194.66) - open/close 23666 (time 1746145226.39 total 111.89 last 171.79) - open/close 25473 (time 1746145236.40 total 121.89 last 180.67) - open/close 27644 (time 1746145246.40 total 131.89 last 217.05) - open/close 29616 (time 1746145256.41 total 141.90 last 197.06) - open/close 30000 (time 1746145258.85 total 144.35 last 157.06) - open/close 31826 (time 1746145268.85 total 154.35 last 182.56) - open/close 33538 (time 1746145278.86 total 164.35 last 171.12) - open/close 35000 (time 1746145288.87 total 174.36 last 146.05) - open/close 36565 (time 1746145298.88 total 184.37 last 156.36) - open/close 38328 (time 1746145308.88 total 194.38 last 176.16) - open/close 39831 (time 1746145318.89 total 204.38 last 150.26) - open/close 42049 (time 1746145328.89 total 214.39 last 221.75) - open/close 44400 (time 1746145338.89 total 224.39 last 235.03) - open/close 46290 (time 1746145348.89 total 234.39 last 188.98) - open/close 48030 (time 1746145358.90 total 244.40 last 173.83) - open/close 49954 (time 1746145368.91 total 254.40 last 192.34) - open/close 51448 (time 1746145378.91 total 264.40 last 149.36) - open/close 53229 (time 1746145388.91 total 274.41 last 178.06) - open/close 54671 (time 1746145398.92 total 284.41 last 144.13) - open/close 56053 (time 1746145408.92 total 294.41 last 138.17) - open/close 57808 (time 1746145418.92 total 304.42 last 175.46) - open/close 60000 (time 1746145427.30 total 312.80 last 261.51) - open/close 61817 (time 1746145437.30 total 322.80 last 181.65) - open/close 63839 (time 1746145447.31 total 332.80 last 202.14) total: 64768 open/close in 338.69 seconds: 191.23 ops/second - /unlink 3009 (time 1746145464.54 total 10.00 last 300.83) - /unlink 7453 (time 1746145474.54 total 20.00 last 444.31) - /unlink 10000 (time 1746145482.14 total 27.61 last 335.09) - /unlink 13783 (time 1746145492.14 total 37.61 last 378.30) - /unlink 15970 (time 1746145502.14 total 47.61 last 218.67) - /unlink 19074 (time 1746145512.14 total 57.61 last 310.35) - /unlink 20000 (time 1746145514.45 total 59.91 last 402.08) - /unlink 22737 (time 1746145524.45 total 69.92 last 273.56) - /unlink 25975 (time 1746145534.45 total 79.92 last 323.69) - /unlink 28814 (time 1746145544.46 total 89.92 last 283.81) - /unlink 30000 (time 1746145549.02 total 94.48 last 260.10) - /unlink 32145 (time 1746145559.02 total 104.48 last 214.47) - /unlink 33829 (time 1746145569.02 total 114.49 last 168.35) - /unlink 35432 (time 1746145579.03 total 124.49 last 160.21) - /unlink 37223 (time 1746145589.03 total 134.49 last 179.08) - /unlink 39489 (time 1746145599.03 total 144.49 last 226.59) - /unlink 40000 (time 1746145601.48 total 146.95 last 208.36) - /unlink 41916 (time 1746145611.48 total 156.95 last 191.58) - /unlink 44221 (time 1746145621.48 total 166.95 last 230.45) - /unlink 46398 (time 1746145631.49 total 176.95 last 217.67) - /unlink 48421 (time 1746145641.49 total 186.95 last 202.23) - /unlink 50000 (time 1746145649.06 total 194.53 last 208.54) - /unlink 52091 (time 1746145659.06 total 204.53 last 209.04) - /unlink 54268 (time 1746145669.07 total 214.53 last 217.66) - /unlink 56169 (time 1746145679.07 total 224.53 last 190.09) - /unlink 58457 (time 1746145689.07 total 234.54 last 228.72) - /unlink 60000 (time 1746145697.13 total 242.59 last 191.48) - /unlink 62173 (time 1746145707.13 total 252.60 last 217.18) total: 64768 /unlink in 261.99 seconds: 247.22 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg403-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg403-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg403-server stop mds service on oleg403-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg403-server unloading modules via unload_modules_local on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-client: ssh exited with exit code 2 pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 PASS 106 (3935s) Stopping clients: oleg403-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg403-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg403-client: oleg403-server: ssh exited with exit code 2 oleg403-server: oleg403-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg403-server' oleg403-server: oleg403-server.virtnet: executing load_modules_local oleg403-server: Loading modules from /home/green/git/lustre-release/lustre oleg403-server: detected 4 online CPUs by sysfs oleg403-server: MODOPTS_LIBCFS= oleg403-server: Force libcfs to create 2 CPU partitions oleg403-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory oleg403-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg403-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 4065 sec ========== 20:29:37 (1746145777) === conf-sanity: start cleanup 20:29:37 (1746145777) === === conf-sanity: finish cleanup 20:29:39 (1746145779) ===