-----============= acceptance-small: conf-sanity ============----- Wed Apr 30 22:17:53 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg428-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg428-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg428-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg428-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg428-server oleg428-server: oleg428-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: detected 4 online CPUs by sysfs oleg428-server: MODOPTS_LIBCFS= oleg428-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg428-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg428-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg428-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg428-server: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg428-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg428-server: oleg428-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg428-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg428-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:19:41 (1746065981) Stopping clients: oleg428-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg428-client.virtnet /mnt/lustre2 (opts:-f) oleg428-server: oleg428-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: detected 4 online CPUs by sysfs oleg428-server: MODOPTS_LIBCFS= oleg428-server: Force libcfs to create 2 CPU partitions oleg428-server: libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg428-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg428-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg428-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg428-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg428-server: oleg428-server.virtnet: executing set_default_debug -1 all pdsh@oleg428-client: oleg428-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg428-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg428-client.virtnet: -o user_xattr,flock 192.168.204.128@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1734 (time 1746066042.53 total 10.00 last 173.35) - open/close 3201 (time 1746066052.53 total 20.00 last 146.69) - open/close 4675 (time 1746066062.54 total 30.01 last 147.32) - open/close 6146 (time 1746066072.54 total 40.01 last 147.09) - open/close 7677 (time 1746066082.54 total 50.01 last 153.09) - open/close 9157 (time 1746066092.55 total 60.02 last 147.90) - open/close 10000 (time 1746066098.45 total 65.92 last 142.88) - open/close 11737 (time 1746066108.45 total 75.92 last 173.67) - open/close 13111 (time 1746066118.45 total 85.92 last 137.34) - open/close 14492 (time 1746066128.46 total 95.93 last 138.04) - open/close 15853 (time 1746066138.46 total 105.93 last 136.07) - open/close 17407 (time 1746066148.46 total 115.94 last 155.31) - open/close 18826 (time 1746066158.47 total 125.94 last 141.90) - open/close 20000 (time 1746066166.45 total 133.92 last 147.06) - open/close 21458 (time 1746066176.45 total 143.93 last 145.72) - open/close 22810 (time 1746066186.45 total 153.93 last 135.20) - open/close 24291 (time 1746066196.46 total 163.93 last 148.05) - open/close 25932 (time 1746066206.47 total 173.94 last 163.98) - open/close 27513 (time 1746066216.47 total 183.94 last 158.08) - open/close 29086 (time 1746066226.47 total 193.94 last 157.22) - open/close 30000 (time 1746066232.39 total 199.87 last 154.34) - open/close 31453 (time 1746066242.40 total 209.87 last 145.24) - open/close 33065 (time 1746066252.40 total 219.87 last 161.17) - open/close 34472 (time 1746066262.41 total 229.88 last 140.62) - open/close 35922 (time 1746066272.41 total 239.88 last 144.94) - open/close 37369 (time 1746066282.41 total 249.89 last 144.63) - open/close 38852 (time 1746066292.42 total 259.89 last 148.28) - open/close 40000 (time 1746066300.58 total 268.05 last 140.67) - open/close 41393 (time 1746066310.58 total 278.05 last 139.26) - open/close 42929 (time 1746066320.59 total 288.06 last 153.51) - open/close 44281 (time 1746066330.59 total 298.06 last 135.19) - open/close 45582 (time 1746066340.59 total 308.06 last 130.09) - open/close 46976 (time 1746066350.59 total 318.06 last 139.36) - open/close 48387 (time 1746066360.60 total 328.07 last 141.02) - open/close 49794 (time 1746066370.60 total 338.07 last 140.59) - open/close 51343 (time 1746066380.61 total 348.08 last 154.83) - open/close 52876 (time 1746066390.61 total 358.09 last 153.20) - open/close 54278 (time 1746066400.62 total 368.09 last 140.15) - open/close 55862 (time 1746066410.62 total 378.09 last 158.37) - open/close 57274 (time 1746066420.62 total 388.09 last 141.19) - open/close 58805 (time 1746066430.63 total 398.10 last 152.99) - open/close 60000 (time 1746066437.43 total 404.90 last 175.69) - open/close 61817 (time 1746066447.43 total 414.90 last 181.61) - open/close 63849 (time 1746066457.44 total 424.91 last 203.03) total: 64768 open/close in 429.86 seconds: 150.67 ops/second - /unlink 2913 (time 1746066473.69 total 10.00 last 291.24) - /unlink 5502 (time 1746066483.70 total 20.00 last 258.86) - /unlink 8168 (time 1746066493.70 total 30.01 last 266.49) - /unlink 10000 (time 1746066501.21 total 37.52 last 243.88) - /unlink 12699 (time 1746066511.21 total 47.52 last 269.87) - /unlink 15494 (time 1746066521.22 total 57.53 last 279.37) - /unlink 17876 (time 1746066531.22 total 67.53 last 238.08) - /unlink 20000 (time 1746066538.22 total 74.52 last 303.70) - /unlink 22678 (time 1746066548.22 total 84.52 last 267.80) - /unlink 25082 (time 1746066558.22 total 94.53 last 240.32) - /unlink 27745 (time 1746066568.22 total 104.53 last 266.20) - /unlink 30000 (time 1746066576.22 total 112.53 last 282.07) - /unlink 32766 (time 1746066586.22 total 122.53 last 276.59) - /unlink 35761 (time 1746066596.22 total 132.53 last 299.47) - /unlink 38579 (time 1746066606.22 total 142.53 last 281.80) - /unlink 40000 (time 1746066611.61 total 147.92 last 263.69) - /unlink 42422 (time 1746066621.61 total 157.92 last 242.17) - /unlink 44911 (time 1746066631.61 total 167.92 last 248.89) - /unlink 47314 (time 1746066641.61 total 177.92 last 240.24) - /unlink 49988 (time 1746066651.61 total 187.92 last 267.39) - /unlink 52913 (time 1746066661.61 total 197.92 last 292.45) - /unlink 55586 (time 1746066671.62 total 207.92 last 267.26) - /unlink 58034 (time 1746066681.62 total 217.93 last 244.75) - /unlink 60000 (time 1746066689.71 total 226.01 last 243.11) - /unlink 62418 (time 1746066699.71 total 236.01 last 241.80) total: 64768 /unlink in 245.17 seconds: 264.18 ops/second Waiting for MDT destroys to complete - open/close 1565 (time 1746066726.92 total 10.01 last 156.35) - open/close 3063 (time 1746066736.93 total 20.01 last 149.79) - open/close 4543 (time 1746066746.93 total 30.01 last 147.94) - open/close 6156 (time 1746066756.93 total 40.02 last 161.28) - open/close 7799 (time 1746066766.93 total 50.02 last 164.29) - open/close 9279 (time 1746066776.94 total 60.02 last 147.91) - open/close 10000 (time 1746066781.46 total 64.54 last 159.46) - open/close 11495 (time 1746066791.46 total 74.54 last 149.50) - open/close 13100 (time 1746066801.47 total 84.55 last 160.36) - open/close 14793 (time 1746066811.47 total 94.55 last 169.29) - open/close 16838 (time 1746066821.47 total 104.56 last 204.47) - open/close 18783 (time 1746066831.47 total 114.56 last 194.47) - open/close 20000 (time 1746066838.52 total 121.61 last 172.63) - open/close 21580 (time 1746066848.53 total 131.61 last 157.92) - open/close 23038 (time 1746066858.53 total 141.62 last 145.74) - open/close 24328 (time 1746066868.54 total 151.62 last 128.94) - open/close 25868 (time 1746066878.54 total 161.63 last 153.92) - open/close 27430 (time 1746066888.55 total 171.63 last 156.12) - open/close 28728 (time 1746066898.55 total 181.63 last 129.77) - open/close 29878 (time 1746066908.55 total 191.63 last 114.98) - open/close 31099 (time 1746066918.55 total 201.64 last 122.07) - open/close 32339 (time 1746066928.56 total 211.64 last 123.93) - open/close 33561 (time 1746066938.56 total 221.64 last 122.19) - open/close 34770 (time 1746066948.56 total 231.65 last 120.82) - open/close 35989 (time 1746066958.57 total 241.65 last 121.84) - open/close 37320 (time 1746066968.58 total 251.66 last 133.00) - open/close 38768 (time 1746066978.58 total 261.67 last 144.71) - open/close 40000 (time 1746066988.29 total 271.37 last 126.94) - open/close 41235 (time 1746066998.29 total 281.38 last 123.47) - open/close 42447 (time 1746067008.29 total 291.38 last 121.20) - open/close 43625 (time 1746067018.29 total 301.38 last 117.79) - open/close 44737 (time 1746067028.30 total 311.38 last 111.15) - open/close 46020 (time 1746067038.30 total 321.38 last 128.30) - open/close 47352 (time 1746067048.30 total 331.39 last 133.14) - open/close 48612 (time 1746067058.31 total 341.39 last 125.94) - open/close 49842 (time 1746067068.31 total 351.40 last 122.94) - open/close 51148 (time 1746067078.31 total 361.40 last 130.57) - open/close 52455 (time 1746067088.31 total 371.40 last 130.70) - open/close 53927 (time 1746067098.32 total 381.40 last 147.14) - open/close 55392 (time 1746067108.32 total 391.41 last 146.45) - open/close 56827 (time 1746067118.32 total 401.41 last 143.47) - open/close 58370 (time 1746067128.33 total 411.41 last 154.24) - open/close 59922 (time 1746067138.33 total 421.42 last 155.12) - open/close 61430 (time 1746067148.33 total 431.42 last 150.79) - open/close 62950 (time 1746067158.34 total 441.42 last 151.90) total: 64768 open/close in 450.23 seconds: 143.86 ops/second - /unlink 3288 (time 1746067178.23 total 10.00 last 328.65) - /unlink 6248 (time 1746067188.23 total 20.00 last 296.00) - /unlink 8790 (time 1746067198.23 total 30.01 last 254.15) - /unlink 10000 (time 1746067202.74 total 34.51 last 268.38) - /unlink 12846 (time 1746067212.74 total 44.51 last 284.60) - /unlink 15759 (time 1746067222.74 total 54.52 last 291.30) - /unlink 18470 (time 1746067232.74 total 64.52 last 270.97) - /unlink 20000 (time 1746067238.09 total 69.87 last 285.96) - /unlink 23879 (time 1746067248.09 total 79.87 last 387.84) - /unlink 27308 (time 1746067258.10 total 89.87 last 342.82) - /unlink 30000 (time 1746067267.58 total 99.35 last 283.94) - /unlink 32490 (time 1746067277.59 total 109.36 last 248.78) - /unlink 35518 (time 1746067287.59 total 119.37 last 302.75) - /unlink 38614 (time 1746067297.59 total 129.37 last 309.56) - /unlink 40000 (time 1746067302.31 total 134.09 last 293.72) - /unlink 43500 (time 1746067312.31 total 144.09 last 349.90) - /unlink 46407 (time 1746067322.31 total 154.09 last 290.64) - /unlink 49672 (time 1746067332.31 total 164.09 last 326.49) - /unlink 53421 (time 1746067342.31 total 174.09 last 374.86) - /unlink 58211 (time 1746067352.31 total 184.09 last 478.95) - /unlink 60000 (time 1746067356.75 total 188.53 last 403.55) - /unlink 64341 (time 1746067366.75 total 198.53 last 434.09) total: 64768 /unlink in 200.05 seconds: 323.75 ops/second Waiting for MDT destroys to complete - open/close 1560 (time 1746067390.13 total 10.00 last 155.96) - open/close 3089 (time 1746067400.13 total 20.00 last 152.89) - open/close 4567 (time 1746067410.13 total 30.00 last 147.78) - open/close 6287 (time 1746067420.13 total 40.01 last 171.98) - open/close 7929 (time 1746067430.13 total 50.01 last 164.20) - open/close 9152 (time 1746067440.13 total 60.01 last 122.28) - open/close 10000 (time 1746067446.88 total 66.76 last 125.58) - open/close 11188 (time 1746067456.89 total 76.76 last 118.79) - open/close 12351 (time 1746067466.91 total 86.78 last 116.07) - open/close 13521 (time 1746067476.92 total 96.79 last 116.85) - open/close 14646 (time 1746067486.92 total 106.80 last 112.43) - open/close 15854 (time 1746067496.93 total 116.80 last 120.75) - open/close 17015 (time 1746067506.93 total 126.81 last 116.07) - open/close 18926 (time 1746067516.93 total 136.81 last 191.04) - open/close 20000 (time 1746067522.82 total 142.70 last 182.34) - open/close 21346 (time 1746067532.83 total 152.70 last 134.55) - open/close 22813 (time 1746067542.84 total 162.71 last 146.59) - open/close 24378 (time 1746067552.84 total 172.72 last 156.43) - open/close 25669 (time 1746067562.84 total 182.72 last 129.06) - open/close 27147 (time 1746067572.85 total 192.72 last 147.77) - open/close 28565 (time 1746067582.85 total 202.73 last 141.75) - open/close 30000 (time 1746067592.69 total 212.57 last 145.79) - open/close 31658 (time 1746067602.69 total 222.57 last 165.78) - open/close 33464 (time 1746067612.70 total 232.58 last 180.50) - open/close 34947 (time 1746067622.71 total 242.58 last 148.17) - open/close 36445 (time 1746067632.72 total 252.59 last 149.67) - open/close 38032 (time 1746067642.72 total 262.59 last 158.69) - open/close 39852 (time 1746067652.72 total 272.60 last 181.95) - open/close 41485 (time 1746067662.72 total 282.60 last 163.28) - open/close 43208 (time 1746067672.72 total 292.60 last 172.26) - open/close 45427 (time 1746067682.73 total 302.60 last 221.81) - open/close 47385 (time 1746067692.73 total 312.61 last 195.73) - open/close 49109 (time 1746067702.74 total 322.61 last 172.33) - open/close 50000 (time 1746067708.15 total 328.02 last 164.64) - open/close 51531 (time 1746067718.15 total 338.03 last 153.01) - open/close 53110 (time 1746067728.16 total 348.03 last 157.85) - open/close 54582 (time 1746067738.16 total 358.04 last 147.15) - open/close 55961 (time 1746067748.17 total 368.05 last 137.73) - open/close 57126 (time 1746067758.17 total 378.05 last 116.48) - open/close 58300 (time 1746067768.18 total 388.06 last 117.32) - open/close 59492 (time 1746067778.18 total 398.06 last 119.19) - open/close 60000 (time 1746067782.08 total 401.96 last 130.23) - open/close 61173 (time 1746067792.09 total 411.97 last 117.17) - open/close 62612 (time 1746067802.09 total 421.97 last 143.87) - open/close 63985 (time 1746067812.10 total 431.98 last 137.22) total: 64768 open/close in 439.05 seconds: 147.52 ops/second - /unlink 1888 (time 1746067830.90 total 10.00 last 188.78) - /unlink 3600 (time 1746067840.91 total 20.00 last 171.18) - /unlink 5299 (time 1746067850.91 total 30.01 last 169.80) - /unlink 7004 (time 1746067860.92 total 40.01 last 170.40) - /unlink 9017 (time 1746067870.92 total 50.02 last 201.21) - /unlink 10000 (time 1746067874.99 total 54.08 last 241.87) - /unlink 12581 (time 1746067884.99 total 64.08 last 258.07) - /unlink 15008 (time 1746067894.99 total 74.09 last 242.65) - /unlink 17007 (time 1746067904.99 total 84.09 last 199.90) - /unlink 19261 (time 1746067914.99 total 94.09 last 225.30) - /unlink 20000 (time 1746067918.20 total 97.29 last 230.57) - /unlink 22427 (time 1746067928.20 total 107.30 last 242.68) - /unlink 25002 (time 1746067938.20 total 117.30 last 257.39) - /unlink 27321 (time 1746067948.20 total 127.30 last 231.89) - /unlink 29889 (time 1746067958.21 total 137.30 last 256.75) - /unlink 33018 (time 1746067968.21 total 147.31 last 312.66) - /unlink 35748 (time 1746067978.22 total 157.31 last 272.94) - /unlink 38957 (time 1746067988.22 total 167.31 last 320.90) - /unlink 40000 (time 1746067991.11 total 170.21 last 360.33) - /unlink 42927 (time 1746068001.11 total 180.21 last 292.62) - /unlink 46184 (time 1746068011.11 total 190.21 last 325.69) - /unlink 49494 (time 1746068021.12 total 200.21 last 330.93) - /unlink 52588 (time 1746068031.12 total 210.21 last 309.38) - /unlink 55496 (time 1746068041.12 total 220.22 last 290.60) - /unlink 58467 (time 1746068051.12 total 230.22 last 297.05) - /unlink 60000 (time 1746068056.38 total 235.48 last 291.57) - /unlink 63516 (time 1746068066.39 total 245.48 last 351.49) total: 64768 /unlink in 249.76 seconds: 259.32 ops/second Waiting for MDT destroys to complete - open/close 1437 (time 1746068090.69 total 10.00 last 143.69) - open/close 2658 (time 1746068100.69 total 20.00 last 122.08) - open/close 4097 (time 1746068110.69 total 30.01 last 143.84) - open/close 5481 (time 1746068120.70 total 40.01 last 138.30) - open/close 6887 (time 1746068130.71 total 50.02 last 140.52) - open/close 8459 (time 1746068140.71 total 60.03 last 157.10) - open/close 9954 (time 1746068150.72 total 70.03 last 149.45) - open/close 11814 (time 1746068160.72 total 80.03 last 185.96) - open/close 13482 (time 1746068170.72 total 90.03 last 166.75) - open/close 14983 (time 1746068180.73 total 100.04 last 150.04) - open/close 16464 (time 1746068190.74 total 110.05 last 147.94) - open/close 17881 (time 1746068200.74 total 120.05 last 141.62) - open/close 19168 (time 1746068210.75 total 130.06 last 128.65) - open/close 20000 (time 1746068217.31 total 136.63 last 126.69) - open/close 21359 (time 1746068227.32 total 146.63 last 135.89) - open/close 22641 (time 1746068237.32 total 156.63 last 128.19) - open/close 23970 (time 1746068247.33 total 166.64 last 132.75) - open/close 25485 (time 1746068257.33 total 176.64 last 151.48) - open/close 27008 (time 1746068267.33 total 186.64 last 152.30) - open/close 28995 (time 1746068277.33 total 196.64 last 198.66) - open/close 30000 (time 1746068283.76 total 203.07 last 156.30) - open/close 31471 (time 1746068293.76 total 213.07 last 147.08) - open/close 33275 (time 1746068303.77 total 223.08 last 180.31) - open/close 35007 (time 1746068313.77 total 233.08 last 173.13) - open/close 36587 (time 1746068323.78 total 243.09 last 157.91) - open/close 38320 (time 1746068333.78 total 253.09 last 173.25) - open/close 39997 (time 1746068343.78 total 263.09 last 167.68) - open/close 41834 (time 1746068353.79 total 273.10 last 183.61) - open/close 43563 (time 1746068363.79 total 283.10 last 172.90) - open/close 45843 (time 1746068373.79 total 293.10 last 227.95) - open/close 47612 (time 1746068383.79 total 303.10 last 176.86) - open/close 49159 (time 1746068393.80 total 313.11 last 154.62) - open/close 50000 (time 1746068397.88 total 317.19 last 205.92) - open/close 51575 (time 1746068407.88 total 327.19 last 157.44) - open/close 53239 (time 1746068417.89 total 337.20 last 166.35) - open/close 54581 (time 1746068427.89 total 347.20 last 134.18) - open/close 56436 (time 1746068437.89 total 357.20 last 185.47) - open/close 58512 (time 1746068447.89 total 367.20 last 207.58) - open/close 60000 (time 1746068454.54 total 373.85 last 223.89) - open/close 62049 (time 1746068464.54 total 383.85 last 204.85) - open/close 63964 (time 1746068474.55 total 393.86 last 191.37) total: 64768 open/close in 398.30 seconds: 162.61 ops/second - /unlink 2800 (time 1746068490.38 total 10.01 last 279.79) - /unlink 6178 (time 1746068500.38 total 20.01 last 337.70) - /unlink 9230 (time 1746068510.38 total 30.01 last 305.18) - /unlink 10000 (time 1746068512.63 total 32.25 last 343.29) - /unlink 13535 (time 1746068522.63 total 42.26 last 353.46) - /unlink 17511 (time 1746068532.63 total 52.26 last 397.54) - /unlink 20000 (time 1746068541.50 total 61.13 last 280.55) - /unlink 22528 (time 1746068551.50 total 71.13 last 252.78) - /unlink 24883 (time 1746068561.50 total 81.13 last 235.45) - /unlink 27132 (time 1746068571.50 total 91.13 last 224.89) - /unlink 29213 (time 1746068581.51 total 101.13 last 208.05) - /unlink 30000 (time 1746068584.66 total 104.29 last 249.21) - /unlink 32370 (time 1746068594.66 total 114.29 last 236.98) - /unlink 34996 (time 1746068604.67 total 124.30 last 262.50) - /unlink 37504 (time 1746068614.67 total 134.30 last 250.76) - /unlink 40000 (time 1746068623.10 total 142.72 last 296.24) - /unlink 43027 (time 1746068633.10 total 152.73 last 302.49) - /unlink 45284 (time 1746068643.11 total 162.74 last 225.61) - /unlink 48016 (time 1746068653.11 total 172.74 last 273.11) - /unlink 50000 (time 1746068659.28 total 178.91 last 321.46) - /unlink 53135 (time 1746068669.28 total 188.91 last 313.42) - /unlink 56167 (time 1746068679.29 total 198.91 last 303.14) - /unlink 59110 (time 1746068689.29 total 208.92 last 294.28) - /unlink 60000 (time 1746068692.66 total 212.29 last 263.78) - /unlink 63082 (time 1746068702.67 total 222.29 last 308.04) total: 64768 /unlink in 229.67 seconds: 282.01 ops/second Waiting for MDT destroys to complete - open/close 1590 (time 1746068727.76 total 10.01 last 158.91) - open/close 3109 (time 1746068737.76 total 20.02 last 151.76) - open/close 4655 (time 1746068747.77 total 30.02 last 154.56) - open/close 6300 (time 1746068757.77 total 40.02 last 164.45) - open/close 7927 (time 1746068767.77 total 50.02 last 162.64) - open/close 9594 (time 1746068777.78 total 60.03 last 166.65) - open/close 10000 (time 1746068780.03 total 62.29 last 179.80) - open/close 11755 (time 1746068790.04 total 72.29 last 175.46) - open/close 13146 (time 1746068800.04 total 82.29 last 139.08) - open/close 14584 (time 1746068810.05 total 92.30 last 143.68) - open/close 15998 (time 1746068820.05 total 102.30 last 141.35) - open/close 17598 (time 1746068830.06 total 112.31 last 159.90) - open/close 19319 (time 1746068840.06 total 122.31 last 172.09) - open/close 20000 (time 1746068843.96 total 126.21 last 174.62) - open/close 21913 (time 1746068853.96 total 136.21 last 191.28) - open/close 24119 (time 1746068863.96 total 146.21 last 220.57) - open/close 26040 (time 1746068873.96 total 156.21 last 192.04) - open/close 27830 (time 1746068883.97 total 166.22 last 178.94) - open/close 29750 (time 1746068893.97 total 176.22 last 191.89) - open/close 31560 (time 1746068903.97 total 186.22 last 180.99) - open/close 33695 (time 1746068913.97 total 196.22 last 213.47) - open/close 35935 (time 1746068923.98 total 206.23 last 223.89) - open/close 38763 (time 1746068933.98 total 216.23 last 282.77) - open/close 40000 (time 1746068938.59 total 220.84 last 268.13) - open/close 42607 (time 1746068948.60 total 230.85 last 260.62) - open/close 45425 (time 1746068958.60 total 240.85 last 281.73) - open/close 47824 (time 1746068968.60 total 250.85 last 239.88) - open/close 49809 (time 1746068978.61 total 260.86 last 198.37) - open/close 52178 (time 1746068988.61 total 270.86 last 236.85) - open/close 54521 (time 1746068998.61 total 280.86 last 234.15) - open/close 56538 (time 1746069008.62 total 290.87 last 201.57) - open/close 58321 (time 1746069018.62 total 300.87 last 178.23) - open/close 59687 (time 1746069028.62 total 310.87 last 136.60) - open/close 61722 (time 1746069038.63 total 320.88 last 203.48) - open/close 64066 (time 1746069048.63 total 330.88 last 234.40) total: 64768 open/close in 334.26 seconds: 193.76 ops/second - /unlink 2540 (time 1746069063.37 total 10.00 last 253.94) - /unlink 4888 (time 1746069073.37 total 20.00 last 234.75) - /unlink 7498 (time 1746069083.37 total 30.00 last 260.99) - /unlink 10000 (time 1746069092.57 total 39.20 last 272.19) - /unlink 12454 (time 1746069102.57 total 49.20 last 245.39) - /unlink 14804 (time 1746069112.57 total 59.20 last 234.95) - /unlink 17170 (time 1746069122.57 total 69.20 last 236.57) - /unlink 19597 (time 1746069132.57 total 79.20 last 242.67) - /unlink 22071 (time 1746069142.58 total 89.21 last 247.30) - /unlink 24426 (time 1746069152.58 total 99.21 last 235.50) - /unlink 26958 (time 1746069162.58 total 109.21 last 253.13) - /unlink 29270 (time 1746069172.58 total 119.21 last 231.16) - /unlink 30000 (time 1746069175.01 total 121.64 last 300.90) - /unlink 33492 (time 1746069185.01 total 131.64 last 349.19) - /unlink 36772 (time 1746069195.01 total 141.64 last 328.00) - /unlink 40000 (time 1746069204.75 total 151.38 last 331.17) - /unlink 42324 (time 1746069214.76 total 161.39 last 232.30) - /unlink 44447 (time 1746069224.76 total 171.39 last 212.23) - /unlink 46762 (time 1746069234.76 total 181.39 last 231.49) - /unlink 49085 (time 1746069244.76 total 191.39 last 232.26) - /unlink 50000 (time 1746069248.53 total 195.17 last 242.62) - /unlink 51998 (time 1746069258.54 total 205.17 last 199.76) - /unlink 54336 (time 1746069268.54 total 215.17 last 233.63) - /unlink 56522 (time 1746069278.55 total 225.18 last 218.46) - /unlink 58981 (time 1746069288.56 total 235.19 last 245.77) - /unlink 60000 (time 1746069292.56 total 239.19 last 254.48) - /unlink 62286 (time 1746069302.56 total 249.19 last 228.56) total: 64768 /unlink in 259.09 seconds: 249.98 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg428-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg428-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg428-server stop mds service on oleg428-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg428-server unloading modules via unload_modules_local on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-client: ssh exited with exit code 2 pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 PASS 106 (3370s) Stopping clients: oleg428-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg428-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg428-client: oleg428-server: ssh exited with exit code 2 oleg428-server: oleg428-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg428-server' oleg428-server: oleg428-server.virtnet: executing load_modules_local oleg428-server: Loading modules from /home/green/git/lustre-release/lustre oleg428-server: detected 4 online CPUs by sysfs oleg428-server: MODOPTS_LIBCFS= oleg428-server: Force libcfs to create 2 CPU partitions oleg428-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg428-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3509 sec ========== 23:16:23 (1746069383) === conf-sanity: start cleanup 23:16:23 (1746069383) === === conf-sanity: finish cleanup 23:16:25 (1746069385) ===