-----============= acceptance-small: conf-sanity ============----- Mon Jun 2 08:28:07 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg417-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg417-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg417-server oleg417-server: oleg417-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/intel_rapl/holders': No such file or directory loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg417-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg417-server: oleg417-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg417-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:29:35 (1748867375) Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f) oleg417-server: oleg417-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: libkmod: kmod_module_get_refcnt: could not read integer from '/sys/module/pcc_cpufreq/refcnt': 'No such device' oleg417-server: libkmod: kmod_module_get_holders: could not open '/sys/module/pcc_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg417-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg417-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg417-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg417-server: oleg417-server.virtnet: executing set_default_debug -1 all pdsh@oleg417-client: oleg417-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg417-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg417-client.virtnet: -o user_xattr,flock 192.168.204.117@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 86842 329 86513 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 32582 326 32256 2% /mnt/lustre[OST:0] filesystem_summary: 32585 329 32256 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1318 (time 1748867426.95 total 10.00 last 131.75) - open/close 2588 (time 1748867436.95 total 20.00 last 126.98) - open/close 3926 (time 1748867446.96 total 30.02 last 133.65) - open/close 5257 (time 1748867456.97 total 40.02 last 133.05) - open/close 6567 (time 1748867466.97 total 50.03 last 130.93) - open/close 8094 (time 1748867476.98 total 60.03 last 152.62) - open/close 9387 (time 1748867486.98 total 70.03 last 129.28) - open/close 10000 (time 1748867491.95 total 75.00 last 123.37) - open/close 11128 (time 1748867501.96 total 85.01 last 112.71) - open/close 12370 (time 1748867511.96 total 95.01 last 124.18) - open/close 13676 (time 1748867521.96 total 105.01 last 130.56) - open/close 14956 (time 1748867531.96 total 115.02 last 127.96) - open/close 16274 (time 1748867541.97 total 125.02 last 131.74) - open/close 17581 (time 1748867551.97 total 135.02 last 130.65) - open/close 18919 (time 1748867561.98 total 145.03 last 133.67) - open/close 20000 (time 1748867568.72 total 151.77 last 160.49) - open/close 21829 (time 1748867578.72 total 161.77 last 182.84) - open/close 23519 (time 1748867588.72 total 171.77 last 168.98) - open/close 25069 (time 1748867598.72 total 181.77 last 154.98) - open/close 26542 (time 1748867608.72 total 191.78 last 147.30) - open/close 27944 (time 1748867618.73 total 201.78 last 140.16) - open/close 29398 (time 1748867628.73 total 211.78 last 145.33) - open/close 30000 (time 1748867632.78 total 215.83 last 148.66) - open/close 31681 (time 1748867642.79 total 225.84 last 167.93) - open/close 33373 (time 1748867652.80 total 235.85 last 169.04) - open/close 34857 (time 1748867662.80 total 245.85 last 148.39) - open/close 36440 (time 1748867672.80 total 255.85 last 158.27) - open/close 38089 (time 1748867682.81 total 265.86 last 164.80) - open/close 39638 (time 1748867692.81 total 275.86 last 154.87) - open/close 40000 (time 1748867695.52 total 278.57 last 133.85) - open/close 41408 (time 1748867705.52 total 288.57 last 140.76) - open/close 42819 (time 1748867715.53 total 298.58 last 140.95) - open/close 44165 (time 1748867725.53 total 308.58 last 134.59) - open/close 45284 (time 1748867735.53 total 318.58 last 111.88) - open/close 46444 (time 1748867745.53 total 328.59 last 115.96) - open/close 47668 (time 1748867755.54 total 338.60 last 122.29) - open/close 48754 (time 1748867765.55 total 348.60 last 108.56) - open/close 49785 (time 1748867775.57 total 358.62 last 102.88) - open/close 50000 (time 1748867777.57 total 360.62 last 107.42) - open/close 51141 (time 1748867787.57 total 370.62 last 114.07) - open/close 52313 (time 1748867797.57 total 380.63 last 117.18) - open/close 53625 (time 1748867807.58 total 390.63 last 131.12) - open/close 54941 (time 1748867817.58 total 400.63 last 131.59) - open/close 56364 (time 1748867827.58 total 410.64 last 142.25) - open/close 57837 (time 1748867837.60 total 420.65 last 147.13) - open/close 59126 (time 1748867847.60 total 430.65 last 128.88) - open/close 60000 (time 1748867853.43 total 436.48 last 149.94) - open/close 61709 (time 1748867863.43 total 446.48 last 170.81) - open/close 63271 (time 1748867873.43 total 456.49 last 156.15) total: 64768 open/close in 466.19 seconds: 138.93 ops/second - /unlink 2795 (time 1748867894.58 total 10.00 last 279.44) - /unlink 5906 (time 1748867904.58 total 20.00 last 311.07) - /unlink 9115 (time 1748867914.58 total 30.00 last 320.86) - /unlink 10000 (time 1748867917.34 total 32.77 last 320.41) - /unlink 13184 (time 1748867927.34 total 42.77 last 318.34) - /unlink 15461 (time 1748867937.35 total 52.77 last 227.70) - /unlink 18256 (time 1748867947.35 total 62.77 last 279.43) - /unlink 20000 (time 1748867952.87 total 68.29 last 315.85) - /unlink 23238 (time 1748867962.87 total 78.29 last 323.74) - /unlink 26230 (time 1748867972.87 total 88.30 last 299.11) - /unlink 28994 (time 1748867982.87 total 98.30 last 276.39) - /unlink 30000 (time 1748867986.25 total 101.68 last 297.70) - /unlink 34096 (time 1748867996.26 total 111.68 last 409.54) - /unlink 38149 (time 1748868006.26 total 121.68 last 405.22) - /unlink 40000 (time 1748868011.05 total 126.47 last 386.50) - /unlink 44631 (time 1748868021.05 total 136.47 last 463.09) - /unlink 50000 (time 1748868030.83 total 146.26 last 548.59) - /unlink 54858 (time 1748868040.83 total 156.26 last 485.75) - /unlink 59330 (time 1748868050.84 total 166.26 last 447.06) - /unlink 64064 (time 1748868060.84 total 176.26 last 473.39) total: 64768 /unlink in 177.91 seconds: 364.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 2132 (time 1748868089.42 total 10.01 last 213.09) - open/close 4013 (time 1748868099.43 total 20.01 last 188.04) - open/close 6439 (time 1748868109.43 total 30.01 last 242.55) - open/close 9206 (time 1748868119.43 total 40.01 last 276.69) - open/close 10000 (time 1748868121.90 total 42.48 last 321.11) - open/close 12641 (time 1748868131.90 total 52.48 last 264.07) - open/close 15243 (time 1748868141.91 total 62.49 last 260.11) - open/close 18016 (time 1748868151.91 total 72.49 last 277.23) - open/close 20000 (time 1748868157.03 total 77.61 last 387.17) - open/close 23583 (time 1748868167.03 total 87.62 last 358.20) - open/close 26253 (time 1748868177.04 total 97.62 last 266.95) - open/close 29097 (time 1748868187.04 total 107.62 last 284.36) - open/close 30000 (time 1748868190.77 total 111.35 last 242.16) - open/close 32980 (time 1748868200.77 total 121.35 last 297.96) - open/close 36315 (time 1748868210.77 total 131.35 last 333.42) - open/close 39309 (time 1748868220.77 total 141.36 last 299.34) - open/close 40000 (time 1748868222.89 total 143.48 last 325.68) - open/close 43309 (time 1748868232.90 total 153.48 last 330.84) - open/close 46439 (time 1748868242.90 total 163.48 last 312.97) - open/close 50000 (time 1748868251.26 total 171.84 last 425.79) - open/close 54482 (time 1748868261.26 total 181.85 last 448.04) - open/close 58038 (time 1748868271.26 total 191.85 last 355.59) - open/close 60000 (time 1748868275.16 total 195.74 last 503.62) - open/close 64711 (time 1748868285.16 total 205.74 last 471.03) total: 64768 open/close in 205.84 seconds: 314.65 ops/second - /unlink 8554 (time 1748868296.12 total 10.00 last 855.38) - /unlink 15958 (time 1748868306.12 total 20.00 last 740.32) - /unlink 20000 (time 1748868312.24 total 26.13 last 659.72) - /unlink 27204 (time 1748868322.24 total 36.13 last 720.38) - /unlink 30000 (time 1748868326.23 total 40.11 last 701.59) - /unlink 36496 (time 1748868336.23 total 50.11 last 649.57) - /unlink 40000 (time 1748868340.27 total 54.16 last 866.43) - /unlink 48706 (time 1748868350.27 total 64.16 last 870.56) - /unlink 57908 (time 1748868360.28 total 74.16 last 920.11) - /unlink 60000 (time 1748868362.58 total 76.47 last 905.95) total: 64768 /unlink in 81.80 seconds: 791.74 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 3093 (time 1748868399.08 total 10.00 last 309.25) - open/close 6934 (time 1748868409.08 total 20.00 last 384.01) - open/close 10000 (time 1748868415.47 total 26.38 last 480.51) - open/close 14454 (time 1748868425.47 total 36.39 last 445.32) - open/close 19292 (time 1748868435.47 total 46.39 last 483.75) - open/close 24005 (time 1748868445.47 total 56.39 last 471.29) - open/close 28594 (time 1748868455.47 total 66.39 last 458.82) - open/close 30000 (time 1748868458.53 total 69.45 last 459.82) - open/close 34791 (time 1748868468.53 total 79.45 last 479.08) - open/close 39263 (time 1748868478.53 total 89.45 last 447.14) - open/close 43693 (time 1748868488.53 total 99.45 last 442.95) - open/close 47323 (time 1748868498.53 total 109.45 last 362.96) - open/close 50000 (time 1748868505.63 total 116.55 last 376.98) - open/close 54454 (time 1748868515.63 total 126.55 last 445.35) - open/close 58808 (time 1748868525.64 total 136.55 last 435.32) - open/close 60000 (time 1748868528.57 total 139.49 last 405.97) - open/close 64139 (time 1748868538.57 total 149.49 last 413.83) total: 64768 open/close in 151.30 seconds: 428.06 ops/second - /unlink 7216 (time 1748868551.35 total 10.00 last 721.56) - /unlink 10000 (time 1748868554.91 total 13.56 last 782.02) - /unlink 17780 (time 1748868564.92 total 23.56 last 777.92) - /unlink 20000 (time 1748868567.62 total 26.27 last 820.42) - /unlink 28138 (time 1748868577.62 total 36.27 last 813.75) - /unlink 30000 (time 1748868579.86 total 38.50 last 832.58) - /unlink 38668 (time 1748868589.86 total 48.51 last 866.75) - /unlink 46895 (time 1748868599.86 total 58.51 last 822.61) - /unlink 50000 (time 1748868603.75 total 62.40 last 798.12) - /unlink 58453 (time 1748868613.75 total 72.40 last 845.30) total: 64768 /unlink in 79.91 seconds: 810.55 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4820 (time 1748868654.73 total 10.00 last 481.95) - open/close 9194 (time 1748868664.74 total 20.00 last 437.31) - open/close 13484 (time 1748868674.78 total 30.05 last 427.10) - open/close 17788 (time 1748868684.78 total 40.05 last 430.39) - open/close 20000 (time 1748868689.66 total 44.93 last 453.41) - open/close 24233 (time 1748868699.66 total 54.93 last 423.29) - open/close 28639 (time 1748868709.66 total 64.93 last 440.59) - open/close 30000 (time 1748868712.60 total 67.87 last 462.79) - open/close 34820 (time 1748868722.60 total 77.87 last 481.94) - open/close 39760 (time 1748868732.60 total 87.87 last 493.94) - open/close 44796 (time 1748868742.60 total 97.87 last 503.59) - open/close 49774 (time 1748868752.60 total 107.87 last 497.79) - open/close 54685 (time 1748868762.61 total 117.87 last 491.05) - open/close 59435 (time 1748868772.61 total 127.87 last 474.92) - open/close 64247 (time 1748868782.61 total 137.87 last 481.15) total: 64768 open/close in 139.03 seconds: 465.86 ops/second - /unlink 8298 (time 1748868794.62 total 10.00 last 829.75) - /unlink 16780 (time 1748868804.62 total 20.00 last 848.19) - /unlink 20000 (time 1748868808.45 total 23.83 last 840.81) - /unlink 28498 (time 1748868818.45 total 33.83 last 849.78) - /unlink 36444 (time 1748868828.45 total 43.83 last 794.53) - /unlink 40000 (time 1748868832.64 total 48.02 last 849.74) - /unlink 48524 (time 1748868842.64 total 58.02 last 852.40) - /unlink 56616 (time 1748868852.64 total 68.02 last 809.19) - /unlink 60000 (time 1748868856.77 total 72.15 last 819.39) total: 64768 /unlink in 77.71 seconds: 833.44 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete - open/close 4828 (time 1748868895.32 total 10.00 last 482.73) - open/close 9562 (time 1748868905.32 total 20.00 last 473.37) - open/close 14173 (time 1748868915.32 total 30.00 last 461.10) - open/close 18885 (time 1748868925.32 total 40.00 last 471.15) - open/close 24277 (time 1748868935.32 total 50.01 last 539.08) - open/close 29292 (time 1748868945.32 total 60.01 last 501.47) - open/close 34151 (time 1748868955.32 total 70.01 last 485.81) - open/close 39155 (time 1748868965.32 total 80.01 last 500.32) - open/close 44225 (time 1748868975.33 total 90.01 last 506.86) - open/close 49094 (time 1748868985.33 total 100.01 last 486.90) - open/close 53699 (time 1748868995.33 total 110.01 last 460.49) - open/close 57663 (time 1748869005.33 total 120.01 last 396.35) - open/close 60000 (time 1748869011.48 total 126.16 last 380.20) total: 64768 open/close in 135.83 seconds: 476.82 ops/second - /unlink 8553 (time 1748869031.95 total 10.00 last 855.22) - /unlink 17413 (time 1748869041.95 total 20.00 last 885.94) - /unlink 20000 (time 1748869044.71 total 22.76 last 937.40) - /unlink 28876 (time 1748869054.71 total 32.76 last 887.54) - /unlink 37865 (time 1748869064.71 total 42.76 last 898.86) - /unlink 40000 (time 1748869067.09 total 45.15 last 895.45) - /unlink 47995 (time 1748869077.10 total 55.15 last 799.42) - /unlink 50000 (time 1748869079.37 total 57.43 last 879.93) - /unlink 59044 (time 1748869089.37 total 67.43 last 904.31) total: 64768 /unlink in 73.70 seconds: 878.80 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg417-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg417-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg417-server stop mds service on oleg417-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg417-server unloading modules via unload_modules_local on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-client: ssh exited with exit code 2 pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 PASS 106 (1765s) Stopping clients: oleg417-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg417-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg417-client: oleg417-server: ssh exited with exit code 2 oleg417-server: oleg417-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg417-server' oleg417-server: oleg417-server.virtnet: executing load_modules_local oleg417-server: Loading modules from /home/green/git/lustre-release/lustre oleg417-server: detected 4 online CPUs by sysfs oleg417-server: MODOPTS_LIBCFS= oleg417-server: Force libcfs to create 2 CPU partitions oleg417-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg417-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 1869 sec ========== 08:59:17 (1748869157) === conf-sanity: start cleanup 08:59:17 (1748869157) === === conf-sanity: finish cleanup 08:59:18 (1748869158) ===