-----============= acceptance-small: conf-sanity ============----- Mon Jun 2 08:28:18 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg260-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg260-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg260-server oleg260-server: oleg260-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg260-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg260-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg260-server: oleg260-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg260-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:30:03 (1748867403) Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) oleg260-server: oleg260-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg260-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg260-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg260-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg260-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg260-server: oleg260-server.virtnet: executing set_default_debug -1 all pdsh@oleg260-client: oleg260-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg260-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg260-client.virtnet: -o user_xattr,flock 192.168.202.160@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 1686 (time 1748867463.46 total 10.01 last 168.51) - open/close 3527 (time 1748867473.47 total 20.01 last 184.02) - open/close 5109 (time 1748867483.47 total 30.01 last 158.18) - open/close 6712 (time 1748867493.47 total 40.01 last 160.24) - open/close 8449 (time 1748867503.47 total 50.02 last 173.66) - open/close 10000 (time 1748867511.77 total 58.31 last 186.99) - open/close 11900 (time 1748867521.77 total 68.32 last 189.89) - open/close 13713 (time 1748867531.78 total 78.32 last 181.25) - open/close 15702 (time 1748867541.78 total 88.32 last 198.87) - open/close 17533 (time 1748867551.78 total 98.32 last 183.09) - open/close 19352 (time 1748867561.78 total 108.32 last 181.90) - open/close 20000 (time 1748867564.92 total 111.46 last 206.25) - open/close 22195 (time 1748867574.92 total 121.47 last 219.40) - open/close 23944 (time 1748867584.92 total 131.47 last 174.90) - open/close 26093 (time 1748867594.93 total 141.48 last 214.72) - open/close 27969 (time 1748867604.94 total 151.48 last 187.50) - open/close 29702 (time 1748867614.94 total 161.48 last 173.28) - open/close 31312 (time 1748867624.94 total 171.48 last 160.98) - open/close 32868 (time 1748867634.94 total 181.49 last 155.56) - open/close 34545 (time 1748867644.94 total 191.49 last 167.67) - open/close 36242 (time 1748867654.95 total 201.49 last 169.62) - open/close 38147 (time 1748867664.95 total 211.50 last 190.42) - open/close 39617 (time 1748867674.96 total 221.50 last 146.96) - open/close 40000 (time 1748867677.38 total 223.93 last 157.84) - open/close 41693 (time 1748867687.38 total 233.93 last 169.25) - open/close 43255 (time 1748867697.39 total 243.93 last 156.12) - open/close 44869 (time 1748867707.39 total 253.94 last 161.32) - open/close 46764 (time 1748867717.40 total 263.94 last 189.48) - open/close 48679 (time 1748867727.40 total 273.94 last 191.45) - open/close 50000 (time 1748867735.71 total 282.26 last 158.87) - open/close 51595 (time 1748867745.72 total 292.26 last 159.46) - open/close 53562 (time 1748867755.72 total 302.26 last 196.70) - open/close 55238 (time 1748867765.72 total 312.26 last 167.60) - open/close 56979 (time 1748867775.72 total 322.26 last 174.06) - open/close 58921 (time 1748867785.72 total 332.27 last 194.09) - open/close 60000 (time 1748867791.46 total 338.01 last 188.05) - open/close 61983 (time 1748867801.46 total 348.01 last 198.30) - open/close 64152 (time 1748867811.47 total 358.01 last 216.80) total: 64768 open/close in 361.10 seconds: 179.36 ops/second - /unlink 3097 (time 1748867825.81 total 10.00 last 309.69) - /unlink 6146 (time 1748867835.82 total 20.00 last 304.83) - /unlink 9208 (time 1748867845.82 total 30.00 last 306.14) - /unlink 13038 (time 1748867855.82 total 40.00 last 382.98) - /unlink 17328 (time 1748867865.82 total 50.01 last 428.98) - /unlink 20000 (time 1748867869.62 total 53.81 last 703.25) - /unlink 26145 (time 1748867879.62 total 63.81 last 614.47) - /unlink 30000 (time 1748867888.17 total 72.36 last 450.58) - /unlink 35209 (time 1748867898.18 total 82.36 last 520.85) - /unlink 40000 (time 1748867906.55 total 90.74 last 572.10) - /unlink 47032 (time 1748867916.55 total 100.74 last 703.14) - /unlink 50000 (time 1748867921.50 total 105.69 last 599.83) - /unlink 55178 (time 1748867931.50 total 115.69 last 517.79) - /unlink 60000 (time 1748867941.26 total 125.45 last 494.01) - /unlink 64522 (time 1748867951.26 total 135.45 last 452.19) total: 64768 /unlink in 136.03 seconds: 476.12 ops/second Waiting for MDT destroys to complete - open/close 2551 (time 1748868000.41 total 10.00 last 255.09) - open/close 4689 (time 1748868010.42 total 20.00 last 213.72) - open/close 6624 (time 1748868020.42 total 30.01 last 193.39) - open/close 8629 (time 1748868030.43 total 40.01 last 200.44) - open/close 10000 (time 1748868037.55 total 47.13 last 192.56) - open/close 12576 (time 1748868047.55 total 57.13 last 257.57) - open/close 15324 (time 1748868057.55 total 67.14 last 274.72) - open/close 17830 (time 1748868067.55 total 77.14 last 250.48) - open/close 20000 (time 1748868074.50 total 84.09 last 312.37) - open/close 22896 (time 1748868084.51 total 94.09 last 289.47) - open/close 26279 (time 1748868094.51 total 104.09 last 338.24) - open/close 29552 (time 1748868104.51 total 114.10 last 327.13) - open/close 32398 (time 1748868114.51 total 124.10 last 284.55) - open/close 35580 (time 1748868124.52 total 134.10 last 318.17) - open/close 38555 (time 1748868134.52 total 144.10 last 297.41) - open/close 40000 (time 1748868139.38 total 148.96 last 297.36) - open/close 43577 (time 1748868149.38 total 158.97 last 357.58) - open/close 46790 (time 1748868159.38 total 168.97 last 321.26) - open/close 49518 (time 1748868169.39 total 178.97 last 272.68) - open/close 52655 (time 1748868179.39 total 188.97 last 313.68) - open/close 55414 (time 1748868189.39 total 198.98 last 275.79) - open/close 58143 (time 1748868199.39 total 208.98 last 272.86) - open/close 60000 (time 1748868205.96 total 215.55 last 282.73) - open/close 62446 (time 1748868215.96 total 225.55 last 244.57) total: 64768 open/close in 232.27 seconds: 278.85 ops/second - /unlink 6019 (time 1748868233.60 total 10.00 last 601.87) - /unlink 10000 (time 1748868240.02 total 16.42 last 620.37) - /unlink 17043 (time 1748868250.02 total 26.42 last 704.27) - /unlink 20000 (time 1748868254.22 total 30.62 last 704.00) - /unlink 25150 (time 1748868264.22 total 40.62 last 514.94) - /unlink 30000 (time 1748868271.01 total 47.41 last 714.29) - /unlink 38331 (time 1748868281.01 total 57.41 last 833.07) - /unlink 45843 (time 1748868291.01 total 67.41 last 751.15) - /unlink 50000 (time 1748868297.98 total 74.38 last 596.57) - /unlink 57499 (time 1748868307.98 total 84.38 last 749.88) - /unlink 60000 (time 1748868310.97 total 87.38 last 834.76) total: 64768 /unlink in 92.78 seconds: 698.06 ops/second Waiting for MDT destroys to complete - open/close 4256 (time 1748868375.89 total 10.00 last 425.56) - open/close 8556 (time 1748868385.89 total 20.00 last 429.93) - open/close 10000 (time 1748868388.53 total 22.65 last 545.75) - open/close 14071 (time 1748868398.54 total 32.65 last 407.02) - open/close 17941 (time 1748868408.54 total 42.66 last 386.80) - open/close 20000 (time 1748868414.57 total 48.69 last 341.22) - open/close 24352 (time 1748868424.57 total 58.69 last 435.19) - open/close 28587 (time 1748868434.58 total 68.69 last 423.44) - open/close 30000 (time 1748868438.59 total 72.70 last 352.21) - open/close 33688 (time 1748868448.59 total 82.70 last 368.75) - open/close 38174 (time 1748868458.59 total 92.71 last 448.52) - open/close 40000 (time 1748868463.52 total 97.63 last 370.60) - open/close 43843 (time 1748868473.52 total 107.63 last 384.25) - open/close 48197 (time 1748868483.52 total 117.63 last 435.39) - open/close 50000 (time 1748868487.96 total 122.07 last 406.09) - open/close 52850 (time 1748868497.96 total 132.08 last 284.97) - open/close 55879 (time 1748868507.97 total 142.08 last 302.73) - open/close 58233 (time 1748868517.97 total 152.08 last 235.39) - open/close 60000 (time 1748868523.20 total 157.32 last 337.51) - open/close 63102 (time 1748868533.21 total 167.32 last 310.10) total: 64768 open/close in 174.50 seconds: 371.16 ops/second - /unlink 4958 (time 1748868551.32 total 10.00 last 495.73) - /unlink 8719 (time 1748868561.32 total 20.00 last 376.10) - /unlink 14570 (time 1748868571.32 total 30.00 last 585.00) - /unlink 19726 (time 1748868581.32 total 40.00 last 515.57) - /unlink 26054 (time 1748868591.32 total 50.00 last 632.78) - /unlink 30000 (time 1748868598.10 total 56.79 last 581.44) - /unlink 38215 (time 1748868608.11 total 66.79 last 821.47) - /unlink 40000 (time 1748868610.27 total 68.95 last 826.13) - /unlink 48444 (time 1748868620.27 total 78.95 last 844.29) - /unlink 57007 (time 1748868630.27 total 88.95 last 856.28) - /unlink 60000 (time 1748868633.41 total 92.10 last 952.14) total: 64768 /unlink in 97.65 seconds: 663.30 ops/second Waiting for MDT destroys to complete - open/close 3222 (time 1748868706.84 total 10.00 last 322.17) - open/close 6829 (time 1748868716.84 total 20.00 last 360.68) - open/close 10000 (time 1748868725.38 total 28.54 last 371.57) - open/close 13894 (time 1748868735.38 total 38.54 last 389.32) - open/close 18121 (time 1748868745.38 total 48.54 last 422.60) - open/close 20000 (time 1748868749.16 total 52.32 last 497.72) - open/close 24698 (time 1748868759.16 total 62.32 last 469.73) - open/close 29417 (time 1748868769.16 total 72.32 last 471.90) - open/close 34127 (time 1748868779.16 total 82.32 last 470.99) - open/close 38781 (time 1748868789.16 total 92.32 last 465.39) - open/close 40000 (time 1748868791.88 total 95.04 last 447.71) - open/close 43457 (time 1748868801.88 total 105.04 last 345.61) - open/close 47377 (time 1748868811.88 total 115.04 last 391.99) - open/close 50000 (time 1748868819.15 total 122.31 last 360.83) - open/close 54531 (time 1748868829.15 total 132.31 last 453.05) - open/close 59140 (time 1748868839.15 total 142.31 last 460.87) - open/close 60000 (time 1748868841.40 total 144.56 last 383.50) - open/close 64431 (time 1748868851.40 total 154.56 last 442.96) total: 64768 open/close in 155.31 seconds: 417.03 ops/second - /unlink 8563 (time 1748868863.10 total 10.00 last 856.22) - /unlink 16699 (time 1748868873.10 total 20.00 last 813.54) - /unlink 20000 (time 1748868876.98 total 23.89 last 849.49) - /unlink 27615 (time 1748868886.98 total 33.89 last 761.43) - /unlink 30000 (time 1748868890.01 total 36.91 last 788.78) - /unlink 37616 (time 1748868900.01 total 46.91 last 761.59) - /unlink 40000 (time 1748868903.47 total 50.37 last 689.03) - /unlink 46186 (time 1748868913.47 total 60.37 last 618.46) - /unlink 50000 (time 1748868920.63 total 67.54 last 532.39) - /unlink 56173 (time 1748868930.63 total 77.54 last 617.30) - /unlink 60000 (time 1748868937.70 total 84.61 last 541.41) total: 64768 /unlink in 90.71 seconds: 713.98 ops/second Waiting for MDT destroys to complete - open/close 5040 (time 1748869021.61 total 10.00 last 503.89) - open/close 10000 (time 1748869030.27 total 18.66 last 572.67) - open/close 15318 (time 1748869040.28 total 28.66 last 531.74) - open/close 20000 (time 1748869049.24 total 37.63 last 522.46) - open/close 25166 (time 1748869059.24 total 47.63 last 516.51) - open/close 30000 (time 1748869068.31 total 56.70 last 533.00) - open/close 35133 (time 1748869078.31 total 66.70 last 513.29) - open/close 40000 (time 1748869088.30 total 76.68 last 487.31) - open/close 43753 (time 1748869098.30 total 86.69 last 375.28) - open/close 48161 (time 1748869108.30 total 96.69 last 440.73) - open/close 50000 (time 1748869111.99 total 100.38 last 497.54) - open/close 54809 (time 1748869121.99 total 110.38 last 480.88) - open/close 59815 (time 1748869132.00 total 120.39 last 500.50) total: 64768 open/close in 130.10 seconds: 497.85 ops/second - /unlink 9840 (time 1748869152.52 total 10.00 last 983.91) - /unlink 20000 (time 1748869162.41 total 19.88 last 1027.95) - /unlink 29231 (time 1748869172.41 total 29.88 last 923.09) - /unlink 38199 (time 1748869182.41 total 39.89 last 896.76) - /unlink 47181 (time 1748869192.41 total 49.89 last 898.14) - /unlink 50000 (time 1748869195.14 total 52.62 last 1031.24) - /unlink 60000 (time 1748869204.87 total 62.35 last 1027.51) total: 64768 /unlink in 67.08 seconds: 965.52 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg260-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg260-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg260-server stop mds service on oleg260-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg260-server unloading modules via unload_modules_local on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-client: ssh exited with exit code 2 pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 PASS 106 (1920s) Stopping clients: oleg260-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg260-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg260-client: oleg260-server: ssh exited with exit code 2 oleg260-server: oleg260-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg260-server' oleg260-server: oleg260-server.virtnet: executing load_modules_local oleg260-server: Loading modules from /home/green/git/lustre-release/lustre oleg260-server: detected 4 online CPUs by sysfs oleg260-server: MODOPTS_LIBCFS= oleg260-server: Force libcfs to create 2 CPU partitions oleg260-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg260-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 2046 sec ========== 09:02:25 (1748869345) === conf-sanity: start cleanup 09:02:26 (1748869346) === === conf-sanity: finish cleanup 09:02:26 (1748869346) ===