-----============= acceptance-small: conf-sanity ============----- Wed May 21 13:19:57 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg407-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg407-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg407-server oleg407-server: oleg407-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg407-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg407-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg407-server: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg407-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg407-server: oleg407-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg407-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:21:22 (1747848082) Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f) oleg407-server: oleg407-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg407-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg407-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg407-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg407-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg407-server: oleg407-server.virtnet: executing set_default_debug -1 all pdsh@oleg407-client: oleg407-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg407-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg407-client.virtnet: -o user_xattr,flock 192.168.204.107@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 2025 (time 1747848130.14 total 10.00 last 202.42) - open/close 3994 (time 1747848140.14 total 20.00 last 196.90) - open/close 6209 (time 1747848150.14 total 30.00 last 221.49) - open/close 8459 (time 1747848160.14 total 40.00 last 224.99) - open/close 10000 (time 1747848167.72 total 47.59 last 203.26) - open/close 12170 (time 1747848177.72 total 57.59 last 216.97) - open/close 14841 (time 1747848187.72 total 67.59 last 267.07) - open/close 17688 (time 1747848197.72 total 77.59 last 284.63) - open/close 20000 (time 1747848207.33 total 87.20 last 240.70) - open/close 22196 (time 1747848217.33 total 97.20 last 219.58) - open/close 24489 (time 1747848227.33 total 107.20 last 229.27) - open/close 26998 (time 1747848237.33 total 117.20 last 250.83) - open/close 29746 (time 1747848247.34 total 127.20 last 274.72) - open/close 32467 (time 1747848257.34 total 137.21 last 272.03) - open/close 34628 (time 1747848267.34 total 147.21 last 216.06) - open/close 36286 (time 1747848277.34 total 157.21 last 165.76) - open/close 37882 (time 1747848287.35 total 167.21 last 159.56) - open/close 39477 (time 1747848297.35 total 177.22 last 159.44) - open/close 40000 (time 1747848300.18 total 180.05 last 184.74) - open/close 41983 (time 1747848310.19 total 190.05 last 198.21) - open/close 43727 (time 1747848320.19 total 200.06 last 174.34) - open/close 45597 (time 1747848330.19 total 210.06 last 186.98) - open/close 48154 (time 1747848340.19 total 220.06 last 255.66) - open/close 50000 (time 1747848347.63 total 227.49 last 248.33) - open/close 52454 (time 1747848357.63 total 237.49 last 245.37) - open/close 54743 (time 1747848367.63 total 247.49 last 228.90) - open/close 57480 (time 1747848377.63 total 257.50 last 273.62) - open/close 60000 (time 1747848386.76 total 266.63 last 275.90) - open/close 62897 (time 1747848396.76 total 276.63 last 289.67) total: 64768 open/close in 283.63 seconds: 228.35 ops/second - /unlink 4685 (time 1747848414.78 total 10.00 last 468.48) - /unlink 8789 (time 1747848424.78 total 20.00 last 410.40) - /unlink 10000 (time 1747848427.77 total 22.99 last 404.83) - /unlink 14166 (time 1747848437.77 total 32.99 last 416.58) - /unlink 20000 (time 1747848447.08 total 42.30 last 626.93) - /unlink 25097 (time 1747848457.08 total 52.30 last 509.67) - /unlink 29532 (time 1747848467.08 total 62.30 last 443.50) - /unlink 35295 (time 1747848477.08 total 72.30 last 576.25) - /unlink 40000 (time 1747848485.13 total 80.35 last 584.64) - /unlink 46698 (time 1747848495.13 total 90.35 last 669.72) - /unlink 50000 (time 1747848499.64 total 94.86 last 732.05) - /unlink 56866 (time 1747848509.64 total 104.86 last 686.59) - /unlink 60000 (time 1747848514.11 total 109.33 last 701.03) total: 64768 /unlink in 116.50 seconds: 555.95 ops/second Waiting for MDT destroys to complete - open/close 3957 (time 1747848562.40 total 10.00 last 395.64) - open/close 7181 (time 1747848572.40 total 20.00 last 322.37) - open/close 10000 (time 1747848579.55 total 27.15 last 394.33) - open/close 14418 (time 1747848589.55 total 37.15 last 441.80) - open/close 18480 (time 1747848599.55 total 47.15 last 406.11) - open/close 20000 (time 1747848603.46 total 51.07 last 388.39) - open/close 23465 (time 1747848613.46 total 61.07 last 346.40) - open/close 26294 (time 1747848623.47 total 71.07 last 282.90) - open/close 30000 (time 1747848633.31 total 80.92 last 376.26) - open/close 33940 (time 1747848643.32 total 90.92 last 393.93) - open/close 37483 (time 1747848653.32 total 100.92 last 354.22) - open/close 40000 (time 1747848659.98 total 107.59 last 377.66) - open/close 44373 (time 1747848669.98 total 117.59 last 437.25) - open/close 49204 (time 1747848679.99 total 127.59 last 483.03) - open/close 53776 (time 1747848689.99 total 137.59 last 457.05) - open/close 57883 (time 1747848699.99 total 147.59 last 410.67) - open/close 60000 (time 1747848705.34 total 152.95 last 395.47) - open/close 63758 (time 1747848715.35 total 162.95 last 375.70) total: 64768 open/close in 166.19 seconds: 389.72 ops/second - /unlink 5572 (time 1747848729.55 total 10.00 last 557.15) - /unlink 10000 (time 1747848737.81 total 18.26 last 535.98) - /unlink 15544 (time 1747848747.81 total 28.26 last 554.34) - /unlink 20000 (time 1747848754.85 total 35.31 last 632.34) - /unlink 26520 (time 1747848764.86 total 45.31 last 651.90) - /unlink 30000 (time 1747848769.41 total 49.86 last 764.76) - /unlink 39879 (time 1747848779.41 total 59.86 last 987.88) - /unlink 49967 (time 1747848789.41 total 69.86 last 1008.71) - /unlink 60000 (time 1747848798.87 total 79.32 last 1060.41) total: 64768 /unlink in 83.74 seconds: 773.45 ops/second Waiting for MDT destroys to complete - open/close 6031 (time 1747848867.81 total 10.00 last 603.02) - open/close 10000 (time 1747848876.71 total 18.90 last 446.19) - open/close 14617 (time 1747848886.71 total 28.90 last 461.65) - open/close 19125 (time 1747848896.71 total 38.90 last 450.74) - open/close 20000 (time 1747848899.35 total 41.54 last 330.80) - open/close 25056 (time 1747848909.36 total 51.55 last 505.49) - open/close 29777 (time 1747848919.36 total 61.55 last 471.95) - open/close 34501 (time 1747848929.36 total 71.55 last 472.36) - open/close 39600 (time 1747848939.36 total 81.55 last 509.85) - open/close 45148 (time 1747848949.36 total 91.55 last 554.79) - open/close 50000 (time 1747848959.26 total 101.45 last 490.29) - open/close 54701 (time 1747848969.26 total 111.45 last 470.08) - open/close 59666 (time 1747848979.26 total 121.45 last 496.47) - open/close 63802 (time 1747848989.26 total 131.45 last 413.55) total: 64768 open/close in 133.49 seconds: 485.19 ops/second - /unlink 8689 (time 1747849002.19 total 10.00 last 868.78) - /unlink 16010 (time 1747849012.19 total 20.00 last 732.09) - /unlink 20000 (time 1747849017.42 total 25.23 last 763.17) - /unlink 26834 (time 1747849027.42 total 35.23 last 683.30) - /unlink 30000 (time 1747849031.88 total 39.69 last 710.69) - /unlink 37562 (time 1747849041.88 total 49.69 last 756.18) - /unlink 40000 (time 1747849045.02 total 52.83 last 774.75) - /unlink 46744 (time 1747849055.02 total 62.83 last 674.33) - /unlink 50000 (time 1747849058.94 total 66.75 last 830.54) - /unlink 59297 (time 1747849068.95 total 76.76 last 928.97) total: 64768 /unlink in 82.71 seconds: 783.06 ops/second Waiting for MDT destroys to complete - open/close 3833 (time 1747849162.49 total 10.00 last 383.22) - open/close 8462 (time 1747849172.49 total 20.00 last 462.84) - open/close 10000 (time 1747849175.38 total 22.90 last 531.47) - open/close 14748 (time 1747849185.38 total 32.90 last 474.69) - open/close 19568 (time 1747849195.38 total 42.90 last 481.97) - open/close 25427 (time 1747849205.38 total 52.90 last 585.82) - open/close 30000 (time 1747849213.34 total 60.85 last 575.03) - open/close 35233 (time 1747849223.34 total 70.86 last 523.21) - open/close 40000 (time 1747849231.55 total 79.07 last 580.31) - open/close 45681 (time 1747849241.55 total 89.07 last 568.03) - open/close 50000 (time 1747849249.98 total 97.50 last 512.73) - open/close 55538 (time 1747849259.98 total 107.50 last 553.77) - open/close 60000 (time 1747849269.34 total 116.86 last 476.64) total: 64768 open/close in 126.59 seconds: 511.65 ops/second - /unlink 9643 (time 1747849289.92 total 10.00 last 964.28) - /unlink 18255 (time 1747849299.92 total 20.00 last 861.20) - /unlink 28495 (time 1747849309.93 total 30.00 last 1023.89) - /unlink 38202 (time 1747849319.93 total 40.00 last 970.67) - /unlink 40000 (time 1747849322.01 total 42.09 last 862.94) - /unlink 49180 (time 1747849332.01 total 52.09 last 917.99) - /unlink 58679 (time 1747849342.01 total 62.09 last 949.86) total: 64768 /unlink in 68.13 seconds: 950.62 ops/second Waiting for MDT destroys to complete - open/close 4216 (time 1747849438.17 total 10.00 last 421.54) - open/close 9628 (time 1747849448.17 total 20.00 last 541.15) - open/close 15108 (time 1747849458.17 total 30.00 last 547.95) - open/close 19858 (time 1747849468.17 total 40.00 last 474.97) - open/close 24739 (time 1747849478.17 total 50.00 last 488.08) - open/close 29642 (time 1747849488.18 total 60.01 last 490.22) - open/close 35296 (time 1747849498.18 total 70.01 last 565.05) - open/close 40000 (time 1747849505.76 total 77.59 last 620.42) - open/close 46045 (time 1747849515.76 total 87.59 last 604.44) - open/close 50000 (time 1747849523.63 total 95.46 last 502.69) - open/close 55293 (time 1747849533.63 total 105.46 last 529.29) - open/close 59989 (time 1747849543.63 total 115.46 last 469.53) total: 64768 open/close in 124.62 seconds: 519.74 ops/second - /unlink 7455 (time 1747849563.63 total 10.00 last 745.50) - /unlink 10000 (time 1747849568.17 total 14.55 last 559.90) - /unlink 18083 (time 1747849578.18 total 24.55 last 808.24) - /unlink 20000 (time 1747849580.55 total 26.92 last 808.99) - /unlink 30000 (time 1747849590.55 total 36.92 last 999.94) - /unlink 38296 (time 1747849600.55 total 46.92 last 829.54) - /unlink 40000 (time 1747849602.58 total 48.95 last 836.90) - /unlink 48146 (time 1747849612.58 total 58.95 last 814.59) - /unlink 50000 (time 1747849614.69 total 61.07 last 877.84) - /unlink 59160 (time 1747849624.69 total 71.07 last 915.99) total: 64768 /unlink in 78.40 seconds: 826.13 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg407-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg407-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg407-server stop mds service on oleg407-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg407-server unloading modules via unload_modules_local on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-client: ssh exited with exit code 2 pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 PASS 106 (1647s) Stopping clients: oleg407-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg407-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg407-client: oleg407-server: ssh exited with exit code 2 oleg407-server: oleg407-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg407-server' oleg407-server: oleg407-server.virtnet: executing load_modules_local oleg407-server: Loading modules from /home/green/git/lustre-release/lustre oleg407-server: detected 4 online CPUs by sysfs oleg407-server: MODOPTS_LIBCFS= oleg407-server: Force libcfs to create 2 CPU partitions oleg407-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg407-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1750 sec ========== 13:49:08 (1747849748) === conf-sanity: start cleanup 13:49:08 (1747849748) === === conf-sanity: finish cleanup 13:49:09 (1747849749) ===