-----============= acceptance-small: conf-sanity ============----- Tue Apr 8 15:28:32 EDT 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg322-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg322-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg322-server oleg322-server: oleg322-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: MODOPTS_LIBCFS= oleg322-server: Force libcfs to create 2 CPU partitions oleg322-server: libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg322-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg322-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg322-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg322-server: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg322-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg322-server: oleg322-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg322-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:29:59 (1744140599) Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:-f) oleg322-server: oleg322-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: MODOPTS_LIBCFS= oleg322-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg322-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg322-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg322-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg322-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg322-server: oleg322-server.virtnet: executing set_default_debug -1 all pdsh@oleg322-client: oleg322-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg322-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg322-client.virtnet: -o user_xattr,flock 192.168.203.122@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5143 (time 1744140638.11 total 10.00 last 514.24) - open/close 10000 (time 1744140647.45 total 19.34 last 520.04) - open/close 15257 (time 1744140657.45 total 29.34 last 525.61) - open/close 20000 (time 1744140666.84 total 38.73 last 505.19) - open/close 25934 (time 1744140676.84 total 48.73 last 593.39) - open/close 30000 (time 1744140684.27 total 56.16 last 547.38) - open/close 35388 (time 1744140694.27 total 66.16 last 538.74) - open/close 40000 (time 1744140702.55 total 74.44 last 557.03) - open/close 45458 (time 1744140712.55 total 84.44 last 545.77) - open/close 50000 (time 1744140720.75 total 92.64 last 554.19) - open/close 55225 (time 1744140730.75 total 102.64 last 522.49) - open/close 60000 (time 1744140739.23 total 111.12 last 562.99) total: 64768 open/close in 119.90 seconds: 540.20 ops/second - /unlink 9775 (time 1744140758.84 total 10.00 last 977.40) - /unlink 19056 (time 1744140768.84 total 20.00 last 928.09) - /unlink 28640 (time 1744140778.84 total 30.00 last 958.40) - /unlink 37917 (time 1744140788.84 total 40.00 last 927.63) - /unlink 40000 (time 1744140791.49 total 42.65 last 785.38) - /unlink 49668 (time 1744140801.49 total 52.65 last 966.78) - /unlink 59141 (time 1744140811.49 total 62.65 last 947.30) total: 64768 /unlink in 68.83 seconds: 940.93 ops/second Waiting for MDT destroys to complete - open/close 5375 (time 1744140922.05 total 10.00 last 537.49) - open/close 10000 (time 1744140930.47 total 18.42 last 549.24) - open/close 15972 (time 1744140940.47 total 28.42 last 597.20) - open/close 20000 (time 1744140947.83 total 35.79 last 546.85) - open/close 25507 (time 1744140957.84 total 45.79 last 550.60) - open/close 30000 (time 1744140965.49 total 53.44 last 586.86) - open/close 35849 (time 1744140975.49 total 63.44 last 584.90) - open/close 40000 (time 1744140982.08 total 70.03 last 630.27) - open/close 46174 (time 1744140992.08 total 80.03 last 617.33) - open/close 50000 (time 1744140998.39 total 86.34 last 606.07) - open/close 55582 (time 1744141008.39 total 96.35 last 558.12) - open/close 60000 (time 1744141015.93 total 103.88 last 586.18) total: 64768 open/close in 112.85 seconds: 573.91 ops/second - /unlink 9232 (time 1744141035.76 total 10.00 last 923.12) - /unlink 18837 (time 1744141045.76 total 20.00 last 960.49) - /unlink 28184 (time 1744141055.76 total 30.00 last 934.64) - /unlink 37817 (time 1744141065.76 total 40.00 last 963.21) - /unlink 40000 (time 1744141068.09 total 42.33 last 937.19) - /unlink 48935 (time 1744141078.09 total 52.33 last 893.43) - /unlink 58650 (time 1744141088.09 total 62.33 last 971.49) total: 64768 /unlink in 68.69 seconds: 942.90 ops/second Waiting for MDT destroys to complete - open/close 5408 (time 1744141197.58 total 10.00 last 540.74) - open/close 10000 (time 1744141206.21 total 18.63 last 532.03) - open/close 15379 (time 1744141216.21 total 28.63 last 537.90) - open/close 20000 (time 1744141224.84 total 37.26 last 535.58) - open/close 25454 (time 1744141234.84 total 47.26 last 545.35) - open/close 30000 (time 1744141243.61 total 56.03 last 518.43) - open/close 35025 (time 1744141253.61 total 66.03 last 502.45) - open/close 40000 (time 1744141263.18 total 75.60 last 519.88) - open/close 44821 (time 1744141273.18 total 85.60 last 482.06) - open/close 49681 (time 1744141283.18 total 95.60 last 485.93) - open/close 54045 (time 1744141293.18 total 105.60 last 436.38) - open/close 58875 (time 1744141303.19 total 115.60 last 482.96) - open/close 60000 (time 1744141305.51 total 117.92 last 484.91) - open/close 64625 (time 1744141315.51 total 127.92 last 462.49) total: 64768 open/close in 128.55 seconds: 503.83 ops/second - /unlink 4523 (time 1744141327.49 total 10.00 last 452.24) - /unlink 10000 (time 1744141334.56 total 17.07 last 774.91) - /unlink 18168 (time 1744141344.56 total 27.07 last 816.68) - /unlink 20000 (time 1744141346.95 total 29.46 last 766.98) - /unlink 26304 (time 1744141356.95 total 39.46 last 630.39) - /unlink 30000 (time 1744141363.65 total 46.16 last 551.50) - /unlink 36391 (time 1744141373.65 total 56.16 last 639.09) - /unlink 40000 (time 1744141378.53 total 61.04 last 739.79) - /unlink 49222 (time 1744141388.53 total 71.04 last 922.19) - /unlink 57814 (time 1744141398.53 total 81.04 last 859.06) - /unlink 60000 (time 1744141400.99 total 83.50 last 888.38) total: 64768 /unlink in 88.72 seconds: 730.02 ops/second Waiting for MDT destroys to complete - open/close 3438 (time 1744141488.02 total 10.00 last 343.77) - open/close 8279 (time 1744141498.02 total 20.00 last 484.08) - open/close 10000 (time 1744141501.20 total 23.19 last 540.08) - open/close 15411 (time 1744141511.20 total 33.19 last 541.10) - open/close 20000 (time 1744141520.31 total 42.30 last 503.78) - open/close 24535 (time 1744141530.31 total 52.30 last 453.49) - open/close 29166 (time 1744141540.31 total 62.30 last 463.08) - open/close 34070 (time 1744141550.31 total 72.30 last 490.37) - open/close 39060 (time 1744141560.31 total 82.30 last 498.93) - open/close 44164 (time 1744141570.32 total 92.30 last 510.33) - open/close 49596 (time 1744141580.32 total 102.30 last 543.18) - open/close 54225 (time 1744141590.32 total 112.30 last 462.86) - open/close 59507 (time 1744141600.32 total 122.30 last 528.11) - open/close 64486 (time 1744141610.32 total 132.30 last 497.89) total: 64768 open/close in 132.82 seconds: 487.63 ops/second - /unlink 8841 (time 1744141621.75 total 10.00 last 884.02) - /unlink 18153 (time 1744141631.75 total 20.00 last 931.12) - /unlink 27230 (time 1744141641.75 total 30.00 last 907.63) - /unlink 30000 (time 1744141645.18 total 33.43 last 807.52) - /unlink 39529 (time 1744141655.19 total 43.43 last 952.82) - /unlink 49267 (time 1744141665.19 total 53.43 last 973.78) - /unlink 58856 (time 1744141675.19 total 63.43 last 958.85) total: 64768 /unlink in 69.53 seconds: 931.58 ops/second Waiting for MDT destroys to complete - open/close 5262 (time 1744141773.63 total 10.00 last 526.20) - open/close 10000 (time 1744141783.00 total 19.38 last 505.18) - open/close 15294 (time 1744141793.00 total 29.38 last 529.37) - open/close 20000 (time 1744141801.62 total 38.00 last 546.20) - open/close 25341 (time 1744141811.62 total 48.00 last 534.08) - open/close 30000 (time 1744141821.41 total 57.78 last 476.03) - open/close 35230 (time 1744141831.41 total 67.78 last 522.92) - open/close 40000 (time 1744141840.74 total 77.12 last 511.15) - open/close 44913 (time 1744141850.74 total 87.12 last 491.21) - open/close 49993 (time 1744141860.75 total 97.12 last 507.91) - open/close 54912 (time 1744141870.75 total 107.12 last 491.89) - open/close 60000 (time 1744141880.49 total 116.87 last 521.89) total: 64768 open/close in 126.15 seconds: 513.44 ops/second - /unlink 9009 (time 1744141900.67 total 10.00 last 900.86) - /unlink 18111 (time 1744141910.67 total 20.00 last 910.12) - /unlink 20000 (time 1744141912.78 total 22.11 last 897.49) - /unlink 29341 (time 1744141922.78 total 32.11 last 934.02) - /unlink 38855 (time 1744141932.78 total 42.11 last 951.33) - /unlink 48611 (time 1744141942.78 total 52.11 last 975.55) - /unlink 58413 (time 1744141952.78 total 62.11 last 980.16) total: 64768 /unlink in 68.99 seconds: 938.84 ops/second Waiting for MDT destroys to complete fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg322-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg322-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg322-server stop mds service on oleg322-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg322-server unloading modules via unload_modules_local on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-client: ssh exited with exit code 2 pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 PASS 106 (1477s) Stopping clients: oleg322-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg322-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg322-client: oleg322-server: ssh exited with exit code 2 oleg322-server: oleg322-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg322-server' oleg322-server: oleg322-server.virtnet: executing load_modules_local oleg322-server: Loading modules from /home/green/git/lustre-release/lustre oleg322-server: detected 4 online CPUs by sysfs oleg322-server: MODOPTS_LIBCFS= oleg322-server: Force libcfs to create 2 CPU partitions oleg322-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg322-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1584 sec ========== 15:54:56 (1744142096) === conf-sanity: start cleanup 15:54:57 (1744142097) === === conf-sanity: finish cleanup 15:54:58 (1744142098) ===