-----============= acceptance-small: conf-sanity ============----- Thu Feb 27 11:20:54 EST 2025 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 oleg357-server: ls: cannot access /home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex: No such file or directory excepting tests: 32newtarball Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg357-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg357-server oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 11:22:19 (1740673339) Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 5520 (time 1740673378.90 total 10.00 last 551.91) - open/close 10000 (time 1740673386.96 total 18.06 last 556.06) - open/close 15697 (time 1740673396.96 total 28.06 last 569.60) - open/close 20000 (time 1740673404.75 total 35.85 last 552.37) - open/close 25355 (time 1740673414.75 total 45.85 last 535.49) - open/close 30000 (time 1740673423.36 total 54.46 last 539.52) - open/close 35053 (time 1740673433.36 total 64.46 last 505.27) - open/close 39431 (time 1740673443.36 total 74.46 last 437.79) - open/close 44165 (time 1740673453.36 total 84.46 last 473.38) - open/close 48990 (time 1740673463.36 total 94.46 last 482.41) - open/close 54430 (time 1740673473.36 total 104.46 last 543.92) - open/close 59879 (time 1740673483.36 total 114.46 last 544.90) total: 64768 open/close in 123.55 seconds: 524.24 ops/second - /unlink 9574 (time 1740673503.36 total 10.00 last 957.31) - /unlink 19292 (time 1740673513.36 total 20.00 last 971.80) - /unlink 28896 (time 1740673523.36 total 30.00 last 960.35) - /unlink 38833 (time 1740673533.36 total 40.00 last 993.62) - /unlink 48684 (time 1740673543.36 total 50.00 last 985.09) - /unlink 58594 (time 1740673553.36 total 60.00 last 990.94) total: 64768 /unlink in 66.11 seconds: 979.74 ops/second Waiting for MDT destroys to complete oleg357-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg357-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 - open/close 5703 (time 1740673666.73 total 10.00 last 570.23) - open/close 10000 (time 1740673674.22 total 17.49 last 573.51) - open/close 15628 (time 1740673684.22 total 27.50 last 562.72) - open/close 20000 (time 1740673692.14 total 35.42 last 551.81) - open/close 25586 (time 1740673702.15 total 45.42 last 558.52) - open/close 30000 (time 1740673709.96 total 53.23 last 565.08) - open/close 35665 (time 1740673719.96 total 63.23 last 566.45) - open/close 40000 (time 1740673727.87 total 71.14 last 547.94) - open/close 45564 (time 1740673737.87 total 81.14 last 556.39) - open/close 50000 (time 1740673745.82 total 89.10 last 557.73) - open/close 54612 (time 1740673755.82 total 99.10 last 461.18) - open/close 58855 (time 1740673765.82 total 109.10 last 424.29) - open/close 60000 (time 1740673768.22 total 111.49 last 478.89) total: 64768 open/close in 120.07 seconds: 539.42 ops/second - /unlink 7248 (time 1740673787.76 total 10.00 last 724.71) - /unlink 10000 (time 1740673790.79 total 13.04 last 905.51) - /unlink 18151 (time 1740673800.79 total 23.04 last 815.09) - /unlink 20000 (time 1740673803.44 total 25.68 last 699.83) - /unlink 30000 (time 1740673813.38 total 35.63 last 1005.27) - /unlink 38920 (time 1740673823.39 total 45.63 last 891.91) - /unlink 47435 (time 1740673833.39 total 55.63 last 851.43) - /unlink 50000 (time 1740673835.94 total 58.19 last 1003.48) - /unlink 58053 (time 1740673845.94 total 68.19 last 805.23) - /unlink 60000 (time 1740673848.58 total 70.82 last 739.11) total: 64768 /unlink in 77.23 seconds: 838.60 ops/second Waiting for MDT destroys to complete oleg357-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported oleg357-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 - open/close 5560 (time 1740673942.41 total 10.00 last 555.95) - open/close 10000 (time 1740673950.34 total 17.92 last 560.36) - open/close 14791 (time 1740673960.34 total 27.93 last 479.02) - open/close 17245 (time 1740673970.34 total 37.93 last 245.36) - open/close 20000 (time 1740673978.40 total 45.99 last 341.70) - open/close 25574 (time 1740673988.40 total 55.99 last 557.40) - open/close 30000 (time 1740673996.10 total 63.69 last 575.04) - open/close 34395 (time 1740674006.10 total 73.69 last 439.46) - open/close 39522 (time 1740674016.10 total 83.69 last 512.66) - open/close 45666 (time 1740674026.10 total 93.69 last 614.40) - open/close 50000 (time 1740674033.08 total 100.67 last 620.79) - open/close 55557 (time 1740674043.08 total 110.67 last 555.65) - open/close 60000 (time 1740674051.14 total 118.73 last 551.62) total: 64768 open/close in 128.43 seconds: 504.32 ops/second - /unlink 10000 (time 1740674071.38 total 9.72 last 1028.62) - /unlink 18876 (time 1740674081.38 total 19.72 last 887.59) - /unlink 27219 (time 1740674091.38 total 29.72 last 834.25) - /unlink 30000 (time 1740674094.60 total 32.95 last 862.48) - /unlink 36847 (time 1740674104.60 total 42.95 last 684.68) - /unlink 40000 (time 1740674109.49 total 47.83 last 645.51) - /unlink 47200 (time 1740674119.49 total 57.83 last 719.96) - /unlink 50000 (time 1740674123.98 total 62.32 last 623.75) - /unlink 56365 (time 1740674133.98 total 72.32 last 636.50) - /unlink 60000 (time 1740674139.14 total 77.49 last 703.68) total: 64768 /unlink in 84.59 seconds: 765.66 ops/second Waiting for MDT destroys to complete oleg357-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 oleg357-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 - open/close 4020 (time 1740674232.87 total 10.00 last 401.95) - open/close 8483 (time 1740674242.87 total 20.00 last 446.23) - open/close 10000 (time 1740674247.26 total 24.39 last 345.72) - open/close 13195 (time 1740674257.26 total 34.39 last 319.46) - open/close 17563 (time 1740674267.26 total 44.40 last 436.67) - open/close 20000 (time 1740674272.63 total 49.76 last 454.25) - open/close 24619 (time 1740674282.63 total 59.76 last 461.84) - open/close 28949 (time 1740674292.63 total 69.76 last 432.97) - open/close 30000 (time 1740674294.94 total 72.07 last 454.83) - open/close 34502 (time 1740674304.94 total 82.07 last 450.14) - open/close 39016 (time 1740674314.94 total 92.08 last 451.33) - open/close 40000 (time 1740674317.25 total 94.38 last 426.46) - open/close 43849 (time 1740674327.25 total 104.38 last 384.90) - open/close 46659 (time 1740674337.25 total 114.38 last 281.00) - open/close 50000 (time 1740674346.39 total 123.53 last 365.33) - open/close 52820 (time 1740674356.40 total 133.53 last 281.90) - open/close 55579 (time 1740674366.40 total 143.53 last 275.89) - open/close 58551 (time 1740674376.40 total 153.54 last 297.10) - open/close 60000 (time 1740674381.74 total 158.87 last 271.39) - open/close 63297 (time 1740674391.74 total 168.88 last 329.57) total: 64768 open/close in 174.14 seconds: 371.93 ops/second - /unlink 7372 (time 1740674407.86 total 10.00 last 737.14) - /unlink 10000 (time 1740674411.34 total 13.48 last 755.40) - /unlink 18120 (time 1740674421.34 total 23.48 last 811.93) - /unlink 20000 (time 1740674423.63 total 25.78 last 819.32) - /unlink 27285 (time 1740674433.64 total 35.78 last 728.35) - /unlink 30000 (time 1740674437.48 total 39.62 last 705.94) - /unlink 36444 (time 1740674447.48 total 49.62 last 644.36) - /unlink 40000 (time 1740674452.49 total 54.63 last 709.92) - /unlink 45529 (time 1740674462.49 total 64.63 last 552.84) - /unlink 50000 (time 1740674469.23 total 71.37 last 663.58) - /unlink 57430 (time 1740674479.23 total 81.37 last 742.92) - /unlink 60000 (time 1740674482.80 total 84.95 last 719.31) total: 64768 /unlink in 91.05 seconds: 711.35 ops/second Waiting for MDT destroys to complete oleg357-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg357-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 - open/close 5351 (time 1740674568.29 total 10.00 last 535.05) - open/close 9865 (time 1740674578.30 total 20.00 last 451.23) - open/close 14343 (time 1740674588.30 total 30.01 last 447.74) - open/close 20000 (time 1740674597.97 total 39.67 last 585.13) - open/close 25391 (time 1740674607.97 total 49.67 last 539.06) - open/close 30000 (time 1740674617.29 total 59.00 last 494.21) - open/close 35533 (time 1740674627.29 total 69.00 last 553.19) - open/close 40000 (time 1740674636.63 total 78.33 last 478.75) - open/close 44741 (time 1740674646.63 total 88.33 last 474.06) - open/close 49542 (time 1740674656.63 total 98.34 last 480.04) - open/close 54213 (time 1740674666.63 total 108.34 last 467.08) - open/close 58829 (time 1740674676.63 total 118.34 last 461.50) - open/close 60000 (time 1740674679.11 total 120.82 last 471.22) total: 64768 open/close in 130.62 seconds: 495.86 ops/second - /unlink 8822 (time 1740674699.82 total 10.00 last 882.12) - /unlink 17957 (time 1740674709.82 total 20.00 last 913.43) - /unlink 20000 (time 1740674712.02 total 22.20 last 928.20) - /unlink 29394 (time 1740674722.02 total 32.20 last 939.36) - /unlink 38630 (time 1740674732.02 total 42.20 last 923.55) - /unlink 47793 (time 1740674742.02 total 52.20 last 916.29) - /unlink 50000 (time 1740674744.56 total 54.74 last 869.64) - /unlink 55030 (time 1740674754.56 total 64.74 last 502.99) - /unlink 58075 (time 1740674764.56 total 74.74 last 304.42) - /unlink 60000 (time 1740674770.22 total 80.40 last 340.45) total: 64768 /unlink in 87.16 seconds: 743.08 ops/second Waiting for MDT destroys to complete oleg357-server: fstrim: /mnt/lustre-ost1: the discard operation is not supported oleg357-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 PASS 106 (1531s) Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 oleg357-server: oleg357-server.virtnet: executing set_hostid Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1636 sec ========== 11:48:11 (1740674891) === conf-sanity: start cleanup 11:48:11 (1740674891) === === conf-sanity: finish cleanup 11:48:12 (1740674892) ===