-----============= acceptance-small: conf-sanity ============----- Wed Oct 22 09:53:48 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg204-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32newtarball Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg204-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg204-server oleg204-server: oleg204-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg204-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg204-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg204-server: oleg204-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg204-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server == conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 09:55:36 (1761141336) Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) oleg204-server: oleg204-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg204-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg204-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg204-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg204-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg204-server: oleg204-server.virtnet: executing set_default_debug -1 all pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg204-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg204-client.virtnet: -o user_xattr,flock 192.168.202.104@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 65808 368 65440 1% /mnt/lustre[OST:0] filesystem_summary: 65969 529 65440 1% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close 4743 (time 1761141388.69 total 10.00 last 474.24) - open/close 9853 (time 1761141398.69 total 20.00 last 510.92) - open/close 14039 (time 1761141408.70 total 30.00 last 418.52) - open/close 18231 (time 1761141418.70 total 40.00 last 419.19) - open/close 20000 (time 1761141422.46 total 43.77 last 469.31) - open/close 24670 (time 1761141432.47 total 53.78 last 466.91) - open/close 29553 (time 1761141442.47 total 63.78 last 488.27) - open/close 33903 (time 1761141452.47 total 73.78 last 434.95) - open/close 38087 (time 1761141462.47 total 83.78 last 418.39) - open/close 40000 (time 1761141466.57 total 87.88 last 466.90) - open/close 44894 (time 1761141476.57 total 97.88 last 489.35) - open/close 49323 (time 1761141486.57 total 107.88 last 442.76) - open/close 53445 (time 1761141496.57 total 117.88 last 412.14) - open/close 57956 (time 1761141506.57 total 127.88 last 451.03) - open/close 60000 (time 1761141511.50 total 132.81 last 415.10) - open/close 63845 (time 1761141521.50 total 142.81 last 384.45) total: 64768 open/close in 145.59 seconds: 444.88 ops/second - /unlink 6337 (time 1761141535.82 total 10.00 last 633.64) - /unlink 10000 (time 1761141541.07 total 15.25 last 697.63) - /unlink 17182 (time 1761141551.07 total 25.25 last 718.17) - /unlink 20000 (time 1761141555.45 total 29.64 last 642.47) - /unlink 27131 (time 1761141565.46 total 39.64 last 713.03) - /unlink 30000 (time 1761141570.06 total 44.24 last 623.54) - /unlink 36163 (time 1761141580.06 total 54.24 last 616.20) - /unlink 40000 (time 1761141585.65 total 59.84 last 685.88) - /unlink 48612 (time 1761141595.65 total 69.84 last 861.15) - /unlink 56381 (time 1761141605.65 total 79.84 last 776.86) - /unlink 60000 (time 1761141610.35 total 84.53 last 771.33) total: 64768 /unlink in 89.98 seconds: 719.83 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 3064 (time 1761141659.60 total 10.00 last 306.35) - open/close 6812 (time 1761141669.60 total 20.00 last 374.79) - open/close 10000 (time 1761141676.90 total 27.30 last 437.00) - open/close 14359 (time 1761141686.90 total 37.30 last 435.79) - open/close 17855 (time 1761141696.90 total 47.30 last 349.57) - open/close 20000 (time 1761141702.25 total 52.65 last 401.15) - open/close 23410 (time 1761141712.25 total 62.65 last 340.95) - open/close 27306 (time 1761141722.25 total 72.65 last 389.51) - open/close 30000 (time 1761141728.57 total 78.97 last 426.23) - open/close 34738 (time 1761141738.58 total 88.97 last 473.70) - open/close 39636 (time 1761141748.58 total 98.98 last 489.73) - open/close 44552 (time 1761141758.58 total 108.98 last 491.49) - open/close 49205 (time 1761141768.58 total 118.98 last 465.28) - open/close 50000 (time 1761141770.63 total 121.02 last 388.67) - open/close 53902 (time 1761141780.63 total 131.02 last 390.17) - open/close 58220 (time 1761141790.63 total 141.03 last 431.70) - open/close 60000 (time 1761141794.53 total 144.93 last 456.35) - open/close 64132 (time 1761141804.53 total 154.93 last 413.13) total: 64768 open/close in 156.61 seconds: 413.56 ops/second - /unlink 7817 (time 1761141817.66 total 10.00 last 781.62) - /unlink 10000 (time 1761141820.71 total 13.04 last 717.38) - /unlink 17950 (time 1761141830.71 total 23.04 last 794.97) - /unlink 20000 (time 1761141833.14 total 25.48 last 842.22) - /unlink 28372 (time 1761141843.14 total 35.48 last 837.15) - /unlink 37046 (time 1761141853.14 total 45.48 last 867.38) - /unlink 40000 (time 1761141857.26 total 49.60 last 717.48) - /unlink 48132 (time 1761141867.26 total 59.60 last 812.67) - /unlink 50000 (time 1761141869.52 total 61.86 last 827.78) - /unlink 58715 (time 1761141879.52 total 71.86 last 871.44) total: 64768 /unlink in 78.26 seconds: 827.60 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 4976 (time 1761141970.44 total 10.00 last 497.51) - open/close 9792 (time 1761141980.44 total 20.00 last 481.59) - open/close 14252 (time 1761141990.44 total 30.00 last 446.00) - open/close 19285 (time 1761142000.44 total 40.00 last 503.24) - open/close 23596 (time 1761142010.44 total 50.00 last 431.09) - open/close 28106 (time 1761142020.44 total 60.00 last 450.94) - open/close 30000 (time 1761142024.25 total 63.81 last 497.25) - open/close 34601 (time 1761142034.25 total 73.82 last 460.02) - open/close 38975 (time 1761142044.25 total 83.82 last 437.28) - open/close 40000 (time 1761142046.43 total 86.00 last 469.79) - open/close 44845 (time 1761142056.43 total 96.00 last 484.47) - open/close 49670 (time 1761142066.43 total 106.00 last 482.48) - open/close 54661 (time 1761142076.44 total 116.00 last 499.00) - open/close 59404 (time 1761142086.44 total 126.00 last 474.25) - open/close 63509 (time 1761142096.44 total 136.01 last 410.42) total: 64768 open/close in 138.75 seconds: 466.80 ops/second - /unlink 8818 (time 1761142110.56 total 10.00 last 881.73) - /unlink 17924 (time 1761142120.56 total 20.00 last 910.56) - /unlink 20000 (time 1761142122.77 total 22.21 last 939.19) - /unlink 29167 (time 1761142132.77 total 32.21 last 916.67) - /unlink 38230 (time 1761142142.77 total 42.21 last 906.29) - /unlink 47525 (time 1761142152.77 total 52.21 last 929.49) - /unlink 50000 (time 1761142155.27 total 54.71 last 990.36) - /unlink 59045 (time 1761142165.27 total 64.71 last 904.42) total: 64768 /unlink in 70.98 seconds: 912.45 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 4608 (time 1761142240.04 total 10.00 last 460.73) - open/close 9165 (time 1761142250.05 total 20.00 last 455.68) - open/close 13823 (time 1761142260.05 total 30.00 last 465.80) - open/close 18246 (time 1761142270.05 total 40.00 last 442.28) - open/close 20000 (time 1761142273.87 total 43.83 last 458.81) - open/close 24590 (time 1761142283.87 total 53.83 last 458.96) - open/close 29042 (time 1761142293.87 total 63.83 last 445.17) - open/close 30000 (time 1761142296.01 total 65.97 last 446.88) - open/close 34672 (time 1761142306.02 total 75.97 last 467.14) - open/close 39387 (time 1761142316.02 total 85.97 last 471.44) - open/close 43888 (time 1761142326.02 total 95.97 last 450.03) - open/close 48214 (time 1761142336.02 total 105.98 last 432.53) - open/close 50000 (time 1761142340.00 total 109.95 last 449.24) - open/close 54511 (time 1761142350.00 total 119.95 last 451.03) - open/close 58923 (time 1761142360.00 total 129.95 last 441.14) - open/close 60000 (time 1761142362.42 total 132.38 last 444.31) - open/close 64662 (time 1761142372.42 total 142.38 last 466.10) total: 64768 open/close in 142.65 seconds: 454.05 ops/second - /unlink 7519 (time 1761142384.13 total 10.00 last 751.84) - /unlink 10000 (time 1761142387.37 total 13.24 last 766.33) - /unlink 18058 (time 1761142397.37 total 23.24 last 805.71) - /unlink 20000 (time 1761142399.80 total 25.66 last 800.98) - /unlink 28044 (time 1761142409.80 total 35.66 last 804.34) - /unlink 30000 (time 1761142412.27 total 38.13 last 792.26) - /unlink 37634 (time 1761142422.27 total 48.13 last 763.35) - /unlink 40000 (time 1761142425.10 total 50.96 last 836.61) - /unlink 47848 (time 1761142435.10 total 60.96 last 784.73) - /unlink 50000 (time 1761142437.68 total 63.55 last 832.31) - /unlink 58088 (time 1761142447.68 total 73.55 last 808.75) - /unlink 60000 (time 1761142450.24 total 76.11 last 747.27) total: 64768 /unlink in 81.87 seconds: 791.10 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 - open/close 4647 (time 1761142538.79 total 10.00 last 464.59) - open/close 9098 (time 1761142548.79 total 20.00 last 445.01) - open/close 10000 (time 1761142550.80 total 22.01 last 449.84) - open/close 14537 (time 1761142560.80 total 32.01 last 453.69) - open/close 19207 (time 1761142570.80 total 42.01 last 466.97) - open/close 23740 (time 1761142580.80 total 52.01 last 453.24) - open/close 28311 (time 1761142590.80 total 62.01 last 457.03) - open/close 30000 (time 1761142594.46 total 65.67 last 462.30) - open/close 34468 (time 1761142604.46 total 75.67 last 446.76) - open/close 38924 (time 1761142614.46 total 85.67 last 445.51) - open/close 40000 (time 1761142616.83 total 88.04 last 453.82) - open/close 44517 (time 1761142626.83 total 98.04 last 451.65) - open/close 48955 (time 1761142636.83 total 108.04 last 443.74) - open/close 50000 (time 1761142639.10 total 110.32 last 459.89) - open/close 54531 (time 1761142649.11 total 120.32 last 453.07) - open/close 59087 (time 1761142659.11 total 130.32 last 455.52) - open/close 63556 (time 1761142669.11 total 140.32 last 446.81) total: 64768 open/close in 143.11 seconds: 452.58 ops/second - /unlink 7544 (time 1761142683.37 total 10.00 last 754.39) - /unlink 10000 (time 1761142686.46 total 13.09 last 796.11) - /unlink 17995 (time 1761142696.46 total 23.09 last 799.46) - /unlink 20000 (time 1761142699.17 total 25.80 last 739.94) - /unlink 28091 (time 1761142709.17 total 35.80 last 809.10) - /unlink 30000 (time 1761142711.46 total 38.09 last 832.17) - /unlink 38009 (time 1761142721.46 total 48.09 last 800.83) - /unlink 40000 (time 1761142723.82 total 50.44 last 845.74) - /unlink 48130 (time 1761142733.82 total 60.44 last 812.97) - /unlink 50000 (time 1761142736.09 total 62.72 last 823.04) - /unlink 57909 (time 1761142746.09 total 72.72 last 790.90) - /unlink 60000 (time 1761142748.60 total 75.23 last 833.62) total: 64768 /unlink in 81.62 seconds: 793.49 ops/second Waiting for MDT destroys to complete oleg204-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg204-client: oleg204-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg204-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg204-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg204-server stop mds service on oleg204-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg204-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing unload_modules_local oleg204-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg204-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-client: ssh exited with exit code 2 pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 PASS 106 (1486s) Stopping clients: oleg204-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg204-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg204-client: oleg204-server: ssh exited with exit code 2 oleg204-server: oleg204-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg204-server' oleg204-server: oleg204-server.virtnet: executing load_modules_local oleg204-server: Loading modules from /home/green/git/lustre-release/lustre oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg204-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg204-server: detected 4 online CPUs by sysfs oleg204-server: MODOPTS_LIBCFS= oleg204-server: Force libcfs to create 2 CPU partitions oleg204-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg204-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 1620 sec ========== 10:20:49 (1761142849) === conf-sanity: start cleanup 10:20:49 (1761142849) === === conf-sanity: finish cleanup 10:20:50 (1761142850) ===