-----============= acceptance-small: conf-sanity ============----- Mon Aug 31 00:59:20 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg402-server: /home/green/git/lustre-release/lustre/tests/except/conf-sanity.ex - see CLIENT_VERSION > v2_15_63-53-g1a6ef725c2 (34682651 > 34553653) for LU-16938, go 81 excepting tests: 32b 32c 32newtarball Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg402-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg402-server oleg402-server: oleg402-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg402-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg402-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg402-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server == conf-sanity test 45: long unlink handling in ptlrpcd == 01:03:01 (1788152581) start mds service on oleg402-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg402-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg402-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg402-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server unloading modules via unload_modules_local on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing unload_modules_local oleg402-server: modules unloaded. pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 PASS 45 (280s) == conf-sanity test 69: replace an OST with the same index ========================================================== 01:07:42 (1788152862) start mds service on oleg402-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg402-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg402-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg402-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 759 (time 1788152955.81 total 10.01 last 75.84) - open/close 1483 (time 1788152965.83 total 20.02 last 72.31) - open/close 2230 (time 1788152975.83 total 30.03 last 74.65) - open/close 3034 (time 1788152985.83 total 40.03 last 80.39) - open/close 3764 (time 1788152995.84 total 50.03 last 72.99) - open/close 4535 (time 1788153005.85 total 60.04 last 77.03) - open/close 5294 (time 1788153015.85 total 70.04 last 75.86) - open/close 5981 (time 1788153025.85 total 80.04 last 68.70) - open/close 6734 (time 1788153035.87 total 90.06 last 75.19) - open/close 7502 (time 1788153045.88 total 100.08 last 76.66) - open/close 8255 (time 1788153055.89 total 110.08 last 75.25) - open/close 9018 (time 1788153065.92 total 120.11 last 76.06) - open/close 9739 (time 1788153075.92 total 130.12 last 72.08) total: 10000 open/close in 133.82 seconds: 74.73 ops/second - unlinked 0 (time 1788153083 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 507 (time 1788153177.75 total 10.00 last 50.69) - open/close 1054 (time 1788153187.75 total 20.01 last 54.66) - open/close 1703 (time 1788153197.76 total 30.02 last 64.84) - open/close 2401 (time 1788153207.77 total 40.03 last 69.75) - open/close 3095 (time 1788153217.77 total 50.03 last 69.38) - open/close 3716 (time 1788153227.78 total 60.03 last 62.06) - open/close 4411 (time 1788153237.78 total 70.04 last 69.48) - open/close 5030 (time 1788153247.79 total 80.05 last 61.84) - open/close 5728 (time 1788153257.80 total 90.06 last 69.72) - open/close 6501 (time 1788153267.80 total 100.06 last 77.30) - open/close 7241 (time 1788153277.83 total 110.08 last 73.82) - open/close 7776 (time 1788153287.84 total 120.10 last 53.43) - open/close 8294 (time 1788153297.87 total 130.12 last 51.65) - open/close 8910 (time 1788153307.87 total 140.13 last 61.57) - open/close 9579 (time 1788153317.89 total 150.15 last 66.76) total: 10000 open/close in 156.30 seconds: 63.98 ops/second - unlinked 0 (time 1788153328 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 604 (time 1788153425.59 total 10.01 last 60.35) - open/close 1146 (time 1788153435.59 total 20.01 last 54.17) - open/close 1860 (time 1788153445.61 total 30.03 last 71.30) - open/close 2583 (time 1788153455.61 total 40.03 last 72.25) - open/close 3304 (time 1788153465.62 total 50.04 last 72.07) - open/close 4015 (time 1788153475.64 total 60.06 last 70.97) - open/close 4742 (time 1788153485.66 total 70.08 last 72.56) - open/close 5463 (time 1788153495.66 total 80.08 last 72.04) - open/close 6193 (time 1788153505.68 total 90.10 last 72.92) - open/close 6881 (time 1788153515.69 total 100.11 last 68.72) - open/close 7504 (time 1788153525.71 total 110.13 last 62.18) - open/close 8050 (time 1788153535.71 total 120.13 last 54.57) - open/close 8873 (time 1788153545.72 total 130.14 last 82.23) - open/close 9516 (time 1788153555.72 total 140.14 last 64.28) total: 10000 open/close in 147.68 seconds: 67.71 ops/second - unlinked 0 (time 1788153567 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 794 (time 1788153664.87 total 10.02 last 79.27) - open/close 1547 (time 1788153674.87 total 20.02 last 75.29) - open/close 2291 (time 1788153684.89 total 30.03 last 74.31) - open/close 3032 (time 1788153694.89 total 40.03 last 74.10) - open/close 3778 (time 1788153704.89 total 50.03 last 74.57) - open/close 4512 (time 1788153714.91 total 60.06 last 73.22) - open/close 5198 (time 1788153724.93 total 70.07 last 68.50) - open/close 5951 (time 1788153734.93 total 80.08 last 75.27) - open/close 6674 (time 1788153744.94 total 90.09 last 72.23) - open/close 7381 (time 1788153754.95 total 100.10 last 70.62) - open/close 8134 (time 1788153764.96 total 110.10 last 75.27) - open/close 8876 (time 1788153774.97 total 120.11 last 74.14) - open/close 9621 (time 1788153784.98 total 130.12 last 74.40) total: 10000 open/close in 134.89 seconds: 74.13 ops/second - unlinked 0 (time 1788153793 ; total 0 ; last 0) total: 10000 unlinks in 77 seconds: 129.870132 unlinks/second - open/close 682 (time 1788153884.83 total 10.01 last 68.14) - open/close 1369 (time 1788153894.83 total 20.01 last 68.65) - open/close 2091 (time 1788153904.83 total 30.02 last 72.19) - open/close 2732 (time 1788153914.84 total 40.02 last 64.06) - open/close 3413 (time 1788153924.85 total 50.03 last 68.06) - open/close 4115 (time 1788153934.87 total 60.05 last 70.05) - open/close 4844 (time 1788153944.88 total 70.06 last 72.83) - open/close 5625 (time 1788153954.88 total 80.06 last 78.10) - open/close 6372 (time 1788153964.90 total 90.08 last 74.53) - open/close 7049 (time 1788153974.91 total 100.10 last 67.60) - open/close 7657 (time 1788153984.93 total 110.12 last 60.68) - open/close 8369 (time 1788153994.95 total 120.14 last 71.06) - open/close 8958 (time 1788154004.97 total 130.15 last 58.83) - open/close 9628 (time 1788154014.97 total 140.15 last 66.97) total: 10000 open/close in 145.26 seconds: 68.84 ops/second - unlinked 0 (time 1788154024 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 723 (time 1788154123.49 total 10.00 last 72.29) - open/close 1420 (time 1788154133.50 total 20.01 last 69.63) - open/close 2093 (time 1788154143.51 total 30.02 last 67.23) - open/close 2769 (time 1788154153.52 total 40.03 last 67.52) - open/close 3515 (time 1788154163.52 total 50.04 last 74.56) - open/close 4281 (time 1788154173.52 total 60.04 last 76.59) - open/close 4975 (time 1788154183.53 total 70.04 last 69.37) - open/close 5556 (time 1788154193.53 total 80.05 last 58.08) - open/close 6112 (time 1788154203.54 total 90.06 last 55.56) - open/close 6723 (time 1788154213.55 total 100.06 last 61.05) - open/close 7419 (time 1788154223.56 total 110.07 last 69.53) - open/close 8012 (time 1788154233.56 total 120.08 last 59.28) - open/close 8640 (time 1788154243.60 total 130.11 last 62.58) - open/close 9141 (time 1788154253.61 total 140.12 last 50.04) - open/close 9689 (time 1788154263.61 total 150.13 last 54.77) total: 10000 open/close in 155.88 seconds: 64.15 ops/second - unlinked 0 (time 1788154274 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 610 (time 1788154379.43 total 10.00 last 60.97) - open/close 1219 (time 1788154389.44 total 20.02 last 60.82) - open/close 1819 (time 1788154399.46 total 30.03 last 59.91) - open/close 2407 (time 1788154409.46 total 40.03 last 58.79) - open/close 2985 (time 1788154419.47 total 50.04 last 57.75) - open/close 3550 (time 1788154429.47 total 60.05 last 56.46) - open/close 4210 (time 1788154439.48 total 70.05 last 65.99) - open/close 4865 (time 1788154449.49 total 80.07 last 65.40) - open/close 5433 (time 1788154459.49 total 90.07 last 56.78) - open/close 6073 (time 1788154469.49 total 100.07 last 63.99) - open/close 6703 (time 1788154479.50 total 110.08 last 62.94) - open/close 7415 (time 1788154489.51 total 120.09 last 71.17) - open/close 8091 (time 1788154499.51 total 130.09 last 67.57) - open/close 8746 (time 1788154509.52 total 140.10 last 65.46) - open/close 9379 (time 1788154519.53 total 150.10 last 63.26) total: 10000 open/close in 159.69 seconds: 62.62 ops/second - unlinked 0 (time 1788154533 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 725 (time 1788154636.86 total 10.02 last 72.35) - open/close 1362 (time 1788154646.87 total 20.03 last 63.65) - open/close 1987 (time 1788154656.89 total 30.04 last 62.41) - open/close 2642 (time 1788154666.89 total 40.05 last 65.44) - open/close 3279 (time 1788154676.90 total 50.06 last 63.66) - open/close 3939 (time 1788154686.90 total 60.06 last 66.00) - open/close 4647 (time 1788154696.91 total 70.06 last 70.77) - open/close 5308 (time 1788154706.91 total 80.07 last 66.05) - open/close 5926 (time 1788154716.91 total 90.07 last 61.80) - open/close 6587 (time 1788154726.92 total 100.07 last 66.08) - open/close 7281 (time 1788154736.92 total 110.08 last 69.36) - open/close 7941 (time 1788154746.94 total 120.09 last 65.91) - open/close 8647 (time 1788154756.95 total 130.10 last 70.53) - open/close 9320 (time 1788154766.95 total 140.11 last 67.28) total: 10000 open/close in 150.07 seconds: 66.64 ops/second - unlinked 0 (time 1788154781 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 663 (time 1788154882.74 total 10.01 last 66.25) - open/close 1285 (time 1788154892.76 total 20.02 last 62.13) - open/close 1876 (time 1788154902.76 total 30.03 last 59.06) - open/close 2584 (time 1788154912.77 total 40.03 last 70.78) - open/close 3248 (time 1788154922.77 total 50.03 last 66.36) - open/close 3917 (time 1788154932.78 total 60.04 last 66.85) - open/close 4544 (time 1788154942.79 total 70.06 last 62.61) - open/close 5193 (time 1788154952.80 total 80.06 last 64.86) - open/close 5825 (time 1788154962.81 total 90.07 last 63.15) - open/close 6477 (time 1788154972.81 total 100.08 last 65.15) - open/close 7164 (time 1788154982.82 total 110.09 last 68.64) - open/close 7811 (time 1788154992.83 total 120.09 last 64.68) - open/close 8404 (time 1788155002.85 total 130.11 last 59.18) - open/close 9086 (time 1788155012.86 total 140.12 last 68.11) - open/close 9783 (time 1788155022.87 total 150.14 last 69.63) total: 10000 open/close in 153.48 seconds: 65.15 ops/second - unlinked 0 (time 1788155031 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 683 (time 1788155129.52 total 10.01 last 68.20) - open/close 1333 (time 1788155139.52 total 20.02 last 64.97) - open/close 2006 (time 1788155149.53 total 30.03 last 67.23) - open/close 2652 (time 1788155159.54 total 40.04 last 64.56) - open/close 3417 (time 1788155169.55 total 50.04 last 76.47) - open/close 4116 (time 1788155179.55 total 60.05 last 69.85) - open/close 4825 (time 1788155189.56 total 70.05 last 70.88) - open/close 5474 (time 1788155199.56 total 80.05 last 64.88) - open/close 6192 (time 1788155209.56 total 90.05 last 71.80) - open/close 6924 (time 1788155219.56 total 100.05 last 73.19) - open/close 7660 (time 1788155229.56 total 110.06 last 73.58) - open/close 8458 (time 1788155239.59 total 120.09 last 79.56) - open/close 9235 (time 1788155249.59 total 130.09 last 77.69) - open/close 9958 (time 1788155259.60 total 140.09 last 72.26) total: 10000 open/close in 140.65 seconds: 71.10 ops/second - unlinked 0 (time 1788155264 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.102@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.102@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.102@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.102@tcp sys.timeout=20 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg402-client.virtnet: -o user_xattr,flock 192.168.204.102@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg402-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server unloading modules via unload_modules_local on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing unload_modules_local oleg402-server: modules unloaded. pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-client: ssh exited with exit code 2 pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 PASS 69 (2639s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 01:51:42 (1788155502) oleg402-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg402-client: oleg402-server: ssh exited with exit code 2 oleg402-server: oleg402-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg402-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg402-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f) oleg402-server: oleg402-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions oleg402-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg402-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg402-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg402-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg402-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg402-server: oleg402-server.virtnet: executing set_default_debug -1 all pdsh@oleg402-client: oleg402-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg402-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg402-server: oleg402-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg402-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg402-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg402-server stop mds service on oleg402-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg402-server SKIP 111 (216s) Stopping clients: oleg402-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg402-client.virtnet /mnt/lustre2 (opts:-f) oleg402-server: oleg402-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg402-server' oleg402-server: oleg402-server.virtnet: executing load_modules_local oleg402-server: Loading modules from /home/green/git/lustre-release/lustre oleg402-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg402-server: detected 4 online CPUs by sysfs oleg402-server: MODOPTS_LIBCFS= oleg402-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3420 sec ========== 01:56:22 (1788155782) === conf-sanity: start cleanup 01:56:25 (1788155785) === === conf-sanity: finish cleanup 01:56:28 (1788155788) ===