-----============= acceptance-small: conf-sanity ============----- Sat May 16 05:16:29 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg357-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg357-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg357-server oleg357-server: oleg357-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg357-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server == conf-sanity test 45: long unlink handling in ptlrpcd == 05:20:16 (1778923216) start mds service on oleg357-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg357-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local oleg357-server: modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 PASS 45 (268s) == conf-sanity test 69: replace an OST with the same index ========================================================== 05:24:45 (1778923485) start mds service on oleg357-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg357-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg357-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 688 (time 1778923584.75 total 10.00 last 68.77) - open/close 1327 (time 1778923594.75 total 20.01 last 63.85) - open/close 1940 (time 1778923604.77 total 30.03 last 61.19) - open/close 2653 (time 1778923614.78 total 40.04 last 71.25) - open/close 3341 (time 1778923624.79 total 50.04 last 68.75) - open/close 3958 (time 1778923634.80 total 60.05 last 61.64) - open/close 4663 (time 1778923644.80 total 70.06 last 70.48) - open/close 5351 (time 1778923654.81 total 80.07 last 68.72) - open/close 6028 (time 1778923664.81 total 90.07 last 67.68) - open/close 6733 (time 1778923674.82 total 100.08 last 70.44) - open/close 7452 (time 1778923684.83 total 110.09 last 71.85) - open/close 8157 (time 1778923694.84 total 120.10 last 70.42) - open/close 8839 (time 1778923704.87 total 130.13 last 68.01) - open/close 9487 (time 1778923714.87 total 140.13 last 64.78) total: 10000 open/close in 147.56 seconds: 67.77 ops/second - unlinked 0 (time 1778923726 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 689 (time 1778923821.72 total 10.00 last 68.88) - open/close 1363 (time 1778923831.73 total 20.01 last 67.36) - open/close 2032 (time 1778923841.75 total 30.03 last 66.76) - open/close 2564 (time 1778923851.75 total 40.03 last 53.19) - open/close 3159 (time 1778923861.75 total 50.03 last 59.49) - open/close 3869 (time 1778923871.76 total 60.04 last 70.96) - open/close 4557 (time 1778923881.76 total 70.04 last 68.78) - open/close 5244 (time 1778923891.77 total 80.05 last 68.61) - open/close 5963 (time 1778923901.79 total 90.07 last 71.78) - open/close 6656 (time 1778923911.80 total 100.08 last 69.22) - open/close 7351 (time 1778923921.82 total 110.11 last 69.35) - open/close 8102 (time 1778923931.83 total 120.11 last 75.04) - open/close 8774 (time 1778923941.86 total 130.14 last 67.00) - open/close 9398 (time 1778923951.87 total 140.16 last 62.32) - open/close 9955 (time 1778923961.90 total 150.18 last 55.55) total: 10000 open/close in 150.93 seconds: 66.26 ops/second - unlinked 0 (time 1778923967 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 676 (time 1778924066.78 total 10.00 last 67.57) - open/close 1364 (time 1778924076.79 total 20.01 last 68.77) - open/close 2076 (time 1778924086.80 total 30.02 last 71.11) - open/close 2744 (time 1778924096.82 total 40.04 last 66.69) - open/close 3345 (time 1778924106.82 total 50.04 last 60.07) - open/close 4014 (time 1778924116.83 total 60.05 last 66.86) - open/close 4692 (time 1778924126.84 total 70.06 last 67.73) - open/close 5327 (time 1778924136.84 total 80.06 last 63.48) - open/close 5944 (time 1778924146.84 total 90.06 last 61.68) - open/close 6599 (time 1778924156.85 total 100.07 last 65.47) - open/close 7378 (time 1778924166.85 total 110.07 last 77.86) - open/close 8090 (time 1778924176.86 total 120.08 last 71.16) - open/close 8795 (time 1778924186.86 total 130.08 last 70.50) - open/close 9477 (time 1778924196.86 total 140.08 last 68.16) total: 10000 open/close in 148.56 seconds: 67.31 ops/second - unlinked 0 (time 1778924210 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 595 (time 1778924310.18 total 10.01 last 59.44) - open/close 1279 (time 1778924320.19 total 20.01 last 68.37) - open/close 1945 (time 1778924330.19 total 30.02 last 66.58) - open/close 2642 (time 1778924340.20 total 40.03 last 69.64) - open/close 3365 (time 1778924350.20 total 50.03 last 72.30) - open/close 4080 (time 1778924360.20 total 60.03 last 71.48) - open/close 4746 (time 1778924370.20 total 70.03 last 66.59) - open/close 5549 (time 1778924380.21 total 80.03 last 80.28) - open/close 6271 (time 1778924390.23 total 90.05 last 72.06) - open/close 6940 (time 1778924400.25 total 100.08 last 66.75) - open/close 7624 (time 1778924410.26 total 110.09 last 68.33) - open/close 8285 (time 1778924420.27 total 120.09 last 66.06) - open/close 8896 (time 1778924430.27 total 130.10 last 61.07) - open/close 9396 (time 1778924440.29 total 140.12 last 49.88) - open/close 9910 (time 1778924450.30 total 150.13 last 51.37) total: 10000 open/close in 151.94 seconds: 65.82 ops/second - unlinked 0 (time 1778924456 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 717 (time 1778924553.88 total 10.00 last 71.68) - open/close 1431 (time 1778924563.91 total 20.03 last 71.19) - open/close 2100 (time 1778924573.94 total 30.06 last 66.71) - open/close 2702 (time 1778924583.95 total 40.07 last 60.18) - open/close 3328 (time 1778924593.97 total 50.09 last 62.46) - open/close 3973 (time 1778924603.98 total 60.10 last 64.40) - open/close 4662 (time 1778924613.99 total 70.11 last 68.86) - open/close 5363 (time 1778924623.99 total 80.11 last 70.06) - open/close 6032 (time 1778924634.01 total 90.13 last 66.82) - open/close 6715 (time 1778924644.02 total 100.13 last 68.23) - open/close 7344 (time 1778924654.03 total 110.15 last 62.83) - open/close 8003 (time 1778924664.04 total 120.16 last 65.84) - open/close 8670 (time 1778924674.04 total 130.16 last 66.69) - open/close 9347 (time 1778924684.05 total 140.17 last 67.63) total: 10000 open/close in 149.41 seconds: 66.93 ops/second - unlinked 0 (time 1778924697 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 568 (time 1778924797.16 total 10.02 last 56.68) - open/close 1197 (time 1778924807.16 total 20.02 last 62.89) - open/close 1861 (time 1778924817.17 total 30.03 last 66.37) - open/close 2470 (time 1778924827.17 total 40.03 last 60.89) - open/close 3163 (time 1778924837.18 total 50.04 last 69.22) - open/close 3848 (time 1778924847.20 total 60.05 last 68.41) - open/close 4500 (time 1778924857.21 total 70.06 last 65.13) - open/close 5205 (time 1778924867.21 total 80.07 last 70.48) - open/close 5871 (time 1778924877.23 total 90.08 last 66.49) - open/close 6659 (time 1778924887.24 total 100.09 last 78.72) - open/close 7416 (time 1778924897.26 total 110.12 last 75.48) - open/close 8070 (time 1778924907.28 total 120.14 last 65.31) - open/close 8675 (time 1778924917.29 total 130.14 last 60.45) - open/close 9258 (time 1778924927.29 total 140.15 last 58.29) - open/close 9944 (time 1778924937.29 total 150.15 last 68.57) total: 10000 open/close in 151.19 seconds: 66.14 ops/second - unlinked 0 (time 1778924942 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 659 (time 1778925044.86 total 10.02 last 65.77) - open/close 1372 (time 1778925054.87 total 20.03 last 71.26) - open/close 1993 (time 1778925064.87 total 30.03 last 62.07) - open/close 2646 (time 1778925074.88 total 40.04 last 65.22) - open/close 3308 (time 1778925084.88 total 50.04 last 66.19) - open/close 3821 (time 1778925094.90 total 60.06 last 51.21) - open/close 4455 (time 1778925104.91 total 70.07 last 63.33) - open/close 5150 (time 1778925114.91 total 80.07 last 69.49) - open/close 5758 (time 1778925124.91 total 90.08 last 60.78) - open/close 6381 (time 1778925134.92 total 100.08 last 62.26) - open/close 7059 (time 1778925144.94 total 110.11 last 67.65) - open/close 7707 (time 1778925154.96 total 120.12 last 64.71) - open/close 8340 (time 1778925164.97 total 130.14 last 63.20) - open/close 9028 (time 1778925175.00 total 140.16 last 68.62) - open/close 9744 (time 1778925185.01 total 150.17 last 71.53) total: 10000 open/close in 153.87 seconds: 64.99 ops/second - unlinked 0 (time 1778925193 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 710 (time 1778925292.30 total 10.03 last 70.79) - open/close 1416 (time 1778925302.32 total 20.05 last 70.43) - open/close 2172 (time 1778925312.33 total 30.06 last 75.54) - open/close 2913 (time 1778925322.35 total 40.08 last 73.98) - open/close 3673 (time 1778925332.36 total 50.09 last 75.93) - open/close 4424 (time 1778925342.37 total 60.10 last 75.03) - open/close 5055 (time 1778925352.37 total 70.10 last 63.09) - open/close 5709 (time 1778925362.38 total 80.11 last 65.32) - open/close 6407 (time 1778925372.38 total 90.11 last 69.80) - open/close 7057 (time 1778925382.39 total 100.12 last 64.96) - open/close 7654 (time 1778925392.39 total 110.12 last 59.68) - open/close 8386 (time 1778925402.39 total 120.12 last 73.17) - open/close 9083 (time 1778925412.40 total 130.13 last 69.68) - open/close 9890 (time 1778925422.40 total 140.13 last 80.70) total: 10000 open/close in 141.61 seconds: 70.61 ops/second - unlinked 0 (time 1778925428 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 612 (time 1778925523.63 total 10.02 last 61.08) - open/close 1201 (time 1778925533.64 total 20.02 last 58.87) - open/close 1829 (time 1778925543.66 total 30.04 last 62.68) - open/close 2463 (time 1778925553.66 total 40.05 last 63.37) - open/close 3174 (time 1778925563.67 total 50.05 last 71.07) - open/close 3924 (time 1778925573.68 total 60.06 last 74.93) - open/close 4635 (time 1778925583.68 total 70.07 last 71.06) - open/close 5299 (time 1778925593.69 total 80.08 last 66.34) - open/close 6019 (time 1778925603.69 total 90.08 last 71.97) - open/close 6695 (time 1778925613.70 total 100.09 last 67.55) - open/close 7435 (time 1778925623.70 total 110.09 last 73.98) - open/close 8107 (time 1778925633.71 total 120.09 last 67.19) - open/close 8792 (time 1778925643.71 total 130.10 last 68.46) - open/close 9542 (time 1778925653.72 total 140.10 last 74.97) total: 10000 open/close in 146.46 seconds: 68.28 ops/second - unlinked 0 (time 1778925665 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 765 (time 1778925759.96 total 10.00 last 76.48) - open/close 1524 (time 1778925769.97 total 20.01 last 75.85) - open/close 2338 (time 1778925779.97 total 30.01 last 81.39) - open/close 3099 (time 1778925789.98 total 40.01 last 76.07) - open/close 3822 (time 1778925799.98 total 50.02 last 72.26) - open/close 4560 (time 1778925810.01 total 60.05 last 73.60) - open/close 5249 (time 1778925820.03 total 70.07 last 68.76) - open/close 5944 (time 1778925830.04 total 80.07 last 69.45) - open/close 6609 (time 1778925840.04 total 90.08 last 66.46) - open/close 7276 (time 1778925850.06 total 100.10 last 66.60) - open/close 7909 (time 1778925860.07 total 110.11 last 63.24) - open/close 8587 (time 1778925870.10 total 120.14 last 67.59) - open/close 9320 (time 1778925880.10 total 130.14 last 73.28) total: 10000 open/close in 139.69 seconds: 71.59 ops/second - unlinked 0 (time 1778925894 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.203.157@tcp sys.timeout=20 start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local oleg357-server: modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 PASS 69 (2650s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 06:08:56 (1778926136) oleg357-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 oleg357-server: oleg357-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions oleg357-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg357-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg357-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) oleg357-server: oleg357-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg357-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg357-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg357-server: oleg357-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg357-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg357-server SKIP 111 (220s) Stopping clients: oleg357-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg357-client.virtnet /mnt/lustre2 (opts:-f) oleg357-server: oleg357-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing load_modules_local oleg357-server: Loading modules from /home/green/git/lustre-release/lustre oleg357-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg357-server: detected 4 online CPUs by sysfs oleg357-server: MODOPTS_LIBCFS= oleg357-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3432 sec ========== 06:13:42 (1778926422) === conf-sanity: start cleanup 06:13:43 (1778926423) === === conf-sanity: finish cleanup 06:13:47 (1778926427) ===