-----============= acceptance-small: conf-sanity ============----- Thu Jun 11 19:31:00 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg419-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg419-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg419-server oleg419-server: oleg419-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg419-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg419-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg419-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server == conf-sanity test 45: long unlink handling in ptlrpcd == 19:34:28 (1781220868) start mds service on oleg419-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg419-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg419-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg419-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg419-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local oleg419-server: modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 PASS 45 (263s) == conf-sanity test 69: replace an OST with the same index ========================================================== 19:38:52 (1781221132) start mds service on oleg419-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg419-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg419-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg419-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49503 inodes available. Want 99872. rc=0 - open/close 697 (time 1781221239.83 total 10.00 last 69.69) - open/close 1346 (time 1781221249.84 total 20.01 last 64.84) - open/close 2039 (time 1781221259.85 total 30.02 last 69.24) - open/close 2704 (time 1781221269.87 total 40.04 last 66.37) - open/close 3403 (time 1781221279.87 total 50.04 last 69.88) - open/close 4065 (time 1781221289.88 total 60.05 last 66.15) - open/close 4816 (time 1781221299.89 total 70.06 last 75.03) - open/close 5511 (time 1781221309.89 total 80.06 last 69.48) - open/close 6167 (time 1781221319.89 total 90.07 last 65.56) - open/close 6905 (time 1781221329.90 total 100.07 last 73.74) - open/close 7629 (time 1781221339.91 total 110.09 last 72.32) - open/close 8329 (time 1781221349.91 total 120.09 last 69.99) - open/close 9057 (time 1781221359.92 total 130.09 last 72.77) - open/close 9692 (time 1781221369.94 total 140.11 last 63.40) total: 10000 open/close in 144.40 seconds: 69.25 ops/second - unlinked 0 (time 1781221379 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 714 (time 1781221485.39 total 10.01 last 71.34) - open/close 1351 (time 1781221495.41 total 20.03 last 63.57) - open/close 2097 (time 1781221505.41 total 30.03 last 74.55) - open/close 2825 (time 1781221515.42 total 40.04 last 72.75) - open/close 3537 (time 1781221525.42 total 50.04 last 71.18) - open/close 4210 (time 1781221535.42 total 60.04 last 67.30) - open/close 4930 (time 1781221545.44 total 70.06 last 71.89) - open/close 5605 (time 1781221555.44 total 80.07 last 67.47) - open/close 6260 (time 1781221565.45 total 90.07 last 65.48) - open/close 6731 (time 1781221575.45 total 100.07 last 47.08) - open/close 7286 (time 1781221585.48 total 110.10 last 55.36) - open/close 7984 (time 1781221595.49 total 120.11 last 69.72) - open/close 8579 (time 1781221605.49 total 130.11 last 59.47) - open/close 9243 (time 1781221615.50 total 140.12 last 66.38) - open/close 9947 (time 1781221625.50 total 150.12 last 70.36) total: 10000 open/close in 150.91 seconds: 66.26 ops/second - unlinked 0 (time 1781221630 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 772 (time 1781221722.07 total 10.01 last 77.11) - open/close 1516 (time 1781221732.09 total 20.03 last 74.28) - open/close 2188 (time 1781221742.09 total 30.03 last 67.18) - open/close 2901 (time 1781221752.09 total 40.03 last 71.29) - open/close 3472 (time 1781221762.11 total 50.05 last 57.01) - open/close 4009 (time 1781221772.14 total 60.07 last 53.56) - open/close 4665 (time 1781221782.16 total 70.09 last 65.46) - open/close 5337 (time 1781221792.16 total 80.10 last 67.20) - open/close 5932 (time 1781221802.17 total 90.11 last 59.42) - open/close 6549 (time 1781221812.18 total 100.12 last 61.64) - open/close 7122 (time 1781221822.18 total 110.12 last 57.29) - open/close 7654 (time 1781221832.20 total 120.13 last 53.13) - open/close 8269 (time 1781221842.22 total 130.15 last 61.38) - open/close 8879 (time 1781221852.22 total 140.16 last 60.96) - open/close 9517 (time 1781221862.22 total 150.16 last 63.79) total: 10000 open/close in 157.14 seconds: 63.64 ops/second - unlinked 0 (time 1781221873 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 483 (time 1781221980.76 total 10.00 last 48.29) - open/close 913 (time 1781221990.77 total 20.01 last 42.97) - open/close 1391 (time 1781222000.78 total 30.01 last 47.78) - open/close 1949 (time 1781222010.78 total 40.02 last 55.76) - open/close 2659 (time 1781222020.79 total 50.03 last 70.96) - open/close 3250 (time 1781222030.79 total 60.03 last 59.08) - open/close 3884 (time 1781222040.80 total 70.04 last 63.36) - open/close 4477 (time 1781222050.81 total 80.05 last 59.24) - open/close 5131 (time 1781222060.82 total 90.05 last 65.34) - open/close 5695 (time 1781222070.84 total 100.08 last 56.26) - open/close 6237 (time 1781222080.86 total 110.10 last 54.08) - open/close 6752 (time 1781222090.88 total 120.12 last 51.40) - open/close 7384 (time 1781222100.89 total 130.13 last 63.16) - open/close 7997 (time 1781222110.90 total 140.14 last 61.22) - open/close 8634 (time 1781222120.92 total 150.15 last 63.62) - open/close 9229 (time 1781222130.92 total 160.15 last 59.50) - open/close 9828 (time 1781222140.93 total 170.17 last 59.82) total: 10000 open/close in 172.82 seconds: 57.86 ops/second - unlinked 0 (time 1781222148 ; total 0 ; last 0) total: 10000 unlinks in 93 seconds: 107.526878 unlinks/second - open/close 552 (time 1781222258.12 total 10.01 last 55.14) - open/close 1050 (time 1781222268.13 total 20.03 last 49.72) - open/close 1499 (time 1781222278.14 total 30.03 last 44.87) - open/close 2012 (time 1781222288.16 total 40.06 last 51.19) - open/close 2457 (time 1781222298.18 total 50.08 last 44.40) - open/close 2905 (time 1781222308.21 total 60.11 last 44.67) - open/close 3443 (time 1781222318.21 total 70.11 last 53.79) - open/close 4022 (time 1781222328.22 total 80.12 last 57.85) - open/close 4594 (time 1781222338.23 total 90.12 last 57.18) - open/close 5199 (time 1781222348.23 total 100.13 last 60.47) - open/close 5823 (time 1781222358.25 total 110.14 last 62.30) - open/close 6361 (time 1781222368.26 total 120.16 last 53.73) - open/close 6908 (time 1781222378.27 total 130.17 last 54.66) - open/close 7455 (time 1781222388.28 total 140.17 last 54.66) - open/close 8051 (time 1781222398.29 total 150.19 last 59.52) - open/close 8562 (time 1781222408.31 total 160.20 last 51.03) - open/close 9084 (time 1781222418.31 total 170.21 last 52.17) - open/close 9644 (time 1781222428.33 total 180.22 last 55.91) total: 10000 open/close in 186.11 seconds: 53.73 ops/second - unlinked 0 (time 1781222439 ; total 0 ; last 0) total: 10000 unlinks in 110 seconds: 90.909088 unlinks/second - open/close 490 (time 1781222565.93 total 10.01 last 48.97) - open/close 1070 (time 1781222575.94 total 20.01 last 57.95) - open/close 1699 (time 1781222585.96 total 30.03 last 62.80) - open/close 2314 (time 1781222595.97 total 40.04 last 61.43) - open/close 2901 (time 1781222605.98 total 50.05 last 58.65) - open/close 3425 (time 1781222615.99 total 60.06 last 52.33) - open/close 4028 (time 1781222626.00 total 70.08 last 60.23) - open/close 4658 (time 1781222636.01 total 80.08 last 62.96) - open/close 5260 (time 1781222646.02 total 90.09 last 60.15) - open/close 5849 (time 1781222656.02 total 100.10 last 58.86) - open/close 6415 (time 1781222666.03 total 110.10 last 56.57) - open/close 7073 (time 1781222676.03 total 120.10 last 65.79) - open/close 7762 (time 1781222686.04 total 130.11 last 68.84) - open/close 8384 (time 1781222696.05 total 140.13 last 62.11) - open/close 8969 (time 1781222706.06 total 150.13 last 58.48) - open/close 9650 (time 1781222716.06 total 160.13 last 68.07) total: 10000 open/close in 166.08 seconds: 60.21 ops/second - unlinked 0 (time 1781222726 ; total 0 ; last 0) total: 10000 unlinks in 92 seconds: 108.695656 unlinks/second - open/close 571 (time 1781222833.92 total 10.01 last 57.02) - open/close 1196 (time 1781222843.93 total 20.02 last 62.44) - open/close 1817 (time 1781222853.95 total 30.05 last 61.95) - open/close 2461 (time 1781222863.95 total 40.05 last 64.39) - open/close 3093 (time 1781222873.96 total 50.06 last 63.11) - open/close 3752 (time 1781222883.98 total 60.07 last 65.83) - open/close 4423 (time 1781222893.99 total 70.09 last 67.03) - open/close 5029 (time 1781222903.99 total 80.09 last 60.57) - open/close 5647 (time 1781222914.00 total 90.09 last 61.77) - open/close 6316 (time 1781222924.00 total 100.10 last 66.86) - open/close 7021 (time 1781222934.01 total 110.11 last 70.45) - open/close 7656 (time 1781222944.01 total 120.11 last 63.50) - open/close 8263 (time 1781222954.03 total 130.13 last 60.59) - open/close 8911 (time 1781222964.03 total 140.13 last 64.79) - open/close 9549 (time 1781222974.03 total 150.13 last 63.77) total: 10000 open/close in 158.40 seconds: 63.13 ops/second - unlinked 0 (time 1781222987 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 600 (time 1781223093.41 total 10.00 last 59.99) - open/close 1176 (time 1781223103.41 total 20.01 last 57.57) - open/close 1782 (time 1781223113.43 total 30.02 last 60.51) - open/close 2311 (time 1781223123.44 total 40.03 last 52.84) - open/close 2884 (time 1781223133.44 total 50.04 last 57.28) - open/close 3445 (time 1781223143.46 total 60.05 last 56.01) - open/close 4111 (time 1781223153.48 total 70.07 last 66.49) - open/close 4759 (time 1781223163.48 total 80.07 last 64.79) - open/close 5515 (time 1781223173.48 total 90.08 last 75.56) - open/close 6276 (time 1781223183.49 total 100.08 last 76.06) - open/close 6970 (time 1781223193.50 total 110.09 last 69.36) - open/close 7689 (time 1781223203.50 total 120.09 last 71.86) - open/close 8323 (time 1781223213.52 total 130.11 last 63.31) - open/close 8965 (time 1781223223.52 total 140.11 last 64.20) - open/close 9537 (time 1781223233.53 total 150.12 last 57.13) total: 10000 open/close in 158.64 seconds: 63.03 ops/second - unlinked 0 (time 1781223247 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 686 (time 1781223350.74 total 10.01 last 68.53) - open/close 1391 (time 1781223360.74 total 20.01 last 70.48) - open/close 2049 (time 1781223370.75 total 30.03 last 65.71) - open/close 2725 (time 1781223380.76 total 40.04 last 67.53) - open/close 3326 (time 1781223390.79 total 50.06 last 59.95) - open/close 3948 (time 1781223400.80 total 60.07 last 62.13) - open/close 4594 (time 1781223410.81 total 70.08 last 64.53) - open/close 5166 (time 1781223420.82 total 80.10 last 57.12) - open/close 5724 (time 1781223430.83 total 90.10 last 55.79) - open/close 6226 (time 1781223440.84 total 100.11 last 50.16) - open/close 6889 (time 1781223450.84 total 110.12 last 66.25) - open/close 7474 (time 1781223460.86 total 120.13 last 58.43) - open/close 8123 (time 1781223470.86 total 130.13 last 64.90) - open/close 8850 (time 1781223480.86 total 140.13 last 72.67) - open/close 9513 (time 1781223490.87 total 150.14 last 66.23) total: 10000 open/close in 158.46 seconds: 63.11 ops/second - unlinked 0 (time 1781223505 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 504 (time 1781223607.14 total 10.00 last 50.39) - open/close 1033 (time 1781223617.15 total 20.01 last 52.85) - open/close 1568 (time 1781223627.16 total 30.02 last 53.45) - open/close 2142 (time 1781223637.17 total 40.03 last 57.32) - open/close 2666 (time 1781223647.18 total 50.04 last 52.35) - open/close 3292 (time 1781223657.18 total 60.05 last 62.56) - open/close 3987 (time 1781223667.19 total 70.06 last 69.44) - open/close 4625 (time 1781223677.20 total 80.07 last 63.74) - open/close 5261 (time 1781223687.22 total 90.08 last 63.50) - open/close 5859 (time 1781223697.23 total 100.09 last 59.75) - open/close 6557 (time 1781223707.23 total 110.09 last 69.79) - open/close 7197 (time 1781223717.23 total 120.10 last 63.96) - open/close 7858 (time 1781223727.25 total 130.12 last 65.97) - open/close 8514 (time 1781223737.27 total 140.13 last 65.50) - open/close 9104 (time 1781223747.27 total 150.14 last 59.00) - open/close 9733 (time 1781223757.27 total 160.14 last 62.88) total: 10000 open/close in 164.16 seconds: 60.92 ops/second - unlinked 0 (time 1781223766 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.119@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg419-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre On OST0, 20430 used inodes rc=0 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.119@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.119@tcp sys.timeout=20 Permanent disk data: Device: /dev/mapper/ost1_flakey Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.119@tcp sys.timeout=20 start ost1 service on oleg419-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local oleg419-server: modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 PASS 69 (2889s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 20:27:02 (1781224022) oleg419-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 oleg419-server: oleg419-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions oleg419-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg419-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg419-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) oleg419-server: oleg419-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg419-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg419-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg419-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg419-server: oleg419-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg419-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server SKIP 111 (239s) Stopping clients: oleg419-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg419-client.virtnet /mnt/lustre2 (opts:-f) oleg419-server: oleg419-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing load_modules_local oleg419-server: Loading modules from /home/green/git/lustre-release/lustre oleg419-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg419-server: detected 4 online CPUs by sysfs oleg419-server: MODOPTS_LIBCFS= oleg419-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3673 sec ========== 20:32:15 (1781224335) === conf-sanity: start cleanup 20:32:17 (1781224337) === === conf-sanity: finish cleanup 20:32:21 (1781224341) ===