-----============= acceptance-small: conf-sanity ============----- Sun Mar 15 15:57:54 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg420-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg420-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg420-server oleg420-server: oleg420-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg420-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg420-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server == conf-sanity test 45: long unlink handling in ptlrpcd == 16:01:55 (1773604915) start mds service on oleg420-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg420-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg420-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg420-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server unloading modules via unload_modules_local on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local oleg420-server: modules unloaded. pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 PASS 45 (309s) == conf-sanity test 69: replace an OST with the same index ========================================================== 16:07:04 (1773605224) start mds service on oleg420-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg420-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg420-server: mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg420-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 749 (time 1773605327.27 total 10.02 last 74.77) - open/close 1393 (time 1773605337.28 total 20.03 last 64.33) - open/close 2011 (time 1773605347.29 total 30.04 last 61.74) - open/close 2729 (time 1773605357.30 total 40.04 last 71.75) - open/close 3487 (time 1773605367.31 total 50.06 last 75.71) - open/close 4262 (time 1773605377.31 total 60.06 last 77.49) - open/close 4926 (time 1773605387.32 total 70.07 last 66.32) - open/close 5600 (time 1773605397.32 total 80.07 last 67.39) - open/close 6338 (time 1773605407.33 total 90.08 last 73.74) - open/close 7086 (time 1773605417.34 total 100.09 last 74.73) - open/close 7697 (time 1773605427.34 total 110.09 last 61.09) - open/close 8431 (time 1773605437.35 total 120.10 last 73.34) - open/close 9118 (time 1773605447.36 total 130.11 last 68.61) - open/close 9897 (time 1773605457.37 total 140.12 last 77.82) total: 10000 open/close in 141.51 seconds: 70.67 ops/second - unlinked 0 (time 1773605463 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 808 (time 1773605569.70 total 10.00 last 80.78) - open/close 1434 (time 1773605579.71 total 20.01 last 62.56) - open/close 2106 (time 1773605589.72 total 30.03 last 67.08) - open/close 2750 (time 1773605599.74 total 40.05 last 64.28) - open/close 3476 (time 1773605609.75 total 50.05 last 72.58) - open/close 4096 (time 1773605619.77 total 60.07 last 61.87) - open/close 4679 (time 1773605629.78 total 70.08 last 58.25) - open/close 5348 (time 1773605639.79 total 80.09 last 66.81) - open/close 6051 (time 1773605649.79 total 90.09 last 70.30) - open/close 6622 (time 1773605659.79 total 100.10 last 57.09) - open/close 7293 (time 1773605669.82 total 110.12 last 66.91) - open/close 7882 (time 1773605679.82 total 120.13 last 58.88) - open/close 8452 (time 1773605689.84 total 130.14 last 56.92) - open/close 9123 (time 1773605699.85 total 140.15 last 67.02) - open/close 9785 (time 1773605709.86 total 150.16 last 66.15) total: 10000 open/close in 153.29 seconds: 65.23 ops/second - unlinked 0 (time 1773605718 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 608 (time 1773605815.67 total 10.02 last 60.67) - open/close 1220 (time 1773605825.68 total 20.03 last 61.14) - open/close 1939 (time 1773605835.69 total 30.04 last 71.86) - open/close 2555 (time 1773605845.70 total 40.05 last 61.53) - open/close 3148 (time 1773605855.70 total 50.06 last 59.27) - open/close 3723 (time 1773605865.71 total 60.06 last 57.46) - open/close 4258 (time 1773605875.75 total 70.10 last 53.28) - open/close 4896 (time 1773605885.77 total 80.12 last 63.68) - open/close 5585 (time 1773605895.77 total 90.13 last 68.88) - open/close 6255 (time 1773605905.78 total 100.13 last 66.95) - open/close 6864 (time 1773605915.79 total 110.14 last 60.85) - open/close 7523 (time 1773605925.80 total 120.15 last 65.84) - open/close 8037 (time 1773605935.81 total 130.16 last 51.35) - open/close 8453 (time 1773605945.81 total 140.16 last 41.60) - open/close 9000 (time 1773605955.81 total 150.17 last 54.68) - open/close 9562 (time 1773605965.82 total 160.17 last 56.18) total: 10000 open/close in 167.36 seconds: 59.75 ops/second - unlinked 0 (time 1773605979 ; total 0 ; last 0) total: 10000 unlinks in 101 seconds: 99.009903 unlinks/second - open/close 649 (time 1773606095.82 total 10.01 last 64.83) - open/close 1384 (time 1773606105.84 total 20.02 last 73.42) - open/close 2184 (time 1773606115.84 total 30.02 last 80.00) - open/close 2964 (time 1773606125.84 total 40.03 last 77.93) - open/close 3781 (time 1773606135.86 total 50.05 last 81.56) - open/close 4496 (time 1773606145.87 total 60.05 last 71.46) - open/close 5129 (time 1773606155.87 total 70.06 last 63.26) - open/close 5665 (time 1773606165.88 total 80.07 last 53.55) - open/close 6248 (time 1773606175.91 total 90.09 last 58.16) - open/close 6736 (time 1773606185.92 total 100.10 last 48.76) - open/close 7323 (time 1773606195.92 total 110.10 last 58.70) - open/close 7952 (time 1773606205.93 total 120.12 last 62.81) - open/close 8625 (time 1773606215.94 total 130.12 last 67.27) - open/close 9323 (time 1773606225.95 total 140.14 last 69.69) total: 10000 open/close in 149.91 seconds: 66.70 ops/second - unlinked 0 (time 1773606241 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second - open/close 623 (time 1773606342.97 total 10.01 last 62.25) - open/close 1277 (time 1773606352.98 total 20.02 last 65.34) - open/close 1823 (time 1773606362.99 total 30.03 last 54.53) - open/close 2221 (time 1773606373.01 total 40.05 last 39.71) - open/close 2639 (time 1773606383.02 total 50.06 last 41.79) - open/close 3261 (time 1773606393.02 total 60.06 last 62.19) - open/close 3994 (time 1773606403.03 total 70.07 last 73.19) - open/close 4636 (time 1773606413.04 total 80.08 last 64.17) - open/close 5180 (time 1773606423.05 total 90.09 last 54.34) - open/close 5746 (time 1773606433.05 total 100.09 last 56.59) - open/close 6239 (time 1773606443.08 total 110.12 last 49.15) - open/close 6950 (time 1773606453.10 total 120.14 last 70.96) - open/close 7632 (time 1773606463.10 total 130.14 last 68.19) - open/close 8323 (time 1773606473.10 total 140.14 last 69.10) - open/close 8930 (time 1773606483.11 total 150.15 last 60.66) - open/close 9572 (time 1773606493.13 total 160.17 last 64.08) total: 10000 open/close in 166.66 seconds: 60.00 ops/second - unlinked 0 (time 1773606504 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 535 (time 1773606606.30 total 10.01 last 53.45) - open/close 959 (time 1773606616.31 total 20.02 last 42.38) - open/close 1562 (time 1773606626.31 total 30.02 last 60.27) - open/close 2261 (time 1773606636.32 total 40.03 last 69.86) - open/close 2956 (time 1773606646.32 total 50.03 last 69.47) - open/close 3527 (time 1773606656.33 total 60.04 last 57.07) - open/close 4144 (time 1773606666.33 total 70.04 last 61.69) - open/close 4747 (time 1773606676.34 total 80.04 last 60.26) - open/close 5352 (time 1773606686.34 total 90.05 last 60.46) - open/close 6011 (time 1773606696.35 total 100.06 last 65.83) - open/close 6582 (time 1773606706.36 total 110.07 last 57.05) - open/close 7248 (time 1773606716.37 total 120.08 last 66.58) - open/close 7892 (time 1773606726.38 total 130.09 last 64.33) - open/close 8437 (time 1773606736.38 total 140.09 last 54.48) - open/close 8956 (time 1773606746.39 total 150.10 last 51.87) - open/close 9575 (time 1773606756.39 total 160.10 last 61.89) total: 10000 open/close in 166.63 seconds: 60.01 ops/second - unlinked 0 (time 1773606770 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 625 (time 1773606876.82 total 10.01 last 62.46) - open/close 1244 (time 1773606886.84 total 20.02 last 61.78) - open/close 1733 (time 1773606896.85 total 30.03 last 48.85) - open/close 2315 (time 1773606906.86 total 40.04 last 58.16) - open/close 2845 (time 1773606916.87 total 50.05 last 52.95) - open/close 3433 (time 1773606926.87 total 60.05 last 58.80) - open/close 4092 (time 1773606936.87 total 70.05 last 65.87) - open/close 4889 (time 1773606946.88 total 80.06 last 79.63) - open/close 5571 (time 1773606956.88 total 90.07 last 68.19) - open/close 6204 (time 1773606966.88 total 100.07 last 63.28) - open/close 6904 (time 1773606976.89 total 110.08 last 69.93) - open/close 7419 (time 1773606986.91 total 120.09 last 51.42) - open/close 7959 (time 1773606996.92 total 130.10 last 53.95) - open/close 8632 (time 1773607006.92 total 140.10 last 67.30) - open/close 9382 (time 1773607016.93 total 150.12 last 74.91) - open/close 9984 (time 1773607026.94 total 160.13 last 60.14) total: 10000 open/close in 160.37 seconds: 62.36 ops/second - unlinked 0 (time 1773607033 ; total 0 ; last 0) total: 10000 unlinks in 83 seconds: 120.481926 unlinks/second - open/close 686 (time 1773607131.61 total 10.00 last 68.60) - open/close 1442 (time 1773607141.62 total 20.01 last 75.54) - open/close 2023 (time 1773607151.63 total 30.02 last 58.05) - open/close 2740 (time 1773607161.64 total 40.02 last 71.65) - open/close 3307 (time 1773607171.64 total 50.03 last 56.65) - open/close 4092 (time 1773607181.65 total 60.04 last 78.47) - open/close 4602 (time 1773607191.65 total 70.04 last 50.97) - open/close 5156 (time 1773607201.65 total 80.04 last 55.40) - open/close 5653 (time 1773607211.67 total 90.06 last 49.60) - open/close 6202 (time 1773607221.68 total 100.07 last 54.87) - open/close 6780 (time 1773607231.70 total 110.09 last 57.70) - open/close 7458 (time 1773607241.72 total 120.11 last 67.64) - open/close 8057 (time 1773607251.72 total 130.11 last 59.90) - open/close 8692 (time 1773607261.72 total 140.11 last 63.48) - open/close 9321 (time 1773607271.74 total 150.13 last 62.83) total: 10000 open/close in 159.87 seconds: 62.55 ops/second - unlinked 0 (time 1773607287 ; total 0 ; last 0) total: 10000 unlinks in 89 seconds: 112.359550 unlinks/second - open/close 655 (time 1773607392.26 total 10.00 last 65.49) - open/close 1280 (time 1773607402.29 total 20.03 last 62.32) - open/close 1769 (time 1773607412.30 total 30.04 last 48.87) - open/close 2344 (time 1773607422.31 total 40.05 last 57.45) - open/close 3099 (time 1773607432.32 total 50.05 last 75.45) - open/close 3762 (time 1773607442.33 total 60.07 last 66.21) - open/close 4228 (time 1773607452.34 total 70.08 last 46.54) - open/close 4789 (time 1773607462.35 total 80.08 last 56.08) - open/close 5395 (time 1773607472.35 total 90.08 last 60.59) - open/close 5972 (time 1773607482.36 total 100.09 last 57.64) - open/close 6722 (time 1773607492.36 total 110.10 last 74.98) - open/close 7388 (time 1773607502.37 total 120.11 last 66.52) - open/close 7998 (time 1773607512.38 total 130.12 last 60.95) - open/close 8647 (time 1773607522.39 total 140.12 last 64.86) - open/close 9310 (time 1773607532.40 total 150.13 last 66.23) - open/close 9918 (time 1773607542.41 total 160.14 last 60.75) total: 10000 open/close in 161.21 seconds: 62.03 ops/second - unlinked 0 (time 1773607548 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 815 (time 1773607649.09 total 10.00 last 81.49) - open/close 1475 (time 1773607659.10 total 20.01 last 65.92) - open/close 2143 (time 1773607669.11 total 30.02 last 66.73) - open/close 2767 (time 1773607679.12 total 40.03 last 62.38) - open/close 3358 (time 1773607689.12 total 50.03 last 59.06) - open/close 4147 (time 1773607699.13 total 60.04 last 78.85) - open/close 4904 (time 1773607709.13 total 70.05 last 75.64) - open/close 5672 (time 1773607719.15 total 80.06 last 76.70) - open/close 6492 (time 1773607729.15 total 90.06 last 82.00) - open/close 7173 (time 1773607739.16 total 100.07 last 68.00) - open/close 7887 (time 1773607749.16 total 110.07 last 71.39) - open/close 8574 (time 1773607759.17 total 120.09 last 68.63) - open/close 9252 (time 1773607769.17 total 130.09 last 67.80) - open/close 9923 (time 1773607779.20 total 140.11 last 66.92) total: 10000 open/close in 141.17 seconds: 70.84 ops/second - unlinked 0 (time 1773607785 ; total 0 ; last 0) total: 10000 unlinks in 86 seconds: 116.279068 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.204.120@tcp sys.timeout=20 start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg420-client.virtnet: -o user_xattr,flock 192.168.204.120@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg420-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server unloading modules via unload_modules_local on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing unload_modules_local oleg420-server: modules unloaded. pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-client: ssh exited with exit code 2 pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 PASS 69 (2817s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 16:54:02 (1773608042) oleg420-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg420-client: oleg420-server: ssh exited with exit code 2 oleg420-server: oleg420-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions oleg420-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg420-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg420-server: mdt/mdt options: 'mdt_enable_flr_ec=1 mdt_enable_flr_immed_mirror=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) oleg420-server: oleg420-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg420-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg420-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg420-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg420-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg420-server: oleg420-server.virtnet: executing set_default_debug -1 all pdsh@oleg420-client: oleg420-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg420-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg420-server: oleg420-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg420-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg420-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg420-server stop mds service on oleg420-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg420-server SKIP 111 (252s) Stopping clients: oleg420-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg420-client.virtnet /mnt/lustre2 (opts:-f) oleg420-server: oleg420-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg420-server' oleg420-server: oleg420-server.virtnet: executing load_modules_local oleg420-server: Loading modules from /home/green/git/lustre-release/lustre oleg420-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1040: echo: write error: Device or resource busy oleg420-server: detected 4 online CPUs by sysfs oleg420-server: MODOPTS_LIBCFS= oleg420-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3687 sec ========== 16:59:23 (1773608363) === conf-sanity: start cleanup 16:59:25 (1773608365) === === conf-sanity: finish cleanup 16:59:29 (1773608369) ===