-----============= acceptance-small: conf-sanity ============----- Tue Apr 1 04:48:25 EDT 2025 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg640-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball Stopping clients: oleg640-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg640-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg640-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg640-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg640-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg640-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg640-server oleg640-server: oleg640-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg640-server' oleg640-server: oleg640-server.virtnet: executing load_modules_local oleg640-server: Loading modules from /home/green/git/lustre-release/lustre oleg640-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg640-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg640-server: detected 4 online CPUs by sysfs oleg640-server: MODOPTS_LIBCFS= oleg640-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg640-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg640-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg640-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg640-server: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg640-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg640-server: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg640-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg640-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg640-server stop mds service on oleg640-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg640-server stop mds service on oleg640-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg640-server == conf-sanity test 45: long unlink handling in ptlrpcd == 04:53:17 (1743497597) start mds service on oleg640-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg640-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg640-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0000 oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg640-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg640-server stop mds service on oleg640-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg640-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg640-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg640-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg640-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg640-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg640-server stop mds service on oleg640-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg640-server stop mds service on oleg640-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg640-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg640-server' oleg640-server: oleg640-server.virtnet: executing unload_modules_local oleg640-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg640-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg640-client: oleg640-client: ssh exited with exit code 2 pdsh@oleg640-client: oleg640-server: ssh exited with exit code 2 pdsh@oleg640-client: oleg640-server: ssh exited with exit code 2 pdsh@oleg640-client: oleg640-client: ssh exited with exit code 2 PASS 45 (331s) == conf-sanity test 69: replace an OST with the same index ========================================================== 04:58:49 (1743497929) start mds service on oleg640-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg640-server' oleg640-server: oleg640-server.virtnet: executing load_modules_local oleg640-server: Loading modules from /home/green/git/lustre-release/lustre oleg640-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg640-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg640-server: detected 4 online CPUs by sysfs oleg640-server: MODOPTS_LIBCFS= oleg640-server: Force libcfs to create 2 CPU partitions oleg640-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg640-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg640-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg640-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0000 oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 666 (time 1743498068.35 total 10.02 last 66.49) - open/close 1320 (time 1743498078.37 total 20.03 last 65.28) - open/close 2001 (time 1743498088.38 total 30.05 last 68.03) - open/close 2673 (time 1743498098.38 total 40.05 last 67.18) - open/close 3350 (time 1743498108.40 total 50.06 last 67.59) - open/close 4002 (time 1743498118.40 total 60.07 last 65.17) - open/close 4689 (time 1743498128.43 total 70.09 last 68.53) - open/close 5363 (time 1743498138.44 total 80.11 last 67.32) - open/close 6019 (time 1743498148.45 total 90.11 last 65.55) - open/close 6672 (time 1743498158.46 total 100.13 last 65.22) - open/close 7360 (time 1743498168.47 total 110.13 last 68.75) - open/close 8005 (time 1743498178.48 total 120.14 last 64.43) - open/close 8659 (time 1743498188.48 total 130.15 last 65.35) - open/close 9331 (time 1743498198.50 total 140.17 last 67.09) total: 10000 open/close in 150.16 seconds: 66.59 ops/second - unlinked 0 (time 1743498214 ; total 0 ; last 0) total: 10000 unlinks in 90 seconds: 111.111115 unlinks/second - open/close 664 (time 1743498320.56 total 10.02 last 66.30) - open/close 1313 (time 1743498330.57 total 20.03 last 64.82) - open/close 1984 (time 1743498340.58 total 30.03 last 67.08) - open/close 2631 (time 1743498350.58 total 40.03 last 64.70) - open/close 3273 (time 1743498360.59 total 50.05 last 64.10) - open/close 3934 (time 1743498370.59 total 60.05 last 66.09) - open/close 4581 (time 1743498380.59 total 70.05 last 64.69) - open/close 5231 (time 1743498390.60 total 80.06 last 64.94) - open/close 5918 (time 1743498400.61 total 90.07 last 68.65) - open/close 6587 (time 1743498410.62 total 100.07 last 66.86) - open/close 7285 (time 1743498420.63 total 110.08 last 69.74) - open/close 7966 (time 1743498430.63 total 120.08 last 68.10) - open/close 8637 (time 1743498440.63 total 130.09 last 67.06) - open/close 9324 (time 1743498450.64 total 140.09 last 68.67) - open/close 9982 (time 1743498460.65 total 150.10 last 65.73) total: 10000 open/close in 150.33 seconds: 66.52 ops/second - unlinked 0 (time 1743498466 ; total 0 ; last 0) total: 10000 unlinks in 91 seconds: 109.890106 unlinks/second - open/close 668 (time 1743498573.85 total 10.02 last 66.66) - open/close 1317 (time 1743498583.85 total 20.03 last 64.84) - open/close 1961 (time 1743498593.86 total 30.04 last 64.36) - open/close 2636 (time 1743498603.86 total 40.04 last 67.49) - open/close 3271 (time 1743498613.87 total 50.05 last 63.44) - open/close 3949 (time 1743498623.88 total 60.05 last 67.75) - open/close 4602 (time 1743498633.89 total 70.07 last 65.22) - open/close 5268 (time 1743498643.89 total 80.07 last 66.60) - open/close 5931 (time 1743498653.90 total 90.07 last 66.25) - open/close 6566 (time 1743498663.91 total 100.08 last 63.46) - open/close 7220 (time 1743498673.93 total 110.10 last 65.26) - open/close 7863 (time 1743498683.93 total 120.11 last 64.24) - open/close 8532 (time 1743498693.94 total 130.12 last 66.85) - open/close 9164 (time 1743498703.95 total 140.13 last 63.15) - open/close 9824 (time 1743498713.95 total 150.13 last 65.99) total: 10000 open/close in 152.80 seconds: 65.44 ops/second - unlinked 0 (time 1743498722 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 680 (time 1743498822.62 total 10.01 last 67.95) - open/close 1347 (time 1743498832.62 total 20.01 last 66.66) - open/close 2057 (time 1743498842.64 total 30.02 last 70.92) - open/close 2710 (time 1743498852.64 total 40.03 last 65.29) - open/close 3399 (time 1743498862.65 total 50.03 last 68.85) - open/close 4075 (time 1743498872.65 total 60.04 last 67.58) - open/close 4777 (time 1743498882.66 total 70.05 last 70.10) - open/close 5457 (time 1743498892.66 total 80.05 last 67.99) - open/close 6173 (time 1743498902.67 total 90.06 last 71.53) - open/close 6901 (time 1743498912.68 total 100.07 last 72.72) - open/close 7627 (time 1743498922.69 total 110.08 last 72.53) - open/close 8336 (time 1743498932.70 total 120.09 last 70.83) - open/close 9050 (time 1743498942.70 total 130.09 last 71.39) - open/close 9763 (time 1743498952.72 total 140.11 last 71.20) total: 10000 open/close in 143.24 seconds: 69.81 ops/second - unlinked 0 (time 1743498961 ; total 0 ; last 0) total: 10000 unlinks in 84 seconds: 119.047623 unlinks/second - open/close 683 (time 1743499061.46 total 10.00 last 68.28) - open/close 1345 (time 1743499071.46 total 20.00 last 66.19) - open/close 2020 (time 1743499081.47 total 30.01 last 67.46) - open/close 2682 (time 1743499091.48 total 40.02 last 66.12) - open/close 3382 (time 1743499101.49 total 50.03 last 69.95) - open/close 4070 (time 1743499111.50 total 60.04 last 68.73) - open/close 4787 (time 1743499121.50 total 70.05 last 71.64) - open/close 5458 (time 1743499131.51 total 80.05 last 67.07) - open/close 6141 (time 1743499141.52 total 90.06 last 68.25) - open/close 6855 (time 1743499151.52 total 100.07 last 71.35) - open/close 7547 (time 1743499161.53 total 110.07 last 69.17) - open/close 8227 (time 1743499171.53 total 120.07 last 68.00) - open/close 8930 (time 1743499181.53 total 130.07 last 70.28) - open/close 9682 (time 1743499191.53 total 140.08 last 75.18) total: 10000 open/close in 144.21 seconds: 69.34 ops/second - unlinked 0 (time 1743499200 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second - open/close 756 (time 1743499295.54 total 10.01 last 75.53) - open/close 1472 (time 1743499305.55 total 20.02 last 71.55) - open/close 2181 (time 1743499315.55 total 30.02 last 70.86) - open/close 2923 (time 1743499325.56 total 40.02 last 74.18) - open/close 3657 (time 1743499335.57 total 50.03 last 73.33) - open/close 4403 (time 1743499345.57 total 60.04 last 74.55) - open/close 5154 (time 1743499355.58 total 70.05 last 75.05) - open/close 5890 (time 1743499365.59 total 80.05 last 73.56) - open/close 6688 (time 1743499375.59 total 90.06 last 79.76) - open/close 7441 (time 1743499385.59 total 100.06 last 75.28) - open/close 8206 (time 1743499395.60 total 110.07 last 76.46) - open/close 8946 (time 1743499405.61 total 120.08 last 73.93) - open/close 9734 (time 1743499415.62 total 130.09 last 78.71) total: 10000 open/close in 133.37 seconds: 74.98 ops/second - unlinked 0 (time 1743499423 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second - open/close 761 (time 1743499513.01 total 10.00 last 76.07) - open/close 1511 (time 1743499523.02 total 20.01 last 74.93) - open/close 2329 (time 1743499533.03 total 30.02 last 81.73) - open/close 3093 (time 1743499543.03 total 40.03 last 76.34) - open/close 3844 (time 1743499553.05 total 50.04 last 75.00) - open/close 4644 (time 1743499563.05 total 60.05 last 79.95) - open/close 5454 (time 1743499573.06 total 70.06 last 80.95) - open/close 6298 (time 1743499583.06 total 80.06 last 84.36) - open/close 7067 (time 1743499593.07 total 90.06 last 76.88) - open/close 7869 (time 1743499603.07 total 100.07 last 80.16) - open/close 8632 (time 1743499613.09 total 110.08 last 76.19) - open/close 9378 (time 1743499623.09 total 120.09 last 74.57) total: 10000 open/close in 128.69 seconds: 77.70 ops/second - unlinked 0 (time 1743499636 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second - open/close 886 (time 1743499727.53 total 10.03 last 88.30) - open/close 1643 (time 1743499737.54 total 20.04 last 75.64) - open/close 2394 (time 1743499747.55 total 30.05 last 75.03) - open/close 3204 (time 1743499757.55 total 40.05 last 81.00) - open/close 3889 (time 1743499767.56 total 50.06 last 68.43) - open/close 4606 (time 1743499777.56 total 60.06 last 71.70) - open/close 5407 (time 1743499787.57 total 70.07 last 80.00) - open/close 6153 (time 1743499797.58 total 80.08 last 74.54) - open/close 6970 (time 1743499807.59 total 90.09 last 81.64) - open/close 7786 (time 1743499817.59 total 100.10 last 81.53) - open/close 8521 (time 1743499827.61 total 110.11 last 73.39) - open/close 9239 (time 1743499837.62 total 120.12 last 71.72) - open/close 9978 (time 1743499847.63 total 130.13 last 73.85) total: 10000 open/close in 130.37 seconds: 76.70 ops/second - unlinked 0 (time 1743499853 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 705 (time 1743499947.91 total 10.01 last 70.45) - open/close 1383 (time 1743499957.91 total 20.01 last 67.76) - open/close 2074 (time 1743499967.93 total 30.03 last 69.02) - open/close 2838 (time 1743499977.93 total 40.03 last 76.35) - open/close 3632 (time 1743499987.94 total 50.04 last 79.34) - open/close 4410 (time 1743499997.94 total 60.04 last 77.77) - open/close 5174 (time 1743500007.96 total 70.06 last 76.27) - open/close 5929 (time 1743500017.97 total 80.07 last 75.40) - open/close 6629 (time 1743500027.99 total 90.09 last 69.91) - open/close 7360 (time 1743500038.00 total 100.10 last 72.98) - open/close 8114 (time 1743500048.02 total 110.12 last 75.30) - open/close 8900 (time 1743500058.02 total 120.12 last 78.54) - open/close 9649 (time 1743500068.03 total 130.13 last 74.87) total: 10000 open/close in 134.27 seconds: 74.48 ops/second - unlinked 0 (time 1743500077 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 716 (time 1743500168.54 total 10.01 last 71.52) - open/close 1473 (time 1743500178.55 total 20.02 last 75.65) - open/close 2259 (time 1743500188.55 total 30.02 last 78.59) - open/close 2970 (time 1743500198.56 total 40.03 last 70.99) - open/close 3716 (time 1743500208.57 total 50.04 last 74.54) - open/close 4492 (time 1743500218.58 total 60.05 last 77.58) - open/close 5281 (time 1743500228.58 total 70.05 last 78.87) - open/close 6032 (time 1743500238.58 total 80.05 last 75.08) - open/close 6778 (time 1743500248.59 total 90.06 last 74.55) - open/close 7573 (time 1743500258.59 total 100.06 last 79.48) - open/close 8344 (time 1743500268.60 total 110.07 last 77.06) - open/close 9124 (time 1743500278.60 total 120.07 last 77.99) - open/close 9889 (time 1743500288.61 total 130.08 last 76.44) total: 10000 open/close in 131.53 seconds: 76.03 ops/second - unlinked 0 (time 1743500294 ; total 0 ; last 0) total: 10000 unlinks in 75 seconds: 133.333328 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg640-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg640-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg640-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.206.140@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg640-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Started lustre-OST0000 oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg640-server: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg640-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg640-server: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg640-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg640-client.virtnet: -o user_xattr,flock 192.168.206.140@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg640-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg640-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg640-server stop mds service on oleg640-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg640-server stop mds service on oleg640-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg640-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg640-server' oleg640-server: oleg640-server.virtnet: executing unload_modules_local oleg640-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg640-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg640-client: oleg640-client: ssh exited with exit code 2 pdsh@oleg640-client: oleg640-server: ssh exited with exit code 2 pdsh@oleg640-client: oleg640-server: ssh exited with exit code 2 pdsh@oleg640-client: oleg640-client: ssh exited with exit code 2 PASS 69 (2539s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 05:41:08 (1743500468) oleg640-server: debugfs 1.47.1-wc2 (08-Nov-2024) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg640-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg640-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg640-client: oleg640-server: ssh exited with exit code 2 oleg640-server: oleg640-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg640-server' oleg640-server: oleg640-server.virtnet: executing load_modules_local oleg640-server: Loading modules from /home/green/git/lustre-release/lustre oleg640-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg640-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg640-server: detected 4 online CPUs by sysfs oleg640-server: MODOPTS_LIBCFS= oleg640-server: Force libcfs to create 2 CPU partitions oleg640-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg640-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg640-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg640-client.virtnet /mnt/lustre2 (opts:-f) oleg640-server: oleg640-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg640-server' oleg640-server: oleg640-server.virtnet: executing load_modules_local oleg640-server: Loading modules from /home/green/git/lustre-release/lustre oleg640-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg640-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg640-server: detected 4 online CPUs by sysfs oleg640-server: MODOPTS_LIBCFS= oleg640-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg640-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg640-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg640-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg640-server: oleg640-server.virtnet: executing set_default_debug -1 all pdsh@oleg640-client: oleg640-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg640-server: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg640-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg640-server: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg640-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg640-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg640-server stop mds service on oleg640-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg640-server stop mds service on oleg640-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg640-server SKIP 111 (201s) Stopping clients: oleg640-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg640-client.virtnet /mnt/lustre2 (opts:-f) oleg640-server: oleg640-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg640-server' oleg640-server: oleg640-server.virtnet: executing load_modules_local oleg640-server: Loading modules from /home/green/git/lustre-release/lustre oleg640-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1001: echo: write error: Device or resource busy oleg640-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1002: echo: write error: Device or resource busy oleg640-server: detected 4 online CPUs by sysfs oleg640-server: MODOPTS_LIBCFS= oleg640-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3422 sec ========== 05:45:31 (1743500731) === conf-sanity: start cleanup 05:45:34 (1743500734) === === conf-sanity: finish cleanup 05:45:38 (1743500738) ===