-----============= acceptance-small: conf-sanity ============----- Thu Apr 23 21:56:35 EDT 2026 mgs: Rocky Linux release 8.10 (Green Obsidian) MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 mds1: Rocky Linux release 8.10 (Green Obsidian) MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky ost1: Rocky Linux release 8.10 (Green Obsidian) OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky client: Rocky Linux release 8.10 (Green Obsidian) CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg159-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg159-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg159-server oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg159-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server == conf-sanity test 45: long unlink handling in ptlrpcd == 22:06:38 (1776996398) start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg159-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg159-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 PASS 45 (610s) == conf-sanity test 69: replace an OST with the same index ========================================================== 22:16:50 (1776997010) start mds service on oleg159-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg159-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg159-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff On OST0, 49504 inodes available. Want 99872. rc=0 - open/close 206 (time 1776997317.14 total 10.06 last 20.48) - open/close 418 (time 1776997327.17 total 20.09 last 21.14) - open/close 605 (time 1776997337.20 total 30.12 last 18.64) - open/close 829 (time 1776997347.25 total 40.17 last 22.29) - open/close 1055 (time 1776997357.27 total 50.19 last 22.56) - open/close 1311 (time 1776997367.31 total 60.23 last 25.49) - open/close 1536 (time 1776997377.34 total 70.26 last 22.44) - open/close 1766 (time 1776997387.36 total 80.29 last 22.94) - open/close 1988 (time 1776997397.39 total 90.31 last 22.14) - open/close 2192 (time 1776997407.42 total 100.34 last 20.34) - open/close 2413 (time 1776997417.44 total 110.36 last 22.06) - open/close 2632 (time 1776997427.45 total 120.37 last 21.88) - open/close 2830 (time 1776997437.50 total 130.42 last 19.70) - open/close 3042 (time 1776997447.51 total 140.43 last 21.18) - open/close 3248 (time 1776997457.56 total 150.48 last 20.51) - open/close 3470 (time 1776997467.58 total 160.50 last 22.13) - open/close 3708 (time 1776997477.65 total 170.57 last 23.64) - open/close 3925 (time 1776997487.71 total 180.63 last 21.58) - open/close 4171 (time 1776997497.71 total 190.63 last 24.60) - open/close 4388 (time 1776997507.73 total 200.65 last 21.66) - open/close 4596 (time 1776997517.80 total 210.72 last 20.65) - open/close 4837 (time 1776997527.85 total 220.77 last 23.98) - open/close 5052 (time 1776997537.90 total 230.82 last 21.39) - open/close 5274 (time 1776997547.91 total 240.83 last 22.18) - open/close 5438 (time 1776997557.94 total 250.87 last 16.34) - open/close 5594 (time 1776997567.98 total 260.90 last 15.54) - open/close 5764 (time 1776997578.01 total 270.93 last 16.95) - open/close 5982 (time 1776997588.02 total 280.94 last 21.77) - open/close 6202 (time 1776997598.06 total 290.98 last 21.93) - open/close 6388 (time 1776997608.06 total 300.98 last 18.59) - open/close 6601 (time 1776997618.08 total 311.00 last 21.27) - open/close 6823 (time 1776997628.10 total 321.02 last 22.15) - open/close 7032 (time 1776997638.10 total 331.02 last 20.89) - open/close 7243 (time 1776997648.13 total 341.05 last 21.05) - open/close 7488 (time 1776997658.13 total 351.06 last 24.48) - open/close 7700 (time 1776997668.16 total 361.08 last 21.15) - open/close 7923 (time 1776997678.22 total 371.14 last 22.18) - open/close 8139 (time 1776997688.24 total 381.16 last 21.54) - open/close 8359 (time 1776997698.27 total 391.19 last 21.94) - open/close 8569 (time 1776997708.33 total 401.25 last 20.88) - open/close 8776 (time 1776997718.36 total 411.28 last 20.63) - open/close 8992 (time 1776997728.37 total 421.29 last 21.58) - open/close 9216 (time 1776997738.42 total 431.34 last 22.29) - open/close 9441 (time 1776997748.42 total 441.34 last 22.50) - open/close 9644 (time 1776997758.50 total 451.42 last 20.15) - open/close 9866 (time 1776997768.53 total 461.45 last 22.13) total: 10000 open/close in 467.41 seconds: 21.39 ops/second - unlinked 0 (time 1776997789 ; total 0 ; last 0) total: 10000 unlinks in 234 seconds: 42.735043 unlinks/second - open/close 222 (time 1776998045.71 total 10.01 last 22.18) - open/close 460 (time 1776998055.72 total 20.03 last 23.76) - open/close 651 (time 1776998065.74 total 30.04 last 19.07) - open/close 871 (time 1776998075.76 total 40.07 last 21.95) - open/close 1099 (time 1776998085.79 total 50.09 last 22.74) - open/close 1306 (time 1776998095.81 total 60.11 last 20.66) - open/close 1563 (time 1776998105.82 total 70.12 last 25.68) - open/close 1803 (time 1776998115.85 total 80.15 last 23.93) - open/close 2045 (time 1776998125.88 total 90.18 last 24.13) - open/close 2295 (time 1776998135.91 total 100.21 last 24.92) - open/close 2546 (time 1776998145.93 total 110.23 last 25.04) - open/close 2788 (time 1776998155.94 total 120.24 last 24.18) - open/close 3001 (time 1776998166.00 total 130.30 last 21.18) - open/close 3221 (time 1776998176.01 total 140.32 last 21.96) - open/close 3413 (time 1776998186.06 total 150.36 last 19.11) - open/close 3616 (time 1776998196.09 total 160.39 last 20.24) - open/close 3807 (time 1776998206.14 total 170.45 last 19.00) - open/close 4018 (time 1776998216.21 total 180.51 last 20.97) - open/close 4234 (time 1776998226.25 total 190.56 last 21.50) - open/close 4463 (time 1776998236.28 total 200.59 last 22.84) - open/close 4684 (time 1776998246.31 total 210.61 last 22.04) - open/close 4914 (time 1776998256.34 total 220.64 last 22.93) - open/close 5116 (time 1776998266.39 total 230.69 last 20.11) - open/close 5348 (time 1776998276.44 total 240.74 last 23.08) - open/close 5560 (time 1776998286.46 total 250.76 last 21.16) - open/close 5767 (time 1776998296.53 total 260.83 last 20.56) - open/close 5961 (time 1776998306.55 total 270.86 last 19.36) - open/close 6170 (time 1776998316.56 total 280.86 last 20.89) - open/close 6385 (time 1776998326.61 total 290.91 last 21.39) - open/close 6590 (time 1776998336.67 total 300.97 last 20.38) - open/close 6789 (time 1776998346.67 total 310.98 last 19.89) - open/close 6963 (time 1776998356.69 total 320.99 last 17.38) - open/close 7147 (time 1776998366.69 total 331.00 last 18.39) - open/close 7291 (time 1776998376.74 total 341.05 last 14.33) - open/close 7506 (time 1776998386.76 total 351.06 last 21.47) - open/close 7723 (time 1776998396.80 total 361.11 last 21.59) - open/close 7925 (time 1776998406.84 total 371.14 last 20.13) - open/close 8135 (time 1776998416.85 total 381.16 last 20.98) - open/close 8351 (time 1776998426.86 total 391.16 last 21.58) - open/close 8559 (time 1776998436.89 total 401.19 last 20.74) - open/close 8764 (time 1776998446.92 total 411.23 last 20.43) - open/close 8983 (time 1776998456.96 total 421.26 last 21.83) - open/close 9185 (time 1776998466.97 total 431.27 last 20.17) - open/close 9385 (time 1776998476.98 total 441.29 last 19.97) - open/close 9591 (time 1776998487.03 total 451.34 last 20.50) - open/close 9762 (time 1776998497.06 total 461.36 last 17.05) - open/close 9968 (time 1776998507.12 total 471.43 last 20.47) total: 10000 open/close in 473.16 seconds: 21.13 ops/second - unlinked 0 (time 1776998524 ; total 0 ; last 0) total: 10000 unlinks in 244 seconds: 40.983608 unlinks/second - open/close 173 (time 1776998793.41 total 10.00 last 17.30) - open/close 345 (time 1776998803.43 total 20.02 last 17.17) - open/close 560 (time 1776998813.44 total 30.04 last 21.46) - open/close 761 (time 1776998823.45 total 40.04 last 20.09) - open/close 971 (time 1776998833.47 total 50.06 last 20.96) - open/close 1193 (time 1776998843.49 total 60.09 last 22.14) - open/close 1399 (time 1776998853.53 total 70.12 last 20.53) - open/close 1618 (time 1776998863.55 total 80.14 last 21.86) - open/close 1855 (time 1776998873.56 total 90.15 last 23.67) - open/close 2109 (time 1776998883.58 total 100.18 last 25.34) - open/close 2320 (time 1776998893.63 total 110.22 last 21.01) - open/close 2524 (time 1776998903.63 total 120.23 last 20.39) - open/close 2722 (time 1776998913.65 total 130.24 last 19.76) - open/close 2896 (time 1776998923.65 total 140.25 last 17.40) - open/close 3104 (time 1776998933.67 total 150.26 last 20.77) - open/close 3308 (time 1776998943.70 total 160.29 last 20.34) - open/close 3510 (time 1776998953.72 total 170.31 last 20.16) - open/close 3689 (time 1776998963.73 total 180.32 last 17.89) - open/close 3879 (time 1776998973.76 total 190.35 last 18.94) - open/close 4058 (time 1776998983.76 total 200.36 last 17.88) - open/close 4265 (time 1776998993.79 total 210.38 last 20.66) - open/close 4447 (time 1776999003.82 total 220.41 last 18.14) - open/close 4644 (time 1776999013.87 total 230.47 last 19.60) - open/close 4802 (time 1776999023.96 total 240.56 last 15.66) - open/close 4971 (time 1776999033.98 total 250.57 last 16.87) - open/close 5128 (time 1776999043.99 total 260.59 last 15.68) - open/close 5315 (time 1776999054.04 total 270.63 last 18.62) - open/close 5522 (time 1776999064.06 total 280.65 last 20.67) - open/close 5735 (time 1776999074.12 total 290.72 last 21.16) - open/close 5936 (time 1776999084.14 total 300.73 last 20.07) - open/close 6130 (time 1776999094.16 total 310.75 last 19.35) - open/close 6354 (time 1776999104.16 total 320.75 last 22.40) - open/close 6546 (time 1776999114.20 total 330.80 last 19.12) - open/close 6743 (time 1776999124.25 total 340.84 last 19.62) - open/close 6949 (time 1776999134.25 total 350.85 last 20.58) - open/close 7152 (time 1776999144.35 total 360.94 last 20.11) - open/close 7384 (time 1776999154.39 total 370.99 last 23.10) - open/close 7590 (time 1776999164.40 total 380.99 last 20.59) - open/close 7810 (time 1776999174.42 total 391.01 last 21.96) - open/close 8016 (time 1776999184.46 total 401.06 last 20.51) - open/close 8223 (time 1776999194.50 total 411.09 last 20.63) - open/close 8432 (time 1776999204.52 total 421.11 last 20.86) - open/close 8657 (time 1776999214.53 total 431.13 last 22.46) - open/close 8849 (time 1776999224.55 total 441.14 last 19.18) - open/close 9063 (time 1776999234.58 total 451.17 last 21.34) - open/close 9274 (time 1776999244.62 total 461.22 last 21.00) - open/close 9514 (time 1776999254.63 total 471.22 last 23.99) - open/close 9713 (time 1776999264.65 total 481.25 last 19.85) - open/close 9928 (time 1776999274.67 total 491.27 last 21.45) total: 10000 open/close in 494.72 seconds: 20.21 ops/second - unlinked 0 (time 1776999291 ; total 0 ; last 0) total: 10000 unlinks in 241 seconds: 41.493774 unlinks/second - open/close 220 (time 1776999555.99 total 10.06 last 21.86) - open/close 439 (time 1776999566.01 total 20.09 last 21.85) - open/close 658 (time 1776999576.01 total 30.09 last 21.90) - open/close 867 (time 1776999586.03 total 40.10 last 20.87) - open/close 1086 (time 1776999596.04 total 50.12 last 21.87) - open/close 1301 (time 1776999606.07 total 60.15 last 21.44) - open/close 1512 (time 1776999616.11 total 70.19 last 21.02) - open/close 1742 (time 1776999626.11 total 80.19 last 23.00) - open/close 1964 (time 1776999636.16 total 90.24 last 22.08) - open/close 2163 (time 1776999646.23 total 100.31 last 19.77) - open/close 2359 (time 1776999656.25 total 110.32 last 19.56) - open/close 2544 (time 1776999666.27 total 120.35 last 18.45) - open/close 2757 (time 1776999676.30 total 130.38 last 21.24) - open/close 2950 (time 1776999686.32 total 140.40 last 19.26) - open/close 3161 (time 1776999696.33 total 150.41 last 21.08) - open/close 3426 (time 1776999706.37 total 160.45 last 26.39) - open/close 3629 (time 1776999716.41 total 170.49 last 20.22) - open/close 3844 (time 1776999726.49 total 180.57 last 21.33) - open/close 4049 (time 1776999736.51 total 190.58 last 20.47) - open/close 4266 (time 1776999746.52 total 200.60 last 21.67) - open/close 4496 (time 1776999756.53 total 210.60 last 22.99) - open/close 4714 (time 1776999766.53 total 220.61 last 21.79) - open/close 4955 (time 1776999776.57 total 230.65 last 24.00) - open/close 5217 (time 1776999786.61 total 240.69 last 26.10) - open/close 5440 (time 1776999796.66 total 250.74 last 22.19) - open/close 5643 (time 1776999806.67 total 260.75 last 20.28) - open/close 5883 (time 1776999816.70 total 270.77 last 23.95) - open/close 6098 (time 1776999826.70 total 280.78 last 21.49) - open/close 6292 (time 1776999836.70 total 290.78 last 19.39) - open/close 6523 (time 1776999846.70 total 300.78 last 23.10) - open/close 6738 (time 1776999856.73 total 310.80 last 21.45) - open/close 6950 (time 1776999866.73 total 320.81 last 21.19) - open/close 7168 (time 1776999876.76 total 330.83 last 21.74) - open/close 7387 (time 1776999886.80 total 340.87 last 21.82) - open/close 7621 (time 1776999896.81 total 350.89 last 23.36) - open/close 7869 (time 1776999906.82 total 360.89 last 24.79) - open/close 8087 (time 1776999916.83 total 370.90 last 21.78) - open/close 8327 (time 1776999926.85 total 380.92 last 23.95) - open/close 8562 (time 1776999936.87 total 390.95 last 23.44) - open/close 8813 (time 1776999946.89 total 400.96 last 25.06) - open/close 9031 (time 1776999956.90 total 410.97 last 21.78) - open/close 9246 (time 1776999966.93 total 421.01 last 21.42) - open/close 9467 (time 1776999976.94 total 431.01 last 22.08) - open/close 9688 (time 1776999986.99 total 441.06 last 21.99) - open/close 9921 (time 1776999997.01 total 451.08 last 23.25) total: 10000 open/close in 454.55 seconds: 22.00 ops/second - unlinked 0 (time 1777000015 ; total 0 ; last 0) total: 10000 unlinks in 227 seconds: 44.052864 unlinks/second - open/close 235 (time 1777000267.76 total 10.02 last 23.46) - open/close 451 (time 1777000277.79 total 20.05 last 21.54) - open/close 673 (time 1777000287.81 total 30.07 last 22.15) - open/close 895 (time 1777000297.84 total 40.10 last 22.14) - open/close 1108 (time 1777000307.88 total 50.13 last 21.23) - open/close 1330 (time 1777000317.91 total 60.17 last 22.13) - open/close 1556 (time 1777000327.92 total 70.18 last 22.57) - open/close 1765 (time 1777000337.98 total 80.24 last 20.78) - open/close 2012 (time 1777000347.99 total 90.25 last 24.67) - open/close 2228 (time 1777000358.02 total 100.28 last 21.54) - open/close 2453 (time 1777000368.04 total 110.29 last 22.47) - open/close 2673 (time 1777000378.06 total 120.31 last 21.96) - open/close 2889 (time 1777000388.08 total 130.34 last 21.55) - open/close 3114 (time 1777000398.09 total 140.35 last 22.47) - open/close 3350 (time 1777000408.11 total 150.36 last 23.57) - open/close 3594 (time 1777000418.15 total 160.41 last 24.28) - open/close 3865 (time 1777000428.17 total 170.43 last 27.04) - open/close 4111 (time 1777000438.19 total 180.45 last 24.55) - open/close 4344 (time 1777000448.21 total 190.46 last 23.27) - open/close 4567 (time 1777000458.24 total 200.49 last 22.23) - open/close 4788 (time 1777000468.26 total 210.51 last 22.06) - open/close 5006 (time 1777000478.35 total 220.61 last 21.59) - open/close 5212 (time 1777000488.38 total 230.63 last 20.55) - open/close 5431 (time 1777000498.42 total 240.68 last 21.79) - open/close 5659 (time 1777000508.48 total 250.74 last 22.67) - open/close 5886 (time 1777000518.51 total 260.77 last 22.62) - open/close 6102 (time 1777000528.53 total 270.79 last 21.56) - open/close 6315 (time 1777000538.53 total 280.79 last 21.30) - open/close 6539 (time 1777000548.58 total 290.84 last 22.30) - open/close 6758 (time 1777000558.59 total 300.85 last 21.87) - open/close 6993 (time 1777000568.64 total 310.89 last 23.40) - open/close 7232 (time 1777000578.65 total 320.90 last 23.88) - open/close 7466 (time 1777000588.66 total 330.92 last 23.37) - open/close 7679 (time 1777000598.73 total 340.99 last 21.14) - open/close 7896 (time 1777000608.74 total 351.00 last 21.69) - open/close 8105 (time 1777000618.75 total 361.00 last 20.88) - open/close 8336 (time 1777000628.76 total 371.02 last 23.07) - open/close 8573 (time 1777000638.77 total 381.03 last 23.67) - open/close 8811 (time 1777000648.82 total 391.08 last 23.69) - open/close 9020 (time 1777000658.86 total 401.12 last 20.81) - open/close 9252 (time 1777000668.90 total 411.15 last 23.12) - open/close 9475 (time 1777000678.92 total 421.18 last 22.24) - open/close 9705 (time 1777000688.93 total 431.19 last 22.98) - open/close 9920 (time 1777000698.94 total 441.20 last 21.48) total: 10000 open/close in 444.36 seconds: 22.50 ops/second - unlinked 0 (time 1777000715 ; total 0 ; last 0) total: 10000 unlinks in 232 seconds: 43.103447 unlinks/second - open/close 196 (time 1777000972.04 total 10.03 last 19.55) - open/close 433 (time 1777000982.06 total 20.05 last 23.64) - open/close 659 (time 1777000992.06 total 30.05 last 22.60) - open/close 866 (time 1777001002.07 total 40.06 last 20.68) - open/close 1063 (time 1777001012.11 total 50.10 last 19.62) - open/close 1210 (time 1777001022.19 total 60.18 last 14.59) - open/close 1317 (time 1777001032.78 total 70.77 last 10.10) - open/close 1424 (time 1777001042.78 total 80.78 last 10.69) - open/close 1545 (time 1777001052.84 total 90.83 last 12.04) - open/close 1670 (time 1777001062.85 total 100.85 last 12.48) - open/close 1873 (time 1777001072.88 total 110.87 last 20.26) - open/close 2064 (time 1777001082.90 total 120.89 last 19.06) - open/close 2277 (time 1777001092.93 total 130.92 last 21.24) - open/close 2489 (time 1777001102.95 total 140.95 last 21.14) - open/close 2700 (time 1777001112.99 total 150.99 last 21.02) - open/close 2904 (time 1777001123.03 total 161.02 last 20.32) - open/close 3108 (time 1777001133.03 total 171.03 last 20.40) - open/close 3307 (time 1777001143.07 total 181.06 last 19.84) - open/close 3543 (time 1777001153.08 total 191.07 last 23.56) - open/close 3767 (time 1777001163.10 total 201.09 last 22.37) - open/close 3994 (time 1777001173.13 total 211.12 last 22.63) - open/close 4190 (time 1777001183.16 total 221.15 last 19.54) - open/close 4418 (time 1777001193.20 total 231.19 last 22.71) - open/close 4631 (time 1777001203.24 total 241.23 last 21.21) - open/close 4849 (time 1777001213.25 total 251.24 last 21.78) - open/close 5055 (time 1777001223.27 total 261.26 last 20.57) - open/close 5258 (time 1777001233.27 total 271.26 last 20.30) - open/close 5460 (time 1777001243.27 total 281.26 last 20.19) - open/close 5676 (time 1777001253.32 total 291.31 last 21.50) - open/close 5866 (time 1777001263.34 total 301.33 last 18.97) - open/close 6069 (time 1777001273.36 total 311.35 last 20.25) - open/close 6270 (time 1777001283.39 total 321.39 last 20.04) - open/close 6480 (time 1777001293.44 total 331.43 last 20.91) - open/close 6704 (time 1777001303.45 total 341.44 last 22.37) - open/close 6919 (time 1777001313.53 total 351.52 last 21.34) - open/close 7141 (time 1777001323.55 total 361.54 last 22.16) - open/close 7357 (time 1777001333.60 total 371.59 last 21.48) - open/close 7575 (time 1777001343.65 total 381.64 last 21.69) - open/close 7794 (time 1777001353.68 total 391.67 last 21.83) - open/close 7998 (time 1777001363.72 total 401.71 last 20.32) - open/close 8230 (time 1777001373.73 total 411.73 last 23.17) - open/close 8448 (time 1777001383.80 total 421.79 last 21.65) - open/close 8685 (time 1777001393.82 total 431.81 last 23.67) - open/close 8918 (time 1777001403.82 total 441.81 last 23.29) - open/close 9145 (time 1777001413.84 total 451.83 last 22.66) - open/close 9363 (time 1777001423.85 total 461.84 last 21.78) - open/close 9569 (time 1777001433.90 total 471.89 last 20.49) - open/close 9784 (time 1777001443.93 total 481.92 last 21.44) - open/close 9991 (time 1777001453.95 total 491.94 last 20.67) total: 10000 open/close in 492.25 seconds: 20.31 ops/second - unlinked 0 (time 1777001469 ; total 0 ; last 0) total: 10000 unlinks in 238 seconds: 42.016808 unlinks/second - open/close 216 (time 1777001733.38 total 10.01 last 21.57) - open/close 445 (time 1777001743.40 total 20.03 last 22.87) - open/close 658 (time 1777001753.42 total 30.05 last 21.26) - open/close 873 (time 1777001763.42 total 40.05 last 21.50) - open/close 1111 (time 1777001773.42 total 50.06 last 23.79) - open/close 1322 (time 1777001783.43 total 60.06 last 21.10) - open/close 1543 (time 1777001793.45 total 70.09 last 22.04) - open/close 1761 (time 1777001803.50 total 80.13 last 21.71) - open/close 1954 (time 1777001813.54 total 90.17 last 19.22) - open/close 2160 (time 1777001823.56 total 100.19 last 20.56) - open/close 2377 (time 1777001833.59 total 110.22 last 21.62) - open/close 2596 (time 1777001843.65 total 120.28 last 21.77) - open/close 2825 (time 1777001853.67 total 130.30 last 22.87) - open/close 3054 (time 1777001863.68 total 140.31 last 22.86) - open/close 3295 (time 1777001873.75 total 150.38 last 23.93) - open/close 3537 (time 1777001883.77 total 160.40 last 24.17) - open/close 3758 (time 1777001893.78 total 170.41 last 22.08) - open/close 3980 (time 1777001903.81 total 180.44 last 22.14) - open/close 4196 (time 1777001913.86 total 190.49 last 21.49) - open/close 4418 (time 1777001923.90 total 200.53 last 22.10) - open/close 4687 (time 1777001933.91 total 210.54 last 26.88) - open/close 4913 (time 1777001943.93 total 220.56 last 22.56) - open/close 5134 (time 1777001953.93 total 230.56 last 22.09) - open/close 5367 (time 1777001963.98 total 240.61 last 23.19) - open/close 5568 (time 1777001973.99 total 250.62 last 20.09) - open/close 5789 (time 1777001984.02 total 260.65 last 22.03) - open/close 6003 (time 1777001994.02 total 270.65 last 21.39) - open/close 6217 (time 1777002004.03 total 280.66 last 21.38) - open/close 6453 (time 1777002014.04 total 290.67 last 23.57) - open/close 6699 (time 1777002024.12 total 300.75 last 24.42) - open/close 6920 (time 1777002034.15 total 310.78 last 22.04) - open/close 7147 (time 1777002044.15 total 320.78 last 22.69) - open/close 7368 (time 1777002054.16 total 330.79 last 22.08) - open/close 7585 (time 1777002064.16 total 340.79 last 21.70) - open/close 7810 (time 1777002074.17 total 350.80 last 22.47) - open/close 8043 (time 1777002084.20 total 360.83 last 23.24) - open/close 8255 (time 1777002094.25 total 370.88 last 21.09) - open/close 8458 (time 1777002104.32 total 380.95 last 20.17) - open/close 8678 (time 1777002114.36 total 390.99 last 21.91) - open/close 8881 (time 1777002124.37 total 401.00 last 20.28) - open/close 9117 (time 1777002134.41 total 411.04 last 23.51) - open/close 9345 (time 1777002144.46 total 421.09 last 22.69) - open/close 9567 (time 1777002154.49 total 431.13 last 22.11) - open/close 9774 (time 1777002164.53 total 441.16 last 20.63) - open/close 9990 (time 1777002174.56 total 451.19 last 21.54) total: 10000 open/close in 451.51 seconds: 22.15 ops/second - unlinked 0 (time 1777002188 ; total 0 ; last 0) total: 10000 unlinks in 224 seconds: 44.642857 unlinks/second - open/close 233 (time 1777002435.75 total 10.06 last 23.17) - open/close 457 (time 1777002445.77 total 20.08 last 22.36) - open/close 671 (time 1777002455.79 total 30.10 last 21.35) - open/close 910 (time 1777002465.81 total 40.11 last 23.86) - open/close 1125 (time 1777002475.83 total 50.14 last 21.45) - open/close 1349 (time 1777002485.85 total 60.16 last 22.35) - open/close 1561 (time 1777002495.86 total 70.16 last 21.19) - open/close 1790 (time 1777002505.86 total 80.17 last 22.89) - open/close 2006 (time 1777002515.87 total 90.18 last 21.58) - open/close 2241 (time 1777002525.90 total 100.20 last 23.44) - open/close 2486 (time 1777002535.90 total 110.21 last 24.49) - open/close 2718 (time 1777002545.90 total 120.21 last 23.20) - open/close 2972 (time 1777002555.92 total 130.22 last 25.36) - open/close 3234 (time 1777002565.93 total 140.24 last 26.17) - open/close 3494 (time 1777002575.98 total 150.29 last 25.86) - open/close 3717 (time 1777002585.99 total 160.29 last 22.29) - open/close 3958 (time 1777002596.04 total 170.34 last 23.98) - open/close 4198 (time 1777002606.07 total 180.38 last 23.92) - open/close 4423 (time 1777002616.12 total 190.43 last 22.39) - open/close 4657 (time 1777002626.14 total 200.45 last 23.35) - open/close 4887 (time 1777002636.19 total 210.49 last 22.89) - open/close 5139 (time 1777002646.23 total 220.53 last 25.11) - open/close 5361 (time 1777002656.25 total 230.56 last 22.14) - open/close 5587 (time 1777002666.30 total 240.61 last 22.48) - open/close 5815 (time 1777002676.31 total 250.62 last 22.79) - open/close 6051 (time 1777002686.32 total 260.63 last 23.57) - open/close 6310 (time 1777002696.36 total 270.67 last 25.79) - open/close 6567 (time 1777002706.41 total 280.72 last 25.58) - open/close 6781 (time 1777002716.44 total 290.75 last 21.34) - open/close 7006 (time 1777002726.46 total 300.76 last 22.46) - open/close 7261 (time 1777002736.48 total 310.78 last 25.45) - open/close 7487 (time 1777002746.51 total 320.82 last 22.52) - open/close 7743 (time 1777002756.51 total 330.82 last 25.59) - open/close 7971 (time 1777002766.56 total 340.86 last 22.70) - open/close 8238 (time 1777002776.59 total 350.89 last 26.62) - open/close 8485 (time 1777002786.62 total 360.93 last 24.62) - open/close 8741 (time 1777002796.62 total 370.93 last 25.59) - open/close 8988 (time 1777002806.63 total 380.94 last 24.68) - open/close 9226 (time 1777002816.69 total 390.99 last 23.67) - open/close 9479 (time 1777002826.70 total 401.01 last 25.26) - open/close 9705 (time 1777002836.72 total 411.02 last 22.57) - open/close 9966 (time 1777002846.73 total 421.03 last 26.07) total: 10000 open/close in 422.41 seconds: 23.67 ops/second - unlinked 0 (time 1777002858 ; total 0 ; last 0) total: 10000 unlinks in 210 seconds: 47.619049 unlinks/second - open/close 263 (time 1777003088.61 total 10.01 last 26.28) - open/close 510 (time 1777003098.62 total 20.01 last 24.69) - open/close 749 (time 1777003108.67 total 30.07 last 23.77) - open/close 991 (time 1777003118.71 total 40.11 last 24.11) - open/close 1243 (time 1777003128.71 total 50.11 last 25.20) - open/close 1501 (time 1777003138.74 total 60.14 last 25.73) - open/close 1766 (time 1777003148.75 total 70.14 last 26.48) - open/close 2040 (time 1777003158.75 total 80.15 last 27.38) - open/close 2283 (time 1777003168.80 total 90.19 last 24.19) - open/close 2545 (time 1777003178.81 total 100.21 last 26.16) - open/close 2786 (time 1777003188.85 total 110.25 last 24.01) - open/close 3024 (time 1777003198.87 total 120.27 last 23.75) - open/close 3276 (time 1777003208.90 total 130.30 last 25.12) - open/close 3531 (time 1777003218.95 total 140.34 last 25.38) - open/close 3751 (time 1777003228.95 total 150.35 last 21.98) - open/close 4016 (time 1777003238.96 total 160.36 last 26.48) - open/close 4257 (time 1777003249.00 total 170.39 last 24.03) - open/close 4487 (time 1777003259.02 total 180.41 last 22.95) - open/close 4726 (time 1777003269.03 total 190.42 last 23.88) - open/close 4973 (time 1777003279.05 total 200.44 last 24.64) - open/close 5218 (time 1777003289.10 total 210.50 last 24.37) - open/close 5445 (time 1777003299.11 total 220.51 last 22.68) - open/close 5700 (time 1777003309.18 total 230.58 last 25.32) - open/close 5949 (time 1777003319.20 total 240.59 last 24.86) - open/close 6187 (time 1777003329.22 total 250.62 last 23.73) - open/close 6449 (time 1777003339.25 total 260.65 last 26.13) - open/close 6716 (time 1777003349.32 total 270.71 last 26.53) - open/close 6971 (time 1777003359.34 total 280.74 last 25.43) - open/close 7215 (time 1777003369.34 total 290.74 last 24.40) - open/close 7484 (time 1777003379.37 total 300.77 last 26.83) - open/close 7738 (time 1777003389.38 total 310.78 last 25.38) - open/close 7987 (time 1777003399.43 total 320.83 last 24.78) - open/close 8222 (time 1777003409.43 total 330.83 last 23.50) - open/close 8460 (time 1777003419.46 total 340.85 last 23.74) - open/close 8681 (time 1777003429.50 total 350.90 last 22.01) - open/close 8931 (time 1777003439.51 total 360.90 last 24.98) - open/close 9185 (time 1777003449.54 total 370.94 last 25.31) - open/close 9440 (time 1777003459.55 total 380.95 last 25.48) - open/close 9700 (time 1777003469.60 total 391.00 last 25.88) - open/close 9982 (time 1777003479.61 total 401.00 last 28.18) total: 10000 open/close in 401.60 seconds: 24.90 ops/second - unlinked 0 (time 1777003491 ; total 1 ; last 1) total: 10000 unlinks in 203 seconds: 49.261086 unlinks/second - open/close 250 (time 1777003713.94 total 10.00 last 24.99) - open/close 497 (time 1777003723.98 total 20.04 last 24.60) - open/close 734 (time 1777003733.99 total 30.06 last 23.66) - open/close 973 (time 1777003744.01 total 40.08 last 23.86) - open/close 1224 (time 1777003754.03 total 50.10 last 25.04) - open/close 1501 (time 1777003764.06 total 60.13 last 27.61) - open/close 1770 (time 1777003774.08 total 70.14 last 26.87) - open/close 2010 (time 1777003784.09 total 80.15 last 23.98) - open/close 2267 (time 1777003794.09 total 90.16 last 25.68) - open/close 2519 (time 1777003804.11 total 100.18 last 25.16) - open/close 2769 (time 1777003814.12 total 110.19 last 24.97) - open/close 3015 (time 1777003824.12 total 120.19 last 24.60) - open/close 3288 (time 1777003834.14 total 130.21 last 27.24) - open/close 3534 (time 1777003844.16 total 140.22 last 24.57) - open/close 3786 (time 1777003854.17 total 150.24 last 25.16) - open/close 4056 (time 1777003864.22 total 160.29 last 26.88) - open/close 4292 (time 1777003874.26 total 170.32 last 23.51) - open/close 4565 (time 1777003884.26 total 180.33 last 27.28) - open/close 4820 (time 1777003894.27 total 190.34 last 25.48) - open/close 5072 (time 1777003904.29 total 200.36 last 25.15) - open/close 5327 (time 1777003914.29 total 210.36 last 25.50) - open/close 5598 (time 1777003924.30 total 220.37 last 27.09) - open/close 5895 (time 1777003934.31 total 230.38 last 29.66) - open/close 6161 (time 1777003944.34 total 240.41 last 26.53) - open/close 6423 (time 1777003954.37 total 250.44 last 26.12) - open/close 6684 (time 1777003964.38 total 260.45 last 26.07) - open/close 6960 (time 1777003974.40 total 270.47 last 27.54) - open/close 7224 (time 1777003984.43 total 280.50 last 26.33) - open/close 7508 (time 1777003994.44 total 290.51 last 28.37) - open/close 7784 (time 1777004004.45 total 300.52 last 27.59) - open/close 8054 (time 1777004014.46 total 310.53 last 26.97) - open/close 8318 (time 1777004024.48 total 320.55 last 26.35) - open/close 8577 (time 1777004034.48 total 330.55 last 25.89) - open/close 8848 (time 1777004044.50 total 340.57 last 27.05) - open/close 9114 (time 1777004054.52 total 350.59 last 26.54) - open/close 9387 (time 1777004064.54 total 360.61 last 27.24) - open/close 9657 (time 1777004074.55 total 370.61 last 26.99) - open/close 9921 (time 1777004084.58 total 380.65 last 26.32) total: 10000 open/close in 383.85 seconds: 26.05 ops/second - unlinked 0 (time 1777004097 ; total 0 ; last 0) total: 10000 unlinks in 202 seconds: 49.504951 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.159@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre On OST0, 20429 used inodes rc=0 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 22 umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.159@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -b 4096 -I 512 -q -O uninit_bg,extents,dir_nlink,quota,project,huge_file,large_dir,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init,packed_meta_blocks -F /dev/mapper/ost1_flakey 200000k checking for existing Lustre data: found Read previous values: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.159@tcp sys.timeout=20 Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: errors=remount-ro Parameters: mgsnode=192.168.201.159@tcp sys.timeout=20 start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg159-client.virtnet: -o user_xattr,flock 192.168.201.159@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg159-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server unloading modules via unload_modules_local on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing unload_modules_local oleg159-server: modules unloaded. pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-client: ssh exited with exit code 2 pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 PASS 69 (7532s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 00:22:23 (1777004543) oleg159-server: debugfs 1.47.3-wc2 (11-Nov-2025) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg159-client: oleg159-server: ssh exited with exit code 2 oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions oleg159-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg159-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg159-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format ost1: /dev/mapper/ost1_flakey pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 SKIP: conf-sanity test_111 /dev/mapper/mds1_flakey too small for 5000000kB MDS Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg159-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg159-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg159-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg159-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg159-server: oleg159-server.virtnet: executing set_default_debug -1 all pdsh@oleg159-client: oleg159-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg159-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg159-server: oleg159-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg159-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg159-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg159-server stop mds service on oleg159-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg159-server SKIP 111 (219s) Stopping clients: oleg159-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg159-client.virtnet /mnt/lustre2 (opts:-f) oleg159-server: oleg159-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg159-server' oleg159-server: oleg159-server.virtnet: executing load_modules_local oleg159-server: Loading modules from /home/green/git/lustre-release/lustre oleg159-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg159-server: detected 4 online CPUs by sysfs oleg159-server: MODOPTS_LIBCFS= oleg159-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 9045 sec ========== 00:27:25 (1777004845) === conf-sanity: start cleanup 00:27:28 (1777004848) === === conf-sanity: finish cleanup 00:27:32 (1777004852) ===