-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 09:46:44 EDT 2025 oleg440-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 110 Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg440-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg440-server oleg440-server: oleg440-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey start mds service on oleg440-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds2_flakey Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg440-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server == conf-sanity test 45: long unlink handling in ptlrpcd == 09:49:38 (1761313778) start mds service on oleg440-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg440-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg440-server: LNET ready to unload oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 PASS 45 (234s) == conf-sanity test 69: replace an OST with the same index ========================================================== 09:53:32 (1761314012) start mds service on oleg440-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg440-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg440-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg440-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre On OST0, 49538 inodes available. Want 99872. rc=0 - open/close 786 (time 1761314090.42 total 10.00 last 78.60) - open/close 1594 (time 1761314100.44 total 20.02 last 80.66) - open/close 2455 (time 1761314110.44 total 30.02 last 86.09) - open/close 3224 (time 1761314120.45 total 40.03 last 76.81) - open/close 3966 (time 1761314130.46 total 50.04 last 74.14) - open/close 4712 (time 1761314140.46 total 60.04 last 74.58) - open/close 5505 (time 1761314150.47 total 70.05 last 79.22) - open/close 6281 (time 1761314160.49 total 80.07 last 77.48) - open/close 7083 (time 1761314170.49 total 90.07 last 80.16) - open/close 7830 (time 1761314180.50 total 100.08 last 74.63) - open/close 8584 (time 1761314190.51 total 110.09 last 75.33) - open/close 9389 (time 1761314200.52 total 120.10 last 80.46) total: 10000 open/close in 127.48 seconds: 78.45 ops/second - unlinked 0 (time 1761314211 ; total 0 ; last 0) total: 10000 unlinks in 66 seconds: 151.515152 unlinks/second - open/close 945 (time 1761314289.53 total 10.00 last 94.49) - open/close 1772 (time 1761314299.54 total 20.01 last 82.61) - open/close 2561 (time 1761314309.55 total 30.02 last 78.87) - open/close 3376 (time 1761314319.55 total 40.02 last 81.45) - open/close 4325 (time 1761314329.56 total 50.03 last 94.86) - open/close 5254 (time 1761314339.56 total 60.03 last 92.90) - open/close 6134 (time 1761314349.57 total 70.04 last 87.92) - open/close 6952 (time 1761314359.57 total 80.04 last 81.73) - open/close 7793 (time 1761314369.58 total 90.05 last 84.04) - open/close 8739 (time 1761314379.58 total 100.05 last 94.58) - open/close 9613 (time 1761314389.59 total 110.06 last 87.36) total: 10000 open/close in 114.81 seconds: 87.10 ops/second - unlinked 0 (time 1761314397 ; total 0 ; last 0) total: 10000 unlinks in 65 seconds: 153.846161 unlinks/second - open/close 800 (time 1761314476.12 total 10.01 last 79.90) - open/close 1684 (time 1761314486.12 total 20.02 last 88.37) - open/close 2486 (time 1761314496.13 total 30.02 last 80.15) - open/close 3387 (time 1761314506.13 total 40.03 last 90.07) - open/close 4194 (time 1761314516.14 total 50.03 last 80.63) - open/close 4971 (time 1761314526.14 total 60.04 last 77.67) - open/close 5822 (time 1761314536.15 total 70.05 last 85.03) - open/close 6641 (time 1761314546.16 total 80.06 last 81.82) - open/close 7376 (time 1761314556.18 total 90.07 last 73.37) - open/close 8084 (time 1761314566.18 total 100.08 last 70.77) - open/close 8805 (time 1761314576.19 total 110.08 last 72.05) - open/close 9520 (time 1761314586.19 total 120.09 last 71.48) total: 10000 open/close in 125.84 seconds: 79.47 ops/second - unlinked 0 (time 1761314595 ; total 0 ; last 0) total: 10000 unlinks in 72 seconds: 138.888885 unlinks/second - open/close 866 (time 1761314680.32 total 10.01 last 86.55) - open/close 1799 (time 1761314690.33 total 20.01 last 93.24) - open/close 2603 (time 1761314700.37 total 30.05 last 80.11) - open/close 3320 (time 1761314710.37 total 40.05 last 71.70) - open/close 4060 (time 1761314720.38 total 50.06 last 73.92) - open/close 4822 (time 1761314730.38 total 60.07 last 76.14) - open/close 5556 (time 1761314740.39 total 70.07 last 73.37) - open/close 6281 (time 1761314750.39 total 80.07 last 72.48) - open/close 7047 (time 1761314760.41 total 90.09 last 76.48) - open/close 7806 (time 1761314770.42 total 100.10 last 75.84) - open/close 8584 (time 1761314780.43 total 110.12 last 77.67) - open/close 9357 (time 1761314790.44 total 120.13 last 77.22) total: 10000 open/close in 128.25 seconds: 77.97 ops/second - unlinked 0 (time 1761314801 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 847 (time 1761314890.45 total 10.01 last 84.65) - open/close 1625 (time 1761314900.46 total 20.01 last 77.76) - open/close 2378 (time 1761314910.46 total 30.01 last 75.28) - open/close 3143 (time 1761314920.47 total 40.02 last 76.45) - open/close 3822 (time 1761314930.49 total 50.04 last 67.74) - open/close 4543 (time 1761314940.49 total 60.04 last 72.10) - open/close 5260 (time 1761314950.49 total 70.04 last 71.69) - open/close 6044 (time 1761314960.50 total 80.05 last 78.32) - open/close 6899 (time 1761314970.50 total 90.06 last 85.49) - open/close 7655 (time 1761314980.51 total 100.06 last 75.54) - open/close 8301 (time 1761314990.51 total 110.07 last 64.57) - open/close 8912 (time 1761315000.52 total 120.07 last 61.09) - open/close 9512 (time 1761315010.52 total 130.07 last 60.00) total: 10000 open/close in 137.12 seconds: 72.93 ops/second - unlinked 0 (time 1761315021 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 786 (time 1761315113.15 total 10.01 last 78.54) - open/close 1575 (time 1761315123.15 total 20.01 last 78.89) - open/close 2382 (time 1761315133.17 total 30.03 last 80.56) - open/close 3078 (time 1761315143.17 total 40.03 last 69.56) - open/close 3821 (time 1761315153.19 total 50.04 last 74.22) - open/close 4425 (time 1761315163.19 total 60.05 last 60.38) - open/close 5121 (time 1761315173.19 total 70.05 last 69.59) - open/close 5865 (time 1761315183.20 total 80.06 last 74.30) - open/close 6605 (time 1761315193.21 total 90.07 last 73.94) - open/close 7392 (time 1761315203.21 total 100.07 last 78.68) - open/close 8098 (time 1761315213.22 total 110.08 last 70.57) - open/close 8856 (time 1761315223.23 total 120.09 last 75.73) - open/close 9588 (time 1761315233.24 total 130.10 last 73.13) total: 10000 open/close in 135.75 seconds: 73.66 ops/second - unlinked 0 (time 1761315241 ; total 0 ; last 0) total: 10000 unlinks in 76 seconds: 131.578949 unlinks/second - open/close 680 (time 1761315331.64 total 10.01 last 67.91) - open/close 1381 (time 1761315341.65 total 20.02 last 70.05) - open/close 2125 (time 1761315351.66 total 30.03 last 74.30) - open/close 2870 (time 1761315361.66 total 40.04 last 74.49) - open/close 3579 (time 1761315371.67 total 50.04 last 70.86) - open/close 4315 (time 1761315381.69 total 60.06 last 73.45) - open/close 4984 (time 1761315391.69 total 70.07 last 66.87) - open/close 5626 (time 1761315401.70 total 80.07 last 64.17) - open/close 6306 (time 1761315411.72 total 90.09 last 67.87) - open/close 6978 (time 1761315421.72 total 100.09 last 67.19) - open/close 7607 (time 1761315431.72 total 110.09 last 62.89) - open/close 8224 (time 1761315441.74 total 120.11 last 61.56) - open/close 8854 (time 1761315451.74 total 130.12 last 62.99) - open/close 9512 (time 1761315461.76 total 140.13 last 65.72) total: 10000 open/close in 148.02 seconds: 67.56 ops/second - unlinked 0 (time 1761315474 ; total 0 ; last 0) total: 10000 unlinks in 87 seconds: 114.942528 unlinks/second - open/close 682 (time 1761315575.94 total 10.02 last 68.08) - open/close 1353 (time 1761315585.95 total 20.03 last 67.03) - open/close 2048 (time 1761315595.96 total 30.04 last 69.45) - open/close 2714 (time 1761315605.98 total 40.06 last 66.46) - open/close 3439 (time 1761315615.99 total 50.07 last 72.38) - open/close 4182 (time 1761315626.00 total 60.08 last 74.27) - open/close 4905 (time 1761315636.01 total 70.09 last 72.21) - open/close 5637 (time 1761315646.02 total 80.10 last 73.14) - open/close 6354 (time 1761315656.03 total 90.11 last 71.65) - open/close 7070 (time 1761315666.04 total 100.12 last 71.52) - open/close 7726 (time 1761315676.04 total 110.12 last 65.57) - open/close 8442 (time 1761315686.04 total 120.12 last 71.59) - open/close 9094 (time 1761315696.05 total 130.13 last 65.15) - open/close 9748 (time 1761315706.05 total 140.13 last 65.38) total: 10000 open/close in 144.06 seconds: 69.42 ops/second - unlinked 0 (time 1761315714 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 749 (time 1761315813.48 total 10.01 last 74.81) - open/close 1497 (time 1761315823.49 total 20.02 last 74.74) - open/close 2219 (time 1761315833.49 total 30.02 last 72.17) - open/close 2855 (time 1761315843.51 total 40.04 last 63.50) - open/close 3571 (time 1761315853.51 total 50.05 last 71.56) - open/close 4285 (time 1761315863.52 total 60.05 last 71.36) - open/close 5051 (time 1761315873.53 total 70.06 last 76.52) - open/close 5736 (time 1761315883.53 total 80.06 last 68.49) - open/close 6346 (time 1761315893.54 total 90.07 last 60.94) - open/close 6896 (time 1761315903.56 total 100.09 last 54.91) - open/close 7444 (time 1761315913.56 total 110.10 last 54.77) - open/close 8113 (time 1761315923.57 total 120.11 last 66.83) - open/close 8728 (time 1761315933.58 total 130.11 last 61.46) - open/close 9409 (time 1761315943.58 total 140.12 last 68.08) total: 10000 open/close in 148.91 seconds: 67.15 ops/second - unlinked 0 (time 1761315956 ; total 0 ; last 0) total: 10000 unlinks in 79 seconds: 126.582275 unlinks/second - open/close 694 (time 1761316050.02 total 10.00 last 69.37) - open/close 1442 (time 1761316060.03 total 20.01 last 74.74) - open/close 2198 (time 1761316070.03 total 30.01 last 75.60) - open/close 2977 (time 1761316080.03 total 40.02 last 77.87) - open/close 3743 (time 1761316090.04 total 50.02 last 76.54) - open/close 4481 (time 1761316100.04 total 60.03 last 73.77) - open/close 5213 (time 1761316110.05 total 70.04 last 73.12) - open/close 5955 (time 1761316120.06 total 80.04 last 74.17) - open/close 6690 (time 1761316130.07 total 90.05 last 73.43) - open/close 7389 (time 1761316140.07 total 100.06 last 69.85) - open/close 8099 (time 1761316150.08 total 110.07 last 70.95) - open/close 8751 (time 1761316160.09 total 120.07 last 65.16) - open/close 9472 (time 1761316170.10 total 130.09 last 72.01) total: 10000 open/close in 137.74 seconds: 72.60 ops/second - unlinked 0 (time 1761316181 ; total 0 ; last 0) total: 10000 unlinks in 80 seconds: 125.000000 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x42 (OST update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.140@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre-OST0000 kilobytes 200000 options -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre-OST0000 -I 512 -q -O extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata start ost1 service on oleg440-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Started lustre-OST0000 oleg440-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg440-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg440-server: oleg440-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 40 oleg440-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre On OST0, 10429 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg440-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg440-server: LNET ready to unload oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 PASS 69 (2334s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 10:32:27 (1761316347) oleg440-server: debugfs 1.47.1-wc2 (08-Nov-2024) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present oleg440-server: debugfs 1.47.1-wc2 (08-Nov-2024) Supported features: dir_prealloc imagic_inodes has_journal ext_attr resize_inode dir_index sparse_super2 fast_commit stable_inodes orphan_file filetype needs_recovery journal_dev meta_bg extent 64bit mmp flex_bg ea_inode dirdata metadata_csum_seed large_dir inline_data encrypt casefold sparse_super large_file huge_file uninit_bg dir_nlink extra_isize quota bigalloc metadata_csum read-only project shared_blocks verity orphan_present umount lustre on /mnt/lustre..... stop ost1 service on oleg440-server stop mds service on oleg440-server stop mds service on oleg440-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. MDT params: --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity --backfstype=ldiskfs --device-size=2400000 --mkfsoptions=\"-O ea_inode,large_dir -E lazy_itable_init\" --reformat /dev/mapper/mds1_flakey /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Permanent disk data: Target: lustre:MDT0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x65 (MDT MGS first_time update ) Persistent mount opts: user_xattr,errors=remount-ro Parameters: sys.timeout=20 mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity device size = 2500MB formatting backing filesystem ldiskfs on /dev/mapper/mds1_flakey target name lustre:MDT0000 kilobytes 2400000 options -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:MDT0000 -J size=93 -I 1024 -i 2560 -q -O ea_inode,large_dir,dirdata,uninit_bg,^extents,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -E lazy_itable_init,lazy_journal_init="0" -F /dev/mapper/mds1_flakey 2400000k Writing CONFIGS/mountdata Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/mds1_flakey Started lustre-MDT0000 OST params: --mgsnode=oleg440-server@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=200000 --mkfsoptions=\"-O large_dir -E lazy_itable_init\" --reformat /dev/mapper/ost1_flakey Permanent disk data: Target: lustre:OST0000 Index: 0 Lustre FS: lustre Mount type: ldiskfs Flags: 0x62 (OST first_time update ) Persistent mount opts: ,errors=remount-ro Parameters: mgsnode=192.168.204.140@tcp sys.timeout=20 device size = 4096MB formatting backing filesystem ldiskfs on /dev/mapper/ost1_flakey target name lustre:OST0000 kilobytes 200000 options -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F mkfs_cmd = mke2fs -j -b 4096 -L lustre:OST0000 -I 512 -q -O large_dir,extents,uninit_bg,dir_nlink,quota,project,huge_file,^fast_commit,flex_bg -G 256 -E lazy_itable_init,resize="4290772992",lazy_journal_init="0" -F /dev/mapper/ost1_flakey 200000k Writing CONFIGS/mountdata Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg440-server: oleg440-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 1 Commit the device label on /dev/mapper/ost1_flakey Started lustre-OST0000 mount lustre on /mnt/lustre..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre Starting client oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre Started clients oleg440-client.virtnet: 192.168.204.140@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) mount lustre on /mnt/lustre2..... Starting client: oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre2 Starting client oleg440-client.virtnet: -o user_xattr,flock oleg440-server@tcp:/lustre /mnt/lustre2 Started clients oleg440-client.virtnet: 192.168.204.140@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1326500 1696 1206392 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 142216 1252 126964 1% /mnt/lustre[OST:0] filesystem_summary: 142216 1252 126964 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 960000 272 959728 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 50000 268 49732 1% /mnt/lustre[OST:0] filesystem_summary: 50004 272 49732 1% /mnt/lustre creating 60000 hardlinks to oleg440-client.virtnet-0 creating 60000 hardlinks to oleg440-client.virtnet-1 waiting for PIDs 27036 27050 to complete - link 819 (time 1761316440.52 total 10.01 last 81.86) - link 1651 (time 1761316450.53 total 20.01 last 83.14) - link 2374 (time 1761316460.53 total 30.01 last 72.29) - link 3092 (time 1761316470.53 total 40.02 last 71.78) - link 3867 (time 1761316480.54 total 50.03 last 77.42) - link 4555 (time 1761316490.55 total 60.03 last 68.75) - link 5203 (time 1761316500.56 total 70.04 last 64.74) - link 5831 (time 1761316510.56 total 80.04 last 62.80) - link 6459 (time 1761316520.57 total 90.06 last 62.72) - link 7181 (time 1761316530.57 total 100.06 last 72.18) - link 7902 (time 1761316540.58 total 110.07 last 72.02) - link 8598 (time 1761316550.58 total 120.07 last 69.60) - link 9275 (time 1761316560.59 total 130.07 last 67.67) - link 9959 (time 1761316570.60 total 140.08 last 68.32) - link 10626 (time 1761316580.60 total 150.09 last 66.69) - link 11282 (time 1761316590.61 total 160.10 last 65.52) - link 12047 (time 1761316600.62 total 170.11 last 76.43) - link 12861 (time 1761316610.62 total 180.11 last 81.37) - link 13544 (time 1761316620.63 total 190.12 last 68.27) - link 14266 (time 1761316630.64 total 200.12 last 72.16) - link 15017 (time 1761316640.64 total 210.12 last 75.08) - link 15758 (time 1761316650.64 total 220.13 last 74.05) - link 16442 (time 1761316660.65 total 230.14 last 68.34) - link 17104 (time 1761316670.66 total 240.15 last 66.13) - link 17644 (time 1761316680.67 total 250.16 last 53.96) - link 18104 (time 1761316690.73 total 260.21 last 45.75) - link 18572 (time 1761316700.73 total 270.21 last 46.79) - link 19292 (time 1761316710.74 total 280.23 last 71.91) - link 20000 (time 1761316720.53 total 290.01 last 72.34) - link 20829 (time 1761316730.54 total 300.02 last 82.81) - link 21626 (time 1761316740.55 total 310.03 last 79.63) - link 22309 (time 1761316750.55 total 320.04 last 68.25) - link 22935 (time 1761316760.57 total 330.06 last 62.49) - link 23589 (time 1761316770.58 total 340.06 last 65.35) - link 24216 (time 1761316780.59 total 350.08 last 62.62) - link 24892 (time 1761316790.61 total 360.09 last 67.50) - link 25606 (time 1761316800.62 total 370.10 last 71.32) - link 26318 (time 1761316810.62 total 380.11 last 71.17) - link 27056 (time 1761316820.63 total 390.12 last 73.73) - link 27810 (time 1761316830.63 total 400.12 last 75.39) - link 28584 (time 1761316840.64 total 410.12 last 77.37) - link 29353 (time 1761316850.64 total 420.13 last 76.87) - link 30000 (time 1761316860.18 total 429.67 last 67.81) - link 30670 (time 1761316870.19 total 439.68 last 66.95) - link 31326 (time 1761316880.20 total 449.69 last 65.51) - link 31954 (time 1761316890.21 total 459.69 last 62.78) - link 32700 (time 1761316900.21 total 469.70 last 74.58) - link 33443 (time 1761316910.22 total 479.71 last 74.23) - link 34165 (time 1761316920.23 total 489.72 last 72.12) - link 34857 (time 1761316930.23 total 499.72 last 69.18) - link 35548 (time 1761316940.25 total 509.73 last 69.00) - link 36229 (time 1761316950.25 total 519.74 last 68.06) - link 36903 (time 1761316960.25 total 529.74 last 67.40) - link 37603 (time 1761316970.25 total 539.74 last 69.99) - link 38233 (time 1761316980.26 total 549.74 last 62.98) - link 38976 (time 1761316990.26 total 559.75 last 74.25) - link 39602 (time 1761317000.29 total 569.77 last 62.47) - link 40000 (time 1761317006.75 total 576.24 last 61.57) - link 40674 (time 1761317016.78 total 586.26 last 67.21) - link 41386 (time 1761317026.78 total 596.27 last 71.16) - link 42106 (time 1761317036.80 total 606.28 last 71.91) - link 42784 (time 1761317046.81 total 616.30 last 67.69) - link 43477 (time 1761317056.83 total 626.31 last 69.21) - link 44280 (time 1761317066.83 total 636.32 last 80.26) - link 45046 (time 1761317076.84 total 646.33 last 76.52) - link 45837 (time 1761317086.86 total 656.34 last 78.96) - link 46618 (time 1761317096.87 total 666.36 last 77.98) - link 47449 (time 1761317106.89 total 676.37 last 82.97) - link 48304 (time 1761317116.89 total 686.37 last 85.50) - link 49033 (time 1761317126.89 total 696.38 last 72.88) - link 49916 (time 1761317136.89 total 706.38 last 88.27) - link 50675 (time 1761317146.90 total 716.38 last 75.87) - link 51526 (time 1761317156.90 total 726.39 last 85.05) - link 52354 (time 1761317166.92 total 736.40 last 82.68) - link 53192 (time 1761317176.92 total 746.40 last 83.80) - link 53938 (time 1761317186.93 total 756.41 last 74.53) - link 54766 (time 1761317196.94 total 766.42 last 82.74) - link 55539 (time 1761317206.94 total 776.43 last 77.23) - link 56272 (time 1761317216.95 total 786.44 last 73.24) - link 56979 (time 1761317226.96 total 796.45 last 70.62) - link 57728 (time 1761317236.97 total 806.46 last 74.84) - link 58470 (time 1761317246.98 total 816.47 last 74.10) - link 59307 (time 1761317256.99 total 826.48 last 83.61) total: 60000 link in 835.50 seconds: 71.81 ops/second - link 802 (time 1761316439.64 total 10.02 last 80.06) - link 1657 (time 1761316449.65 total 20.03 last 85.41) - link 2393 (time 1761316459.66 total 30.04 last 73.52) - link 3124 (time 1761316469.66 total 40.05 last 73.05) - link 3874 (time 1761316479.67 total 50.06 last 74.92) - link 4559 (time 1761316489.68 total 60.06 last 68.45) - link 5210 (time 1761316499.68 total 70.06 last 65.09) - link 5817 (time 1761316509.69 total 80.07 last 60.67) - link 6445 (time 1761316519.69 total 90.07 last 62.80) - link 7174 (time 1761316529.70 total 100.08 last 72.83) - link 7899 (time 1761316539.71 total 110.09 last 72.40) - link 8616 (time 1761316549.71 total 120.09 last 71.69) - link 9289 (time 1761316559.72 total 130.10 last 67.23) - link 9979 (time 1761316569.74 total 140.12 last 68.89) - link 10639 (time 1761316579.74 total 150.13 last 65.96) - link 11324 (time 1761316589.76 total 160.14 last 68.37) - link 12062 (time 1761316599.77 total 170.16 last 73.72) - link 12885 (time 1761316609.78 total 180.17 last 82.22) - link 13598 (time 1761316619.79 total 190.17 last 71.25) - link 14311 (time 1761316629.80 total 200.18 last 71.25) - link 15059 (time 1761316639.81 total 210.19 last 74.74) - link 15793 (time 1761316649.82 total 220.20 last 73.33) - link 16491 (time 1761316659.84 total 230.22 last 69.63) - link 17160 (time 1761316669.86 total 240.24 last 66.78) - link 17680 (time 1761316679.86 total 250.24 last 52.00) - link 18152 (time 1761316689.87 total 260.25 last 47.15) - link 18604 (time 1761316699.89 total 270.27 last 45.12) - link 19313 (time 1761316709.90 total 280.28 last 70.83) - link 20000 (time 1761316719.69 total 290.07 last 70.17) - link 20807 (time 1761316729.69 total 300.08 last 80.65) - link 21604 (time 1761316739.70 total 310.08 last 79.66) - link 22307 (time 1761316749.71 total 320.09 last 70.21) - link 22946 (time 1761316759.73 total 330.11 last 63.80) - link 23592 (time 1761316769.74 total 340.12 last 64.55) - link 24236 (time 1761316779.74 total 350.12 last 64.39) - link 24910 (time 1761316789.76 total 360.14 last 67.28) - link 25594 (time 1761316799.76 total 370.14 last 68.40) - link 26320 (time 1761316809.76 total 380.15 last 72.54) - link 27068 (time 1761316819.77 total 390.15 last 74.76) - link 27821 (time 1761316829.77 total 400.15 last 75.29) - link 28570 (time 1761316839.79 total 410.17 last 74.77) - link 29358 (time 1761316849.79 total 420.17 last 78.79) - link 30000 (time 1761316859.41 total 429.80 last 66.71) - link 30688 (time 1761316869.43 total 439.81 last 68.70) - link 31318 (time 1761316879.45 total 449.83 last 62.84) - link 31949 (time 1761316889.47 total 459.85 last 63.01) - link 32700 (time 1761316899.47 total 469.85 last 75.08) - link 33431 (time 1761316909.48 total 479.87 last 73.00) - link 34157 (time 1761316919.48 total 489.87 last 72.59) - link 34843 (time 1761316929.50 total 499.88 last 68.48) - link 35550 (time 1761316939.52 total 509.90 last 70.61) - link 36224 (time 1761316949.52 total 519.90 last 67.38) - link 36892 (time 1761316959.53 total 529.91 last 66.74) - link 37588 (time 1761316969.53 total 539.91 last 69.56) - link 38237 (time 1761316979.54 total 549.92 last 64.84) - link 38949 (time 1761316989.55 total 559.93 last 71.16) - link 39584 (time 1761316999.55 total 569.93 last 63.48) - link 40000 (time 1761317006.42 total 576.80 last 60.59) - link 40659 (time 1761317016.43 total 586.82 last 65.78) - link 41365 (time 1761317026.44 total 596.82 last 70.58) - link 42086 (time 1761317036.45 total 606.84 last 71.98) - link 42771 (time 1761317046.47 total 616.85 last 68.40) - link 43467 (time 1761317056.47 total 626.86 last 69.56) - link 44288 (time 1761317066.49 total 636.87 last 81.94) - link 45052 (time 1761317076.51 total 646.90 last 76.24) - link 45832 (time 1761317086.52 total 656.90 last 77.96) - link 46608 (time 1761317096.54 total 666.92 last 77.43) - link 47425 (time 1761317106.55 total 676.93 last 81.66) - link 48288 (time 1761317116.57 total 686.95 last 86.13) - link 49027 (time 1761317126.58 total 696.97 last 73.76) - link 49909 (time 1761317136.59 total 706.97 last 88.18) - link 50683 (time 1761317146.60 total 716.98 last 77.33) - link 51534 (time 1761317156.66 total 727.05 last 84.53) - link 52382 (time 1761317166.67 total 737.05 last 84.76) - link 53226 (time 1761317176.69 total 747.07 last 84.21) - link 53979 (time 1761317186.71 total 757.09 last 75.19) - link 54814 (time 1761317196.74 total 767.12 last 83.23) - link 55571 (time 1761317206.76 total 777.14 last 75.54) - link 56285 (time 1761317216.78 total 787.17 last 71.23) - link 56980 (time 1761317226.80 total 797.18 last 69.40) - link 57720 (time 1761317236.82 total 807.20 last 73.87) - link 58471 (time 1761317246.83 total 817.21 last 75.03) - link 59308 (time 1761317256.84 total 827.23 last 83.55) total: 60000 link in 836.42 seconds: 71.73 ops/second estimate 35206s left after 120000 files / 837s umount lustre on /mnt/lustre2..... Stopping client oleg440-client.virtnet /mnt/lustre2 (opts:-f) umount lustre on /mnt/lustre..... Stopping client oleg440-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg440-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg440-server stop mds service on oleg440-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg440-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing unload_modules_local oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg440-server: LNET ready to unload oleg440-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. ETA 35206s after 120000 files / 837s is too long e2fsck -d -v -t -t -f -n /dev/mapper/mds1_flakey -m8 oleg440-server: e2fsck 1.47.1-wc2 (08-Nov-2024) oleg440-server: Use max possible thread num: 1 instead Pass 1: Checking inodes, blocks, and sizes [Thread 0] Scan group range [0, 29], used inodes 276/960000 [Thread 0] jumping to group 0 [Thread 0] e2fsck_pass1_run:2675: increase inode 116 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 117 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 118 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 152 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 186 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 187 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 188 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 189 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 190 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 191 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 192 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 193 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 194 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 195 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 196 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 197 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 198 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 199 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 200 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 201 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 202 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 203 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 204 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 205 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 206 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 207 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 208 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 209 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 210 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 211 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 212 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 213 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 214 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 215 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 216 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 217 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 218 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 219 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 220 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 221 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 223 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 224 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 225 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 226 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 227 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 228 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 229 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 230 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 231 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 232 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 233 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 234 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 235 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 236 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 237 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 238 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 239 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 240 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 241 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 242 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 243 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 244 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 245 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 246 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 247 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 248 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 249 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 250 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 251 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 252 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 253 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 254 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 255 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 256 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 257 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 258 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 259 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 260 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 261 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 262 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 263 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 264 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 265 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 266 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 267 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 268 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 269 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 270 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 271 badness 0 to 2 for 10084 [Thread 0] e2fsck_pass1_run:2655: increase inode 276 badness 0 to 1 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 276 badness 1 to 3 for 10084 [Thread 0] e2fsck_pass1_run:2655: increase inode 277 badness 0 to 1 for 10084 [Thread 0] e2fsck_pass1_run:2675: increase inode 277 badness 1 to 3 for 10084 [Thread 0] group 0 finished [Thread 0] group 1 finished [Thread 0] group 2 finished [Thread 0] group 3 finished [Thread 0] group 4 finished [Thread 0] group 5 finished [Thread 0] group 6 finished [Thread 0] group 7 finished [Thread 0] group 8 finished [Thread 0] group 9 finished [Thread 0] group 10 finished [Thread 0] group 11 finished [Thread 0] group 12 finished [Thread 0] group 13 finished [Thread 0] group 14 finished [Thread 0] group 15 finished [Thread 0] group 16 finished [Thread 0] group 17 finished [Thread 0] group 18 finished [Thread 0] group 19 finished [Thread 0] group 20 finished [Thread 0] group 21 finished [Thread 0] group 22 finished [Thread 0] group 23 finished [Thread 0] group 24 finished [Thread 0] group 25 finished [Thread 0] group 26 finished [Thread 0] group 27 finished [Thread 0] group 28 finished [Thread 0] group 29 finished [Thread 0] Pass 1: Memory used: 352k/1212k (126k/227k), time: 0.07/ 0.00/ 0.00 [Thread 0] Pass 1: I/O read: 2MB, write: 0MB, rate: 29.06MB/s [Thread 0] Scanned group range [0, 29], used inodes 276/281 Pass 2: Checking directory structure Pass 2: Memory used: 352k/304k (82k/271k), time: 0.94/ 0.31/ 0.09 Pass 2: I/O read: 49MB, write: 0MB, rate: 52.09MB/s Pass 3: Checking directory connectivity Peak memory: Memory used: 352k/304k (82k/271k), time: 1.10/ 0.33/ 0.10 Pass 3: Memory used: 352k/304k (81k/272k), time: 0.00/ 0.00/ 0.00 Pass 3: I/O read: 0MB, write: 0MB, rate: 0.00MB/s Pass 4: Checking reference counts Pass 4: Memory used: 352k/0k (79k/274k), time: 0.03/ 0.02/ 0.00 Pass 4: I/O read: 1MB, write: 0MB, rate: 28.98MB/s Pass 5: Checking group summary information Pass 5: Memory used: 492k/0k (78k/415k), time: 0.01/ 0.00/ 0.00 Pass 5: I/O read: 1MB, write: 0MB, rate: 122.52MB/s 276 inodes used (0.03%, out of 960000) 4 non-contiguous files (1.4%) 1 non-contiguous directory (0.4%) # of inodes with ind/dind/tind blocks: 1/1/0 278375 blocks used (46.40%, out of 600000) 0 bad blocks 1 large file 148 regular files 118 directories 0 character device files 0 block device files 0 fifos 120000 links 0 symbolic links (0 fast symbolic links) 0 sockets ------------ 120264 files Memory used: 492k/0k (80k/412k), time: 1.15/ 0.36/ 0.10 I/O read: 49MB, write: 0MB, rate: 42.44MB/s pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 pdsh@oleg440-client: oleg440-client: ssh exited with exit code 2 Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg440-client: oleg440-server: ssh exited with exit code 2 oleg440-server: oleg440-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions oleg440-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg440-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey PASS 111 (1045s) Stopping clients: oleg440-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg440-client.virtnet /mnt/lustre2 (opts:-f) oleg440-server: oleg440-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg440-server' oleg440-server: oleg440-server.virtnet: executing load_modules_local oleg440-server: Loading modules from /home/green/git/lustre-release/lustre oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg440-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg440-server: detected 4 online CPUs by sysfs oleg440-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: /dev/mapper/mds1_flakey Format mds2: /dev/mapper/mds2_flakey Format ost1: /dev/mapper/ost1_flakey Format ost2: /dev/mapper/ost2_flakey == conf-sanity test complete, duration 3843 sec ========== 10:50:49 (1761317449)