-----============= acceptance-small: conf-sanity ============----- Fri Oct 24 08:59:18 EDT 2025 oleg137-server: ls: cannot access '/home/green/git/lustre-release/lustre/tests/except/conf-sanity.*ex': No such file or directory excepting tests: 32b 32c 32newtarball 110 108a 112a 112b 113 117 119 121 122a 123aa 123ab 123ac 123ad 123ae 123af 123ag 123ah 123ahi 123F 123G 123H 126 129 132 133 135 136 137 150 152 153a 153b 153c 155 802a Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping client oleg137-client.virtnet /mnt/lustre opts:-f Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg137-server oleg137-server: oleg137-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg137-server: detected 4 online CPUs by sysfs oleg137-server: Force libcfs to create 2 CPU partitions Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 start mds service on oleg137-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-mdt1/mdt1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Commit the device label on lustre-ost1/ost1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg137-server: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg137-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server == conf-sanity test 45: long unlink handling in ptlrpcd == 09:01:37 (1761310897) start mds service on oleg137-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock oleg137-server@tcp:/lustre /mnt/lustre setup single mount lustre success stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server sleep 60 sec fail_loc=0x8000050f sleep 10 sec manual umount lustre on /mnt/lustre.... df: /mnt/lustre: Input/output error fail_loc=0x0 start mds service on oleg137-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock oleg137-server@tcp:/lustre /mnt/lustre umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local oleg137-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg137-server: LNET ready to unload oleg137-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 PASS 45 (179s) == conf-sanity test 69: replace an OST with the same index ========================================================== 09:04:37 (1761311077) start mds service on oleg137-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg137-server: detected 4 online CPUs by sysfs oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg137-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg137-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock oleg137-server@tcp:/lustre /mnt/lustre On OST0, 11872 inodes available. Want 99872. rc=0 - open/close 2001 (time 1761311122.82 total 10.00 last 200.07) - open/close 4540 (time 1761311132.82 total 20.00 last 253.90) - open/close 7738 (time 1761311142.82 total 30.00 last 319.75) - open/close 9691 (time 1761311152.83 total 40.01 last 195.25) total: 10000 open/close in 41.54 seconds: 240.72 ops/second - unlinked 0 (time 1761311156 ; total 0 ; last 0) total: 10000 unlinks in 25 seconds: 400.000000 unlinks/second - open/close 1846 (time 1761311193.10 total 10.00 last 184.59) - open/close 4462 (time 1761311203.10 total 20.00 last 261.53) - open/close 6647 (time 1761311213.10 total 30.00 last 218.47) - open/close 8812 (time 1761311223.11 total 40.01 last 216.43) total: 10000 open/close in 44.48 seconds: 224.80 ops/second - unlinked 0 (time 1761311229 ; total 0 ; last 0) total: 10000 unlinks in 23 seconds: 434.782623 unlinks/second - open/close 2024 (time 1761311265.02 total 10.00 last 202.34) - open/close 4598 (time 1761311275.02 total 20.00 last 257.38) - open/close 7672 (time 1761311285.02 total 30.01 last 307.34) - open/close 8587 (time 1761311295.03 total 40.02 last 91.40) - open/close 9268 (time 1761311305.03 total 50.02 last 68.07) total: 10000 open/close in 59.57 seconds: 167.87 ops/second - unlinked 0 (time 1761311318 ; total 0 ; last 0) total: 10000 unlinks in 78 seconds: 128.205124 unlinks/second - open/close 494 (time 1761311411.37 total 10.01 last 49.37) - open/close 1145 (time 1761311421.38 total 20.01 last 65.05) - open/close 1804 (time 1761311431.38 total 30.01 last 65.90) - open/close 2382 (time 1761311441.40 total 40.03 last 57.69) - open/close 3036 (time 1761311451.41 total 50.05 last 65.30) - open/close 3749 (time 1761311461.42 total 60.06 last 71.25) - open/close 4453 (time 1761311471.43 total 70.07 last 70.32) - open/close 5204 (time 1761311481.44 total 80.07 last 75.07) - open/close 5981 (time 1761311491.44 total 90.08 last 77.66) - open/close 6695 (time 1761311501.45 total 100.08 last 71.37) - open/close 7430 (time 1761311511.45 total 110.09 last 73.46) - open/close 8187 (time 1761311521.46 total 120.09 last 75.65) - open/close 8873 (time 1761311531.47 total 130.10 last 68.56) - open/close 9604 (time 1761311541.47 total 140.10 last 73.10) total: 10000 open/close in 145.45 seconds: 68.75 ops/second - unlinked 0 (time 1761311550 ; total 0 ; last 0) total: 10000 unlinks in 81 seconds: 123.456787 unlinks/second - open/close 735 (time 1761311644.80 total 10.00 last 73.48) - open/close 1409 (time 1761311654.80 total 20.01 last 67.34) - open/close 2076 (time 1761311664.81 total 30.02 last 66.66) - open/close 2832 (time 1761311674.81 total 40.02 last 75.59) - open/close 3540 (time 1761311684.82 total 50.02 last 70.76) - open/close 4263 (time 1761311694.82 total 60.03 last 72.29) - open/close 4955 (time 1761311704.83 total 70.04 last 69.14) - open/close 5637 (time 1761311714.84 total 80.04 last 68.15) - open/close 6253 (time 1761311724.84 total 90.05 last 61.55) - open/close 6862 (time 1761311734.85 total 100.06 last 60.85) - open/close 7488 (time 1761311744.85 total 110.06 last 62.60) - open/close 8080 (time 1761311754.86 total 120.06 last 59.17) - open/close 8738 (time 1761311764.86 total 130.07 last 65.78) - open/close 9403 (time 1761311774.87 total 140.07 last 66.46) total: 10000 open/close in 148.76 seconds: 67.22 ops/second - unlinked 0 (time 1761311787 ; total 0 ; last 0) total: 10000 unlinks in 88 seconds: 113.636360 unlinks/second - open/close 639 (time 1761311889.72 total 10.00 last 63.90) - open/close 1327 (time 1761311899.72 total 20.00 last 68.79) - open/close 1994 (time 1761311909.73 total 30.01 last 66.66) - open/close 2640 (time 1761311919.74 total 40.01 last 64.56) - open/close 3339 (time 1761311929.75 total 50.03 last 69.80) - open/close 4012 (time 1761311939.75 total 60.03 last 67.28) - open/close 4706 (time 1761311949.76 total 70.04 last 69.34) - open/close 5408 (time 1761311959.77 total 80.05 last 70.15) - open/close 6074 (time 1761311969.77 total 90.05 last 66.58) - open/close 6770 (time 1761311979.77 total 100.05 last 69.59) - open/close 7461 (time 1761311989.78 total 110.06 last 69.07) - open/close 8126 (time 1761311999.78 total 120.06 last 66.45) - open/close 8793 (time 1761312009.79 total 130.07 last 66.66) - open/close 9422 (time 1761312019.79 total 140.07 last 62.87) total: 10000 open/close in 148.64 seconds: 67.28 ops/second - unlinked 0 (time 1761312031 ; total 0 ; last 0) total: 10000 unlinks in 85 seconds: 117.647057 unlinks/second - open/close 681 (time 1761312131.19 total 10.00 last 68.08) - open/close 1324 (time 1761312141.19 total 20.01 last 64.28) - open/close 1993 (time 1761312151.20 total 30.01 last 66.86) - open/close 2643 (time 1761312161.20 total 40.02 last 64.97) - open/close 3337 (time 1761312171.21 total 50.03 last 69.33) - open/close 4006 (time 1761312181.22 total 60.04 last 66.85) - open/close 4692 (time 1761312191.23 total 70.05 last 68.54) - open/close 5381 (time 1761312201.24 total 80.06 last 68.83) - open/close 6061 (time 1761312211.24 total 90.06 last 67.97) - open/close 6740 (time 1761312221.24 total 100.06 last 67.88) - open/close 7409 (time 1761312231.26 total 110.08 last 66.77) - open/close 8029 (time 1761312241.27 total 120.09 last 61.94) - open/close 8662 (time 1761312251.28 total 130.10 last 63.26) - open/close 9331 (time 1761312261.29 total 140.11 last 66.82) total: 10000 open/close in 150.03 seconds: 66.65 ops/second - unlinked 0 (time 1761312275 ; total 0 ; last 0) total: 10000 unlinks in 82 seconds: 121.951218 unlinks/second - open/close 680 (time 1761312371.77 total 10.01 last 67.91) - open/close 1356 (time 1761312381.79 total 20.03 last 67.52) - open/close 2071 (time 1761312391.79 total 30.03 last 71.46) - open/close 2778 (time 1761312401.80 total 40.04 last 70.64) - open/close 3496 (time 1761312411.82 total 50.06 last 71.67) - open/close 4172 (time 1761312421.82 total 60.06 last 67.60) - open/close 4832 (time 1761312431.83 total 70.07 last 65.94) - open/close 5495 (time 1761312441.83 total 80.07 last 66.30) - open/close 6203 (time 1761312451.83 total 90.07 last 70.76) - open/close 6920 (time 1761312461.85 total 100.09 last 71.60) - open/close 7658 (time 1761312471.86 total 110.10 last 73.74) - open/close 8354 (time 1761312481.86 total 120.10 last 69.53) - open/close 9044 (time 1761312491.88 total 130.12 last 68.89) - open/close 9777 (time 1761312501.88 total 140.12 last 73.27) total: 10000 open/close in 143.20 seconds: 69.83 ops/second - unlinked 0 (time 1761312508 ; total 0 ; last 0) total: 10000 unlinks in 73 seconds: 136.986298 unlinks/second - open/close 839 (time 1761312594.71 total 10.00 last 83.89) - open/close 1745 (time 1761312604.72 total 20.01 last 90.52) - open/close 2653 (time 1761312614.72 total 30.02 last 90.74) - open/close 3499 (time 1761312624.73 total 40.02 last 84.56) - open/close 4268 (time 1761312634.73 total 50.02 last 76.87) - open/close 5171 (time 1761312644.73 total 60.03 last 90.28) - open/close 6027 (time 1761312654.74 total 70.03 last 85.56) - open/close 6801 (time 1761312664.75 total 80.04 last 77.31) - open/close 7598 (time 1761312674.75 total 90.05 last 79.66) - open/close 8350 (time 1761312684.76 total 100.05 last 75.17) - open/close 9197 (time 1761312694.76 total 110.05 last 84.69) total: 10000 open/close in 119.15 seconds: 83.93 ops/second - unlinked 0 (time 1761312706 ; total 0 ; last 0) total: 10000 unlinks in 69 seconds: 144.927536 unlinks/second - open/close 796 (time 1761312788.90 total 10.00 last 79.59) - open/close 1640 (time 1761312798.90 total 20.00 last 84.38) - open/close 2417 (time 1761312808.91 total 30.01 last 77.68) - open/close 3216 (time 1761312818.93 total 40.03 last 79.74) - open/close 3969 (time 1761312828.94 total 50.04 last 75.23) - open/close 4756 (time 1761312838.94 total 60.04 last 78.66) - open/close 5531 (time 1761312848.94 total 70.04 last 77.50) - open/close 6282 (time 1761312858.95 total 80.05 last 75.05) - open/close 7037 (time 1761312868.95 total 90.05 last 75.47) - open/close 7774 (time 1761312878.96 total 100.06 last 73.66) - open/close 8473 (time 1761312888.97 total 110.07 last 69.84) - open/close 9298 (time 1761312898.97 total 120.07 last 82.49) total: 10000 open/close in 129.64 seconds: 77.14 ops/second - unlinked 0 (time 1761312911 ; total 0 ; last 0) total: 10000 unlinks in 74 seconds: 135.135132 unlinks/second rm: missing operand Try 'rm --help' for more information. umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server Permanent disk data: Target: lustre-OST0000 Index: 0 Lustre FS: lustre Mount type: zfs Flags: 0x42 (OST update ) Persistent mount opts: Parameters: mgsnode=192.168.201.137@tcp sys.timeout=20 autodegrade=on mkfs_cmd = zpool create -o feature@longname=disabled -f -O canmount=off lustre-ost1 /dev/vde mkfs_cmd = zfs create -o canmount=off -o quota=409600000 lustre-ost1/ost1 xattr=sa dnodesize=auto recordsize=1M compression=off Writing lustre-ost1/ost1 properties lustre:mgsnode=192.168.201.137@tcp lustre:sys.timeout=20 lustre:autodegrade=on lustre:version=1 lustre:flags=66 lustre:index=0 lustre:fsname=lustre lustre:svname=lustre-OST0000 start ost1 service on oleg137-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg137-server: oleg137-server.virtnet: executing set_default_debug -1 all 8 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 1 Started lustre-OST0000 oleg137-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid oleg137-server: oleg137-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 40 oleg137-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec mount lustre on /mnt/lustre..... Starting client: oleg137-client.virtnet: -o user_xattr,flock oleg137-server@tcp:/lustre /mnt/lustre On OST0, 10357 used inodes rc=0 umount lustre on /mnt/lustre..... Stopping client oleg137-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg137-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg137-server stop mds service on oleg137-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg137-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy LNET ready to unload /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy unloading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing unload_modules_local oleg137-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 99: echo: write error: Device or resource busy oleg137-server: LNET ready to unload oleg137-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 141: echo: write error: Device or resource busy modules unloaded. pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-client: ssh exited with exit code 2 pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 PASS 69 (1986s) == conf-sanity test 111: Adding large_dir with over 2GB directory ========================================================== 09:37:44 (1761313064) SKIP: conf-sanity test_111 Only applicable to ldiskfs-based MDTs SKIP 111 (4s) Stopping clients: oleg137-client.virtnet /mnt/lustre (opts:-f) Stopping clients: oleg137-client.virtnet /mnt/lustre2 (opts:-f) pdsh@oleg137-client: oleg137-server: ssh exited with exit code 2 oleg137-server: oleg137-server.virtnet: executing set_hostid /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' loading modules on: 'oleg137-server' oleg137-server: oleg137-server.virtnet: executing load_modules_local oleg137-server: Loading modules from /home/green/git/lustre-release/lustre oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg137-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg137-server: detected 4 online CPUs by sysfs oleg137-server: Force libcfs to create 2 CPU partitions oleg137-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg137-server: quota/lquota options: 'hash_lqs_cur_bits=3' Formatting mgs, mds, osts Format mds1: lustre-mdt1/mdt1 Format ost1: lustre-ost1/ost1 Format ost2: lustre-ost2/ost2 == conf-sanity test complete, duration 2362 sec ========== 09:38:40 (1761313120)