== sanity-quota test 12b: Inode quota rebalancing ======== 18:24:08 (1778538248) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Waiting 90s for 'u' Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000_UUID 0 - 0 - 0 - 0 - lustre-MDT0001_UUID 0 - 0 - 0 - 0 - lustre-OST0000_UUID 0 - 0 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 Create 2048 files on mdt0... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [createmany] [-m] [/mnt/lustre/d12b.sanity-quota/f12b.sanity-quota] [2048] mdt0 created 2048 Create files on mdt1... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [createmany] [-m] [/mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota] [1] mknod(/mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota0) error: Disk quota exceeded total: 0 create in 0.04 seconds: 0.00 ops/second Free space from mdt0... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [unlinkmany] [/mnt/lustre/d12b.sanity-quota/f12b.sanity-quota] [2048] - unlinked 0 (time 1778538273 ; total 0 ; last 0) total: 2048 unlinks in 21 seconds: 97.523811 unlinks/second sleep 5 for ZFS MDS Waiting for MDT destroys to complete Create files on mdt1 after space freed from mdt0... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: [createmany] [-m] [/mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota] [1024] mknod(/mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota988) error: Disk quota exceeded total: 988 create in 4.62 seconds: 213.96 ops/second Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 1470 0 0 - 980 0 2048 - lustre-MDT0000_UUID 0 - 0 - 0 - 1024 - lustre-MDT0001_UUID 1470 - 0 - 980 - 1024 - lustre-OST0000_UUID 0 - 0 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 2048, total allocated block limit: 0 Files for user (quota_usr): File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota182 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798761659 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-05-11 18:25:03.000000000 -0400 Modify: 2026-05-11 18:25:03.000000000 -0400 Change: 2026-05-11 18:25:03.000000000 -0400 Birth: 2026-05-11 18:25:04.000000000 -0400 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota971 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798762448 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-05-11 18:25:07.000000000 -0400 Modify: 2026-05-11 18:25:07.000000000 -0400 Change: 2026-05-11 18:25:07.000000000 -0400 Birth: 2026-05-11 18:25:08.000000000 -0400 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota906 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798762383 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-05-11 18:25:07.000000000 -0400 Modify: 2026-05-11 18:25:07.000000000 -0400 Change: 2026-05-11 18:25:07.000000000 -0400 Birth: 2026-05-11 18:25:07.000000000 -0400 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota892 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798762369 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-05-11 18:25:07.000000000 -0400 Modify: 2026-05-11 18:25:07.000000000 -0400 Change: 2026-05-11 18:25:07.000000000 -0400 Birth: 2026-05-11 18:25:07.000000000 -0400 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 1470 0 0 - 980 0 0 - lustre-MDT0000_UUID 0 - 0 - 0 - 0 - lustre-MDT0001_UUID 1470 - 0 - 980 - 0 - lustre-OST0000_UUID 0 - 0 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 Files for group (quota_usr): File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota182 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798761659 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-05-11 18:25:03.000000000 -0400 Modify: 2026-05-11 18:25:03.000000000 -0400 Change: 2026-05-11 18:25:03.000000000 -0400 Birth: 2026-05-11 18:25:04.000000000 -0400 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota971 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798762448 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-05-11 18:25:07.000000000 -0400 Modify: 2026-05-11 18:25:07.000000000 -0400 Change: 2026-05-11 18:25:07.000000000 -0400 Birth: 2026-05-11 18:25:08.000000000 -0400 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota906 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798762383 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-05-11 18:25:07.000000000 -0400 Modify: 2026-05-11 18:25:07.000000000 -0400 Change: 2026-05-11 18:25:07.000000000 -0400 Birth: 2026-05-11 18:25:07.000000000 -0400 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota892 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798762369 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-05-11 18:25:07.000000000 -0400 Modify: 2026-05-11 18:25:07.000000000 -0400 Change: 2026-05-11 18:25:07.000000000 -0400 Birth: 2026-05-11 18:25:07.000000000 -0400 Disk quotas for prj 1000 (pid 1000): Filesystem kbytes quota limit grace files quota limit grace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000_UUID 0 - 0 - 0 - 0 - lustre-MDT0001_UUID 0 - 0 - 0 - 0 - lustre-OST0000_UUID 0 - 0 - - - - - lustre-OST0001_UUID 0 - 0 - - - - - Total allocated inode limit: 0, total allocated block limit: 0 Files for project (1000): sanity-quota test_12b: @@@@@@ FAIL: rebalancing failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6459:error() = /home/green/git/lustre-release/lustre/tests/sanity-quota.sh:156:quota_error() = /home/green/git/lustre-release/lustre/tests/sanity-quota.sh:2523:test_12b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6817:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6864:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6705:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-quota.sh:2529:main() Dumping lctl log to /tmp/testlogs//sanity-quota.test_12b.*.1778538325.log rsync: chown "/tmp/testlogs/.sanity-quota.test_12b.debug_log.oleg415-server.1778538325.log.kTAz4k" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-quota.test_12b.dmesg.oleg415-server.1778538325.log.mgxytR" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS Waiting for MDT destroys to complete