== sanity-quota test 12b: Inode quota rebalancing ======== 16:53:01 (1768254781) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 Waiting 90s for 'u' Updated after 2s: want 'u' got 'u' Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 -u quota_usr inodes:0 Create 2048 files on mdt0... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [createmany] [-m] [/mnt/lustre/d12b.sanity-quota/f12b.sanity-quota] [2048] mdt0 created 1536 Create files on mdt1... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [createmany] [-m] [/mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota] [1] mknod(/mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota0) error: Disk quota exceeded total: 0 create in 0.04 seconds: 0.00 ops/second Free space from mdt0... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [unlinkmany] [/mnt/lustre/d12b.sanity-quota/f12b.sanity-quota] [1536] - unlinked 0 (time 1768254805 ; total 0 ; last 0) total: 1536 unlinks in 15 seconds: 102.400002 unlinks/second sleep 5 for ZFS MDS Waiting for MDT destroys to complete Create files on mdt1 after space freed from mdt0... running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [createmany] [-m] [/mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota] [1024] mknod(/mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota674) error: Disk quota exceeded total: 674 create in 4.69 seconds: 143.82 ops/second Disk quotas for usr quota_usr (uid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 1002 0 0 - 668 0 2048 - lustre-MDT0000 0 - 0 - 0 - 1024 - lustre-MDT0001 1002 - 0 - 668 - 1024 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 2048, total allocated block limit: 0 Files for user (quota_usr), count=674: File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota661 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798763163 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-01-12 16:53:53.000000000 -0500 Modify: 2026-01-12 16:53:53.000000000 -0500 Change: 2026-01-12 16:53:53.000000000 -0500 Birth: 2026-01-12 16:53:53.000000000 -0500 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota616 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798763118 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-01-12 16:53:52.000000000 -0500 Modify: 2026-01-12 16:53:52.000000000 -0500 Change: 2026-01-12 16:53:52.000000000 -0500 Birth: 2026-01-12 16:53:53.000000000 -0500 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota251 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798762753 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-01-12 16:53:50.000000000 -0500 Modify: 2026-01-12 16:53:50.000000000 -0500 Change: 2026-01-12 16:53:50.000000000 -0500 Birth: 2026-01-12 16:53:50.000000000 -0500 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota226 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798762728 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-01-12 16:53:49.000000000 -0500 Modify: 2026-01-12 16:53:49.000000000 -0500 Change: 2026-01-12 16:53:49.000000000 -0500 Birth: 2026-01-12 16:53:50.000000000 -0500 Disk quotas for grp quota_usr (gid 60000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 1002 0 0 - 668 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 1002 - 0 - 668 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Files for group (quota_usr), count=674: File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota661 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798763163 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-01-12 16:53:53.000000000 -0500 Modify: 2026-01-12 16:53:53.000000000 -0500 Change: 2026-01-12 16:53:53.000000000 -0500 Birth: 2026-01-12 16:53:53.000000000 -0500 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota616 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798763118 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-01-12 16:53:52.000000000 -0500 Modify: 2026-01-12 16:53:52.000000000 -0500 Change: 2026-01-12 16:53:52.000000000 -0500 Birth: 2026-01-12 16:53:53.000000000 -0500 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota251 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798762753 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-01-12 16:53:50.000000000 -0500 Modify: 2026-01-12 16:53:50.000000000 -0500 Change: 2026-01-12 16:53:50.000000000 -0500 Birth: 2026-01-12 16:53:50.000000000 -0500 File: /mnt/lustre/d12b.sanity-quota-1/f12b.sanity-quota226 Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 162129603798762728 Links: 1 Access: (0444/-r--r--r--) Uid: (60000/quota_usr) Gid: (60000/quota_usr) Access: 2026-01-12 16:53:49.000000000 -0500 Modify: 2026-01-12 16:53:49.000000000 -0500 Change: 2026-01-12 16:53:49.000000000 -0500 Birth: 2026-01-12 16:53:50.000000000 -0500 Disk quotas for prj (pid 1000): Filesystem kbytes bquota blimit bgrace files iquota ilimit igrace /mnt/lustre 0 0 0 - 0 0 0 - lustre-MDT0000 0 - 0 - 0 - 0 - lustre-MDT0001 0 - 0 - 0 - 0 - lustre-OST0000 0 - 0 - 0 - 0 - lustre-OST0001 0 - 0 - 0 - 0 - Total allocated inode limit: 0, total allocated block limit: 0 Files for project (1000), count=0: sanity-quota test_12b: @@@@@@ FAIL: rebalancing failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7317:error() = /home/green/git/lustre-release/lustre/tests/sanity-quota.sh:156:quota_error() = /home/green/git/lustre-release/lustre/tests/sanity-quota.sh:2772:test_12b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7693:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7756:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7571:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-quota.sh:2778:main() Dumping lctl log to /tmp/testlogs//sanity-quota.test_12b.*.1768254868.log Delete files... Wait for unlink objects finished... rsync: chown "/tmp/testlogs/.sanity-quota.test_12b.debug_log.oleg646-server.1768254868.log.4dLMYk" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-quota.test_12b.dmesg.oleg646-server.1768254868.log.2zfh9p" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] sleep 5 for ZFS MDS Waiting for MDT destroys to complete