== sanityn test 119b: ltrash_purge: space-based purging with --auto-ost ========================================================== 00:18:46 (1779077926) mdd.lustre-MDT0000.trash_can_enable=1 mdd.lustre-MDT0001.trash_can_enable=1 mdd.lustre-MDT0000.trash_can_type=plain mdd.lustre-MDT0001.trash_can_type=plain Trash Can Type: plain Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Waiting for MDT destroys to complete OST0 has highest usage: 100% Increase OST0 space usage by creating 74 MB /mnt/lustre/d119b.sanityn/f1 74+0 records in 74+0 records out 77594624 bytes (78 MB, 74 MiB) copied, 1.15708 s, 67.1 MB/s Waiting for orphan cleanup... osp.lustre-OST0000-osc-MDT0000.old_sync_processed osp.lustre-OST0000-osc-MDT0001.old_sync_processed osp.lustre-OST0001-osc-MDT0000.old_sync_processed osp.lustre-OST0001-osc-MDT0001.old_sync_processed wait 40 secs maximumly for oleg135-server mds-ost sync done. Stripe /mnt/lustre/d119b.sanityn/f2 across all OSTs f2 Test --auto-ost with high threshold (no OSTs selected) ltrash_purge configuration for /mnt/lustre: - MDT_usage=0.0%, OST_usage=99.0%, max_age=0(s) - scan_interval=60(s), threads=4, del_count=10000 - enable_stats=off, auto_ost=on, auto_mdt=off - mode=dry-run trash projid is 4294967294 Trash can usage on projid 4294967294: blocks 0 inodes 1 MDT0000 stats - blocks=353529, free=347725, avail=312022, files=1024000, ffree=1010507 - usage=1.8%, i_usage=1.3% MDT0001 stats - blocks=353529, free=352067, avail=316364, files=1024000, ffree=1014644 - usage=0.5%, i_usage=0.9% OST0000 stats - blocks=958279, free=56733, avail=209, files=262144, ffree=258202 - usage=100.0%, i_usage=1.5% OST0001 stats - blocks=958279, free=927976, avail=870840, files=262144, ffree=258203 - usage=3.4%, i_usage=1.5% Current space usage in lustre: - Max MDT usage: 1.8%, Max OST usage: 100.0% OST0000 (lustre-OST0000_UUID): usage=100.0%, i_usage=1.5%, max=100.0% OST0001 (lustre-OST0001_UUID): usage=3.4%, i_usage=1.5%, max=3.4% - OST0000 (lustre-OST0000_UUID) with usage 100.0% Auto-selected 1 OST(s) for purging Start scanning and purging at 1779077942: Skip: /mnt/lustre/.ltrash is not a trash directory Add path /mnt/lustre/.ltrash/MDT0000 (depth=0) Add path /mnt/lustre/.ltrash/MDT0001 (depth=0) Add path /mnt/lustre/.ltrash/MDT0001/0x240000405:0xc:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/tnm (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x240000405:0x41:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x71:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6d:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x69:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6b:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb61e:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/500 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000007:0x1:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60b:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60c:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0 (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/tnm (depth=1) Add path /mnt/lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0 (depth=1) Purging object: /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb61e:0x0/f2 dry-run: would purge /mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb61e:0x0/f2 (size=0) Collected 55 dirs totally Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x79:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f8:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xe5:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/tnm' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60c:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe7:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbe3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc93:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc39:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc37:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc2e:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1c:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb60b:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc33:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x10bc7:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xa:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x401:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000007:0x1:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/500' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3f3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x3ee:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd0:0x657:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x1:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc95:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbec:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc1f:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc0c:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbf3:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xea:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x7e:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x77:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0x84:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbcc:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc25:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x240000404:0x5b1:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b73:0xb61e:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6b:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200001b71:0xf0:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x69:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x6d:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc06:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x240000404:0x5af:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x2400013a1:0x71:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x240000405:0x41:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc12:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/tnm' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x2000013a1:0x5:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xc28:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000/0x200000bd1:0xbd2:0x0' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001/0x240000405:0xc:0x0' Deleted 53 directories, 0 errors in depth 1 Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0000' Cannot read unrm xattr for '/mnt/lustre/.ltrash/MDT0001' Deleted 2 directories, 0 errors in depth 0 Totally deleted 55 directories, 0 errors ltrash_purge ended with exit code: 0 sanityn test_119b: @@@@@@ FAIL: Expected 'No OSTs exceed threshold' message not found Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7318:error() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:10654:test_119b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7694:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7757:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7572:run_test() = /home/green/git/lustre-release/lustre/tests/sanityn.sh:10708:main() Dumping lctl log to /tmp/testlogs//sanityn.test_119b.*.1779077943.log rsync: chown "/tmp/testlogs/.sanityn.test_119b.debug_log.oleg135-server.1779077943.log.8t500B" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanityn.test_119b.dmesg.oleg135-server.1779077943.log.4srWZs" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] mdd.lustre-MDT0000.trash_can_enable=0 mdd.lustre-MDT0001.trash_can_enable=0 mdd.lustre-MDT0000.trash_can_type=uid mdd.lustre-MDT0001.trash_can_type=uid