== sanity-quota test 16c: lfs quota should preserve usage with an unavailable OST ========================================================== 16:14:32 (1789503272) sleep 5 for ZFS MDS Waiting for MDT destroys to complete Creating test directory fail_val=0 fail_loc=0 running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000 [dd] [if=/dev/urandom] [bs=1M] [of=/mnt/lustre/d16c.sanity-quota/f16c.sanity-quota] [count=50] [oflag=sync] 50+0 records in 50+0 records out 52428800 bytes (52 MB, 50 MiB) copied, 14.4021 s, 3.6 MB/s Stopping /mnt/lustre-ost1 (opts:) on oleg151-server Stopping client oleg151-client.virtnet /mnt/lustre (opts:) Starting client: oleg151-client.virtnet: -o user_xattr,flock 192.168.201.151@tcp:/lustre /mnt/lustre oleg151-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff976a5a6ec000.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff976a5a6ec000.ost_server_uuid in DISCONN state after 0 sec Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg151-server: oleg151-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 Started lustre-OST0000 affected facets: ost1 oleg151-server: oleg151-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1405 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1400 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1395 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1390 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: RECOVERING oleg151-server: Waiting 1385 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING oleg151-server: *.lustre-OST0000.recovery_status status: COMPLETE 192.168.201.151@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 oleg151-server: oleg151-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg151-server: rpc test_16c: @@@@@@ FAIL: can't put import for os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid into FULL state after 50 sec, have oleg151-server: Trace dump: oleg151-server: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7530:error() oleg151-server: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:9068:_wait_import_state() oleg151-server: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:9090:wait_import_state() oleg151-server: = /home/green/git/lustre-release/lustre/tests/rpc.sh:20:main() oleg151-server: oleg151-server.virtnet: executing check_logdir /tmp/test_logs/1789503434 oleg151-server: Dumping lctl log to /tmp/test_logs/1789503434/rpc.test_16c.*.1789503490.log oleg151-server: Dumping logs only on local client. pdsh@oleg151-client: oleg151-server: ssh exited with exit code 1 sanity-quota test_16c: @@@@@@ FAIL: mds1: import is not in FULL state after 50 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7530:error() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:9183:_wait_osc_import_state() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:9198:wait_osc_import_state() = /home/green/git/lustre-release/lustre/tests/sanity-quota.sh:3275:test_16c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7907:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7970:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7784:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-quota.sh:3280:main() Dumping lctl log to /tmp/testlogs//sanity-quota.test_16c.*.1789503495.log rsync: chown "/tmp/testlogs/.sanity-quota.test_16c.debug_log.oleg151-server.1789503495.log.wMlSzy" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-quota.test_16c.dmesg.oleg151-server.1789503495.log.vV8h95" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] Delete files... Wait for unlink objects finished... sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete