== sanity-quota test 16c: lfs quota should preserve usage with an unavailable OST ========================================================== 16:02:53 (1789502573)
Waiting for MDT destroys to complete
Creating test directory
fail_val=0
fail_loc=0
running as uid/gid/euid/egid 60000/60000/60000/60000, groups: 60000
 [dd] [if=/dev/zero] [bs=1M] [of=/mnt/lustre/d16c.sanity-quota/f16c.sanity-quota] [count=50] [oflag=sync]
50+0 records in
50+0 records out
52428800 bytes (52 MB, 50 MiB) copied, 7.94898 s, 6.6 MB/s
Stopping /mnt/lustre-ost1 (opts:) on oleg344-server
Stopping client oleg344-client.virtnet /mnt/lustre (opts:)
Starting client: oleg344-client.virtnet:  -o user_xattr,flock 192.168.203.144@tcp:/lustre /mnt/lustre
oleg344-client.virtnet: executing wait_import_state (DISCONN|IDLE) osc.lustre-OST0000-osc-ffff9587c62db800.ost_server_uuid 50
osc.lustre-OST0000-osc-ffff9587c62db800.ost_server_uuid in DISCONN state after 0 sec
Start ost1: mount -t lustre -o localrecov  /dev/mapper/ost1_flakey /mnt/lustre-ost1
seq.cli-lustre-OST0000-super.width=65536
oleg344-server: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1
Started lustre-OST0000
affected facets: ost1
oleg344-server: oleg344-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1470 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1465 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1460 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1455 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1450 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1445 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1440 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1435 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1430 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1425 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1420 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1415 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1410 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1405 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1400 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1395 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1390 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: RECOVERING
oleg344-server: Waiting 1385 secs for *.lustre-OST0000.recovery_status recovery done. status: RECOVERING
oleg344-server: *.lustre-OST0000.recovery_status status: COMPLETE
192.168.203.144@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0
oleg344-server: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50
oleg344-server:  rpc test_16c: @@@@@@ FAIL: can't put import for os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid into FULL state after 50 sec, have  
oleg344-server:   Trace dump:
oleg344-server:   = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7530:error()
oleg344-server:   = /home/green/git/lustre-release/lustre/tests/test-framework.sh:9068:_wait_import_state()
oleg344-server:   = /home/green/git/lustre-release/lustre/tests/test-framework.sh:9090:wait_import_state()
oleg344-server:   = /home/green/git/lustre-release/lustre/tests/rpc.sh:20:main()
oleg344-server: oleg344-server.virtnet: executing check_logdir /tmp/test_logs/1789502726
oleg344-server: Dumping lctl log to /tmp/test_logs/1789502726/rpc.test_16c.*.1789502780.log
oleg344-server: Dumping logs only on local client.
pdsh@oleg344-client: oleg344-server: ssh exited with exit code 1
 sanity-quota test_16c: @@@@@@ FAIL: mds1: import is not in FULL state after 50 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7530:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:9183:_wait_osc_import_state()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:9198:wait_osc_import_state()
  = /home/green/git/lustre-release/lustre/tests/sanity-quota.sh:3275:test_16c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7907:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7970:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7784:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-quota.sh:3280:main()
Dumping lctl log to /tmp/testlogs//sanity-quota.test_16c.*.1789502787.log
rsync: chown "/tmp/testlogs/.sanity-quota.test_16c.debug_log.oleg344-server.1789502787.log.g2EPKW" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-quota.test_16c.dmesg.oleg344-server.1789502787.log.WeaCcx" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]
Delete files...
Wait for unlink objects finished...
Waiting for MDT destroys to complete