== sanity-hsm test 31c: Restore a large aligned file and check size during restore ========================================================== 19:14:58 (1715210098)
Starting copytool 'agt1' on 'oleg123-client.virtnet' with cmdline 'lhsmtool_posix --archive-format=v2 --hsm-root=/tmp/arc1/sanity-hsm.test_31c/ --daemon --pid-file=/var/run/lhsmtool_posix.pid  "/mnt/lustre2"'
lfs mkdir: cannot resolve path '/mnt/lustre/d31c.sanity-hsm': Cannot send after transport endpoint shutdown (108)
lfs mkdir: '/mnt/lustre/d31c.sanity-hsm' is not on a Lustre filesystem: Cannot send after transport endpoint shutdown (108)
lfs setdirstripe: cannot create dir '/mnt/lustre/d31c.sanity-hsm': Cannot send after transport endpoint shutdown
mkdir: cannot create directory '/mnt/lustre': Cannot send after transport endpoint shutdown
rm: cannot remove '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
dd: failed to open '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
cannot create file '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm'
Cannot stat /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm: Cannot send after transport endpoint shutdown
Cannot stat /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm: Cannot send after transport endpoint shutdown
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2662:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210100.log.67X1iW" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210100.log.hUMNzf" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
Cannot stat /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm: Cannot send after transport endpoint shutdown
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210112.log.KLMEm3" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210112.log.ROgOcR" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210124.log.QqygMl" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210124.log.ymuwdE" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210135.log.WBXMzy" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210135.log.wmt3Kl" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210147.log.JbiWTF" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210147.log.KdMXEX" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210159.log.qDRGrR" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210159.log.5YaGGD" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210170.log.nlZEu3" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210170.log.hWhWtk" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210182.log.Pi2Nzc" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210182.log.qtHJfY" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210194.log.nP9Vax" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210194.log.vDiH6P" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210207.log.5i4nuR" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210207.log.RIZrnF" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210218.log.o1AX31" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210218.log.350hJk" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
stat: cannot stat '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm': Cannot send after transport endpoint shutdown
size seen is right:  == 
lfs hsm_state: get HSM state for '/mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm' failed: Cannot send after transport endpoint shutdown
 sanity-hsm test_31c: @@@@@@ FAIL: /home/green/git/lustre-release/lustre/utils/lfs hsm_state /mnt/lustre/d31c.sanity-hsm/f31c.sanity-hsm failed (run as root) 
  Trace dump:
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7030:error()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:11429:get_hsm_flags()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2678:restore_and_check_size()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2739:test_31c()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7375:run_one()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7435:run_one_logged()
  = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7261:run_test()
  = /home/green/git/lustre-release/lustre/tests/sanity-hsm.sh:2744:main()
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.debug_log.oleg123-server.1715210230.log.aIdaoe" failed: Operation not permitted (1)
rsync: chown "/tmp/testlogs/.sanity-hsm.test_31c.dmesg.oleg123-server.1715210230.log.LU2HO1" failed: Operation not permitted (1)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]
 restore is too long