== sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 10:48:38 (1761317318) Checking servers environments Checking clients oleg421-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions loading modules on: 'oleg421-server' oleg421-server: oleg421-server.virtnet: executing load_modules_local oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 712: echo: write error: Device or resource busy oleg421-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 713: echo: write error: Device or resource busy oleg421-server: Loading modules from /home/green/git/lustre-release/lustre oleg421-server: detected 4 online CPUs by sysfs oleg421-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-MDT0001 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg421-server: mount.lustre: according to /etc/mtab /dev/mapper/ost1_flakey is already mounted on /mnt/lustre-ost1 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 17 Start of /dev/mapper/ost1_flakey on ost1 failed 17 Starting ost2: -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg421-client.virtnet: -o user_xattr,flock oleg421-server@tcp:/lustre /mnt/lustre Starting client oleg421-client.virtnet: -o user_xattr,flock oleg421-server@tcp:/lustre /mnt/lustre Started clients oleg421-client.virtnet: 192.168.204.121@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8d01c553e000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8d01c553e000.idle_timeout=debug disable quota as required set fail_loc=0x160d to skip the updating LAST_ID on-disk fail_loc=0x160d - open/close 0 (time 1761317425.31 total 20.23 last 0.00) - open/close 1 (time 1761317445.79 total 40.71 last 0.05) - open/close 2 (time 1761317466.27 total 61.19 last 0.05) - open/close 3 (time 1761317486.74 total 81.66 last 0.05) - open/close 4 (time 1761317507.23 total 102.15 last 0.05) - open/close 5 (time 1761317527.71 total 122.63 last 0.05) - open/close 6 (time 1761317548.19 total 143.11 last 0.05) - open/close 7 (time 1761317568.66 total 163.58 last 0.05) - open/close 8 (time 1761317589.15 total 184.07 last 0.05) - open/close 9 (time 1761317609.63 total 204.55 last 0.05) - open/close 10 (time 1761317630.11 total 225.03 last 0.05) - open/close 11 (time 1761317650.59 total 245.51 last 0.05) - open/close 12 (time 1761317671.07 total 265.99 last 0.05) - open/close 13 (time 1761317691.55 total 286.47 last 0.05) - open/close 14 (time 1761317712.03 total 306.95 last 0.05) - open/close 15 (time 1761317732.50 total 327.42 last 0.05) - open/close 16 (time 1761317752.99 total 347.91 last 0.05) - open/close 17 (time 1761317773.46 total 368.38 last 0.05) - open/close 18 (time 1761317793.95 total 388.87 last 0.05) - open/close 19 (time 1761317814.42 total 409.34 last 0.05) - open/close 20 (time 1761317834.91 total 429.83 last 0.05) - open/close 21 (time 1761317855.38 total 450.30 last 0.05) - open/close 22 (time 1761317875.88 total 470.80 last 0.05) - open/close 23 (time 1761317896.34 total 491.26 last 0.05) - open/close 24 (time 1761317916.84 total 511.76 last 0.05) - open/close 25 (time 1761317937.30 total 532.22 last 0.05) - open/close 26 (time 1761317957.78 total 552.70 last 0.05) - open/close 27 (time 1761317978.28 total 573.20 last 0.05) - open/close 28 (time 1761317998.74 total 593.66 last 0.05) - open/close 29 (time 1761318019.22 total 614.14 last 0.05) - open/close 30 (time 1761318039.71 total 634.63 last 0.05) - open/close 31 (time 1761318060.20 total 655.12 last 0.05) total: 32 open/close in 655.12 seconds: 0.05 ops/second 192.168.204.121@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,encrypt 0 0 Stopping client oleg421-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost1 (opts:) on oleg421-server fail_loc=0x215 Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 oleg421-server: oleg421-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg421-client: oleg421-server: ssh exited with exit code 1 Started lustre-OST0000 the on-disk LAST_ID should be smaller than the expected one /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh: line 1502: [: 1: unary operator expected sanity-lfsck test_11b: @@@@@@ FAIL: (4) expect id_used '1' > id_ost1 '' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6482:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:1503:test_11b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6840:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6887:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6728:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:1532:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_11b.*.1761318194.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_11b.debug_log.oleg421-server.1761318194.log.oTLWgp" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_11b.dmesg.oleg421-server.1761318194.log.5Yjh1S" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3]