== sanity-lfsck test 11b: LFSCK can rebuild crashed last_id ========================================================== 18:49:01 (1736984941) Checking servers environments Checking clients oleg448-client.virtnet environments Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory loading modules on: 'oleg448-server' oleg448-server: oleg448-server.virtnet: executing load_modules_local oleg448-server: Loading modules from /home/green/git/lustre-release/lustre oleg448-server: detected 4 online CPUs by sysfs oleg448-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-MDT0000 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg448-server: mount.lustre: according to /etc/mtab lustre-ost1/ost1 is already mounted on /mnt/lustre-ost1 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 17 Start of lustre-ost1/ost1 on ost1 failed 17 Starting ost2: -o localrecov lustre-ost2/ost2 /mnt/lustre-ost2 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg448-client.virtnet: -o user_xattr,flock oleg448-server@tcp:/lustre /mnt/lustre Starting client oleg448-client.virtnet: -o user_xattr,flock oleg448-server@tcp:/lustre /mnt/lustre Started clients oleg448-client.virtnet: 192.168.204.148@tcp:/lustre on /mnt/lustre type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800a95ef800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800a95ef800.idle_timeout=debug disable quota as required set fail_loc=0x160d to skip the updating LAST_ID on-disk fail_loc=0x160d - open/close 0 (time 1736984992.77 total 20.02 last 0.00) - open/close 1 (time 1736985012.79 total 40.03 last 0.05) - open/close 2 (time 1736985032.80 total 60.04 last 0.05) - open/close 3 (time 1736985052.81 total 80.05 last 0.05) - open/close 4 (time 1736985072.82 total 100.07 last 0.05) - open/close 5 (time 1736985092.83 total 120.08 last 0.05) - open/close 6 (time 1736985112.84 total 140.08 last 0.05) - open/close 7 (time 1736985132.85 total 160.09 last 0.05) - open/close 8 (time 1736985152.86 total 180.10 last 0.05) - open/close 9 (time 1736985172.87 total 200.11 last 0.05) - open/close 10 (time 1736985192.88 total 220.12 last 0.05) - open/close 11 (time 1736985212.89 total 240.13 last 0.05) - open/close 12 (time 1736985232.90 total 260.15 last 0.05) - open/close 13 (time 1736985252.91 total 280.16 last 0.05) - open/close 14 (time 1736985272.92 total 300.17 last 0.05) - open/close 15 (time 1736985292.93 total 320.17 last 0.05) - open/close 16 (time 1736985312.93 total 340.18 last 0.05) - open/close 17 (time 1736985332.94 total 360.19 last 0.05) - open/close 18 (time 1736985352.95 total 380.19 last 0.05) - open/close 19 (time 1736985372.96 total 400.20 last 0.05) - open/close 20 (time 1736985392.97 total 420.21 last 0.05) - open/close 21 (time 1736985412.98 total 440.22 last 0.05) - open/close 22 (time 1736985432.99 total 460.23 last 0.05) - open/close 23 (time 1736985452.99 total 480.24 last 0.05) - open/close 24 (time 1736985473.00 total 500.24 last 0.05) - open/close 25 (time 1736985493.00 total 520.25 last 0.05) - open/close 26 (time 1736985513.01 total 540.26 last 0.05) - open/close 27 (time 1736985533.02 total 560.26 last 0.05) - open/close 28 (time 1736985553.02 total 580.27 last 0.05) - open/close 29 (time 1736985573.03 total 600.27 last 0.05) - open/close 30 (time 1736985593.03 total 620.28 last 0.05) - open/close 31 (time 1736985613.04 total 640.28 last 0.05) total: 32 open/close in 640.28 seconds: 0.05 ops/second 192.168.204.148@tcp:/lustre /mnt/lustre lustre rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt 0 0 Stopping client oleg448-client.virtnet /mnt/lustre (opts:) Stopping /mnt/lustre-ost1 (opts:) on oleg448-server fail_loc=0x215 Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 oleg448-server: oleg448-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8 pdsh@oleg448-client: oleg448-server: ssh exited with exit code 1 Started lustre-OST0000 the on-disk LAST_ID should be smaller than the expected one /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh: line 1502: [: 1: unary operator expected sanity-lfsck test_11b: @@@@@@ FAIL: (4) expect id_used '1' > id_ost1 '' Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6421:error() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:1503:test_11b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6756:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6803:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:6644:run_test() = /home/green/git/lustre-release/lustre/tests/sanity-lfsck.sh:1532:main() Dumping lctl log to /tmp/testlogs//sanity-lfsck.test_11b.*.1736985695.log rsync: chown "/tmp/testlogs/.sanity-lfsck.test_11b.debug_log.oleg448-server.1736985695.log.sC8xRR" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity-lfsck.test_11b.dmesg.oleg448-server.1736985695.log.SgIiLn" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1651) [generator=3.1.2]