-----============= acceptance-small: sanity ============----- Wed Aug 26 06:04:23 EDT 2026 MGS_OS_ID_LIKE=rhel centos fedora rocky MGS_OS_VERSION_ID=8.10 MGS_OS_ID=rocky MGS_OS_VERSION_CODE=134873088 MDS1_OS_VERSION_ID=8.10 MDS1_OS_VERSION_CODE=134873088 MDS1_OS_ID_LIKE=rhel centos fedora rocky MDS1_OS_ID=rocky OST1_OS_VERSION_CODE=134873088 OST1_OS_ID_LIKE=rhel centos fedora rocky OST1_OS_VERSION_ID=8.10 OST1_OS_ID=rocky CLIENT_OS_ID=rocky CLIENT_OS_VERSION_CODE=134873088 CLIENT_OS_VERSION_ID=8.10 CLIENT_OS_ID_LIKE=rhel centos fedora rocky oleg315-server: /home/green/git/lustre-release/lustre/tests/except/sanity.ex - see CLIENT_VERSION > v2_15_63-134-gdacc4b6d38 (34683210 > 34553734) for LU-15963, go 312 - see CLIENT_VERSION > v2_15_63-239-gab4ede45b4 (34683210 > 34553839) for LU-17856, go 56wa 56xe - see CLIENT_VERSION >= v2_15_7-16-gab522d54ad (34683210 >= 34539280) for LU-16203, go 60i - see CLIENT_VERSION > v2_16_54-54-ga2e3a2f5a3 (34683210 > 34616886) for LU-14590, go 0d - see CLIENT_VERSION >= v2_17_50-196-g63ee4af3bb (34683210 >= 34681540) for LU-19620, go 60b - see CLIENT_VERSION >= v2_17_50-34-g089f438ba9 (34683210 >= 34681378) for LU-19692, go 77k - see CLIENT_VERSION >= v2_17_50-71-g2cbe29c282 (34683210 >= 34681415) for LU-19644, go 150ib - see CLIENT_VERSION >= v2_17_50-72-ge1f272796c (34683210 >= 34681416) for LU-19743, go 27z - see CLIENT_VERSION >= v2_17_52-99-g84242c4358 (34683210 >= 34681955) for LU-20010, go 56aa - see CLIENT_VERSION >= v2_17_55-19-g2868627130 (34683210 >= 34682643) for LU-20435, go 430a - need MDS1_VERSION <= 2.16.61-1-g89cf292a8c2 (34683210 <= 34618625) for LU-18938, skip 360 - see MDS1_VERSION <= CLIENT_VERSION (34683210 <= 34683210) for LU-18562, go 270a - need MDS1_VERSION < v2_14_55-100-g8a84c7f9c7 (34683210 < 34486116) for LU-14927, skip 0f - need OST1_VERSION < v2_17_50-410-gbd8e7b9555 (34683210 < 34681754) for LU-12550, skip 216 excepting tests: 42a 42c 42b 118c 118d 407 119i 216 817 411a skipping tests SLOW=no: 27m 60i 64b 68 71 135 136 230d 300o 842 51c 51e 834 === sanity: start setup 06:04:49 (1787738689) === oleg315-client.virtnet: executing check_config_client /mnt/lustre oleg315-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg315-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93f688a9d800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93f688a9d800.idle_timeout=debug disable quota as required oleg315-server: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity: finish setup 06:05:30 (1787738730) === running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8239] preparing for tests involving mounts mke2fs 1.47.3-wc2 (11-Nov-2025) debug=all == sanity test 60a: llog_test run from kernel module and test llog_reader ========================================================== 06:05:36 (1787738736) SKIP: sanity test_60a missing subtest run-llog.sh SKIP 60a (4s) == sanity test 60b: limit repeated messages from CERROR/CWARN ========================================================== 06:05:40 (1787738740) PASS 60b (7s) == sanity test 60c: unlink file when mds full ============ 06:05:47 (1787738747) create 5000 files - open/close 512 (time 1787738761.85 total 10.02 last 51.09) - open/close 975 (time 1787738771.86 total 20.03 last 46.25) - open/close 1442 (time 1787738781.89 total 30.06 last 46.59) - open/close 1865 (time 1787738791.89 total 40.06 last 42.29) - open/close 2384 (time 1787738801.92 total 50.08 last 51.76) - open/close 2804 (time 1787738811.94 total 60.10 last 41.92) - open/close 3319 (time 1787738821.95 total 70.12 last 51.42) - open/close 3846 (time 1787738831.97 total 80.14 last 52.60) - open/close 4345 (time 1787738842.00 total 90.17 last 49.76) - open/close 4787 (time 1787738852.02 total 100.18 last 44.13) total: 5000 open/close in 104.13 seconds: 48.02 ops/second fail_loc=0x80000137 - unlinked 0 (time 1787738860 ; total 0 ; last 0) total: 5000 unlinks in 69 seconds: 72.463768 unlinks/second fail_loc=0 PASS 60c (188s) == sanity test 60d: test printk console message masking == 06:08:56 (1787738936) printk=0 emerg PASS 60d (7s) == sanity test 60e: no space while new llog is being created ========================================================== 06:09:03 (1787738943) fail_loc=0x15b PASS 60e (8s) == sanity test 60f: change debug_path works ============== 06:09:11 (1787738951) debug_path=/tmp/f60f.sanity fail_loc=0x8000050e striped dir -i0 -c2 -H crush /mnt/lustre/d60f.sanity ls: cannot access '/tmp/f60f.sanity*': No such file or directory 0 /tmp/f60f.sanity.1787738953.14787 debug_path=/tmp/lustre-log PASS 60f (7s) == sanity test 60g: transaction abort won't cause MDT hung ========================================================== 06:09:18 (1787738958) striped dir -i0 -c2 -H crush /mnt/lustre/d60g.sanity /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 5174: 15551 Killed ( local index=0; while true; do $LFS setdirstripe -i $(($index % $MDSCOUNT)) -c $MDSCOUNT $DIR/$tdir/subdir$index 2> /dev/null; mkdir $DIR/$tdir/subdir$index 2> /dev/null; rmdir $DIR/$tdir/subdir$index 2> /dev/null; index=$((index + 1)); done ) Started LFSCK on the device lustre-MDT0000: scrub namespace /mnt/lustre/d60g.sanity: subdir112 subdir208 subdir218 subdir22 subdir229 subdir302 subdir306 subdir314 subdir33 subdir340 subdir384 subdir387 subdir61 subdir83 /mnt/lustre/d60g.sanity/subdir112: ls: closing directory '/mnt/lustre/d60g.sanity/subdir112': No such file or directory /mnt/lustre/d60g.sanity/subdir208: /mnt/lustre/d60g.sanity/subdir218: /mnt/lustre/d60g.sanity/subdir22: /mnt/lustre/d60g.sanity/subdir229: /mnt/lustre/d60g.sanity/subdir302: /mnt/lustre/d60g.sanity/subdir306: /mnt/lustre/d60g.sanity/subdir314: /mnt/lustre/d60g.sanity/subdir33: /mnt/lustre/d60g.sanity/subdir340: /mnt/lustre/d60g.sanity/subdir384: /mnt/lustre/d60g.sanity/subdir387: /mnt/lustre/d60g.sanity/subdir61: /mnt/lustre/d60g.sanity/subdir83: PASS 60g (121s) == sanity test 60h: striped directory with missing stripes can be accessed ========================================================== 06:11:19 (1787739079) fail_loc=0x80000188 /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10941: /mnt/lustre/d60h.sanity-0x80000188/2: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10941: /mnt/lustre/d60h.sanity-0x80000188/3: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10941: /mnt/lustre/d60h.sanity-0x80000188/4: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10941: /mnt/lustre/d60h.sanity-0x80000188/7: No such device /home/green/git/lustre-release/lustre/tests/sanity.sh: line 10941: /mnt/lustre/d60h.sanity-0x80000188/8: No such device lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush mdtidx FID[seq:oid:ver] 0 [0x200000400:0xce:0x0] 0 [0:0x0:0x0] lmv_stripe_count: 3 lmv_stripe_offset: 1 lmv_hash_type: crush,migrating,fixed mdtidx FID[seq:oid:ver] 1 [0x240000400:0xcc:0x0] 0 [0x200000400:0xce:0x0] 0 [0:0x0:0x0] /mnt/lustre/d60h.sanity-0x80000188 ~ cmp: EOF on 0 which is empty sanity test_60h: @@@@@@ FAIL: 0 data mismatch Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:10951:test_60h() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:10957:main() Dumping lctl log to /tmp/testlogs//sanity.test_60h.*.1787739086.log rsync: chown "/tmp/testlogs/.sanity.test_60h.debug_log.oleg315-server.1787739086.log.i0gXPw" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_60h.dmesg.oleg315-server.1787739086.log.VHH3IB" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 60h (14s) SKIP: sanity test_60i skipping SLOW test 60i == sanity test 60j: llog_reader reports corruptions ====== 06:11:35 (1787739095) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' mdd.lustre-MDT0000.changelog_mask=ALL mdd.lustre-MDT0001.changelog_mask=ALL lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records total: 100 open/close in 1.94 seconds: 51.68 ops/second - unlinked 0 (time 1787739108 ; total 0 ; last 0) total: 100 unlinks in 1 seconds: 100.000000 unlinks/second oleg315-server: debugfs 1.47.3-wc2 (11-Nov-2025) SKIP: sanity test_60j path oi.1/0x1:0xc:0x0 is not in 'O/1/d/' format lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 SKIP 60j (24s) == sanity test 61a: mmap() writes don't make sync hang ========================================================================== 06:12:00 (1787739120) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0316086 s, 130 kB/s PASS 61a (8s) == sanity test 61b: mmap() of unstriped file is successful ========================================================== 06:12:08 (1787739128) PASS 61b (7s) == sanity test 63a: Verify oig_wait interruption does not crash ================================================================= 06:12:15 (1787739135) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20825 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20830 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20834 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20837 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20841 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20845 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20849 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20853 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20857 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11075: 20861 Terminated dd if=/dev/zero of=$DIR/f63 bs=8k checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4044 1267260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2900 1268404 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 13864 3592964 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 15412 7198436 1% /mnt/lustre pass grant check: client:246775808 server:246775808 PASS 63a (68s) == sanity test 63b: async write errors should be returned to fsync ============================================================= 06:13:24 (1787739204) debug=-1 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0161097 s, 254 kB/s fail_loc=0x80000406 fsync: Input/output error checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4044 1267260 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2900 1268404 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1548 3605472 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 1576 3605252 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 3124 7210724 1% /mnt/lustre pass grant check: client:246775808 server:246775808 PASS 63b (12s) == sanity test 63c: test sync_on_close=1 ================= 06:13:36 (1787739216) llite.lustre-ffff93f688a9d800.sync_on_close=1 total: 20 open/close in 0.52 seconds: 38.11 ops/second total 92 -rw-r--r-- 1 root root 90112 Aug 26 06:13 f63c.sanity-1 -rw-r--r-- 1 root root 6 Aug 26 06:13 f63c.sanity-empty-0 -rw-r--r-- 1 root root 6 Aug 26 06:13 f63c.sanity-empty-1 -rw-r--r-- 1 root root 7 Aug 26 06:13 f63c.sanity-empty-10 -rw-r--r-- 1 root root 7 Aug 26 06:13 f63c.sanity-empty-11 -rw-r--r-- 1 root root 7 Aug 26 06:13 f63c.sanity-empty-12 -rw-r--r-- 1 root root 7 Aug 26 06:13 f63c.sanity-empty-13 -rw-r--r-- 1 root root 7 Aug 26 06:13 f63c.sanity-empty-14 -rw-r--r-- 1 root root 7 Aug 26 06:13 f63c.sanity-empty-15 -rw-r--r-- 1 root root 7 Aug 26 06:13 f63c.sanity-empty-16 -rw-r--r-- 1 root root 7 Aug 26 06:13 f63c.sanity-empty-17 -rw-r--r-- 1 root root 7 Aug 26 06:13 f63c.sanity-empty-18 -rw-r--r-- 1 root root 7 Aug 26 06:13 f63c.sanity-empty-19 -rw-r--r-- 1 root root 6 Aug 26 06:13 f63c.sanity-empty-2 -rw-r--r-- 1 root root 6 Aug 26 06:13 f63c.sanity-empty-3 -rw-r--r-- 1 root root 6 Aug 26 06:13 f63c.sanity-empty-4 -rw-r--r-- 1 root root 6 Aug 26 06:13 f63c.sanity-empty-5 -rw-r--r-- 1 root root 6 Aug 26 06:13 f63c.sanity-empty-6 -rw-r--r-- 1 root root 6 Aug 26 06:13 f63c.sanity-empty-7 -rw-r--r-- 1 root root 6 Aug 26 06:13 f63c.sanity-empty-8 -rw-r--r-- 1 root root 6 Aug 26 06:13 f63c.sanity-empty-9 error: set_param: setting /sys/kernel/debug/lnet/debug=rpctrace+dlmtrace+inode+entry: Invalid argument error: set_param: setting 'debug'='rpctrace+dlmtrace+inode+entry': Invalid argument 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 2.50807 s, 6.7 MB/s llite.lustre-ffff93f688a9d800.sync_on_close=0 PASS 63c (17s) == sanity test 64a: verify filter grant calculations (in kernel) =============================================================== 06:13:53 (1787739233) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4048 1267256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2904 1268400 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9740 3597280 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9860 3597160 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19600 7194440 1% /mnt/lustre osc.lustre-OST0000-osc-ffff93f688a9d800.cur_grant_bytes=142983168 osc.lustre-OST0001-osc-ffff93f688a9d800.cur_grant_bytes=484638720 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4048 1267256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2904 1268400 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9740 3597280 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9860 3597160 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19600 7194440 1% /mnt/lustre pass grant check: client:627621888 server:627621888 PASS 64a (8s) SKIP: sanity test_64b skipping SLOW test 64b == sanity test 64c: verify grant shrink ================== 06:14:03 (1787739243) osc.lustre-OST0000-osc-ffff93f688a9d800.cur_grant_bytes=0 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4048 1267256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2904 1268400 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9740 3597280 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9860 3597160 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19600 7194440 1% /mnt/lustre pass grant check: client:627621888 server:627621888 PASS 64c (8s) == sanity test 64d: check grant limit exceed ============= 06:14:11 (1787739251) 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB, 1000 MiB) copied, 36.5833 s, 28.7 MB/s /home/green/git/lustre-release/lustre/tests/sanity.sh: line 11329: kill: (24459) - No such process checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4048 1267256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2904 1268400 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 1033744 2560916 29% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9860 3597160 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 1043604 6158076 15% /mnt/lustre pass grant check: client:969277440 server:969277440 Waiting for MDT destroys to complete PASS 64d (63s) == sanity test 64e: check grant consumption (no grant allocation) ========================================================== 06:15:14 (1787739314) debug=+cache Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre fail_loc=0x725 1+0 records in 1+0 records out 1519616 bytes (1.5 MB, 1.4 MiB) copied, 0.110628 s, 13.7 MB/s fail_loc=0 Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre fail_loc=0x725 fail_loc=0 PASS 64e (19s) == sanity test 64f: check grant consumption (with grant allocation) ========================================================== 06:15:33 (1787739333) debug=+cache Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre 1+0 records in 1+0 records out 3051520 bytes (3.1 MB, 2.9 MiB) copied, 0.178541 s, 17.1 MB/s Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre llite.lustre-ffff93f685481800.hybrid_io=0 fail_loc=0x50a fail_val=3 1+0 records in 1+0 records out 3051520 bytes (3.1 MB, 2.9 MiB) copied, 0.440802 s, 6.9 MB/s fail_loc=0 fail_val=0 PASS 64f (15s) == sanity test 64g: grant shrink on MDT ================== 06:15:49 (1787739349) 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0175434 s, 7.5 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0111853 s, 11.7 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0259698 s, 5.0 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0195578 s, 6.7 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0288082 s, 4.5 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0132776 s, 9.9 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0198599 s, 6.6 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0228122 s, 5.7 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0348889 s, 3.8 MB/s 1+0 records in 1+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0308167 s, 4.3 MB/s 1089536 grants, 0 pages 1089536 grants, 8 pages 1089536 grants, 8 pages 1089536 grants, 8 pages mdc.lustre-MDT0000-mdc-ffff93f685481800.grant_shrink_interval=5 mdc.lustre-MDT0001-mdc-ffff93f685481800.grant_shrink_interval=5 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 8 pages 0 grants, 0 pages mdc.lustre-MDT0000-mdc-ffff93f685481800.grant_shrink_interval=1200 mdc.lustre-MDT0001-mdc-ffff93f685481800.grant_shrink_interval=1200 PASS 64g (50s) == sanity test 64h: grant shrink on read ================= 06:16:40 (1787739400) osc.lustre-OST0000-osc-ffff93f685481800.grant_shrink=1 osc.lustre-OST0000-osc-ffff93f685481800.grant_shrink_interval=10 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.83954 s, 5.7 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0336098 s, 122 kB/s PASS 64h (19s) == sanity test 64i: shrink on reconnect ================== 06:16:59 (1787739419) 64+0 records in 64+0 records out 67108864 bytes (67 MB, 64 MiB) copied, 10.1649 s, 6.6 MB/s fail_loc=0x80000513 fail_val=17 osc.lustre-OST0000-osc-ffff93f685481800.cur_grant_bytes=70574080B Failing ost1 on oleg315-server Stopping /mnt/lustre-ost1 (opts:) on oleg315-server 06:17:18 (1787739438) shut down facet: ost1 facet_host: oleg315-server facet_failover_host: oleg315-server Failover ost1 to oleg315-server mount facets: ost1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 06:17:44 (1787739464) targets are mounted 06:17:44 (1787739464) facet_failover done oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.990676 s, 8.5 MB/s PASS 64i (67s) == sanity test 64j: check grants on re-done rpc ========== 06:18:05 (1787739485) fail_loc=0x80000256 checking grant......UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4048 1267256 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 2904 1268400 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9760 3597232 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9860 3597160 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19620 7194392 1% /mnt/lustre pass grant check: client:42082304 server:42082304 PASS 64j (11s) == sanity test 65a: directory with no stripe info ======== 06:18:16 (1787739496) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d65a.sanity default stripe 1, ost count 2 PASS 65a (8s) == sanity test 65b: directory setstripe -S stripe_size*2 -i 0 -c 1 ========================================================== 06:18:24 (1787739504) striped dir -i1 -c2 -H crush2 /mnt/lustre/d65b.sanity dir stripe 1, default stripe 1, ost count 2 PASS 65b (9s) == sanity test 65c: directory setstripe -S stripe_size*4 -i 1 -c 1 ========================================================== 06:18:33 (1787739513) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d65c.sanity dir stripe 1, default stripe 1, ost count 2 PASS 65c (9s) == sanity test 65d: directory setstripe -S stripe_size -c stripe_count ========================================================== 06:18:42 (1787739522) striped dir -i1 -c2 -H crush2 /mnt/lustre/d65d.sanity dir stripe 0, default stripe 1, ost count 2 PASS 65d (8s) == sanity test 65e: directory setstripe defaults ========= 06:18:51 (1787739531) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d65e.sanity (Default) /mnt/lustre/d65e.sanity default stripe 1, ost count 2 PASS 65e (8s) == sanity test 65f: dir setstripe permission (should return error) ============================================================= 06:18:58 (1787739538) striped dir -i1 -c2 -H crush /mnt/lustre/d65f.sanityf running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [/mnt/lustre/d65f.sanityf] lfs setstripe: unable to open '/mnt/lustre/d65f.sanityf': Permission denied (13) PASS 65f (7s) == sanity test 65g: directory setstripe -d =============== 06:19:06 (1787739546) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d65g.sanity (Default) /mnt/lustre/d65g.sanity PASS 65g (8s) == sanity test 65h: directory stripe info inherit ============================================================================== 06:19:14 (1787739554) striped dir -i1 -c2 -H crush2 /mnt/lustre/d65h.sanity striped dir -i1 -c2 -H crush /mnt/lustre/d65h.sanity/dd1 PASS 65h (6s) == sanity test 65i: various tests to set root directory striping ========================================================== 06:19:20 (1787739560) /mnt/lustre stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f61 lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2557 0x9fd 0x280000401 /mnt/lustre/d60j.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f60b.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x2c0000401 /mnt/lustre/d65c.sanity stripe_count: 1 stripe_size: 16777216 pattern: raid0 stripe_offset: 1 /mnt/lustre/d65b.sanity stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d65h.sanity stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d65f.sanityf stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65e.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d60h.sanity-0x80000188 stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65a.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f64j.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2566 0xa06 0x280000401 /mnt/lustre/d60f.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65g.sanity stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f63b.sanity lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2558 0x9fe 0x2c0000401 /mnt/lustre/f61b.sanity has no stripe info /mnt/lustre/d63c.sanity stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65d.sanity stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre lmm_fid: [0x200000007:0x1:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f61 lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x1597 lmm_fid: [0x200000402:0x1597:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2557 0x9fd 0x280000401 /mnt/lustre/d60j.sanity lmm_fid: [0x200000402:0x1532:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f60b.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x3 lmm_fid: [0x200000402:0x3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2 0x2 0x2c0000401 /mnt/lustre/d65c.sanity lmm_fid: [0x240000404:0x4:0x0] stripe_count: 1 stripe_size: 16777216 pattern: raid0 stripe_offset: 1 /mnt/lustre/d65b.sanity lmm_fid: [0x240000404:0x3:0x0] stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d65h.sanity lmm_fid: [0x240000404:0xb:0x0] stripe_count: 1 stripe_size: 8388608 pattern: raid0 stripe_offset: 0 /mnt/lustre/d65f.sanityf lmm_fid: [0x240000404:0x9:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65e.sanity lmm_fid: [0x240000404:0x8:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d60h.sanity-0x80000188 lmm_fid: [0x240000402:0x186:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65a.sanity lmm_fid: [0x240000404:0x1:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f64j.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000407 lmm_object_id: 0x1c lmm_fid: [0x200000407:0x1c:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 2566 0xa06 0x280000401 /mnt/lustre/d60f.sanity lmm_fid: [0x200000402:0x138d:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65g.sanity lmm_fid: [0x240000404:0xa:0x0] stripe_count: -1 stripe_size: 65536 pattern: raid0 stripe_offset: -1 /mnt/lustre/f63b.sanity lmm_magic: 0x0BD10BD0 lmm_seq: 0x200000402 lmm_object_id: 0x15a3 lmm_fid: [0x200000402:0x15a3:0x0] lmm_stripe_count: 1 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 obdidx objid objid group 1 2558 0x9fe 0x2c0000401 /mnt/lustre/f61b.sanity has no stripe info /mnt/lustre/d63c.sanity lmm_fid: [0x240000402:0x18d:0x0] stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 /mnt/lustre/d65d.sanity lmm_fid: [0x240000404:0x6:0x0] stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 PASS 65i (10s) == sanity test 65j: set default striping on root directory (bug 6367)=========================================================== 06:19:30 (1787739570) PASS 65j (10s) == sanity test 65k: validate manual striping works properly with deactivated OSCs ========================================================== 06:19:40 (1787739580) Check OST status: lustre-OST0000-osc-MDT0001 is active lustre-OST0000-osc-MDT0000 is active lustre-OST0001-osc-MDT0001 is active lustre-OST0001-osc-MDT0000 is active - open/close 669 (time 1787739603.47 total 10.01 last 66.81) total: 1000 open/close in 14.31 seconds: 69.87 ops/second Deactivate: lustre-OST0000-osc-MDT0001 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1787739616 ; total 0 ; last 0) total: 1000 unlinks in 7 seconds: 142.857147 unlinks/second lustre-OST0000-osc-MDT0001 is Activate oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec - open/close 654 (time 1787739655.23 total 10.01 last 65.30) total: 1000 open/close in 14.71 seconds: 68.00 ops/second Deactivate: lustre-OST0000-osc-MDT0000 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1787739668 ; total 0 ; last 0) total: 1000 unlinks in 7 seconds: 142.857147 unlinks/second lustre-OST0000-osc-MDT0000 is Activate oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec - open/close 725 (time 1787739702.70 total 10.01 last 72.44) total: 1000 open/close in 13.57 seconds: 73.68 ops/second Deactivate: lustre-OST0001-osc-MDT0001 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1787739714 ; total 1 ; last 1) total: 1000 unlinks in 8 seconds: 125.000000 unlinks/second lustre-OST0001-osc-MDT0001 is Activate oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec - open/close 697 (time 1787739749.11 total 10.00 last 69.67) total: 1000 open/close in 14.62 seconds: 68.39 ops/second Deactivate: lustre-OST0001-osc-MDT0000 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 0 -c 1 /mnt/lustre/d65k.sanity/0 /home/green/git/lustre-release/lustre/utils/lfs setstripe -i 1 -c 1 /mnt/lustre/d65k.sanity/1 - unlinked 0 (time 1787739764 ; total 0 ; last 0) total: 1000 unlinks in 9 seconds: 111.111115 unlinks/second lustre-OST0001-osc-MDT0000 is Activate oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec oleg315-server: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 oleg315-server: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec PASS 65k (213s) == sanity test 65l: lfs find on -1 stripe dir ================================================================================== 06:23:13 (1787739793) striped dir -i1 -c2 -H all_char /mnt/lustre/d65l.sanity/test_dir PASS 65l (8s) == sanity test 65m: normal user can't set filesystem default stripe ========================================================== 06:23:21 (1787739801) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-c] [2] [/mnt/lustre] lfs setstripe: setstripe error for '/mnt/lustre': Operation not permitted PASS 65m (7s) == sanity test 65n: don't inherit default layout from root for new subdirectories ========================================================== 06:23:28 (1787739808) Creating new pool test_65n oleg315-server: Pool lustre.test_65n created Adding targets to pool oleg315-server: OST lustre-OST0000_UUID added to pool lustre.test_65n oleg315-server: OST lustre-OST0001_UUID added to pool lustre.test_65n /home/green/git/lustre-release/lustre/utils/lfs getstripe -d /mnt/lustre/d65n.sanity-4 stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65n /home/green/git/lustre-release/lustre/utils/lfs getstripe -d /mnt/lustre stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65n Destroy the created pools: test_65n lustre.test_65n oleg315-server: OST lustre-OST0000_UUID removed from pool lustre.test_65n oleg315-server: OST lustre-OST0001_UUID removed from pool lustre.test_65n oleg315-server: Pool lustre.test_65n destroyed PASS 65n (31s) == sanity test 65o: pool inheritance for mdt component === 06:23:59 (1787739839) Creating new pool test_65o oleg315-server: Pool lustre.test_65o created Adding targets to pool oleg315-server: OST lustre-OST0000_UUID added to pool lustre.test_65o oleg315-server: OST lustre-OST0001_UUID added to pool lustre.test_65o /mnt/lustre/d65o.sanity lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 stripe_count: 0 stripe_size: 1048576 pattern: mdt mdt_index: -1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o /mnt/lustre/d65o.sanity/dir2 lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 2 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 stripe_count: 0 stripe_size: 1048576 pattern: mdt mdt_index: -1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o lcm_layout_gen: 0 lcm_mirror_count: 1 lcm_entry_count: 1 lcme_id: N/A lcme_mirror_id: N/A lcme_flags: 0 lcme_extent.e_start: 0 lcme_extent.e_end: EOF stripe_count: 1 stripe_size: 4194304 pattern: raid0 stripe_offset: -1 pool: test_65o Destroy the created pools: test_65o lustre.test_65o oleg315-server: OST lustre-OST0000_UUID removed from pool lustre.test_65o oleg315-server: OST lustre-OST0001_UUID removed from pool lustre.test_65o oleg315-server: Pool lustre.test_65o destroyed PASS 65o (32s) == sanity test 65p: setstripe with yaml file and huge number ========================================================== 06:24:31 (1787739871) striped dir -i1 -c2 -H crush2 /mnt/lustre/d65p.sanity/src_dir striped dir -i1 -c2 -H all_char /mnt/lustre/d65p.sanity/dst_dir PASS 65p (8s) == sanity test 65q: setstripe with >=8E offset should fail ========================================================== 06:24:39 (1787739879) striped dir -i1 -c2 -H crush2 /mnt/lustre/d65q.sanity/src_dir Cannot set layout EA: Invalid argument lfs setstripe: lfs setstripe: cannot set default composite layout for '/mnt/lustre/d65q.sanity/src_dir': Invalid argument (22) PASS 65q (8s) == sanity test 65r: prevent all-zero offsets ============= 06:24:47 (1787739887) striped dir -i1 -c2 -H all_char /mnt/lustre/d65r.sanity PASS 65r (8s) == sanity test 66: update inode blocks count on client ========================================================================= 06:24:55 (1787739895) 8+0 records in 8+0 records out 8192 bytes (8.2 kB, 8.0 KiB) copied, 0.0301894 s, 271 kB/s PASS 66 (16s) == sanity test 69: verify oa2dentry return -ENOENT doesn't LBUG ================================================================ 06:25:11 (1787739911) directio on /mnt/lustre/f69.sanity.2 for 1x4194304 bytes PASS fail_loc=0x217 directio on /mnt/lustre/f69.sanity for 2x4194304 bytes Write error No such file or directory (rc = -1, len = 8388608) fail_loc=0 directio on /mnt/lustre/f69.sanity for 2x4194304 bytes PASS directio on /mnt/lustre/f69.sanity for 1x4194304 bytes PASS fail_loc=0x217 directio on /mnt/lustre/f69.sanity for 1x4194304 bytes Read error: No such file or directory rc = -1 fail_loc=0 PASS 69 (14s) == sanity test 70a: verify health_check, health_write don't explode (on OST) ========================================================== 06:25:26 (1787739926) enable_health_write=off enable_health_write=0 enable_health_write=on enable_health_write=1 enable_health_write=0 PASS 70a (24s) SKIP: sanity test_71 skipping SLOW test 71 == sanity test 72a: Test that remove suid works properly (bug5695) ============================================================== 06:25:54 (1787739954) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8239] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [dd] [if=/dev/zero] [of=/mnt/lustre/f72a.sanity] [bs=512] [count=1] 1+0 records in 1+0 records out 512 bytes copied, 0.0180297 s, 28.4 kB/s PASS 72a (13s) == sanity test 72b: Test that we keep mode setting if without file data changed (bug 24226) ========================================================== 06:26:06 (1787739966) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [true] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [/mnt/lustre/d0_runas_test/f8239] striped dir -i0 -c2 -H crush2 /mnt/lustre/f72b.sanity-dg striped dir -i0 -c2 -H crush /mnt/lustre/f72b.sanity-du running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [2777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-fg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-fu] chmod: changing permissions of '/mnt/lustre/f72b.sanity-fu': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-dg] chmod: changing permissions of '/mnt/lustre/f72b.sanity-dg': Operation not permitted running as uid/gid/euid/egid 500/500/500/500, groups: 500 [chmod] [4777] [/mnt/lustre/f72b.sanity-du] chmod: changing permissions of '/mnt/lustre/f72b.sanity-du': Operation not permitted PASS 72b (10s) == sanity test 73: multiple MDC requests (should not deadlock) ========================================================== 06:26:16 (1787739976) striped dir -i1 -c2 -H crush /mnt/lustre/d73-1 striped dir -i1 -c2 -H all_char /mnt/lustre/d73-2 multiop /mnt/lustre/d73-1/f73-1 vO_c TMPPIPE=/tmp/multiop_open_wait_pipe.8239 fail_loc=0x80000129 fail_loc=0 /mnt/lustre/d73-1/f73-1 has type file OK /mnt/lustre/d73-1/f73-2 has type file OK /mnt/lustre/d73-2/f73-3 has type file OK PASS 73 (35s) == sanity test 74a: ldlm_enqueue freed-export error path, ls (shouldn't LBUG) ========================================================== 06:26:51 (1787740011) fail_loc=0x8000030e /mnt/lustre/f74a fail_loc=0 PASS 74a (8s) == sanity test 74b: ldlm_enqueue freed-export error path, touch (shouldn't LBUG) ========================================================== 06:26:59 (1787740019) fail_loc=0x8000030e fail_loc=0 PASS 74b (7s) == sanity test 74c: ldlm_lock_create error path, (shouldn't LBUG) ========================================================== 06:27:06 (1787740026) fail_loc=0x319 touch: cannot touch '/mnt/lustre/f74c.sanity': No such file or directory fail_loc=0 PASS 74c (7s) == sanity test 76a: confirm clients recycle inodes properly ============================================================== 06:27:14 (1787740034) before slab objects: 93 created: 512, after slab objects: 93 PASS 76a (88s) == sanity test 76b: confirm clients recycle directory inodes properly ============================================================== 06:28:41 (1787740121) slab objects before: 93, after: 93 PASS 76b (55s) == sanity test 77a: normal checksum read/write operation ========================================================== 06:29:36 (1787740176) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.438004 s, 19.2 MB/s 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.971144 s, 8.6 MB/s PASS 77a (9s) == sanity test 77b: checksum error on client write, read ========================================================== 06:29:45 (1787740185) fail_loc=0x80000409 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.7954 s, 10.5 MB/s fail_loc=0 set checksum type to crc32, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to adler, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to crc32c, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10ip512, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10ip4K, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10crc512, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to t10crc4K, rc = 0 fail_loc=0x80000408 fail_loc=0 set checksum type to crc32c, rc = 0 PASS 77b (49s) == sanity test 77c: checksum error on client read with debug ========================================================== 06:30:35 (1787740235) debug_path=/tmp/lustre-log 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.22187 s, 6.9 MB/s osc.lustre-OST0000-osc-ffff93f685481800.checksum_dump=1 osc.lustre-OST0001-osc-ffff93f685481800.checksum_dump=1 obdfilter.lustre-OST0000.checksum_dump=1 obdfilter.lustre-OST0001.checksum_dump=1 fail_loc=0x80000408 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 7.09009 s, 1.2 MB/s fail_loc=0 osc.lustre-OST0000-osc-ffff93f685481800.checksum_dump=0 osc.lustre-OST0001-osc-ffff93f685481800.checksum_dump=0 obdfilter.lustre-OST0000.checksum_dump=0 obdfilter.lustre-OST0001.checksum_dump=0 PASS 77c (46s) == sanity test 77d: checksum error on OST direct write, read ========================================================== 06:31:20 (1787740280) fail_loc=0x80000409 directio on /mnt/lustre/f77d.sanity for 8x1048576 bytes PASS fail_loc=0 fail_loc=0x80000408 directio on /mnt/lustre/f77d.sanity for 8x1048576 bytes PASS fail_loc=0 PASS 77d (13s) == sanity test 77f: repeat checksum error on write (expect error) ========================================================== 06:31:33 (1787740293) osc.lustre-OST0000-osc-ffff93f685481800.resend_count=1 osc.lustre-OST0001-osc-ffff93f685481800.resend_count=1 set checksum type to crc32, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.crc32 for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to adler, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.adler for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to crc32c, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.crc32c for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10ip512, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.t10ip512 for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10ip4K, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.t10ip4K for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10crc512, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.t10crc512 for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to t10crc4K, rc = 0 fail_loc=0x409 directio on /mnt/lustre/f77f.sanity.t10crc4K for 8x1048576 bytes Write error Input/output error (rc = -1, len = 8388608) fail_loc=0 set checksum type to crc32c, rc = 0 osc.lustre-OST0000-osc-ffff93f685481800.resend_count=10 osc.lustre-OST0001-osc-ffff93f685481800.resend_count=10 PASS 77f (38s) == sanity test 77g: checksum error on OST write, read ==== 06:32:12 (1787740332) fail_loc=0x8000021a 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 2.99211 s, 2.8 MB/s fail_loc=0 fail_loc=0x8000021b fail_loc=0 PASS 77g (20s) == sanity test 77k: enable/disable checksum correctly ==== 06:32:32 (1787740352) Waiting 90s for '0' Updated after 2s: want '0' got '0' remount client, checksum should be 0 Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre Waiting 90s for '1' remount client, checksum should be 1 Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre remount client with option checksum, checksum should be 1 192.168.203.115@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock,checksum 192.168.203.115@tcp:/lustre /mnt/lustre remount client with option nochecksum, checksum should be 0 192.168.203.115@tcp:/lustre /mnt/lustre lustre rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose 0 0 Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock,nochecksum 192.168.203.115@tcp:/lustre /mnt/lustre Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre Waiting 90s for '0' PASS 77k (42s) == sanity test 77l: preferred checksum type is remembered after reconnected ========================================================== 06:33:15 (1787740395) osc.lustre-OST0000-osc-ffff93f68614a800.idle_timeout=10 osc.lustre-OST0001-osc-ffff93f68614a800.idle_timeout=10 error: set_param: setting /sys/fs/lustre/osc/lustre-OST0000-osc-ffff93f68614a800/checksum_type=invalid: Invalid argument error: set_param: setting /sys/fs/lustre/osc/lustre-OST0001-osc-ffff93f68614a800/checksum_type=invalid: Invalid argument error: set_param: setting 'osc/*osc-[^mM]*/checksum_type'='invalid': Invalid argument set checksum type to invalid, rc = 22 set checksum type to crc32, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s oleg315-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in IDLE state after 0 sec oleg315-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in FULL state after 0 sec set checksum type to adler, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg315-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in IDLE state after 6 sec oleg315-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in FULL state after 0 sec set checksum type to crc32c, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg315-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in IDLE state after 5 sec oleg315-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in FULL state after 0 sec set checksum type to t10ip512, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg315-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in IDLE state after 7 sec oleg315-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in FULL state after 0 sec set checksum type to t10ip4K, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg315-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in IDLE state after 4 sec oleg315-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in FULL state after 0 sec set checksum type to t10crc512, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg315-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in IDLE state after 4 sec oleg315-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in FULL state after 0 sec set checksum type to t10crc4K, rc = 0 ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=3900000->3900s oleg315-client.virtnet: executing wait_import_state IDLE osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in IDLE state after 6 sec oleg315-client.virtnet: executing wait_import_state FULL osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f68614a800.ost_server_uuid in FULL state after 0 sec osc.lustre-OST0000-osc-ffff93f68614a800.idle_timeout=20 osc.lustre-OST0001-osc-ffff93f68614a800.idle_timeout=20 set checksum type to crc32c, rc = 0 PASS 77l (193s) == sanity test 77m: Verify checksum_speed is correctly read ========================================================== 06:36:27 (1787740587) checksum_speed= adler32: 631 crc32: 679 crc32c: 5103 PASS 77m (8s) == sanity test 77n: Verify read from a hole inside contiguous blocks with T10PI ========================================================== 06:36:35 (1787740595) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0075323 s, 544 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00456711 s, 897 kB/s set checksum type to t10ip512, rc = 0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0521095 s, 236 kB/s set checksum type to t10ip4K, rc = 0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0323446 s, 380 kB/s set checksum type to t10crc512, rc = 0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0265262 s, 463 kB/s set checksum type to t10crc4K, rc = 0 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0194261 s, 633 kB/s set checksum type to crc32c, rc = 0 PASS 77n (19s) == sanity test 77o: Verify checksum_type for server (mdt and ofd(obdfilter)) ========================================================== 06:36:54 (1787740614) obdfilter.lustre-*.checksum_type: crc32 adler [crc32c] t10ip512 t10ip4K t10crc512 t10crc4K crc32 adler [crc32c] t10ip512 t10ip4K t10crc512 t10crc4K mdt.lustre-*.checksum_type: crc32 adler [crc32c] t10ip512 t10ip4K t10crc512 t10crc4K crc32 adler [crc32c] t10ip512 t10ip4K t10crc512 t10crc4K PASS 77o (13s) == sanity test 78: handle large O_DIRECT writes correctly ====================================================================== 06:37:07 (1787740627) MemFree: 2488, Max file size: 600000 MemTotal: 3693 Mem to use for directio: 1718 Smallest OST: 3597148 File size: 32 directIO rdwr round 1 of 1 directio on /mnt/lustre/f78.sanity for 32x1048576 bytes PASS PASS 78 (10s) == sanity test 79: df report consistency check =========== 06:37:17 (1787740637) Waiting for MDT destroys to complete UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 4552 1266752 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3332 1267972 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9764 3597256 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9872 3597148 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19636 7194404 1% /mnt/lustre 192.168.203.115@tcp:/lustre 7666232 19636 7194404 1% /mnt/lustre llite.lustre-ffff93f68614a800.maxbytes=17592186040320 lmv.lustre-clilmv-ffff93f68614a800.maxbytes=17592186040320 lov.lustre-clilov-ffff93f68614a800.maxbytes=17592186040320 mdc.lustre-MDT0000-mdc-ffff93f68614a800.maxbytes=17592186040320 mdc.lustre-MDT0001-mdc-ffff93f68614a800.maxbytes=17592186040320 osc.lustre-OST0000-osc-ffff93f68614a800.maxbytes=17592186040320 osc.lustre-OST0001-osc-ffff93f68614a800.maxbytes=17592186040320 llite.lustre-ffff93f68614a800.namelen_max=255 lmv.lustre-clilmv-ffff93f68614a800.namelen_max=255 lov.lustre-clilov-ffff93f68614a800.namelen_max=255 mdc.lustre-MDT0000-mdc-ffff93f68614a800.namelen_max=255 mdc.lustre-MDT0001-mdc-ffff93f68614a800.namelen_max=255 osc.lustre-OST0000-osc-ffff93f68614a800.namelen_max=255 osc.lustre-OST0001-osc-ffff93f68614a800.namelen_max=255 lod.lustre-MDT0000-mdtlov.maxbytes=17592186040320 lod.lustre-MDT0001-mdtlov.maxbytes=17592186040320 lov.lustre-MDT0000-mdtlov.maxbytes=17592186040320 lov.lustre-MDT0001-mdtlov.maxbytes=17592186040320 obdfilter.lustre-OST0000.maxbytes=17592186040320 obdfilter.lustre-OST0001.maxbytes=17592186040320 osc.lustre-OST0000-osc-MDT0000.maxbytes=17592186040320 osc.lustre-OST0000-osc-MDT0001.maxbytes=17592186040320 osc.lustre-OST0001-osc-MDT0000.maxbytes=17592186040320 osc.lustre-OST0001-osc-MDT0001.maxbytes=17592186040320 osd-ldiskfs.lustre-MDT0000.maxbytes=17592186040320 osd-ldiskfs.lustre-MDT0001.maxbytes=17592186040320 osd-ldiskfs.lustre-OST0000.maxbytes=17592186040320 osd-ldiskfs.lustre-OST0001.maxbytes=17592186040320 osp.lustre-MDT0000-osp-MDT0001.maxbytes=17592186040320 osp.lustre-MDT0001-osp-MDT0000.maxbytes=17592186040320 osp.lustre-OST0000-osc-MDT0000.maxbytes=17592186040320 osp.lustre-OST0000-osc-MDT0001.maxbytes=17592186040320 osp.lustre-OST0001-osc-MDT0000.maxbytes=17592186040320 osp.lustre-OST0001-osc-MDT0001.maxbytes=17592186040320 lod.lustre-MDT0000-mdtlov.namelen_max=255 lod.lustre-MDT0001-mdtlov.namelen_max=255 lov.lustre-MDT0000-mdtlov.namelen_max=255 lov.lustre-MDT0001-mdtlov.namelen_max=255 obdfilter.lustre-OST0000.namelen_max=255 obdfilter.lustre-OST0001.namelen_max=255 osc.lustre-OST0000-osc-MDT0000.namelen_max=255 osc.lustre-OST0000-osc-MDT0001.namelen_max=255 osc.lustre-OST0001-osc-MDT0000.namelen_max=255 osc.lustre-OST0001-osc-MDT0001.namelen_max=255 osd-ldiskfs.lustre-MDT0000.namelen_max=255 osd-ldiskfs.lustre-MDT0001.namelen_max=255 osd-ldiskfs.lustre-OST0000.namelen_max=255 osd-ldiskfs.lustre-OST0001.namelen_max=255 osp.lustre-MDT0000-osp-MDT0001.namelen_max=255 osp.lustre-MDT0001-osp-MDT0000.namelen_max=255 osp.lustre-OST0000-osc-MDT0000.namelen_max=255 osp.lustre-OST0000-osc-MDT0001.namelen_max=255 osp.lustre-OST0001-osc-MDT0000.namelen_max=255 osp.lustre-OST0001-osc-MDT0001.namelen_max=255 lod.lustre-MDT0000-mdtlov.maxbytes=17592186040320 lod.lustre-MDT0001-mdtlov.maxbytes=17592186040320 lov.lustre-MDT0000-mdtlov.maxbytes=17592186040320 lov.lustre-MDT0001-mdtlov.maxbytes=17592186040320 obdfilter.lustre-OST0000.maxbytes=17592186040320 obdfilter.lustre-OST0001.maxbytes=17592186040320 osc.lustre-OST0000-osc-MDT0000.maxbytes=17592186040320 osc.lustre-OST0000-osc-MDT0001.maxbytes=17592186040320 osc.lustre-OST0001-osc-MDT0000.maxbytes=17592186040320 osc.lustre-OST0001-osc-MDT0001.maxbytes=17592186040320 osd-ldiskfs.lustre-MDT0000.maxbytes=17592186040320 osd-ldiskfs.lustre-MDT0001.maxbytes=17592186040320 osd-ldiskfs.lustre-OST0000.maxbytes=17592186040320 osd-ldiskfs.lustre-OST0001.maxbytes=17592186040320 osp.lustre-MDT0000-osp-MDT0001.maxbytes=17592186040320 osp.lustre-MDT0001-osp-MDT0000.maxbytes=17592186040320 osp.lustre-OST0000-osc-MDT0000.maxbytes=17592186040320 osp.lustre-OST0000-osc-MDT0001.maxbytes=17592186040320 osp.lustre-OST0001-osc-MDT0000.maxbytes=17592186040320 osp.lustre-OST0001-osc-MDT0001.maxbytes=17592186040320 lod.lustre-MDT0000-mdtlov.namelen_max=255 lod.lustre-MDT0001-mdtlov.namelen_max=255 lov.lustre-MDT0000-mdtlov.namelen_max=255 lov.lustre-MDT0001-mdtlov.namelen_max=255 obdfilter.lustre-OST0000.namelen_max=255 obdfilter.lustre-OST0001.namelen_max=255 osc.lustre-OST0000-osc-MDT0000.namelen_max=255 osc.lustre-OST0000-osc-MDT0001.namelen_max=255 osc.lustre-OST0001-osc-MDT0000.namelen_max=255 osc.lustre-OST0001-osc-MDT0001.namelen_max=255 osd-ldiskfs.lustre-MDT0000.namelen_max=255 osd-ldiskfs.lustre-MDT0001.namelen_max=255 osd-ldiskfs.lustre-OST0000.namelen_max=255 osd-ldiskfs.lustre-OST0001.namelen_max=255 osp.lustre-MDT0000-osp-MDT0001.namelen_max=255 osp.lustre-MDT0001-osp-MDT0000.namelen_max=255 osp.lustre-OST0000-osc-MDT0000.namelen_max=255 osp.lustre-OST0000-osc-MDT0001.namelen_max=255 osp.lustre-OST0001-osc-MDT0000.namelen_max=255 osp.lustre-OST0001-osc-MDT0001.namelen_max=255 PASS 79 (18s) == sanity test 80: Page eviction is equally fast at high offsets too ========================================================== 06:37:35 (1787740655) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.15106 s, 6.9 MB/s PASS 80 (8s) == sanity test 81a: OST should retry write when get -ENOSPC ========================================================================= 06:37:43 (1787740663) fail_loc=0x80000228 PASS 81a (9s) == sanity test 81b: OST should return -ENOSPC when retry still fails ================================================================= 06:37:53 (1787740673) fail_loc=0x228 write: No space left on device PASS 81b (11s) == sanity test 99: cvs strange file/directory operations ========================================================== 06:38:03 (1787740683) striped dir -i1 -c2 -H all_char /mnt/lustre/d99.sanity.cvsroot running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [-d] [/mnt/lustre/d99.sanity.cvsroot] [init] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [-d] [/mnt/lustre/d99.sanity.cvsroot] [import] [-m] [nomesg] [d99.sanity.reposname] [vtag] [rtag] cvs import: cannot open /root/.cvsignore: Permission denied N d99.sanity.reposname/README N d99.sanity.reposname/functions No conflicts created by this import striped dir -i1 -c2 -H all_char /mnt/lustre/d99.sanity.reposname running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [-d] [/mnt/lustre/d99.sanity.cvsroot] [co] [d99.sanity.reposname] cvs checkout: cannot open /root/.cvsignore: Permission denied cvs checkout: Updating d99.sanity.reposname U d99.sanity.reposname/README U d99.sanity.reposname/functions running as uid/gid/euid/egid 500/500/500/500, groups: 500 [touch] [foo99] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [add] [-m] [addmsg] [foo99] cvs add: scheduling file `foo99' for addition cvs add: use 'cvs commit' to add this file permanently running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [update] cvs update: cannot open /root/.cvsignore: Permission denied cvs update: Updating . A foo99 running as uid/gid/euid/egid 500/500/500/500, groups: 500 [cvs] [commit] [-m] [nomsg] [foo99] RCS file: /mnt/lustre/d99.sanity.cvsroot/d99.sanity.reposname/foo99,v done Checking in foo99; /mnt/lustre/d99.sanity.cvsroot/d99.sanity.reposname/foo99,v <-- foo99 initial revision: 1.1 done PASS 99 (32s) == sanity test 100: check local port using privileged port ========================================================== 06:38:36 (1787740716) PASS 100 (7s) == sanity test 101a: check read-ahead for random reads === 06:38:44 (1787740724) nreads: 10000 file size: 96MB 180.228573s, 2.98283MB/s osc.lustre-OST0000-osc-ffff93f68614a800.rpc_stats= snapshot_time: 1787740922.065332489 secs.nsecs start_time: 1787740726.070097985 secs.nsecs elapsed_time: 195.995234504 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 1 0 0 | 1 3 3 2: 0 0 0 | 0 0 3 4: 1 0 0 | 0 0 3 8: 9 0 0 | 0 0 3 16: 5380 99 100 | 0 0 3 32: 0 0 100 | 0 0 3 64: 0 0 100 | 0 0 3 128: 0 0 100 | 0 0 3 256: 0 0 100 | 0 0 3 512: 0 0 100 | 0 0 3 1024: 0 0 100 | 25 96 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 5391 100 100 | 26 100 100 read write offset rpcs % cum % | rpcs % cum % 0: 6 0 0 | 2 7 7 1: 0 0 0 | 0 0 7 2: 0 0 0 | 0 0 7 4: 0 0 0 | 0 0 7 8: 0 0 0 | 0 0 7 16: 4 0 0 | 0 0 7 32: 7 0 0 | 0 0 7 64: 15 0 0 | 0 0 7 128: 23 0 1 | 0 0 7 256: 55 1 2 | 0 0 7 512: 115 2 4 | 0 0 7 1024: 199 3 7 | 1 3 11 2048: 467 8 16 | 2 7 19 4096: 884 16 32 | 4 15 34 8192: 1763 32 65 | 8 30 65 16384: 1853 34 100 | 9 34 100 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 0 0 0 | 0 0 0 4096: 248 4 4 | 0 0 0 8192: 2519 46 51 | 0 0 0 16384: 2189 40 91 | 1 3 3 32768: 529 9 101 | 4 15 19 65536: 33 0 102 | 19 73 92 131072: 0 0 102 | 2 7 100 262144: 0 0 102 | 0 0 100 524288: 0 0 102 | 0 0 100 1048576: 0 0 102 | 0 0 100 2097152: 0 0 102 | 0 0 100 4194304: 0 0 102 | 0 0 100 8388608: 0 0 102 | 0 0 100 16777216: 0 0 102 | 0 0 100 33554432: 0 0 102 | 0 0 100 67108864: 0 0 102 | 0 0 100 134217728: 0 0 102 | 0 0 100 268435456: 0 0 102 | 0 0 100 536870912: 0 0 102 | 0 0 100 1073741824: 0 0 102 | 0 0 100 osc.lustre-OST0001-osc-ffff93f68614a800.rpc_stats= snapshot_time: 1787740922.065483225 secs.nsecs start_time: 1787740726.070267055 secs.nsecs elapsed_time: 195.995216170 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 3 75 75 2: 0 0 0 | 0 0 75 4: 0 0 0 | 0 0 75 8: 0 0 0 | 1 25 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 0 0 0 | 3 75 75 2: 0 0 0 | 1 25 100 read write offset rpcs % cum % | rpcs % cum % 0: 0 0 0 | 4 100 100 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 0 0 0 | 0 0 0 4096: 0 0 0 | 0 0 0 8192: 0 0 0 | 1 25 25 16384: 0 0 0 | 1 25 50 32768: 0 0 0 | 1 25 75 65536: 0 0 0 | 1 25 100 llite.lustre-ffff93f68614a800.read_ahead_stats= snapshot_time 1787740922.075536496 secs.nsecs start_time 1787740726.103106855 secs.nsecs elapsed_time 195.972429641 secs.nsecs hits 80726 samples [pages] misses 5391 samples [pages] readpage_not_consecutive 8198 samples [pages] zero_size_window 80726 samples [pages] failed_to_fast_read 5391 samples [pages] readahead_pages 5390 samples [pages] 2 15 80726 PASS 101a (204s) == sanity test 101b: check stride-io mode read-ahead =========================================================================== 06:42:08 (1787740928) striped dir -i1 -c2 -H crush /mnt/lustre/d101b.sanity 1.425046s, 5.88655MB/s Read-ahead success for size 8192 1.834862s, 4.57179MB/s Read-ahead success for size 16384 1.835598s, 4.56996MB/s Read-ahead success for size 32768 1.536355s, 5.46007MB/s Read-ahead success for size 65536 1.389939s, 6.03523MB/s Read-ahead success for size 131072 1.268934s, 6.61075MB/s Read-ahead success for size 262144 1.332973s, 6.29316MB/s Read-ahead success for size 524288 1.050627s, 7.98438MB/s Read-ahead success for size 1048576 PASS 101b (29s) == sanity test 101c: check stripe_size aligned read-ahead ========================================================== 06:42:37 (1787740957) striped dir -i1 -c2 -H crush2 /mnt/lustre/d101c.sanity osc.lustre-OST0000-osc-ffff93f68614a800.rpc_stats=0 osc.lustre-OST0001-osc-ffff93f68614a800.rpc_stats=0 59.599624s, 10.996MB/s osc.lustre-OST0000-osc-ffff93f68614a800.rpc_stats= snapshot_time: 1787741039.864802113 secs.nsecs start_time: 1787740980.139264452 secs.nsecs elapsed_time: 59.725537661 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 798 100 100 | 0 0 0 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 798 100 100 | 0 0 0 read write offset rpcs % cum % | rpcs % cum % 0: 1 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 1 0 0 | 0 0 0 32: 2 0 0 | 0 0 0 64: 4 0 1 | 0 0 0 128: 8 1 2 | 0 0 0 256: 16 2 4 | 0 0 0 512: 32 4 8 | 0 0 0 1024: 63 7 15 | 0 0 0 2048: 128 16 31 | 0 0 0 4096: 255 31 63 | 0 0 0 8192: 288 36 100 | 0 0 0 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 0 0 0 | 0 0 0 4096: 17 2 2 | 0 0 0 8192: 326 40 42 | 0 0 0 16384: 353 44 87 | 0 0 0 32768: 109 13 100 | 0 0 0 65536: 10 1 102 | 0 0 0 131072: 0 0 102 | 0 0 0 262144: 0 0 102 | 0 0 0 524288: 0 0 102 | 0 0 0 1048576: 0 0 102 | 0 0 0 2097152: 0 0 102 | 0 0 0 4194304: 0 0 102 | 0 0 0 8388608: 0 0 102 | 0 0 0 16777216: 0 0 102 | 0 0 0 33554432: 0 0 102 | 0 0 0 67108864: 0 0 102 | 0 0 0 134217728: 0 0 102 | 0 0 0 268435456: 0 0 102 | 0 0 0 536870912: 0 0 102 | 0 0 0 1073741824: 0 0 102 | 0 0 0 osc.lustre-OST0001-osc-ffff93f68614a800.rpc_stats= snapshot_time: 1787741039.864931014 secs.nsecs start_time: 1787740980.139354327 secs.nsecs elapsed_time: 59.725576687 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 799 100 100 | 0 0 0 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 799 100 100 | 0 0 0 read write offset rpcs % cum % | rpcs % cum % 0: 1 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 1 0 0 | 0 0 0 32: 2 0 0 | 0 0 0 64: 4 0 1 | 0 0 0 128: 8 1 2 | 0 0 0 256: 16 2 4 | 0 0 0 512: 32 4 8 | 0 0 0 1024: 64 8 16 | 0 0 0 2048: 128 16 32 | 0 0 0 4096: 255 31 63 | 0 0 0 8192: 288 36 100 | 0 0 0 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 0 0 0 | 0 0 0 4096: 27 3 3 | 0 0 0 8192: 328 41 44 | 0 0 0 16384: 331 41 85 | 0 0 0 32768: 112 14 99 | 0 0 0 65536: 17 2 102 | 0 0 0 131072: 0 0 102 | 0 0 0 262144: 0 0 102 | 0 0 0 524288: 0 0 102 | 0 0 0 1048576: 0 0 102 | 0 0 0 2097152: 0 0 102 | 0 0 0 4194304: 0 0 102 | 0 0 0 8388608: 0 0 102 | 0 0 0 16777216: 0 0 102 | 0 0 0 33554432: 0 0 102 | 0 0 0 67108864: 0 0 102 | 0 0 0 134217728: 0 0 102 | 0 0 0 268435456: 0 0 102 | 0 0 0 536870912: 0 0 102 | 0 0 0 1073741824: 0 0 102 | 0 0 0 osc.lustre-OST0000-osc-ffff93f68614a800.rpc_stats check passed! osc.lustre-OST0001-osc-ffff93f68614a800.rpc_stats check passed! PASS 101c (92s) == sanity test 101d: file read with and without read-ahead enabled ========================================================== 06:44:09 (1787741049) Create test file /mnt/lustre/f101d.sanity size 80M, 7025M free 80+0 records in 80+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 8.61074 s, 9.7 MB/s Cancel LRU locks on lustre client to flush the client cache Disable read-ahead 0 Reading the test file /mnt/lustre/f101d.sanity with read-ahead disabled Cancel LRU locks on lustre client to flush the client cache Enable read-ahead with 40MB Reading the test file /mnt/lustre/f101d.sanity with read-ahead enabled read-ahead disabled time read '252.715' read-ahead enabled time read '7.64244' Waiting for MDT destroys to complete PASS 101d (299s) == sanity test 101e: check read-ahead for small read(1k) for small files(500k) ========================================================== 06:49:08 (1787741348) Creating 100 500K test files Cancel LRU locks on lustre client to flush the client cache Reset readahead stats llite.lustre-ffff93f68614a800.max_cached_mb= users: 6 max_cached_mb: 1846 used_mb: 49 unused_mb: 1797 unevict_mb: 0 reclaim_count: 0 max_read_ahead_mb: 256 used_read_ahead_mb: 0 llite.lustre-ffff93f68614a800.read_ahead_stats= snapshot_time 1787741421.545111604 secs.nsecs start_time 1787741403.868718531 secs.nsecs elapsed_time 17.676393073 secs.nsecs hits 12300 samples [pages] misses 200 samples [pages] zero_size_window 100 samples [pages] failed_to_fast_read 200 samples [pages] readahead_pages 100 samples [pages] 123 123 12300 PASS 101e (86s) == sanity test 101f: check mmap read performance ========= 06:50:34 (1787741434) /opt/iozone/bin/iozone debug=reada mmap Cancel LRU locks on lustre client to flush the client cache Reset readahead stats mmap read the file with small block size checking missing pages llite.lustre-ffff93f68614a800.read_ahead_stats= snapshot_time 1787741439.633089694 secs.nsecs start_time 1787741439.296130727 secs.nsecs elapsed_time 0.336958967 secs.nsecs debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout PASS 101f (11s) == sanity test 101g: Big bulk(4/16 MiB) readahead ======== 06:50:45 (1787741445) remount client to enable new RPC size Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre osc.lustre-OST0000-osc-ffff93f6b40cf800.max_pages_per_rpc=16M osc.lustre-OST0001-osc-ffff93f6b40cf800.max_pages_per_rpc=16M 10+0 records in 10+0 records out 167772160 bytes (168 MB, 160 MiB) copied, 6.40702 s, 26.2 MB/s 10+0 records in 10+0 records out 167772160 bytes (168 MB, 160 MiB) copied, 7.11124 s, 23.6 MB/s osc.lustre-OST0000-osc-ffff93f6b40cf800.max_pages_per_rpc=8M osc.lustre-OST0001-osc-ffff93f6b40cf800.max_pages_per_rpc=8M 10+0 records in 10+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 3.61259 s, 23.2 MB/s 10+0 records in 10+0 records out 83886080 bytes (84 MB, 80 MiB) copied, 3.64226 s, 23.0 MB/s osc.lustre-OST0000-osc-ffff93f6b40cf800.max_pages_per_rpc=4M osc.lustre-OST0001-osc-ffff93f6b40cf800.max_pages_per_rpc=4M 10+0 records in 10+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 2.33134 s, 18.0 MB/s 10+0 records in 10+0 records out 41943040 bytes (42 MB, 40 MiB) copied, 5.16894 s, 8.1 MB/s Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre Waiting for MDT destroys to complete Waiting for MDT destroys to complete Waiting for MDT destroys to complete PASS 101g (75s) == sanity test 101h: Readahead should cover current read window ========================================================== 06:52:00 (1787741520) llite.lustre-ffff93f6a0950800.hybrid_io=0 70+0 records in 70+0 records out 73400320 bytes (73 MB, 70 MiB) copied, 9.69362 s, 7.6 MB/s Cancel LRU locks on lustre client to flush the client cache Reset readahead stats Read 10M of data but cross 64M bundary 1+0 records in 1+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.4768 s, 7.1 MB/s PASS 101h (22s) == sanity test 101i: allow current readahead to exceed reservation ========================================================== 06:52:22 (1787741542) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.4191 s, 7.4 MB/s llite.lustre-ffff93f6a0950800.max_read_ahead_per_file_mb=1 Reset readahead stats llite.lustre-ffff93f6a0950800.read_ahead_stats=0 5+0 records in 5+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 2.43677 s, 4.3 MB/s llite.lustre-ffff93f6a0950800.read_ahead_stats= snapshot_time 1787741548.735366561 secs.nsecs start_time 1787741546.218082234 secs.nsecs elapsed_time 2.517284327 secs.nsecs hits 2555 samples [pages] misses 5 samples [pages] zero_size_window 2555 samples [pages] failed_to_fast_read 6 samples [pages] readahead_pages 5 samples [pages] 511 511 2555 llite.lustre-ffff93f6a0950800.max_read_ahead_per_file_mb=64 PASS 101i (13s) == sanity test 101j: A complete read block should be submitted when no RA ========================================================== 06:52:35 (1787741555) llite.lustre-ffff93f6a0950800.hybrid_io=0 Disable read-ahead 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 2.34597 s, 7.2 MB/s Reset readahead stats 4096+0 records in 4096+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 47.8328 s, 351 kB/s snapshot_time 1787741609.158021942 secs.nsecs start_time 1787741561.200897689 secs.nsecs elapsed_time 47.957124253 secs.nsecs failed_to_fast_read 4096 samples [pages] Reset readahead stats 16+0 records in 16+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.70243 s, 9.9 MB/s snapshot_time 1787741611.580737306 secs.nsecs start_time 1787741609.764847085 secs.nsecs elapsed_time 1.815890221 secs.nsecs failed_to_fast_read 16 samples [pages] forceread_pages 16 samples [pages] 255 255 4080 Reset readahead stats 1+0 records in 1+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.14902 s, 14.6 MB/s snapshot_time 1787741613.282495593 secs.nsecs start_time 1787741612.063164156 secs.nsecs elapsed_time 1.219331437 secs.nsecs failed_to_fast_read 1 samples [pages] forceread_pages 1 samples [pages] 4095 4095 4095 PASS 101j (63s) == sanity test 101m: read ahead for small file and last stripe of the file ========================================================== 06:53:38 (1787741618) keep default fallocate mode: 0 Test readahead: size=4096 ramax= iosz=1048576 short read: 0 ->+ 1048576 -> 4096 4096 short read: 4096 ->+ 1044480 -> 4096 0 snapshot_time 1787741620.629741548 secs.nsecs start_time 1787741620.360351267 secs.nsecs elapsed_time 0.269390281 secs.nsecs misses 2 samples [pages] zero_size_window 2 samples [pages] failed_to_fast_read 1 samples [pages] Test readahead: size=16384 ramax= iosz=1048576 short read: 0 ->+ 1048576 -> 16384 16384 short read: 16384 ->+ 1032192 -> 16384 0 snapshot_time 1787741621.001242880 secs.nsecs start_time 1787741620.734502508 secs.nsecs elapsed_time 0.266740372 secs.nsecs hits 3 samples [pages] misses 2 samples [pages] zero_size_window 4 samples [pages] failed_to_fast_read 1 samples [pages] readahead_pages 1 samples [pages] 3 3 3 Test readahead: size=16385 ramax= iosz=1048576 short read: 0 ->+ 1048576 -> 16385 16385 short read: 16385 ->+ 1032191 -> 16385 0 snapshot_time 1787741621.293128541 secs.nsecs start_time 1787741621.077846633 secs.nsecs elapsed_time 0.215281908 secs.nsecs hits 4 samples [pages] misses 1 samples [pages] zero_size_window 4 samples [pages] readahead_to_eof 1 samples [pages] failed_to_fast_read 1 samples [pages] readahead_pages 1 samples [pages] 4 4 4 Test readahead: size=16383 ramax= iosz=1048576 short read: 0 ->+ 1048576 -> 16383 16383 short read: 16383 ->+ 1032193 -> 16383 0 snapshot_time 1787741621.681255448 secs.nsecs start_time 1787741621.389322529 secs.nsecs elapsed_time 0.291932919 secs.nsecs hits 3 samples [pages] misses 1 samples [pages] zero_size_window 3 samples [pages] readahead_to_eof 1 samples [pages] failed_to_fast_read 1 samples [pages] readahead_pages 1 samples [pages] 3 3 3 Test readahead: size=1048577 ramax= iosz=2097152 short read: 0 ->+ 2097152 -> 1048577 1048577 short read: 1048577 ->+ 1048575 -> 1048577 0 snapshot_time 1787741622.305658976 secs.nsecs start_time 1787741621.805637987 secs.nsecs elapsed_time 0.500020989 secs.nsecs hits 256 samples [pages] misses 1 samples [pages] zero_size_window 256 samples [pages] readahead_to_eof 1 samples [pages] failed_to_fast_read 1 samples [pages] readahead_pages 1 samples [pages] 256 256 256 Test readahead: size=1064960 ramax= iosz=2097152 short read: 0 ->+ 2097152 -> 1064960 1064960 short read: 1064960 ->+ 1032192 -> 1064960 0 snapshot_time 1787741622.897869361 secs.nsecs start_time 1787741622.418238350 secs.nsecs elapsed_time 0.479631011 secs.nsecs hits 259 samples [pages] misses 2 samples [pages] zero_size_window 260 samples [pages] failed_to_fast_read 1 samples [pages] readahead_pages 1 samples [pages] 259 259 259 Test readahead: size=1064960 ramax= iosz=2097152 short read: 0 ->+ 2097152 -> 1064960 1064960 short read: 1064960 ->+ 1032192 -> 1064960 0 snapshot_time 1787741623.549097080 secs.nsecs start_time 1787741623.068120736 secs.nsecs elapsed_time 0.480976344 secs.nsecs hits 259 samples [pages] misses 2 samples [pages] zero_size_window 260 samples [pages] failed_to_fast_read 1 samples [pages] readahead_pages 1 samples [pages] 259 259 259 Test readahead: size=2113536 ramax= iosz=3145728 short read: 0 ->+ 3145728 -> 2113536 2113536 short read: 2113536 ->+ 1032192 -> 2113536 0 snapshot_time 1787741624.485223186 secs.nsecs start_time 1787741623.744791223 secs.nsecs elapsed_time 0.740431963 secs.nsecs hits 515 samples [pages] misses 2 samples [pages] zero_size_window 516 samples [pages] failed_to_fast_read 1 samples [pages] readahead_pages 1 samples [pages] 515 515 515 Test readahead: size=4210688 ramax= iosz=5242880 short read: 0 ->+ 5242880 -> 4210688 4210688 short read: 4210688 ->+ 1032192 -> 4210688 0 snapshot_time 1787741625.682277899 secs.nsecs start_time 1787741624.748340856 secs.nsecs elapsed_time 0.933937043 secs.nsecs hits 1026 samples [pages] misses 3 samples [pages] readpage_not_consecutive 1 samples [pages] zero_size_window 1027 samples [pages] failed_to_fast_read 1 samples [pages] readahead_pages 2 samples [pages] 3 1023 1026 PASS 101m (12s) == sanity test 102a: user xattr test ============================================================================================ 06:53:51 (1787741631) set/get xattr... trusted.name1="value1" user.author1="author1" listxattr... remove xattr... set lustre special xattr ... lfs setstripe: setstripe error for '/mnt/lustre/f102a.sanity': stripe already set getfattr: Removing leading '/' from absolute path names setfattr: /mnt/lustre/f102a.sanity: Numerical result out of range PASS 102a (8s) == sanity test 102b: getfattr/setfattr for trusted.lov EAs ========================================================== 06:53:59 (1787741639) test layout '-S 65536 -i 1 -c 2' lmm_stripe_count: 2 lmm_stripe_size: 65536 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xf36:0x0 - l_ost_idx: 0 l_fid: 0x280000401:0xf49:0x0 get/set/list trusted.lov xattr ... getfattr: Removing leading '/' from absolute path names setfattr 4 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 6 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 8 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 10 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 12 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 14 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 16 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 18 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 20 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 22 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 24 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 26 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 28 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 30 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 32 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 34 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 36 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 38 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 40 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 42 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 44 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 46 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 48 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 50 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 52 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 54 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 56 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 58 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 60 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 62 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 64 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 66 /mnt/lustre/f102b.sanity.2 setfattr 68 /mnt/lustre/f102b.sanity.2 setfattr 70 /mnt/lustre/f102b.sanity.2 setfattr 72 /mnt/lustre/f102b.sanity.2 setfattr 74 /mnt/lustre/f102b.sanity.2 setfattr 76 /mnt/lustre/f102b.sanity.2 setfattr 78 /mnt/lustre/f102b.sanity.2 setfattr 80 /mnt/lustre/f102b.sanity.2 setfattr 82 /mnt/lustre/f102b.sanity.2 setfattr 84 /mnt/lustre/f102b.sanity.2 setfattr 86 /mnt/lustre/f102b.sanity.2 setfattr 88 /mnt/lustre/f102b.sanity.2 setfattr 90 /mnt/lustre/f102b.sanity.2 setfattr 92 /mnt/lustre/f102b.sanity.2 setfattr 94 /mnt/lustre/f102b.sanity.2 setfattr 96 /mnt/lustre/f102b.sanity.2 setfattr 98 /mnt/lustre/f102b.sanity.2 setfattr 100 /mnt/lustre/f102b.sanity.2 setfattr 102 /mnt/lustre/f102b.sanity.2 setfattr 104 /mnt/lustre/f102b.sanity.2 setfattr 106 /mnt/lustre/f102b.sanity.2 setfattr 108 /mnt/lustre/f102b.sanity.2 setfattr 110 /mnt/lustre/f102b.sanity.2 setfattr 112 /mnt/lustre/f102b.sanity.2 setfattr 114 /mnt/lustre/f102b.sanity.2 setfattr 116 /mnt/lustre/f102b.sanity.2 setfattr 118 /mnt/lustre/f102b.sanity.2 setfattr 120 /mnt/lustre/f102b.sanity.2 setfattr 122 /mnt/lustre/f102b.sanity.2 setfattr 124 /mnt/lustre/f102b.sanity.2 setfattr 126 /mnt/lustre/f102b.sanity.2 setfattr 128 /mnt/lustre/f102b.sanity.2 setfattr 130 /mnt/lustre/f102b.sanity.2 setfattr 132 /mnt/lustre/f102b.sanity.2 setfattr 134 /mnt/lustre/f102b.sanity.2 setfattr 136 /mnt/lustre/f102b.sanity.2 setfattr 138 /mnt/lustre/f102b.sanity.2 setfattr 140 /mnt/lustre/f102b.sanity.2 setfattr 142 /mnt/lustre/f102b.sanity.2 setfattr 144 /mnt/lustre/f102b.sanity.2 setfattr 146 /mnt/lustre/f102b.sanity.2 setfattr 148 /mnt/lustre/f102b.sanity.2 setfattr 150 /mnt/lustre/f102b.sanity.2 setfattr 152 /mnt/lustre/f102b.sanity.2 setfattr 154 /mnt/lustre/f102b.sanity.2 setfattr 156 /mnt/lustre/f102b.sanity.2 setfattr 158 /mnt/lustre/f102b.sanity.2 setfattr 160 /mnt/lustre/f102b.sanity.2 setfattr 162 /mnt/lustre/f102b.sanity.2 test layout '-E 1M -S 65536 -i 1 -c 2 -Eeof -S4M' lcm_layout_gen: 2 lcm_mirror_count: 1 lcm_entry_count: 2 components: - lcme_id: 1 lcme_mirror_id: 0 lcme_flags: init lcme_extent.e_start: 0 lcme_extent.e_end: 1048576 sub_layout: lmm_stripe_count: 2 lmm_stripe_size: 65536 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 1 lmm_objects: - l_ost_idx: 1 l_fid: 0x2c0000401:0xf38:0x0 - l_ost_idx: 0 l_fid: 0x280000401:0xf4b:0x0 - lcme_id: 2 lcme_mirror_id: 0 lcme_flags: 0 lcme_extent.e_start: 1048576 lcme_extent.e_end: EOF sub_layout: lmm_stripe_count: 2 lmm_stripe_size: 4194304 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: -1 get/set/list trusted.lov xattr ... getfattr: Removing leading '/' from absolute path names setfattr 4 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 6 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 8 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 10 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 12 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 14 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 16 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 18 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 20 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 22 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 24 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 26 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 28 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 30 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 32 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 34 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 36 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 38 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 40 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 42 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 44 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 46 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 48 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 50 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 52 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 54 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 56 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 58 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 60 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 62 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 64 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 66 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 68 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 70 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 72 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 74 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 76 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 78 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 80 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 82 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 84 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 86 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 88 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 90 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 92 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 94 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 96 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 98 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 100 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 102 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 104 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 106 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 108 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 110 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 112 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 114 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 116 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 118 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 120 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 122 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 124 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 126 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 128 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 130 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 132 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 134 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 136 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 138 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 140 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 142 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 144 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 146 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 148 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 150 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 152 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 154 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 156 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 158 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 160 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 162 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 164 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 166 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 168 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 170 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 172 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 174 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 176 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 178 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 180 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 182 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 184 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 186 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 188 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 190 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 192 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 194 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 196 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 198 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 200 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 202 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 204 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 206 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 208 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 210 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 212 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 214 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 216 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 218 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 220 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 222 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 224 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 226 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 228 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 230 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 232 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 234 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 236 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 238 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 240 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 242 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 244 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 246 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 248 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 250 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 252 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 254 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 256 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 258 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 260 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 262 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 264 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 266 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 268 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 270 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 272 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 274 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 276 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 278 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 280 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 282 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 284 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 286 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 288 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 290 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 292 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 294 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 296 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 298 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 300 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 302 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 304 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 306 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 308 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 310 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 312 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 314 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 316 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 318 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 320 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 322 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 324 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 326 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 328 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 330 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 332 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 334 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 336 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 338 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 340 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 342 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 344 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 346 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 348 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 350 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 352 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 354 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 356 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 358 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 360 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 362 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 364 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 366 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 368 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 370 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 372 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 374 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 376 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 378 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 380 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 382 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 384 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 386 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 388 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 390 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 392 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 394 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 396 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 398 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 400 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 402 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 404 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 406 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 408 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 410 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 412 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 414 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 416 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 418 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 420 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 422 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 424 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 426 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 428 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 430 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 432 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 434 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 436 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 438 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 440 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 442 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 444 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 446 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 448 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 450 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 452 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 454 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 456 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 458 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 460 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 462 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 464 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 466 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 468 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 470 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 472 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 474 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 476 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 478 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 480 /mnt/lustre/f102b.sanity.2 setfattr: /mnt/lustre/f102b.sanity.2: Numerical result out of range setfattr 482 /mnt/lustre/f102b.sanity.2 PASS 102b (15s) == sanity test 102c: non-root getfattr/setfattr for lustre.lov EAs ===================================================================== 06:54:14 (1787741654) get/set/list lustre.lov xattr ... striped dir -i0 -c2 -H crush /mnt/lustre/d102c.sanity running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [65536] [-i] [1] [-c] [2] [/mnt/lustre/d102c.sanity/f102c.sanity] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [getstripe] [-c] [/mnt/lustre/d102c.sanity/f102c.sanity] lustre.lov=0s0AvRCwEAAAAZAAAAAAAAAAoEAAACAAAAAAABAAIA//8BBADAAgAAADoPAAAAAAAAAAAAAAEAAAABBACAAgAAAE0PAAAAAAAAAAAAAAAAAAA= running as uid/gid/euid/egid 500/500/500/500, groups: 500 [mcreate] [/mnt/lustre/d102c.sanity/f102c.sanity2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [setfattr] [-n] [lustre.lov] [-v] [0s0AvRCwEAAAAZAAAAAAAAAAoEAAACAAAAAAABAAIA//8BBADAAgAAADoPAAAAAAAAAAAAAAEAAAABBACAAgAAAE0PAAAAAAAAAAAAAAAAAAA=] [/mnt/lustre/d102c.sanity/f102c.sanity2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [getstripe] [-S] [/mnt/lustre/d102c.sanity/f102c.sanity2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [getstripe] [-c] [/mnt/lustre/d102c.sanity/f102c.sanity2] PASS 102c (9s) == sanity test 102d: tar restore stripe info from tarfile,not keep osts ========================================================== 06:54:24 (1787741664) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d102d.sanity PASS 102d (32s) == sanity test 102f: tar copy files, not keep osts ======= 06:54:55 (1787741695) striped dir -i0 -c2 -H crush /mnt/lustre/d102f.sanity striped dir -i0 -c2 -H all_char /mnt/lustre/d102f.sanity.restore PASS 102f (27s) == sanity test 102h: grow xattr from inside inode to external block ========================================================== 06:55:22 (1787741722) save trusted.big on /mnt/lustre/f102h.sanity save trusted.sml on /mnt/lustre/f102h.sanity grow trusted.sml on /mnt/lustre/f102h.sanity trusted.big still valid after growing trusted.sml PASS 102h (17s) == sanity test 102ha: grow xattr from inside inode to external inode ========================================================== 06:55:39 (1787741739) setting xattr of max xattr size: 65536 save trusted.big on /mnt/lustre/f102ha.sanity save trusted.sml on /mnt/lustre/f102ha.sanity grow trusted.sml on /mnt/lustre/f102ha.sanity trusted.big still valid after growing trusted.sml setting xattr of > max xattr size: 65536 + 10 This should fail: save trusted.big on /mnt/lustre/f102ha.sanity setfattr: /mnt/lustre/f102ha.sanity: Argument list too long PASS 102ha (19s) == sanity test 102i: lgetxattr test on symbolic link ====================================================================== 06:55:58 (1787741758) getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f102i.sanity trusted.lov=0s0AvRCwEAAABWAAAAAAAAAAoEAAACAAAAAABAAAEA//8BBACAAgAAAGoPAAAAAAAAAAAAAAAAAAA= /mnt/lustre/f102i.sanitylink: trusted.lov: No such attribute PASS 102i (7s) == sanity test 102j: non-root tar restore stripe info from tarfile, not keep osts ============================================================= 06:56:05 (1787741765) striped dir -i0 -c2 -H crush2 /mnt/lustre/d102j.sanity running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [65536] [-i] [1] [-c] [2] [d102j.sanity] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [65536] [-i] [0] [-c] [1] [file1-0-1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [65536] [-i] [1] [-c] [1] [file1-1-1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [65536] [-i] [0] [-c] [2] [file1-0-2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [65536] [-i] [1] [-c] [2] [file1-1-2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [131072] [-i] [0] [-c] [1] [file2-0-1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [131072] [-i] [1] [-c] [1] [file2-1-1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [131072] [-i] [0] [-c] [2] [file2-0-2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [131072] [-i] [1] [-c] [2] [file2-1-2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [196608] [-i] [0] [-c] [1] [file3-0-1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [196608] [-i] [1] [-c] [1] [file3-1-1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [196608] [-i] [0] [-c] [2] [file3-0-2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [196608] [-i] [1] [-c] [2] [file3-1-2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [262144] [-i] [0] [-c] [1] [file4-0-1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [262144] [-i] [1] [-c] [1] [file4-1-1] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [262144] [-i] [0] [-c] [2] [file4-0-2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lfs] [setstripe] [-S] [262144] [-i] [1] [-c] [2] [file4-1-2] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [tar] [cf] [/tmp/f102.tar] [d102j.sanity] [--xattrs] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [tar] [xf] [/tmp/f102.tar] [-C] [/mnt/lustre/d102j.sanity] [--xattrs] [--xattrs-include=lustre.*] PASS 102j (18s) == sanity test 102k: setfattr without parameter of value shouldn't cause a crash ========================================================== 06:56:23 (1787741783) striped dir -i0 -c2 -H crush /mnt/lustre/d102k.sanity PASS 102k (9s) == sanity test 102l: listxattr size test ============================================================================================ 06:56:32 (1787741792) listxattr as user... PASS 102l (6s) == sanity test 102m: Ensure listxattr fails on small bufffer ================================================================== 06:56:38 (1787741798) PASS 102m (7s) == sanity test 102n: silently ignore setxattr on internal trusted xattrs ========================================================== 06:56:45 (1787741805) setfattr: /mnt/lustre/f102n.sanity.1: Numerical result out of range PASS 102n (9s) == sanity test 102p: check setxattr(2) correctly fails without permission ========================================================== 06:56:54 (1787741814) setfacl as user... running as uid/gid/euid/egid 500/500/500/500, groups: 500 [setfacl] [-m] [u:500:rwx] [/mnt/lustre/f102p.sanity] setfacl: /mnt/lustre/f102p.sanity: Operation not permitted setfattr as user... running as uid/gid/euid/egid 500/500/500/500, groups: 500 [setfattr] [-x] [system.posix_acl_access] [/mnt/lustre/f102p.sanity] setfattr: /mnt/lustre/f102p.sanity: Operation not permitted PASS 102p (8s) == sanity test 102q: flistxattr should not return trusted.link EAs for orphans ========================================================== 06:57:03 (1787741823) PASS 102q (9s) == sanity test 102r: set EAs with empty values =========== 06:57:11 (1787741831) getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/f102r.sanity user.f102r.sanity="" getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d102r.sanity user.d102r.sanity="" striped dir -i0 -c2 -H all_char /mnt/lustre/d102r.sanity getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/d102r.sanity user.d102r.sanity="" PASS 102r (10s) == sanity test 102s: getting nonexistent xattrs should fail ========================================================== 06:57:21 (1787741841) llite.lustre-ffff93f6a0950800.xattr_cache=0 /mnt/lustre/f102s.sanity: lustre.n102s: Operation not supported /mnt/lustre/f102s.sanity: security.n102s: No such attribute /mnt/lustre/f102s.sanity: system.n102s: Operation not supported /mnt/lustre/f102s.sanity: trusted.n102s: No such attribute /mnt/lustre/f102s.sanity: user.n102s: No such attribute llite.lustre-ffff93f6a0950800.xattr_cache=1 /mnt/lustre/f102s.sanity: lustre.n102s: No such attribute /mnt/lustre/f102s.sanity: security.n102s: No such attribute /mnt/lustre/f102s.sanity: system.n102s: Operation not supported /mnt/lustre/f102s.sanity: trusted.n102s: No such attribute /mnt/lustre/f102s.sanity: user.n102s: No such attribute PASS 102s (8s) == sanity test 102t: zero length xattr values handled correctly ========================================================== 06:57:30 (1787741850) llite.lustre-ffff93f6a0950800.xattr_cache=0 llite.lustre-ffff93f6a0950800.xattr_cache=1 PASS 102t (11s) == sanity test 103a: acl test ============================ 06:57:41 (1787741861) /usr/bin/setfacl mdt.lustre-MDT0000.job_xattr=NONE mdt.lustre-MDT0001.job_xattr=NONE uid=1(bin) gid=1(bin) groups=1(bin) uid=2(daemon) gid=2(daemon) groups=2(daemon) users:x:100: Adding user daemon to group bin Adding user daemon to group bin performing cp with bin='bin' daemon='daemon' users='users'... [3] $ umask 022 -- ok [4] $ mkdir d -- ok [5] $ cd d -- ok [6] $ touch f -- ok [7] $ setfacl -m u:bin:rw f -- ok [8] $ ls -l f | awk -- '{ print $1 }' -- ok [11] $ cp f g -- ok [12] $ ls -l g | awk -- '{sub(/\./, "", $1); print $1 }' -- ok [15] $ rm g -- ok [16] $ cp -p f g -- ok [17] $ ls -l f | awk -- '{ print $1 }' -- ok [20] $ mkdir h -- ok [21] $ echo blubb > h/x -- ok [22] $ cp -rp h i -- ok [23] $ cat i/x -- ok [26] $ rm -r i -- ok [31] $ setfacl -R -m u:bin:rwx h -- ok [32] $ getfacl --omit-header h/x -- ok [40] $ cp -rp h i -- ok [41] $ getfacl --omit-header i/x -- ok [49] $ cd .. -- ok [50] $ rm -r d -- ok 22 commands (22 passed, 0 failed) performing getfacl-noacl with bin='bin' daemon='daemon' users='users'... [4] $ mkdir test -- ok [5] $ cd test -- ok [6] $ umask 027 -- ok [7] $ touch x -- ok [8] $ getfacl --omit-header x -- ok [14] $ getfacl --omit-header --access x -- ok [20] $ getfacl --omit-header -d x -- ok [21] $ getfacl --omit-header -d . -- ok [22] $ getfacl --omit-header -d / -- ok [25] $ getfacl --skip-base x -- ok [26] $ getfacl --omit-header --all-effective x -- ok [32] $ getfacl --omit-header --no-effective x -- ok [38] $ mkdir d -- ok [39] $ touch d/y -- ok [46] $ getfacl -dRP . | grep file | sort -- ok [51] $ ln -s d l -- ok [53] $ ln -s l ll -- ok [62] $ rm l ll x -- ok [63] $ rm -rf d -- ok [64] $ cd .. -- ok [65] $ rmdir test -- ok 21 commands (21 passed, 0 failed) performing misc with bin='bin' daemon='daemon' users='users'... [6] $ umask 027 -- ok [7] $ touch f -- ok [10] $ setfacl -m u::r f -- ok [11] $ setfacl -m u::rw,u:bin:rw f -- ok [12] $ ls -dl f | awk '{print $1}' -- ok [15] $ getfacl --omit-header f -- ok [23] $ rm f -- ok [24] $ umask 022 -- ok [25] $ touch f -- ok [26] $ setfacl -m u:bin:rw f -- ok [27] $ ls -dl f | awk '{print $1}' -- ok [30] $ getfacl --omit-header f -- ok [38] $ rm f -- ok [39] $ umask 027 -- ok [40] $ mkdir d -- ok [41] $ setfacl -m u:bin:rwx d -- ok [42] $ ls -dl d | awk '{print $1}' -- ok [45] $ getfacl --omit-header d -- ok [53] $ rmdir d -- ok [54] $ umask 022 -- ok [55] $ mkdir d -- ok [56] $ setfacl -m u:bin:rwx d -- ok [57] $ ls -dl d | awk '{print $1}' -- ok [60] $ getfacl --omit-header d -- ok [68] $ rmdir d -- ok [73] $ umask 022 -- ok [74] $ touch f -- ok [75] $ setfacl -m u:bin:rw,u:daemon:r f -- ok [76] $ ls -dl f | awk '{print $1}' -- ok [79] $ getfacl --omit-header f -- ok [90] $ setfacl -m g:users:rw,g:daemon:r f -- ok [91] $ ls -dl f | awk '{print $1}' -- ok [94] $ getfacl --omit-header f -- ok [107] $ setfacl -x g:users f -- ok [108] $ ls -dl f | awk '{print $1}' -- ok [111] $ getfacl --omit-header f -- ok [123] $ setfacl -x u:daemon f -- ok [124] $ ls -dl f | awk '{print $1}' -- ok [127] $ getfacl --omit-header f -- ok [136] $ rm f -- ok [140] $ umask 027 -- ok [141] $ mkdir d -- ok [142] $ setfacl -m u:bin:rwx,u:daemon:rw,d:u:bin:rwx,d:m:rx d -- ok [143] $ ls -dl d | awk '{print $1}' -- ok [146] $ getfacl --omit-header d -- ok [162] $ umask 027 -- ok [163] $ touch d/f -- ok [164] $ ls -dl d/f | awk '{print $1}' -- ok [167] $ getfacl --omit-header d/f -- ok [175] $ rm d/f -- ok [176] $ umask 022 -- ok [177] $ touch d/f -- ok [178] $ ls -dl d/f | awk '{print $1}' -- ok [181] $ getfacl --omit-header d/f -- ok [189] $ rm d/f -- ok [193] $ umask 000 -- ok [194] $ mkdir d/d -- ok [195] $ ls -dl d/d | awk '{print $1}' -- ok [198] $ getfacl --omit-header d/d -- ok [211] $ rmdir d/d -- ok [212] $ umask 022 -- ok [213] $ mkdir d/d -- ok [214] $ ls -dl d/d | awk '{print $1}' -- ok [217] $ getfacl --omit-header d/d -- ok [232] $ setfacl -nm u:daemon:rx,d:u:daemon:rx,g:users:rx,g:daemon:rwx d/d -- ok [233] $ ls -dl d/d | awk '{print $1}' -- ok [236] $ getfacl --omit-header d/d -- ok [256] $ ln -s d d/l -- ok [257] $ ls -dl d/l | awk '{ sub(/\.$/, "", $1); print $1 }' -- ok [260] $ ls -dl -L d/l | awk '{print $1}' -- ok [265] $ cd d -- ok [266] $ getfacl --omit-header l -- ok [283] $ cd .. -- ok [285] $ rm d/l -- ok [289] $ setfacl -m g:daemon:rx,u:bin:rx d/d -- ok [290] $ ls -dl d/d | awk '{print $1}' -- ok [293] $ getfacl --omit-header d/d -- ok [310] $ setfacl -m d:u:bin:rwx d/d -- ok [311] $ ls -dl d/d | awk '{print $1}' -- ok [314] $ getfacl --omit-header d/d -- ok [331] $ rmdir d/d -- ok [335] $ setfacl -k d -- ok [336] $ ls -dl d | awk '{print $1}' -- ok [339] $ getfacl --omit-header d -- ok [350] $ setfacl -b d -- ok [351] $ ls -dl d | awk '{sub(/\./, "", $1); print $1}' -- ok [354] $ getfacl --omit-header d -- ok [362] $ chmod 775 d -- ok [363] $ ls -dl d | awk '{sub(/\./, "", $1); print $1}' -- ok [366] $ getfacl --omit-header d -- ok [372] $ rmdir d -- ok [373] $ umask 002 -- ok [374] $ mkdir d -- ok [375] $ setfacl -m u:daemon:rwx,u:bin:rx,d:u:daemon:rwx,d:u:bin:rx d -- ok [376] $ ls -dl d | awk '{print $1}' -- ok [379] $ getfacl --omit-header d -- ok [394] $ chmod 750 d -- ok [395] $ ls -dl d | awk '{print $1}' -- ok [398] $ getfacl --omit-header d -- ok [413] $ chmod 750 d -- ok [414] $ ls -dl d | awk '{print $1}' -- ok [417] $ getfacl --omit-header d -- ok [432] $ rmdir d -- ok 103 commands (103 passed, 0 failed) performing permissions with bin='bin' daemon='daemon' users='users'... [12] $ id -u -- ok [19] $ mkdir d -- ok [20] $ cd d -- ok [21] $ umask 027 -- ok [22] $ touch f -- ok [23] $ ls -l f | awk -- '{sub(/\./, "", $1); print $1, $3, $4 }' -- ok [30] $ echo root > f -- ok [32] $ su daemon -- ok [33] $ echo daemon >> f -- ok [36] $ su -- ok [42] $ chown bin:bin f -- ok [43] $ ls -l f | awk -- '{sub(/\./, "", $1); print $1, $3, $4 }' -- ok [45] $ su bin -- ok [46] $ echo bin >> f -- ok [52] $ su daemon -- ok [53] $ cat f -- ok [57] $ echo daemon >> f -- ok [64] $ su bin -- ok [65] $ setfacl -m u:daemon:rw f -- ok [66] $ getfacl --omit-header f -- ok [77] $ su daemon -- ok [78] $ echo daemon >> f -- ok [79] $ cat f -- ok [88] $ su bin -- ok [89] $ chmod g-w f -- ok [90] $ getfacl --omit-header f -- ok [98] $ su daemon -- ok [99] $ echo daemon >> f -- ok [108] $ su bin -- ok [109] $ setfacl -m u:daemon:r,g:daemon:rw-,o::rw- f -- ok [111] $ su daemon -- ok [112] $ echo daemon >> f -- ok [119] $ su bin -- ok [120] $ setfacl -x u:daemon f -- ok [122] $ su daemon -- ok [123] $ echo daemon2 >> f -- ok [124] $ cat f -- ok [134] $ su bin -- ok [135] $ setfacl -m g:daemon:r f -- ok [137] $ su daemon -- ok [138] $ echo daemon3 >> f -- ok [145] $ su bin -- ok [146] $ setfacl -x g:daemon f -- ok [148] $ su daemon -- ok [149] $ echo daemon4 >> f -- ok [156] $ su -- ok [157] $ chgrp root f -- ok [159] $ su daemon -- ok [160] $ echo daemon5 >> f -- ok [161] $ cat f -- ok [172] $ su -- ok [173] $ setfacl -m g:bin:r,g:daemon:w f -- ok [175] $ su daemon -- ok [176] $ : < f -- ok [177] $ : > f -- ok [178] $ : <> f -- ok [186] $ su -- ok [187] $ mkdir -m 750 e -- ok [188] $ touch e/h -- ok [190] $ su bin -- ok [191] $ shopt -s nullglob ; echo e/* -- ok [194] $ echo i > e/i -- ok [197] $ su -- ok [198] $ setfacl -m u:bin:rx e -- ok [200] $ su bin -- ok [201] $ echo e/* -- ok [208] $ touch e/i 2>&1 | sed -e "s/touch .*e\/i.*:/touch \'e\/i\':/" -- ok [211] $ su -- ok [212] $ setfacl -m u:bin:rwx e -- ok [214] $ su bin -- ok [215] $ echo i > e/i -- ok [220] $ su -- ok [221] $ touch g -- ok [222] $ ln -s g l -- ok [223] $ setfacl -m u:bin:rw l -- ok [224] $ ls -l g | awk -- '{ print $1, $3, $4 }' -- ok [234] $ mknod -m 0660 hdt b 91 64 -- ok [235] $ mknod -m 0660 null c 1 3 -- ok [236] $ mkfifo -m 0660 fifo -- ok [238] $ su bin -- ok [239] $ : < hdt -- ok [241] $ : < null -- ok [243] $ : < fifo -- ok [246] $ su -- ok [247] $ setfacl -m u:bin:rw hdt null fifo -- ok [249] $ su bin -- ok [250] $ : < hdt -- ok [252] $ : < null -- ok [253] $ ( echo blah > fifo & ) ; cat fifo -- ok [261] $ su -- ok [262] $ mkdir -m 600 x -- ok [263] $ chown daemon:daemon x -- ok [264] $ echo j > x/j -- ok [265] $ ls -l x/j | awk -- '{sub(/\./, "", $1); print $1, $3, $4 }' -- ok [268] $ setfacl -m u:daemon:r x -- ok [270] $ ls -l x/j | awk -- '{sub(/\./, "", $1); print $1, $3, $4 }' -- ok [274] $ echo k > x/k -- ok [277] $ chmod 750 x -- ok [282] $ su -- ok [283] $ cd .. -- ok [284] $ rm -rf d -- ok 101 commands (101 passed, 0 failed) 65534 nobody:x:65534: /usr/bin/setfattr performing permissions_xattr with bin='bin' daemon='daemon' users='users'... [11] $ id -u -- ok [19] $ mkdir d -- ok [20] $ cd d -- ok [21] $ umask 027 -- ok [22] $ touch f -- ok [23] $ chown nobody:nobody f -- ok [24] $ ls -l f | awk -- '{ sub(/\.$/, "", $1); print $1, $3, $4 }' -- ok [26] $ su nobody -- ok [27] $ echo nobody > f -- ok [33] $ su bin -- ok [34] $ setfattr -n user.test.xattr -v 123456 f -- ok [41] $ su nobody -- ok [42] $ setfacl -m g:bin:rw f -- ok [43] $ getfacl --omit-header f -- ok [55] $ su bin -- ok [56] $ setfattr -n user.test.xattr -v 123456 f -- ok [57] $ getfattr -d f -- ok [66] $ su -- ok [67] $ ln -s f l -- ok [68] $ ls -l l | awk -- '{ sub(/\.$/, "", $1); print $1, $3, $4 }' -- ok [70] $ su bin -- ok [71] $ getfattr -d l -- ok [81] $ su -- ok [82] $ mkdir t -- ok [83] $ chown nobody:nobody t -- ok [84] $ chmod 1750 t -- ok [85] $ ls -dl t | awk -- '{ sub(/\.$/, "", $1); print $1, $3, $4 }' -- ok [87] $ su nobody -- ok [88] $ setfacl -m g:bin:rwx t -- ok [89] $ getfacl --omit-header t -- ok [96] $ su bin -- ok [97] $ setfattr -n user.test.xattr -v 654321 t -- ok [105] $ su -- ok [106] $ mkdir d -- ok [107] $ chown nobody:nobody d -- ok [108] $ chmod 750 d -- ok [109] $ ls -dl d | awk -- '{ sub(/\.$/, "", $1); print $1, $3, $4 }' -- ok [111] $ su nobody -- ok [112] $ setfacl -m g:bin:rwx d -- ok [113] $ getfacl --omit-header d -- ok [120] $ su bin -- ok [121] $ setfattr -n user.test.xattr -v 654321 d -- ok [122] $ getfattr -d d -- ok [131] $ su -- ok [132] $ mknod -m 0660 hdt b 91 64 -- ok [133] $ mknod -m 0660 null c 1 3 -- ok [134] $ mkfifo -m 0660 fifo -- ok [135] $ setfattr -n user.test.xattr -v 123456 hdt -- ok [137] $ setfattr -n user.test.xattr -v 123456 null -- ok [139] $ setfattr -n user.test.xattr -v 123456 fifo -- ok [145] $ su -- ok [146] $ cd .. -- ok [147] $ rm -rf d -- ok 53 commands (53 passed, 0 failed) performing setfacl with bin='bin' daemon='daemon' users='users'... [3] $ mkdir d -- ok [4] $ chown bin:bin d -- ok [5] $ cd d -- ok [7] $ su bin -- ok [8] $ sg bin -- [(1,0)(1 1,1 1)]ok [9] $ umask 027 -- ok [10] $ touch g -- ok [11] $ ls -dl g | awk '{sub(/\./, "", $1); print $1}' -- ok [14] $ setfacl -m m:- g -- ok [15] $ ls -dl g | awk '{print $1}' -- ok [18] $ getfacl g -- ok [28] $ setfacl -x m g -- ok [29] $ getfacl g -- ok [38] $ setfacl -m u:daemon:rw g -- ok [39] $ getfacl g -- ok [50] $ setfacl -m u::rwx,g::r-x,o:- g -- ok [51] $ getfacl g -- ok [62] $ setfacl -m u::rwx,g::r-x,o:-,m:- g -- ok [63] $ getfacl g -- ok [74] $ setfacl -m u::rwx,g::r-x,o:-,u:root:-,m:- g -- ok [75] $ getfacl g -- ok [87] $ setfacl -m u::rwx,g::r-x,o:-,u:root:-,m:- g -- ok [88] $ getfacl g -- ok [100] $ setfacl -m u::rwx,g::r-x,o:-,u:root:- g -- ok [101] $ getfacl g -- ok [113] $ setfacl --test -x u: g -- ok [116] $ setfacl --test -x u:x -- ok [119] $ setfacl -m d:u:root:rwx g -- ok [122] $ setfacl -x m g -- ok [129] $ mkdir d -- ok [130] $ setfacl --test -m u::rwx,u:bin:rwx,g::r-x,o::--- d -- ok [133] $ setfacl --test -m u::rwx,u:bin:rwx,g::r-x,m::---,o::--- d -- ok [136] $ setfacl --test -d -m u::rwx,u:bin:rwx,g::r-x,o::--- d -- ok [139] $ setfacl --test -d -m u::rwx,u:bin:rwx,g::r-x,m::---,o::--- d -- ok [142] $ su -- ok [143] $ cd .. -- ok [144] $ rm -r d -- ok 37 commands (37 passed, 0 failed) performing inheritance with bin='bin' daemon='daemon' users='users'... [4] $ id -u -- ok [7] $ mkdir d -- ok [8] $ setfacl -d -m group:bin:r-x d -- ok [9] $ getfacl d -- ok [23] $ mkdir d/subdir -- ok [24] $ getfacl d/subdir -- ok [40] $ touch d/f -- ok [41] $ ls -l d/f | awk -- '{ print $1 }' -- ok [43] $ getfacl d/f -- ok [54] $ su bin -- ok [55] $ echo i >> d/f -- ok [62] $ su -- ok [63] $ rm d/f -- ok [64] $ rmdir d/subdir -- ok [65] $ mv d tree -- ok [66] $ ./make-tree -- ok [67] $ getfacl tree/dir0/dir5/file4 -- ok [77] $ getfacl tree/dir0/dir6/file4 -- ok [87] $ echo i >> tree/dir6/dir2/file2 -- ok [88] $ echo i > tree/dir1/f -- ok [89] $ ls -l tree/dir1/f | awk -- '{ print $1 }' -- ok [98] $ rm -rf tree -- ok 22 commands (22 passed, 0 failed) LU-974 ignore umask when acl is enabled... performing 974 with bin='bin' daemon='daemon' users='users'... [3] $ umask 022 -- ok [4] $ mkdir 974 -- ok [6] $ touch 974/f1 -- ok [7] $ ls -dl 974/f1 | awk '{sub(/\./, "", $1); print $1 }' -- ok [10] $ setfacl -R -d -m mask:007 974 -- ok [11] $ touch 974/f2 -- ok [12] $ ls -dl 974/f2 | awk '{ print $1 }' -- ok [15] $ umask 077 -- ok [16] $ touch f3 -- ok [17] $ ls -dl f3 | awk '{sub(/\./, "", $1); print $1 }' -- ok [20] $ rm -rf 974 -- ok 11 commands (11 passed, 0 failed) performing 974_remote with bin='bin' daemon='daemon' users='users'... [4] $ umask 022 -- ok [5] $ lfs mkdir -i 1 974 -- ok [7] $ touch 974/f1 -- ok [8] $ ls -dl 974/f1 | awk '{ sub(/\.$/, "", $1); print $1 }' -- ok [11] $ setfacl -R -d -m mask:007 974 -- ok [12] $ touch 974/f2 -- ok [13] $ ls -dl 974/f2 | awk '{ sub(/\.$/, "", $1); print $1 }' -- ok [16] $ umask 077 -- ok [17] $ touch f3 -- ok [18] $ ls -dl f3 | awk '{ sub(/\.$/, "", $1); print $1 }' -- ok [21] $ rm -rf 974 -- ok 11 commands (11 passed, 0 failed) LU-2561 newly created file is same size as directory... performing 2561 with bin='bin' daemon='daemon' users='users'... [3] $ mkdir -p 2561 -- ok [4] $ cd 2561 -- ok [5] $ getfacl --access . | setfacl -d -M- . -- ok [6] $ touch f1 -- ok [7] $ stat -c '%s' f1 -- ok [9] $ cd .. -- ok [10] $ rm -rf 2561 -- ok 7 commands (7 passed, 0 failed) performing 4924 with bin='bin' daemon='daemon' users='users'... [3] $ mkdir 4924 -- ok [4] $ cd 4924 -- ok [5] $ touch f -- ok [6] $ chmod u=rwx,g=rwxs f -- ok [7] $ ls -l f | awk -- '{sub(/\./, "", $1); print $1, $3, $4 }' -- ok [9] $ touch f -- ok [10] $ ls -l f | awk -- '{sub(/\./, "", $1); print $1, $3, $4 }' -- ok [12] $ cd .. -- ok [13] $ rm -rf 4924 -- ok 9 commands (9 passed, 0 failed) mdt.lustre-MDT0000.job_xattr=user.job mdt.lustre-MDT0001.job_xattr=user.job PASS 103a (333s) == sanity test 103b: umask lfs setstripe ================= 07:03:14 (1787742194) PASS 103b (217s) == sanity test 103c: 'cp -rp' won't set empty acl ======== 07:06:52 (1787742412) getfattr: Removing leading '/' from absolute path names getfattr: Removing leading '/' from absolute path names PASS 103c (13s) == sanity test 103e: inheritance of big amount of default ACLs ========================================================== 07:07:04 (1787742424) mdc.lustre-MDT0000-mdc-ffff93f6a0950800.stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.stats=clear debug=0 6861 default ACLs created File: /mnt/lustre/d103e.sanity Size: 4096 Blocks: 120 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205423500081 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 07:07:08.000000000 -0400 Modify: 2026-08-26 07:07:08.000000000 -0400 Change: 2026-08-26 07:07:24.000000000 -0400 Birth: 2026-08-26 07:07:08.000000000 -0400 File: /mnt/lustre/d103e.sanity/f103e.sanity Size: 0 Blocks: 0 IO Block: 4194304 regular empty file Device: 2c54f966h/743766374d Inode: 144115205423500083 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 07:07:26.000000000 -0400 Modify: 2026-08-26 07:07:26.000000000 -0400 Change: 2026-08-26 07:07:26.000000000 -0400 Birth: 2026-08-26 07:07:25.000000000 -0400 6861 ACLs were inherited Added 0 more ACLs to the file Total 6862 ACLs in file mdc.lustre-MDT0000-mdc-ffff93f6a0950800.stats= snapshot_time 1787742458.028872714 secs.nsecs start_time 1787742429.289959054 secs.nsecs elapsed_time 28.738913660 secs.nsecs req_waittime 839 samples [usecs] 2070 603763 12731805 924037802215 req_active 910 samples [reqs] 1 3 1259 1963 ldlm_ibits_enqueue 283 samples [reqs] 1 1 283 283 mds_close 1 samples [usecs] 8897 8897 8897 79156609 mds_getxattr 204 samples [usecs] 2096 53998 2393738 39706287460 mds_batch 1 samples [usecs] 15683 15683 15683 245956489 ldlm_cancel 275 samples [usecs] 2023 38201 3077633 50231446445 obd_ping 1 samples [usecs] 19721 19721 19721 388917841 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.stats= snapshot_time 1787742458.028957381 secs.nsecs start_time 1787742429.290094181 secs.nsecs elapsed_time 28.738863200 secs.nsecs req_waittime 6 samples [usecs] 6192 19121 76259 1167246787 req_active 6 samples [reqs] 1 1 6 6 obd_ping 6 samples [usecs] 6091 18627 73343 1092433321 debug=trace inode super iotrace malloc cache info ioctl neterror net warning buffs other dentry nettrace page dlmtrace error emerg ha rpctrace vfstrace reada mmap config console quota sec lfsck hsm snapshot layout PASS 103e (34s) == sanity test 103f: changelog doesn't interfere with default ACLs buffers ========================================================== 07:07:38 (1787742458) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 PASS 103f (22s) == sanity test 103g: no MDS_GETXATTR storm for inodes without ACL_ACCESS (LU-17238) ========================================================== 07:08:00 (1787742480) /usr/bin/getfacl running as uid/gid/euid/egid 500/500/500/500, groups: 500 [test] [-r] [/mnt/lustre] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [test] [-r] [/mnt/lustre/f103g.sanity] getxattr ACL RPCs: 0 PASS 103g (8s) == sanity test 104a: lfs df [-ih] [path] test =================================================================================== 07:08:09 (1787742489) UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5020 1266284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3412 1267892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9768 3597252 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9888 3597104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194356 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 459 999.6K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 402 999.6K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 492 255.5K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 456 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.9K 861 511.1K 1% /mnt/lustre UUID bytes Used Available Use% Mounted on lustre-MDT0000_UUID 1.3G 4.9M 1.2G 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1.3G 3.3M 1.2G 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3.7G 9.5M 3.4G 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3.7G 9.7M 3.4G 1% /mnt/lustre[OST:1] filesystem_summary: 7.3G 19.2M 6.9G 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1024000 459 1023541 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1024000 402 1023598 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 262144 492 261652 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 262144 456 261688 1% /mnt/lustre[OST:1] filesystem_summary: 524201 861 523340 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5020 1266284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3412 1267892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9768 3597252 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9888 3597104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194356 1% /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 1000.0K 459 999.6K 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1000.0K 402 999.6K 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 256.0K 492 255.5K 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 256.0K 456 255.6K 1% /mnt/lustre[OST:1] filesystem_summary: 511.9K 861 511.1K 1% /mnt/lustre UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5020 1266284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3412 1267892 1% /mnt/lustre[MDT:1] lustre-OST0001_UUID 3833116 9888 3597104 1% /mnt/lustre[OST:1] filesystem_summary: 3833116 9888 3597104 1% /mnt/lustre oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff93f6a0950800.ost_server_uuid 50 osc.lustre-OST0000-osc-ffff93f6a0950800.ost_server_uuid in FULL state after 0 sec UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 5020 1266284 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 3412 1267892 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 9768 3597252 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 9888 3597104 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 19656 7194356 1% /mnt/lustre PASS 104a (17s) == sanity test 104b: runas -u 500 -g 500 lfs check servers test ============================================================================== 07:08:26 (1787742506) PASS 104b (7s) == sanity test 104c: Verify df vs lfs_df stays same after recordsize change ========================================================== 07:08:33 (1787742513) SKIP: sanity test_104c zfs only test SKIP 104c (3s) == sanity test 104d: runas -u 500 -g 500 lctl dl test ==== 07:08:36 (1787742516) running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lctl] [dl] == running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lctl] [dl] 5 UP osc lustre-OST0000-osc-ffff93f6a0950800 93404b66-1099-449f-b183-1d25741371ef 5 6 UP osc lustre-OST0001-osc-ffff93f6a0950800 93404b66-1099-449f-b183-1d25741371ef 5 == running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lctl] [dl] == running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lctl] [dl] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lctl] [dl] running as uid/gid/euid/egid 500/500/500/500, groups: 500 [/home/green/git/lustre-release/lustre/utils/lctl] [dl] PASS 104d (8s) == sanity test 105a: flock when mounted without -o flock test ================================================================== 07:08:44 (1787742524) PASS 105a (7s) == sanity test 105b: fcntl when mounted without -o flock test ================================================================== 07:08:51 (1787742531) PASS 105b (9s) == sanity test 105c: lockf when mounted without -o flock test ========================================================== 07:09:00 (1787742540) PASS 105c (7s) == sanity test 105d: flock race (should not freeze) ================================================================== 07:09:07 (1787742547) striped dir -i1 -c2 -H crush2 /mnt/lustre/d105d.sanity fail_loc=0x80000315 fcntl cmd 5 failed: Invalid argument thread 1: set write lock (blocking): rc = 0 thread 2: unlock: rc = 0 thread 2: unlock done: rc = 0 thread 2: set write lock (non-blocking): rc = 0 thread 2: set write lock done: rc = 0 thread 1: set write lock done: rc = 0 thread 1: unlock: rc = 0 thread 1: unlock done: rc = 0 PASS 105d (17s) == sanity test 105e: Two conflicting flocks from same process ========================================================== 07:09:25 (1787742565) PASS 105e (8s) == sanity test 105f: Enqueue same range flocks =========== 07:09:32 (1787742572) Time for processing 1.943s Time for processing 1.905s Time for processing 1.988s Time for processing 2.013s Time for processing 2.074s Time for processing 2.131s Time for processing 2.065s Time for processing 2.128s Time for processing 2.051s Time for processing 1.958s Time for processing 2.068s Time for processing 2.205s Time for processing 2.109s Time for processing 2.097s Time for processing 2.076s Time for processing 2.038s Time for processing 2.099s Time for processing 2.043s Time for processing 2.117s Time for processing 2.072s Time for processing 2.091s PASS 105f (13s) == sanity test 105g: ldlm_lock_debug stack test ========== 07:09:45 (1787742585) fail_loc=0x8000032f thread 1: set write lock (blocking): rc = 0 thread 1: set write lock done: rc = 0 thread 1: unlock: rc = 0 thread 1: unlock done: rc = 0 thread 2: unlock: rc = 0 thread 2: unlock done: rc = 0 thread 2: set write lock (non-blocking): rc = 0 thread 2: set write lock done: rc = 0 PASS 105g (11s) == sanity test 105h: Flock functional verify ============= 07:09:56 (1787742596) Time for processing 0.068s Time for processing 0.110s Time for processing 0.047s Time for processing 0.095s Time for processing 0.083s Time for processing 0.065s Time for processing 0.076s PASS 105h (9s) == sanity test 105i: Flock deadlock verify =============== 07:10:05 (1787742605) flocks_test: cannot set lock(35): Resource deadlock avoided PASS 105i (14s) == sanity test 106: attempt exec of dir followed by chown of that dir ========================================================== 07:10:19 (1787742619) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d106.sanity /home/green/git/lustre-release/lustre/tests/sanity.sh: line 14854: /mnt/lustre/d106.sanity: Is a directory PASS 106 (8s) == sanity test 107: Coredump on SIG ====================== 07:10:28 (1787742628) kernel.core_pattern = core kernel.core_uses_pid = 0 /home/green/git/lustre-release/lustre/tests/sanity.sh: line 14880: 121433 Segmentation fault (core dumped) sleep 60 kernel.core_pattern = |/usr/lib/systemd/systemd-coredump sysctl: "%P" must be of the form name=value sysctl: "%u" must be of the form name=value sysctl: "%g" must be of the form name=value sysctl: "%s" must be of the form name=value sysctl: "%t" must be of the form name=value sysctl: "%c" must be of the form name=value sysctl: "%h" must be of the form name=value sysctl: "%e" must be of the form name=value kernel.core_uses_pid = 1 PASS 107 (9s) == sanity test 110: filename length checking ============= 07:10:37 (1787742637) striped dir -i0 -c2 -H crush2 /mnt/lustre/d110.sanity striped dir -i0 -c2 -H all_char /mnt/lustre/d110.sanity/aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa lfs mkdir: dirstripe error on '/mnt/lustre/d110.sanity/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb': File name too long lfs setdirstripe: cannot create dir '/mnt/lustre/d110.sanity/bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb': File name too long touch: cannot touch '/mnt/lustre/d110.sanity/yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy': File name too long total 8 drwxr-xr-x 2 root root 8192 Aug 26 07:10 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa -rw-r--r-- 1 root root 0 Aug 26 07:10 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx PASS 110 (10s) == sanity test 116a: stripe QOS: free space balance ============================================================================= 07:10:47 (1787742647) Free space priority 90% Waiting for MDT destroys to complete OST kbytes available: 3597252 3597104 Min free space: OST 1: 3597104 Max free space: OST 0: 3597252 striped dir -i0 -c2 -H crush /mnt/lustre/d116a.sanity/OST1 Check for uneven OSTs: diff=148KB (0%) must be > 17% ...no Fill 19% remaining space in OST1 with 683449KB Waiting for MDT destroys to complete OST kbytes available: 3597252 2915120 Min free space: OST 1: 2915120 Max free space: OST 0: 3597252 diff=682132=23% must be > 17% for QOS mode...ok writing 600 files to QOS-assigned OSTs wrote 600 200k files Note: free space may not be updated, so measurements might be off Waiting for MDT destroys to complete OST kbytes available: 3530652 2861720 Min free space: OST 1: 2861720 Max free space: OST 0: 3530652 free space delta: orig 682132 final 668932 Wrote 53400KB to smaller OST 1 Wrote 66600KB to larger OST 0 Wrote 24% more data to larger OST 0 lustre-OST0001_UUID 267 files created on smaller OST 1 lustre-OST0000_UUID 333 files created on larger OST 0 Wrote 24% more files to larger OST 0 Waiting for MDT destroys to complete cleanup time 34 PASS 116a (186s) == sanity test 116b: QoS shouldn't LBUG if not enough OSTs found on the 2nd pass ========================================================== 07:13:53 (1787742833) lod.lustre-MDT0000-mdtlov.qos_threshold_rr=0 lov.lustre-MDT0000-mdtlov.qos_threshold_rr=0 fail_loc=0x147 total: 20 open/close in 1.12 seconds: 17.88 ops/second fail_loc=0 lod.lustre-MDT0000-mdtlov.qos_threshold_rr=17% lov.lustre-MDT0000-mdtlov.qos_threshold_rr=17% PASS 116b (19s) == sanity test 117: verify osd extend ==================== 07:14:12 (1787742852) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.287065 s, 3.7 MB/s fail_loc=0x21e fail_loc=0 Truncate succeeded. PASS 117 (9s) resend_count is set to 4 4 == sanity test 118a: verify O_SYNC works ================= 07:14:22 (1787742862) 2+0 records in 2+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0261791 s, 5.0 MB/s PASS 118a (7s) == sanity test 118b: Reclaim dirty pages on fatal error ==================================================================== 07:14:30 (1787742870) 2+0 records in 2+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0497817 s, 2.6 MB/s fail_val=0 fail_loc=0x217 open: No such file or directory fail_val=0 fail_loc=0 Dirty pages not leaked on ENOENT PASS 118b (13s) SKIP: sanity test_118c skipping ALWAYS excluded test 118c resend_count is set to 4 4 SKIP: sanity test_118d skipping ALWAYS excluded test 118d == sanity test 118f: Simulate unrecoverable OSC side error ==================================================================== 07:14:46 (1787742886) 2+0 records in 2+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0800314 s, 1.6 MB/s fail_loc=0x8000040a write: Input/output error fail_loc=0x0 No pages locked after fsync 2+0 records in 2+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0299638 s, 4.4 MB/s PASS 118f (11s) == sanity test 118g: Don't stay in wait if we got local -ENOMEM ==================================================================== 07:14:58 (1787742898) 2+0 records in 2+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0798087 s, 1.6 MB/s fail_loc=0x406 write: Input/output error fail_loc=0 No pages locked after fsync 2+0 records in 2+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0569921 s, 2.3 MB/s PASS 118g (13s) == sanity test 118h: Verify timeout in handling recoverables errors ==================================================================== 07:15:10 (1787742910) 2+0 records in 2+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.042027 s, 3.1 MB/s fail_val=0 fail_loc=0x20e write: Input/output error fail_val=0 fail_loc=0 No pages locked after fsync PASS 118h (26s) resend_count is set to 4 4 == sanity test 118i: Fix error before timeout in recoverable error ==================================================================== 07:15:36 (1787742936) 2+0 records in 2+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0525617 s, 2.5 MB/s fail_val=0 fail_loc=0x20e fail_val=0 fail_loc=0 No pages locked after fsync PASS 118i (22s) resend_count is set to 4 4 == sanity test 118j: Simulate unrecoverable OST side error ==================================================================== 07:15:58 (1787742958) 2+0 records in 2+0 records out 131072 bytes (131 kB, 128 KiB) copied, 0.0290068 s, 4.5 MB/s fail_val=0 fail_loc=0x220 write: Bad address fail_val=0 fail_loc=0x0 No pages locked after fsync PASS 118j (12s) == sanity test 118k: bio alloc -ENOMEM and IO TERM handling =================================================================== 07:16:10 (1787742970) fail_val=0 fail_loc=0x20e striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d118k.sanity /home/green/git/lustre-release/lustre/tests/sanity.sh: line 15429: 131454 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 15429: 131457 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 15429: 131460 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 15429: 131463 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.90798 s, 5.5 MB/s /home/green/git/lustre-release/lustre/tests/sanity.sh: line 15429: 131466 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.8864 s, 5.6 MB/s /home/green/git/lustre-release/lustre/tests/sanity.sh: line 15429: 131469 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 15429: 131473 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 15429: 131476 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 15429: 131479 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) /home/green/git/lustre-release/lustre/tests/sanity.sh: line 15429: 131482 Terminated ( dd if=/dev/zero of=$DIR/$tdir/$tfile-$i bs=1M count=10 || error "dd to $DIR/$tdir/$tfile-$i failed" ) fail_val=0 fail_loc=0 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 4.20455 s, 2.5 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 7.78484 s, 1.3 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 7.19654 s, 1.5 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 8.7516 s, 1.2 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 8.25438 s, 1.3 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 12.4721 s, 841 kB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 11.9679 s, 876 kB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 14.3834 s, 729 kB/s PASS 118k (27s) == sanity test 118l: fsync dir =========================== 07:16:37 (1787742997) striped dir -i0 -c2 -H crush2 /mnt/lustre/d118l.sanity PASS 118l (8s) == sanity test 118m: fdatasync dir ======================= 07:16:45 (1787743005) striped dir -i0 -c2 -H crush /mnt/lustre/d118m.sanity PASS 118m (8s) resend_count is set to 4 4 == sanity test 118n: statfs() sends OST_STATFS requests in parallel ========================================================== 07:16:53 (1787743013) fail_val=0 fail_loc=0x242 fail_val=0 fail_loc=0 PASS 118n (12s) == sanity test 119a: Short directIO read must return actual read amount ========================================================== 07:17:05 (1787743025) directio on /mnt/lustre/f119a.sanity for 1x524288 bytes PASS PASS 119a (8s) == sanity test 119b: Sparse directIO read must return actual read amount ========================================================== 07:17:13 (1787743033) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.134381 s, 7.8 MB/s PASS 119b (8s) == sanity test 119c: Testing for direct read hitting hole ========================================================== 07:17:21 (1787743041) directio on /mnt/lustre/f119c.sanity for 1x1048576 bytes PASS directio on /mnt/lustre/f119c.sanity for 2x1048576 bytes PASS PASS 119c (8s) == sanity test 119e: Basic tests of dio read and write at various sizes ========================================================== 07:17:29 (1787743049) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.56101 s, 16.8 MB/s 4+0 records in 4+0 records out 16380 bytes (16 kB, 16 KiB) copied, 0.396802 s, 41.3 kB/s llite.lustre-ffff93f6a0950800.unaligned_dio=0 testing disabling unaligned DIO - 'invalid argument' expected: dd: error reading '/mnt/lustre/f119e.sanity.1': Invalid argument 0+0 records in 0+0 records out 0 bytes copied, 0.00243 s, 0.0 kB/s llite.lustre-ffff93f6a0950800.unaligned_dio=1 Read/write with DIO at size 1044480 25+1 records in 25+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.60344 s, 7.3 MB/s -rw-r--r-- 1 root root 26214400 Aug 26 07:17 /mnt/lustre/f119e.sanity.1 -rw-r--r-- 1 root root 26214400 Aug 26 07:17 /mnt/lustre/f119e.sanity.2 /mnt/lustre/f119e.sanity.2 has type file OK /mnt/lustre/f119e.sanity.2 has size 26214400 OK Read/write with DIO at size 1048576 25+0 records in 25+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 3.76352 s, 7.0 MB/s -rw-r--r-- 1 root root 26214400 Aug 26 07:17 /mnt/lustre/f119e.sanity.1 -rw-r--r-- 1 root root 26214400 Aug 26 07:17 /mnt/lustre/f119e.sanity.2 /mnt/lustre/f119e.sanity.2 has type file OK /mnt/lustre/f119e.sanity.2 has size 26214400 OK Read/write with DIO at size 1049600 24+1 records in 24+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.14872 s, 6.3 MB/s -rw-r--r-- 1 root root 26214400 Aug 26 07:17 /mnt/lustre/f119e.sanity.1 -rw-r--r-- 1 root root 26214400 Aug 26 07:17 /mnt/lustre/f119e.sanity.2 /mnt/lustre/f119e.sanity.2 has type file OK /mnt/lustre/f119e.sanity.2 has size 26214400 OK 1048577 24 0 1048578 314 0 1048579 212 0 1048580 242 0 1048581 250 0 1048582 343 0 1048583 220 0 1048584 241 0 1048585 344 0 1048586 326 0 1048587 343 0 1048588 356 0 1048589 361 0 1048590 264 0 1048591 253 0 1048592 101 0 1048593 54 0 1048594 313 0 1048595 351 0 1048596 175 0 1048597 352 0 1048598 321 0 1048599 354 0 1048600 34 0 1048601 322 0 1048602 117 0 1048603 320 0 1048604 120 0 1048605 226 0 1048606 215 0 1048607 344 0 1048608 350 0 1048609 167 0 1048610 274 0 1048611 307 0 1048612 151 0 1048613 166 0 1048614 46 0 1048615 325 0 1048616 143 0 1048617 247 0 1048618 37 0 1048619 21 0 1048620 47 0 1048621 1 0 1048622 123 0 1048623 136 0 1048624 275 0 1048625 177 0 1048626 307 0 1048627 264 0 1048628 116 0 1048629 261 0 1048630 66 0 1048631 53 0 1048632 344 0 1048633 347 0 1048634 12 0 1048635 66 0 1048636 365 0 1048637 267 0 1048638 355 0 1048639 255 0 1048640 145 0 1048641 234 0 1048642 166 0 1048643 145 0 1048644 366 0 1048645 47 0 1048646 73 0 1048647 203 0 1048648 223 0 1048649 31 0 1048650 320 0 1048651 331 0 1048652 325 0 1048653 267 0 1048654 231 0 1048655 64 0 1048656 161 0 1048657 207 0 1048659 152 0 1048660 2 0 1048661 22 0 1048662 264 0 1048663 145 0 1048664 136 0 1048665 334 0 1048666 351 0 1048667 115 0 1048668 41 0 1048669 24 0 1048670 16 0 1048671 264 0 1048672 304 0 1048673 366 0 1048674 304 0 1048675 342 0 1048676 106 0 1048677 22 0 1048678 320 0 1048679 173 0 1048680 174 0 1048681 373 0 1048682 306 0 1048683 162 0 1048684 327 0 1048685 361 0 1048686 141 0 1048687 60 0 1048688 351 0 1048689 360 0 1048690 212 0 1048691 322 0 1048692 274 0 1048693 237 0 1048694 5 0 1048695 33 0 1048696 12 0 1048697 115 0 1048698 227 0 1048699 31 0 1048700 351 0 1048701 72 0 1048702 342 0 1048703 76 0 1048704 252 0 1048705 224 0 1048706 303 0 1048707 27 0 1048708 201 0 1048709 15 0 1048710 144 0 1048711 261 0 1048712 62 0 1048713 342 0 1048714 361 0 1048715 112 0 1048716 133 0 1048717 114 0 1048718 261 0 1048719 133 0 1048720 142 0 1048721 355 0 1048722 162 0 1048723 277 0 1048724 363 0 1048725 202 0 1048726 354 0 1048727 345 0 1048728 175 0 1048729 271 0 1048730 115 0 1048731 62 0 1048732 341 0 1048733 275 0 1048734 275 0 1048735 171 0 1048736 17 0 1048737 271 0 1048738 213 0 1048739 160 0 1048740 111 0 1048741 132 0 1048742 224 0 1048743 50 0 1048744 50 0 1048745 317 0 1048746 241 0 1048747 101 0 1048748 21 0 1048749 75 0 1048750 126 0 1048751 305 0 1048752 47 0 1048753 337 0 1048754 35 0 1048755 342 0 1048756 44 0 1048757 357 0 1048758 263 0 1048759 55 0 1048760 221 0 1048761 152 0 1048762 252 0 1048763 115 0 1048764 70 0 1048765 345 0 1048766 215 0 1048767 56 0 1048768 312 0 1048769 155 0 1048770 371 0 1048771 142 0 1048772 254 0 1048773 156 0 1048774 237 0 1048775 206 0 1048776 233 0 1048777 202 0 1048778 166 0 1048779 77 0 1048780 53 0 1048781 50 0 1048782 50 0 1048783 100 0 1048784 117 0 1048785 113 0 1048786 27 0 1048787 225 0 1048788 207 0 1048789 115 0 1048790 365 0 1048791 247 0 1048792 365 0 1048793 304 0 1048794 12 0 1048795 131 0 1048796 267 0 1048797 112 0 1048798 227 0 1048799 251 0 1048800 5 0 1048801 7 0 1048802 201 0 1048803 212 0 1048804 3 0 1048805 243 0 1048806 12 0 1048807 144 0 1048808 143 0 1048809 223 0 1048810 232 0 1048811 201 0 1048812 27 0 1048813 237 0 1048814 3 0 1048815 110 0 1048816 334 0 1048817 130 0 1048818 304 0 1048819 316 0 1048820 205 0 1048821 63 0 1048822 372 0 1048823 333 0 1048824 66 0 1048825 32 0 1048826 45 0 1048827 153 0 1048828 266 0 1048829 307 0 1048830 15 0 1048831 123 0 1048832 374 0 1048833 277 0 1048834 270 0 1048835 106 0 1048836 365 0 1048837 61 0 1048838 364 0 1048839 6 0 1048840 340 0 1048841 115 0 1048842 221 0 1048843 10 0 1048844 234 0 1048845 137 0 1048846 113 0 1048847 110 0 1048848 371 0 1048849 327 0 1048850 306 0 1048851 312 0 1048852 41 0 1048853 66 0 1048854 230 0 1048855 332 0 1048856 77 0 1048857 134 0 1048858 27 0 1048859 14 0 1048860 333 0 1048861 201 0 1048862 71 0 1048863 316 0 1048864 352 0 1048865 314 0 1048866 73 0 1048867 43 0 1048868 226 0 1048869 72 0 1048870 324 0 1048871 125 0 1048872 243 0 1048873 121 0 1048874 130 0 1048875 11 0 1048876 221 0 1048877 146 0 1048878 254 0 1048879 73 0 1048880 17 0 1048881 237 0 1048882 361 0 1048883 35 0 1048884 32 0 1048885 151 0 1048886 62 0 1048887 231 0 1048888 267 0 1048889 133 0 1048890 312 0 1048891 153 0 1048892 267 0 1048893 333 0 1048894 334 0 1048895 52 0 1048896 105 0 1048897 211 0 1048898 110 0 1048899 72 0 1048900 24 0 1048901 254 0 1048902 27 0 1048903 267 0 1048904 155 0 1048905 146 0 1048906 310 0 1048907 157 0 1048908 332 0 1048909 2 0 1048910 75 0 1048911 307 0 1048912 372 0 1048913 246 0 1048914 377 0 1048915 205 0 1048916 267 0 1048917 146 0 1048918 104 0 1048919 220 0 1048920 220 0 1048921 241 0 1048922 22 0 1048923 14 0 1048924 237 0 1048925 355 0 1048926 213 0 1048927 276 0 1048928 215 0 1048929 306 0 1048930 226 0 1048931 112 0 1048932 151 0 1048933 10 0 1048934 121 0 1048935 233 0 1048936 310 0 1048937 53 0 1048938 22 0 1048939 130 0 1048940 34 0 1048941 22 0 1048942 143 0 1048943 226 0 1048944 173 0 1048945 377 0 1048946 130 0 1048947 34 0 1048948 74 0 1048949 275 0 1048950 73 0 1048951 161 0 1048952 71 0 1048953 207 0 1048954 221 0 1048955 104 0 1048956 113 0 1048957 3 0 1048958 174 0 1048959 57 0 1048960 34 0 1048961 124 0 1048962 64 0 1048963 341 0 1048964 170 0 1048965 12 0 1048966 232 0 1048967 321 0 1048968 105 0 1048969 43 0 1048970 237 0 1048971 242 0 1048972 364 0 1048973 327 0 1048974 56 0 1048975 150 0 1048976 53 0 1048977 45 0 1048978 141 0 1048979 251 0 1048980 200 0 1048981 370 0 1048982 302 0 1048983 124 0 1048984 237 0 1048985 174 0 1048986 312 0 1048987 223 0 1048988 357 0 1048989 233 0 1048990 54 0 1048991 262 0 1048992 112 0 1048993 311 0 1048994 154 0 1048995 333 0 1048996 161 0 1048997 52 0 1048998 364 0 1048999 102 0 1049000 350 0 1049001 34 0 1049002 373 0 1049003 226 0 1049004 171 0 1049005 163 0 1049006 143 0 1049007 213 0 1049008 152 0 1049009 124 0 1049010 53 0 1049011 114 0 1049012 266 0 1049013 207 0 1049014 5 0 1049015 125 0 1049016 136 0 1049017 1 0 1049018 132 0 1049019 2 0 1049020 167 0 1049021 235 0 1049022 73 0 1049023 12 0 1049024 167 0 1049025 262 0 1049026 23 0 1049027 202 0 1049028 13 0 1049029 347 0 1049030 127 0 1049031 325 0 1049032 126 0 1049033 130 0 1049034 121 0 1049035 266 0 1049036 357 0 1049037 261 0 1049038 347 0 1049039 23 0 1049040 321 0 1049041 53 0 1049042 23 0 1049043 275 0 1049044 307 0 1049045 54 0 1049046 142 0 1049047 243 0 1049048 274 0 1049049 377 0 1049050 177 0 1049051 206 0 1049052 56 0 1049053 57 0 1049054 35 0 1049055 165 0 1049056 57 0 1049057 223 0 1049058 344 0 1049059 204 0 1049060 344 0 1049061 264 0 1049062 312 0 1049063 361 0 1049064 232 0 1049065 250 0 1049066 40 0 1049067 265 0 1049068 346 0 1049069 2 0 1049070 265 0 1049071 51 0 1049072 304 0 1049073 306 0 1049074 371 0 1049075 22 0 1049076 1 0 1049077 57 0 1049078 154 0 1049079 61 0 1049080 16 0 1049081 36 0 1049082 307 0 1049083 104 0 1049084 245 0 1049085 351 0 1049086 311 0 1049087 75 0 1049088 67 0 1049089 20 0 1049090 74 0 1049091 375 0 1049092 221 0 1049093 277 0 1049094 15 0 1049095 370 0 1049096 266 0 1049097 14 0 1049098 361 0 1049099 120 0 1049100 16 0 1049101 44 0 1049102 14 0 1049103 234 0 1049104 210 0 1049105 153 0 1049106 331 0 1049107 72 0 1049108 354 0 1049109 345 0 1049110 155 0 1049111 201 0 1049112 354 0 1049113 135 0 1049114 71 0 1049115 177 0 1049116 343 0 1049117 37 0 1049118 37 0 1049119 263 0 1049120 114 0 1049121 127 0 1049122 104 0 1049123 215 0 1049124 40 0 1049125 242 0 1049126 230 0 1049127 70 0 1049128 232 0 1049129 20 0 1049130 116 0 1049131 242 0 1049132 351 0 1049133 274 0 1049134 143 0 1049135 213 0 1049136 1 0 1049137 105 0 1049138 242 0 1049139 321 0 1049140 333 0 1049141 43 0 1049142 176 0 1049143 7 0 1049144 76 0 1049145 367 0 1049146 144 0 1049147 345 0 1049148 343 0 1049149 222 0 1049150 246 0 1049151 133 0 1049152 346 0 1049153 172 0 1049154 74 0 1049155 214 0 1049156 305 0 1049157 347 0 1049158 334 0 1049159 12 0 1049160 373 0 1049161 100 0 1049162 362 0 1049163 266 0 1049164 34 0 1049165 326 0 1049166 51 0 1049167 27 0 1049168 245 0 1049169 277 0 1049170 256 0 1049171 277 0 1049172 56 0 1049173 102 0 1049174 10 0 1049175 201 0 1049176 120 0 1049177 72 0 1049178 316 0 1049179 313 0 1049180 317 0 1049181 55 0 1049182 27 0 1049183 344 0 1049185 256 0 1049187 265 0 1049188 20 0 1049189 256 0 1049190 226 0 1049191 300 0 1049192 235 0 1049193 325 0 1049194 40 0 1049195 10 0 1049196 67 0 1049197 317 0 1049198 161 0 1049199 72 0 1049200 146 0 1049201 53 0 1049202 150 0 1049203 337 0 1049204 60 0 1049205 153 0 1049206 72 0 1049207 121 0 1049208 33 0 1049209 63 0 1049210 47 0 1049211 160 0 1049212 125 0 1049213 260 0 1049214 264 0 1049215 24 0 1049216 230 0 1049217 75 0 1049218 130 0 1049219 141 0 1049220 53 0 1049221 116 0 1049222 71 0 1049223 36 0 1049224 65 0 1049225 12 0 1049226 2 0 1049227 155 0 1049228 231 0 1049229 175 0 1049230 23 0 1049231 107 0 1049232 153 0 1049233 156 0 1049234 10 0 1049235 145 0 1049236 162 0 1049237 145 0 1049238 144 0 1049239 356 0 1049240 160 0 1049241 136 0 1049242 147 0 1049243 317 0 1049244 304 0 1049245 136 0 1049246 22 0 1049247 145 0 1049248 320 0 1049249 254 0 1049250 64 0 1049251 107 0 1049252 5 0 1049253 373 0 1049254 1 0 1049255 127 0 1049256 314 0 1049257 73 0 1049258 175 0 1049259 115 0 1049260 273 0 1049261 217 0 1049262 255 0 1049263 7 0 1049264 113 0 1049265 244 0 1049266 47 0 1049267 76 0 1049268 162 0 1049269 46 0 1049270 21 0 1049271 207 0 1049272 10 0 1049273 377 0 1049274 303 0 1049275 272 0 1049276 165 0 1049277 62 0 1049278 4 0 1049279 171 0 1049280 247 0 1049281 6 0 1049282 224 0 1049283 334 0 1049284 313 0 1049285 320 0 1049286 362 0 1049287 43 0 1049288 40 0 1049289 260 0 1049290 342 0 1049291 117 0 1049292 73 0 1049293 174 0 1049294 217 0 1049295 357 0 1049296 116 0 1049297 37 0 1049298 123 0 1049299 220 0 1049300 331 0 1049301 54 0 1049302 157 0 1049303 111 0 1049304 144 0 1049305 105 0 1049306 110 0 1049307 15 0 1049308 1 0 1049309 140 0 1049310 160 0 1049311 14 0 1049312 310 0 1049313 242 0 1049314 121 0 1049315 251 0 1049316 55 0 1049317 265 0 1049318 351 0 1049319 144 0 1049320 207 0 1049321 210 0 1049322 354 0 1049323 103 0 1049324 4 0 1049325 67 0 1049326 154 0 1049327 247 0 1049328 176 0 1049329 176 0 1049330 24 0 1049331 135 0 1049332 133 0 1049333 321 0 1049334 242 0 1049335 100 0 1049336 242 0 1049337 315 0 1049338 341 0 1049339 266 0 1049340 17 0 1049341 101 0 1049342 131 0 1049343 263 0 1049344 61 0 1049345 237 0 1049346 246 0 1049347 44 0 1049348 365 0 1049349 44 0 1049350 300 0 1049351 41 0 1049352 76 0 1049353 232 0 1049354 334 0 1049355 322 0 1049356 334 0 1049358 335 0 1049359 223 0 1049360 363 0 1049361 271 0 1049362 245 0 1049363 304 0 1049364 260 0 1049365 233 0 1049366 201 0 1049367 310 0 1049368 64 0 1049369 331 0 1049370 115 0 1049371 265 0 1049372 152 0 1049373 42 0 1049374 102 0 1049375 115 0 1049376 11 0 1049377 310 0 1049378 167 0 1049379 372 0 1049380 301 0 1049381 312 0 1049382 217 0 1049383 102 0 1049384 241 0 1049385 57 0 1049386 163 0 1049387 25 0 1049388 64 0 1049389 114 0 1049390 150 0 1049391 335 0 1049392 221 0 1049393 235 0 1049394 1 0 1049395 254 0 1049396 363 0 1049397 121 0 1049398 367 0 1049399 316 0 1049400 142 0 1049401 370 0 1049402 311 0 1049403 346 0 1049404 161 0 1049405 323 0 1049406 241 0 1049407 325 0 1049408 345 0 1049409 344 0 1049410 137 0 1049411 130 0 1049412 24 0 1049413 177 0 1049414 342 0 1049415 10 0 1049416 136 0 1049417 343 0 1049418 61 0 1049419 35 0 1049420 62 0 1049421 232 0 1049422 247 0 1049423 253 0 1049424 232 0 1049425 263 0 1049426 140 0 1049427 115 0 1049428 351 0 1049429 324 0 1049430 273 0 1049431 240 0 1049432 207 0 1049433 304 0 1049434 2 0 1049435 120 0 1049436 204 0 1049437 205 0 1049438 207 0 1049439 264 0 1049440 300 0 1049441 340 0 1049442 317 0 1049443 202 0 1049444 275 0 1049445 30 0 1049446 242 0 1049447 323 0 1049448 174 0 1049449 175 0 1049450 331 0 1049451 216 0 1049452 7 0 1049453 110 0 1049454 51 0 1049455 105 0 1049456 45 0 1049457 166 0 1049458 37 0 1049459 1 0 1049460 202 0 1049461 225 0 1049462 17 0 1049463 337 0 1049464 30 0 1049465 227 0 1049466 5 0 1049467 111 0 1049468 130 0 1049469 61 0 1049470 100 0 1049471 260 0 1049472 104 0 1049473 202 0 1049474 364 0 1049475 74 0 1049476 247 0 1049477 162 0 1049478 166 0 1049479 42 0 1049480 27 0 1049481 276 0 1049482 370 0 1049483 331 0 1049484 110 0 1049485 315 0 1049486 174 0 1049487 234 0 1049488 361 0 1049489 261 0 1049490 234 0 1049491 100 0 1049492 330 0 1049493 121 0 1049494 214 0 1049495 333 0 1049496 77 0 1049497 270 0 1049498 70 0 1049499 225 0 1049500 173 0 1049501 66 0 1049502 376 0 1049503 366 0 1049504 352 0 1049505 302 0 1049506 156 0 1049507 253 0 1049508 167 0 1049509 275 0 1049510 143 0 1049511 342 0 1049512 226 0 1049513 63 0 1049514 346 0 1049515 153 0 1049516 67 0 1049517 223 0 1049518 321 0 1049519 332 0 1049520 213 0 1049521 104 0 1049522 373 0 1049523 346 0 1049524 42 0 1049525 244 0 1049526 42 0 1049527 27 0 1049528 256 0 1049529 65 0 1049530 326 0 1049531 136 0 1049532 262 0 1049533 225 0 1049534 106 0 1049535 156 0 1049536 127 0 1049537 157 0 1049538 3 0 1049539 252 0 1049540 314 0 1049541 330 0 1049542 152 0 1049543 20 0 1049544 335 0 1049545 164 0 1049546 76 0 1049547 227 0 1049548 132 0 1049549 244 0 1049550 336 0 1049551 204 0 1049552 73 0 1049553 172 0 1049554 63 0 1049555 350 0 1049556 331 0 1049557 152 0 1049558 7 0 1049559 213 0 1049560 316 0 1049561 122 0 1049562 160 0 1049563 326 0 1049564 127 0 1049565 142 0 1049566 136 0 1049567 271 0 1049568 247 0 1049569 257 0 1049570 256 0 1049571 40 0 1049572 41 0 1049573 123 0 1049574 313 0 1049575 153 0 1049576 252 0 1049577 177 0 1049578 362 0 1049579 160 0 1049580 262 0 1049581 266 0 1049582 56 0 1049583 263 0 1049584 304 0 1049585 22 0 1049586 151 0 1049587 23 0 1049588 210 0 1049589 125 0 1049590 156 0 1049591 301 0 1049592 373 0 1049593 111 0 1049594 26 0 1049595 136 0 1049596 203 0 1049597 363 0 1049598 152 0 1049599 306 0 1049600 57 0 1049601 167 0 1049602 44 0 1049603 115 0 1049604 203 0 1049605 354 0 1049606 162 0 1049607 177 0 1049608 314 0 1049609 212 0 1049610 101 0 1049611 56 0 1049612 221 0 1049613 336 0 1049614 302 0 1049615 117 0 1049616 61 0 1049617 53 0 1049618 277 0 1049619 207 0 1049620 370 0 1049621 23 0 1049622 60 0 1049623 242 0 1049624 51 0 1049625 155 0 1049626 200 0 1049627 141 0 1049628 244 0 1049629 67 0 1049630 75 0 1049631 105 0 1049632 332 0 1049633 224 0 1049634 324 0 1049635 253 0 1049636 362 0 1049637 360 0 1049638 114 0 1049639 336 0 1049640 362 0 1049641 237 0 1049642 374 0 1049643 15 0 1049644 35 0 1049645 313 0 1049646 5 0 1049647 254 0 1049648 363 0 1049649 11 0 1049650 47 0 1049651 222 0 1049652 31 0 1049653 64 0 1049654 234 0 1049655 137 0 1049656 321 0 1049657 220 0 1049658 54 0 1049659 177 0 1049660 264 0 1049661 313 0 1049662 244 0 1049663 245 0 1049664 61 0 1049665 374 0 1049666 226 0 1049667 14 0 1049668 160 0 1049669 76 0 1049670 253 0 1049671 373 0 1049672 32 0 1049673 161 0 1049674 46 0 1049675 140 0 1049676 124 0 1049677 254 0 1049678 70 0 1049679 210 0 1049680 240 0 1049681 361 0 1049682 66 0 1049683 144 0 1049684 100 0 1049685 321 0 1049686 253 0 1049687 5 0 1049688 5 0 1049689 367 0 1049690 155 0 1049691 120 0 1049692 50 0 1049693 340 0 1049694 143 0 1049695 150 0 1049696 366 0 1049697 73 0 1049698 201 0 1049699 372 0 1049700 45 0 1049701 167 0 1049702 172 0 1049703 267 0 1049704 114 0 1049705 143 0 1049706 137 0 1049707 52 0 1049708 156 0 1049709 263 0 1049710 215 0 1049711 374 0 1049712 256 0 1049713 15 0 1049714 276 0 1049715 107 0 1049716 150 0 1049717 40 0 1049718 226 0 1049719 170 0 1049720 132 0 1049721 335 0 1049722 267 0 1049723 27 0 1049724 223 0 1049725 270 0 1049726 210 0 1049727 77 0 1049728 40 0 1049729 235 0 1049730 322 0 1049731 61 0 1049732 13 0 1049733 174 0 1049734 77 0 1049735 177 0 1049736 344 0 1049737 45 0 1049738 311 0 1049739 53 0 1049740 351 0 1049741 371 0 1049742 71 0 1049743 34 0 1049744 1 0 1049745 321 0 1049746 367 0 1049747 146 0 1049748 153 0 1049749 360 0 1049750 370 0 1049751 350 0 1049752 376 0 1049753 247 0 1049754 104 0 1049755 34 0 1049756 214 0 1049757 222 0 1049758 261 0 1049759 307 0 1049760 20 0 1049761 245 0 1049762 353 0 1049763 364 0 1049764 122 0 1049765 245 0 1049766 23 0 1049767 33 0 1049768 156 0 1049769 215 0 1049770 31 0 1049771 164 0 1049772 313 0 1049773 76 0 1049774 344 0 1049775 373 0 1049776 331 0 1049777 332 0 1049778 142 0 1049779 340 0 1049780 277 0 1049781 345 0 1049782 112 0 1049783 337 0 1049784 255 0 1049785 31 0 1049786 351 0 1049787 222 0 1049788 230 0 1049789 202 0 1049790 101 0 1049791 347 0 1049792 235 0 1049793 234 0 1049794 51 0 1049795 147 0 1049796 136 0 1049797 113 0 1049798 67 0 1049799 104 0 1049800 125 0 1049801 232 0 1049802 161 0 1049803 51 0 1049804 163 0 1049805 320 0 1049806 356 0 1049807 325 0 1049808 107 0 1049809 356 0 1049810 266 0 1049811 126 0 1049812 252 0 1049813 143 0 1049814 235 0 1049815 143 0 1049816 62 0 1049817 353 0 1049818 46 0 1049819 25 0 1049820 16 0 1049821 207 0 1049822 300 0 1049823 162 0 1049824 372 0 1049825 275 0 1049826 252 0 1049827 321 0 1049828 373 0 1049829 255 0 1049830 353 0 1049831 60 0 1049832 204 0 1049833 321 0 1049834 174 0 1049835 11 0 1049836 356 0 1049837 141 0 1049838 220 0 1049839 315 0 1049840 36 0 1049841 354 0 1049842 136 0 1049843 145 0 1049844 31 0 1049845 243 0 1049846 326 0 1049847 333 0 1049848 112 0 1049849 303 0 1049850 7 0 1049851 62 0 1049852 357 0 1049853 376 0 1049854 217 0 1049855 173 0 1049856 373 0 1049857 304 0 1049858 351 0 1049859 62 0 1049860 353 0 1049861 344 0 1049862 221 0 1049863 105 0 1049864 115 0 1049865 325 0 1049866 312 0 1049867 60 0 1049868 146 0 1049869 24 0 1049870 115 0 1049871 50 0 1049872 375 0 1049873 322 0 1049874 33 0 1049875 11 0 1049877 133 0 1049878 266 0 1049879 165 0 1049880 60 0 1049881 312 0 1049882 311 0 1049883 101 0 1049884 226 0 1049885 331 0 1049886 130 0 1049887 372 0 1049888 376 0 1049889 36 0 1049890 14 0 1049891 136 0 1049892 137 0 1049893 346 0 1049894 342 0 1049895 47 0 1049896 77 0 1049897 33 0 1049898 71 0 1049899 203 0 1049900 302 0 1049901 302 0 1049902 244 0 1049903 15 0 1049904 75 0 1049905 373 0 1049906 166 0 1049907 330 0 1049908 125 0 1049909 363 0 1049910 161 0 1049911 67 0 1049912 72 0 1049913 203 0 1049914 43 0 1049915 37 0 1049916 203 0 1049917 237 0 1049918 276 0 1049919 104 0 1049920 204 0 1049921 47 0 1049922 172 0 1049923 135 0 1049924 345 0 1049925 373 0 1049926 232 0 1049927 167 0 1049928 371 0 1049929 355 0 1049930 144 0 1049931 141 0 1049932 261 0 1049933 12 0 1049934 34 0 1049935 46 0 1049936 246 0 1049937 331 0 1049938 245 0 1049939 115 0 1049940 217 0 1049941 236 0 1049942 356 0 1049943 211 0 1049944 35 0 1049945 365 0 1049946 134 0 1049947 227 0 1049948 21 0 1049949 320 0 1049950 26 0 1049951 61 0 1049952 347 0 1049953 17 0 1049954 316 0 1049955 216 0 1049956 42 0 1049957 77 0 1049958 5 0 1049959 352 0 1049960 41 0 1049961 124 0 1049962 100 0 1049963 145 0 1049964 247 0 1049965 111 0 1049966 70 0 1049967 130 0 1049968 23 0 1049969 245 0 1049970 111 0 1049971 133 0 1049972 124 0 1049973 311 0 1049974 361 0 1049975 356 0 1049976 152 0 1049977 362 0 1049978 176 0 1049979 250 0 1049980 224 0 1049981 14 0 1049982 73 0 1049983 202 0 1049985 265 0 1049986 230 0 1049987 177 0 1049988 16 0 1049989 10 0 1049990 33 0 1049991 33 0 1049992 140 0 1049993 150 0 1049994 365 0 1049995 11 0 1049996 255 0 1049997 103 0 1049998 114 0 1049999 156 0 1050000 151 0 1050001 227 0 1050002 223 0 1050003 351 0 1050004 322 0 1050005 151 0 1050006 165 0 1050007 254 0 1050008 222 0 1050009 324 0 1050010 163 0 1050011 141 0 1050012 120 0 1050013 140 0 1050014 361 0 1050015 122 0 1050016 10 0 1050017 13 0 1050018 73 0 1050019 50 0 1050020 150 0 1050021 206 0 1050022 177 0 1050023 205 0 1050024 233 0 1050025 163 0 1050026 232 0 1050027 302 0 1050028 230 0 1050029 101 0 1050030 346 0 1050031 226 0 1050032 331 0 1050033 44 0 1050034 257 0 1050035 235 0 1050036 176 0 1050037 100 0 1050038 242 0 1050039 227 0 1050040 12 0 1050041 154 0 1050042 35 0 1050043 42 0 1050044 172 0 1050045 143 0 1050046 172 0 1050047 106 0 1050048 51 0 1050049 335 0 1050050 243 0 1050051 375 0 1050052 140 0 1050053 261 0 1050054 277 0 1050055 375 0 1050056 327 0 1050057 253 0 1050058 106 0 1050059 321 0 1050060 236 0 1050061 153 0 1050062 143 0 1050063 120 0 1050064 137 0 1050065 5 0 1050066 205 0 1050067 307 0 1050068 230 0 1050069 103 0 1050070 331 0 1050071 235 0 1050072 332 0 1050073 56 0 1050074 42 0 1050075 270 0 1050076 154 0 1050077 256 0 1050078 231 0 1050079 67 0 1050080 322 0 1050081 16 0 1050082 323 0 1050083 102 0 1050084 307 0 1050085 277 0 1050086 164 0 1050087 242 0 1050088 233 0 1050089 251 0 1050090 240 0 1050091 337 0 1050092 200 0 1050093 126 0 1050094 306 0 1050095 252 0 1050096 262 0 1050097 36 0 1050098 363 0 1050099 366 0 1050100 333 0 1050101 61 0 1050102 216 0 1050103 334 0 1050104 330 0 1050105 261 0 1050106 42 0 1050107 176 0 1050108 36 0 1050109 105 0 1050110 336 0 1050111 317 0 1050112 23 0 1050113 44 0 1050114 142 0 1050115 205 0 1050116 277 0 1050117 57 0 1050118 261 0 1050119 3 0 1050120 302 0 1050121 140 0 1050122 16 0 1050123 11 0 1050124 76 0 1050125 342 0 1050126 16 0 1050127 271 0 1050128 64 0 1050129 251 0 1050130 113 0 1050131 235 0 1050132 125 0 1050133 130 0 1050134 22 0 1050135 275 0 1050136 107 0 1050137 70 0 1050138 352 0 1050139 114 0 1050140 155 0 1050141 154 0 1050142 314 0 1050143 11 0 1050144 74 0 1050145 140 0 1050146 131 0 1050147 344 0 1050148 354 0 1050149 347 0 1050150 216 0 1050151 15 0 1050152 252 0 1050153 62 0 1050154 313 0 1050155 205 0 1050156 71 0 1050157 113 0 1050158 241 0 1050159 355 0 1050160 300 0 1050161 172 0 1050162 204 0 1050163 341 0 1050164 52 0 1050165 376 0 1050166 16 0 1050167 62 0 1050168 152 0 1050169 74 0 1050170 264 0 1050171 345 0 1050172 260 0 1050173 355 0 1050174 174 0 1050175 163 0 1050176 233 0 1050177 300 0 1050178 264 0 1050179 126 0 1050180 340 0 1050181 123 0 1050182 71 0 1050183 75 0 1050184 215 0 1050185 34 0 1050186 322 0 1050187 67 0 1050188 24 0 1050189 302 0 1050190 25 0 1050191 323 0 1050192 256 0 1050193 170 0 1050194 175 0 1050195 216 0 1050196 124 0 1050197 41 0 1050198 271 0 1050199 63 0 1050200 255 0 1050201 163 0 1050202 335 0 1050203 66 0 1050204 63 0 1050205 212 0 1050206 212 0 1050207 370 0 1050208 365 0 1050209 116 0 1050210 324 0 1050211 214 0 1050212 234 0 1050213 377 0 1050214 16 0 1050215 357 0 1050216 212 0 1050217 5 0 1050218 272 0 1050219 267 0 1050220 71 0 1050221 171 0 1050222 102 0 1050223 232 0 1050224 351 0 1050225 346 0 1050226 234 0 1050227 257 0 1050228 66 0 1050229 223 0 1050230 344 0 1050231 70 0 1050232 254 0 1050233 174 0 1050234 130 0 1050235 274 0 1050236 75 0 1050237 174 0 1050238 100 0 1050239 77 0 1050240 377 0 1050241 223 0 1050242 203 0 1050243 60 0 1050244 147 0 1050245 7 0 1050246 333 0 1050247 17 0 1050248 342 0 1050249 15 0 1050250 307 0 1050251 224 0 1050252 351 0 1050253 235 0 1050254 205 0 1050255 20 0 1050256 37 0 1050257 134 0 1050258 212 0 1050259 135 0 1050260 221 0 1050261 33 0 1050262 341 0 1050263 36 0 1050264 311 0 1050265 332 0 1050266 61 0 1050267 301 0 1050268 1 0 1050269 22 0 1050270 57 0 1050271 25 0 1050272 136 0 1050273 255 0 1050274 366 0 1050275 71 0 1050276 236 0 1050277 135 0 1050278 215 0 1050279 157 0 1050280 360 0 1050281 136 0 1050282 20 0 1050283 224 0 1050284 21 0 1050285 173 0 1050286 137 0 1050287 266 0 1050288 340 0 1050289 305 0 1050290 162 0 1050291 170 0 1050292 117 0 1050293 164 0 1050294 77 0 1050295 350 0 1050296 355 0 1050297 375 0 1050298 17 0 1050299 276 0 1050300 327 0 1050301 277 0 1050302 254 0 1050303 364 0 1050304 76 0 1050305 105 0 1050306 362 0 1050307 232 0 1050308 46 0 1050309 337 0 1050310 212 0 1050311 211 0 1050312 206 0 1050313 362 0 1050314 44 0 1050315 6 0 1050316 50 0 1050317 241 0 1050318 306 0 1050319 41 0 1050320 272 0 1050321 26 0 1050322 43 0 1050323 324 0 1050324 31 0 1050325 335 0 1050326 222 0 1050327 275 0 1050328 106 0 1050329 137 0 1050330 130 0 1050331 231 0 1050332 110 0 1050333 104 0 1050334 22 0 1050335 207 0 1050336 45 0 1050337 113 0 1050338 330 0 1050339 32 0 1050340 313 0 1050341 224 0 1050342 147 0 1050343 141 0 1050344 201 0 1050345 334 0 1050346 172 0 1050347 322 0 1050348 323 0 1050349 100 0 1050350 210 0 1050351 301 0 1050352 202 0 1050353 250 0 1050354 102 0 1050355 62 0 1050356 117 0 1050357 136 0 1050358 223 0 1050359 12 0 1050360 212 0 1050361 173 0 1050362 232 0 1050363 305 0 1050364 37 0 1050365 225 0 1050366 52 0 1050367 4 0 1050368 343 0 1050369 62 0 1050370 42 0 1050371 312 0 1050372 206 0 1050373 215 0 1050374 113 0 1050375 1 0 1050376 331 0 1050377 75 0 1050378 362 0 1050379 244 0 1050380 332 0 1050381 56 0 1050382 344 0 1050383 264 0 1050384 61 0 1050385 353 0 1050386 301 0 1050387 107 0 1050388 123 0 1050389 76 0 1050390 50 0 1050391 116 0 1050392 365 0 1050393 21 0 1050394 201 0 1050395 50 0 1050396 320 0 1050397 70 0 1050398 171 0 1050399 200 0 1050400 76 0 1050401 56 0 1050402 313 0 1050403 37 0 1050404 364 0 1050405 176 0 1050406 1 0 1050407 203 0 1050408 217 0 1050409 127 0 1050410 15 0 1050411 155 0 1050412 337 0 1050413 315 0 1050414 306 0 1050415 74 0 1050416 336 0 1050417 2 0 1050418 63 0 1050419 314 0 1050420 260 0 1050421 143 0 1050422 103 0 1050423 362 0 1050424 335 0 1050425 262 0 1050426 353 0 1050427 43 0 1050428 354 0 1050429 50 0 1050430 127 0 1050431 244 0 1050432 210 0 1050433 227 0 1050434 147 0 1050435 32 0 1050436 170 0 1050437 273 0 1050438 247 0 1050439 176 0 1050440 123 0 1050441 202 0 1050442 5 0 1050443 325 0 1050444 264 0 1050445 117 0 1050446 237 0 1050447 335 0 1050448 16 0 1050449 250 0 1050450 254 0 1050451 307 0 1050452 370 0 1050453 141 0 1050454 140 0 1050455 237 0 1050456 120 0 1050457 275 0 1050458 221 0 1050459 167 0 1050460 256 0 1050461 335 0 1050462 314 0 1050463 150 0 1050464 243 0 1050465 44 0 1050466 17 0 1050467 11 0 1050468 75 0 1050469 157 0 1050470 62 0 1050471 123 0 1050472 312 0 1050473 145 0 1050474 31 0 1050475 61 0 1050476 277 0 1050477 60 0 1050478 153 0 1050479 112 0 1050480 272 0 1050481 254 0 1050482 241 0 1050483 34 0 1050484 350 0 1050485 327 0 1050486 211 0 1050487 251 0 1050488 231 0 1050489 373 0 1050490 376 0 1050491 144 0 1050492 21 0 1050493 333 0 1050494 103 0 1050495 326 0 1050496 344 0 1050497 114 0 1050498 44 0 1050499 102 0 1050500 222 0 1050501 115 0 1050502 117 0 1050503 150 0 1050504 312 0 1050505 340 0 1050506 70 0 1050507 141 0 1050508 233 0 1050509 366 0 1050510 200 0 1050511 252 0 1050512 257 0 1050513 172 0 1050514 42 0 1050515 33 0 1050516 254 0 1050517 230 0 1050518 67 0 1050519 74 0 1050520 150 0 1050521 217 0 1050523 230 0 1050524 141 0 1050525 164 0 1050526 310 0 1050527 276 0 1050528 200 0 1050529 304 0 1050530 313 0 1050531 142 0 1050532 235 0 1050533 211 0 1050534 116 0 1050535 376 0 1050536 116 0 1050537 357 0 1050538 43 0 1050539 24 0 1050540 116 0 1050541 212 0 1050542 163 0 1050543 223 0 1050544 220 0 1050545 322 0 1050546 164 0 1050547 272 0 1050548 235 0 1050549 22 0 1050550 327 0 1050551 300 0 1050552 112 0 1050553 200 0 1050554 163 0 1050555 206 0 1050556 35 0 1050557 57 0 1050558 363 0 1050559 230 0 1050560 42 0 1050561 376 0 1050562 117 0 1050563 377 0 1050564 145 0 1050565 377 0 1050566 104 0 1050567 55 0 1050568 275 0 1050569 371 0 1050570 114 0 1050571 176 0 1050572 107 0 1050573 320 0 1050574 116 0 1050575 131 0 1050576 240 0 1050577 236 0 1050578 62 0 1050579 160 0 1050580 37 0 1050581 16 0 1050582 227 0 1050583 101 0 1050584 62 0 1050585 333 0 1050586 105 0 1050587 314 0 1050588 117 0 1050589 171 0 1050590 146 0 1050591 33 0 1050592 151 0 1050593 72 0 1050594 32 0 1050595 50 0 1050596 357 0 1050597 120 0 1050598 141 0 1050599 217 0 1050600 225 0 1050601 121 0 1050602 11 0 1050603 32 0 1050604 353 0 1050605 36 0 1050606 60 0 1050607 331 0 1050608 65 0 1050609 46 0 1050610 234 0 1050611 41 0 1050612 350 0 1050613 100 0 1050614 360 0 1050615 36 0 1050616 316 0 1050617 224 0 1050618 42 0 1050619 137 0 1050620 62 0 1050621 165 0 1050622 367 0 1050623 174 0 1050624 245 0 1050625 261 0 1050626 30 0 1050627 325 0 1050628 332 0 1050629 106 0 1050630 275 0 1050631 351 0 1050632 312 0 1050633 373 0 1050634 164 0 1050635 77 0 1050636 30 0 1050637 325 0 1050638 27 0 1050639 5 0 1050640 312 0 1050641 246 0 1050642 324 0 1050643 246 0 1050644 10 0 1050645 151 0 1050646 26 0 1050647 145 0 1050648 311 0 1050649 337 0 1050650 1 0 1050651 17 0 1050652 135 0 1050653 14 0 1050654 51 0 1050655 323 0 1050657 27 0 1050658 352 0 1050659 12 0 1050660 144 0 1050661 235 0 1050662 137 0 1050663 257 0 1050664 233 0 1050665 247 0 1050666 144 0 1050667 336 0 1050668 45 0 1050669 147 0 1050670 164 0 1050671 52 0 1050672 274 0 1050673 74 0 1050674 271 0 1050675 137 0 1050676 43 0 1050677 100 0 1050678 216 0 1050679 67 0 1050680 151 0 1050681 374 0 1050682 317 0 1050683 241 0 1050684 145 0 1050685 146 0 1050686 314 0 1050687 200 0 1050688 307 0 1050689 325 0 1050690 46 0 1050691 14 0 1050692 141 0 1050693 104 0 1050694 61 0 1050695 243 0 1050696 32 0 1050697 100 0 1050698 252 0 1050699 33 0 1050700 105 0 1050701 162 0 1050702 263 0 1050703 27 0 1050704 263 0 1050705 155 0 1050706 262 0 1050707 322 0 1050708 51 0 1050709 227 0 1050710 21 0 1050711 213 0 1050712 155 0 1050713 302 0 1050714 21 0 1050715 303 0 1050716 107 0 1050717 254 0 1050718 354 0 1050719 277 0 1050720 76 0 1050721 131 0 1050722 115 0 1050723 302 0 1050724 205 0 1050725 221 0 1050726 127 0 1050727 36 0 1050728 40 0 1050729 311 0 1050730 130 0 1050731 376 0 1050732 263 0 1050733 55 0 1050734 302 0 1050735 107 0 1050736 75 0 1050737 155 0 1050738 161 0 1050739 55 0 1050740 32 0 1050741 320 0 1050742 312 0 1050743 115 0 1050744 43 0 1050745 240 0 1050746 260 0 1050747 16 0 1050748 43 0 1050749 41 0 1050750 63 0 1050751 136 0 1050752 116 0 1050753 37 0 1050754 151 0 1050755 234 0 1050756 361 0 1050757 117 0 1050758 2 0 1050759 363 0 1050760 315 0 1050761 373 0 1050762 372 0 1050763 305 0 1050764 3 0 1050765 137 0 1050766 10 0 1050767 125 0 1050768 357 0 1050769 25 0 1050770 137 0 1050771 376 0 1050772 73 0 1050773 306 0 1050774 121 0 1050775 73 0 1050776 125 0 1050777 135 0 1050778 134 0 1050779 21 0 1050780 56 0 1050781 204 0 1050782 104 0 1050783 154 0 1050784 40 0 1050785 336 0 1050786 346 0 1050787 66 0 1050788 243 0 1050789 76 0 1050790 333 0 1050791 352 0 1050792 160 0 1050793 220 0 1050794 375 0 1050795 154 0 1050796 163 0 1050797 320 0 1050798 251 0 1050799 243 0 1050800 174 0 1050801 316 0 1050802 315 0 1050803 135 0 1050804 54 0 1050805 315 0 1050806 100 0 1050807 125 0 1050808 117 0 1050809 373 0 1050810 104 0 1050811 373 0 1050812 323 0 1050813 245 0 1050814 25 0 1050815 207 0 1050816 272 0 1050817 47 0 1050818 106 0 1050819 64 0 1050820 334 0 1050821 216 0 1050822 7 0 1050823 176 0 1050824 334 0 1050825 171 0 1050826 61 0 1050827 117 0 1050828 35 0 1050829 60 0 1050830 337 0 1050831 77 0 1050832 273 0 1050833 223 0 1050834 14 0 1050835 313 0 1050836 124 0 1050837 133 0 1050838 71 0 1050839 304 0 1050840 3 0 1050841 114 0 1050842 156 0 1050843 210 0 1050844 154 0 1050845 356 0 1050846 226 0 1050847 136 0 1050848 317 0 1050849 126 0 1050850 370 0 1050851 67 0 1050852 112 0 1050853 376 0 1050854 125 0 1050855 164 0 1050856 366 0 1050857 17 0 1050858 43 0 1050859 23 0 1050860 105 0 1050861 237 0 1050862 253 0 1050863 173 0 1050864 206 0 1050865 357 0 1050866 321 0 1050867 100 0 1050868 156 0 1050869 201 0 1050870 51 0 1050871 1 0 1050872 224 0 1050873 147 0 1050874 33 0 1050875 63 0 1050876 176 0 1050877 200 0 1050878 254 0 1050879 100 0 1050880 204 0 1050881 176 0 1050882 307 0 1050883 272 0 1050884 32 0 1050885 72 0 1050886 103 0 1050887 101 0 1050888 341 0 1050889 164 0 1050890 161 0 1050891 315 0 1050892 126 0 1050893 217 0 1050894 130 0 1050895 150 0 1050896 162 0 1050897 317 0 1050898 243 0 1050899 367 0 1050900 210 0 1050901 210 0 1050902 137 0 1050903 120 0 1050904 230 0 1050905 364 0 1050906 34 0 1050907 40 0 1050908 105 0 1050909 112 0 1050910 105 0 1050911 264 0 1050912 102 0 1050913 371 0 1050914 202 0 1050915 155 0 1050916 253 0 1050917 37 0 1050918 45 0 1050919 21 0 1050920 253 0 1050921 267 0 1050922 6 0 1050923 317 0 1050924 102 0 1050925 37 0 1050926 44 0 1050927 161 0 1050928 251 0 1050929 67 0 1050930 145 0 1050931 156 0 1050932 342 0 1050933 353 0 1050934 77 0 1050935 10 0 1050936 127 0 1050937 173 0 1050938 54 0 1050939 117 0 1050940 143 0 1050941 365 0 1050942 222 0 1050943 335 0 1050944 356 0 1050945 234 0 1050946 134 0 1050947 57 0 1050948 272 0 1050949 32 0 1050950 105 0 1050951 274 0 1050952 136 0 1050953 150 0 1050954 322 0 1050955 323 0 1050956 45 0 1050957 31 0 1050958 20 0 1050959 50 0 1050960 35 0 1050961 163 0 1050962 255 0 1050963 233 0 1050964 156 0 1050965 20 0 1050966 170 0 1050967 26 0 1050968 163 0 1050969 126 0 1050970 134 0 1050971 42 0 1050972 111 0 1050973 43 0 1050974 37 0 1050975 306 0 1050976 47 0 1050977 356 0 1050978 75 0 1050979 270 0 1050980 211 0 1050981 350 0 1050982 30 0 1050983 213 0 1050984 156 0 1050985 160 0 1050986 131 0 1050987 164 0 1050988 230 0 1050989 41 0 1050990 36 0 1050991 313 0 1050992 355 0 1050993 55 0 1050994 13 0 1050995 371 0 1050996 143 0 1050998 126 0 1050999 206 0 1051000 110 0 1051001 231 0 1051002 304 0 1051003 221 0 1051004 140 0 1051005 254 0 1051006 237 0 1051007 163 0 1051008 131 0 1051009 44 0 1051010 316 0 1051011 205 0 1051012 230 0 1051013 101 0 1051014 232 0 1051015 72 0 1051016 152 0 1051017 3 0 1051018 253 0 1051019 121 0 1051020 236 0 1051022 115 0 1051023 40 0 1051024 66 0 1051025 150 0 1051026 206 0 1051027 335 0 1051028 14 0 1051029 237 0 1051030 123 0 1051031 57 0 1051032 254 0 1051033 115 0 1051034 253 0 1051035 136 0 1051036 275 0 1051037 170 0 1051038 220 0 1051039 115 0 1051040 41 0 1051041 45 0 1051042 154 0 1051043 246 0 1051044 322 0 1051045 10 0 1051046 162 0 1051047 4 0 1051048 31 0 1051049 324 0 1051050 223 0 1051051 370 0 1051052 175 0 1051053 113 0 1051054 321 0 1051055 122 0 1051056 155 0 1051057 260 0 1051058 147 0 1051059 215 0 1051060 326 0 1051061 237 0 1051062 1 0 1051063 10 0 1051064 206 0 1051065 244 0 1051066 322 0 1051067 235 0 1051068 57 0 1051069 235 0 1051070 372 0 1051071 142 0 1051072 304 0 1051073 103 0 1051074 321 0 1051075 71 0 1051076 255 0 1051077 351 0 1051078 11 0 1051079 30 0 1051080 272 0 1051081 76 0 1051082 103 0 1051083 175 0 1051084 355 0 1051085 342 0 1051086 270 0 1051087 342 0 1051088 77 0 1051090 11 0 1051091 250 0 1051092 376 0 1051093 261 0 1051094 124 0 1051095 103 0 1051096 162 0 1051097 353 0 1051098 343 0 1051099 41 0 1051100 350 0 1051101 71 0 1051102 201 0 1051103 161 0 1051104 354 0 1051105 233 0 1051106 37 0 1051107 50 0 1051108 256 0 1051109 126 0 1051110 171 0 1051111 304 0 1051112 362 0 1051113 343 0 1051114 42 0 1051115 124 0 1051116 142 0 1051117 317 0 1051118 46 0 1051119 256 0 1051120 337 0 1051121 374 0 1051122 43 0 1051123 37 0 1051124 14 0 1051125 263 0 1051126 146 0 1051127 341 0 1051128 363 0 1051129 3 0 1051130 311 0 1051131 307 0 1051132 44 0 1051133 250 0 1051134 347 0 1051135 317 0 1051136 220 0 1051137 244 0 1051138 322 0 1051139 71 0 1051140 204 0 1051141 310 0 1051142 24 0 1051143 20 0 1051144 340 0 1051145 204 0 1051146 14 0 1051147 331 0 1051148 341 0 1051149 243 0 1051150 342 0 1051151 330 0 1051152 215 0 1051153 341 0 1051154 356 0 1051155 317 0 1051156 217 0 1051157 101 0 1051158 366 0 1051159 354 0 1051160 102 0 1051161 112 0 1051162 324 0 1051163 364 0 1051164 204 0 1051165 21 0 1051166 315 0 1051167 174 0 1051168 142 0 1051169 132 0 1051170 123 0 1051171 206 0 1051172 313 0 1051173 216 0 1051174 24 0 1051175 105 0 1051176 365 0 1051177 260 0 1051178 45 0 1051179 263 0 1051180 337 0 1051181 311 0 1051182 213 0 1051183 370 0 1051184 25 0 1051185 134 0 1051186 311 0 1051187 351 0 1051188 312 0 1051189 37 0 1051190 41 0 1051191 117 0 1051192 341 0 1051193 254 0 1051194 55 0 1051195 26 0 1051196 137 0 1051197 204 0 1051198 56 0 1051199 170 0 1051200 224 0 1051201 145 0 1051202 223 0 1051203 207 0 1051204 247 0 1051205 303 0 1051206 203 0 1051207 70 0 1051208 101 0 1051209 4 0 1051210 213 0 1051211 112 0 1051212 366 0 1051213 120 0 1051214 266 0 1051215 255 0 1051216 171 0 1051217 4 0 1051218 370 0 1051219 177 0 1051220 57 0 1051221 247 0 1051222 270 0 1051223 360 0 1051224 30 0 1051225 152 0 1051226 257 0 1051227 15 0 1051228 232 0 1051229 113 0 1051230 260 0 1051231 100 0 1051232 246 0 1051233 236 0 1051234 374 0 1051235 313 0 1051236 252 0 1051237 112 0 1051238 320 0 1051239 70 0 1051240 215 0 1051241 222 0 1051242 354 0 1051243 327 0 1051244 304 0 1051245 36 0 1051246 62 0 1051247 12 0 1051248 141 0 1051249 6 0 1051250 103 0 1051251 31 0 1051252 312 0 1051253 45 0 1051254 145 0 1051255 1 0 1051256 303 0 1051257 170 0 1051258 35 0 1051259 274 0 1051260 313 0 1051261 21 0 1051262 44 0 1051263 166 0 1051264 40 0 1051265 12 0 1051266 11 0 1051267 214 0 1051268 55 0 1051269 115 0 1051270 271 0 1051271 136 0 1051272 317 0 1051273 165 0 1051274 10 0 1051275 236 0 1051276 165 0 1051277 6 0 1051278 160 0 1051279 254 0 1051280 167 0 1051281 327 0 1051282 155 0 1051283 22 0 1051284 360 0 1051285 356 0 1051286 355 0 1051287 166 0 1051288 320 0 1051289 11 0 1051290 264 0 1051291 351 0 1051292 113 0 1051293 311 0 1051294 101 0 1051295 155 0 1051296 301 0 1051297 57 0 1051298 256 0 1051299 341 0 1051300 31 0 1051301 15 0 1051302 321 0 1051303 63 0 1051304 61 0 1051305 42 0 1051306 100 0 1051307 76 0 1051308 46 0 1051309 133 0 1051310 103 0 1051311 303 0 1051312 306 0 1051313 177 0 1051314 50 0 1051315 220 0 1051316 345 0 1051317 174 0 1051318 30 0 1051319 263 0 1051320 274 0 1051321 215 0 1051322 344 0 1051323 162 0 1051324 102 0 1051325 276 0 1051326 27 0 1051327 324 0 1051328 166 0 1051329 111 0 1051330 325 0 1051331 234 0 1051332 335 0 1051333 312 0 1051334 111 0 1051335 335 0 1051336 327 0 1051338 317 0 1051339 243 0 1051340 102 0 1051341 104 0 1051342 130 0 1051343 334 0 1051344 152 0 1051345 56 0 1051346 311 0 1051347 66 0 1051348 221 0 1051349 144 0 1051350 330 0 1051351 35 0 1051352 6 0 1051353 5 0 1051354 62 0 1051355 236 0 1051356 216 0 1051357 146 0 1051358 364 0 1051359 267 0 1051360 146 0 1051361 134 0 1051362 325 0 1051363 275 0 1051364 246 0 1051365 245 0 1051366 47 0 1051367 120 0 1051368 32 0 1051369 203 0 1051370 27 0 1051371 254 0 1051372 122 0 1051373 67 0 1051374 103 0 1051375 22 0 1051376 230 0 1051377 356 0 1051378 122 0 1051379 40 0 1051380 343 0 1051381 317 0 1051382 316 0 1051383 261 0 1051384 32 0 1051385 270 0 1051386 12 0 1051387 141 0 1051388 226 0 1051389 151 0 1051390 222 0 1051391 141 0 1051392 223 0 1051393 4 0 1051394 346 0 1051395 125 0 1051396 327 0 1051397 45 0 1051398 161 0 1051399 215 0 1051400 120 0 1051401 240 0 1051402 51 0 1051403 173 0 1051404 222 0 1051405 51 0 1051406 271 0 1051407 367 0 1051408 317 0 1051409 226 0 1051410 367 0 1051411 23 0 1051412 203 0 1051413 235 0 1051414 142 0 1051415 356 0 1051416 371 0 1051417 225 0 1051418 257 0 1051419 304 0 1051420 133 0 1051421 20 0 1051422 134 0 1051423 62 0 1051424 361 0 1051425 116 0 1051426 315 0 1051427 140 0 1051428 343 0 1051429 21 0 1051430 30 0 1051431 345 0 1051432 37 0 1051433 210 0 1051434 307 0 1051435 174 0 1051436 112 0 1051437 145 0 1051438 4 0 1051439 54 0 1051440 372 0 1051441 114 0 1051442 74 0 1051443 310 0 1051444 306 0 1051445 235 0 1051446 273 0 1051447 216 0 1051448 203 0 1051449 241 0 1051450 23 0 1051451 67 0 1051452 346 0 1051453 342 0 1051454 362 0 1051455 50 0 1051456 63 0 1051457 121 0 1051458 376 0 1051459 227 0 1051460 360 0 1051461 166 0 1051462 220 0 1051463 220 0 1051464 202 0 1051465 363 0 1051466 327 0 1051467 260 0 1051468 310 0 1051469 277 0 1051470 13 0 1051471 363 0 1051472 115 0 1051473 13 0 1051474 151 0 1051475 107 0 1051476 52 0 1051477 301 0 1051478 374 0 1051479 26 0 1051480 275 0 1051481 341 0 1051482 151 0 1051483 130 0 1051484 40 0 1051485 171 0 1051486 375 0 1051487 32 0 1051488 242 0 1051489 146 0 1051490 273 0 1051491 136 0 1051492 254 0 1051493 126 0 1051494 240 0 1051495 233 0 1051496 211 0 1051497 322 0 1051498 347 0 1051499 335 0 1051500 30 0 1051501 220 0 1051502 251 0 1051503 345 0 1051504 134 0 1051505 122 0 1051506 266 0 1051507 34 0 1051508 122 0 1051509 10 0 1051510 265 0 1051511 343 0 1051512 270 0 1051513 161 0 1051514 132 0 1051515 205 0 1051516 37 0 1051517 143 0 1051518 146 0 1051519 150 0 1051520 130 0 1051521 224 0 1051522 175 0 1051523 316 0 1051524 255 0 1051525 101 0 1051526 201 0 1051527 217 0 1051528 361 0 1051529 301 0 1051530 300 0 1051531 75 0 1051532 265 0 1051533 63 0 1051534 15 0 1051535 42 0 1051536 75 0 1051537 130 0 1051538 151 0 1051539 215 0 1051540 173 0 1051541 364 0 1051542 71 0 1051543 27 0 1051544 364 0 1051545 206 0 1051546 31 0 1051547 41 0 1051548 21 0 1051549 6 0 1051550 205 0 1051551 147 0 1051552 46 0 1051553 210 0 1051554 365 0 1051555 212 0 1051556 330 0 1051557 343 0 1051558 374 0 1051559 114 0 1051560 114 0 1051561 23 0 1051562 276 0 1051563 255 0 1051564 144 0 1051565 331 0 1051566 175 0 1051567 354 0 1051568 155 0 1051569 304 0 1051570 130 0 1051571 24 0 1051572 154 0 1051573 121 0 1051574 61 0 1051575 302 0 1051576 166 0 1051577 75 0 1051578 256 0 1051579 340 0 1051580 16 0 1051581 326 0 1051582 70 0 1051583 337 0 1051584 142 0 1051585 127 0 1051586 157 0 1051587 62 0 1051588 325 0 1051589 337 0 1051590 264 0 1051591 165 0 1051592 350 0 1051593 103 0 1051594 315 0 1051595 157 0 1051596 5 0 1051597 46 0 1051598 104 0 1051599 340 0 1051600 126 0 1051601 156 0 1051602 31 0 1051603 163 0 1051604 21 0 1051605 103 0 1051606 364 0 1051607 345 0 1051608 234 0 1051609 337 0 1051610 347 0 1051611 367 0 1051612 11 0 1051613 174 0 1051614 234 0 1051615 347 0 1051616 351 0 1051617 130 0 1051618 304 0 1051619 361 0 1051620 13 0 1051621 70 0 1051622 261 0 1051623 166 0 1051624 150 0 1051625 143 0 1051626 223 0 1051627 33 0 1051628 232 0 1051629 122 0 1051630 235 0 1051631 43 0 1051632 20 0 1051633 26 0 1051634 125 0 1051635 226 0 1051636 37 0 1051637 350 0 1051638 234 0 1051639 100 0 1051640 312 0 1051641 125 0 1051642 354 0 1051643 13 0 1051644 111 0 1051645 271 0 1051646 202 0 1051647 200 0 1051648 40 0 1051649 67 0 1051650 311 0 1051651 336 0 1051652 341 0 1051653 63 0 1051654 341 0 1051655 166 0 1051656 265 0 1051657 201 0 1051658 270 0 1051659 205 0 1051660 150 0 1051661 313 0 1051662 266 0 1051663 145 0 1051664 372 0 1051665 242 0 1051666 314 0 1051667 100 0 1051668 241 0 1051669 136 0 1051670 65 0 1051671 221 0 1051672 261 0 1051673 51 0 1051674 120 0 1051675 56 0 1051676 226 0 1051677 371 0 1051678 22 0 1051679 142 0 1051680 2 0 1051681 154 0 1051682 132 0 1051683 123 0 1051684 210 0 1051685 356 0 1051686 333 0 1051687 227 0 1051688 340 0 1051689 106 0 1051690 340 0 1051691 171 0 1051692 15 0 1051693 112 0 1051694 101 0 1051695 17 0 1051696 52 0 1051697 214 0 1051698 263 0 1051699 276 0 1051700 170 0 1051701 20 0 1051702 214 0 1051703 163 0 1051704 201 0 1051705 224 0 1051706 61 0 1051707 201 0 1051708 13 0 1051709 316 0 1051710 133 0 1051711 43 0 1051712 202 0 1051713 276 0 1051714 105 0 1051715 77 0 1051716 364 0 1051717 224 0 1051718 260 0 1051719 265 0 1051720 151 0 1051721 255 0 1051722 70 0 1051723 37 0 1051724 127 0 1051725 144 0 1051726 266 0 1051727 161 0 1051728 147 0 1051729 125 0 1051730 217 0 1051731 350 0 1051732 233 0 1051733 223 0 1051734 3 0 1051735 16 0 1051736 4 0 1051737 46 0 1051738 175 0 1051739 365 0 1051740 26 0 1051741 316 0 1051742 160 0 1051743 135 0 1051744 173 0 1051745 355 0 1051746 215 0 1051747 43 0 1051748 117 0 1051749 45 0 1051750 162 0 1051751 123 0 1051752 325 0 1051753 166 0 1051754 337 0 1051755 132 0 1051756 315 0 1051757 57 0 1051758 374 0 1051759 327 0 1051760 374 0 1051761 230 0 1051762 140 0 1051763 5 0 1051764 254 0 1051765 24 0 1051766 254 0 1051767 143 0 1051768 112 0 1051769 72 0 1051770 40 0 1051771 274 0 1051772 105 0 1051773 232 0 1051774 253 0 1051775 124 0 1051776 136 0 1051777 25 0 1051778 60 0 1051779 111 0 1051780 327 0 1051781 176 0 1051782 322 0 1051783 275 0 1051784 4 0 1051785 316 0 1051786 40 0 1051787 234 0 1051788 376 0 1051789 235 0 1051790 375 0 1051791 320 0 1051792 377 0 1051793 11 0 1051794 262 0 1051795 212 0 1051796 345 0 1051797 122 0 1051798 7 0 1051799 64 0 1051800 20 0 1051801 143 0 1051802 67 0 1051803 263 0 1051804 344 0 1051805 25 0 1051806 310 0 1051807 141 0 1051808 345 0 1051809 175 0 1051810 315 0 1051811 225 0 1051812 151 0 1051813 166 0 1051814 10 0 1051815 320 0 1051816 252 0 1051817 62 0 1051818 253 0 1051819 162 0 1051820 173 0 1051821 361 0 1051822 177 0 1051823 55 0 1051824 7 0 1051825 252 0 1051826 355 0 1051827 335 0 1051828 267 0 1051829 302 0 1051830 361 0 1051831 271 0 1051832 233 0 1051833 201 0 1051834 66 0 1051835 41 0 1051836 222 0 1051837 244 0 1051838 1 0 1051839 11 0 1051840 141 0 1051841 313 0 1051842 313 0 1051843 251 0 1051844 51 0 1051845 260 0 1051846 266 0 1051847 205 0 1051848 141 0 1051849 366 0 1051850 305 0 1051851 164 0 1051852 331 0 1051853 43 0 1051854 346 0 1051855 56 0 1051856 376 0 1051857 270 0 1051858 73 0 1051859 301 0 1051860 345 0 1051861 373 0 1051862 121 0 1051863 321 0 1051864 332 0 1051865 230 0 1051866 263 0 1051867 17 0 1051868 7 0 1051869 46 0 1051870 72 0 1051871 60 0 1051872 115 0 1051873 160 0 1051874 7 0 1051875 17 0 1051876 226 0 1051877 20 0 1051878 100 0 1051879 15 0 1051880 176 0 1051881 232 0 1051882 161 0 1051883 76 0 1051884 1 0 1051885 67 0 1051886 71 0 1051887 264 0 1051888 135 0 1051889 76 0 1051890 103 0 1051891 167 0 1051892 177 0 1051893 112 0 1051894 260 0 1051895 72 0 1051896 171 0 1051897 142 0 1051898 121 0 1051899 15 0 1051900 7 0 1051901 162 0 1051902 261 0 1051903 6 0 1051904 265 0 1051905 167 0 1051906 72 0 1051907 366 0 1051908 246 0 1051909 45 0 1051910 364 0 1051911 21 0 1051912 376 0 1051913 211 0 1051914 222 0 1051915 323 0 1051916 362 0 1051917 173 0 1051918 351 0 1051919 232 0 1051920 240 0 1051921 214 0 1051922 332 0 1051924 232 0 1051925 315 0 1051926 331 0 1051927 60 0 1051928 310 0 1051929 226 0 1051930 233 0 1051931 232 0 1051932 224 0 1051933 354 0 1051934 37 0 1051935 3 0 1051936 131 0 1051937 22 0 1051938 152 0 1051939 32 0 1051940 4 0 1051941 11 0 1051942 172 0 1051943 165 0 1051944 150 0 1051945 64 0 1051946 340 0 1051947 375 0 1051948 14 0 1051949 163 0 1051950 104 0 1051951 164 0 1051952 3 0 1051953 130 0 1051954 365 0 1051955 226 0 1051956 275 0 1051957 3 0 1051958 30 0 1051959 33 0 1051960 54 0 1051961 322 0 1051962 342 0 1051963 107 0 1051964 323 0 1051965 257 0 1051966 23 0 1051967 106 0 1051968 352 0 1051969 140 0 1051970 150 0 1051971 252 0 1051972 177 0 1051973 176 0 1051974 111 0 1051975 224 0 1051976 117 0 1051977 63 0 1051978 373 0 1051979 246 0 1051980 224 0 1051981 13 0 1051982 301 0 1051983 125 0 1051984 312 0 1051985 174 0 1051986 300 0 1051987 266 0 1051988 246 0 1051989 226 0 1051990 260 0 1051991 202 0 1051992 340 0 1051993 15 0 1051994 351 0 1051995 113 0 1051996 172 0 1051997 44 0 1051998 333 0 1051999 111 0 1052000 311 0 1052001 72 0 1052002 174 0 1052003 306 0 1052004 104 0 1052005 135 0 1052006 142 0 1052007 313 0 1052008 267 0 1052009 62 0 1052010 151 0 1052011 272 0 1052012 105 0 1052013 243 0 1052014 216 0 1052015 311 0 1052016 336 0 1052017 275 0 1052018 103 0 1052019 31 0 1052020 45 0 1052021 133 0 1052022 347 0 1052023 317 0 1052024 55 0 1052025 254 0 1052026 33 0 1052027 367 0 1052028 205 0 1052029 71 0 1052030 110 0 1052031 25 0 1052032 103 0 1052033 4 0 1052034 336 0 1052035 33 0 1052036 32 0 1052037 311 0 1052038 366 0 1052039 224 0 1052040 107 0 1052041 232 0 1052042 104 0 1052043 324 0 1052044 276 0 1052045 362 0 1052046 341 0 1052047 372 0 1052048 77 0 1052049 122 0 1052050 343 0 1052051 112 0 1052052 15 0 1052053 12 0 1052054 36 0 1052055 236 0 1052056 242 0 1052057 60 0 1052058 206 0 1052059 62 0 1052060 156 0 1052061 301 0 1052062 273 0 1052063 257 0 1052064 264 0 1052065 212 0 1052066 364 0 1052067 136 0 1052068 237 0 1052069 133 0 1052070 62 0 1052071 276 0 1052072 370 0 1052073 166 0 1052074 2 0 1052075 36 0 1052076 256 0 1052077 373 0 1052078 203 0 1052079 150 0 1052080 26 0 1052081 304 0 1052082 77 0 1052083 132 0 1052084 12 0 1052085 121 0 1052086 347 0 1052087 161 0 1052088 144 0 1052089 222 0 1052090 124 0 1052091 144 0 1052092 367 0 1052093 362 0 1052094 154 0 1052095 233 0 1052096 54 0 1052097 12 0 1052098 137 0 1052099 104 0 1052100 60 0 1052101 217 0 1052102 266 0 1052103 56 0 1052104 344 0 1052105 231 0 1052106 66 0 1052107 44 0 1052108 163 0 1052109 140 0 1052110 127 0 1052111 106 0 1052112 276 0 1052113 377 0 1052114 47 0 1052115 17 0 1052116 306 0 1052117 165 0 1052118 275 0 1052119 353 0 1052120 355 0 1052121 111 0 1052122 32 0 1052123 312 0 1052124 6 0 1052125 201 0 1052126 104 0 1052127 307 0 1052128 107 0 1052129 27 0 1052130 361 0 1052131 247 0 1052132 12 0 1052133 144 0 1052134 347 0 1052135 17 0 1052136 365 0 1052137 72 0 1052138 275 0 1052139 261 0 1052140 5 0 1052141 25 0 1052142 222 0 1052143 375 0 1052144 235 0 1052145 277 0 1052146 213 0 1052147 205 0 1052148 122 0 1052149 101 0 1052150 254 0 1052151 365 0 1052152 176 0 1052153 327 0 1052154 275 0 1052155 235 0 1052156 160 0 1052157 226 0 1052158 273 0 1052159 150 0 1052160 335 0 1052161 131 0 1052162 6 0 1052163 253 0 1052164 260 0 1052165 4 0 1052166 267 0 1052167 100 0 1052168 370 0 1052169 50 0 1052170 243 0 1052171 304 0 1052172 334 0 1052173 64 0 1052174 155 0 1052175 332 0 1052176 353 0 1052177 207 0 1052178 344 0 1052179 230 0 1052180 124 0 1052181 3 0 1052182 126 0 1052183 271 0 1052184 272 0 1052185 43 0 1052186 261 0 1052187 132 0 1052188 333 0 1052189 41 0 1052190 136 0 1052191 262 0 1052192 236 0 1052193 23 0 1052194 50 0 1052195 367 0 1052196 71 0 1052197 61 0 1052198 316 0 1052199 107 0 1052200 121 0 1052201 61 0 1052202 131 0 1052203 227 0 1052204 200 0 1052205 107 0 1052206 152 0 1052207 170 0 1052208 343 0 1052209 34 0 1052210 25 0 1052211 336 0 1052212 251 0 1052213 166 0 1052214 227 0 1052215 67 0 1052216 230 0 1052217 301 0 1052218 14 0 1052219 166 0 1052220 31 0 1052221 114 0 1052222 210 0 1052223 271 0 1052224 25 0 1052225 105 0 1052226 150 0 1052227 21 0 1052228 302 0 1052229 145 0 1052230 307 0 1052231 357 0 1052232 74 0 1052233 260 0 1052234 323 0 1052235 172 0 1052236 334 0 1052237 106 0 1052238 226 0 1052239 65 0 1052240 105 0 1052241 320 0 1052242 44 0 1052243 207 0 1052244 103 0 1052245 362 0 1052246 241 0 1052247 233 0 1052248 50 0 1052249 361 0 1052251 160 0 1052252 362 0 1052253 102 0 1052254 274 0 1052255 34 0 1052256 217 0 1052257 3 0 1052258 320 0 1052259 261 0 1052260 101 0 1052261 353 0 1052262 115 0 1052263 316 0 1052264 225 0 1052265 44 0 1052266 36 0 1052267 267 0 1052268 214 0 1052269 45 0 1052270 112 0 1052271 117 0 1052272 251 0 1052273 171 0 1052274 371 0 1052275 336 0 1052276 362 0 1052277 4 0 1052278 104 0 1052279 153 0 1052280 271 0 1052281 136 0 1052282 240 0 1052283 136 0 1052284 313 0 1052285 312 0 1052286 275 0 1052287 103 0 1052288 54 0 1052289 137 0 1052290 127 0 1052291 137 0 1052292 2 0 1052293 114 0 1052294 173 0 1052295 140 0 1052296 24 0 1052297 357 0 1052298 276 0 1052299 111 0 1052300 306 0 1052301 37 0 1052302 15 0 1052303 162 0 1052304 237 0 1052305 262 0 1052306 336 0 1052307 174 0 1052308 346 0 1052309 135 0 1052310 354 0 1052311 16 0 1052312 313 0 1052313 11 0 1052314 354 0 1052315 221 0 1052316 344 0 1052317 235 0 1052318 113 0 1052319 313 0 1052320 162 0 1052321 305 0 1052322 332 0 1052323 71 0 1052324 116 0 1052325 34 0 1052326 41 0 1052327 346 0 1052328 375 0 1052329 204 0 1052330 274 0 1052331 357 0 1052332 350 0 1052333 233 0 1052334 200 0 1052335 275 0 1052336 116 0 1052337 141 0 1052338 364 0 1052339 316 0 1052340 321 0 1052341 164 0 1052342 112 0 1052343 34 0 1052344 330 0 1052345 264 0 1052346 341 0 1052347 221 0 1052348 177 0 1052349 320 0 1052350 312 0 1052351 341 0 1052352 143 0 1052353 31 0 1052354 100 0 1052355 236 0 1052356 223 0 1052357 20 0 1052358 226 0 1052359 105 0 1052360 23 0 1052361 336 0 1052362 307 0 1052363 307 0 1052364 4 0 1052365 336 0 1052366 136 0 1052367 276 0 1052368 227 0 1052369 275 0 1052370 26 0 1052371 272 0 1052372 117 0 1052373 174 0 1052374 246 0 1052375 253 0 1052376 310 0 1052377 1 0 1052378 116 0 1052379 62 0 1052380 77 0 1052381 207 0 1052382 202 0 1052383 24 0 1052384 63 0 1052385 65 0 1052386 363 0 1052387 304 0 1052388 266 0 1052389 60 0 1052390 63 0 1052391 67 0 1052392 230 0 1052393 4 0 1052394 1 0 1052395 122 0 1052396 304 0 1052397 35 0 1052398 40 0 1052399 375 0 1052400 247 0 1052401 117 0 1052402 112 0 1052403 240 0 1052404 242 0 1052405 123 0 1052406 170 0 1052407 141 0 1052408 366 0 1052409 116 0 1052410 265 0 1052411 356 0 1052412 240 0 1052413 256 0 1052414 311 0 1052415 251 0 1052416 127 0 1052417 256 0 1052418 367 0 1052419 343 0 1052420 251 0 1052421 160 0 1052422 121 0 1052423 367 0 1052424 356 0 1052425 112 0 1052426 76 0 1052427 141 0 1052428 114 0 1052429 147 0 1052430 326 0 1052431 163 0 1052432 72 0 1052433 210 0 1052434 320 0 1052435 167 0 1052436 61 0 1052437 361 0 1052438 70 0 1052439 326 0 1052440 207 0 1052441 34 0 1052442 166 0 1052443 304 0 1052444 142 0 1052445 23 0 1052446 70 0 1052447 113 0 1052448 272 0 1052449 7 0 1052450 337 0 1052451 264 0 1052452 34 0 1052453 305 0 1052454 263 0 1052455 46 0 1052456 5 0 1052457 271 0 1052458 142 0 1052459 376 0 1052460 220 0 1052461 121 0 1052462 67 0 1052463 60 0 1052464 267 0 1052465 333 0 1052466 377 0 1052467 41 0 1052468 70 0 1052469 163 0 1052470 143 0 1052471 155 0 1052472 67 0 1052473 163 0 1052474 333 0 1052475 146 0 1052476 64 0 1052477 170 0 1052478 46 0 1052479 161 0 1052480 232 0 1052481 72 0 1052482 102 0 1052483 212 0 1052484 235 0 1052485 267 0 1052486 77 0 1052487 371 0 1052488 103 0 1052489 225 0 1052490 345 0 1052491 345 0 1052492 277 0 1052493 46 0 1052494 260 0 1052495 311 0 1052496 73 0 1052497 116 0 1052498 50 0 1052499 343 0 1052500 30 0 1052501 153 0 1052502 155 0 1052503 35 0 1052504 325 0 1052505 123 0 1052506 167 0 1052507 341 0 1052508 204 0 1052509 34 0 1052510 67 0 1052511 21 0 1052512 312 0 1052513 24 0 1052514 335 0 1052515 11 0 1052516 273 0 1052517 43 0 1052518 242 0 1052519 304 0 1052520 205 0 1052521 32 0 1052522 304 0 1052523 25 0 1052524 223 0 1052525 275 0 1052526 367 0 1052527 260 0 1052528 74 0 1052529 370 0 1052530 234 0 1052531 310 0 1052532 27 0 1052533 362 0 1052534 210 0 1052535 331 0 1052536 135 0 1052537 112 0 1052538 320 0 1052539 31 0 1052540 142 0 1052541 353 0 1052542 24 0 1052543 211 0 1052544 216 0 1052545 364 0 1052546 105 0 1052547 116 0 1052548 64 0 1052549 115 0 1052550 125 0 1052551 135 0 1052552 211 0 1052553 172 0 1052554 361 0 1052555 335 0 1052556 271 0 1052557 206 0 1052558 127 0 1052559 166 0 1052560 325 0 1052561 126 0 1052562 101 0 1052563 202 0 1052564 177 0 1052565 201 0 1052566 167 0 1052567 222 0 1052568 225 0 1052569 361 0 1052570 207 0 1052571 116 0 1052572 67 0 1052573 226 0 1052574 226 0 1052575 146 0 1052576 357 0 1052577 366 0 1052578 16 0 1052579 365 0 1052580 23 0 1052581 130 0 1052582 330 0 1052583 250 0 1052584 113 0 1052586 24 0 1052587 123 0 1052588 120 0 1052589 155 0 1052590 324 0 1052591 175 0 1052592 236 0 1052593 154 0 1052594 265 0 1052595 276 0 1052596 345 0 1052597 173 0 1052598 122 0 1052599 35 0 1052600 164 0 1052601 250 0 1052602 161 0 1052603 240 0 1052604 252 0 1052605 371 0 1052606 47 0 1052607 245 0 1052608 165 0 1052609 303 0 1052610 145 0 1052611 251 0 1052612 317 0 1052613 160 0 1052614 322 0 1052615 70 0 1052616 307 0 1052617 25 0 1052618 60 0 1052619 371 0 1052620 167 0 1052621 116 0 1052622 267 0 1052623 336 0 1052624 61 0 1052625 253 0 1052626 161 0 1052627 237 0 1052628 270 0 1052629 111 0 1052630 322 0 1052631 257 0 1052632 254 0 1052633 113 0 1052634 346 0 1052635 70 0 1052636 163 0 1052637 205 0 1052638 306 0 1052639 275 0 1052640 253 0 1052641 2 0 1052642 345 0 1052643 336 0 1052644 333 0 1052645 235 0 1052646 270 0 1052647 155 0 1052648 247 0 1052649 2 0 1052650 314 0 1052651 224 0 1052652 234 0 1052653 266 0 1052654 371 0 1052655 352 0 1052656 105 0 1052657 111 0 1052658 161 0 1052660 306 0 1052661 76 0 1052662 362 0 1052663 357 0 1052664 265 0 1052665 136 0 1052666 25 0 1052667 103 0 1052668 364 0 1052669 242 0 1052670 216 0 1052671 3 0 1052672 314 0 sanity test_119e: @@@@@@ FAIL: files differ, bsize 1049600 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:15629:test_119e() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:15633:main() Dumping lctl log to /tmp/testlogs//sanity.test_119e.*.1787743082.log llite.lustre-ffff93f6a0950800.unaligned_dio=1 rsync: chown "/tmp/testlogs/.sanity.test_119e.debug_log.oleg315-server.1787743082.log.YNrACS" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_119e.dmesg.oleg315-server.1787743082.log.wP1y9l" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 119e (117s) == sanity test 119f: dio vs dio race ===================== 07:19:26 (1787743166) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.333 s, 19.7 MB/s bs: 1044480 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.08199 s, 6.4 MB/s 25+1 records in 25+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.32433 s, 4.9 MB/s /mnt/lustre/f119f.sanity.2 has type file OK /mnt/lustre/f119f.sanity.2 has size 26214400 OK bs: 1048576 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.03933 s, 6.5 MB/s 25+0 records in 25+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.18725 s, 5.1 MB/s /mnt/lustre/f119f.sanity.2 has type file OK /mnt/lustre/f119f.sanity.2 has size 26214400 OK bs: 1049600 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.18984 s, 6.3 MB/s 24+1 records in 24+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 5.84584 s, 4.5 MB/s /mnt/lustre/f119f.sanity.2 has type file OK /mnt/lustre/f119f.sanity.2 has size 26214400 OK PASS 119f (46s) == sanity test 119g: dio vs buffered I/O race ============ 07:20:12 (1787743212) 1+0 records in 1+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.6666 s, 15.7 MB/s bs: 1044480 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 7.59931 s, 3.4 MB/s 25+1 records in 25+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 9.93468 s, 2.6 MB/s /mnt/lustre/f119g.sanity.2 has type file OK /mnt/lustre/f119g.sanity.2 has size 26214400 OK bs: 1048576 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.03395 s, 12.9 MB/s 25+0 records in 25+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 4.56662 s, 5.7 MB/s /mnt/lustre/f119g.sanity.2 has type file OK /mnt/lustre/f119g.sanity.2 has size 26214400 OK bs: 1049600 12+1 records in 12+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 2.78486 s, 9.4 MB/s 24+1 records in 24+1 records out 26214400 bytes (26 MB, 25 MiB) copied, 6.74392 s, 3.9 MB/s /mnt/lustre/f119g.sanity.2 has type file OK /mnt/lustre/f119g.sanity.2 has size 26214400 OK PASS 119g (52s) == sanity test 119h: basic tests of memory unaligned dio ========================================================== 07:21:05 (1787743265) unaligned writes of blocksize: 1044480 unaligned writes of blocksize: 1048576 unaligned writes of blocksize: 1049600 5+0 records in 5+0 records out 26214400 bytes (26 MB, 25 MiB) copied, 1.05127 s, 24.9 MB/s unaligned reads of blocksize: 1044480 unaligned reads of blocksize: 1048576 unaligned reads of blocksize: 1049600 PASS 119h (38s) SKIP: sanity test_119i skipping ALWAYS excluded test 119i == sanity test 119j: basic tests of hybrid IO switching == 07:21:44 (1787743304) 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0281945 s, 145 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000426737 s, 9.6 MB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.000662795 s, 6.2 MB/s osc.lustre-OST0000-osc-ffff93f6a0950800.rpc_stats= snapshot_time: 1787743306.211885796 secs.nsecs start_time: 1787743306.188039009 secs.nsecs elapsed_time: 0.023846787 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 read write offset rpcs % cum % | rpcs % cum % 0: 0 0 0 | 0 0 0 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 osc.lustre-OST0001-osc-ffff93f6a0950800.rpc_stats= snapshot_time: 1787743306.211957169 secs.nsecs start_time: 1787743306.188071411 secs.nsecs elapsed_time: 0.023885758 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 read write offset rpcs % cum % | rpcs % cum % 0: 0 0 0 | 0 0 0 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 fail_loc=0x1429 1+0 records in 1+0 records out 8 bytes copied, 0.021167 s, 0.4 kB/s osc.lustre-OST0000-osc-ffff93f6a0950800.rpc_stats= snapshot_time: 1787743306.378577122 secs.nsecs start_time: 1787743306.188039009 secs.nsecs elapsed_time: 0.190538113 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 1 100 100 | 0 0 0 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 1 100 100 | 0 0 0 read write offset rpcs % cum % | rpcs % cum % 0: 1 100 100 | 0 0 0 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 0 0 0 | 0 0 0 4096: 1 100 100 | 0 0 0 osc.lustre-OST0001-osc-ffff93f6a0950800.rpc_stats= snapshot_time: 1787743306.379866261 secs.nsecs start_time: 1787743306.188071411 secs.nsecs elapsed_time: 0.191794850 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 read write offset rpcs % cum % | rpcs % cum % 0: 0 0 0 | 0 0 0 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 fail_loc=0x1429 1+0 records in 1+0 records out 8 bytes copied, 0.0196478 s, 0.4 kB/s osc.lustre-OST0000-osc-ffff93f6a0950800.rpc_stats= snapshot_time: 1787743306.628322007 secs.nsecs start_time: 1787743306.188039009 secs.nsecs elapsed_time: 0.440282998 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 1 100 100 | 1 100 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 1 100 100 | 1 100 100 read write offset rpcs % cum % | rpcs % cum % 0: 1 100 100 | 1 100 100 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 0 0 0 | 0 0 0 4096: 1 100 100 | 0 0 0 8192: 0 0 100 | 0 0 0 16384: 0 0 100 | 1 100 100 osc.lustre-OST0001-osc-ffff93f6a0950800.rpc_stats= snapshot_time: 1787743306.628492625 secs.nsecs start_time: 1787743306.188071411 secs.nsecs elapsed_time: 0.440421214 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 read write offset rpcs % cum % | rpcs % cum % 0: 0 0 0 | 0 0 0 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 PASS 119j (8s) == sanity test 119k: hybrid IO counting with stats and disabling ========================================================== 07:21:52 (1787743312) llite.lustre-ffff93f6a0950800.hybrid_io=1 llite.lustre-ffff93f6a0950800.stats=c 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.13915 s, 7.4 MB/s llite.lustre-ffff93f6a0950800.stats= snapshot_time 1787743316.277514173 secs.nsecs start_time 1787743314.972103511 secs.nsecs elapsed_time 1.305410662 secs.nsecs write_bytes 8 samples [bytes] 1048576 1048576 8388608 8796093022208 write 8 samples [usecs] 109270 186118 1125484 162732947406 open 1 samples [usecs] 97 97 97 9409 close 1 samples [usecs] 9657 9657 9657 93257649 mknod 1 samples [usecs] 18375 18375 18375 337640625 inode_permission 3 samples [usecs] 5 13364 13586 178643610 opencount 1 samples [reqs] 1 1 1 1 hybrid_noswitch 8 samples [reqs] 8+0 records in 8+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 1.431 s, 11.7 MB/s llite.lustre-ffff93f6a0950800.stats= snapshot_time 1787743318.346454088 secs.nsecs start_time 1787743314.972103511 secs.nsecs elapsed_time 3.374350577 secs.nsecs write_bytes 16 samples [bytes] 1048576 2097152 25165824 43980465111040 hybrid_write_bytes 8 samples [bytes] 2097152 2097152 16777216 35184372088832 write 16 samples [usecs] 109270 209845 2516150 411105087044 open 2 samples [usecs] 97 28203 28300 795418618 close 2 samples [usecs] 6707 9657 16364 138241498 truncate 1 samples [usecs] 484775 484775 484775 235006800625 mknod 1 samples [usecs] 18375 18375 18375 337640625 getxattr 1 samples [usecs] 9972 9972 9972 99440784 inode_permission 5 samples [usecs] 4 23171 36761 715538867 opencount 2 samples [reqs] 1 2 3 5 openclosetime 1 samples [usecs] 166655 166655 166655 27773889025 hybrid_noswitch 8 samples [reqs] hybrid_writesize_switch 8 samples [reqs] llite.lustre-ffff93f6a0950800.stats=c 2+0 records in 2+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.84791 s, 4.5 MB/s llite.lustre-ffff93f6a0950800.stats= snapshot_time 1787743320.364619106 secs.nsecs start_time 1787743318.358716959 secs.nsecs elapsed_time 2.005902147 secs.nsecs read_bytes 2 samples [bytes] 4194304 4194304 8388608 35184372088832 read 2 samples [usecs] 839790 996605 1836395 1698468770125 open 1 samples [usecs] 1595 1595 1595 2544025 close 1 samples [usecs] 11032 11032 11032 121705024 seek 1 samples [usecs] 18 18 18 324 inode_permission 2 samples [usecs] 4 7439 7443 55338737 opencount 1 samples [reqs] 3 3 3 9 openclosetime 1 samples [usecs] 165341 165341 165341 27337646281 hybrid_noswitch 2 samples [reqs] 2+0 records in 2+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 0.894685 s, 18.8 MB/s llite.lustre-ffff93f6a0950800.stats= snapshot_time 1787743321.849095811 secs.nsecs start_time 1787743318.358716959 secs.nsecs elapsed_time 3.490378852 secs.nsecs read_bytes 4 samples [bytes] 4194304 8388608 25165824 175921860444160 hybrid_read_bytes 2 samples [bytes] 8388608 8388608 16777216 140737488355328 read 4 samples [usecs] 422189 996605 2724550 2093836635002 open 2 samples [usecs] 1595 15074 16669 229769501 close 2 samples [usecs] 5771 11032 16803 155009465 seek 2 samples [usecs] 18 31 49 1285 inode_permission 4 samples [usecs] 3 7439 7687 55396827 opencount 2 samples [reqs] 3 4 7 25 openclosetime 2 samples [usecs] 165341 567366 732707 349241824237 hybrid_noswitch 2 samples [reqs] hybrid_readsize_switch 2 samples [reqs] llite.lustre-ffff93f6a0950800.stats=c llite.lustre-ffff93f6a0950800.hybrid_io=0 8+0 records in 8+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 2.91943 s, 5.7 MB/s llite.lustre-ffff93f6a0950800.stats= snapshot_time 1787743325.165251935 secs.nsecs start_time 1787743321.879279103 secs.nsecs elapsed_time 3.285972832 secs.nsecs write_bytes 8 samples [bytes] 2097152 2097152 16777216 35184372088832 write 8 samples [usecs] 282163 484030 2899443 1090143757637 open 1 samples [usecs] 42253 42253 42253 1785316009 close 1 samples [usecs] 111 111 111 12321 truncate 1 samples [usecs] 142092 142092 142092 20190136464 getxattr 1 samples [usecs] 31 31 31 961 getxattr_hits 1 samples [reqs] inode_permission 2 samples [usecs] 4 929 933 863057 opencount 1 samples [reqs] 5 5 5 25 openclosetime 1 samples [usecs] 184529 184529 184529 34050951841 2+0 records in 2+0 records out 16777216 bytes (17 MB, 16 MiB) copied, 3.94566 s, 4.3 MB/s llite.lustre-ffff93f6a0950800.stats= snapshot_time 1787743330.450285851 secs.nsecs start_time 1787743321.879279103 secs.nsecs elapsed_time 8.571006748 secs.nsecs read_bytes 2 samples [bytes] 8388608 8388608 16777216 140737488355328 write_bytes 8 samples [bytes] 2097152 2097152 16777216 35184372088832 read 2 samples [usecs] 1872395 2049541 3921936 7706481346706 write 8 samples [usecs] 282163 484030 2899443 1090143757637 open 2 samples [usecs] 62 42253 42315 1785319853 close 2 samples [usecs] 111 14230 14341 202505221 seek 1 samples [usecs] 15 15 15 225 truncate 1 samples [usecs] 142092 142092 142092 20190136464 getxattr 1 samples [usecs] 31 31 31 961 getxattr_hits 1 samples [reqs] inode_permission 4 samples [usecs] 4 15999 16937 256831083 opencount 2 samples [reqs] 5 6 11 61 openclosetime 2 samples [usecs] 184529 1395052 1579581 1980221034545 PASS 119k (27s) == sanity test 119m: Test DIO readv/writev: exercise iter duplication ========================================================== 07:22:19 (1787743339) PASS 119m (12s) == sanity test 119n: Test Unaligned DIO readv() and writev() with unpatched ZFS ========================================================== 07:22:31 (1787743351) SKIP: sanity test_119n need ZFS server without unaligned_dio support SKIP 119n (5s) == sanity test 119o: Test Unaligned DIO readv() and writev() with unpatched servers ========================================================== 07:22:37 (1787743357) SKIP: sanity test_119o need ldiskfs without unaligned_dio support. SKIP 119o (6s) == sanity test 119p: Test Unaligned DIO readv() and writev() with patched servers ========================================================== 07:22:43 (1787743363) Data mismatch 0: ffffffba sanity test_119p: @@@@@@ FAIL: DIO unaligned readv failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:16151:test_119p() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:16154:main() Dumping lctl log to /tmp/testlogs//sanity.test_119p.*.1787743371.log rsync: chown "/tmp/testlogs/.sanity.test_119p.debug_log.oleg315-server.1787743371.log.1gMvV0" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_119p.dmesg.oleg315-server.1787743371.log.OydgVQ" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 119p (103s) == sanity test 119q: Test patchded Unaligned DIO readv() and writev() ========================================================== 07:24:27 (1787743467) RWV interop with 64k + 0x1000 +/- 1 writev: 69631 0x100001 0x100000 readv: 0x100001 69631 0x100000 Data mismatch 0: ffffffba sanity test_119q: @@@@@@ FAIL: DIO unaligned readv failed: 69631 Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:16178:test_119q() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:16200:main() Dumping lctl log to /tmp/testlogs//sanity.test_119q.*.1787743472.log rsync: chown "/tmp/testlogs/.sanity.test_119q.debug_log.oleg315-server.1787743472.log.V0SiNB" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_119q.dmesg.oleg315-server.1787743472.log.NFwibq" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 119q (11s) == sanity test 119r: Test error handling in unaligned DIO user copy ========================================================== 07:24:38 (1787743478) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.154627 s, 6.8 MB/s fail_loc=0x1437 fail_loc=0 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0504331 s, 81.2 kB/s fail_loc=0 PASS 119r (9s) == sanity test 119s: full-size unaligned DIO packs matching bulk MDs ========================================================== 07:24:47 (1787743487) SKIP: sanity test_119s need client page size larger than the server's SKIP 119s (4s) == sanity test 120a: Early Lock Cancel: mkdir test ======= 07:24:51 (1787743491) striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d120a.sanity ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s striped dir -i0 -c1 -H crush /mnt/lustre/d120a.sanity/d1 PASS 120a (13s) == sanity test 120b: Early Lock Cancel: create test ====== 07:25:04 (1787743504) striped dir -i0 -c2 -H all_char /mnt/lustre/d120b.sanity ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s PASS 120b (12s) == sanity test 120c: Early Lock Cancel: link test ======== 07:25:16 (1787743516) striped dir -i0 -c1 -H all_char /mnt/lustre/d120c.sanity ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s striped dir -i0 -c1 -H crush /mnt/lustre/d120c.sanity/d1 striped dir -i0 -c1 -H crush /mnt/lustre/d120c.sanity/d2 PASS 120c (13s) == sanity test 120d: Early Lock Cancel: setattr test ===== 07:25:29 (1787743529) striped dir -i0 -c1 -H all_char /mnt/lustre/d120d.sanity ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s PASS 120d (10s) == sanity test 120e: Early Lock Cancel: unlink test ====== 07:25:39 (1787743539) striped dir -i0 -c1 -H crush /mnt/lustre/d120e.sanity ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s 1+0 records in 1+0 records out 512 bytes copied, 0.0488954 s, 10.5 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.0409968 s, 0.0 kB/s PASS 120e (20s) == sanity test 120f: Early Lock Cancel: rename test ====== 07:25:59 (1787743559) striped dir -i0 -c1 -H all_char /mnt/lustre/d120f.sanity ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s striped dir -i0 -c1 -H crush /mnt/lustre/d120f.sanity/d1 striped dir -i0 -c1 -H all_char /mnt/lustre/d120f.sanity/d2 1+0 records in 1+0 records out 512 bytes copied, 0.0412536 s, 12.4 kB/s 1+0 records in 1+0 records out 512 bytes copied, 0.0396817 s, 12.9 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.0248762 s, 0.0 kB/s 0+0 records in 0+0 records out 0 bytes copied, 0.0411885 s, 0.0 kB/s PASS 120f (17s) == sanity test 120g: Early Lock Cancel: performance test ========================================================== 07:26:16 (1787743576) ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s ldlm.namespaces.*osc*.lru_size=0->400 ldlm.namespaces.*osc*.lru_max_age=600000->3900s create 10000 files striped dir -i0 -c2 -H all_char /mnt/lustre/d120g.sanity - open/close 718 (time 1787743592.60 total 10.00 last 71.79) - open/close 1456 (time 1787743602.60 total 20.00 last 73.78) - open/close 2178 (time 1787743612.62 total 30.02 last 72.06) - open/close 2899 (time 1787743622.63 total 40.03 last 72.02) - open/close 3632 (time 1787743632.64 total 50.04 last 73.23) - open/close 4395 (time 1787743642.66 total 60.06 last 76.17) - open/close 5148 (time 1787743652.66 total 70.07 last 75.23) - open/close 5905 (time 1787743662.68 total 80.08 last 75.60) - open/close 6647 (time 1787743672.70 total 90.10 last 74.07) - open/close 7339 (time 1787743682.70 total 100.10 last 69.20) - open/close 8056 (time 1787743692.70 total 110.11 last 71.65) - open/close 8835 (time 1787743702.71 total 120.12 last 77.84) - open/close 9620 (time 1787743712.72 total 130.13 last 78.40) total: 10000 open/close in 135.43 seconds: 73.84 ops/second total: 1 cancels, 0 blockings rm 10000 files total: 10000 removes in 313 total: 2 cancels, 0 blockings Waiting for MDT destroys to complete PASS 120g (474s) == sanity test 121: read cancel race ===================== 07:34:11 (1787744051) fail_loc=0x310 fail_loc=0 sanity test_121: @@@@@@ FAIL: read 0 blocks, must be 1 blocks Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:16602:test_121() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:16604:main() Dumping lctl log to /tmp/testlogs//sanity.test_121.*.1787744055.log rsync: chown "/tmp/testlogs/.sanity.test_121.debug_log.oleg315-server.1787744055.log.AoyZ6L" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_121.dmesg.oleg315-server.1787744055.log.U4v6yK" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 121 (79s) == sanity test 123aa: verify statahead work ============== 07:35:30 (1787744130) seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123aa.sanity total: 100 open/close in 3.60 seconds: 27.76 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 101 real 0m2.846s user 0m0.004s sys 0m0.348s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff93f6a0950800.statahead_max=128 llite.lustre-ffff93f6a0950800.statahead_max=128 101 real 0m1.024s user 0m0.004s sys 0m0.181s 'ls -l' 100 files with statahead: 1 sec statahead total: 157 statahead wrong: 2 agl total: 152 list_total: 25 fname_total: 112 hit_total: 1068 miss_total: 777 swrong=2 ewrong=2 - open/close 560 (time 1787744162.33 total 10.01 last 55.96) total: 900 open/close in 15.87 seconds: 56.72 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 1001 real 0m25.861s user 0m0.041s sys 0m3.149s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff93f6a0950800.statahead_max=128 llite.lustre-ffff93f6a0950800.statahead_max=128 1001 real 0m6.018s user 0m0.029s sys 0m1.388s 'ls -l' 1000 files with statahead: 6 sec statahead total: 158 statahead wrong: 2 agl total: 153 list_total: 26 fname_total: 112 hit_total: 2067 miss_total: 778 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123aa.sanity: 24 seconds llite.lustre-ffff93f6a0950800.statahead_stats= statahead total: 158 statahead wrong: 2 agl total: 153 list_total: 26 fname_total: 112 hit_total: 2067 miss_total: 778 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787744236.986431754 secs.nsecs start_time: 1787744136.053769879 secs.nsecs elapsed_time: 100.932661875 secs.nsecs subreqs per batch batches %% cum %% 1: 5 20 20 2: 1 4 24 4: 3 12 36 8: 1 4 40 16: 1 4 44 32: 2 8 52 64: 12 48 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787744236.986487316 secs.nsecs start_time: 1787744136.053879742 secs.nsecs elapsed_time: 100.932607574 secs.nsecs subreqs per batch batches %% cum %% 1: 4 15 15 2: 1 3 19 4: 3 11 30 8: 4 15 46 16: 0 0 46 32: 5 19 65 64: 9 34 100 unlink(/mnt/lustre/d123aa.sanity/f123aa.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 PASS 123aa (118s) == sanity test 123ab: verify statahead work by using statx ========================================================== 07:37:28 (1787744248) seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ab.sanity total: 100 open/close in 2.84 seconds: 35.21 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 100 real 0m2.411s user 0m0.012s sys 0m0.277s 'statx -l' 100 files without statahead: 2 sec llite.lustre-ffff93f6a0950800.statahead_max=128 llite.lustre-ffff93f6a0950800.statahead_max=128 100 real 0m1.107s user 0m0.006s sys 0m0.122s 'statx -l' 100 files with statahead: 1 sec statahead total: 159 statahead wrong: 2 agl total: 154 list_total: 27 fname_total: 112 hit_total: 2166 miss_total: 778 swrong=2 ewrong=2 - open/close 711 (time 1787744278.51 total 10.01 last 71.06) total: 900 open/close in 12.74 seconds: 70.67 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 1000 real 0m29.998s user 0m0.085s sys 0m2.776s 'statx -l' 1000 files without statahead: 30 sec llite.lustre-ffff93f6a0950800.statahead_max=128 llite.lustre-ffff93f6a0950800.statahead_max=128 1000 real 0m5.958s user 0m0.050s sys 0m1.175s 'statx -l' 1000 files with statahead: 6 sec statahead total: 160 statahead wrong: 2 agl total: 155 list_total: 28 fname_total: 112 hit_total: 3165 miss_total: 778 swrong=2 ewrong=2 'statx -l' done unlink + rm -r /mnt/lustre/d123ab.sanity: 32 seconds llite.lustre-ffff93f6a0950800.statahead_stats= statahead total: 160 statahead wrong: 2 agl total: 155 list_total: 28 fname_total: 112 hit_total: 3165 miss_total: 778 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787744362.701688085 secs.nsecs start_time: 1787744254.802303019 secs.nsecs elapsed_time: 107.899385066 secs.nsecs subreqs per batch batches %% cum %% 1: 2 8 8 2: 0 0 8 4: 1 4 13 8: 4 17 30 16: 1 4 34 32: 3 13 47 64: 12 52 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787744362.703599043 secs.nsecs start_time: 1787744254.802391364 secs.nsecs elapsed_time: 107.901207679 secs.nsecs subreqs per batch batches %% cum %% 1: 4 19 19 2: 3 14 33 4: 1 4 38 8: 0 0 38 16: 0 0 38 32: 6 28 66 64: 7 33 100 unlink(/mnt/lustre/d123ab.sanity/f123ab.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 PASS 123ab (130s) == sanity test 123ac: verify statahead work by using statx without glimpse RPCs ========================================================== 07:39:38 (1787744378) seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123ac.sanity total: 100 open/close in 2.85 seconds: 35.13 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 208 real 0m1.097s user 0m0.003s sys 0m0.193s 'statx -c "%n %i %A %h %u %g %W %X %Z" -D' 100 files without statahead: 1 sec llite.lustre-ffff93f6a0950800.statahead_max=128 llite.lustre-ffff93f6a0950800.statahead_max=128 208 real 0m0.833s user 0m0.001s sys 0m0.081s 'statx -c "%n %i %A %h %u %g %W %X %Z" -D' 100 files with statahead: 0 sec statahead total: 161 statahead wrong: 2 agl total: 155 list_total: 29 fname_total: 112 hit_total: 3263 miss_total: 779 swrong=2 ewrong=2 - open/close 649 (time 1787744405.21 total 10.01 last 64.81) total: 900 open/close in 14.23 seconds: 63.23 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 2008 real 0m20.312s user 0m0.030s sys 0m2.078s 'statx -c "%n %i %A %h %u %g %W %X %Z" -D' 1000 files without statahead: 20 sec llite.lustre-ffff93f6a0950800.statahead_max=128 llite.lustre-ffff93f6a0950800.statahead_max=128 2008 real 0m4.029s user 0m0.016s sys 0m0.594s 'statx -c "%n %i %A %h %u %g %W %X %Z" -D' 1000 files with statahead: 4 sec statahead total: 162 statahead wrong: 2 agl total: 155 list_total: 30 fname_total: 112 hit_total: 4261 miss_total: 780 swrong=2 ewrong=2 'statx -c "%n %i %A %h %u %g %W %X %Z" -D' done unlink + rm -r /mnt/lustre/d123ac.sanity: 24 seconds llite.lustre-ffff93f6a0950800.statahead_stats= statahead total: 162 statahead wrong: 2 agl total: 155 list_total: 30 fname_total: 112 hit_total: 4261 miss_total: 780 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787744470.707708201 secs.nsecs start_time: 1787744383.637260355 secs.nsecs elapsed_time: 87.070447846 secs.nsecs subreqs per batch batches %% cum %% 1: 1 4 4 2: 3 12 16 4: 6 24 40 8: 3 12 52 16: 0 0 52 32: 0 0 52 64: 12 48 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787744470.708032529 secs.nsecs start_time: 1787744383.637344028 secs.nsecs elapsed_time: 87.070688501 secs.nsecs subreqs per batch batches %% cum %% 1: 1 4 4 2: 3 12 16 4: 4 16 32 8: 5 20 52 16: 0 0 52 32: 0 0 52 64: 12 48 100 kvm mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ac.sanity total: 100 open/close in 3.53 seconds: 28.32 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 100 real 0m0.330s user 0m0.002s sys 0m0.030s 'statx --cached=always -D' 100 files without statahead: 1 sec llite.lustre-ffff93f6a0950800.statahead_max=128 llite.lustre-ffff93f6a0950800.statahead_max=128 100 real 0m0.308s user 0m0.003s sys 0m0.029s 'statx --cached=always -D' 100 files with statahead: 0 sec statahead total: 162 statahead wrong: 2 agl total: 155 list_total: 30 fname_total: 112 hit_total: 4261 miss_total: 780 swrong=2 ewrong=2 - open/close 684 (time 1787744500.28 total 10.01 last 68.36) total: 900 open/close in 13.24 seconds: 67.97 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 1000 real 0m0.256s user 0m0.002s sys 0m0.056s 'statx --cached=always -D' 1000 files without statahead: 1 sec llite.lustre-ffff93f6a0950800.statahead_max=128 llite.lustre-ffff93f6a0950800.statahead_max=128 1000 real 0m0.188s user 0m0.002s sys 0m0.033s 'statx --cached=always -D' 1000 files with statahead: 0 sec statahead total: 162 statahead wrong: 2 agl total: 155 list_total: 30 fname_total: 112 hit_total: 4261 miss_total: 780 swrong=2 ewrong=2 - open/close 628 (time 1787744523.26 total 10.00 last 62.77) - open/close 1320 (time 1787744533.28 total 20.02 last 69.10) - open/close 2055 (time 1787744543.29 total 30.03 last 73.44) - open/close 2655 (time 1787744553.29 total 40.03 last 59.99) - open/close 3353 (time 1787744563.29 total 50.04 last 69.74) - open/close 3989 (time 1787744573.30 total 60.04 last 63.56) - open/close 4580 (time 1787744583.31 total 70.06 last 59.02) - open/close 5172 (time 1787744593.32 total 80.06 last 59.17) - open/close 5810 (time 1787744603.32 total 90.06 last 63.77) - open/close 6422 (time 1787744613.35 total 100.09 last 61.06) - open/close 7041 (time 1787744623.35 total 110.09 last 61.88) - open/close 7733 (time 1787744633.36 total 120.10 last 69.16) - open/close 8304 (time 1787744643.36 total 130.10 last 57.06) - open/close 8916 (time 1787744653.37 total 140.12 last 61.12) total: 9000 open/close in 141.59 seconds: 63.56 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 10000 real 0m0.780s user 0m0.001s sys 0m0.120s 'statx --cached=always -D' 10000 files without statahead: 0 sec llite.lustre-ffff93f6a0950800.statahead_max=128 llite.lustre-ffff93f6a0950800.statahead_max=128 10000 real 0m0.809s user 0m0.006s sys 0m0.120s 'statx --cached=always -D' 10000 files with statahead: 1 sec statahead total: 162 statahead wrong: 2 agl total: 155 list_total: 30 fname_total: 112 hit_total: 4261 miss_total: 780 swrong=2 ewrong=2 'statx --cached=always -D' done unlink + rm -r /mnt/lustre/d123ac.sanity: 397 seconds llite.lustre-ffff93f6a0950800.statahead_stats= statahead total: 162 statahead wrong: 2 agl total: 155 list_total: 30 fname_total: 112 hit_total: 4261 miss_total: 780 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787745067.971210692 secs.nsecs start_time: 1787744473.229227241 secs.nsecs elapsed_time: 594.741983451 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787745067.971282978 secs.nsecs start_time: 1787744473.229310357 secs.nsecs elapsed_time: 594.741972621 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ac.sanity/f123ac.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ac.sanity/f123ac.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 PASS 123ac (705s) == sanity test 123ad: Verify batching statahead works correctly ========================================================== 07:51:23 (1787745083) batch: statahead_max=32 statahead_batch_max=32 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 llite.lustre-ffff93f6a0950800.statahead_max=32 llite.lustre-ffff93f6a0950800.statahead_batch_max=32 seq.cli-lustre-OST0000-super.width=0x1ffffff seq.cli-lustre-OST0001-super.width=0x1ffffff kvm mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 striped dir -i1 -c2 -H all_char /mnt/lustre/d123ad.sanity total: 100 open/close in 2.86 seconds: 34.99 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 101 real 0m2.715s user 0m0.005s sys 0m0.305s 'ls -l' 100 files without statahead: 2 sec llite.lustre-ffff93f6a0950800.statahead_max=32 llite.lustre-ffff93f6a0950800.statahead_max=32 101 real 0m1.322s user 0m0.004s sys 0m0.151s 'ls -l' 100 files with statahead: 2 sec statahead total: 163 statahead wrong: 2 agl total: 156 list_total: 31 fname_total: 112 hit_total: 4360 miss_total: 781 swrong=2 ewrong=2 - open/close 680 (time 1787745113.28 total 10.00 last 68.00) total: 900 open/close in 14.03 seconds: 64.14 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 1001 real 0m31.551s user 0m0.037s sys 0m2.862s 'ls -l' 1000 files without statahead: 32 sec llite.lustre-ffff93f6a0950800.statahead_max=32 llite.lustre-ffff93f6a0950800.statahead_max=32 1001 real 0m9.922s user 0m0.020s sys 0m1.393s 'ls -l' 1000 files with statahead: 10 sec statahead total: 164 statahead wrong: 2 agl total: 157 list_total: 32 fname_total: 112 hit_total: 5354 miss_total: 787 swrong=2 ewrong=2 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 21 seconds llite.lustre-ffff93f6a0950800.statahead_stats= statahead total: 164 statahead wrong: 2 agl total: 157 list_total: 32 fname_total: 112 hit_total: 5354 miss_total: 787 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787745195.372439914 secs.nsecs start_time: 1787745088.626704931 secs.nsecs elapsed_time: 106.745734983 secs.nsecs subreqs per batch batches %% cum %% 1: 1 2 2 2: 0 0 2 4: 3 7 10 8: 4 10 20 16: 20 50 70 32: 12 30 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787745195.372506433 secs.nsecs start_time: 1787745088.626844544 secs.nsecs elapsed_time: 106.745661889 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 3 7 7 8: 2 5 12 16: 21 53 66 32: 13 33 100 - open/close 759 (time 1787745207.87 total 10.00 last 75.90) - open/close 1484 (time 1787745217.88 total 20.01 last 72.45) - open/close 2199 (time 1787745227.89 total 30.02 last 71.40) - open/close 2823 (time 1787745237.92 total 40.05 last 62.20) - open/close 3425 (time 1787745247.93 total 50.06 last 60.15) - open/close 3875 (time 1787745257.94 total 60.07 last 44.98) - open/close 4359 (time 1787745267.96 total 70.08 last 48.31) - open/close 4815 (time 1787745277.96 total 80.09 last 45.59) - open/close 5371 (time 1787745288.00 total 90.13 last 55.38) - open/close 6009 (time 1787745298.00 total 100.13 last 63.76) - open/close 6603 (time 1787745308.01 total 110.13 last 59.38) - open/close 7098 (time 1787745318.04 total 120.17 last 49.35) - open/close 7518 (time 1787745328.04 total 130.17 last 41.99) - open/close 8017 (time 1787745338.04 total 140.17 last 49.89) - open/close 8620 (time 1787745348.05 total 150.18 last 60.26) - open/close 9187 (time 1787745358.06 total 160.19 last 56.65) - open/close 9849 (time 1787745368.06 total 170.19 last 66.18) total: 10000 open/close in 172.28 seconds: 58.05 ops/second llite.lustre-ffff93f6a0950800.statahead_batch_max=0 llite.lustre-ffff93f6a0950800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m22.127s user 0m0.319s sys 0m14.297s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff93f6a0950800.statahead_batch_max=32 llite.lustre-ffff93f6a0950800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.stats=clear ls -l files in d123ad.sanity: 10001 real 2m9.080s user 0m0.284s sys 0m13.983s batched statahead RPC= hit count=9971 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787745604.274694705 secs.nsecs start_time: 1787745463.247936796 secs.nsecs elapsed_time: 141.026757909 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 0 0 16: 2 0 0 32: 313 99 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787745604.276170873 secs.nsecs start_time: 1787745463.248038767 secs.nsecs elapsed_time: 141.028132106 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 batch: statahead_max=2048 statahead_batch_max=256 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 llite.lustre-ffff93f6a0950800.statahead_max=2048 llite.lustre-ffff93f6a0950800.statahead_batch_max=256 kvm mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=0 striped dir -i1 -c2 -H crush2 /mnt/lustre/d123ad.sanity total: 100 open/close in 4.04 seconds: 24.74 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 101 real 0m2.728s user 0m0.005s sys 0m0.320s 'ls -l' 100 files without statahead: 3 sec llite.lustre-ffff93f6a0950800.statahead_max=2048 llite.lustre-ffff93f6a0950800.statahead_max=2048 101 real 0m0.878s user 0m0.005s sys 0m0.158s 'ls -l' 100 files with statahead: 1 sec statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10070 miss_total: 30 swrong=0 ewrong=0 - open/close 788 (time 1787745999.89 total 10.01 last 78.74) total: 900 open/close in 11.80 seconds: 76.27 ops/second llite.lustre-ffff93f6a0950800.statahead_max=0 1001 real 0m25.168s user 0m0.032s sys 0m2.909s 'ls -l' 1000 files without statahead: 25 sec llite.lustre-ffff93f6a0950800.statahead_max=2048 llite.lustre-ffff93f6a0950800.statahead_max=2048 1001 real 0m7.135s user 0m0.042s sys 0m1.427s 'ls -l' 1000 files with statahead: 7 sec statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11069 miss_total: 31 swrong=0 ewrong=0 'ls -l' done unlink + rm -r /mnt/lustre/d123ad.sanity: 24 seconds llite.lustre-ffff93f6a0950800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 11069 miss_total: 31 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787746072.336635249 secs.nsecs start_time: 1787745607.404672935 secs.nsecs elapsed_time: 464.931962314 secs.nsecs subreqs per batch batches %% cum %% 1: 1 14 14 2: 0 0 14 4: 1 14 28 8: 2 28 57 16: 1 14 71 32: 0 0 71 64: 0 0 71 128: 0 0 71 256: 2 28 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787746072.336708717 secs.nsecs start_time: 1787745607.404842305 secs.nsecs elapsed_time: 464.931866412 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 2 22 22 8: 2 22 44 16: 1 11 55 32: 0 0 55 64: 1 11 66 128: 1 11 77 256: 2 22 100 - open/close 723 (time 1787746085.04 total 10.00 last 72.30) - open/close 1418 (time 1787746095.05 total 20.01 last 69.43) - open/close 2128 (time 1787746105.05 total 30.01 last 70.99) - open/close 2737 (time 1787746115.06 total 40.01 last 60.89) - open/close 3373 (time 1787746125.07 total 50.02 last 63.53) - open/close 4108 (time 1787746135.07 total 60.03 last 73.49) - open/close 4834 (time 1787746145.07 total 70.03 last 72.60) - open/close 5537 (time 1787746155.07 total 80.03 last 70.29) - open/close 6160 (time 1787746165.08 total 90.04 last 62.23) - open/close 6870 (time 1787746175.09 total 100.04 last 70.99) - open/close 7494 (time 1787746185.09 total 110.05 last 62.37) - open/close 8088 (time 1787746195.10 total 120.05 last 59.36) - open/close 8631 (time 1787746205.11 total 130.06 last 54.25) - open/close 9143 (time 1787746215.11 total 140.07 last 51.18) - open/close 9793 (time 1787746225.11 total 150.07 last 65.00) total: 10000 open/close in 152.92 seconds: 65.39 ops/second llite.lustre-ffff93f6a0950800.statahead_batch_max=0 llite.lustre-ffff93f6a0950800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.stats=clear ls -l files in d123ad.sanity: 10001 real 1m12.049s user 0m0.334s sys 0m14.554s unbatched statahead RPC=10004 hit count=9999 llite.lustre-ffff93f6a0950800.statahead_batch_max=256 llite.lustre-ffff93f6a0950800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats=clear mdc.lustre-MDT0000-mdc-ffff93f6a0950800.stats=clear mdc.lustre-MDT0001-mdc-ffff93f6a0950800.stats=clear ls -l files in d123ad.sanity: 10001 real 0m57.939s user 0m0.303s sys 0m14.403s batched statahead RPC= hit count=9999 mdc.lustre-MDT0000-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787746379.082422980 secs.nsecs start_time: 1787746310.687025182 secs.nsecs elapsed_time: 68.395397798 secs.nsecs subreqs per batch batches %% cum %% 1: 45 33 33 2: 11 8 42 4: 8 6 48 8: 7 5 53 16: 4 3 56 32: 14 10 66 64: 5 3 70 128: 4 3 73 256: 35 26 100 mdc.lustre-MDT0001-mdc-ffff93f6a0950800.batch_stats= snapshot_time: 1787746379.082488935 secs.nsecs start_time: 1787746310.687115959 secs.nsecs elapsed_time: 68.395372976 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second unlink(/mnt/lustre/d123ad.sanity/f123ad.sanity0) error: No such file or directory total: 0 unlinks in 0 seconds: -nan unlinks/second seq.cli-lustre-OST0000-super.width=65536 seq.cli-lustre-OST0001-super.width=65536 llite.lustre-ffff93f6a0950800.statahead_batch_max=64 llite.lustre-ffff93f6a0950800.statahead_max=128 PASS 123ad (1544s) == sanity test 123b: not panic with network error in statahead enqueue (bug 15027) ========================================================== 08:17:07 (1787746627) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d123b.sanity - open/close 761 (time 1787746642.21 total 10.00 last 76.09) total: 1000 open/close in 13.35 seconds: 74.89 ops/second fail_loc=0x80000803 ls done fail_loc=0x0 statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 10998 miss_total: 2 PASS 123b (59s) == sanity test 123c: Can not initialize inode warning on DNE statahead ========================================================== 08:18:07 (1787746687) striped dir -i0 -c1 -H crush2 /mnt/lustre/d123c.sanity.0 striped dir -i1 -c1 -H all_char /mnt/lustre/d123c.sanity.1 Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre PASS 123c (17s) == sanity test 123d: Statahead on striped directories works correctly ========================================================== 08:18:24 (1787746704) striped dir -i1 -c1 -H fnv_1a_64 /mnt/lustre/d123d.sanity total: 100 mkdir in 5.39 seconds: 18.56 ops/second Stopping client oleg315-client.virtnet /mnt/lustre (opts:) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre llite.lustre-ffff93f5a5e0f800.statahead_max=128 llite.lustre-ffff93f5a5e0f800.statahead_stats=0 total 800 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity0 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity1 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity10 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity11 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity12 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity13 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity14 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity15 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity16 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity17 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity18 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity19 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity2 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity20 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity21 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity22 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity23 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity24 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity25 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity26 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity27 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity28 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity29 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity3 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity30 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity31 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity32 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity33 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity34 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity35 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity36 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity37 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity38 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity39 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity4 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity40 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity41 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity42 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity43 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity44 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity45 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity46 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity47 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity48 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity49 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity5 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity50 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity51 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity52 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity53 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity54 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity55 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity56 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity57 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity58 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity59 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity6 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity60 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity61 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity62 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity63 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity64 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity65 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity66 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity67 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity68 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity69 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity7 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity70 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity71 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity72 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity73 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity74 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity75 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity76 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity77 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity78 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity79 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity8 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity80 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity81 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity82 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity83 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity84 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity85 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity86 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity87 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity88 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity89 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity9 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity90 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity91 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity92 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity93 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity94 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity95 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity96 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity97 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity98 drwxr-xr-x 2 root root 8192 Aug 26 08:18 f123d.sanity99 statahead total: 1 statahead wrong: 0 agl total: 1 list_total: 1 fname_total: 0 hit_total: 99 miss_total: 1 - unlinked 0 (time 1787746730 ; total 1 ; last 1) total: 100 unlinks in 8 seconds: 12.500000 unlinks/second PASS 123d (34s) == sanity test 123e: statahead with large wide striping == 08:18:57 (1787746737) llite.lustre-ffff93f5a5e0f800.statahead_max=2048 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=1024 total 0 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.0 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.1 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.10 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.100 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.1000 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.101 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.102 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.103 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.104 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.105 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.106 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.107 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.108 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.109 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.11 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.110 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.111 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.112 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.113 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.114 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.115 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.116 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.117 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.118 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.119 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.12 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.120 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.121 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.122 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.123 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.124 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.125 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.126 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.127 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.128 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.129 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.13 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.130 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.131 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.132 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.133 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.134 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.135 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.136 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.137 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.138 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.139 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.14 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.140 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.141 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.142 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.143 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.144 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.145 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.146 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.147 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.148 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.149 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.15 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.150 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.151 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.152 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.153 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.154 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.155 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.156 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.157 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.158 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.159 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.16 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.160 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.161 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.162 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.163 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.164 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.165 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.166 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.167 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.168 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.169 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.17 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.170 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.171 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.172 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.173 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.174 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.175 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.176 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.177 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.178 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.179 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.18 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.180 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.181 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.182 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.183 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.184 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.185 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.186 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.187 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.188 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.189 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.19 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.190 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.191 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.192 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.193 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.194 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.195 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.196 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.197 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.198 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.199 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.2 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.20 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.200 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.201 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.202 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.203 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.204 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.205 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.206 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.207 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.208 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.209 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.21 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.210 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.211 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.212 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.213 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.214 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.215 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.216 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.217 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.218 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.219 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.22 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.220 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.221 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.222 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.223 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.224 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.225 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.226 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.227 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.228 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.229 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.23 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.230 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.231 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.232 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.233 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.234 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.235 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.236 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.237 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.238 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.239 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.24 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.240 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.241 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.242 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.243 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.244 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.245 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.246 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.247 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.248 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.249 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.25 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.250 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.251 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.252 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.253 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.254 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.255 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.256 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.257 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.258 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.259 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.26 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.260 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.261 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.262 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.263 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.264 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.265 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.266 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.267 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.268 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.269 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.27 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.270 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.271 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.272 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.273 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.274 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.275 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.276 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.277 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.278 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.279 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.28 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.280 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.281 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.282 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.283 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.284 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.285 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.286 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.287 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.288 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.289 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.29 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.290 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.291 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.292 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.293 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.294 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.295 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.296 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.297 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.298 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.299 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.3 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.30 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.300 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.301 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.302 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.303 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.304 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.305 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.306 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.307 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.308 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.309 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.31 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.310 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.311 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.312 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.313 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.314 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.315 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.316 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.317 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.318 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.319 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.32 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.320 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.321 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.322 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.323 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.324 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.325 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.326 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.327 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.328 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.329 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.33 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.330 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.331 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.332 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.333 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.334 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.335 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.336 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.337 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.338 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.339 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.34 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.340 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.341 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.342 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.343 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.344 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.345 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.346 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.347 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.348 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.349 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.35 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.350 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.351 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.352 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.353 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.354 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.355 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.356 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.357 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.358 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.359 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.36 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.360 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.361 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.362 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.363 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.364 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.365 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.366 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.367 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.368 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.369 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.37 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.370 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.371 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.372 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.373 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.374 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.375 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.376 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.377 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.378 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.379 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.38 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.380 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.381 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.382 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.383 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.384 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.385 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.386 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.387 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.388 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.389 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.39 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.390 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.391 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.392 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.393 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.394 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.395 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.396 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.397 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.398 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.399 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.4 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.40 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.400 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.401 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.402 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.403 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.404 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.405 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.406 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.407 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.408 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.409 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.41 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.410 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.411 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.412 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.413 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.414 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.415 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.416 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.417 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.418 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.419 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.42 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.420 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.421 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.422 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.423 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.424 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.425 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.426 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.427 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.428 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.429 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.43 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.430 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.431 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.432 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.433 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.434 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.435 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.436 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.437 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.438 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.439 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.44 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.440 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.441 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.442 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.443 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.444 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.445 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.446 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.447 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.448 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.449 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.45 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.450 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.451 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.452 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.453 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.454 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.455 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.456 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.457 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.458 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.459 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.46 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.460 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.461 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.462 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.463 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.464 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.465 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.466 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.467 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.468 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.469 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.47 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.470 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.471 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.472 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.473 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.474 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.475 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.476 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.477 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.478 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.479 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.48 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.480 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.481 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.482 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.483 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.484 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.485 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.486 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.487 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.488 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.489 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.49 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.490 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.491 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.492 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.493 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.494 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.495 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.496 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.497 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.498 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.499 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.5 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.50 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.500 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.501 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.502 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.503 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.504 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.505 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.506 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.507 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.508 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.509 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.51 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.510 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.511 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.512 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.513 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.514 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.515 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.516 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.517 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.518 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.519 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.52 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.520 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.521 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.522 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.523 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.524 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.525 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.526 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.527 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.528 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.529 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.53 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.530 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.531 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.532 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.533 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.534 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.535 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.536 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.537 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.538 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.539 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.54 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.540 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.541 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.542 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.543 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.544 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.545 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.546 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.547 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.548 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.549 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.55 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.550 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.551 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.552 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.553 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.554 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.555 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.556 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.557 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.558 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.559 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.56 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.560 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.561 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.562 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.563 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.564 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.565 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.566 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.567 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.568 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.569 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.57 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.570 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.571 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.572 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.573 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.574 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.575 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.576 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.577 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.578 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.579 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.58 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.580 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.581 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.582 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.583 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.584 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.585 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.586 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.587 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.588 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.589 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.59 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.590 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.591 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.592 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.593 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.594 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.595 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.596 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.597 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.598 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.599 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.6 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.60 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.600 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.601 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.602 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.603 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.604 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.605 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.606 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.607 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.608 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.609 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.61 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.610 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.611 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.612 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.613 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.614 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.615 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.616 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.617 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.618 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.619 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.62 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.620 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.621 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.622 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.623 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.624 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.625 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.626 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.627 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.628 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.629 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.63 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.630 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.631 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.632 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.633 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.634 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.635 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.636 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.637 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.638 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.639 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.64 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.640 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.641 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.642 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.643 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.644 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.645 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.646 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.647 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.648 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.649 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.65 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.650 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.651 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.652 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.653 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.654 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.655 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.656 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.657 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.658 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.659 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.66 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.660 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.661 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.662 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.663 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.664 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.665 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.666 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.667 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.668 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.669 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.67 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.670 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.671 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.672 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.673 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.674 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.675 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.676 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.677 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.678 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.679 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.68 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.680 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.681 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.682 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.683 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.684 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.685 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.686 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.687 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.688 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.689 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.69 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.690 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.691 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.692 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.693 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.694 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.695 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.696 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.697 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.698 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.699 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.7 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.70 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.700 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.701 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.702 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.703 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.704 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.705 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.706 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.707 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.708 -rw-r--r-- 1 root root 0 Aug 26 08:20 f123e.sanity.709 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.71 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.710 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.711 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.712 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.713 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.714 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.715 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.716 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.717 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.718 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.719 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.72 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.720 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.721 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.722 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.723 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.724 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.725 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.726 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.727 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.728 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.729 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.73 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.730 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.731 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.732 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.733 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.734 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.735 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.736 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.737 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.738 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.739 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.74 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.740 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.741 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.742 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.743 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.744 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.745 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.746 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.747 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.748 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.749 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.75 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.750 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.751 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.752 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.753 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.754 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.755 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.756 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.757 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.758 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.759 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.76 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.760 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.761 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.762 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.763 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.764 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.765 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.766 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.767 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.768 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.769 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.77 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.770 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.771 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.772 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.773 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.774 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.775 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.776 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.777 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.778 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.779 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.78 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.780 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.781 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.782 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.783 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.784 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.785 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.786 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.787 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.788 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.789 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.79 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.790 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.791 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.792 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.793 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.794 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.795 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.796 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.797 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.798 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.799 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.8 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.80 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.800 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.801 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.802 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.803 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.804 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.805 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.806 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.807 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.808 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.809 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.81 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.810 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.811 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.812 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.813 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.814 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.815 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.816 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.817 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.818 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.819 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.82 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.820 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.821 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.822 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.823 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.824 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.825 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.826 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.827 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.828 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.829 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.83 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.830 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.831 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.832 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.833 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.834 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.835 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.836 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.837 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.838 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.839 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.84 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.840 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.841 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.842 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.843 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.844 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.845 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.846 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.847 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.848 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.849 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.85 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.850 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.851 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.852 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.853 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.854 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.855 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.856 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.857 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.858 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.859 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.86 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.860 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.861 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.862 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.863 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.864 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.865 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.866 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.867 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.868 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.869 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.87 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.870 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.871 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.872 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.873 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.874 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.875 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.876 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.877 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.878 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.879 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.88 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.880 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.881 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.882 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.883 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.884 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.885 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.886 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.887 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.888 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.889 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.89 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.890 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.891 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.892 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.893 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.894 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.895 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.896 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.897 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.898 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.899 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.9 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.90 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.900 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.901 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.902 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.903 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.904 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.905 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.906 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.907 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.908 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.909 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.91 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.910 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.911 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.912 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.913 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.914 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.915 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.916 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.917 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.918 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.919 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.92 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.920 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.921 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.922 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.923 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.924 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.925 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.926 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.927 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.928 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.929 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.93 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.930 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.931 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.932 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.933 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.934 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.935 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.936 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.937 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.938 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.939 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.94 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.940 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.941 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.942 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.943 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.944 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.945 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.946 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.947 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.948 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.949 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.95 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.950 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.951 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.952 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.953 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.954 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.955 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.956 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.957 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.958 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.959 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.96 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.960 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.961 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.962 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.963 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.964 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.965 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.966 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.967 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.968 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.969 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.97 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.970 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.971 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.972 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.973 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.974 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.975 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.976 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.977 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.978 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.979 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.98 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.980 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.981 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.982 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.983 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.984 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.985 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.986 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.987 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.988 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.989 -rw-r--r-- 1 root root 0 Aug 26 08:19 f123e.sanity.99 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.990 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.991 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.992 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.993 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.994 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.995 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.996 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.997 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.998 -rw-r--r-- 1 root root 0 Aug 26 08:21 f123e.sanity.999 mdc.lustre-MDT0000-mdc-ffff93f5a5e0f800.batch_stats= snapshot_time: 1787746967.027537282 secs.nsecs start_time: 0.000000000 secs.nsecs elapsed_time: 1787746967.027537282 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 20 20 16: 1 20 40 32: 1 20 60 64: 0 0 60 128: 1 20 80 256: 0 0 80 512: 0 0 80 1024: 1 20 100 mdc.lustre-MDT0001-mdc-ffff93f5a5e0f800.batch_stats= snapshot_time: 1787746967.027615709 secs.nsecs start_time: 0.000000000 secs.nsecs elapsed_time: 1787746967.027615709 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 20 20 16: 1 20 40 32: 2 40 80 64: 1 20 100 llite.lustre-ffff93f5a5e0f800.statahead_agl=1 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=1024 llite.lustre-ffff93f5a5e0f800.statahead_fname_match_hit=4 llite.lustre-ffff93f5a5e0f800.statahead_fname_predict_hit=2 llite.lustre-ffff93f5a5e0f800.statahead_max=2048 llite.lustre-ffff93f5a5e0f800.statahead_min=8 llite.lustre-ffff93f5a5e0f800.statahead_running_max=16 llite.lustre-ffff93f5a5e0f800.statahead_timeout=30 llite.lustre-ffff93f5a5e0f800.statahead_stats= statahead total: 2 statahead wrong: 0 agl total: 2 list_total: 2 fname_total: 0 hit_total: 1099 miss_total: 2 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=64 llite.lustre-ffff93f5a5e0f800.statahead_max=128 PASS 123e (362s) == sanity test 123f: Retry mechanism with large wide striping files ========================================================== 08:24:59 (1787747099) llite.lustre-ffff93f5a5e0f800.statahead_max=64 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=64 total 0 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.0 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.1 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.10 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.100 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.101 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.102 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.103 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.104 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.105 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.106 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.107 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.108 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.109 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.11 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.110 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.111 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.112 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.113 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.114 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.115 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.116 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.117 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.118 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.119 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.12 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.120 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.121 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.122 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.123 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.124 -rw-r--r-- 1 root root 0 Aug 26 08:31 f123f.sanity.125 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.126 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.127 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.128 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.129 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.13 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.130 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.131 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.132 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.133 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.134 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.135 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.136 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.137 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.138 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.139 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.14 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.140 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.141 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.142 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.143 -rw-r--r-- 1 root root 0 Aug 26 08:32 f123f.sanity.144 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.145 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.146 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.147 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.148 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.149 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.15 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.150 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.151 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.152 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.153 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.154 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.155 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.156 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.157 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.158 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.159 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.16 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.160 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.161 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.162 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.163 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.164 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.165 -rw-r--r-- 1 root root 0 Aug 26 08:33 f123f.sanity.166 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.167 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.168 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.169 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.17 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.170 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.171 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.172 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.173 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.174 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.175 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.176 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.177 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.178 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.179 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.18 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.180 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.181 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.182 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.183 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.184 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.185 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.186 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.187 -rw-r--r-- 1 root root 0 Aug 26 08:34 f123f.sanity.188 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.189 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.19 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.190 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.191 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.192 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.193 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.194 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.195 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.196 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.197 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.198 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.199 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.2 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.20 -rw-r--r-- 1 root root 0 Aug 26 08:35 f123f.sanity.200 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.21 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.22 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.23 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.24 -rw-r--r-- 1 root root 0 Aug 26 08:26 f123f.sanity.25 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.26 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.27 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.28 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.29 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.3 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.30 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.31 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.32 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.33 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.34 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.35 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.36 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.37 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.38 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.39 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.4 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.40 -rw-r--r-- 1 root root 0 Aug 26 08:27 f123f.sanity.41 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.42 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.43 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.44 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.45 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.46 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.47 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.48 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.49 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.5 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.50 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.51 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.52 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.53 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.54 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.55 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.56 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.57 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.58 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.59 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.6 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.60 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.61 -rw-r--r-- 1 root root 0 Aug 26 08:28 f123f.sanity.62 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.63 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.64 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.65 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.66 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.67 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.68 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.69 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.7 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.70 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.71 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.72 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.73 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.74 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.75 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.76 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.77 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.78 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.79 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.8 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.80 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.81 -rw-r--r-- 1 root root 0 Aug 26 08:29 f123f.sanity.82 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.83 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.84 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.85 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.86 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.87 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.88 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.89 -rw-r--r-- 1 root root 0 Aug 26 08:25 f123f.sanity.9 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.90 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.91 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.92 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.93 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.94 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.95 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.96 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.97 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.98 -rw-r--r-- 1 root root 0 Aug 26 08:30 f123f.sanity.99 mdc.lustre-MDT0000-mdc-ffff93f5a5e0f800.batch_stats= snapshot_time: 1787748165.959605494 secs.nsecs start_time: 0.000000000 secs.nsecs elapsed_time: 1787748165.959605494 secs.nsecs subreqs per batch batches %% cum %% 1: 47 70 70 2: 11 16 86 4: 0 0 86 8: 2 2 89 16: 3 4 94 32: 1 1 95 64: 1 1 97 128: 1 1 98 256: 0 0 98 512: 0 0 98 1024: 1 1 100 mdc.lustre-MDT0001-mdc-ffff93f5a5e0f800.batch_stats= snapshot_time: 1787748165.960424614 secs.nsecs start_time: 0.000000000 secs.nsecs elapsed_time: 1787748165.960424614 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 2: 0 0 0 4: 0 0 0 8: 1 20 20 16: 1 20 40 32: 2 40 80 64: 1 20 100 llite.lustre-ffff93f5a5e0f800.statahead_agl=1 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=64 llite.lustre-ffff93f5a5e0f800.statahead_fname_match_hit=4 llite.lustre-ffff93f5a5e0f800.statahead_fname_predict_hit=2 llite.lustre-ffff93f5a5e0f800.statahead_max=64 llite.lustre-ffff93f5a5e0f800.statahead_min=8 llite.lustre-ffff93f5a5e0f800.statahead_running_max=16 llite.lustre-ffff93f5a5e0f800.statahead_timeout=30 llite.lustre-ffff93f5a5e0f800.statahead_stats= statahead total: 3 statahead wrong: 0 agl total: 3 list_total: 3 fname_total: 0 hit_total: 1197 miss_total: 3 llite.lustre-ffff93f5a5e0f800.statahead_max=128 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=64 PASS 123f (1388s) == sanity test 123g: Test for stat-ahead advise ========== 08:48:07 (1787748487) - open/close 265 (time 1787748511.63 total 10.02 last 26.45) - open/close 491 (time 1787748521.65 total 20.04 last 22.54) - open/close 699 (time 1787748531.72 total 30.11 last 20.67) - open/close 878 (time 1787748541.74 total 40.13 last 17.85) total: 1000 open/close in 46.57 seconds: 21.47 ops/second llite.lustre-ffff93f5a5e0f800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f5a5e0f800.batch_stats=clear mdc.lustre-MDT0001-mdc-ffff93f5a5e0f800.batch_stats=clear statahead total: 1 statahead wrong: 0 agl total: 1 list_total: 0 fname_total: 0 hit_total: 1000 miss_total: 0 snapshot_time: 1787748581.589348895 secs.nsecs start_time: 1787748568.856331803 secs.nsecs elapsed_time: 12.733017092 secs.nsecs subreqs per batch batches %% cum %% 1: 3 11 11 2: 2 7 18 4: 1 3 22 8: 4 14 37 16: 2 7 44 32: 0 0 44 64: 15 55 100 snapshot_time: 1787748581.589577026 secs.nsecs start_time: 1787748568.856489526 secs.nsecs elapsed_time: 12.733087500 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 Hit total: 1000 PASS 123g (107s) == sanity test 123h: Verify statahead work with the fname pattern via du ========================================================== 08:49:54 (1787748594) llite.lustre-ffff93f5a5e0f800.enable_statahead_fname=1 Scan a directory with number regularized fname llite.lustre-ffff93f5a5e0f800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f5a5e0f800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f5a5e0f800.batch_stats=0 llite.lustre-ffff93f5a5e0f800.statahead_max=1024 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=1024 statahead total: 0 statahead wrong: 0 agl total: 0 list_total: 0 fname_total: 0 hit_total: 0 miss_total: 0 Wait statahead thread (ll_sa_xxx) to exit... Waiting 35s for '' statahead total: 1 statahead wrong: 0 agl total: 1 list_total: 0 fname_total: 1 hit_total: 9992 miss_total: 1 snapshot_time: 1787749195.254660054 secs.nsecs start_time: 1787749121.407239868 secs.nsecs elapsed_time: 73.847420186 secs.nsecs subreqs per batch batches %% cum %% 1: 1 7 7 2: 0 0 7 4: 0 0 7 8: 1 7 14 16: 1 7 21 32: 1 7 28 64: 0 0 28 128: 0 0 28 256: 0 0 28 512: 0 0 28 1024: 10 71 100 snapshot_time: 1787749195.254733595 secs.nsecs start_time: 1787749121.407655576 secs.nsecs elapsed_time: 73.847078019 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 Scan a directory with zeroed padding number regularized fname llite.lustre-ffff93f5a5e0f800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f5a5e0f800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f5a5e0f800.batch_stats=0 llite.lustre-ffff93f5a5e0f800.statahead_max=1024 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=1024 statahead total: 0 statahead wrong: 0 agl total: 0 list_total: 0 fname_total: 0 hit_total: 0 miss_total: 0 Wait statahead thread (ll_sa_xxx) to exit... Waiting 35s for '' statahead total: 1 statahead wrong: 0 agl total: 1 list_total: 0 fname_total: 1 hit_total: 9990 miss_total: 3 snapshot_time: 1787749986.415897989 secs.nsecs start_time: 1787749912.571744744 secs.nsecs elapsed_time: 73.844153245 secs.nsecs subreqs per batch batches %% cum %% 1: 1 7 7 2: 0 0 7 4: 0 0 7 8: 1 7 15 16: 1 7 23 32: 0 0 23 64: 0 0 23 128: 0 0 23 256: 0 0 23 512: 0 0 23 1024: 10 76 100 snapshot_time: 1787749986.415961893 secs.nsecs start_time: 1787749912.572353494 secs.nsecs elapsed_time: 73.843608399 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 llite.lustre-ffff93f5a5e0f800.enable_statahead_fname=1 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=64 llite.lustre-ffff93f5a5e0f800.statahead_max=128 PASS 123h (1591s) == sanity test 123i: Verify statahead work with the fname indexing pattern ========================================================== 09:16:25 (1787750185) llite.lustre-ffff93f5a5e0f800.statahead_max=1024 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=32 llite.lustre-ffff93f5a5e0f800.statahead_min=64 llite.lustre-ffff93f5a5e0f800.enable_statahead_fname=1 Command: - createmany -m /mnt/lustre/d123i.sanity/f123i.sanity.%06d 1000 - ls /mnt/lustre/d123i.sanity/* > /dev/null total: 1000 create in 5.66 seconds: 176.56 ops/second llite.lustre-ffff93f5a5e0f800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f5a5e0f800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f5a5e0f800.batch_stats=0 statahead_stats (Pre): statahead total: 0 statahead wrong: 0 agl total: 0 list_total: 0 fname_total: 0 hit_total: 0 miss_total: 0 statahead_stats (Post): statahead total: 1 statahead wrong: 0 agl total: 0 list_total: 0 fname_total: 1 hit_total: 0 miss_total: 0 snapshot_time: 1787750201.557411660 secs.nsecs start_time: 1787750197.055431427 secs.nsecs elapsed_time: 4.501980233 secs.nsecs subreqs per batch batches %% cum %% 1: 2 2 2 2: 3 3 6 4: 8 9 15 8: 4 4 20 16: 6 7 28 32: 59 71 100 snapshot_time: 1787750201.557477624 secs.nsecs start_time: 1787750197.055507722 secs.nsecs elapsed_time: 4.501969902 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 Wait the statahead thread (ll_sa_xxx) to exit ... Waiting 35s for '' Waiting 15s for '' statahead total: 1 statahead wrong: 0 agl total: 0 list_total: 0 fname_total: 1 hit_total: 994 miss_total: 1 snapshot_time: 1787750231.958039328 secs.nsecs start_time: 1787750197.055431427 secs.nsecs elapsed_time: 34.902607901 secs.nsecs subreqs per batch batches %% cum %% 1: 3 3 3 2: 3 3 7 4: 8 9 16 8: 4 4 21 16: 6 7 28 32: 59 71 100 snapshot_time: 1787750231.958096748 secs.nsecs start_time: 1787750197.055507722 secs.nsecs elapsed_time: 34.902589026 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 Command: - createmany -m /mnt/lustre/d123i.sanity/f123i.sanity 1000 - aheadmany -c stat -N -s 0 -e 1000 -b f123i.sanity -d /mnt/lustre/d123i.sanity total: 1000 create in 6.43 seconds: 155.48 ops/second llite.lustre-ffff93f5a5e0f800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f5a5e0f800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f5a5e0f800.batch_stats=0 statahead_stats (Pre): statahead total: 0 statahead wrong: 0 agl total: 0 list_total: 0 fname_total: 0 hit_total: 0 miss_total: 0 statahead_stats (Post): statahead total: 1 statahead wrong: 0 agl total: 1 list_total: 0 fname_total: 1 hit_total: 0 miss_total: 0 snapshot_time: 1787750247.539653455 secs.nsecs start_time: 1787750243.825677952 secs.nsecs elapsed_time: 3.713975503 secs.nsecs subreqs per batch batches %% cum %% 1: 5 5 5 2: 3 3 9 4: 5 5 15 8: 8 9 24 16: 4 4 29 32: 60 70 100 snapshot_time: 1787750247.539715223 secs.nsecs start_time: 1787750243.826016997 secs.nsecs elapsed_time: 3.713698226 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 Wait the statahead thread (ll_sa_xxx) to exit ... Waiting 35s for '' Waiting 15s for '' Waiting 5s for '' statahead total: 1 statahead wrong: 0 agl total: 1 list_total: 0 fname_total: 1 hit_total: 994 miss_total: 1 snapshot_time: 1787750278.396503389 secs.nsecs start_time: 1787750243.825677952 secs.nsecs elapsed_time: 34.570825437 secs.nsecs subreqs per batch batches %% cum %% 1: 6 6 6 2: 3 3 10 4: 5 5 16 8: 8 9 25 16: 4 4 30 32: 60 69 100 snapshot_time: 1787750278.396570084 secs.nsecs start_time: 1787750243.826016997 secs.nsecs elapsed_time: 34.570553087 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 - unlinked 0 (time 1787750289 ; total 0 ; last 0) total: 1000 unlinks in 8 seconds: 125.000000 unlinks/second - unlinked 0 (time 1787750302 ; total 1 ; last 1) total: 1000 unlinks in 11 seconds: 90.909088 unlinks/second llite.lustre-ffff93f5a5e0f800.enable_statahead_fname=1 llite.lustre-ffff93f5a5e0f800.statahead_min=8 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=64 llite.lustre-ffff93f5a5e0f800.statahead_max=128 PASS 123i (129s) == sanity test 123j: -ENOENT error from batched statahead be handled correctly ========================================================== 09:18:35 (1787750315) llite.lustre-ffff93f5a5e0f800.enable_statahead_fname=1 File: /mnt/lustre/d123j.sanity.1 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205440298645 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 09:18:37.000000000 -0400 Modify: 2026-08-26 09:18:37.000000000 -0400 Change: 2026-08-26 09:18:37.000000000 -0400 Birth: 2026-08-26 09:18:36.000000000 -0400 File: /mnt/lustre/d123j.sanity.2 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205440298646 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 09:18:37.000000000 -0400 Modify: 2026-08-26 09:18:37.000000000 -0400 Change: 2026-08-26 09:18:37.000000000 -0400 Birth: 2026-08-26 09:18:36.000000000 -0400 File: /mnt/lustre/d123j.sanity.3 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205440298647 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 09:18:37.000000000 -0400 Modify: 2026-08-26 09:18:37.000000000 -0400 Change: 2026-08-26 09:18:37.000000000 -0400 Birth: 2026-08-26 09:18:36.000000000 -0400 File: /mnt/lustre/d123j.sanity.4 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205440298648 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 09:18:37.000000000 -0400 Modify: 2026-08-26 09:18:37.000000000 -0400 Change: 2026-08-26 09:18:37.000000000 -0400 Birth: 2026-08-26 09:18:36.000000000 -0400 File: /mnt/lustre/d123j.sanity.5 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205440298649 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 09:18:37.000000000 -0400 Modify: 2026-08-26 09:18:37.000000000 -0400 Change: 2026-08-26 09:18:37.000000000 -0400 Birth: 2026-08-26 09:18:36.000000000 -0400 File: /mnt/lustre/d123j.sanity.6 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205440298650 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 09:18:37.000000000 -0400 Modify: 2026-08-26 09:18:37.000000000 -0400 Change: 2026-08-26 09:18:37.000000000 -0400 Birth: 2026-08-26 09:18:36.000000000 -0400 File: /mnt/lustre/d123j.sanity.7 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205440298651 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 09:18:37.000000000 -0400 Modify: 2026-08-26 09:18:37.000000000 -0400 Change: 2026-08-26 09:18:37.000000000 -0400 Birth: 2026-08-26 09:18:36.000000000 -0400 File: /mnt/lustre/d123j.sanity.8 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205440298652 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 09:18:37.000000000 -0400 Modify: 2026-08-26 09:18:37.000000000 -0400 Change: 2026-08-26 09:18:37.000000000 -0400 Birth: 2026-08-26 09:18:36.000000000 -0400 File: /mnt/lustre/d123j.sanity.9 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205440298653 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 09:18:37.000000000 -0400 Modify: 2026-08-26 09:18:37.000000000 -0400 Change: 2026-08-26 09:18:37.000000000 -0400 Birth: 2026-08-26 09:18:36.000000000 -0400 File: /mnt/lustre/d123j.sanity.10 Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205440298654 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 09:18:37.000000000 -0400 Modify: 2026-08-26 09:18:37.000000000 -0400 Change: 2026-08-26 09:18:37.000000000 -0400 Birth: 2026-08-26 09:18:36.000000000 -0400 stat: cannot statx '/mnt/lustre/d123j.sanity.11': No such file or directory llite.lustre-ffff93f5a5e0f800.enable_statahead_fname=1 PASS 123j (10s) == sanity test 123k: Verify statahead work with mdtest shared stat() mode ========================================================== 09:18:45 (1787750325) SKIP: sanity test_123k mdtest not found SKIP 123k (4s) == sanity test 123l: Avoid panic when revalidate a local cached entry ========================================================== 09:18:49 (1787750329) llite.lustre-ffff93f5a5e0f800.statahead_stats=clear mdc.lustre-MDT0000-mdc-ffff93f5a5e0f800.batch_stats=0 mdc.lustre-MDT0001-mdc-ffff93f5a5e0f800.batch_stats=0 llite.lustre-ffff93f5a5e0f800.enable_statahead_fname=1 llite.lustre-ffff93f5a5e0f800.statahead_max=1024 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=32 statahead total: 0 statahead wrong: 0 agl total: 0 list_total: 0 fname_total: 0 hit_total: 0 miss_total: 0 fail_loc=0x80001433 fail_val=35 statahead total: 2 statahead wrong: 0 agl total: 0 list_total: 0 fname_total: 2 hit_total: 5 miss_total: 3 snapshot_time: 1787750373.744458255 secs.nsecs start_time: 1787750337.366667767 secs.nsecs elapsed_time: 36.377790488 secs.nsecs subreqs per batch batches %% cum %% 1: 2 5 5 2: 0 0 5 4: 0 0 5 8: 2 5 11 16: 1 2 14 32: 30 85 100 snapshot_time: 1787750373.744545061 secs.nsecs start_time: 1787750337.366838360 secs.nsecs elapsed_time: 36.377706701 secs.nsecs subreqs per batch batches %% cum %% 1: 0 0 0 Sleep to wait the statahead thread (ll_sa_xxx) to exit ... Waiting 35s for '' Waiting 15s for '' Waiting 5s for '' statahead total: 2 statahead wrong: 0 agl total: 0 list_total: 0 fname_total: 2 hit_total: 92 miss_total: 3 llite.lustre-ffff93f5a5e0f800.enable_statahead_fname=1 llite.lustre-ffff93f5a5e0f800.statahead_batch_max=64 llite.lustre-ffff93f5a5e0f800.statahead_max=128 PASS 123l (83s) == sanity test 124a: lru resize ================================================================================================= 09:20:12 (1787750412) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d124a.sanity create 2000 files at /mnt/lustre/d124a.sanity - open/close 871 (time 1787750428.21 total 10.01 last 87.05) - open/close 1646 (time 1787750438.22 total 20.01 last 77.45) total: 2000 open/close in 24.26 seconds: 82.43 ops/second NSDIR=ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800 NS=ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800 LRU=1004 LIMIT=46162 LVF=5517300 OLD_LVF=100 Sleep 50 sec ...1004...1004...1004...792...792...654...654...557...557...470 Dropped 534 locks in 50s unlink 2000 files at /mnt/lustre/d124a.sanity - unlinked 0 (time 1787750526 ; total 0 ; last 0) total: 2000 unlinks in 21 seconds: 95.238098 unlinks/second PASS 124a (146s) == sanity test 124b: lru resize (performance test) ================================================================================= 09:22:39 (1787750559) ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s striped dir -i0 -c2 -H crush /mnt/lustre/d124b.sanity/disable_lru_resize - open/close 715 (time 1787750574.66 total 10.01 last 71.46) - open/close 1500 (time 1787750584.67 total 20.01 last 78.45) - open/close 2252 (time 1787750594.68 total 30.02 last 75.11) - open/close 3005 (time 1787750604.69 total 40.03 last 75.24) - open/close 3720 (time 1787750614.69 total 50.03 last 71.48) - open/close 4465 (time 1787750624.69 total 60.04 last 74.47) - open/close 5166 (time 1787750634.69 total 70.04 last 70.09) - open/close 5815 (time 1787750644.70 total 80.04 last 64.88) - open/close 6412 (time 1787750654.70 total 90.05 last 59.68) - open/close 7111 (time 1787750664.71 total 100.05 last 69.86) - open/close 7895 (time 1787750674.72 total 110.07 last 78.28) total: 8000 open/close in 111.45 seconds: 71.78 ops/second doing ls -la /mnt/lustre/d124b.sanity/disable_lru_resize 3 times ls -la time: 163 seconds lru_size = 400 400 - unlinked 0 (time 1787750850 ; total 0 ; last 0) total: 8000 unlinks in 108 seconds: 74.074074 unlinks/second striped dir -i0 -c2 -H crush /mnt/lustre/d124b.sanity/enable_lru_resize - open/close 586 (time 1787750975.25 total 10.00 last 58.59) - open/close 1208 (time 1787750985.27 total 20.02 last 62.11) - open/close 1887 (time 1787750995.27 total 30.02 last 67.87) - open/close 2652 (time 1787751005.28 total 40.03 last 76.43) - open/close 3462 (time 1787751015.29 total 50.04 last 80.90) - open/close 4177 (time 1787751025.29 total 60.04 last 71.49) - open/close 4968 (time 1787751035.30 total 70.05 last 79.03) - open/close 5718 (time 1787751045.31 total 80.06 last 74.96) - open/close 6327 (time 1787751055.31 total 90.06 last 60.88) - open/close 6925 (time 1787751065.31 total 100.06 last 59.79) - open/close 7550 (time 1787751075.32 total 110.07 last 62.47) total: 8000 open/close in 118.08 seconds: 67.75 ops/second doing ls -la /mnt/lustre/d124b.sanity/enable_lru_resize 3 times ls -la time: 92 seconds lru_size = 4052 3955 ls -la is 43% faster with lru resize enabled - unlinked 0 (time 1787751196 ; total 0 ; last 0) total: 8000 unlinks in 62 seconds: 129.032257 unlinks/second PASS 124b (702s) == sanity test 124c: LRUR cancel very aged locks ========= 09:34:21 (1787751261) striped dir -i0 -c2 -H all_char /mnt/lustre/d124c.sanity total: 100 open/close in 3.19 seconds: 31.37 ops/second unused=55, max_age=600000, recalc_p=10 ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800.lru_max_age=1000 sleep 20 seconds... - unlinked 0 (time 1787751288 ; total 0 ; last 0) total: 100 unlinks in 3 seconds: 33.333332 unlinks/second PASS 124c (39s) == sanity test 124d: cancel very aged locks if lru-resize disabled ========================================================== 09:35:01 (1787751301) ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d124d.sanity total: 100 open/close in 3.52 seconds: 28.42 ops/second unused=55, max_age=3900000, recalc_p=10 ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800.lru_max_age=1000 sleep 20 seconds... - unlinked 0 (time 1787751334 ; total 0 ; last 0) total: 100 unlinks in 2 seconds: 50.000000 unlinks/second PASS 124d (35s) == sanity test 124e: LFRU keep priv locks from eviction == 09:35:36 (1787751336) max_unused=400 ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800.lru_priv_score_threshold=1 - open/close 760 (time 1787751352.31 total 10.00 last 75.99) total: 800 open/close in 10.65 seconds: 75.15 ops/second - unlinked 0 (time 1787751384 ; total 0 ; last 0) total: 800 unlinks in 9 seconds: 88.888885 unlinks/second ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800.lru_priv_score_threshold=1 PASS 124e (58s) == sanity test 124f: LFRU priv threshold inc/dec adjustment ========================================================== 09:36:34 (1787751394) max_unused=400 ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s - open/close 862 (time 1787751410.37 total 10.01 last 86.09) total: 1600 open/close in 19.72 seconds: 81.16 ops/second ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800.lru_priv_score_threshold=7 - unlinked 0 (time 1787751467 ; total 0 ; last 0) total: 1600 unlinks in 19 seconds: 84.210526 unlinks/second ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800.lru_priv_score_threshold=1 PASS 124f (95s) == sanity test 124g: LFRU performance test =============== 09:38:09 (1787751489) ldlm.namespaces.*mdc*.lru_size=0->400 ldlm.namespaces.*mdc*.lru_max_age=600000->3900s llite.lustre-ffff93f5a5e0f800.enable_statahead_fname=0 ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800.lock_cache_policy=LFRU total: 400 open/close in 4.29 seconds: 93.24 ops/second total: 100 open/close in 2.97 seconds: 33.63 ops/second ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800.lock_unused_count=0 mdt.lustre-MDT0000.exports.192.168.203.15@tcp.ldlm_stats= snapshot_time 1787751509.323854791 secs.nsecs start_time 1787738625.561467070 secs.nsecs elapsed_time 12883.762387721 secs.nsecs ldlm_enqueue 263089 samples [reqs] ldlm_convert 10 samples [reqs] ldlm_cancel 8145 samples [reqs] ldlm_bl_callback 3559 samples [reqs] ldlm_cp_callback 2 samples [reqs] start stat & ls ops... round0 done, before_priv_cnt=10, priv_cnt=10, inc_enq=507 round1 done, before_priv_cnt=19, priv_cnt=19, inc_enq=494 round2 done, before_priv_cnt=28, priv_cnt=28, inc_enq=485 round3 done, before_priv_cnt=37, priv_cnt=37, inc_enq=478 round4 done, before_priv_cnt=46, priv_cnt=46, inc_enq=468 round5 done, before_priv_cnt=55, priv_cnt=55, inc_enq=458 round6 done, before_priv_cnt=64, priv_cnt=64, inc_enq=451 round7 done, before_priv_cnt=73, priv_cnt=73, inc_enq=441 round8 done, before_priv_cnt=82, priv_cnt=82, inc_enq=431 round9 done, before_priv_cnt=91, priv_cnt=91, inc_enq=424 base_dir=/mnt/lustre/d124g.sanity/LFRU, policy=LFRU o_enq=263089, n_enq=267726, ref_enq_res=4637 Waiting for MDT destroys to complete ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800.lock_cache_policy=LRU total: 400 open/close in 6.12 seconds: 65.35 ops/second total: 100 open/close in 3.00 seconds: 33.29 ops/second ldlm.namespaces.lustre-MDT0000-mdc-ffff93f5a5e0f800.lock_unused_count=0 mdt.lustre-MDT0000.exports.192.168.203.15@tcp.ldlm_stats= snapshot_time 1787751991.429511814 secs.nsecs start_time 1787738625.561467070 secs.nsecs elapsed_time 13365.868044744 secs.nsecs ldlm_enqueue 268352 samples [reqs] ldlm_convert 10 samples [reqs] ldlm_cancel 8147 samples [reqs] ldlm_bl_callback 3559 samples [reqs] ldlm_cp_callback 2 samples [reqs] start stat & ls ops... round0 done, before_priv_cnt=0, priv_cnt=0, inc_enq=507 round1 done, before_priv_cnt=0, priv_cnt=0, inc_enq=503 round2 done, before_priv_cnt=0, priv_cnt=0, inc_enq=503 round3 done, before_priv_cnt=0, priv_cnt=0, inc_enq=505 round4 done, before_priv_cnt=0, priv_cnt=0, inc_enq=504 round5 done, before_priv_cnt=0, priv_cnt=0, inc_enq=503 round6 done, before_priv_cnt=0, priv_cnt=0, inc_enq=503 round7 done, before_priv_cnt=0, priv_cnt=0, inc_enq=505 round8 done, before_priv_cnt=0, priv_cnt=0, inc_enq=504 round9 done, before_priv_cnt=0, priv_cnt=0, inc_enq=503 base_dir=/mnt/lustre/d124g.sanity/LRU, policy=LRU o_enq=268352, n_enq=273392, ref_enq_res=5040 Waiting for MDT destroys to complete >> lfru_enqueue=4637, lru_enqueue=5040 PASS 124g (963s) == sanity test 125: don't return EPROTO when a dir has a non-default striping and ACLs ========================================================== 09:54:13 (1787752453) uid=500(sanityusr) gid=500(sanityusr) groups=500(sanityusr) striped dir -i1 -c2 -H fnv_1a_64 /mnt/lustre/d125.sanity drwxrwxr-x+ 2 root root 8192 Aug 26 09:54 /mnt/lustre/d125.sanity PASS 125 (11s) == sanity test 126: check that the fsgid provided by the client is taken into account ========================================================== 09:54:24 (1787752464) running as uid/gid/euid/egid 0/1/0/1, groups: 1 [touch] [/mnt/lustre/f126.sanity] PASS 126 (8s) == sanity test 127a: verify the client stats are sane ==== 09:54:33 (1787752473) enable_stats_header=1 stats before reset osc.lustre-OST0000-osc-ffff93f5a5e0f800.stats= snapshot_time 1787752475.306708182 secs.nsecs start_time 1787746716.444333141 secs.nsecs elapsed_time 5758.862375041 secs.nsecs req_waittime 320072 samples [usecs] 1741 4178840 175120645019 197349368509691819 req_active 320072 samples [reqs] 1 798 47664057 13623469835 ldlm_glimpse_enqueue 141324 samples [reqs] 1 1 141324 141324 ost_setattr 129701 samples [usecs] 2445 2300442 58578148765 39382875588068643 ost_connect 2 samples [usecs] 9176 9762 18938 179495620 ost_disconnect 1 samples [usecs] 15868 15868 15868 251793424 ost_statfs 1 samples [usecs] 42025 42025 42025 1766100625 ost_set_info 4 samples [usecs] 6909 16419 45034 580066252 ldlm_cancel 48482 samples [usecs] 1674 1620468 2811191461 568847685362525 obd_ping 557 samples [usecs] 2040 1545430 14835015 4433182007673 osc.lustre-OST0001-osc-ffff93f5a5e0f800.stats= snapshot_time 1787752475.307117465 secs.nsecs start_time 1787746716.445361030 secs.nsecs elapsed_time 5758.861756435 secs.nsecs req_waittime 309139 samples [usecs] 1494 4105441 165376155644 180709986408455474 req_active 309139 samples [reqs] 1 707 41967062 11108011516 ldlm_glimpse_enqueue 134918 samples [reqs] 1 1 134918 134918 ost_setattr 123284 samples [usecs] 2410 2343344 55983378399 38132892011962483 ost_connect 2 samples [usecs] 9297 19281 28578 458191170 ost_disconnect 1 samples [usecs] 15781 15781 15781 249039961 ost_statfs 1 samples [usecs] 28967 28967 28967 839087089 ost_set_info 4 samples [usecs] 5509 18294 44080 575511994 ldlm_cancel 50381 samples [usecs] 1457 1515044 2766599300 512094526322926 obd_ping 548 samples [usecs] 1990 1220942 13909384 3551084951276 osc.lustre-OST0000-osc-ffff93f5a5e0f800.stats=0 osc.lustre-OST0001-osc-ffff93f5a5e0f800.stats=0 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.159052 s, 13.2 MB/s 1+0 records in 1+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.436826 s, 4.8 MB/s got name=req_waittime count=7 unit=[usecs] min=3053 max=64359 got name=req_active count=7 unit=[reqs] min=1 max=2 got name=ldlm_extent_enqueue count=2 unit=[reqs] min=1 max=1 got name=read_bytes count=1 unit=[bytes] min=2097152 max=2097152 got name=write_bytes count=1 unit=[bytes] min=2097152 max=2097152 got name=ost_read count=1 unit=[usecs] min=52138 max=52138 got name=ost_write count=1 unit=[usecs] min=64184 max=64184 got name=ost_punch count=1 unit=[usecs] min=10798 max=10798 got name=ldlm_cancel count=2 unit=[usecs] min=6616 max=8320 PASS 127a (8s) == sanity test 127b: verify the llite client stats are sane ========================================================== 09:54:41 (1787752481) stats before reset llite.lustre-ffff93f5a5e0f800.stats= snapshot_time 1787752483.096210372 secs.nsecs start_time 1787746716.299492534 secs.nsecs elapsed_time 5766.796717838 secs.nsecs read_bytes 1 samples [bytes] 2097152 2097152 2097152 4398046511104 write_bytes 1 samples [bytes] 2097152 2097152 2097152 4398046511104 hybrid_read_bytes 1 samples [bytes] 2097152 2097152 2097152 4398046511104 hybrid_write_bytes 1 samples [bytes] 2097152 2097152 2097152 4398046511104 read 1 samples [usecs] 413120 413120 413120 170668134400 write 1 samples [usecs] 139297 139297 139297 19403654209 ioctl 152 samples [reqs] open 43973 samples [usecs] 11 53876 5014346 21712974646 close 43973 samples [usecs] 53 153946 328784228 4269540314408 seek 1 samples [usecs] 16 16 16 256 readdir 90 samples [usecs] 6 632802 4617814 1444567096068 setattr 21307 samples [usecs] 10517 3333454 1090747092 920033309795200 truncate 1 samples [usecs] 22466 22466 22466 504721156 getattr 107871 samples [usecs] 107 4690461 741837810 1092873102770530 create 2000 samples [usecs] 2013 45961 11676742 94653203498 unlink 44908 samples [usecs] 1962 1940162 838651574 236215101040356 mkdir 18 samples [usecs] 6432 138045 474950 31007972474 rmdir 120 samples [usecs] 6043 155091 8010326 632435918756 mknod 45908 samples [usecs] 2011 2802019 696033168 214484872392362 statfs 16 samples [usecs] 121 23709 162387 2303026515 setxattr 1 samples [usecs] 69058 69058 69058 4769007364 getxattr 128 samples [usecs] 23 114182 2424837 76462917709 getxattr_hits 14 samples [reqs] inode_permission 782177 samples [usecs] 0 109172 138596022 693790829170 opencount 43982 samples [reqs] 1 3 44006 44064 openclosetime 26 samples [usecs] 4684 627175855 2671675788 1221326397933057694 hybrid_noswitch 2 samples [reqs] hybrid_writesize_switch 1 samples [reqs] llite.lustre-ffff93f5a5e0f800.stats=0 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0292463 s, 140 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.00415054 s, 987 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0213933 s, 191 kB/s 1+0 records in 1+0 records out 4096 bytes (4.1 kB, 4.0 KiB) copied, 0.0109951 s, 373 kB/s got name=read_bytes count=2 unit=[bytes] min=4096 max=4096 got name=write_bytes count=2 unit=[bytes] min=4096 max=4096 got name=hybrid_read_bytes count=1 unit=[bytes] min=4096 max=4096 got name=cached_read_bytes count=1 unit=[bytes] min=4096 max=4096 got name=read count=2 unit=[usecs] min=61 max=13430 got name=write count=2 unit=[usecs] min=2864 max=19907 got name=open count=4 unit=[usecs] min=62 max=28061 got name=close count=4 unit=[usecs] min=294 max=9781 got name=seek count=2 unit=[usecs] min=17 max=17 got name=truncate count=1 unit=[usecs] min=39379 max=39379 got name=mknod count=1 unit=[usecs] min=22916 max=22916 got name=getxattr count=1 unit=[usecs] min=13599 max=13599 got name=inode_permission count=9 unit=[usecs] min=4 max=12873 got name=opencount count=4 unit=[reqs] min=1 max=4 got name=openclosetime count=3 unit=[usecs] min=27856 max=106146 got name=hybrid_noswitch count=3 unit=[reqs] min= max= PASS 127b (7s) == sanity test 127c: test llite extent stats with regular & mmap i/o ========================================================== 09:54:48 (1787752488) llite.lustre-ffff93f5a5e0f800.extents_stats=1 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00899124 s, 342 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0225069 s, 136 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.00624184 s, 492 kB/s 1+0 records in 1+0 records out 3072 bytes (3.1 kB, 3.0 KiB) copied, 0.0008699 s, 3.5 MB/s 1+0 records in 1+0 records out 6144 bytes (6.1 kB, 6.0 KiB) copied, 0.0318765 s, 193 kB/s 1+0 records in 1+0 records out 6144 bytes (6.1 kB, 6.0 KiB) copied, 0.00293822 s, 2.1 MB/s 1+0 records in 1+0 records out 6144 bytes (6.1 kB, 6.0 KiB) copied, 0.00624994 s, 983 kB/s 1+0 records in 1+0 records out 6144 bytes (6.1 kB, 6.0 KiB) copied, 0.00668726 s, 919 kB/s 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.013485 s, 911 kB/s 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00316936 s, 3.9 MB/s 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00683161 s, 1.8 MB/s 1+0 records in 1+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.00945674 s, 1.3 MB/s 1+0 records in 1+0 records out 24576 bytes (25 kB, 24 KiB) copied, 0.0139956 s, 1.8 MB/s 1+0 records in 1+0 records out 24576 bytes (25 kB, 24 KiB) copied, 0.00406003 s, 6.1 MB/s 1+0 records in 1+0 records out 24576 bytes (25 kB, 24 KiB) copied, 0.00718053 s, 3.4 MB/s 1+0 records in 1+0 records out 24576 bytes (25 kB, 24 KiB) copied, 0.00777762 s, 3.2 MB/s 1+0 records in 1+0 records out 49152 bytes (49 kB, 48 KiB) copied, 0.0208373 s, 2.4 MB/s 1+0 records in 1+0 records out 49152 bytes (49 kB, 48 KiB) copied, 0.00558482 s, 8.8 MB/s 1+0 records in 1+0 records out 49152 bytes (49 kB, 48 KiB) copied, 0.00716291 s, 6.9 MB/s 1+0 records in 1+0 records out 49152 bytes (49 kB, 48 KiB) copied, 0.00561249 s, 8.8 MB/s 1+0 records in 1+0 records out 98304 bytes (98 kB, 96 KiB) copied, 0.0191111 s, 5.1 MB/s 1+0 records in 1+0 records out 98304 bytes (98 kB, 96 KiB) copied, 0.0151597 s, 6.5 MB/s 1+0 records in 1+0 records out 98304 bytes (98 kB, 96 KiB) copied, 0.0263133 s, 3.7 MB/s 1+0 records in 1+0 records out 98304 bytes (98 kB, 96 KiB) copied, 0.00931178 s, 10.6 MB/s 1+0 records in 1+0 records out 196608 bytes (197 kB, 192 KiB) copied, 0.0279845 s, 7.0 MB/s 1+0 records in 1+0 records out 196608 bytes (197 kB, 192 KiB) copied, 0.013491 s, 14.6 MB/s 1+0 records in 1+0 records out 196608 bytes (197 kB, 192 KiB) copied, 0.0101895 s, 19.3 MB/s 1+0 records in 1+0 records out 196608 bytes (197 kB, 192 KiB) copied, 0.00970419 s, 20.3 MB/s 1+0 records in 1+0 records out 393216 bytes (393 kB, 384 KiB) copied, 0.0418063 s, 9.4 MB/s 1+0 records in 1+0 records out 393216 bytes (393 kB, 384 KiB) copied, 0.0506325 s, 7.8 MB/s 1+0 records in 1+0 records out 393216 bytes (393 kB, 384 KiB) copied, 0.0119556 s, 32.9 MB/s 1+0 records in 1+0 records out 393216 bytes (393 kB, 384 KiB) copied, 0.0229589 s, 17.1 MB/s 1+0 records in 1+0 records out 786432 bytes (786 kB, 768 KiB) copied, 0.0717669 s, 11.0 MB/s 1+0 records in 1+0 records out 786432 bytes (786 kB, 768 KiB) copied, 0.13074 s, 6.0 MB/s 1+0 records in 1+0 records out 786432 bytes (786 kB, 768 KiB) copied, 0.0167569 s, 46.9 MB/s 1+0 records in 1+0 records out 786432 bytes (786 kB, 768 KiB) copied, 0.0266014 s, 29.6 MB/s 1+0 records in 1+0 records out 1572864 bytes (1.6 MB, 1.5 MiB) copied, 0.255062 s, 6.2 MB/s 1+0 records in 1+0 records out 1572864 bytes (1.6 MB, 1.5 MiB) copied, 0.179057 s, 8.8 MB/s 1+0 records in 1+0 records out 1572864 bytes (1.6 MB, 1.5 MiB) copied, 0.0211804 s, 74.3 MB/s 1+0 records in 1+0 records out 1572864 bytes (1.6 MB, 1.5 MiB) copied, 0.0169974 s, 92.5 MB/s 1+0 records in 1+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.163682 s, 19.2 MB/s 1+0 records in 1+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.166497 s, 18.9 MB/s 1+0 records in 1+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.709545 s, 4.4 MB/s 1+0 records in 1+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.046423 s, 67.8 MB/s 1+0 records in 1+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.368572 s, 17.1 MB/s 1+0 records in 1+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.362763 s, 17.3 MB/s 1+0 records in 1+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 1.28674 s, 4.9 MB/s 1+0 records in 1+0 records out 6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.038775 s, 162 MB/s 1+0 records in 1+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.402336 s, 31.3 MB/s 1+0 records in 1+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.401 s, 31.4 MB/s 1+0 records in 1+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.317789 s, 39.6 MB/s 1+0 records in 1+0 records out 12582912 bytes (13 MB, 12 MiB) copied, 0.591538 s, 21.3 MB/s 1+0 records in 1+0 records out 25165824 bytes (25 MB, 24 MiB) copied, 0.984185 s, 25.6 MB/s 1+0 records in 1+0 records out 25165824 bytes (25 MB, 24 MiB) copied, 1.10607 s, 22.8 MB/s 1+0 records in 1+0 records out 25165824 bytes (25 MB, 24 MiB) copied, 0.957382 s, 26.3 MB/s 1+0 records in 1+0 records out 25165824 bytes (25 MB, 24 MiB) copied, 1.04086 s, 24.2 MB/s 1+0 records in 1+0 records out 50331648 bytes (50 MB, 48 MiB) copied, 1.81599 s, 27.7 MB/s 1+0 records in 1+0 records out 50331648 bytes (50 MB, 48 MiB) copied, 1.84253 s, 27.3 MB/s 1+0 records in 1+0 records out 50331648 bytes (50 MB, 48 MiB) copied, 1.80523 s, 27.9 MB/s 1+0 records in 1+0 records out 50331648 bytes (50 MB, 48 MiB) copied, 1.7462 s, 28.8 MB/s llite.lustre-ffff93f5a5e0f800.extents_stats= snapshot_time: 1787752512.769745365 secs.nsecs start_time: 1787752490.582807657 secs.nsecs elapsed_time: 22.186937708 secs.nsecs read | write extents calls % cum% | calls % cum% 0K - 4K : 2 6 6 | 2 6 6 4K - 8K : 2 6 13 | 2 6 13 8K - 16K : 2 6 20 | 2 6 20 16K - 32K : 2 6 26 | 2 6 26 32K - 64K : 2 6 33 | 2 6 33 64K - 128K : 2 6 40 | 2 6 40 128K - 256K : 2 6 46 | 2 6 46 256K - 512K : 2 6 53 | 2 6 53 512K - 1024K : 2 6 60 | 2 6 60 1M - 2M : 2 6 66 | 2 6 66 2M - 4M : 2 6 73 | 2 6 73 4M - 8M : 2 6 80 | 2 6 80 8M - 16M : 2 6 86 | 2 6 86 16M - 32M : 2 6 93 | 2 6 93 32M - 64M : 2 6 100 | 2 6 100 llite.lustre-ffff93f5a5e0f800.extents_stats=c 1+0 records in 1+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.0610669 s, 8.6 MB/s llite.lustre-ffff93f5a5e0f800.extents_stats= snapshot_time: 1787752514.814636262 secs.nsecs start_time: 1787752514.403588413 secs.nsecs elapsed_time: 0.411047849 secs.nsecs read | write extents calls % cum% | calls % cum% 0K - 4K : 0 0 0 | 0 0 0 4K - 8K : 256 100 100 | 128 99 99 8K - 16K : 0 0 100 | 0 0 99 16K - 32K : 0 0 100 | 0 0 99 32K - 64K : 0 0 100 | 0 0 99 64K - 128K : 0 0 100 | 0 0 99 128K - 256K : 0 0 100 | 0 0 99 256K - 512K : 0 0 100 | 0 0 99 512K - 1024K : 0 0 100 | 1 0 100 llite.lustre-ffff93f5a5e0f800.extents_stats=0 PASS 127c (32s) == sanity test 127d: OSC RPC latency histograms for read and write latency ========================================================== 09:55:20 (1787752520) osc.lustre-OST0000-osc-ffff93f5a5e0f800.rpc_stats=clear osc.lustre-OST0001-osc-ffff93f5a5e0f800.rpc_stats=clear 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.8172 s, 5.8 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.54385 s, 6.8 MB/s RPC stats after I/O: snapshot_time: 1787752526.380382456 secs.nsecs start_time: 1787752522.476537795 secs.nsecs elapsed_time: 3.903844661 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 1 100 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 0 0 0 | 1 100 100 read write offset rpcs % cum % | rpcs % cum % 0: 0 0 0 | 1 100 100 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 0 0 0 | 0 0 0 4096: 0 0 0 | 0 0 0 8192: 0 0 0 | 0 0 0 16384: 0 0 0 | 0 0 0 32768: 0 0 0 | 1 100 100 snapshot_time: 1787752526.380485657 secs.nsecs start_time: 1787752522.476813877 secs.nsecs elapsed_time: 3.903671780 secs.nsecs read RPCs in flight: 0 write RPCs in flight: 0 DIO RPCs in flight: 0 pending write pages: 0 pending read pages: 0 read write pages per rpc rpcs % cum % | rpcs % cum % 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 1 25 25 | 0 0 0 512: 1 25 50 | 1 33 33 1024: 2 50 100 | 2 66 100 read write rpcs in flight rpcs % cum % | rpcs % cum % 1: 3 75 75 | 3 100 100 2: 1 25 100 | 0 0 100 read write offset rpcs % cum % | rpcs % cum % 0: 1 25 25 | 1 33 33 1: 0 0 25 | 0 0 33 2: 0 0 25 | 0 0 33 4: 0 0 25 | 0 0 33 8: 0 0 25 | 0 0 33 16: 0 0 25 | 0 0 33 32: 0 0 25 | 0 0 33 64: 0 0 25 | 0 0 33 128: 0 0 25 | 0 0 33 256: 1 25 50 | 0 0 33 512: 0 0 50 | 0 0 33 1024: 1 25 75 | 1 33 66 2048: 1 25 100 | 1 33 100 read write RPC latency (us) count % cum % | count % cum % 0: 0 0 0 | 0 0 0 1: 0 0 0 | 0 0 0 2: 0 0 0 | 0 0 0 4: 0 0 0 | 0 0 0 8: 0 0 0 | 0 0 0 16: 0 0 0 | 0 0 0 32: 0 0 0 | 0 0 0 64: 0 0 0 | 0 0 0 128: 0 0 0 | 0 0 0 256: 0 0 0 | 0 0 0 512: 0 0 0 | 0 0 0 1024: 0 0 0 | 0 0 0 2048: 0 0 0 | 0 0 0 4096: 0 0 0 | 0 0 0 8192: 0 0 0 | 0 0 0 16384: 1 25 25 | 0 0 0 32768: 2 50 75 | 0 0 0 65536: 1 25 100 | 2 66 66 131072: 0 0 100 | 1 33 100 Read latency found: 1 Write latency found: 1 PASS 127d (11s) == sanity test 127e: client IO latency histograms by size ========================================================== 09:55:31 (1787752531) osc.lustre-OST0000-osc-ffff93f5a5e0f800.io_latency_stats=clear Generating I/O with size 4K 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.272901 s, 150 kB/s 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.129538 s, 316 kB/s Generating I/O with size 64K 10+0 records in 10+0 records out 655360 bytes (655 kB, 640 KiB) copied, 0.397197 s, 1.6 MB/s 10+0 records in 10+0 records out 655360 bytes (655 kB, 640 KiB) copied, 0.330098 s, 2.0 MB/s Generating I/O with size 1024K 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 1.19676 s, 8.8 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.577694 s, 18.2 MB/s osc.lustre-OST0000-*.io_latency_stats stats after I/O: osc.lustre-OST0000-osc-ffff93f5a5e0f800.io_latency_stats= io_latency_by_size: snapshot_time: 1787752543.209590435 start_time: 1787752534.260379663 elapsed_time: 8.949210772 rd_4K: { 4096us: 5, 8192us: 5, } rd_64K: { 8192us: 3, 16384us: 5, 32768us: 2, } rd_1024K: { 8192us: 3, 16384us: 4, 32768us: 2, 65536us: 1, } wr_4K: { 8192us: 1, 16384us: 9, } wr_64K: { 16384us: 7, 32768us: 3, } wr_1024K: { 16384us: 1, 32768us: 7, 65536us: 2, } PASS 127e (18s) == sanity test 127f: OST IO latency histograms by size === 09:55:50 (1787752550) osd-ldiskfs.lustre-OST0000.read_cache_enable=0 osd-ldiskfs.lustre-OST0000.writethrough_cache_enable=0 osd-ldiskfs.lustre-OST0000.io_latency_stats=clear osd-ldiskfs.lustre-OST0000.stats=clear osd-ldiskfs.lustre-OST0001.stats=clear Generating I/O with size 4K 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.468824 s, 87.4 kB/s 10+0 records in 10+0 records out 40960 bytes (41 kB, 40 KiB) copied, 0.170642 s, 240 kB/s Generating I/O with size 64K 10+0 records in 10+0 records out 655360 bytes (655 kB, 640 KiB) copied, 0.516059 s, 1.3 MB/s 10+0 records in 10+0 records out 655360 bytes (655 kB, 640 KiB) copied, 0.282579 s, 2.3 MB/s Generating I/O with size 1024K 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.991571 s, 10.6 MB/s 10+0 records in 10+0 records out 10485760 bytes (10 MB, 10 MiB) copied, 0.835277 s, 12.6 MB/s osd-*.lustre-OST0000.io_latency_stats stats after I/O: osd-ldiskfs.lustre-OST0000.io_latency_stats= io_latency_by_size: snapshot_time: 1787752567.235978636 start_time: 1787752555.868823327 elapsed_time: 11.367155309 rd_4K: { 256us: 1, 512us: 5, 1024us: 4, } rd_64K: { 256us: 2, 512us: 2, 1024us: 3, 2048us: 1, 4096us: 2, } rd_1024K: { 1024us: 6, 2048us: 2, 4096us: 2, } wr_4K: { 256us: 2, 512us: 4, 1024us: 4, } wr_64K: { 256us: 1, 512us: 3, 1024us: 2, 2048us: 3, 4096us: 1, } wr_1024K: { 512us: 1, 1024us: 2, 2048us: 5, 4096us: 2, } osd-ldiskfs.lustre-OST0000.read_cache_enable=1 osd-ldiskfs.lustre-OST0000.writethrough_cache_enable=1 PASS 127f (26s) == sanity test 127g: cached_read_bytes tracks page cache hits ========================================================== 09:56:16 (1787752576) llite.lustre-ffff93f5a5e0f800.stats=0 (a) cold: cached= total=4096 llite.lustre-ffff93f5a5e0f800.stats=0 (b) warm read: cached=16384 total=16384 count=cached_read_bytes llite.lustre-ffff93f5a5e0f800.stats=0 (c) fast_read=0: cached= llite.lustre-ffff93f5a5e0f800.stats=0 (d) readahead: cached=57344 total=65536 miss=2/16 PASS 127g (18s) == sanity test 128: interactive lfs for 2 consecutive find's ========================================================== 09:56:33 (1787752593) lfs: failed for 'find': No such file or directory /mnt/lustre/f128.sanity PASS 128 (7s) == sanity test 129: test directory size limit ================================================================================== 09:56:40 (1787752600) pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 mcreate: cannot create `/mnt/lustre/d129.sanity/file_base_497' with mode 0100644: No space left on device pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 rc=28 returned as expected after 497 files total: 5 open/close in 0.21 seconds: 23.33 ops/second [14210.389461] Lustre: 60854:0:(osd_handler.c:613:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513275, FID: [0x20000040b:0x906f:0x0]) is approaching max size limit [14212.328176] Lustre: 6524:0:(osd_handler.c:613:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513275, FID: [0x20000040b:0x906f:0x0]) is approaching max size limit [14231.177895] Lustre: 6524:0:(osd_handler.c:609:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513275, FID: [0x20000040b:0x906f:0x0]) has reached max size limit [14231.177895] Lustre: 6524:0:(osd_handler.c:609:osd_ldiskfs_add_entry()) lustre-MDT0000: directory (inode: 513275, FID: [0x20000040b:0x906f:0x0]) has reached max size limit - unlinked 0 (time 1787752664 ; total 0 ; last 0) unlink(/mnt/lustre/d129.sanity/file_base_497) error: No such file or directory total: 497 unlinks in 6 seconds: 82.833336 unlinks/second pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 PASS 129 (80s) == sanity test 130a: FIEMAP (1-stripe file) ============== 09:58:00 (1787752680) 1+0 records in 1+0 records out 65536 bytes (66 kB, 64 KiB) copied, 0.0274161 s, 2.4 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f130a.sanity is 65536 (64 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 63: 379008.. 379071: 64: 0000: last,net,eof /mnt/lustre/f130a.sanity: 1 extent found FIEMAP on single striped file succeeded PASS 130a (8s) == sanity test 130b: FIEMAP (2-stripe file) ============== 09:58:08 (1787752688) 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.279964 s, 7.5 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f130b.sanity is 2097152 (2048 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 1023: 1835008.. 1836031: 1024: 0000: net 1: 0.. 1023: 1183744.. 1184767: 1024: 0001: last,net /mnt/lustre/f130b.sanity: 2 extents found FIEMAP on 2-stripe file succeeded PASS 130b (11s) == sanity test 130c: FIEMAP (2-stripe file with hole) ==== 09:58:19 (1787752699) 1+0 records in 1+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0706447 s, 14.8 MB/s Filesystem type is: bd00bd0 File size of /mnt/lustre/f130c.sanity is 2097152 (2048 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 512.. 1023: 351704.. 352215: 512: 0001: net 1: 512.. 1023: 378180.. 378691: 512: 0000: last,net /mnt/lustre/f130c.sanity: 2 extents found FIEMAP on 2-stripe file with hole succeeded PASS 130c (10s) == sanity test 130d: FIEMAP (N-stripe file) ============== 09:58:29 (1787752709) SKIP: sanity test_130d needs >= 3 OSTs SKIP 130d (5s) == sanity test 130e: FIEMAP (test continuation FIEMAP calls) ========================================================== 09:58:34 (1787752714) Filesystem type is: bd00bd0 File size of /mnt/lustre/f130e.sanity is 67043328 (65472 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 63: 355704.. 355767: 64: 0000: net 1: 128.. 191: 355768.. 355831: 64: 0000: net 2: 256.. 319: 355832.. 355895: 64: 0000: net 3: 384.. 447: 355896.. 355959: 64: 0000: net 4: 512.. 575: 355960.. 356023: 64: 0000: net 5: 640.. 703: 356024.. 356087: 64: 0000: net 6: 768.. 831: 356088.. 356151: 64: 0000: net 7: 896.. 959: 356152.. 356215: 64: 0000: net 8: 1024.. 1087: 356216.. 356279: 64: 0000: net 9: 1152.. 1215: 356280.. 356343: 64: 0000: net 10: 1280.. 1343: 380928.. 380991: 64: 0000: net 11: 1408.. 1471: 380992.. 381055: 64: 0000: net 12: 1536.. 1599: 381056.. 381119: 64: 0000: net 13: 1664.. 1727: 381120.. 381183: 64: 0000: net 14: 1792.. 1855: 381184.. 381247: 64: 0000: net 15: 1920.. 1983: 381248.. 381311: 64: 0000: net 16: 2048.. 2111: 381312.. 381375: 64: 0000: net 17: 2176.. 2239: 381376.. 381439: 64: 0000: net 18: 2304.. 2367: 381440.. 381503: 64: 0000: net 19: 2432.. 2495: 381504.. 381567: 64: 0000: net 20: 2560.. 2623: 381568.. 381631: 64: 0000: net 21: 2688.. 2751: 381632.. 381695: 64: 0000: net 22: 2816.. 2879: 381696.. 381759: 64: 0000: net 23: 2944.. 3007: 381760.. 381823: 64: 0000: net 24: 3072.. 3135: 381824.. 381887: 64: 0000: net 25: 3200.. 3263: 381888.. 381951: 64: 0000: net 26: 3328.. 3391: 381952.. 382015: 64: 0000: net 27: 3456.. 3519: 382016.. 382079: 64: 0000: net 28: 3584.. 3647: 382080.. 382143: 64: 0000: net 29: 3712.. 3775: 382144.. 382207: 64: 0000: net 30: 3840.. 3903: 382208.. 382271: 64: 0000: net 31: 3968.. 4031: 382272.. 382335: 64: 0000: net 32: 4096.. 4159: 1835072.. 1835135: 64: 0000: net 33: 4224.. 4287: 1835200.. 1835263: 64: 0000: net 34: 4352.. 4415: 1835328.. 1835391: 64: 0000: net 35: 4480.. 4543: 1835456.. 1835519: 64: 0000: net 36: 4608.. 4671: 1835584.. 1835647: 64: 0000: net 37: 4736.. 4799: 1835712.. 1835775: 64: 0000: net 38: 4864.. 4927: 1835840.. 1835903: 64: 0000: net 39: 4992.. 5055: 1835968.. 1836031: 64: 0000: net 40: 5120.. 5183: 1836096.. 1836159: 64: 0000: net 41: 5248.. 5311: 1836224.. 1836287: 64: 0000: net 42: 5376.. 5439: 1836352.. 1836415: 64: 0000: net 43: 5504.. 5567: 1836480.. 1836543: 64: 0000: net 44: 5632.. 5695: 1836608.. 1836671: 64: 0000: net 45: 5760.. 5823: 1836736.. 1836799: 64: 0000: net 46: 5888.. 5951: 1836864.. 1836927: 64: 0000: net 47: 6016.. 6079: 1836992.. 1837055: 64: 0000: net 48: 6144.. 6207: 1837120.. 1837183: 64: 0000: net 49: 6272.. 6335: 1837248.. 1837311: 64: 0000: net 50: 6400.. 6463: 1837376.. 1837439: 64: 0000: net 51: 6528.. 6591: 1837504.. 1837567: 64: 0000: net 52: 6656.. 6719: 1837632.. 1837695: 64: 0000: net 53: 6784.. 6847: 1837760.. 1837823: 64: 0000: net 54: 6912.. 6975: 1837888.. 1837951: 64: 0000: net 55: 7040.. 7103: 1838016.. 1838079: 64: 0000: net 56: 7168.. 7231: 1838144.. 1838207: 64: 0000: net 57: 7296.. 7359: 1838272.. 1838335: 64: 0000: net 58: 7424.. 7487: 1838400.. 1838463: 64: 0000: net 59: 7552.. 7615: 1838528.. 1838591: 64: 0000: net 60: 7680.. 7743: 1838656.. 1838719: 64: 0000: net 61: 7808.. 7871: 1838784.. 1838847: 64: 0000: net 62: 7936.. 7999: 1838912.. 1838975: 64: 0000: net 63: 8064.. 8127: 1839040.. 1839103: 64: 0000: net 64: 8192.. 8255: 1851392.. 1851455: 64: 0000: net 65: 8320.. 8383: 1851520.. 1851583: 64: 0000: net 66: 8448.. 8511: 1851648.. 1851711: 64: 0000: net 67: 8576.. 8639: 1851776.. 1851839: 64: 0000: net 68: 8704.. 8767: 1851904.. 1851967: 64: 0000: net 69: 8832.. 8895: 1852032.. 1852095: 64: 0000: net 70: 8960.. 9023: 1852160.. 1852223: 64: 0000: net 71: 9088.. 9151: 1852288.. 1852351: 64: 0000: net 72: 9216.. 9279: 1852416.. 1852479: 64: 0000: net 73: 9344.. 9407: 1852544.. 1852607: 64: 0000: net 74: 9472.. 9535: 1852672.. 1852735: 64: 0000: net 75: 9600.. 9663: 1852800.. 1852863: 64: 0000: net 76: 9728.. 9791: 1852928.. 1852991: 64: 0000: net 77: 9856.. 9919: 1853056.. 1853119: 64: 0000: net 78: 9984.. 10047: 1853184.. 1853247: 64: 0000: net 79: 10112.. 10175: 1853312.. 1853375: 64: 0000: net 80: 10240.. 10303: 1853440.. 1853503: 64: 0000: net 81: 10368.. 10431: 1853568.. 1853631: 64: 0000: net 82: 10496.. 10559: 1853696.. 1853759: 64: 0000: net 83: 10624.. 10687: 1853824.. 1853887: 64: 0000: net 84: 10752.. 10815: 1853952.. 1854015: 64: 0000: net 85: 10880.. 10943: 1854080.. 1854143: 64: 0000: net 86: 11008.. 11071: 1854208.. 1854271: 64: 0000: net 87: 11136.. 11199: 1854336.. 1854399: 64: 0000: net 88: 11264.. 11327: 1854464.. 1854527: 64: 0000: net 89: 11392.. 11455: 1854592.. 1854655: 64: 0000: net 90: 11520.. 11583: 1854720.. 1854783: 64: 0000: net 91: 11648.. 11711: 1854848.. 1854911: 64: 0000: net 92: 11776.. 11839: 1854976.. 1855039: 64: 0000: net 93: 11904.. 11967: 1855104.. 1855167: 64: 0000: net 94: 12032.. 12095: 1855232.. 1855295: 64: 0000: net 95: 12160.. 12223: 1855360.. 1855423: 64: 0000: net 96: 12288.. 12351: 1855488.. 1855551: 64: 0000: net 97: 12416.. 12479: 1855616.. 1855679: 64: 0000: net 98: 12544.. 12607: 1855744.. 1855807: 64: 0000: net 99: 12672.. 12735: 1855872.. 1855935: 64: 0000: net 100: 12800.. 12863: 1856000.. 1856063: 64: 0000: net 101: 12928.. 12991: 1856128.. 1856191: 64: 0000: net 102: 13056.. 13119: 1856256.. 1856319: 64: 0000: net 103: 13184.. 13247: 1856384.. 1856447: 64: 0000: net 104: 13312.. 13375: 1856512.. 1856575: 64: 0000: net 105: 13440.. 13503: 1856640.. 1856703: 64: 0000: net 106: 13568.. 13631: 1856768.. 1856831: 64: 0000: net 107: 13696.. 13759: 1856896.. 1856959: 64: 0000: net 108: 13824.. 13887: 1857024.. 1857087: 64: 0000: net 109: 13952.. 14015: 1857152.. 1857215: 64: 0000: net 110: 14080.. 14143: 1857280.. 1857343: 64: 0000: net 111: 14208.. 14271: 1857408.. 1857471: 64: 0000: net 112: 14336.. 14399: 1857536.. 1857599: 64: 0000: net 113: 14464.. 14527: 1857664.. 1857727: 64: 0000: net 114: 14592.. 14655: 1857792.. 1857855: 64: 0000: net 115: 14720.. 14783: 1857920.. 1857983: 64: 0000: net 116: 14848.. 14911: 1858048.. 1858111: 64: 0000: net 117: 14976.. 15039: 1858176.. 1858239: 64: 0000: net 118: 15104.. 15167: 1858304.. 1858367: 64: 0000: net 119: 15232.. 15295: 1858432.. 1858495: 64: 0000: net 120: 15360.. 15423: 1858560.. 1858623: 64: 0000: net 121: 15488.. 15551: 1858688.. 1858751: 64: 0000: net 122: 15616.. 15679: 1858816.. 1858879: 64: 0000: net 123: 15744.. 15807: 1858944.. 1859007: 64: 0000: net 124: 15872.. 15935: 1859072.. 1859135: 64: 0000: net 125: 16000.. 16063: 1859200.. 1859263: 64: 0000: net 126: 16128.. 16191: 1859328.. 1859391: 64: 0000: net 127: 16256.. 16319: 1859456.. 1859519: 64: 0000: net 128: 16384.. 16447: 1859584.. 1859647: 64: 0000: net 129: 16512.. 16575: 1859712.. 1859775: 64: 0000: net 130: 16640.. 16703: 1859840.. 1859903: 64: 0000: net 131: 16768.. 16831: 1859968.. 1860031: 64: 0000: net 132: 16896.. 16959: 1860096.. 1860159: 64: 0000: net 133: 17024.. 17087: 1860224.. 1860287: 64: 0000: net 134: 17152.. 17215: 1860352.. 1860415: 64: 0000: net 135: 17280.. 17343: 1860480.. 1860543: 64: 0000: net 136: 17408.. 17471: 1860608.. 1860671: 64: 0000: net 137: 17536.. 17599: 1860736.. 1860799: 64: 0000: net 138: 17664.. 17727: 1860864.. 1860927: 64: 0000: net 139: 17792.. 17855: 1860992.. 1861055: 64: 0000: net 140: 17920.. 17983: 1861120.. 1861183: 64: 0000: net 141: 18048.. 18111: 1861248.. 1861311: 64: 0000: net 142: 18176.. 18239: 1861376.. 1861439: 64: 0000: net 143: 18304.. 18367: 1861504.. 1861567: 64: 0000: net 144: 18432.. 18495: 1861632.. 1861695: 64: 0000: net 145: 18560.. 18623: 1861760.. 1861823: 64: 0000: net 146: 18688.. 18751: 1861888.. 1861951: 64: 0000: net 147: 18816.. 18879: 1862016.. 1862079: 64: 0000: net 148: 18944.. 19007: 1862144.. 1862207: 64: 0000: net 149: 19072.. 19135: 1862272.. 1862335: 64: 0000: net 150: 19200.. 19263: 1862400.. 1862463: 64: 0000: net 151: 19328.. 19391: 1862528.. 1862591: 64: 0000: net 152: 19456.. 19519: 1862656.. 1862719: 64: 0000: net 153: 19584.. 19647: 1862784.. 1862847: 64: 0000: net 154: 19712.. 19775: 1862912.. 1862975: 64: 0000: net 155: 19840.. 19903: 1863040.. 1863103: 64: 0000: net 156: 19968.. 20031: 1863168.. 1863231: 64: 0000: net 157: 20096.. 20159: 1863296.. 1863359: 64: 0000: net 158: 20224.. 20287: 1863424.. 1863487: 64: 0000: net 159: 20352.. 20415: 1863552.. 1863615: 64: 0000: net 160: 20480.. 20543: 1863680.. 1863743: 64: 0000: net 161: 20608.. 20671: 1863808.. 1863871: 64: 0000: net 162: 20736.. 20799: 1863936.. 1863999: 64: 0000: net 163: 20864.. 20927: 1864064.. 1864127: 64: 0000: net 164: 20992.. 21055: 1864192.. 1864255: 64: 0000: net 165: 21120.. 21183: 1864320.. 1864383: 64: 0000: net 166: 21248.. 21311: 1864448.. 1864511: 64: 0000: net 167: 21376.. 21439: 1864576.. 1864639: 64: 0000: net 168: 21504.. 21567: 1864704.. 1864767: 64: 0000: net 169: 21632.. 21695: 1864832.. 1864895: 64: 0000: net 170: 21760.. 21823: 1864960.. 1865023: 64: 0000: net 171: 21888.. 21951: 1865088.. 1865151: 64: 0000: net 172: 22016.. 22079: 1865216.. 1865279: 64: 0000: net 173: 22144.. 22207: 1865344.. 1865407: 64: 0000: net 174: 22272.. 22335: 1865472.. 1865535: 64: 0000: net 175: 22400.. 22463: 1865600.. 1865663: 64: 0000: net 176: 22528.. 22591: 1865728.. 1865791: 64: 0000: net 177: 22656.. 22719: 1865856.. 1865919: 64: 0000: net 178: 22784.. 22847: 1865984.. 1866047: 64: 0000: net 179: 22912.. 22975: 1866112.. 1866175: 64: 0000: net 180: 23040.. 23103: 1866240.. 1866303: 64: 0000: net 181: 23168.. 23231: 1866368.. 1866431: 64: 0000: net 182: 23296.. 23359: 1866496.. 1866559: 64: 0000: net 183: 23424.. 23487: 1866624.. 1866687: 64: 0000: net 184: 23552.. 23615: 1866752.. 1866815: 64: 0000: net 185: 23680.. 23743: 1866880.. 1866943: 64: 0000: net 186: 23808.. 23871: 1867008.. 1867071: 64: 0000: net 187: 23936.. 23999: 1867136.. 1867199: 64: 0000: net 188: 24064.. 24127: 1867264.. 1867327: 64: 0000: net 189: 24192.. 24255: 1867392.. 1867455: 64: 0000: net 190: 24320.. 24383: 1867520.. 1867583: 64: 0000: net 191: 24448.. 24511: 1867648.. 1867711: 64: 0000: net 192: 24576.. 24639: 1843200.. 1843263: 64: 0000: net 193: 24704.. 24767: 1843328.. 1843391: 64: 0000: net 194: 24832.. 24895: 1843456.. 1843519: 64: 0000: net 195: 24960.. 25023: 1843584.. 1843647: 64: 0000: net 196: 25088.. 25151: 1843712.. 1843775: 64: 0000: net 197: 25216.. 25279: 1843840.. 1843903: 64: 0000: net 198: 25344.. 25407: 1843968.. 1844031: 64: 0000: net 199: 25472.. 25535: 1844096.. 1844159: 64: 0000: net 200: 25600.. 25663: 1844224.. 1844287: 64: 0000: net 201: 25728.. 25791: 1844352.. 1844415: 64: 0000: net 202: 25856.. 25919: 1844480.. 1844543: 64: 0000: net 203: 25984.. 26047: 1844608.. 1844671: 64: 0000: net 204: 26112.. 26175: 1844736.. 1844799: 64: 0000: net 205: 26240.. 26303: 1844864.. 1844927: 64: 0000: net 206: 26368.. 26431: 1844992.. 1845055: 64: 0000: net 207: 26496.. 26559: 1845120.. 1845183: 64: 0000: net 208: 26624.. 26687: 1845248.. 1845311: 64: 0000: net 209: 26752.. 26815: 1845376.. 1845439: 64: 0000: net 210: 26880.. 26943: 1845504.. 1845567: 64: 0000: net 211: 27008.. 27071: 1845632.. 1845695: 64: 0000: net 212: 27136.. 27199: 1845760.. 1845823: 64: 0000: net 213: 27264.. 27327: 1845888.. 1845951: 64: 0000: net 214: 27392.. 27455: 1846016.. 1846079: 64: 0000: net 215: 27520.. 27583: 1846144.. 1846207: 64: 0000: net 216: 27648.. 27711: 1846272.. 1846335: 64: 0000: net 217: 27776.. 27839: 1846400.. 1846463: 64: 0000: net 218: 27904.. 27967: 1846528.. 1846591: 64: 0000: net 219: 28032.. 28095: 1846656.. 1846719: 64: 0000: net 220: 28160.. 28223: 1846784.. 1846847: 64: 0000: net 221: 28288.. 28351: 1846912.. 1846975: 64: 0000: net 222: 28416.. 28479: 1847040.. 1847103: 64: 0000: net 223: 28544.. 28607: 1847168.. 1847231: 64: 0000: net 224: 28672.. 28735: 1847296.. 1847359: 64: 0000: net 225: 28800.. 28863: 1847424.. 1847487: 64: 0000: net 226: 28928.. 28991: 1847552.. 1847615: 64: 0000: net 227: 29056.. 29119: 1847680.. 1847743: 64: 0000: net 228: 29184.. 29247: 1847808.. 1847871: 64: 0000: net 229: 29312.. 29375: 1847936.. 1847999: 64: 0000: net 230: 29440.. 29503: 1848064.. 1848127: 64: 0000: net 231: 29568.. 29631: 1848192.. 1848255: 64: 0000: net 232: 29696.. 29759: 1848320.. 1848383: 64: 0000: net 233: 29824.. 29887: 1848448.. 1848511: 64: 0000: net 234: 29952.. 30015: 1848576.. 1848639: 64: 0000: net 235: 30080.. 30143: 1848704.. 1848767: 64: 0000: net 236: 30208.. 30271: 1848832.. 1848895: 64: 0000: net 237: 30336.. 30399: 1848960.. 1849023: 64: 0000: net 238: 30464.. 30527: 1849088.. 1849151: 64: 0000: net 239: 30592.. 30655: 1849216.. 1849279: 64: 0000: net 240: 30720.. 30783: 1849344.. 1849407: 64: 0000: net 241: 30848.. 30911: 1849472.. 1849535: 64: 0000: net 242: 30976.. 31039: 1849600.. 1849663: 64: 0000: net 243: 31104.. 31167: 1849728.. 1849791: 64: 0000: net 244: 31232.. 31295: 1849856.. 1849919: 64: 0000: net 245: 31360.. 31423: 1849984.. 1850047: 64: 0000: net 246: 31488.. 31551: 1850112.. 1850175: 64: 0000: net 247: 31616.. 31679: 1850240.. 1850303: 64: 0000: net 248: 31744.. 31807: 1850368.. 1850431: 64: 0000: net 249: 31872.. 31935: 1850496.. 1850559: 64: 0000: net 250: 32000.. 32063: 1850624.. 1850687: 64: 0000: net 251: 32128.. 32191: 1850752.. 1850815: 64: 0000: net 252: 32256.. 32319: 1850880.. 1850943: 64: 0000: net 253: 32384.. 32447: 1851008.. 1851071: 64: 0000: net 254: 32512.. 32575: 1851136.. 1851199: 64: 0000: net 255: 32640.. 32703: 1851264.. 1851327: 64: 0000: net 256: 0.. 63: 349948.. 350011: 64: 0001: net 257: 128.. 191: 350012.. 350075: 64: 0001: net 258: 256.. 319: 350076.. 350139: 64: 0001: net 259: 384.. 447: 350140.. 350203: 64: 0001: net 260: 512.. 575: 360448.. 360511: 64: 0001: net 261: 640.. 703: 360512.. 360575: 64: 0001: net 262: 768.. 831: 360576.. 360639: 64: 0001: net 263: 896.. 959: 360640.. 360703: 64: 0001: net 264: 1024.. 1087: 360704.. 360767: 64: 0001: net 265: 1152.. 1215: 360768.. 360831: 64: 0001: net 266: 1280.. 1343: 360832.. 360895: 64: 0001: net 267: 1408.. 1471: 360896.. 360959: 64: 0001: net 268: 1536.. 1599: 360960.. 361023: 64: 0001: net 269: 1664.. 1727: 361024.. 361087: 64: 0001: net 270: 1792.. 1855: 361088.. 361151: 64: 0001: net 271: 1920.. 1983: 361152.. 361215: 64: 0001: net 272: 2048.. 2111: 361216.. 361279: 64: 0001: net 273: 2176.. 2239: 361280.. 361343: 64: 0001: net 274: 2304.. 2367: 361344.. 361407: 64: 0001: net 275: 2432.. 2495: 361408.. 361471: 64: 0001: net 276: 2560.. 2623: 361472.. 361535: 64: 0001: net 277: 2688.. 2751: 361536.. 361599: 64: 0001: net 278: 2816.. 2879: 361600.. 361663: 64: 0001: net 279: 2944.. 3007: 361664.. 361727: 64: 0001: net 280: 3072.. 3135: 361728.. 361791: 64: 0001: net 281: 3200.. 3263: 361792.. 361855: 64: 0001: net 282: 3328.. 3391: 361856.. 361919: 64: 0001: net 283: 3456.. 3519: 361920.. 361983: 64: 0001: net 284: 3584.. 3647: 361984.. 362047: 64: 0001: net 285: 3712.. 3775: 362048.. 362111: 64: 0001: net 286: 3840.. 3903: 362112.. 362175: 64: 0001: net 287: 3968.. 4031: 362176.. 362239: 64: 0001: net 288: 4096.. 4159: 1183808.. 1183871: 64: 0001: net 289: 4224.. 4287: 1183936.. 1183999: 64: 0001: net 290: 4352.. 4415: 1184064.. 1184127: 64: 0001: net 291: 4480.. 4543: 1184192.. 1184255: 64: 0001: net 292: 4608.. 4671: 1184320.. 1184383: 64: 0001: net 293: 4736.. 4799: 1184448.. 1184511: 64: 0001: net 294: 4864.. 4927: 1184576.. 1184639: 64: 0001: net 295: 4992.. 5055: 1184704.. 1184767: 64: 0001: net 296: 5120.. 5183: 1184832.. 1184895: 64: 0001: net 297: 5248.. 5311: 1184960.. 1185023: 64: 0001: net 298: 5376.. 5439: 1185088.. 1185151: 64: 0001: net 299: 5504.. 5567: 1185216.. 1185279: 64: 0001: net 300: 5632.. 5695: 1185344.. 1185407: 64: 0001: net 301: 5760.. 5823: 1185472.. 1185535: 64: 0001: net 302: 5888.. 5951: 1185600.. 1185663: 64: 0001: net 303: 6016.. 6079: 1185728.. 1185791: 64: 0001: net 304: 6144.. 6207: 1185856.. 1185919: 64: 0001: net 305: 6272.. 6335: 1185984.. 1186047: 64: 0001: net 306: 6400.. 6463: 1186112.. 1186175: 64: 0001: net 307: 6528.. 6591: 1186240.. 1186303: 64: 0001: net 308: 6656.. 6719: 1186368.. 1186431: 64: 0001: net 309: 6784.. 6847: 1186496.. 1186559: 64: 0001: net 310: 6912.. 6975: 1186624.. 1186687: 64: 0001: net 311: 7040.. 7103: 1186752.. 1186815: 64: 0001: net 312: 7168.. 7231: 1186880.. 1186943: 64: 0001: net 313: 7296.. 7359: 1187008.. 1187071: 64: 0001: net 314: 7424.. 7487: 1187136.. 1187199: 64: 0001: net 315: 7552.. 7615: 1187264.. 1187327: 64: 0001: net 316: 7680.. 7743: 1187392.. 1187455: 64: 0001: net 317: 7808.. 7871: 1187520.. 1187583: 64: 0001: net 318: 7936.. 7999: 1187648.. 1187711: 64: 0001: net 319: 8064.. 8127: 1187776.. 1187839: 64: 0001: net 320: 8192.. 8255: 1196032.. 1196095: 64: 0001: net 321: 8320.. 8383: 1196160.. 1196223: 64: 0001: net 322: 8448.. 8511: 1196288.. 1196351: 64: 0001: net 323: 8576.. 8639: 1196416.. 1196479: 64: 0001: net 324: 8704.. 8767: 1196544.. 1196607: 64: 0001: net 325: 8832.. 8895: 1196672.. 1196735: 64: 0001: net 326: 8960.. 9023: 1196800.. 1196863: 64: 0001: net 327: 9088.. 9151: 1196928.. 1196991: 64: 0001: net 328: 9216.. 9279: 1197056.. 1197119: 64: 0001: net 329: 9344.. 9407: 1197184.. 1197247: 64: 0001: net 330: 9472.. 9535: 1197312.. 1197375: 64: 0001: net 331: 9600.. 9663: 1197440.. 1197503: 64: 0001: net 332: 9728.. 9791: 1197568.. 1197631: 64: 0001: net 333: 9856.. 9919: 1197696.. 1197759: 64: 0001: net 334: 9984.. 10047: 1197824.. 1197887: 64: 0001: net 335: 10112.. 10175: 1197952.. 1198015: 64: 0001: net 336: 10240.. 10303: 1198080.. 1198143: 64: 0001: net 337: 10368.. 10431: 1198208.. 1198271: 64: 0001: net 338: 10496.. 10559: 1198336.. 1198399: 64: 0001: net 339: 10624.. 10687: 1198464.. 1198527: 64: 0001: net 340: 10752.. 10815: 1198592.. 1198655: 64: 0001: net 341: 10880.. 10943: 1198720.. 1198783: 64: 0001: net 342: 11008.. 11071: 1198848.. 1198911: 64: 0001: net 343: 11136.. 11199: 1198976.. 1199039: 64: 0001: net 344: 11264.. 11327: 1199104.. 1199167: 64: 0001: net 345: 11392.. 11455: 1199232.. 1199295: 64: 0001: net 346: 11520.. 11583: 1199360.. 1199423: 64: 0001: net 347: 11648.. 11711: 1199488.. 1199551: 64: 0001: net 348: 11776.. 11839: 1199616.. 1199679: 64: 0001: net 349: 11904.. 11967: 1199744.. 1199807: 64: 0001: net 350: 12032.. 12095: 1199872.. 1199935: 64: 0001: net 351: 12160.. 12223: 1200000.. 1200063: 64: 0001: net 352: 12288.. 12351: 1200128.. 1200191: 64: 0001: net 353: 12416.. 12479: 1200256.. 1200319: 64: 0001: net 354: 12544.. 12607: 1200384.. 1200447: 64: 0001: net 355: 12672.. 12735: 1200512.. 1200575: 64: 0001: net 356: 12800.. 12863: 1200640.. 1200703: 64: 0001: net 357: 12928.. 12991: 1200768.. 1200831: 64: 0001: net 358: 13056.. 13119: 1200896.. 1200959: 64: 0001: net 359: 13184.. 13247: 1201024.. 1201087: 64: 0001: net 360: 13312.. 13375: 1201152.. 1201215: 64: 0001: net 361: 13440.. 13503: 1201280.. 1201343: 64: 0001: net 362: 13568.. 13631: 1201408.. 1201471: 64: 0001: net 363: 13696.. 13759: 1201536.. 1201599: 64: 0001: net 364: 13824.. 13887: 1201664.. 1201727: 64: 0001: net 365: 13952.. 14015: 1201792.. 1201855: 64: 0001: net 366: 14080.. 14143: 1201920.. 1201983: 64: 0001: net 367: 14208.. 14271: 1202048.. 1202111: 64: 0001: net 368: 14336.. 14399: 1202176.. 1202239: 64: 0001: net 369: 14464.. 14527: 1202304.. 1202367: 64: 0001: net 370: 14592.. 14655: 1202432.. 1202495: 64: 0001: net 371: 14720.. 14783: 1202560.. 1202623: 64: 0001: net 372: 14848.. 14911: 1202688.. 1202751: 64: 0001: net 373: 14976.. 15039: 1202816.. 1202879: 64: 0001: net 374: 15104.. 15167: 1202944.. 1203007: 64: 0001: net 375: 15232.. 15295: 1203072.. 1203135: 64: 0001: net 376: 15360.. 15423: 1203200.. 1203263: 64: 0001: net 377: 15488.. 15551: 1203328.. 1203391: 64: 0001: net 378: 15616.. 15679: 1203456.. 1203519: 64: 0001: net 379: 15744.. 15807: 1203584.. 1203647: 64: 0001: net 380: 15872.. 15935: 1203712.. 1203775: 64: 0001: net 381: 16000.. 16063: 1203840.. 1203903: 64: 0001: net 382: 16128.. 16191: 1203968.. 1204031: 64: 0001: net 383: 16256.. 16319: 1204096.. 1204159: 64: 0001: net 384: 16384.. 16447: 1212416.. 1212479: 64: 0001: net 385: 16512.. 16575: 1212544.. 1212607: 64: 0001: net 386: 16640.. 16703: 1212672.. 1212735: 64: 0001: net 387: 16768.. 16831: 1212800.. 1212863: 64: 0001: net 388: 16896.. 16959: 1212928.. 1212991: 64: 0001: net 389: 17024.. 17087: 1213056.. 1213119: 64: 0001: net 390: 17152.. 17215: 1213184.. 1213247: 64: 0001: net 391: 17280.. 17343: 1213312.. 1213375: 64: 0001: net 392: 17408.. 17471: 1213440.. 1213503: 64: 0001: net 393: 17536.. 17599: 1213568.. 1213631: 64: 0001: net 394: 17664.. 17727: 1213696.. 1213759: 64: 0001: net 395: 17792.. 17855: 1213824.. 1213887: 64: 0001: net 396: 17920.. 17983: 1213952.. 1214015: 64: 0001: net 397: 18048.. 18111: 1214080.. 1214143: 64: 0001: net 398: 18176.. 18239: 1214208.. 1214271: 64: 0001: net 399: 18304.. 18367: 1214336.. 1214399: 64: 0001: net 400: 18432.. 18495: 1214464.. 1214527: 64: 0001: net 401: 18560.. 18623: 1214592.. 1214655: 64: 0001: net 402: 18688.. 18751: 1214720.. 1214783: 64: 0001: net 403: 18816.. 18879: 1214848.. 1214911: 64: 0001: net 404: 18944.. 19007: 1214976.. 1215039: 64: 0001: net 405: 19072.. 19135: 1215104.. 1215167: 64: 0001: net 406: 19200.. 19263: 1215232.. 1215295: 64: 0001: net 407: 19328.. 19391: 1215360.. 1215423: 64: 0001: net 408: 19456.. 19519: 1215488.. 1215551: 64: 0001: net 409: 19584.. 19647: 1215616.. 1215679: 64: 0001: net 410: 19712.. 19775: 1215744.. 1215807: 64: 0001: net 411: 19840.. 19903: 1215872.. 1215935: 64: 0001: net 412: 19968.. 20031: 1216000.. 1216063: 64: 0001: net 413: 20096.. 20159: 1216128.. 1216191: 64: 0001: net 414: 20224.. 20287: 1216256.. 1216319: 64: 0001: net 415: 20352.. 20415: 1216384.. 1216447: 64: 0001: net 416: 20480.. 20543: 1216512.. 1216575: 64: 0001: net 417: 20608.. 20671: 1216640.. 1216703: 64: 0001: net 418: 20736.. 20799: 1216768.. 1216831: 64: 0001: net 419: 20864.. 20927: 1216896.. 1216959: 64: 0001: net 420: 20992.. 21055: 1217024.. 1217087: 64: 0001: net 421: 21120.. 21183: 1217152.. 1217215: 64: 0001: net 422: 21248.. 21311: 1217280.. 1217343: 64: 0001: net 423: 21376.. 21439: 1217408.. 1217471: 64: 0001: net 424: 21504.. 21567: 1217536.. 1217599: 64: 0001: net 425: 21632.. 21695: 1217664.. 1217727: 64: 0001: net 426: 21760.. 21823: 1217792.. 1217855: 64: 0001: net 427: 21888.. 21951: 1217920.. 1217983: 64: 0001: net 428: 22016.. 22079: 1218048.. 1218111: 64: 0001: net 429: 22144.. 22207: 1218176.. 1218239: 64: 0001: net 430: 22272.. 22335: 1218304.. 1218367: 64: 0001: net 431: 22400.. 22463: 1218432.. 1218495: 64: 0001: net 432: 22528.. 22591: 1218560.. 1218623: 64: 0001: net 433: 22656.. 22719: 1218688.. 1218751: 64: 0001: net 434: 22784.. 22847: 1218816.. 1218879: 64: 0001: net 435: 22912.. 22975: 1218944.. 1219007: 64: 0001: net 436: 23040.. 23103: 1219072.. 1219135: 64: 0001: net 437: 23168.. 23231: 1219200.. 1219263: 64: 0001: net 438: 23296.. 23359: 1219328.. 1219391: 64: 0001: net 439: 23424.. 23487: 1219456.. 1219519: 64: 0001: net 440: 23552.. 23615: 1219584.. 1219647: 64: 0001: net 441: 23680.. 23743: 1219712.. 1219775: 64: 0001: net 442: 23808.. 23871: 1219840.. 1219903: 64: 0001: net 443: 23936.. 23999: 1219968.. 1220031: 64: 0001: net 444: 24064.. 24127: 1220096.. 1220159: 64: 0001: net 445: 24192.. 24255: 1220224.. 1220287: 64: 0001: net 446: 24320.. 24383: 1220352.. 1220415: 64: 0001: net 447: 24448.. 24511: 1220480.. 1220543: 64: 0001: net 448: 24576.. 24639: 1204224.. 1204287: 64: 0001: net 449: 24704.. 24767: 1204352.. 1204415: 64: 0001: net 450: 24832.. 24895: 1204480.. 1204543: 64: 0001: net 451: 24960.. 25023: 1204608.. 1204671: 64: 0001: net 452: 25088.. 25151: 1204736.. 1204799: 64: 0001: net 453: 25216.. 25279: 1204864.. 1204927: 64: 0001: net 454: 25344.. 25407: 1204992.. 1205055: 64: 0001: net 455: 25472.. 25535: 1205120.. 1205183: 64: 0001: net 456: 25600.. 25663: 1205248.. 1205311: 64: 0001: net 457: 25728.. 25791: 1205376.. 1205439: 64: 0001: net 458: 25856.. 25919: 1205504.. 1205567: 64: 0001: net 459: 25984.. 26047: 1205632.. 1205695: 64: 0001: net 460: 26112.. 26175: 1205760.. 1205823: 64: 0001: net 461: 26240.. 26303: 1205888.. 1205951: 64: 0001: net 462: 26368.. 26431: 1206016.. 1206079: 64: 0001: net 463: 26496.. 26559: 1206144.. 1206207: 64: 0001: net 464: 26624.. 26687: 1206272.. 1206335: 64: 0001: net 465: 26752.. 26815: 1206400.. 1206463: 64: 0001: net 466: 26880.. 26943: 1206528.. 1206591: 64: 0001: net 467: 27008.. 27071: 1206656.. 1206719: 64: 0001: net 468: 27136.. 27199: 1206784.. 1206847: 64: 0001: net 469: 27264.. 27327: 1206912.. 1206975: 64: 0001: net 470: 27392.. 27455: 1207040.. 1207103: 64: 0001: net 471: 27520.. 27583: 1207168.. 1207231: 64: 0001: net 472: 27648.. 27711: 1207296.. 1207359: 64: 0001: net 473: 27776.. 27839: 1207424.. 1207487: 64: 0001: net 474: 27904.. 27967: 1207552.. 1207615: 64: 0001: net 475: 28032.. 28095: 1207680.. 1207743: 64: 0001: net 476: 28160.. 28223: 1207808.. 1207871: 64: 0001: net 477: 28288.. 28351: 1207936.. 1207999: 64: 0001: net 478: 28416.. 28479: 1208064.. 1208127: 64: 0001: net 479: 28544.. 28607: 1208192.. 1208255: 64: 0001: net 480: 28672.. 28735: 1208320.. 1208383: 64: 0001: net 481: 28800.. 28863: 1208448.. 1208511: 64: 0001: net 482: 28928.. 28991: 1208576.. 1208639: 64: 0001: net 483: 29056.. 29119: 1208704.. 1208767: 64: 0001: net 484: 29184.. 29247: 1208832.. 1208895: 64: 0001: net 485: 29312.. 29375: 1208960.. 1209023: 64: 0001: net 486: 29440.. 29503: 1209088.. 1209151: 64: 0001: net 487: 29568.. 29631: 1209216.. 1209279: 64: 0001: net 488: 29696.. 29759: 1209344.. 1209407: 64: 0001: net 489: 29824.. 29887: 1209472.. 1209535: 64: 0001: net 490: 29952.. 30015: 1209600.. 1209663: 64: 0001: net 491: 30080.. 30143: 1209728.. 1209791: 64: 0001: net 492: 30208.. 30271: 1209856.. 1209919: 64: 0001: net 493: 30336.. 30399: 1209984.. 1210047: 64: 0001: net 494: 30464.. 30527: 1210112.. 1210175: 64: 0001: net 495: 30592.. 30655: 1210240.. 1210303: 64: 0001: net 496: 30720.. 30783: 1210368.. 1210431: 64: 0001: net 497: 30848.. 30911: 1210496.. 1210559: 64: 0001: net 498: 30976.. 31039: 1210624.. 1210687: 64: 0001: net 499: 31104.. 31167: 1210752.. 1210815: 64: 0001: net 500: 31232.. 31295: 1210880.. 1210943: 64: 0001: net 501: 31360.. 31423: 1211008.. 1211071: 64: 0001: net 502: 31488.. 31551: 1211136.. 1211199: 64: 0001: net 503: 31616.. 31679: 1211264.. 1211327: 64: 0001: net 504: 31744.. 31807: 1211392.. 1211455: 64: 0001: net 505: 31872.. 31935: 1211520.. 1211583: 64: 0001: net 506: 32000.. 32063: 1211648.. 1211711: 64: 0001: net 507: 32128.. 32191: 1211776.. 1211839: 64: 0001: net 508: 32256.. 32319: 1211904.. 1211967: 64: 0001: net 509: 32384.. 32447: 1212032.. 1212095: 64: 0001: net 510: 32512.. 32575: 1212160.. 1212223: 64: 0001: net 511: 32640.. 32703: 1212288.. 1212351: 64: 0001: last,net /mnt/lustre/f130e.sanity: 11 extents found FIEMAP with continuation calls succeeded PASS 130e (73s) == sanity test 130f: FIEMAP (unstriped file) ============= 09:59:47 (1787752787) Filesystem type is: bd00bd0 File size of /mnt/lustre/f130f.sanity is 33554432 (32768 blocks of 1024 bytes) /mnt/lustre/f130f.sanity: 0 extents found PASS 130f (9s) == sanity test 130g: FIEMAP (overstripe file) ============ 09:59:56 (1787752796) 200+0 records in 200+0 records out 209715200 bytes (210 MB, 200 MiB) copied, 26.1085 s, 8.0 MB/s filefrag list 200 extents in file with stripecount 200 PASS 130g (46s) == sanity test 130h: FIEMAP deadlock ===================== 10:00:42 (1787752842) /mnt/lustre/f130h.sanity lmm_stripe_count: 2 lmm_stripe_size: 1048576 lmm_pattern: raid0 lmm_layout_gen: 0 lmm_stripe_offset: 0 obdidx objid objid group 0 28479 0x6f3f 0x280000403 1 22082 0x5642 0x2c0000403 2+0 records in 2+0 records out 2097152 bytes (2.1 MB, 2.0 MiB) copied, 0.318 s, 6.6 MB/s ldlm.namespaces.lustre-OST0000-osc-ffff93f5a5e0f800.lru_size=clear fail_loc=0x80000418 fail_val=5 3+0 records in 3+0 records out 3145728 bytes (3.1 MB, 3.0 MiB) copied, 0.376397 s, 8.4 MB/s extent 0 in offset 0, length 1048576 flags (0x80000001): LAST NET No unwritten extents, extents number 1, sum of lengths 1048576, expected sum 2097152 fail_loc=0 fail_val=0 PASS 130h (15s) == sanity test 130i: FIEMAP (DoM file) =================== 10:00:57 (1787752857) BLKS: 384 PAGE_SIZE: 4096 PAGE_KB: 4 EXPECTED: 512 Filesystem type is: bd00bd0 File size of /mnt/lustre/f130i.sanity is 3141632 (3068 blocks of 1024 bytes) ext: device_logical: physical_offset: length: dev: flags: 0: 0.. 3: 1219216.. 1219219: 4: 0000: net 1: 8.. 11: 1219220.. 1219223: 4: 0000: net 2: 16.. 19: 1219224.. 1219227: 4: 0000: net 3: 24.. 27: 1219228.. 1219231: 4: 0000: net 380: 1504.. 1507: 382660.. 382663: 4: 0000: net 381: 1512.. 1515: 382664.. 382667: 4: 0000: net 382: 1520.. 1523: 382668.. 382671: 4: 0000: net 383: 1528.. 1531: 382672.. 382675: 4: 0000: last,net /mnt/lustre/f130i.sanity: 17 extents found LUN_LEN: 512 FIEMAP on DoM file succeeded PASS 130i (56s) == sanity test 131a: test iov's crossing stripe boundary for writev/readv ========================================================== 10:01:53 (1787752913) PASS 131a (10s) == sanity test 131b: test append writev ================== 10:02:03 (1787752923) /mnt/lustre/f131b.sanity has type file OK /mnt/lustre/f131b.sanity has size 3145728 OK /mnt/lustre/f131b.sanity has type file OK /mnt/lustre/f131b.sanity has size 5767168 OK PASS 131b (9s) == sanity test 131c: test read/write on file w/o objects ========================================================== 10:02:12 (1787752932) Write error: Bad file descriptor (rc = -1, len = 1048576) PASS 131c (8s) == sanity test 131d: test short read ===================== 10:02:20 (1787752940) PASS 131d (7s) == sanity test 131e: test read hitting hole ============== 10:02:27 (1787752947) PASS 131e (7s) == sanity test 133a: Verifying MDT stats ================================================================================================== 10:02:35 (1787752955) mdt.lustre-MDT0000.rename_stats mdt.lustre-MDT0001.rename_stats mdt.lustre-MDT0000.md_stats=clear mdt.lustre-MDT0001.md_stats=clear obdfilter.lustre-OST0000.stats=clear obdfilter.lustre-OST0001.stats=clear mdt.lustre-MDT0000.md_stats=clear mdt.lustre-MDT0001.md_stats=clear /mnt/lustre/d133a.sanity/stats_testdir: total 0 -rw-r--r-- 1 root root 0 Aug 26 10:02 f133a.sanity PASS 133a (22s) == sanity test 133b: Verifying extra MDT stats ============================================================================================ 10:02:57 (1787752977) mdt.lustre-MDT0000.md_stats=clear mdt.lustre-MDT0001.md_stats=clear obdfilter.lustre-OST0000.stats=clear obdfilter.lustre-OST0001.stats=clear mdt.lustre-MDT0000.md_stats=clear mdt.lustre-MDT0001.md_stats=clear PASS 133b (15s) == sanity test 133c: Verifying OST stats ================================================================================================== 10:03:12 (1787752992) striped dir -i1 -c2 -H crush2 /mnt/lustre/d133c.sanity/stats_testdir Waiting for MDT destroys to complete mdt.lustre-MDT0000.md_stats=clear mdt.lustre-MDT0001.md_stats=clear obdfilter.lustre-OST0000.stats=clear obdfilter.lustre-OST0001.stats=clear 1+0 records in 1+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.10302 s, 5.1 MB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB, 1.0 KiB) copied, 0.0272149 s, 37.6 kB/s Waiting for MDT destroys to complete PASS 133c (32s) == sanity test 133d: Verifying rename_stats ================================================================================================== 10:03:44 (1787753024) mdt.lustre-MDT0000.rename_stats mdt.lustre-MDT0001.rename_stats mdt.lustre-MDT0000.rename_stats=clear mdt.lustre-MDT0001.rename_stats=clear total: 512 open/close in 6.02 seconds: 85.12 ops/second source rename dir size: 64K target rename dir size: 4K mdt.lustre-MDT0000.rename_stats= rename_stats: - snapshot_time: 1787753039.099545723 - start_time: 1787753028.025574134 - elapsed_time: 11.073971589 - same_dir: 64KB: { sample: 1, pct: 100, cum_pct: 100 } Check same dir rename stats success mdt.lustre-MDT0000.rename_stats=clear mdt.lustre-MDT0001.rename_stats=clear source rename dir size: 64K target rename dir size: 4K mdt.lustre-MDT0000.rename_stats= rename_stats: - snapshot_time: 1787753043.457656848 - start_time: 1787753041.981388082 - elapsed_time: 1.476268766 - crossdir_src: 64KB: { sample: 1, pct: 100, cum_pct: 100 } - crossdir_tgt: 4KB: { sample: 1, pct: 100, cum_pct: 100 } Check cross dir rename stats success PASS 133d (39s) == sanity test 133e: Verifying OST {read,write}_bytes nid stats =========================================================================== 10:04:24 (1787753064) 42+0 records in 42+0 records out 1376256 bytes (1.4 MB, 1.3 MiB) copied, 0.188578 s, 7.3 MB/s 42+0 records in 42+0 records out 1376256 bytes (1.4 MB, 1.3 MiB) copied, 0.29482 s, 4.7 MB/s PASS 133e (15s) == sanity test 133f: Check reads/writes of client lustre proc files with bad area io ========================================================== 10:04:38 (1787753078) cln..Stopping clients: oleg315-client.virtnet /mnt/lustre (opts:) Stopping client oleg315-client.virtnet /mnt/lustre opts: Stopping clients: oleg315-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg315-server unloading modules via unload_modules_local on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local oleg315-server: modules unloaded. mnt../home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg315-server: detected 4 online CPUs by sysfs oleg315-server: MODOPTS_LIBCFS= oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg315-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Checking servers environments Checking clients oleg315-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg315-server: detected 4 online CPUs by sysfs oleg315-server: MODOPTS_LIBCFS= oleg315-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre Starting client oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre Started clients oleg315-client.virtnet: 192.168.203.115@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93f6830f1800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93f6830f1800.idle_timeout=debug disable quota as required done PASS 133f (229s) == sanity test 133g: Check reads/writes of server lustre proc files with bad area io ========================================================== 10:08:27 (1787753307) cln..Stopping clients: oleg315-client.virtnet /mnt/lustre (opts:) Stopping client oleg315-client.virtnet /mnt/lustre opts: Stopping clients: oleg315-client.virtnet /mnt/lustre2 (opts:) Stopping /mnt/lustre-mds1 (opts:-f) on oleg315-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg315-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg315-server Stopping /mnt/lustre-ost2 (opts:-f) on oleg315-server unloading modules via unload_modules_local on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing unload_modules_local oleg315-server: modules unloaded. mnt../home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ../libcfs/libcfs/libcfs options: 'cpu_npartitions=2' ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg315-server: detected 4 online CPUs by sysfs oleg315-server: MODOPTS_LIBCFS= oleg315-server: Force libcfs to create 2 CPU partitions oleg315-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg315-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg315-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Checking servers environments Checking clients oleg315-client.virtnet environments /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions loading modules on: 'oleg315-server' oleg315-server: oleg315-server.virtnet: executing load_modules_local oleg315-server: Loading modules from /home/green/git/lustre-release/lustre oleg315-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1049: echo: write error: Device or resource busy oleg315-server: detected 4 online CPUs by sysfs oleg315-server: MODOPTS_LIBCFS= oleg315-server: Force libcfs to create 2 CPU partitions Setup mgs, mdt, osts pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0000 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Start ost2: mount -t lustre -o localrecov /dev/mapper/ost2_flakey /mnt/lustre-ost2 seq.cli-lustre-OST0001-super.width=65536 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-OST0001 Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre Starting client oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre Started clients oleg315-client.virtnet: 192.168.203.115@tcp:/lustre on /mnt/lustre type lustre (rw,flock,checksum,encrypt,lazystatfs,lruresize,nolock,statfs_project,nosync_on_close,nouser_fid2path,user_xattr,verbose) Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff93f692896800.idle_timeout=debug osc.lustre-OST0001-osc-ffff93f692896800.idle_timeout=debug disable quota as required done PASS 133g (340s) == sanity test 133h: Proc files should end with newlines ========================================================== 10:14:08 (1787753648) PASS 133h (1371s) == sanity test 134a: Server reclaims locks when reaching lock_reclaim_threshold ========================================================== 10:36:59 (1787755019) - open/close 851 (time 1787755033.92 total 10.01 last 85.02) total: 1000 open/close in 12.06 seconds: 82.92 ops/second fail_loc=0x327 fail_val=500 sleep 10 seconds ... fail_loc=0 fail_val=0 - unlinked 0 (time 1787755059 ; total 0 ; last 0) total: 1000 unlinks in 13 seconds: 76.923080 unlinks/second PASS 134a (62s) == sanity test 134b: Server rejects lock request when reaching lock_limit_mb ========================================================== 10:38:01 (1787755081) ldlm.lock_reclaim_threshold_mb=0 fail_loc=0x328 fail_val=500 debug=+trace Sleep 20 seconds ... fail_loc=0 fail_val=0 oleg315-server: error: set_param: setting /sys/fs/lustre/ldlm/lock_reclaim_threshold_mb=739m: Invalid argument oleg315-server: error: set_param: setting 'ldlm/lock_reclaim_threshold_mb'='739m': Invalid argument pdsh@oleg315-client: oleg315-server: ssh exited with exit code 22 - open/close 404 (time 1787755102.10 total 10.76 last 37.56) - open/close 534 (time 1787755112.11 total 20.77 last 12.99) total: 600 open/close in 21.74 seconds: 27.60 ops/second - unlinked 0 (time 1787755117 ; total 0 ; last 0) total: 600 unlinks in 3 seconds: 200.000000 unlinks/second PASS 134b (46s) == sanity test 134c: Lock memory accounting includes associated structures ========================================================== 10:38:47 (1787755127) per-lock overhead: 896B (lock=512 res=256 ibits=128) ldlm.lock_reclaim_threshold_mb=20 ldlm.lock_limit_mb=50 reclaim_threshold: 23616 locks ≈ 20MB (set 20MB, 100%) lock_limit: 59041 locks ≈ 50MB (set 50MB, 100%) ldlm.lock_limit_mb=1108 ldlm.lock_reclaim_threshold_mb=0 PASS 134c (17s) SKIP: sanity test_135 skipping SLOW test 135 SKIP: sanity test_136 skipping SLOW test 136 == sanity test 140: Check reasonable stack depth (shouldn't LBUG) ============================================================== 10:39:07 (1787755147) striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d140.sanity striped dir -i0 -c2 -H crush2 1 striped dir -i0 -c2 -H crush 2 striped dir -i0 -c2 -H crush 3 striped dir -i0 -c2 -H fnv_1a_64 4 striped dir -i0 -c2 -H crush2 5 striped dir -i0 -c2 -H crush2 6 striped dir -i0 -c2 -H all_char 7 striped dir -i0 -c2 -H all_char 8 striped dir -i0 -c2 -H fnv_1a_64 9 striped dir -i0 -c2 -H crush2 10 striped dir -i0 -c2 -H fnv_1a_64 11 striped dir -i0 -c2 -H all_char 12 striped dir -i0 -c2 -H crush 13 striped dir -i0 -c2 -H crush2 14 striped dir -i0 -c2 -H fnv_1a_64 15 striped dir -i0 -c2 -H all_char 16 striped dir -i0 -c2 -H crush2 17 striped dir -i0 -c2 -H fnv_1a_64 18 striped dir -i0 -c2 -H crush 19 striped dir -i0 -c2 -H all_char 20 striped dir -i0 -c2 -H crush2 21 striped dir -i0 -c2 -H crush2 22 striped dir -i0 -c2 -H fnv_1a_64 23 striped dir -i0 -c2 -H crush 24 striped dir -i0 -c2 -H all_char 25 striped dir -i0 -c2 -H all_char 26 striped dir -i0 -c2 -H fnv_1a_64 27 striped dir -i0 -c2 -H all_char 28 striped dir -i0 -c2 -H crush 29 striped dir -i0 -c2 -H fnv_1a_64 30 striped dir -i0 -c2 -H fnv_1a_64 31 striped dir -i0 -c2 -H fnv_1a_64 32 striped dir -i0 -c2 -H crush2 33 striped dir -i0 -c2 -H crush2 34 striped dir -i0 -c2 -H crush 35 striped dir -i0 -c2 -H fnv_1a_64 36 striped dir -i0 -c2 -H crush2 37 striped dir -i0 -c2 -H crush2 38 striped dir -i0 -c2 -H all_char 39 striped dir -i0 -c2 -H crush 40 striped dir -i0 -c2 -H all_char 41 The symlink depth = 40 open symlink_self returns 40 PASS 140 (57s) == sanity test 150a: truncate/append tests =============== 10:40:04 (1787755204) 1+0 records in 1+0 records out 6096 bytes (6.1 kB, 6.0 KiB) copied, 0.000684151 s, 8.9 MB/s cmp: EOF on /mnt/lustre/f150a.sanity after byte 4096, in line 19 sanity test_150a: @@@@@@ FAIL: /tmp/f150a.sanity /mnt/lustre/f150a.sanity differ Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:19479:test_150a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:19499:main() Dumping lctl log to /tmp/testlogs//sanity.test_150a.*.1787755210.log rsync: chown "/tmp/testlogs/.sanity.test_150a.debug_log.oleg315-server.1787755210.log.xNiFYy" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_150a.dmesg.oleg315-server.1787755210.log.EjilVv" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] Waiting for MDT destroys to complete FAIL 150a (33s) == sanity test 150b: Verify fallocate (prealloc) functionality ========================================================== 10:40:37 (1787755237) keep default fallocate mode: 0 Waiting for MDT destroys to complete PASS 150b (40s) == sanity test 150bb: Verify fallocate modes both zero space ========================================================== 10:41:17 (1787755277) keep default fallocate mode: 0 20+0 records in 20+0 records out 20971520 bytes (21 MB, 20 MiB) copied, 2.31184 s, 9.1 MB/s osd-ldiskfs.lustre-MDT0000.fallocate_zero_blocks=1 osd-ldiskfs.lustre-MDT0001.fallocate_zero_blocks=1 osd-ldiskfs.lustre-OST0000.fallocate_zero_blocks=1 osd-ldiskfs.lustre-OST0001.fallocate_zero_blocks=1 osd-ldiskfs.lustre-MDT0000.fallocate_zero_blocks=0 osd-ldiskfs.lustre-MDT0001.fallocate_zero_blocks=0 osd-ldiskfs.lustre-OST0000.fallocate_zero_blocks=0 osd-ldiskfs.lustre-OST0001.fallocate_zero_blocks=0 Waiting for MDT destroys to complete PASS 150bb (46s) == sanity test 150c: Verify fallocate Size and Blocks ==== 10:42:03 (1787755323) keep default fallocate mode: 0 verify fallocate on PFL file Waiting for MDT destroys to complete PASS 150c (24s) == sanity test 150d: Verify fallocate Size and Blocks - Non zero start ========================================================== 10:42:28 (1787755348) keep default fallocate mode: 0 Waiting for MDT destroys to complete PASS 150d (21s) == sanity test 150e: Verify 60% of available OST space consumed by fallocate ========================================================== 10:42:48 (1787755368) keep default fallocate mode: 0 df before: UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 7252 1264052 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4968 1266336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14672 3592348 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 31324 3575696 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 45996 7168044 1% /mnt/lustre 'fallocate -l 204800k /mnt/lustre/f150e.sanity' succeeded df after fallocate: UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 7252 1264052 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4968 1266336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 117072 3489948 4% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 133724 3473296 4% /mnt/lustre[OST:1] filesystem_summary: 7666232 250796 6963244 4% /mnt/lustre Waiting for MDT destroys to complete df after unlink: UUID 1K-blocks Used Available Use% Mounted on lustre-MDT0000_UUID 1414116 7252 1264052 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 1414116 4968 1266336 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 3833116 14672 3592348 1% /mnt/lustre[OST:0] lustre-OST0001_UUID 3833116 31324 3575696 1% /mnt/lustre[OST:1] filesystem_summary: 7666232 45996 7168044 1% /mnt/lustre Waiting for MDT destroys to complete PASS 150e (32s) == sanity test 150f: Verify fallocate punch functionality ========================================================== 10:43:20 (1787755400) keep default fallocate mode: 0 Verify fallocate punch: Range within the file range 5+0 records in 5+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.0323273 s, 634 kB/s Verify fallocate punch: Range overlapping and less than blocksize 5+0 records in 5+0 records out 20480 bytes (20 kB, 20 KiB) copied, 0.014912 s, 1.4 MB/s Waiting for MDT destroys to complete PASS 150f (24s) == sanity test 150g: Verify fallocate punch on large range ========================================================== 10:43:45 (1787755425) keep default fallocate mode: 0 Verify fallocate punch: Very large Range 256+0 records in 256+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.865242 s, 1.2 MB/s 256+0 records in 256+0 records out 1048576 bytes (1.0 MB, 1.0 MiB) copied, 1.18293 s, 886 kB/s 1024+0 records in 1024+0 records out 4194304 bytes (4.2 MB, 4.0 MiB) copied, 3.93869 s, 1.1 MB/s punch_size = 109043712 size - punch_size: 8192 size - punch_size in blocks: 2 fallocate -p --offset 4096 -l 109043712 /mnt/lustre/f150g.sanity Hole at [4096, 109047808) Waiting for MDT destroys to complete PASS 150g (32s) == sanity test 150h: Verify extend fallocate updates the file size ========================================================== 10:44:17 (1787755457) keep default fallocate mode: 0 PASS 150h (11s) == sanity test 150ia: Verify fallocate zero-range ZERO functionality ========================================================== 10:44:27 (1787755467) keep default fallocate mode: 0 Verify fallocate(zero): range within the file 8+0 records in 8+0 records out 524288 bytes (524 kB, 512 KiB) copied, 0.151715 s, 3.5 MB/s Waiting for MDT destroys to complete PASS 150ia (23s) == sanity test 150ib: Verify fallocate zero-range PREALLOC functionality ========================================================== 10:44:51 (1787755491) keep default fallocate mode: 0 oleg315-server: debugfs 1.47.3-wc2 (11-Nov-2025) filesystem features: Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg ea_inode dirdata large_dir sparse_super large_file huge_file uninit_bg dir_nlink quota project Verify fallocate(zero): range within the file 8+0 records in 8+0 records out 32768 bytes (33 kB, 32 KiB) copied, 0.104035 s, 315 kB/s Waiting for MDT destroys to complete PASS 150ib (26s) == sanity test 150ic: Verify fallocate LARGE zero PREALLOC functionality ========================================================== 10:45:17 (1787755517) SKIP: sanity test_150ic only check on DoM component SKIP 150ic (5s) == sanity test 150id: fallocate that fails must not leave a dirty page behind ========================================================== 10:45:23 (1787755523) keep default fallocate mode: 0 fail_loc=0x2303 Chance of close/open is 1 in 50 Seed set to 5534 truncating to largest ever: 0x3fbbd8 truncating to largest ever: 0x4a9514 truncating to largest ever: 0x4aa105 fallocating to largest ever: 0x50f791 fallocating to largest ever: 0x514470 fallocating to largest ever: 0x517d7b truncating to largest ever: 0x51e419 truncating to largest ever: 0x524024 fallocating to largest ever: 0x5283bd skipping zero size read 010000 1787755915.720285 read 0x0a67b8 thru 0x0b359e (0xcde7 bytes) 020000 1787756307.451263 punch from 0x062766 to 0x06685c (0x40f7 bytes) All operations completed A-OK! fail_loc=0 Waiting for MDT destroys to complete PASS 150id (805s) == sanity test 151: test cache on oss and controls ========================================================================================= 10:58:48 (1787756328) striped dir -i1 -c2 -H all_char /mnt/lustre/d151.sanity-check 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0207322 s, 790 kB/s BEFORE:375379 AFTER:375383 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0923649 s, 177 kB/s BEFORE:10245 AFTER:10249 fail_loc=0x609 3+0 records in 3+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0344341 s, 357 kB/s fail_loc=0 PASS 151 (29s) == sanity test 152: test read/write with enomem ====================================================================================== 10:59:17 (1787756357) fail_loc=0x80000226 1+0 records in 1+0 records out 6096 bytes (6.1 kB, 6.0 KiB) copied, 0.000315702 s, 19.3 MB/s fail_loc=0 fail_loc=0x80000226 cmp: EOF on /mnt/lustre/f152.sanity after byte 4096, in line 12 sanity test_152: @@@@@@ FAIL: cmp failed Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20175:test_152() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20180:main() Dumping lctl log to /tmp/testlogs//sanity.test_152.*.1787756361.log rsync: chown "/tmp/testlogs/.sanity.test_152.debug_log.oleg315-server.1787756361.log.NMPrUX" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_152.dmesg.oleg315-server.1787756361.log.poaY2f" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 152 (11s) == sanity test 153: test if fdatasync does not crash ================================================================================= 10:59:28 (1787756368) PASS 153 (8s) == sanity test 154A: lfs path2fid and fid2path basic checks ========================================================== 10:59:36 (1787756376) /mnt/lustre [0x2000013a1:0x6ad:0x0] /mnt/lustre/// [0x2000013a1:0x6ad:0x0] /mnt/lustre/f154A.sanity [0x2000013a1:0x6ad:0x0] lfs fid2path: cannot resolve mount point for '/mnt/lustre_wrong': No such device PASS 154A (10s) == sanity test 154B: verify the ll_decode_linkea tool ==== 10:59:46 (1787756386) PFID: [0x2000013a1:0x6ae:0x0], name: f154B.sanity PASS 154B (9s) == sanity test 154C: lfs fid2path on OST FID ============= 10:59:56 (1787756396) 8+0 records in 8+0 records out 8388608 bytes (8.4 MB, 8.0 MiB) copied, 1.48608 s, 5.6 MB/s ost_fid=0x280000403:0x7450:0x0 with outpath=/mnt/lustre/f154C.sanity ost_fid=0x2c0000403:0x5ad2:0x0 with outpath=/mnt/lustre/f154C.sanity Waiting for MDT destroys to complete PASS 154C (26s) == sanity test 154a: Open-by-FID ========================= 11:00:21 (1787756421) stat fid [0x2000013a1:0x6b2:0x0] File: /mnt/lustre/.lustre/fid/[0x2000013a1:0x6b2:0x0] Size: 158 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115272381367986 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 11:00:23.000000000 -0400 Modify: 2026-08-26 11:00:23.000000000 -0400 Change: 2026-08-26 11:00:23.000000000 -0400 Birth: 2026-08-26 11:00:23.000000000 -0400 touch fid [0x2000013a1:0x6b2:0x0] write to fid [0x2000013a1:0x6b2:0x0] read fid [0x2000013a1:0x6b2:0x0] append write to fid [0x2000013a1:0x6b2:0x0] rename fid [0x2000013a1:0x6b2:0x0] mv: cannot move '/mnt/lustre/.lustre/fid/[0x2000013a1:0x6b2:0x0]' to '/mnt/lustre/f154a.sanity.1': Operation not permitted mv: cannot move '/mnt/lustre/f154a.sanity.1' to '/mnt/lustre/.lustre/fid/[0x2000013a1:0x6b2:0x0]': Operation not permitted truncate fid [0x2000013a1:0x6b2:0x0] link fid [0x2000013a1:0x6b2:0x0] uid=500(sanityusr) gid=500(sanityusr) groups=500(sanityusr) setfacl fid [0x2000013a1:0x6b2:0x0] getfacl fid [0x2000013a1:0x6b2:0x0] getfacl: Removing leading '/' from absolute path names # file: mnt/lustre/.lustre/fid/[0x2000013a1:0x6b2:0x0] # owner: root # group: root user::rw- user:sanityusr:rwx group::r-- mask::rwx other::r-- unlink fid [0x2000013a1:0x6b2:0x0] unlink: cannot unlink '/mnt/lustre/.lustre/fid/[0x2000013a1:0x6b2:0x0]': Operation not permitted mknod fid [0x2000013a1:0x6b2:0x0] mknod: '/mnt/lustre/.lustre/fid/[0x2000013a1:0x6b2:0x0]': Operation not permitted stat non-exist fid [0xf00000400:0x1:0x0] stat: cannot statx '/mnt/lustre/.lustre/fid/[0xf00000400:0x1:0x0]': No such file or directory write to non-exist fid [0xf00000400:0x1:0x0] /home/green/git/lustre-release/lustre/tests/sanity.sh: line 20232: /mnt/lustre/.lustre/fid/[0xf00000400:0x1:0x0]: Operation not permitted link new fid [0xf00000400:0x1:0x0] ln: failed to create hard link '/mnt/lustre/.lustre/fid/[0xf00000400:0x1:0x0]' => '/mnt/lustre/f154a.sanity': Operation not permitted ls [0x2400013a1:0xe:0x0] f154a.sanity touch [0x2400013a1:0xe:0x0]/f154a.sanity.1 touch /mnt/lustre/.lustre/fid/f154a.sanity touch: setting times of '/mnt/lustre/.lustre/fid/f154a.sanity': No such file or directory setxattr to /mnt/lustre/.lustre/fid listxattr for /mnt/lustre/.lustre/fid getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/.lustre/fid trusted.lma=0sAAAAAAAAAAACAAAAAgAAAAIAAAAAAAAA trusted.name1="value1" delxattr from /mnt/lustre/.lustre/fid touch invalid fid: /mnt/lustre/.lustre/fid/[0x200000400:0x2:0x3] touch: setting times of '/mnt/lustre/.lustre/fid/[0x200000400:0x2:0x3]': No such file or directory touch non-normal fid: /mnt/lustre/.lustre/fid/[0x1:0x2:0x0] touch: setting times of '/mnt/lustre/.lustre/fid/[0x1:0x2:0x0]': No such file or directory rename d154a.sanity to /mnt/lustre/.lustre/fid rename '/mnt/lustre/d154a.sanity' returned -1: Operation not permitted change mode of /mnt/lustre/.lustre/fid to 777 restore mode of /mnt/lustre/.lustre/fid to 100 Succeed in opening file "/mnt/lustre/f154a.sanity-2"(flags=O_LOV_DELAY_CREATE) cp /etc/passwd /mnt/lustre/.lustre/fid/[0x2000013a1:0x6ba:0x0] cp /etc/passwd /mnt/lustre/f154a.sanity-2 diff /etc/passwd /mnt/lustre/.lustre/fid/[0x2000013a1:0x6ba:0x0] rm: cannot remove '/mnt/lustre/.lustre/lost+found/MDT0000': Operation not permitted rm: cannot remove '/mnt/lustre/.lustre/lost+found/MDT0001': Operation not permitted rm: cannot remove '/mnt/lustre/.lustre/fid': Operation not permitted touch: setting times of '/mnt/lustre/.lustre/file': No such file or directory mkdir: cannot create directory '/mnt/lustre/.lustre/dir': Operation not permitted PASS 154a (12s) == sanity test 154b: Open-by-FID for remote directory ==== 11:00:34 (1787756434) stat fid [0x2400013a1:0x13:0x0] File: /mnt/lustre/.lustre/fid/[0x2400013a1:0x13:0x0] Size: 158 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 162129670890848275 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2026-08-26 11:00:36.000000000 -0400 Modify: 2026-08-26 11:00:36.000000000 -0400 Change: 2026-08-26 11:00:36.000000000 -0400 Birth: 2026-08-26 11:00:35.000000000 -0400 touch fid [0x2400013a1:0x13:0x0] write to fid [0x2400013a1:0x13:0x0] read fid [0x2400013a1:0x13:0x0] append write to fid [0x2400013a1:0x13:0x0] rename fid [0x2400013a1:0x13:0x0] mv: cannot move '/mnt/lustre/.lustre/fid/[0x2400013a1:0x13:0x0]' to '/mnt/lustre/d154b.sanity/remote_dir/f154b.sanity.1': Operation not permitted mv: cannot move '/mnt/lustre/d154b.sanity/remote_dir/f154b.sanity.1' to '/mnt/lustre/.lustre/fid/[0x2400013a1:0x13:0x0]': Operation not permitted truncate fid [0x2400013a1:0x13:0x0] link fid [0x2400013a1:0x13:0x0] uid=500(sanityusr) gid=500(sanityusr) groups=500(sanityusr) setfacl fid [0x2400013a1:0x13:0x0] getfacl fid [0x2400013a1:0x13:0x0] getfacl: Removing leading '/' from absolute path names # file: mnt/lustre/.lustre/fid/[0x2400013a1:0x13:0x0] # owner: root # group: root user::rw- user:sanityusr:rwx group::r-- mask::rwx other::r-- unlink fid [0x2400013a1:0x13:0x0] unlink: cannot unlink '/mnt/lustre/.lustre/fid/[0x2400013a1:0x13:0x0]': Operation not permitted mknod fid [0x2400013a1:0x13:0x0] mknod: '/mnt/lustre/.lustre/fid/[0x2400013a1:0x13:0x0]': Operation not permitted stat non-exist fid [0xf00000400:0x1:0x0] stat: cannot statx '/mnt/lustre/.lustre/fid/[0xf00000400:0x1:0x0]': No such file or directory write to non-exist fid [0xf00000400:0x1:0x0] /home/green/git/lustre-release/lustre/tests/sanity.sh: line 20232: /mnt/lustre/.lustre/fid/[0xf00000400:0x1:0x0]: Operation not permitted link new fid [0xf00000400:0x1:0x0] ln: failed to create hard link '/mnt/lustre/.lustre/fid/[0xf00000400:0x1:0x0]' => '/mnt/lustre/d154b.sanity/remote_dir/f154b.sanity': Operation not permitted ls [0x2400013a1:0x15:0x0] f154b.sanity touch [0x2400013a1:0x15:0x0]/f154b.sanity.1 touch /mnt/lustre/.lustre/fid/f154b.sanity touch: setting times of '/mnt/lustre/.lustre/fid/f154b.sanity': No such file or directory setxattr to /mnt/lustre/.lustre/fid listxattr for /mnt/lustre/.lustre/fid getfattr: Removing leading '/' from absolute path names # file: mnt/lustre/.lustre/fid trusted.lma=0sAAAAAAAAAAACAAAAAgAAAAIAAAAAAAAA trusted.name1="value1" delxattr from /mnt/lustre/.lustre/fid touch invalid fid: /mnt/lustre/.lustre/fid/[0x200000400:0x2:0x3] touch: setting times of '/mnt/lustre/.lustre/fid/[0x200000400:0x2:0x3]': No such file or directory touch non-normal fid: /mnt/lustre/.lustre/fid/[0x1:0x2:0x0] touch: setting times of '/mnt/lustre/.lustre/fid/[0x1:0x2:0x0]': No such file or directory rename d154b.sanity to /mnt/lustre/.lustre/fid rename '/mnt/lustre/d154b.sanity/remote_dir/d154b.sanity' returned -1: Operation not permitted change mode of /mnt/lustre/.lustre/fid to 777 restore mode of /mnt/lustre/.lustre/fid to 100 Succeed in opening file "/mnt/lustre/d154b.sanity/remote_dir/f154b.sanity-2"(flags=O_LOV_DELAY_CREATE) cp /etc/passwd /mnt/lustre/.lustre/fid/[0x2400013a1:0x18:0x0] cp /etc/passwd /mnt/lustre/d154b.sanity/remote_dir/f154b.sanity-2 diff /etc/passwd /mnt/lustre/.lustre/fid/[0x2400013a1:0x18:0x0] PASS 154b (12s) == sanity test 154c: lfs path2fid and fid2path multiple arguments ========================================================== 11:00:46 (1787756446) PASS 154c (8s) == sanity test 154d: Verify open file fid ================ 11:00:53 (1787756453) mdt.lustre-MDT0000.exports.192.168.203.15@tcp.open_files= [0x200000002:0x1:0x0] [0x200000002:0x3:0x0] [0x200000002:0x2:0x0] [0x2000013a1:0x6c6:0x0] mdt.lustre-MDT0001.exports.192.168.203.15@tcp.open_files=[0x2400013a1:0x1:0x0] PASS 154d (8s) == sanity test 154db: fid is stored in dir entries ======= 11:01:01 (1787756461) striped dir -i0 -c1 -H crush2 /mnt/lustre/d154db.sanity total: 100 open/close in 2.53 seconds: 39.58 ops/second PASS 154db (13s) == sanity test 154e: .lustre is not returned by readdir == 11:01:14 (1787756474) PASS 154e (9s) == sanity test 154ea: .lustre is not returned by readdir (2) ========================================================== 11:01:24 (1787756484) PASS 154ea (73s) == sanity test 154f: get parent fids by reading link ea == 11:02:37 (1787756557) [0x2000013a1:0x73c:0x0]/f154f.sanity [0x2000013a1:0x73d:0x0]/link [0x2000013a1:0x73c:0x0]/f154f.sanity [0x2000013a1:0x73d:0x0]/link [0x2000013a1:0x73c:0x0]/f154f.sanity [0x2000013a1:0x73d:0x0]/link [0x2000013a1:0x73c:0x0]/f154f.sanity [0x2000013a1:0x73d:0x0]/link [0x200000007:0x1:0x0]/f llite.lustre-ffff93f692896800.xattr_cache=1 [0x2000013a1:0x73d:0x0]/link [0x2000013a1:0x73d:0x0]/f154f.sanity.moved PASS 154f (10s) == sanity test 154g: various llapi FID tests ============= 11:02:48 (1787756568) Starting test test10 at 1787756570 Finishing test test10 at 1787756570 Starting test test11 at 1787756570 Finishing test test11 at 1787756570 Starting test test12 at 1787756570 Finishing test test12 at 1787756570 Starting test test20 at 1787756570 Finishing test test20 at 1787757613 Starting test test30 at 1787757797 Was able to store 155 links in the EA Finishing test test30 at 1787757842 Starting test test31 at 1787757868 Finishing test test31 at 1787757868 Starting test test40 at 1787757868 Finishing test test40 at 1787757869 Starting test test41 at 1787757869 Finishing test test41 at 1787757869 Starting test test42 at 1787757869 Finishing test test42 at 1787757876 PASS 154g (1323s) == sanity test 154h: Verify interactive path2fid ========= 11:24:51 (1787757891) [0x2000013a1:0xf9b:0x0] PASS 154h (8s) == sanity test 154i: fid2path for path longer than PATH_MAX ========================================================== 11:24:59 (1787757899) mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst0' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst1' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst2' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst3' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst4' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst5' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst6' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst7' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst8' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst9' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst10' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst11' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst12' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst13' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst14' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst15' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst16' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst17' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst18' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst19' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst20' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst21' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst22' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst23' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst24' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst25' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst26' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst27' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst28' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst29' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst30' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst31' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst32' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst33' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst34' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst35' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst36' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst37' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst38' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst39' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst40' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst41' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst42' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst43' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst44' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst45' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst46' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst47' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst48' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst49' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst50' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst51' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst52' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst53' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst54' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst55' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst56' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst57' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst58' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst59' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst60' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst61' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst62' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst63' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst64' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst65' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst66' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst67' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst68' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst69' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst70' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst71' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst72' mkdir: created directory 'thislongpathnameisforaverydeepsubdirthatwewanttotestagainst73' Path for fid [0x2000013a1:0xfe5:0x0] is: /mnt/lustre/d154i.sanity/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst0/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst1/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst2/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst3/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst4/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst5/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst6/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst7/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst8/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst9/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst10/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst11/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst12/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst13/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst14/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst15/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst16/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst17/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst18/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst19/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst20/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst21/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst22/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst23/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst24/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst25/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst26/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst27/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst28/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst29/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst30/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst31/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst32/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst33/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst34/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst35/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst36/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst37/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst38/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst39/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst40/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst41/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst42/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst43/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst44/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst45/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst46/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst47/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst48/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst49/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst50/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst51/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst52/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst53/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst54/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst55/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst56/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst57/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst58/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst59/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst60/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst61/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst62/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst63/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst64/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst65/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst66/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst67/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst68/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst69/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst70/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst71/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst72/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst73 Path for fid [0x2000013a1:0xfe5:0x0] is: /mnt/lustre/d154i.sanity/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst0/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst1/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst2/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst3/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst4/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst5/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst6/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst7/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst8/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst9/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst10/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst11/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst12/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst13/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst14/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst15/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst16/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst17/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst18/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst19/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst20/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst21/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst22/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst23/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst24/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst25/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst26/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst27/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst28/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst29/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst30/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst31/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst32/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst33/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst34/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst35/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst36/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst37/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst38/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst39/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst40/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst41/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst42/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst43/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst44/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst45/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst46/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst47/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst48/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst49/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst50/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst51/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst52/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst53/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst54/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst55/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst56/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst57/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst58/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst59/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst60/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst61/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst62/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst63/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst64/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst65/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst66/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst67/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst68/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst69/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst70/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst71/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst72/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst73 PASS 154i (45s) == sanity test 154j: fid2path for long path crossing MDT boundary ========================================================== 11:25:44 (1787757944) Path for fid [0x2400013a1:0x5d:0x0] is: /mnt/lustre/d154j.sanity/remote/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst0/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst1/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst2/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst3/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst4/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst5/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst6/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst7/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst8/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst9/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst10/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst11/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst12/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst13/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst14/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst15/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst16/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst17/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst18/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst19/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst20/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst21/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst22/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst23/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst24/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst25/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst26/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst27/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst28/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst29/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst30/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst31/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst32/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst33/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst34/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst35/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst36/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst37/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst38/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst39/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst40/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst41/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst42/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst43/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst44/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst45/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst46/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst47/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst48/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst49/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst50/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst51/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst52/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst53/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst54/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst55/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst56/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst57/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst58/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst59/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst60/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst61/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst62/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst63/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst64/thislongpathnameisforaverydeepsubdirthatwewanttotestagainst65 PASS 154j (29s) == sanity test 155a: Verify small file correctness: read cache:on write_cache:on ========================================================== 11:26:13 (1787757973) 1+0 records in 1+0 records out 6096 bytes (6.1 kB, 6.0 KiB) copied, 0.00299404 s, 2.0 MB/s cmp: EOF on /mnt/lustre/f155a.sanity after byte 4096, in line 17 sanity test_155a: @@@@@@ FAIL: /tmp/f155a.sanity /mnt/lustre/f155a.sanity differ Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20749:test_155_small_load() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20819:test_155a() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20823:main() Dumping lctl log to /tmp/testlogs//sanity.test_155a.*.1787757983.log rsync: chown "/tmp/testlogs/.sanity.test_155a.debug_log.oleg315-server.1787757983.log.GBu2Qe" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_155a.dmesg.oleg315-server.1787757983.log.GXPHYY" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 155a (17s) == sanity test 155b: Verify small file correctness: read cache:on write_cache:off ========================================================== 11:26:31 (1787757991) 1+0 records in 1+0 records out 6096 bytes (6.1 kB, 6.0 KiB) copied, 0.000950525 s, 6.4 MB/s cmp: EOF on /mnt/lustre/f155b.sanity after byte 4096, in line 25 sanity test_155b: @@@@@@ FAIL: /tmp/f155b.sanity /mnt/lustre/f155b.sanity differ Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20749:test_155_small_load() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20835:test_155b() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20839:main() Dumping lctl log to /tmp/testlogs//sanity.test_155b.*.1787758000.log rsync: chown "/tmp/testlogs/.sanity.test_155b.debug_log.oleg315-server.1787758000.log.q0UjWM" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_155b.dmesg.oleg315-server.1787758000.log.Lwf5kp" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 155b (13s) == sanity test 155c: Verify small file correctness: read cache:off write_cache:on ========================================================== 11:26:44 (1787758004) 1+0 records in 1+0 records out 6096 bytes (6.1 kB, 6.0 KiB) copied, 0.000379441 s, 16.1 MB/s cmp: EOF on /mnt/lustre/f155c.sanity after byte 4096, in line 19 sanity test_155c: @@@@@@ FAIL: /tmp/f155c.sanity /mnt/lustre/f155c.sanity differ Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20749:test_155_small_load() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20851:test_155c() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20855:main() Dumping lctl log to /tmp/testlogs//sanity.test_155c.*.1787758016.log rsync: chown "/tmp/testlogs/.sanity.test_155c.debug_log.oleg315-server.1787758016.log.MAQWlq" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_155c.dmesg.oleg315-server.1787758016.log.2osYae" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 155c (15s) == sanity test 155d: Verify small file correctness: read cache:off write_cache:off ========================================================== 11:27:00 (1787758020) 1+0 records in 1+0 records out 6096 bytes (6.1 kB, 6.0 KiB) copied, 0.000330225 s, 18.5 MB/s cmp: EOF on /mnt/lustre/f155d.sanity after byte 4096, in line 15 sanity test_155d: @@@@@@ FAIL: /tmp/f155d.sanity /mnt/lustre/f155d.sanity differ Trace dump: = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7549:error() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20749:test_155_small_load() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20867:test_155d() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7926:run_one() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7989:run_one_logged() = /home/green/git/lustre-release/lustre/tests/test-framework.sh:7788:run_test() = /home/green/git/lustre-release/lustre/tests/sanity.sh:20871:main() Dumping lctl log to /tmp/testlogs//sanity.test_155d.*.1787758028.log rsync: chown "/tmp/testlogs/.sanity.test_155d.debug_log.oleg315-server.1787758028.log.rEKjyo" failed: Operation not permitted (1) rsync: chown "/tmp/testlogs/.sanity.test_155d.dmesg.oleg315-server.1787758028.log.VOzbro" failed: Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1668) [generator=3.1.3] FAIL 155d (12s) == sanity test 155e: Verify big file correctness: read cache:on write_cache:on ========================================================== 11:27:11 (1787758031) Waiting for MDT destroys to complete OST kbytes available: 3600612 3583972 Min free space: OST 1: 3583972 Max free space: OST 0: 3600612 OSS cache size: 65536 KB Large file size: 131072 KB 1024+0 records in 1024+0 records out 134217728 bytes (134 MB, 128 MiB) copied, 4.45873 s, 30.1 MB/s -rw-r--r-- 1 root root 128M Aug 26 11:27 /mnt/lustre/f155e.sanity -rw-r--r-- 1 root root 128M Aug 26 11:27 /tmp/f155e.sanity PASS 155e (49s) == sanity test 155f: Verify big file correctness: read cache:on write_cache:off ========================================================== 11:28:00 (1787758080) Waiting for MDT destroys to complete OST kbytes available: 3600612 3583972 Min free space: OST 1: 3583972 Max free space: OST 0: 3600612 OSS cache size: 65536 KB Large file size: 131072 KB 1024+0 records in 1024+0 records out 134217728 bytes (134 MB, 128 MiB) copied, 3.67807 s, 36.5 MB/s -rw-r--r-- 1 root root 128M Aug 26 11:28 /mnt/lustre/f155f.sanity -rw-r--r-- 1 root root 128M Aug 26 11:28 /tmp/f155f.sanity PASS 155f (53s) == sanity test 155g: Verify big file correctness: read cache:off write_cache:on ========================================================== 11:28:53 (1787758133) Waiting for MDT destroys to complete OST kbytes available: 3600612 3583972 Min free space: OST 1: 3583972 Max free space: OST 0: 3600612 OSS cache size: 65536 KB Large file size: 131072 KB 1024+0 records in 1024+0 records out 134217728 bytes (134 MB, 128 MiB) copied, 5.23647 s, 25.6 MB/s -rw-r--r-- 1 root root 128M Aug 26 11:29 /mnt/lustre/f155g.sanity -rw-r--r-- 1 root root 128M Aug 26 11:29 /tmp/f155g.sanity PASS 155g (54s) == sanity test 155h: Verify big file correctness: read cache:off write_cache:off ========================================================== 11:29:47 (1787758187) Waiting for MDT destroys to complete OST kbytes available: 3600612 3583972 Min free space: OST 1: 3583972 Max free space: OST 0: 3600612 OSS cache size: 65536 KB Large file size: 131072 KB 1024+0 records in 1024+0 records out 134217728 bytes (134 MB, 128 MiB) copied, 3.20988 s, 41.8 MB/s -rw-r--r-- 1 root root 128M Aug 26 11:30 /mnt/lustre/f155h.sanity -rw-r--r-- 1 root root 128M Aug 26 11:30 /tmp/f155h.sanity PASS 155h (54s) == sanity test 156: Verification of tunables ============= 11:30:42 (1787758242) striped dir -i0 -c2 -H all_char /mnt/lustre/d156.sanity-check 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0256699 s, 638 kB/s BEFORE:506458 AFTER:506462 4+0 records in 4+0 records out 16384 bytes (16 kB, 16 KiB) copied, 0.0489525 s, 335 kB/s BEFORE:10260 AFTER:10264 Turn on read and write cache Write data and read it back. Read should be satisfied from the cache. 3+0 records in 3+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0334036 s, 368 kB/s cache hits: before: 28690, after: 28693 Read again; it should be satisfied from the cache. cache hits:: before: 28693, after: 28696 Turn off the read cache and turn on the write cache Read again; it should be satisfied from the cache. cache hits:: before: 28696, after: 28699 Write data and read it back. Read should be satisfied from the cache. 3+0 records in 3+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0123216 s, 997 kB/s cache hits:: before: 28699, after: 28702 Turn off read and write cache Write data and read it back It should not be satisfied from the cache. 3+0 records in 3+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.063436 s, 194 kB/s cache hits:: before: 28702, after: 28702 Turn on the read cache and turn off the write cache Write data and read it back It should not be satisfied from the cache. 3+0 records in 3+0 records out 12288 bytes (12 kB, 12 KiB) copied, 0.0255329 s, 481 kB/s cache hits:: before: 28702, after: 28702 Read again; it should be satisfied from the cache. cache hits:: before: 28702, after: 28705 PASS 156 (95s) == sanity test 157a: llapi pool pinning API tests ======== 11:32:17 (1787758337) test0 @1787758340: llapi_pool_pin_file/llapi_pool_unpin_file basic behavior ... pass test1 @1787758340: llapi_pool_pin_file already pinned ......................... pass test2 @1787758340: pool pins coexist with PCC hsm pins ........................ pass test3 @1787758340: llapi_pool_pin_fd, llapi_pool_unpin_fd and llapi_pool_is_pinned_fd pass test4 @1787758340: llapi_pool_pin_fid, llapi_pool_unpin_fid and llapi_pool_is_pinned_fid pass PASS 157a (9s) == sanity test 157b: lustre.pin inheritance on create ==== 11:32:27 (1787758347) striped dir -i1 -c2 -H crush /mnt/lustre/d157b.sanity PASS 157b (11s) == sanity test 160a: changelog sanity ==================== 11:32:38 (1787758358) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl1 cl1' striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d160a.sanity/pics/2008/zachy lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records verifying changelog mask mdd.lustre-MDT0000.changelog_mask=-MKDIR mdd.lustre-MDT0001.changelog_mask=-MKDIR mdd.lustre-MDT0000.changelog_mask=-CLOSE mdd.lustre-MDT0001.changelog_mask=-CLOSE striped dir -i0 -c2 -H all_char /mnt/lustre/d160a.sanity/pics/zach/sofia mdd.lustre-MDT0000.changelog_mask=+MKDIR mdd.lustre-MDT0001.changelog_mask=+MKDIR mdd.lustre-MDT0000.changelog_mask=+CLOSE mdd.lustre-MDT0001.changelog_mask=+CLOSE striped dir -i0 -c2 -H crush2 /mnt/lustre/d160a.sanity/pics/2008/sofia verifying target fid verifying parent fid getting records for cl1 current_index: 9 ID index (idle) mask cl1 4 (10) lustre-MDT0000: clear the changelog for cl1 to record #7 verifying user clear: 4 + 3 == 7 lustre-MDT0001.3 01CREAT 15:32:48.767696147 2026.08.26 0x0 t=[0x2400013a1:0x61:0x0] j=cp.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x2000013a1:0x1001:0x0] pic1.jpg lustre-MDT0001.4 11CLOSE 15:32:48.845918431 2026.08.26 0xc2 t=[0x2400013a1:0x61:0x0] j=cp.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp lustre-MDT0001.5 01CREAT 15:32:52.447081680 2026.08.26 0x0 t=[0x2400013a1:0x62:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x2000013a1:0x1001:0x0] file lustre-MDT0001.6 13TRUNC 15:32:56.005179392 2026.08.26 0xe t=[0x2400013a1:0x62:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x2400013a0:0x2e:0x0] lustre-MDT0001.7 11CLOSE 15:32:56.054759757 2026.08.26 0x242 t=[0x2400013a1:0x62:0x0] j=bash.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp verifying user min purge: 7 + 1 == 8 lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0001: clear the changelog for cl1 of all records Stopping /mnt/lustre-mds1 (opts:) on oleg315-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 verifying index survives MDT restart: 9 == 9 verifying users from this test are deregistered lustre-MDT0000: clear the changelog for cl1 of all records lustre-MDT0000: Deregistered changelog user #1 lustre-MDT0001: clear the changelog for cl1 of all records lustre-MDT0001: Deregistered changelog user #1 current_index: 9 ID index (idle) mask other changelog users; can't verify off lustre-MDT0001: changelog user 'cl1' not found lustre-MDT0000: changelog user 'cl1' not found PASS 160a (63s) == sanity test 160b: Verify that very long rename doesn't crash in changelog ========================================================== 11:33:42 (1787758422) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl2 cl2' creating very long named file renaming very long named file lustre-MDT0000.12 08RENME 15:33:50.011376444 2026.08.26 0x0 t=[0:0x0:0x0] j=mv.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x200000007:0x1:0x0] bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb s=[0x2000013a1:0x1005:0x0] sp=[0x200000007:0x1:0x0] aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa lustre-MDT0001: clear the changelog for cl2 of all records lustre-MDT0001: Deregistered changelog user #2 lustre-MDT0000: clear the changelog for cl2 of all records lustre-MDT0000: Deregistered changelog user #2 PASS 160b (21s) == sanity test 160c: verify that changelog log catch the truncate event ========================================================== 11:34:03 (1787758443) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl3 cl3' mdd.lustre-MDT0000.changelog_mask=-TRUNC mdd.lustre-MDT0001.changelog_mask=-TRUNC mdd.lustre-MDT0000.changelog_mask=+TRUNC mdd.lustre-MDT0001.changelog_mask=+TRUNC lustre-MDT0001.8 02MKDIR 15:34:12.479691485 2026.08.26 0x0 t=[0x2400013a1:0x63:0x0] j=mkdir.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x200000007:0x1:0x0] d160c.sanity lustre-MDT0001.9 01CREAT 15:34:12.581927007 2026.08.26 0x0 t=[0x2400013a1:0x64:0x0] j=mcreate.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x2400013a1:0x63:0x0] foo_160c lustre-MDT0001.10 14SATTR 15:34:14.454270060 2026.08.26 0xe t=[0x2400013a1:0x64:0x0] j=truncate.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x2400013a1:0x63:0x0] lustre-MDT0001.11 13TRUNC 15:34:16.207630957 2026.08.26 0xe t=[0x2400013a1:0x64:0x0] j=truncate.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x2400013a1:0x63:0x0] lustre-MDT0001: clear the changelog for cl3 of all records lustre-MDT0001: Deregistered changelog user #3 lustre-MDT0000: clear the changelog for cl3 of all records lustre-MDT0000: Deregistered changelog user #3 PASS 160c (24s) == sanity test 160d: verify that changelog log catch the migrate event ========================================================== 11:34:27 (1787758467) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl4 cl4' lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0000.15 20MIGRT 15:34:34.318378376 2026.08.26 0x0 t=[0x2000013a1:0x1007:0x0] j=lfs.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x2000013a1:0x1006:0x0] migrate_dir s=[0x2400013a1:0x65:0x0] sp=[0x2000013a1:0x1006:0x0] migrate_dir lustre-MDT0000.16 12LYOUT 15:34:34.539343632 2026.08.26 0x0 t=[0x2000013a1:0x1007:0x0] j=lfs.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x2400013a1:0x63:0x0] lustre-MDT0001: clear the changelog for cl4 of all records lustre-MDT0001: Deregistered changelog user #4 lustre-MDT0000: clear the changelog for cl4 of all records lustre-MDT0000: Deregistered changelog user #4 PASS 160d (19s) == sanity test 160e: changelog negative testing (should return errors) ========================================================== 11:34:46 (1787758486) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl5 cl5' pdsh@oleg315-client: oleg315-server: ssh exited with exit code 4 deregister an existing changelog user usage: {--device MDTNAME} changelog_deregister [ID|clID] [--help|-h] [--user|-u USERNAME] run after connecting to device --device oleg315-server: error: changelog_deregister: User not found pdsh@oleg315-client: oleg315-server: ssh exited with exit code 2 lfs changelog_clear: cannot purge records for 'cl5': Invalid argument (22) changelog_clear: record out of range: 1000000000 lustre-MDT0001: clear the changelog for cl5 of all records lustre-MDT0001: Deregistered changelog user #5 lustre-MDT0000: clear the changelog for cl5 of all records lustre-MDT0000: Deregistered changelog user #5 PASS 160e (21s) == sanity test 160f: changelog garbage collect (timestamped users) ========================================================== 11:35:07 (1787758507) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl6' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl6 cl7 cl6 cl7' striped dir -i0 -c2 -H all_char /mnt/lustre/d160f.sanity 1787758521: creating first dirs mdd.lustre-MDT0000.changelog_max_idle_time=15 mdd.lustre-MDT0001.changelog_max_idle_time=15 mdd.lustre-MDT0000.changelog_gc=1 mdd.lustre-MDT0001.changelog_gc=1 mdd.lustre-MDT0000.changelog_min_gc_interval=2 mdd.lustre-MDT0001.changelog_min_gc_interval=2 mdd.lustre-MDT0000.changelog_min_free_cat_entries=3 mdd.lustre-MDT0001.changelog_min_free_cat_entries=3 1787758533: sleep1 7/15s fail_loc=0x1313 fail_val=3 lustre-MDT0000: clear the changelog for cl6 to record #18 mds1: verifying user cl6 clear: 16 + 2 == 18 lustre-MDT0001: clear the changelog for cl6 to record #14 mds2: verifying user cl6 clear: 12 + 2 == 14 1787758550: sleep2 -11/15s sleep: invalid option -- '1' Try 'sleep --help' for more information. 1787758550: creating 4 dirs 1787758550: sleep3 4/15s pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 mds1: 1787758560 verify rec 18+1 == 19 mds2: 1787758564 verify rec 14+1 == 15 mdd.lustre-MDT0000.changelog_min_free_cat_entries=2 mdd.lustre-MDT0001.changelog_min_free_cat_entries=2 mdd.lustre-MDT0000.changelog_min_gc_interval=3600 mdd.lustre-MDT0001.changelog_min_gc_interval=3600 mdd.lustre-MDT0000.changelog_gc=1 mdd.lustre-MDT0001.changelog_gc=1 mdd.lustre-MDT0000.changelog_max_idle_time=2592000 mdd.lustre-MDT0001.changelog_max_idle_time=2592000 lustre-MDT0001: changelog user 'cl7' not found lustre-MDT0000: changelog user 'cl7' not found lustre-MDT0001: clear the changelog for cl6 of all records lustre-MDT0001: Deregistered changelog user #6 lustre-MDT0000: clear the changelog for cl6 of all records lustre-MDT0000: Deregistered changelog user #6 PASS 160f (79s) == sanity test 160g: changelog garbage collect on idle records ========================================================== 11:36:26 (1787758586) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl8 cl8' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl8 cl9 cl8 cl9' striped dir -i0 -c2 -H all_char /mnt/lustre/d160g.sanity mdd.lustre-MDT0000.changelog_max_idle_indexes=2 mdd.lustre-MDT0001.changelog_max_idle_indexes=2 mdd.lustre-MDT0000.changelog_gc=1 mdd.lustre-MDT0001.changelog_gc=1 mdd.lustre-MDT0000.changelog_min_gc_interval=2 mdd.lustre-MDT0001.changelog_min_gc_interval=2 lustre-MDT0000: clear the changelog for cl8 to record #23 mds1: verifying user1 cl8 clear: 21 + 2 == 23 lustre-MDT0001: clear the changelog for cl8 to record #18 mds2: verifying user1 cl8 clear: 16 + 2 == 18 pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 mds1: 1787758620 verify rec 23+1 == 24 mds2: 1787758623 verify rec 18+1 == 19 mdd.lustre-MDT0000.changelog_min_gc_interval=3600 mdd.lustre-MDT0001.changelog_min_gc_interval=3600 mdd.lustre-MDT0000.changelog_gc=1 mdd.lustre-MDT0001.changelog_gc=1 mdd.lustre-MDT0000.changelog_max_idle_indexes=2097446912 mdd.lustre-MDT0001.changelog_max_idle_indexes=2097446912 lustre-MDT0001: changelog user 'cl9' not found lustre-MDT0000: changelog user 'cl9' not found lustre-MDT0001: clear the changelog for cl8 of all records lustre-MDT0001: Deregistered changelog user #8 lustre-MDT0000: clear the changelog for cl8 of all records lustre-MDT0000: Deregistered changelog user #8 PASS 160g (58s) == sanity test 160h: changelog gc thread stop upon umount, orphan records delete ========================================================== 11:37:24 (1787758644) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl10 cl10' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl10 cl11 cl10 cl11' striped dir -i0 -c2 -H all_char /mnt/lustre/d160h.sanity mdd.lustre-MDT0000.changelog_max_idle_time=10 mdd.lustre-MDT0001.changelog_max_idle_time=10 mdd.lustre-MDT0000.changelog_gc=1 mdd.lustre-MDT0001.changelog_gc=1 mdd.lustre-MDT0000.changelog_min_gc_interval=2 mdd.lustre-MDT0001.changelog_min_gc_interval=2 lustre-MDT0000: clear the changelog for cl10 to record #27 mds1: verifying user cl10 clear: 25 + 2 == 27 lustre-MDT0001: clear the changelog for cl10 to record #21 mds2: verifying user cl10 clear: 19 + 2 == 21 fail_loc=0x1316 total: 4 create in 0.11 seconds: 38.08 ops/second Stopping /mnt/lustre-mds2 (opts:) on oleg315-server Stopping /mnt/lustre-mds1 (opts:) on oleg315-server fail_loc=0 Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0000 Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg315-server: oleg315-server.virtnet: executing set_default_debug all all pdsh@oleg315-client: oleg315-server: ssh exited with exit code 1 Started lustre-MDT0001 mds1: verifying first index 27 + 1 == 28 mds2: verifying first index 21 + 1 == 22 mdd.lustre-MDT0000.changelog_min_gc_interval=3600 mdd.lustre-MDT0001.changelog_min_gc_interval=3600 mdd.lustre-MDT0000.changelog_gc=1 mdd.lustre-MDT0001.changelog_gc=1 mdd.lustre-MDT0000.changelog_max_idle_time=2592000 mdd.lustre-MDT0001.changelog_max_idle_time=2592000 lustre-MDT0001: changelog user 'cl11' not found lustre-MDT0000: changelog user 'cl11' not found lustre-MDT0001: clear the changelog for cl10 of all records lustre-MDT0001: Deregistered changelog user #10 lustre-MDT0000: clear the changelog for cl10 of all records lustre-MDT0000: Deregistered changelog user #10 PASS 160h (126s) == sanity test 160i: changelog user register/unregister race ========================================================== 11:39:30 (1787758770) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl11 cl11' striped dir -i0 -c2 -H all_char /mnt/lustre/d160i.sanity fail_loc=0x10001315 fail_val=1 lustre-MDT0000: clear the changelog for cl11 of all records mdd.lustre-MDT0000.changelog_mask=+hsm lustre-MDT0000: Deregistered changelog user #11 lustre-MDT0001: clear the changelog for cl11 of all records mdd.lustre-MDT0001.changelog_mask=+hsm lustre-MDT0001: Deregistered changelog user #11 Registered 2 changelog users: 'cl11 cl12 cl11 cl12' cl12 33 (4) cl12 25 (2) total: 4 create in 0.13 seconds: 30.13 ops/second verify changelogs are on: 35 != 33 verify changelogs are on: 35 != 33 lustre-MDT0001: clear the changelog for cl12 of all records lustre-MDT0001: Deregistered changelog user #12 lustre-MDT0000: clear the changelog for cl12 of all records lustre-MDT0000: Deregistered changelog user #12 lustre-MDT0001: changelog user 'cl11' not found lustre-MDT0000: changelog user 'cl11' not found PASS 160i (42s) == sanity test 160j: client can be umounted while its chanangelog is being used ========================================================== 11:40:12 (1787758812) Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre2 mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl13 cl13' striped dir -i0 -c2 -H all_char /mnt/lustre/d160j.sanity Starting client: oleg315-client.virtnet: -o user_xattr,flock 192.168.203.115@tcp:/lustre /mnt/lustre lustre-MDT0000: clear the changelog for cl13 of all records lustre-MDT0000: Deregistered changelog user #13 lustre-MDT0001: clear the changelog for cl13 of all records lustre-MDT0001: Deregistered changelog user #13 lustre-MDT0001: changelog user 'cl13' not found lustre-MDT0000: changelog user 'cl13' not found PASS 160j (25s) == sanity test 160k: Verify that changelog records are not lost ========================================================== 11:40:37 (1787758837) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl14 cl14' fail_loc=0x8000015d fail_val=3 lustre-MDT0000.43 07RMDIR 15:40:48.308472213 2026.08.26 0x1 t=[0x200002342:0x2:0x0] j=rmdir.0 ef=0x13 u=0:0 nid=192.168.203.15@tcp p=[0x240001b72:0x1:0x0] 1 lustre-MDT0001: clear the changelog for cl14 of all records lustre-MDT0001: Deregistered changelog user #14 lustre-MDT0000: clear the changelog for cl14 of all records lustre-MDT0000: Deregistered changelog user #14 PASS 160k (33s) == sanity test 160l: Verify that MTIME changelog records contain the parent FID ========================================================== 11:41:10 (1787758870) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl15 cl15' mdd.lustre-MDT0000.changelog_mask=-CREAT mdd.lustre-MDT0001.changelog_mask=-CREAT mdd.lustre-MDT0000.changelog_mask=-CLOSE mdd.lustre-MDT0001.changelog_mask=-CLOSE striped dir -i0 -c2 -H fnv_1a_64 /mnt/lustre/d160l.sanity lustre-MDT0001: clear the changelog for cl15 of all records lustre-MDT0001: Deregistered changelog user #15 lustre-MDT0000: clear the changelog for cl15 of all records lustre-MDT0000: Deregistered changelog user #15 PASS 160l (29s) == sanity test 160m: Changelog clear race ================ 11:41:39 (1787758899) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl16 cl16' mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl16 cl17 cl16 cl17' striped dir -i0 -c1 -H fnv_1a_64 /mnt/lustre/d160m.sanity total: 50 create in 0.63 seconds: 79.73 ops/second - unlinked 0 (time 1787758914 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second rm: cannot remove '/mnt/lustre/d160m.sanity': Is a directory fail_loc=0x8000015f fail_val=0 lustre-MDT0000: clear the changelog for cl16 to record #55 lustre-MDT0000: clear the changelog for cl17 of all records lustre-MDT0000: clear the changelog for cl16 of all records lustre-MDT0001: clear the changelog for cl17 of all records lustre-MDT0001: Deregistered changelog user #17 lustre-MDT0000: clear the changelog for cl17 of all records lustre-MDT0000: Deregistered changelog user #17 lustre-MDT0001: clear the changelog for cl16 of all records lustre-MDT0001: Deregistered changelog user #16 lustre-MDT0000: clear the changelog for cl16 of all records lustre-MDT0000: Deregistered changelog user #16 PASS 160m (36s) == sanity test 160n: Changelog destroy race ============== 11:42:15 (1787758935) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl18 cl18' striped dir -i0 -c1 -H crush2 /mnt/lustre/d160n.sanity - create 1545 (time 1787758956.91 total 10.00 last 154.50) - create 2763 (time 1787758966.92 total 20.01 last 121.71) - create 3948 (time 1787758976.93 total 30.02 last 118.35) - create 5069 (time 1787758986.94 total 40.03 last 112.03) - create 6493 (time 1787758996.94 total 50.03 last 142.34) - create 7969 (time 1787759006.94 total 60.03 last 147.57) - create 9646 (time 1787759016.94 total 70.03 last 167.69) total: 10000 create in 72.26 seconds: 138.39 ops/second rename '/mnt/lustre/d160n.sanity/f160n.sanity10000' returned -1: No such file or directory - unlinked 0 (time 1787759576 ; total 0 ; last 0) total: 10000 unlinks in 128 seconds: 78.125000 unlinks/second last record 30147 - create 1206 (time 1787759721.47 total 10.00 last 120.57) - create 2329 (time 1787759731.48 total 20.01 last 112.21) - create 3971 (time 1787759741.48 total 30.01 last 164.19) - create 5485 (time 1787759751.48 total 40.01 last 151.40) - create 6734 (time 1787759761.48 total 50.01 last 124.87) - create 7806 (time 1787759771.49 total 60.02 last 107.15) - create 8887 (time 1787759781.49 total 70.02 last 108.10) total: 10000 create in 78.70 seconds: 127.06 ops/second rename '/mnt/lustre/d160n.sanity/f160n.sanity10000' returned -1: No such file or directory - unlinked 0 (time 1787760347 ; total 0 ; last 0) total: 10000 unlinks in 136 seconds: 73.529411 unlinks/second last record 60147 - create 1202 (time 1787760501.07 total 10.04 last 119.77) - create 2395 (time 1787760511.08 total 20.04 last 119.29) - create 3854 (time 1787760521.08 total 30.05 last 145.76) - create 5366 (time 1787760531.10 total 40.06 last 151.03) - create 6574 (time 1787760541.10 total 50.06 last 120.77) - create 8110 (time 1787760551.10 total 60.06 last 153.57) - create 9476 (time 1787760561.10 total 70.06 last 136.59) total: 10000 create in 74.02 seconds: 135.10 ops/second rename '/mnt/lustre/d160n.sanity/f160n.sanity10000' returned -1: No such file or directory - unlinked 0 (time 1787761121 ; total 0 ; last 0) total: 10000 unlinks in 103 seconds: 97.087379 unlinks/second last record 90147 fail_loc=0x8000016c fail_val=0 lustre-MDT0000: clear the changelog for cl18 of all records lustre-MDT0000: clear the changelog for cl18 of all records lustre-MDT0001: clear the changelog for cl18 of all records lustre-MDT0001: Deregistered changelog user #18 lustre-MDT0000: clear the changelog for cl18 of all records lustre-MDT0000: Deregistered changelog user #18 PASS 160n (2311s) == sanity test 160o: changelog user name and mask ======== 12:20:47 (1787761247) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl19-test_160o cl19-test_160o' oleg315-server: error: changelog_register: Invalid argument pdsh@oleg315-client: oleg315-server: ssh exited with exit code 22 oleg315-server: error: changelog_register: User exists pdsh@oleg315-client: oleg315-server: ssh exited with exit code 17 oleg315-server: error: changelog_register: File name too long pdsh@oleg315-client: oleg315-server: ssh exited with exit code 36 mdd.lustre-MDT0000.changelog_mask=MARK+HSM mdd.lustre-MDT0001.changelog_mask=MARK+HSM error: get_param: param_path 'mdd/*/changelog*mask': No such file or directory lustre-MDT0000: clear the changelog for cl19-test_160o of all records lustre-MDT0001: clear the changelog for cl19-test_160o of all records striped dir -i0 -c1 -H crush /mnt/lustre/d160o.sanity mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl19-test_160o cl20 cl19-test_160o cl23' mdd.lustre-MDT0000.changelog_mask=MARK mdd.lustre-MDT0001.changelog_mask=MARK mdd.lustre-MDT0000.changelog_mask=CLOSE,UNLNK mdd.lustre-MDT0001.changelog_mask=CLOSE,UNLNK lustre-MDT0000: Deregistered changelog user #19 lustre-MDT0001: clear the changelog for cl20 of all records lustre-MDT0001: Deregistered changelog user #20 lustre-MDT0000: clear the changelog for cl23 of all records lustre-MDT0000: Deregistered changelog user #23 lustre-MDT0001: clear the changelog for cl19-test_160o of all records lustre-MDT0001: Deregistered changelog user #19 lustre-MDT0000: changelog user 'cl19-test_160o' not found PASS 160o (21s) == sanity test 160p: Changelog orphan cleanup with no users ========================================================== 12:21:08 (1787761268) mdd.lustre-MDT0000.changelog_mask=+hsm mdd.lustre-MDT0001.changelog_mask=+hsm Registered 2 changelog users: 'cl21 cl24' striped dir -i0 -c1 -H crush2 /mnt/lustre/d160p.sanity total: 50 create in 0.16 seconds: 315.91 ops/second - unlinked 0 (time 1787761270 ; total 0 ; last 0) total: 50 unlinks in 1 seconds: 50.000000 unlinks/second Stopping /mnt/lustre-mds1 (opts:) on oleg315-server