-----============= acceptance-small: sanity-wbc ============----- Wed Jul 24 23:25:36 EDT 2024 mgs: CentOS Linux release 7.9.2009 (Core) MGS_OS_VERSION_ID=7 MGS_OS_ID=centos MGS_OS_VERSION_CODE=117440512 MGS_OS_ID_LIKE=rhel fedora centos mds1: CentOS Linux release 7.9.2009 (Core) MDS1_OS_ID_LIKE=rhel fedora centos MDS1_OS_ID=centos MDS1_OS_VERSION_ID=7 MDS1_OS_VERSION_CODE=117440512 ost1: CentOS Linux release 7.9.2009 (Core) OST1_OS_VERSION_CODE=117440512 OST1_OS_VERSION_ID=7 OST1_OS_ID_LIKE=rhel fedora centos OST1_OS_ID=centos client: CentOS Linux release 7.9.2009 (Core) CLIENT_OS_ID=centos CLIENT_OS_ID_LIKE=rhel fedora centos CLIENT_OS_VERSION_ID=7 CLIENT_OS_VERSION_CODE=117440512 === sanity-wbc: start setup 23:25:39 (1721877939) === Starting client oleg205-client.virtnet: -o user_xattr,flock oleg205-server@tcp:/lustre /mnt/lustre2 Started clients oleg205-client.virtnet: 192.168.202.105@tcp:/lustre on /mnt/lustre2 type lustre (rw,checksum,flock,user_xattr,lruresize,lazystatfs,nouser_fid2path,verbose,noencrypt,statfs_project) oleg205-client.virtnet: executing check_config_client /mnt/lustre oleg205-client.virtnet: Checking config lustre mounted on /mnt/lustre Checking servers environments Checking clients oleg205-client.virtnet environments Using TIMEOUT=20 osc.lustre-OST0000-osc-ffff8800aa8ee800.idle_timeout=debug osc.lustre-OST0000-osc-ffff8800b6034000.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800aa8ee800.idle_timeout=debug osc.lustre-OST0001-osc-ffff8800b6034000.idle_timeout=debug disable quota as required oleg205-server: oleg205-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all osd-ldiskfs.track_declares_assert=1 === sanity-wbc: finish setup 23:25:45 (1721877945) === running as uid/gid/euid/egid 500/500/500/500, groups: [true] running as uid/gid/euid/egid 500/500/500/500, groups: [touch] [/mnt/lustre/d0_runas_test/f6830] nobody:x:99: excepting tests: == sanity-wbc test 1: Basic test for WBC with LAZY flush mode ========================================================== 23:25:46 (1721877946) llite.lustre-ffff8800aa8ee800.wbc.conf=enable llite.lustre-ffff8800b6034000.wbc.conf=enable llite.lustre-ffff8800aa8ee800.wbc.conf= cache_mode: memfs flush_mode: lazy_drop max_rpcs: 0 llite.lustre-ffff8800b6034000.wbc.conf= cache_mode: memfs flush_mode: lazy_drop max_rpcs: 0 /mnt/lustre/d1.sanity-wbc, state: (0x0000000f) root protected sync complete, metadata: lustre, flush_mode: lazy_drop 2+0 records in 2+0 records out 8192 bytes (8.2 kB) copied, 0.000238931 s, 34.3 MB/s 2+0 records in 2+0 records out 8194 bytes (8.2 kB) copied, 0.000201781 s, 40.6 MB/s 2+0 records in 2+0 records out 2097152 bytes (2.1 MB) copied, 0.00353476 s, 593 MB/s 2+0 records in 2+0 records out 2097158 bytes (2.1 MB) copied, 0.00309006 s, 679 MB/s oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: ROOT/d1.sanity-wbc/file1: File not found by ext2_lookup oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: ROOT/d1.sanity-wbc/dir1/file2: File not found by ext2_lookup oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: ROOT/d1.sanity-wbc/dir1/dir2: File not found by ext2_lookup oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: ROOT/d1.sanity-wbc/file3: File not found by ext2_lookup oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: ROOT/d1.sanity-wbc/file4: File not found by ext2_lookup oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: ROOT/d1.sanity-wbc/file5: File not found by ext2_lookup oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: ROOT/d1.sanity-wbc/file6: File not found by ext2_lookup stat /mnt/lustre2/d1.sanity-wbc File: '/mnt/lustre2/d1.sanity-wbc' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272502275 Links: 3 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-24 23:25:46.000000000 -0400 Modify: 2024-07-24 23:25:47.000000000 -0400 Change: 2024-07-24 23:25:47.000000000 -0400 Birth: - oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps Inode: 512112 Type: regular Mode: 0666 Flags: 0x0 Inode: 512110 Type: directory Mode: 0777 Flags: 0x0 Inode: 512111 Type: regular Mode: 0666 Flags: 0x0 Inode: 512109 Type: regular Mode: 0666 Flags: 0x0 Inode: 512107 Type: regular Mode: 0666 Flags: 0x0 Inode: 512108 Type: regular Mode: 0666 Flags: 0x0 stat /mnt/lustre2/d1.sanity-wbc/dir1 File: '/mnt/lustre2/d1.sanity-wbc/dir1' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272502277 Links: 3 Access: (0777/drwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-24 23:25:47.000000000 -0400 Modify: 2024-07-24 23:25:47.000000000 -0400 Change: 2024-07-24 23:25:47.000000000 -0400 Birth: - oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps oleg205-server: debugfs 1.46.2.wc5 (26-Mar-2022) oleg205-server: /dev/mapper/mds1_flakey: catastrophic mode - not reading inode or group bitmaps Inode: 512113 Type: regular Mode: 0666 Flags: 0x0 Inode: 512114 Type: directory Mode: 0777 Flags: 0x0 /mnt/lustre/d1.sanity-wbc/file1 has size 6 OK /mnt/lustre/d1.sanity-wbc/dir1/file2 has size 8192 OK /mnt/lustre/d1.sanity-wbc/file3 has size 0 OK /mnt/lustre/d1.sanity-wbc/file4 has size 8194 OK /mnt/lustre2/d1.sanity-wbc/file1 has size 6 OK /mnt/lustre2/d1.sanity-wbc/dir1/file2 has size 8192 OK /mnt/lustre2/d1.sanity-wbc/file3 has size 0 OK /mnt/lustre2/d1.sanity-wbc/file4 has size 8194 OK /mnt/lustre2/d1.sanity-wbc/file5 has size 2097152 OK /mnt/lustre2/d1.sanity-wbc/file6 has size 2097158 OK llite.lustre-ffff8800aa8ee800.wbc.conf=disable llite.lustre-ffff8800b6034000.wbc.conf=disable PASS 1 (3s) == sanity-wbc test 2: Verify remote read works correctly ========================================================== 23:25:49 (1721877949) llite.lustre-ffff8800aa8ee800.wbc.conf=enable llite.lustre-ffff8800b6034000.wbc.conf=enable llite.lustre-ffff8800aa8ee800.wbc.conf= cache_mode: memfs flush_mode: lazy_drop max_rpcs: 0 llite.lustre-ffff8800b6034000.wbc.conf= cache_mode: memfs flush_mode: lazy_drop max_rpcs: 0 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00018391 s, 5.6 MB/s 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000201613 s, 5.1 MB/s File: '/mnt/lustre2/d2.sanity-wbc' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272502286 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-24 23:25:49.000000000 -0400 Modify: 2024-07-24 23:25:49.000000000 -0400 Change: 2024-07-24 23:25:49.000000000 -0400 Birth: - File: '/mnt/lustre2/d2.sanity-wbc' Size: 4096 Blocks: 8 IO Block: 1048576 directory Device: 2c54f966h/743766374d Inode: 144115205272502288 Links: 2 Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-24 23:25:49.000000000 -0400 Modify: 2024-07-24 23:25:49.000000000 -0400 Change: 2024-07-24 23:25:49.000000000 -0400 Birth: - llite.lustre-ffff8800aa8ee800.wbc.conf=disable llite.lustre-ffff8800b6034000.wbc.conf=disable PASS 2 (1s) == sanity-wbc test 3: Remote read for WBC cached regular file with holes ========================================================== 23:25:50 (1721877950) llite.lustre-ffff8800aa8ee800.wbc.conf=enable llite.lustre-ffff8800b6034000.wbc.conf=enable llite.lustre-ffff8800aa8ee800.wbc.conf= cache_mode: memfs flush_mode: lazy_drop max_rpcs: 0 llite.lustre-ffff8800b6034000.wbc.conf= cache_mode: memfs flush_mode: lazy_drop max_rpcs: 0 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.000255162 s, 4.0 MB/s /mnt/lustre/d3.sanity-wbc/f3.sanity-wbc has size 1049600 OK File: '/mnt/lustre2/d3.sanity-wbc/f3.sanity-wbc' Size: 1049600 Blocks: 1 IO Block: 4194304 regular file Device: 2c54f966h/743766374d Inode: 144115205272502293 Links: 1 Access: (0666/-rw-rw-rw-) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2024-07-24 23:25:51.000000000 -0400 Modify: 2024-07-24 23:25:51.000000000 -0400 Change: 2024-07-24 23:25:51.000000000 -0400 Birth: - /mnt/lustre2/d3.sanity-wbc/f3.sanity-wbc has size 1049600 OK 1+0 records in 1+0 records out 1024 bytes (1.0 kB) copied, 0.00017957 s, 5.7 MB/s /mnt/lustre/d3.sanity-wbc/f3.sanity-wbc has size 1049600 OK /mnt/lustre2/d3.sanity-wbc/f3.sanity-wbc has size 1049600 OK llite.lustre-ffff8800aa8ee800.wbc.conf=disable llite.lustre-ffff8800b6034000.wbc.conf=disable PASS 3 (2s) == sanity-wbc test 4: Verify unlink() works correctly ==== 23:25:52 (1721877952) llite.lustre-ffff8800aa8ee800.wbc.conf=enable llite.lustre-ffff8800b6034000.wbc.conf=enable llite.lustre-ffff8800aa8ee800.wbc.conf= cache_mode: memfs flush_mode: lazy_drop max_rpcs: 0 llite.lustre-ffff8800b6034000.wbc.conf= cache_mode: memfs flush_mode: lazy_drop max_rpcs: 0 stat: cannot stat '/mnt/lustre/d4.sanity-wbc/f4.sanity-wbc': No such file or directory stat: cannot stat '/mnt/lustre/d4.sanity-wbc/dir2/file2': No such file or directory stat: cannot stat '/mnt/lustre/d4.sanity-wbc/dir2': No such file or directory stat: cannot stat '/mnt/lustre/d4.sanity-wbc': No such file or directory stat: cannot stat '/mnt/lustre/d4.sanity-wbc/dir2': No such file or directory stat: cannot stat '/mnt/lustre/d4.sanity-wbc': No such file or directory stat: cannot stat '/mnt/lustre/d4.sanity-wbc/dir2/file2': No such file or directory llite.lustre-ffff8800aa8ee800.wbc.conf=disable llite.lustre-ffff8800b6034000.wbc.conf=disable PASS 4 (2s) == sanity-wbc test 5: Hanle -ENOENT lookup failure correctly ========================================================== 23:25:54 (1721877954) stat: cannot stat 'indirect': No such file or directory PASS 5 (1s) cleanup: ====================================================== == sanity-wbc test complete, duration 19 sec ============= 23:25:55 (1721877955) === sanity-wbc: start cleanup 23:25:56 (1721877956) === Stopping clients: oleg205-client.virtnet /mnt/lustre2 (opts:) Stopping client oleg205-client.virtnet /mnt/lustre2 opts: === sanity-wbc: finish cleanup 23:25:56 (1721877956) ===