[ 2032.907795] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2034.248383] Lustre: DEBUG MARKER: Write 100M (buffered) ...
[ 2036.114645] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2036.455160] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2036.914067] Lustre: Failing over lustre-MDT0000
[ 2036.979142] LustreError: 24944:0:(obd_class.h:473:obd_check_dev()) Device 22 not setup
[ 2036.981020] LustreError: 24944:0:(obd_class.h:473:obd_check_dev()) Skipped 3 previous similar messages
[ 2036.998708] Lustre: server umount lustre-MDT0000 complete
[ 2037.569073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2037.572294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2037.575749] Lustre: Skipped 1 previous similar message
[ 2037.577562] LustreError: 8369:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2037.586045] LustreError: 8369:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 3 previous similar messages
[ 2050.042707] LDISKFS-fs (dm-0): recovery complete
[ 2050.043893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2050.070045] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2050.128791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2050.136434] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2050.926317] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2052.924036] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2055.137967] Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to  (at 0@lo)
[ 2055.139731] Lustre: Skipped 1 previous similar message
[ 2055.149748] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted.
[ 2055.180053] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2048 to 0x2c0000401:2081)
[ 2055.180104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2051 to 0x280000401:2081)
[ 2056.030797] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2056.439183] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2058.626321] Lustre: DEBUG MARKER: (dd_pid=15547, time=0, timeout=600)
[ 2065.683083] Lustre: DEBUG MARKER: User quota (limit: 200)
[ 2066.985099] Lustre: DEBUG MARKER: Write 100M (directio) ...
[ 2069.126796] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2069.595490] Lustre: DEBUG MARKER: Fail mds for 40 seconds
[ 2070.137945] Lustre: Failing over lustre-MDT0000
[ 2070.160920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2070.207893] Lustre: server umount lustre-MDT0000 complete
[ 2072.957030] LustreError: 8361:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.12@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2072.961975] LustreError: 8361:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 15 previous similar messages
[ 2083.039824] LDISKFS-fs (dm-0): recovery complete
[ 2083.041091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2083.066662] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2083.125170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2083.793263] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2088.150488] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2048 to 0x2c0000401:2113)
[ 2088.150495] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2083 to 0x280000401:2113)
[ 2089.031935] Lustre: DEBUG MARKER: oleg312-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2089.446258] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2091.571399] Lustre: DEBUG MARKER: (dd_pid=17840, time=0, timeout=600)
[ 2101.676323] Lustre: DEBUG MARKER: == sanity-quota test 19: Updating admin limits doesn't zero operational limits(b14790) ========================================================== 05:41:03 (1744537263)
[ 2121.937237] Lustre: DEBUG MARKER: == sanity-quota test 20: Test if setquota specifiers work properly (b15754) ========================================================== 05:41:23 (1744537283)
[ 2129.644663] Lustre: DEBUG MARKER: == sanity-quota test 21: Setquota while writing
[ 2132.763596] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for user: quota_usr
[ 2133.280819] Lustre: DEBUG MARKER: Set limit(block:10G; file:1000000) for group: quota_usr
[ 2133.304019] LustreError: 26289:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0001: Delaying reintegration for qtype:1 until pending updates are flushed.
[ 2133.311514] LustreError: 26289:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 9 previous similar messages
[ 2133.831025] Lustre: DEBUG MARKER: Set limit(block:10G; file:) for project: 1000
[ 2134.290031] Lustre: DEBUG MARKER: Set quota for 1 times
[ 2135.720478] Lustre: DEBUG MARKER: Set quota for 2 times
[ 2137.161393] Lustre: DEBUG MARKER: Set quota for 3 times
[ 2138.522466] Lustre: DEBUG MARKER: Set quota for 4 times
[ 2139.900537] Lustre: DEBUG MARKER: Set quota for 5 times
[ 2141.275727] Lustre: DEBUG MARKER: Set quota for 6 times
[ 2142.672178] Lustre: DEBUG MARKER: Set quota for 7 times
[ 2144.091690] Lustre: DEBUG MARKER: Set quota for 8 times
[ 2145.453966] Lustre: DEBUG MARKER: Set quota for 9 times
[ 2146.902997] Lustre: DEBUG MARKER: Set quota for 10 times
[ 2148.366120] Lustre: DEBUG MARKER: Set quota for 11 times
[ 2149.828182] Lustre: DEBUG MARKER: Set quota for 12 times
[ 2151.219828] Lustre: DEBUG MARKER: Set quota for 13 times
[ 2152.610859] Lustre: DEBUG MARKER: Set quota for 14 times
[ 2153.968719] Lustre: DEBUG MARKER: Set quota for 15 times
[ 2155.410755] Lustre: DEBUG MARKER: Set quota for 16 times
[ 2156.863080] Lustre: DEBUG MARKER: Set quota for 17 times
[ 2158.273438] Lustre: DEBUG MARKER: Set quota for 18 times
[ 2159.719247] Lustre: DEBUG MARKER: Set quota for 19 times
[ 2161.118401] Lustre: DEBUG MARKER: Set quota for 20 times
[ 2162.486887] Lustre: DEBUG MARKER: Set quota for 21 times
[ 2181.859676] Lustre: DEBUG MARKER: == sanity-quota test 22: enable/disable quota by 'lctl conf_param/set_param -P' ========================================================== 05:42:23 (1744537343)
[ 2188.289213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2188.292369] Lustre: Skipped 6 previous similar messages
[ 2189.008164] LustreError: 5786:0:(obd_class.h:473:obd_check_dev()) Device 22 not setup
[ 2189.010107] LustreError: 5786:0:(obd_class.h:473:obd_check_dev()) Skipped 15 previous similar messages
[ 2189.030654] Lustre: server umount lustre-MDT0000 complete
[ 2190.319554] LustreError: 30051:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1744537352 with bad export cookie 8651158972775442811
[ 2190.321148] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2190.331292] LustreError: 30051:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 2190.501448] Lustre: server umount lustre-MDT0001 complete
[ 2202.332881] Lustre: server umount lustre-OST0000 complete
[ 2213.599576] Lustre: server umount lustre-OST0001 complete
[ 2218.141869] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local
[ 2220.772263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2220.940684] LustreError: 7995:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2220.945817] LustreError: 7995:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 10 previous similar messages
[ 2220.970163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2221.815569] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2224.764158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2224.805437] Lustre: 8865:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.112@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 2224.951444] mount.lustre (8865) used greatest stack depth: 9808 bytes left
[ 2226.059099] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2227.008953] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 2228.797117] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 2228.803854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2230.652655] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2230.941914] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2116 to 0x280000401:2145)
[ 2233.470842] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 2233.478803] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2234.792927] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2115 to 0x2c0000401:2145)
[ 2235.339799] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2235.793839] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97)
[ 2238.428510] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2239.717788] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 2251.958258] Lustre: server umount lustre-MDT0000 complete
[ 2253.312076] LustreError: 9341:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1744537415 with bad export cookie 8651158972775468186
[ 2253.314641] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2253.324596] LustreError: 9341:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2255.825063] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 2255.828704] Lustre: Skipped 1 previous similar message
[ 2257.829096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 2257.832916] Lustre: Skipped 1 previous similar message
[ 2259.480737] Lustre: server umount lustre-MDT0001 complete
[ 2260.931714] Lustre: server umount lustre-OST0000 complete
[ 2262.380276] Lustre: server umount lustre-OST0001 complete
[ 2267.209418] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local
[ 2270.452638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2270.699596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2270.704155] Lustre: Skipped 3 previous similar messages
[ 2271.675833] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2274.684747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2274.729989] Lustre: 15843:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.112@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 2274.736372] Lustre: 15843:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 2 previous similar messages
[ 2276.037201] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2277.039805] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 2279.004512] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 2279.012155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2279.152263] mount.lustre (16767) used greatest stack depth: 9328 bytes left
[ 2280.141488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2116 to 0x280000401:2177)
[ 2280.775997] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2283.919731] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 2283.925613] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2285.006734] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:129)
[ 2285.014148] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2115 to 0x2c0000401:2177)
[ 2285.711959] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2288.982396] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2290.289139] Lustre: Modifying parameter general.lod.*.mdt_hash in log params
[ 2298.780736] Lustre: DEBUG MARKER: == sanity-quota test 23: Quota should be honored with directIO (b16125) ========================================================== 05:44:20 (1744537460)
[ 2299.326084] Lustre: DEBUG MARKER: OST0_SIZE: 3605408 required: 6144
[ 2299.844037] Lustre: DEBUG MARKER: run for 4MB test file
[ 2304.768714] Lustre: DEBUG MARKER: User quota (limit: 4 MB)
[ 2306.447770] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2306.987527] Lustre: DEBUG MARKER: Write half of file
[ 2307.619910] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2308.239391] Lustre: DEBUG MARKER: Step1: done
[ 2308.786923] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2309.374092] Lustre: DEBUG MARKER: Step2: done
[ 2324.327489] Lustre: DEBUG MARKER: OST0_SIZE: 3605408 required: 61440
[ 2324.850016] Lustre: DEBUG MARKER: run for 40MB test file
[ 2327.973297] Lustre: DEBUG MARKER: User quota (limit: 40 MB)
[ 2329.419813] Lustre: DEBUG MARKER: Step1: trigger EDQUOT with O_DIRECT
[ 2329.930304] Lustre: DEBUG MARKER: Write half of file
[ 2330.934009] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2332.015646] Lustre: DEBUG MARKER: Step1: done
[ 2332.547684] Lustre: DEBUG MARKER: Step2: rewrite should succeed
[ 2332.961380] Lustre: DEBUG MARKER: Step2: done
[ 2352.519609] Lustre: DEBUG MARKER: == sanity-quota test 24: lfs draws an asterix when limit is reached (b16646) ========================================================== 05:45:13 (1744537513)
[ 2369.626402] Lustre: DEBUG MARKER: == sanity-quota test 25: check indexes versions ========== 05:45:31 (1744537531)
[ 2380.964852] Lustre: DEBUG MARKER: Write...
[ 2381.549582] Lustre: DEBUG MARKER: Write out of block quota ...
[ 2408.809126] Lustre: DEBUG MARKER: == sanity-quota test 27a: lfs quota/setquota should handle wrong arguments (b19612) ========================================================== 05:46:10 (1744537570)
[ 2410.989766] Lustre: DEBUG MARKER: == sanity-quota test 27b: lfs quota/setquota should handle user/group/project ID (b20200) ========================================================== 05:46:12 (1744537572)
[ 2415.029456] Lustre: DEBUG MARKER: == sanity-quota test 27c: lfs quota should support human-readable output ========================================================== 05:46:16 (1744537576)
[ 2418.527298] Lustre: DEBUG MARKER: == sanity-quota test 27d: lfs setquota should support fraction block limit ========================================================== 05:46:19 (1744537579)
[ 2421.751863] Lustre: DEBUG MARKER: == sanity-quota test 30: Hard limit updates should not reset grace times ========================================================== 05:46:23 (1744537583)
[ 2447.984313] Lustre: DEBUG MARKER: == sanity-quota test 33: Basic usage tracking for user
[ 2489.312321] Lustre: DEBUG MARKER: == sanity-quota test 34: Usage transfer for user
[ 2553.999784] Lustre: DEBUG MARKER: == sanity-quota test 35: Usage is still accessible across reboot ========================================================== 05:48:35 (1744537715)
[ 2574.041757] Lustre: DEBUG MARKER: Restart...
[ 2575.329186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 2575.334062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2575.342559] Lustre: Skipped 15 previous similar messages
[ 2575.346548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 2580.936843] LustreError: 10332:0:(obd_class.h:473:obd_check_dev()) Device 10 not setup
[ 2580.940854] LustreError: 10332:0:(obd_class.h:473:obd_check_dev()) Skipped 51 previous similar messages
[ 2580.973440] Lustre: server umount lustre-MDT0000 complete
[ 2582.353655] LustreError: 16307:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1744537744 with bad export cookie 8651158972775470489
[ 2582.356015] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2582.368782] LustreError: 16307:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 2582.536799] Lustre: server umount lustre-MDT0001 complete
[ 2594.523416] Lustre: server umount lustre-OST0000 complete
[ 2605.971623] Lustre: server umount lustre-OST0001 complete
[ 2610.593848] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local
[ 2613.725091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2613.896569] LustreError: 12544:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2613.907267] LustreError: 12544:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 8 previous similar messages
[ 2613.927330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2613.931181] Lustre: Skipped 3 previous similar messages
[ 2615.021128] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2618.125451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2618.173766] Lustre: 13407:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.112@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 2618.179639] Lustre: 13407:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 2 previous similar messages
[ 2619.464149] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2620.534312] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 2622.902264] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 2622.908982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2624.068022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:2189 to 0x280000401:2209)
[ 2624.866967] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2628.040666] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 2628.046782] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 2628.067438] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:161)
[ 2629.167320] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:2185 to 0x2c0000401:2209)
[ 2629.906257] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 2633.128929] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 2662.789744] Lustre: DEBUG MARKER: == sanity-quota test 37: Quota accounted properly for file created by 'lfs setstripe' ========================================================== 05:50:24 (1744537824)
[ 2686.970263] Lustre: DEBUG MARKER: == sanity-quota test 38: Quota accounting iterator doesn't skip id entries ========================================================== 05:50:48 (1744537848)
[ 3187.477651] Lustre: DEBUG MARKER: == sanity-quota test 39: Project ID interface works correctly ========================================================== 05:59:08 (1744538348)
[ 3194.049105] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3194.050052] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3194.050055] Lustre: Skipped 7 previous similar messages
[ 3194.062772] Lustre: Skipped 6 previous similar messages
[ 3196.008763] LustreError: 22687:0:(obd_class.h:473:obd_check_dev()) Device 10 not setup
[ 3196.012974] LustreError: 22687:0:(obd_class.h:473:obd_check_dev()) Skipped 25 previous similar messages
[ 3196.075606] Lustre: server umount lustre-MDT0000 complete
[ 3197.439666] LustreError: 13874:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1744538359 with bad export cookie 8651158972775479288
[ 3197.441472] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3197.452296] LustreError: 13874:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3197.601141] Lustre: server umount lustre-MDT0001 complete
[ 3209.113973] Lustre: server umount lustre-OST0000 complete
[ 3220.613924] Lustre: server umount lustre-OST0001 complete
[ 3225.496850] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local
[ 3228.842018] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3229.023471] LustreError: 24898:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3229.031809] LustreError: 24898:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 3 previous similar messages
[ 3229.058636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3229.062005] Lustre: Skipped 3 previous similar messages
[ 3230.293389] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3233.416504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3233.459402] Lustre: 25762:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.112@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3233.465637] Lustre: 25762:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 2 previous similar messages
[ 3234.757277] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3235.849777] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3235.853466] Lustre: Skipped 1 previous similar message
[ 3238.154257] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3238.161010] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3239.328771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:7211 to 0x280000401:7233)
[ 3240.154199] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3243.254340] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3243.263332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3243.326239] Lustre: lustre-OST0001: Not available for connect from 0@lo (not set up)
[ 3243.330321] Lustre: Skipped 3 previous similar messages
[ 3243.335063] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:193)
[ 3245.160172] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3245.179897] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:7211 to 0x2c0000401:7233)
[ 3248.439900] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3267.860785] Lustre: DEBUG MARKER: == sanity-quota test 40a: Hard link across different project ID ========================================================== 06:00:29 (1744538429)
[ 3282.571111] Lustre: DEBUG MARKER: == sanity-quota test 40b: Mv across different project ID ========================================================== 06:00:43 (1744538443)
[ 3297.008030] Lustre: DEBUG MARKER: == sanity-quota test 40c: Remote child Dir inherit project quota properly ========================================================== 06:00:58 (1744538458)
[ 3312.497936] Lustre: DEBUG MARKER: == sanity-quota test 40d: Stripe Directory inherit project quota properly ========================================================== 06:01:13 (1744538473)
[ 3327.372863] Lustre: DEBUG MARKER: == sanity-quota test 41: df should return projid-specific values ========================================================== 06:01:28 (1744538488)
[ 3330.741344] LustreError: 2442:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3330.750118] LustreError: 2442:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 9 previous similar messages
[ 3354.337484] Lustre: DEBUG MARKER: == sanity-quota test 42: lfs quota should include default quota info ========================================================== 06:01:55 (1744538515)
[ 3364.581172] Lustre: DEBUG MARKER: == sanity-quota test 48: lfs quota --delete should delete quota project ID ========================================================== 06:02:05 (1744538525)
[ 3402.416076] Lustre: DEBUG MARKER: == sanity-quota test 49: lfs quota -a prints the quota usage for all quota IDs ========================================================== 06:02:43 (1744538563)
[ 3404.110061] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3404.112823] Lustre: Skipped 1 previous similar message
[ 3405.115061] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3405.117399] Lustre: Skipped 161 previous similar messages
[ 3407.115531] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3407.117926] Lustre: Skipped 343 previous similar messages
[ 3411.117886] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3411.120356] Lustre: Skipped 639 previous similar messages
[ 3419.122675] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3419.125097] Lustre: Skipped 1351 previous similar messages
[ 3435.134459] Lustre: *** cfs_fail_loc=a09, val=0***
[ 3435.136773] Lustre: Skipped 2507 previous similar messages
[ 3578.897767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3578.901658] Lustre: Skipped 3 previous similar messages
[ 3581.123963] Lustre: server umount lustre-MDT0000 complete
[ 3582.404704] LustreError: 2290:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1744538744 with bad export cookie 8651158972777226957
[ 3582.406999] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3582.417326] LustreError: 2290:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 3583.905842] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 3583.909489] Lustre: Skipped 1 previous similar message
[ 3588.586165] Lustre: server umount lustre-MDT0001 complete
[ 3590.185702] Lustre: server umount lustre-OST0000 complete
[ 3591.753205] Lustre: server umount lustre-OST0001 complete
[ 3593.792602] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_hostid
[ 3595.727904] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local
[ 3600.963036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3605.590802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3608.091743] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3608.095402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3610.713947] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3610.717499] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3614.713155] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local
[ 3617.873166] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3617.905326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3618.025144] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000
[ 3618.029682] Lustre: Skipped 1 previous similar message
[ 3618.043220] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space.
[ 3618.100612] Lustre: lustre-MDT0000: new disk, initializing
[ 3618.131559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3618.135412] Lustre: Skipped 3 previous similar messages
[ 3618.143941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt
[ 3619.296951] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3623.653232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3623.681640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3623.720890] Lustre: 15191:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.203.112@tcp: no remote llog for lustre-sptlrpc, check MGS config
[ 3623.727211] Lustre: 15191:0:(mgc_request_server.c:550:mgc_llog_local_copy()) Skipped 2 previous similar messages
[ 3623.736486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space.
[ 3623.739933] Lustre: Skipped 1 previous similar message
[ 3623.792619] Lustre: lustre-MDT0001: new disk, initializing
[ 3623.825682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt
[ 3623.830754] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt]
[ 3624.990036] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3629.811824] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3629.818283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3629.839376] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3629.844903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3629.944678] Lustre: lustre-OST0000: new disk, initializing
[ 3629.947842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
[ 3631.017718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost
[ 3631.023456] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost]
[ 3631.044213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401
[ 3631.739733] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3636.154762] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3636.163445] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro
[ 3636.189462] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3636.195274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3636.241023] Lustre: lustre-OST0001: new disk, initializing
[ 3636.244012] Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
[ 3637.937605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost
[ 3637.940889] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost]
[ 3637.954940] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401
[ 3638.058445] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3642.684863] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3657.190945] Lustre: DEBUG MARKER: == sanity-quota test 50: Test if lfs find --projid works ========================================================== 06:06:58 (1744538818)
[ 3672.372546] Lustre: DEBUG MARKER: == sanity-quota test 51: Test project accounting with mv/cp ========================================================== 06:07:13 (1744538833)
[ 3692.022052] Lustre: DEBUG MARKER: == sanity-quota test 52: Rename normal file across project ID ========================================================== 06:07:33 (1744538853)
[ 3697.206025] Lustre: DEBUG MARKER: rename directory return 255
[ 3712.329191] Lustre: DEBUG MARKER: == sanity-quota test 53: Project inherit attribute could be cleared ========================================================== 06:07:53 (1744538873)
[ 3718.826721] Lustre: DEBUG MARKER: == sanity-quota test 54: basic lfs project interface test ========================================================== 06:08:00 (1744538880)
[ 3731.411296] Lustre: DEBUG MARKER: == sanity-quota test 55: Chgrp should be affected by group quota ========================================================== 06:08:12 (1744538892)
[ 3751.207599] LustreError: 25774:0:(qmt_entry.c:552:qmt_adjust_edquot()) $$$ set revoke_time explicitly  qmt:lustre-QMT0000 pool:dt-0x0 id:60001 enforced:1 hard:51200 soft:0 granted:51200 time:0 qunit: 1024 edquot:0 may_rel:0 revoke:0 default:no
[ 3762.468513] Lustre: DEBUG MARKER: == sanity-quota test 56: lfs quota -t should work well === 06:08:43 (1744538923)
[ 3770.985689] Lustre: DEBUG MARKER: == sanity-quota test 57: lfs project could tolerate errors ========================================================== 06:08:52 (1744538932)
[ 3786.656274] Lustre: DEBUG MARKER: == sanity-quota test 58: project ID should be kept for new mirrors created by FID ========================================================== 06:09:08 (1744538948)
[ 3834.646163] Lustre: DEBUG MARKER: == sanity-quota test 59: lfs project dosen't crash kernel with project disabled ========================================================== 06:09:56 (1744538996)
[ 3838.272771] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3838.273226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3838.273227] Lustre: Skipped 2 previous similar messages
[ 3838.280825] Lustre: Skipped 9 previous similar messages
[ 3841.616374] LustreError: 616:0:(obd_class.h:473:obd_check_dev()) Device 28 not setup
[ 3841.618224] LustreError: 616:0:(obd_class.h:473:obd_check_dev()) Skipped 51 previous similar messages
[ 3841.635287] Lustre: server umount lustre-MDT0000 complete
[ 3842.528966] LustreError: 15204:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1744539004 with bad export cookie 8651158972777598587
[ 3842.529972] LustreError: MGC192.168.203.112@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3842.533697] LustreError: 15204:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 3842.628895] Lustre: server umount lustre-MDT0001 complete
[ 3853.467161] Lustre: server umount lustre-OST0000 complete
[ 3864.433227] Lustre: server umount lustre-OST0001 complete
[ 3868.730172] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local
[ 3870.915998] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3871.025653] LustreError: 3114:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3871.030291] LustreError: 3114:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 5 previous similar messages
[ 3871.040548] LustreError: lustre-MDT0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3871.682789] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3873.906547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3873.995123] LustreError: lustre-MDT0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3874.672045] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3875.324256] Lustre: Modifying parameter general.debug_raw_pointers in log params
[ 3875.325854] Lustre: Skipped 3 previous similar messages
[ 3876.750595] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3876.753334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3876.820396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
[ 3876.822410] Lustre: Skipped 5 previous similar messages
[ 3876.823851] LustreError: lustre-OST0000: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3877.840275] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3879.651778] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3879.654333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3879.687659] LustreError: lustre-OST0001: can't enable quota enforcement since space accounting isn't functional. Please run tunefs.lustre --quota on an unmounted filesystem if not done already
[ 3880.687170] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:65)
[ 3880.690102] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:20 to 0x2c0000401:65)
[ 3880.708675] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3887.140023] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3895.429894] LustreError: 5934:0:(osd_handler.c:3264:osd_quota_transfer()) lustre-MDT0000: quota transfer failed. Is project enforcement enabled on the ldiskfs filesystem? rc = -95
[ 3898.706271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3898.709863] Lustre: Skipped 6 previous similar messages
[ 3902.509613] Lustre: server umount lustre-MDT0000 complete
[ 3903.798483] LustreError: 3097:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1744539066 with bad export cookie 8651158972777647034
[ 3903.805108] LustreError: 3097:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 3903.954026] Lustre: server umount lustre-MDT0001 complete
[ 3914.750759] Lustre: server umount lustre-OST0000 complete
[ 3926.137918] Lustre: server umount lustre-OST0001 complete
[ 3932.125146] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing load_modules_local
[ 3935.191072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3936.508080] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3939.446011] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3940.647781] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3943.940658] LDISKFS-fs (dm-2): file extents enabled, maximum tree depth=5
[ 3943.946625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3945.062135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:20 to 0x280000401:97)
[ 3945.771193] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3948.800900] LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5
[ 3948.807241] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc
[ 3949.880379] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97)
[ 3950.611056] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
[ 3953.785130] Lustre: DEBUG MARKER: Using TIMEOUT=20
[ 3974.460767] Lustre: DEBUG MARKER: == sanity-quota test 60: Test quota for root with setgid ========================================================== 06:12:15 (1744539135)
[ 3978.066771] LustreError: 14791:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed.
[ 3978.073274] LustreError: 14791:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 7 previous similar messages
[ 3993.635938] Lustre: DEBUG MARKER: SKIP: sanity-quota test_61 skipping SLOW test 61
[ 3994.198160] Lustre: DEBUG MARKER: == sanity-quota test 62: Project inherit should be only changed by root ========================================================== 06:12:35 (1744539155)
[ 4000.561583] Lustre: DEBUG MARKER: SKIP: sanity-quota test_63 skipping excluded test 63
[ 4001.132891] Lustre: DEBUG MARKER: == sanity-quota test 64: lfs project on non dir/files should succeed ========================================================== 06:12:42 (1744539162)
[ 4019.255486] Lustre: DEBUG MARKER: SKIP: sanity-quota test_65 skipping excluded test 65
[ 4019.609012] Lustre: DEBUG MARKER: == sanity-quota test 66: nonroot user can not change project state in default ========================================================== 06:13:01 (1744539181)
[ 4034.056402] Lustre: DEBUG MARKER: == sanity-quota test 67: quota pools recalculation ======= 06:13:15 (1744539195)
[ 4037.483612] Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
[ 4038.970031] Lustre: 20798:0:(qsd_reint.c:240:qsd_reint_index()) lustre-OST0000: index version for fid [0x200000005:0x1007:0x0] is 0, but index isn't empty (1)
[ 4039.681542] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
[ 4040.626465] Lustre: DEBUG MARKER: oleg312-server.virtnet: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
[ 4041.627339] Lustre: DEBUG MARKER: Write...
[ 4049.997716] LustreError: 22286:0:(mgs_handler.c:848:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 4053.813782] Lustre: DEBUG MARKER: Write...
[ 4056.988749] Lustre: DEBUG MARKER: Write...
[ 4093.960391] Lustre: DEBUG MARKER: == sanity-quota test 68: slave number in quota pool changed after each add/remove OST ========================================================== 06:14:15 (1744539255)
[ 4103.534119] LustreError: 27247:0:(mgs_handler.c:848:mgs_iocontrol_pool()) OBD_IOC_POOL err -17, cmd CE021 for pool lustre.qpool1
[ 4123.833085] Lustre: DEBUG MARKER: == sanity-quota test 69: EDQUOT at one of pools shouldn't affect DOM ========================================================== 06:14:45 (1744539285)
[ 4128.171209] LustreError: 30016:0:(qsd_reint.c:617:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed.
[ 4128.177845] LustreError: 30016:0:(qsd_reint.c:617:qqi_reint_delayed()) Skipped 7 previous similar messages
[ 4134.645031] Lustre: DEBUG MARKER: User quota (block hardlimit:200 MB)
[ 4135.170204] Lustre: DEBUG MARKER: User quota (block hardlimit:10 MB)
[ 4168.669320] Lustre: DEBUG MARKER: == sanity-quota test 70a: check lfs setquota/quota with a pool option ========================================================== 06:15:29 (1744539329)
[ 4187.758063] Lustre: DEBUG MARKER: == sanity-quota test 70b: lfs setquota pool works properly ========================================================== 06:15:49 (1744539349)
[ 4202.518778] Lustre: DEBUG MARKER: == sanity-quota test 71a: Check PFL with quota pools ===== 06:16:03 (1744539363)
[ 4205.915296] Lustre: DEBUG MARKER: User quota (block hardlimit:100 MB)
[ 4217.238281] Lustre: DEBUG MARKER: Write...
[ 4217.973627] Lustre: DEBUG MARKER: Files for user (quota_usr), count=1:
[ 4218.537414] Lustre: DEBUG MARKER: sanity-quota test_71a: @@@@@@ FAIL: user write failure, but expect success