00010000:00000001:2.0F:1773946216.993845:0:121148:0:(ldlm_pool.c:458:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:2.0:1773946216.993858:0:121148:0:(ldlm_pool.c:462:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0F:1773946216.993890:0:193418:0:(genops.c:1914:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1773946216.993898:0:193418:0:(genops.c:1928:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1773946216.993901:0:193418:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00000001:2.0:1773946217.206501:0:193823:0:(llite_lib.c:4092:ll_show_options()) Process leaving (rc=0 : 0 : 0) 00000080:00000001:1.0F:1773946217.579869:0:193889:0:(llite_lib.c:4092:ll_show_options()) Process leaving (rc=0 : 0 : 0) 00000080:00000001:2.0:1773946217.938868:0:193982:0:(llite_lib.c:4092:ll_show_options()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:2.0:1773946218.017859:0:121148:0:(ldlm_pool.c:458:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:2.0:1773946218.017864:0:121148:0:(ldlm_pool.c:462:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1773946218.017900:0:193419:0:(genops.c:1914:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1773946218.017905:0:193419:0:(genops.c:1928:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1773946218.017908:0:193419:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:0.0:1773946218.017956:0:191265:0:(lib-socket.c:453:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.204.165:988 00000400:00000100:0.0:1773946218.017992:0:191265:0:(acceptor.c:111:lnet_connect_console_error()) Connection to 192.168.204.165@tcp at host 192.168.204.165 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000400:00000200:0.0:1773946218.018004:0:191265:0:(router.c:1750:lnet_notify()) 192.168.204.55@tcp notifying 192.168.204.165@tcp: down 00000800:00000200:0.0:1773946218.018016:0:191265:0:(socklnd.c:1681:ksocknal_notify_gw_down()) gw 192.168.204.165@tcp down 00000800:00000100:0.0:1773946218.018024:0:191265:0:(socklnd_cb.c:422:ksocknal_txlist_done()) Deleting packet type 2 len 0 0@<0:0>->192.168.204.165@tcp 00000800:00000010:0.0:1773946218.018028:0:191265:0:(socklnd_cb.c:98:ksocknal_free_tx()) kfreed 'tx': 232 at 000000005cb2b769. 00000400:00000200:0.0:1773946218.018034:0:191265:0:(lib-msg.c:1048:lnet_is_health_check()) health check = 1, status = -113, hstatus = 8 00000400:00000200:0.0:1773946218.018040:0:191265:0:(lib-msg.c:824:lnet_health_check()) health check: 192.168.204.55@tcp->192.168.204.165@tcp: GET: REMOTE_DROPPED 00000400:00000200:0.0:1773946218.018047:0:191265:0:(peer.c:3081:lnet_discovery_event_handler()) Received event: 5 00000400:00000200:0.0:1773946218.018051:0:191265:0:(peer.c:3033:lnet_discovery_event_send()) Ping Send to 192.168.204.165@tcp: 1 00000400:00000200:0.0:1773946218.018057:0:191265:0:(peer.c:2405:lnet_peer_queue_for_discovery()) Queue peer 192.168.204.165@tcp: -114 00000400:00000200:1.0:1773946218.018322:0:191274:0:(peer.c:3988:lnet_peer_discovery_wait_for_work()) woken: 0 00000400:00000200:1.0:1773946218.018335:0:191274:0:(peer.c:4100:lnet_peer_discovery()) peer 192.168.204.165@tcp(000000008b369f39) state 0x841 00000400:00000001:1.0:1773946218.018341:0:191274:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:1.0:1773946218.018342:0:191274:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000400:00000200:1.0:1773946218.018345:0:191274:0:(lib-md.c:46:lnet_md_unlink()) Unlinking md 0000000043bdd2e0 00000400:00000010:1.0:1773946218.018347:0:191274:0:(lib-lnet.h:312:lnet_md_free()) slab-freed 'md': 136 at 0000000043bdd2e0. 00000400:00000200:1.0:1773946218.018353:0:191274:0:(peer.c:3081:lnet_discovery_event_handler()) Received event: 6 00000400:00000200:1.0:1773946218.018356:0:191274:0:(peer.c:2405:lnet_peer_queue_for_discovery()) Queue peer 192.168.204.165@tcp: -114 00000400:00000010:1.0:1773946218.018360:0:191274:0:(api-ni.c:1839:lnet_ping_buffer_free()) kfreed 'pbuf': 281 at 000000006fa39dc6. 00000400:00000200:1.0:1773946218.018365:0:191274:0:(peer.c:3747:lnet_peer_ping_failed()) peer 192.168.204.165@tcp:-113 00000400:00000200:1.0:1773946218.018368:0:191274:0:(peer.c:4118:lnet_peer_discovery()) peer 192.168.204.165@tcp(000000008b369f39) state 0x2041 rc -113 00000400:00000200:1.0:1773946218.018372:0:191274:0:(peer.c:2421:lnet_peer_discovery_complete()) Discovery complete. Dequeue peer 192.168.204.165@tcp 00000001:00000001:1.0:1773946218.063373:0:194018:0:(debug.c:644:libcfs_debug_mark_buffer()) ************************************************** 00000001:02000400:1.0:1773946218.063379:0:194018:0:(debug.c:645:libcfs_debug_mark_buffer()) DEBUG MARKER: == conf-sanity test 162: replace nids with -o noclient === 14:50:17 (1773946217) 00000001:00000001:1.0:1773946218.066827:0:194018:0:(debug.c:647:libcfs_debug_mark_buffer()) ************************************************** 00010000:00000001:2.0:1773946219.041849:0:121148:0:(ldlm_pool.c:458:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:2.0:1773946219.041856:0:121148:0:(ldlm_pool.c:462:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:0.0:1773946219.041921:0:193418:0:(genops.c:1914:obd_stale_export_get()) Process entered 00000020:00000001:0.0:1773946219.041925:0:193418:0:(genops.c:1928:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:0.0:1773946219.041929:0:193418:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000040:2.0:1773946220.065874:0:152776:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e4604cb-9f63-473b-92ae-61bdc80df14a->MGS: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1773946220.065886:0:152776:0:(pinger.c:119:ptlrpc_ping()) Process entered 00000100:00000010:2.0:1773946220.065893:0:152776:0:(client.c:592:ptlrpc_request_cache_alloc()) slab-alloced 'req': 888 at 0000000024352b9e. 00000020:00000040:2.0:1773946220.065900:0:152776:0:(genops.c:1229:class_import_get()) import 00000000bab840cb refcount=4 obd=MGC192.168.204.155@tcp 00000100:00000001:2.0:1773946220.065906:0:152776:0:(client.c:844:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1773946220.065909:0:152776:0:(sec.c:440:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1773946220.065914:0:152776:0:(sec.c:464:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000010:2.0:1773946220.065922:0:152776:0:(sec_null.c:146:null_alloc_reqbuf()) kmalloced '(req->rq_reqbuf)': 256 at 000000009502c4c6. 00000100:00000001:2.0:1773946220.065930:0:152776:0:(client.c:924:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1773946220.065941:0:152776:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging 8e4604cb-9f63-473b-92ae-61bdc80df14a->MGS req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00000001:2.0:1773946220.065954:0:152776:0:(jobid.c:936:lustre_get_jobid()) Process entered 00000100:00000001:2.0:1773946220.065956:0:152776:0:(jobid.c:947:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1773946220.065963:0:152776:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [0000000024352b9e] to pc [ptlrpcd_00_02+2] req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1773946220.066003:0:152776:0:(pinger.c:142:ptlrpc_ping()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1773946220.066007:0:152776:0:(pinger.c:229:ptlrpc_pinger_process_import()) c8ed80ad-4461-452a-b4bd-ffd70870ac7a->lustre-MDT0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:2.0:1773946220.066011:0:152776:0:(pinger.c:119:ptlrpc_ping()) Process entered 00000100:00000001:1.0:1773946220.066013:0:191279:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000010:2.0:1773946220.066015:0:152776:0:(client.c:592:ptlrpc_request_cache_alloc()) slab-alloced 'req': 888 at 000000003782171d. 00000020:00000040:2.0:1773946220.066020:0:152776:0:(genops.c:1229:class_import_get()) import 000000000f7b6da0 refcount=4 obd=lustre-MDT0000-mdc-ffff9caa6b722800 00000100:00000001:2.0:1773946220.066023:0:152776:0:(client.c:844:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1773946220.066025:0:152776:0:(sec.c:440:sptlrpc_req_get_ctx()) Process entered 00000008:00000010:1.0:1773946220.066025:0:191279:0:(osc_dev.c:88:osc_key_init()) slab-alloced 'info': 2856 at 00000000cbe5ac76. 02000000:00000001:2.0:1773946220.066027:0:152776:0:(sec.c:464:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00020000:00000010:1.0:1773946220.066033:0:191279:0:(lov_dev.c:78:lov_key_init()) slab-alloced 'info': 192 at 00000000d4af1d4e. 00000080:00000010:1.0:1773946220.066039:0:191279:0:(vvp_dev.c:68:ll_thread_key_init()) slab-alloced 'lti': 168 at 000000009e687c6d. 02000000:00000010:2.0:1773946220.066044:0:152776:0:(sec_null.c:146:null_alloc_reqbuf()) kmalloced '(req->rq_reqbuf)': 256 at 0000000032c738f6. 00000080:00000010:1.0:1773946220.066045:0:191279:0:(vvp_dev.c:119:vvp_thread_key_init()) slab-alloced 'vti': 584 at 00000000358bbd40. 00000100:00000001:2.0:1773946220.066051:0:152776:0:(client.c:924:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000008:00000010:1.0:1773946220.066053:0:191279:0:(osc_dev.c:115:osc_session_init()) slab-alloced 'info': 352 at 000000004f1e578c. 00020000:00000010:1.0:1773946220.066058:0:191279:0:(lov_dev.c:116:lov_session_key_init()) slab-alloced 'info': 576 at 0000000048eba467. 00000100:00000040:2.0:1773946220.066059:0:152776:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging c8ed80ad-4461-452a-b4bd-ffd70870ac7a->lustre-MDT0000_UUID req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000080:00000010:1.0:1773946220.066064:0:191279:0:(vvp_dev.c:94:vvp_session_key_init()) slab-alloced 'session': 240 at 0000000057bf63b2. 00000100:00100000:1.0:1773946220.066068:0:191279:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [2->3] 00000100:00000001:1.0:1773946220.066070:0:191279:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1773946220.066078:0:191279:0:(client.c:2614:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1773946220.066082:0:191279:0:(client.c:2642:ptlrpc_set_next_timeout()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1773946220.066086:0:152776:0:(jobid.c:936:lustre_get_jobid()) Process entered 00000100:00000001:1.0:1773946220.066087:0:191279:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1773946220.066091:0:152776:0:(jobid.c:947:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1773946220.066092:0:191279:0:(client.c:1974:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1773946220.066098:0:191279:0:(client.c:1797:ptlrpc_send_new_req()) Process entered 00000100:00000040:2.0:1773946220.066101:0:152776:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [000000003782171d] to pc [ptlrpcd_00_03+3] req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1773946220.066108:0:191279:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1773946220.066123:0:152776:0:(pinger.c:142:ptlrpc_ping()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1773946220.066126:0:191279:0:(client.c:1363:ptlrpc_import_delay_req()) Process entered 00000100:00000040:2.0:1773946220.066128:0:152776:0:(pinger.c:229:ptlrpc_pinger_process_import()) c8ed80ad-4461-452a-b4bd-ffd70870ac7a->lustre-MDT0001_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:1.0:1773946220.066131:0:191279:0:(client.c:1421:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.066133:0:152776:0:(pinger.c:119:ptlrpc_ping()) Process entered 00000100:00000010:2.0:1773946220.066138:0:152776:0:(client.c:592:ptlrpc_request_cache_alloc()) slab-alloced 'req': 888 at 00000000d4f3130d. 00000020:00000040:2.0:1773946220.066143:0:152776:0:(genops.c:1229:class_import_get()) import 0000000063e09825 refcount=4 obd=lustre-MDT0001-mdc-ffff9caa6b722800 02000000:00000001:1.0:1773946220.066145:0:191279:0:(sec.c:715:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1773946220.066147:0:152776:0:(client.c:844:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:1.0:1773946220.066149:0:191279:0:(sec.c:746:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 02000000:00000001:2.0:1773946220.066151:0:152776:0:(sec.c:440:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1773946220.066156:0:152776:0:(sec.c:464:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:1.0:1773946220.066156:0:191279:0:(sec.c:774:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0F:1773946220.066164:0:191278:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000010:2.0:1773946220.066165:0:152776:0:(sec_null.c:146:null_alloc_reqbuf()) kmalloced '(req->rq_reqbuf)': 256 at 00000000c4aa4116. 00000100:00100000:1.0:1773946220.066165:0:191279:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@0000000024352b9e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:8e4604cb-9f63-473b-92ae-61bdc80df14a:191279:1860117379162624:192.168.204.155@tcp:400: 00000100:00000001:2.0:1773946220.066171:0:152776:0:(client.c:924:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1773946220.066173:0:191279:0:(niobuf.c:865:ptl_send_rpc()) Process entered 00000008:00000010:3.0:1773946220.066196:0:191278:0:(osc_dev.c:88:osc_key_init()) slab-alloced 'info': 2856 at 00000000ccf01804. 00000100:00000040:2.0:1773946220.066198:0:152776:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging c8ed80ad-4461-452a-b4bd-ffd70870ac7a->lustre-MDT0001_UUID req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:1.0:1773946220.066201:0:191279:0:(sec.c:1068:sptlrpc_cli_wrap_request()) Process entered 00020000:00000010:3.0:1773946220.066204:0:191278:0:(lov_dev.c:78:lov_key_init()) slab-alloced 'info': 192 at 000000008b36385d. 02000000:00000001:1.0:1773946220.066206:0:191279:0:(sec.c:1105:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000080:00000010:3.0:1773946220.066212:0:191278:0:(vvp_dev.c:68:ll_thread_key_init()) slab-alloced 'lti': 168 at 00000000c6c97fae. 00000100:00000001:2.0:1773946220.066215:0:152776:0:(jobid.c:936:lustre_get_jobid()) Process entered 02000000:00000001:1.0:1773946220.066217:0:191279:0:(sec.c:1852:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:2.0:1773946220.066219:0:152776:0:(jobid.c:947:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000080:00000010:3.0:1773946220.066220:0:191278:0:(vvp_dev.c:119:vvp_thread_key_init()) slab-alloced 'vti': 584 at 0000000052917739. 02000000:00000010:1.0:1773946220.066224:0:191279:0:(sec_null.c:189:null_alloc_repbuf()) kmalloced '(req->rq_repbuf)': 512 at 0000000037440cf2. 00000008:00000010:3.0:1773946220.066229:0:191278:0:(osc_dev.c:115:osc_session_init()) slab-alloced 'info': 352 at 00000000c30efbbd. 00000100:00000040:2.0:1773946220.066229:0:152776:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000d4f3130d] to pc [ptlrpcd_00_00+0] req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 02000000:00000001:1.0:1773946220.066230:0:191279:0:(sec.c:1862:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00020000:00000010:3.0:1773946220.066237:0:191278:0:(lov_dev.c:116:lov_session_key_init()) slab-alloced 'info': 576 at 00000000a257286c. 00000400:00000010:1.0:1773946220.066242:0:191279:0:(lib-me.c:70:LNetMEAttach()) slab-alloced 'me': 88 at 00000000f7b94d97. 00000080:00000010:3.0:1773946220.066246:0:191278:0:(vvp_dev.c:94:vvp_session_key_init()) slab-alloced 'session': 240 at 00000000dcac3d12. 00000100:00100000:3.0:1773946220.066251:0:191278:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [3->2] 00000400:00000010:1.0:1773946220.066253:0:191279:0:(lib-md.c:131:lnet_md_build()) slab-alloced 'lmd': 136 at 0000000048355ba7. 00000100:00000001:3.0:1773946220.066254:0:191278:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1773946220.066261:0:191278:0:(client.c:2614:ptlrpc_set_next_timeout()) Process entered 00000100:00000200:1.0:1773946220.066263:0:191279:0:(niobuf.c:1096:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1860117379162624, portal 25 00000100:00000001:3.0:1773946220.066264:0:191278:0:(client.c:2642:ptlrpc_set_next_timeout()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1773946220.066268:0:152776:0:(pinger.c:142:ptlrpc_ping()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1773946220.066270:0:191278:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:1.0:1773946220.066270:0:191279:0:(client.c:3244:ptlrpc_request_addref()) Process entered 00000100:00000040:2.0:1773946220.066274:0:152776:0:(pinger.c:229:ptlrpc_pinger_process_import()) c8ed80ad-4461-452a-b4bd-ffd70870ac7a->lustre-OST0000_UUID: level FULL/9 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00000001:3.0:1773946220.066275:0:191278:0:(client.c:1974:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1773946220.066275:0:191279:0:(client.c:3246:ptlrpc_request_addref()) Process leaving (rc=18446634853835497344 : -109219874054272 : ffff9caa44345f80) 00000100:00000001:3.0:1773946220.066279:0:191278:0:(client.c:1797:ptlrpc_send_new_req()) Process entered 00000100:00000001:2.0:1773946220.066284:0:152776:0:(pinger.c:119:ptlrpc_ping()) Process entered 00000100:00000040:1.0:1773946220.066287:0:191279:0:(niobuf.c:1116:ptl_send_rpc()) @@@ send flags=200 req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1773946220.066289:0:191278:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000010:2.0:1773946220.066291:0:152776:0:(client.c:592:ptlrpc_request_cache_alloc()) slab-alloced 'req': 888 at 00000000e7bddb4a. 00000020:00000040:2.0:1773946220.066302:0:152776:0:(genops.c:1229:class_import_get()) import 00000000dcd78867 refcount=3 obd=lustre-OST0000-osc-ffff9caa6b722800 00000100:00000001:2.0:1773946220.066310:0:152776:0:(client.c:844:ptlrpc_request_bufs_pack()) Process entered 02000000:00000001:2.0:1773946220.066314:0:152776:0:(sec.c:440:sptlrpc_req_get_ctx()) Process entered 02000000:00000001:2.0:1773946220.066317:0:152776:0:(sec.c:464:sptlrpc_req_get_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1773946220.066321:0:191279:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000001:3.0:1773946220.066325:0:191278:0:(client.c:1363:ptlrpc_import_delay_req()) Process entered 02000000:00000010:2.0:1773946220.066326:0:152776:0:(sec_null.c:146:null_alloc_reqbuf()) kmalloced '(req->rq_reqbuf)': 256 at 0000000080043654. 00000100:00000001:3.0:1773946220.066329:0:191278:0:(client.c:1421:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:1.0:1773946220.066330:0:191279:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.204.155@tcp 00000400:00000010:1.0:1773946220.066336:0:191279:0:(lib-md.c:131:lnet_md_build()) slab-alloced 'lmd': 136 at 00000000d733aa62. 02000000:00000001:3.0:1773946220.066337:0:191278:0:(sec.c:715:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:2.0:1773946220.066339:0:152776:0:(client.c:924:ptlrpc_request_bufs_pack()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1773946220.066342:0:191278:0:(sec.c:746:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000200:1.0:1773946220.066342:0:191279:0:(niobuf.c:94:ptl_send_buf()) Sending 224 bytes to portal 26, xid 1860117379162624, offset 0 00000100:00000040:2.0:1773946220.066348:0:152776:0:(pinger.c:133:ptlrpc_ping()) @@@ pinging c8ed80ad-4461-452a-b4bd-ffd70870ac7a->lustre-OST0000_UUID req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 02000000:00000001:3.0:1773946220.066350:0:191278:0:(sec.c:774:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:1.0:1773946220.066357:0:191279:0:(lib-move.c:5316:LNetPut()) LNetPut -> 12345-192.168.204.155@tcp 00000100:00100000:3.0:1773946220.066360:0:191278:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@000000003782171d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:c8ed80ad-4461-452a-b4bd-ffd70870ac7a:191278:1860117379162752:192.168.204.155@tcp:400: 00000100:00000001:3.0:1773946220.066371:0:191278:0:(niobuf.c:865:ptl_send_rpc()) Process entered 00000100:00000001:2.0:1773946220.066374:0:152776:0:(jobid.c:936:lustre_get_jobid()) Process entered 02000000:00000001:3.0:1773946220.066377:0:191278:0:(sec.c:1068:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:2.0:1773946220.066378:0:152776:0:(jobid.c:947:lustre_get_jobid()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:1.0:1773946220.066378:0:191279:0:(lib-move.c:2973:lnet_handle_send_case_locked()) Source ANY to MR: 192.168.204.155@tcp local destination 02000000:00000001:3.0:1773946220.066381:0:191278:0:(sec.c:1105:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1773946220.066387:0:191278:0:(sec.c:1852:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000040:2.0:1773946220.066389:0:152776:0:(ptlrpcd.c:235:ptlrpcd_add_req()) @@@ add req [00000000e7bddb4a] to pc [ptlrpcd_00_01+1] req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000400:00000200:1.0:1773946220.066389:0:191279:0:(lib-move.c:2603:lnet_find_best_ni_on_local_net()) selected best_lpn tcp 02000000:00000010:3.0:1773946220.066393:0:191278:0:(sec_null.c:189:null_alloc_repbuf()) kmalloced '(req->rq_repbuf)': 512 at 000000004a58fcba. 00000400:00000200:1.0:1773946220.066397:0:191279:0:(lib-move.c:1789:lnet_get_best_ni()) selected best_ni 192.168.204.55@tcp 02000000:00000001:3.0:1773946220.066399:0:191278:0:(sec.c:1862:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:1.0:1773946220.066405:0:191279:0:(lib-move.c:1343:lnet_select_peer_ni()) 192.168.204.55@tcp lpni_is_preferred = 0 00000400:00000010:3.0:1773946220.066408:0:191278:0:(lib-me.c:70:LNetMEAttach()) slab-alloced 'me': 88 at 00000000458d2b58. 00000400:00000200:1.0:1773946220.066413:0:191279:0:(lib-move.c:1412:lnet_select_peer_ni()) sd_best_lpni = 192.168.204.155@tcp 00000400:00000010:3.0:1773946220.066423:0:191278:0:(lib-md.c:131:lnet_md_build()) slab-alloced 'lmd': 136 at 0000000073d78dc2. 00000100:00000001:2.0:1773946220.066427:0:152776:0:(pinger.c:142:ptlrpc_ping()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:1.0:1773946220.066428:0:191279:0:(lib-move.c:1909:lnet_handle_send()) 192.168.204.55@tcp NI seq info: [72:72:256:4294967295] 192.168.204.155@tcp LPNI seq info [71:71:8:4294967295] 00000100:00000200:3.0:1773946220.066431:0:191278:0:(niobuf.c:1096:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1860117379162752, portal 10 00000100:00000040:2.0:1773946220.066434:0:152776:0:(pinger.c:302:ptlrpc_pinger_main()) next wakeup in 5 (4972) 00000100:00000001:3.0:1773946220.066439:0:191278:0:(client.c:3244:ptlrpc_request_addref()) Process entered 00000100:00000001:3.0:1773946220.066443:0:191278:0:(client.c:3246:ptlrpc_request_addref()) Process leaving (rc=18446634853835491968 : -109219874059648 : ffff9caa44344a80) 00000400:00000200:1.0:1773946220.066451:0:191279:0:(lib-move.c:2024:lnet_handle_send()) TRACE: 192.168.204.55@tcp(192.168.204.55@tcp:) -> 192.168.204.155@tcp(192.168.204.155@tcp:192.168.204.155@tcp) : PUT try# 0 00000100:00000001:0.0:1773946220.066451:0:191276:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000020:00000010:3.0:1773946220.066456:0:191278:0:(lprocfs_status.c:1408:lprocfs_stats_alloc_one()) alloc '(stats->ls_percpu[cpuid])': 4608 at 00000000e029d7c3. 00000080:00000010:0.0:1773946220.066461:0:191276:0:(vvp_dev.c:68:ll_thread_key_init()) slab-alloced 'lti': 168 at 00000000cf1f7fdf. 00000800:00000200:1.0:1773946220.066462:0:191279:0:(socklnd_cb.c:1001:ksocknal_send()) sending 224 bytes in 1 frags to 12345-192.168.204.155@tcp 00010000:00000001:2.0:1773946220.066463:0:121148:0:(ldlm_pool.c:458:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:2.0:1773946220.066470:0:121148:0:(ldlm_pool.c:462:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000080:00000010:0.0:1773946220.066471:0:191276:0:(vvp_dev.c:119:vvp_thread_key_init()) slab-alloced 'vti': 584 at 0000000052f2717f. 00000800:00000010:1.0:1773946220.066473:0:191279:0:(socklnd_cb.c:41:ksocknal_alloc_tx()) alloc '(tx)': 248 at 00000000828a44c4. 00000100:00000040:3.0:1773946220.066477:0:191278:0:(niobuf.c:1116:ptl_send_rpc()) @@@ send flags=200 req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000080:00000010:0.0:1773946220.066481:0:191276:0:(vvp_dev.c:94:vvp_session_key_init()) slab-alloced 'session': 240 at 000000005e50d12e. 00000800:00000200:1.0:1773946220.066488:0:191279:0:(socklnd.c:214:ksocknal_find_peer_locked()) got peer_ni [00000000ed7f6003] -> 12345-192.168.204.155@tcp (5) 00000100:00000001:0.0:1773946220.066488:0:191276:0:(client.c:1974:ptlrpc_check_set()) Process entered 00000100:00000001:0.0:1773946220.066492:0:191276:0:(client.c:1797:ptlrpc_send_new_req()) Process entered 00000100:00000001:3.0:1773946220.066496:0:191278:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000040:0.0:1773946220.066502:0:191276:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1773946220.066506:0:191278:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.204.155@tcp 00000020:00000001:2.0:1773946220.066506:0:193419:0:(genops.c:1914:obd_stale_export_get()) Process entered 00000020:00000001:2.0:1773946220.066513:0:193419:0:(genops.c:1928:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00000800:00000200:1.0:1773946220.066513:0:191279:0:(socklnd_cb.c:748:ksocknal_queue_tx_locked()) Sending to 12345-192.168.204.155@tcp ip 192.168.204.155:988 00000400:00000010:3.0:1773946220.066516:0:191278:0:(lib-md.c:131:lnet_md_build()) slab-alloced 'lmd': 136 at 000000002fa680cc. 00010000:00010000:2.0:1773946220.066520:0:193419:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000200:1.0:1773946220.066522:0:191279:0:(socklnd_cb.c:767:ksocknal_queue_tx_locked()) Packet 00000000828a44c4 type 1, nob 320 niov 1 nkiov 1 00000100:00000200:3.0:1773946220.066523:0:191278:0:(niobuf.c:94:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1860117379162752, offset 0 00000100:00000001:0.0:1773946220.066532:0:191276:0:(client.c:1363:ptlrpc_import_delay_req()) Process entered 00000100:00000001:0.0:1773946220.066534:0:191276:0:(client.c:1421:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.066538:0:191277:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000400:00000200:3.0:1773946220.066539:0:191278:0:(lib-move.c:5316:LNetPut()) LNetPut -> 12345-192.168.204.155@tcp 02000000:00000001:0.0:1773946220.066540:0:191276:0:(sec.c:715:sptlrpc_req_refresh_ctx()) Process entered 00000100:00000001:1.0:1773946220.066544:0:191279:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1773946220.066546:0:191276:0:(sec.c:746:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000080:00000010:2.0:1773946220.066548:0:191277:0:(vvp_dev.c:68:ll_thread_key_init()) slab-alloced 'lti': 168 at 00000000bca1bffa. 00000100:00000001:1.0:1773946220.066551:0:191279:0:(niobuf.c:662:kick_cpu_latency()) MGC192.168.204.155@tcp: PM QoS new deadline estimation for cpu 0 is 4299640375 00000400:00000200:3.0:1773946220.066553:0:191278:0:(lib-move.c:2973:lnet_handle_send_case_locked()) Source ANY to MR: 192.168.204.155@tcp local destination 02000000:00000001:0.0:1773946220.066553:0:191276:0:(sec.c:774:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000080:00000010:2.0:1773946220.066556:0:191277:0:(vvp_dev.c:119:vvp_thread_key_init()) slab-alloced 'vti': 584 at 00000000f279d83c. 00000100:00100000:0.0:1773946220.066563:0:191276:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@00000000d4f3130d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:c8ed80ad-4461-452a-b4bd-ffd70870ac7a:191276:1860117379162880:192.168.204.155@tcp:400: 00000400:00000200:3.0:1773946220.066564:0:191278:0:(lib-move.c:2603:lnet_find_best_ni_on_local_net()) selected best_lpn tcp 00000100:00000001:1.0:1773946220.066564:0:191279:0:(niobuf.c:684:kick_cpu_latency()) MGC192.168.204.155@tcp: PM QoS request active for cpu 0, simply extend its deadline from 4299635273 00000080:00000010:2.0:1773946220.066565:0:191277:0:(vvp_dev.c:94:vvp_session_key_init()) slab-alloced 'session': 240 at 00000000f4cf8080. 00000100:00000001:2.0:1773946220.066571:0:191277:0:(client.c:1974:ptlrpc_check_set()) Process entered 00000400:00000200:3.0:1773946220.066573:0:191278:0:(lib-move.c:1789:lnet_get_best_ni()) selected best_ni 192.168.204.55@tcp 00000100:00000001:1.0:1773946220.066574:0:191279:0:(niobuf.c:662:kick_cpu_latency()) MGC192.168.204.155@tcp: PM QoS new deadline estimation for cpu 1 is 4299640375 00000100:00000001:2.0:1773946220.066575:0:191277:0:(client.c:1797:ptlrpc_send_new_req()) Process entered 00000100:00000001:0.0:1773946220.066577:0:191276:0:(niobuf.c:865:ptl_send_rpc()) Process entered 00000400:00000200:3.0:1773946220.066581:0:191278:0:(lib-move.c:1343:lnet_select_peer_ni()) 192.168.204.55@tcp lpni_is_preferred = 0 02000000:00000001:0.0:1773946220.066582:0:191276:0:(sec.c:1068:sptlrpc_cli_wrap_request()) Process entered 00000100:00000001:1.0:1773946220.066583:0:191279:0:(niobuf.c:684:kick_cpu_latency()) MGC192.168.204.155@tcp: PM QoS request active for cpu 1, simply extend its deadline from 4299635273 00000100:00000040:2.0:1773946220.066586:0:191277:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from New to Rpc req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 0 ref 1 fl New:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 02000000:00000001:0.0:1773946220.066586:0:191276:0:(sec.c:1105:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:3.0:1773946220.066591:0:191278:0:(lib-move.c:1412:lnet_select_peer_ni()) sd_best_lpni = 192.168.204.155@tcp 02000000:00000001:0.0:1773946220.066592:0:191276:0:(sec.c:1852:sptlrpc_cli_alloc_repbuf()) Process entered 00000100:00000001:1.0:1773946220.066593:0:191279:0:(niobuf.c:662:kick_cpu_latency()) MGC192.168.204.155@tcp: PM QoS new deadline estimation for cpu 2 is 4299640375 02000000:00000010:0.0:1773946220.066599:0:191276:0:(sec_null.c:189:null_alloc_repbuf()) kmalloced '(req->rq_repbuf)': 512 at 0000000069a065bb. 00000100:00000001:1.0:1773946220.066600:0:191279:0:(niobuf.c:684:kick_cpu_latency()) MGC192.168.204.155@tcp: PM QoS request active for cpu 2, simply extend its deadline from 4299635273 00000400:00000200:3.0:1773946220.066604:0:191278:0:(lib-move.c:1909:lnet_handle_send()) 192.168.204.55@tcp NI seq info: [73:73:255:4294967295] 192.168.204.155@tcp LPNI seq info [72:72:7:4294967295] 02000000:00000001:0.0:1773946220.066607:0:191276:0:(sec.c:1862:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1773946220.066609:0:191279:0:(niobuf.c:662:kick_cpu_latency()) MGC192.168.204.155@tcp: PM QoS new deadline estimation for cpu 3 is 4299640375 00000400:00000010:0.0:1773946220.066616:0:191276:0:(lib-me.c:70:LNetMEAttach()) slab-alloced 'me': 88 at 00000000f10cd56f. 00000100:00000001:2.0:1773946220.066618:0:191277:0:(client.c:1363:ptlrpc_import_delay_req()) Process entered 00000100:00000001:1.0:1773946220.066618:0:191279:0:(niobuf.c:684:kick_cpu_latency()) MGC192.168.204.155@tcp: PM QoS request active for cpu 3, simply extend its deadline from 4299635273 00000100:00000001:2.0:1773946220.066623:0:191277:0:(client.c:1421:ptlrpc_import_delay_req()) Process leaving (rc=0 : 0 : 0) 00000400:00000010:0.0:1773946220.066624:0:191276:0:(lib-md.c:131:lnet_md_build()) slab-alloced 'lmd': 136 at 000000005c421093. 02000000:00000001:2.0:1773946220.066630:0:191277:0:(sec.c:715:sptlrpc_req_refresh_ctx()) Process entered 00000400:00000200:3.0:1773946220.066631:0:191278:0:(lib-move.c:2024:lnet_handle_send()) TRACE: 192.168.204.55@tcp(192.168.204.55@tcp:) -> 192.168.204.155@tcp(192.168.204.155@tcp:192.168.204.155@tcp) : PUT try# 0 00000100:00000001:1.0:1773946220.066633:0:191279:0:(niobuf.c:1134:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000200:0.0:1773946220.066633:0:191276:0:(niobuf.c:1096:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1860117379162880, portal 10 02000000:00000001:2.0:1773946220.066636:0:191277:0:(sec.c:746:sptlrpc_req_refresh_ctx()) Process leaving via out_sec_put (rc=0 : 0 : 0x0) 00000100:00000001:0.0:1773946220.066641:0:191276:0:(client.c:3244:ptlrpc_request_addref()) Process entered 00000100:00000001:1.0:1773946220.066642:0:191279:0:(client.c:1925:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1773946220.066644:0:191277:0:(sec.c:774:sptlrpc_req_refresh_ctx()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1773946220.066645:0:191276:0:(client.c:3246:ptlrpc_request_addref()) Process leaving (rc=18446634853835489280 : -109219874062336 : ffff9caa44344000) 00000100:00000001:1.0:1773946220.066647:0:191279:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000800:00000200:3.0:1773946220.066650:0:191278:0:(socklnd_cb.c:1001:ksocknal_send()) sending 224 bytes in 1 frags to 12345-192.168.204.155@tcp 00000100:00000001:1.0:1773946220.066654:0:191279:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00100000:2.0:1773946220.066655:0:191277:0:(client.c:1903:ptlrpc_send_new_req()) Sending RPC req@00000000e7bddb4a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:c8ed80ad-4461-452a-b4bd-ffd70870ac7a:191277:1860117379163008:192.168.204.155@tcp:400: 00000800:00000010:3.0:1773946220.066658:0:191278:0:(socklnd_cb.c:41:ksocknal_alloc_tx()) alloc '(tx)': 248 at 0000000083731e81. 00000100:00000001:1.0:1773946220.066660:0:191279:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000040:0.0:1773946220.066660:0:191276:0:(niobuf.c:1116:ptl_send_rpc()) @@@ send flags=200 req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1773946220.066665:0:191279:0:(client.c:1974:ptlrpc_check_set()) Process entered 00000100:00000001:2.0:1773946220.066667:0:191277:0:(niobuf.c:865:ptl_send_rpc()) Process entered 00000100:00000001:1.0:1773946220.066670:0:191279:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000800:00000200:3.0:1773946220.066672:0:191278:0:(socklnd.c:214:ksocknal_find_peer_locked()) got peer_ni [00000000ed7f6003] -> 12345-192.168.204.155@tcp (5) 02000000:00000001:2.0:1773946220.066672:0:191277:0:(sec.c:1068:sptlrpc_cli_wrap_request()) Process entered 02000000:00000001:2.0:1773946220.066678:0:191277:0:(sec.c:1105:sptlrpc_cli_wrap_request()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1773946220.066680:0:191279:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1773946220.066684:0:191277:0:(sec.c:1852:sptlrpc_cli_alloc_repbuf()) Process entered 00000800:00000200:3.0:1773946220.066686:0:191278:0:(socklnd_cb.c:748:ksocknal_queue_tx_locked()) Sending to 12345-192.168.204.155@tcp ip 192.168.204.155:988 00000100:00000001:0.0:1773946220.066690:0:191276:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000800:00000200:3.0:1773946220.066692:0:191278:0:(socklnd_cb.c:767:ksocknal_queue_tx_locked()) Packet 0000000083731e81 type 1, nob 320 niov 1 nkiov 1 02000000:00000010:2.0:1773946220.066696:0:191277:0:(sec_null.c:189:null_alloc_repbuf()) kmalloced '(req->rq_repbuf)': 512 at 00000000c47662ca. 00000100:00000001:3.0:1773946220.066699:0:191278:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:0.0:1773946220.066699:0:191276:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.204.155@tcp 02000000:00000001:2.0:1773946220.066702:0:191277:0:(sec.c:1862:sptlrpc_cli_alloc_repbuf()) Process leaving (rc=0 : 0 : 0) 00000400:00000010:0.0:1773946220.066705:0:191276:0:(lib-md.c:131:lnet_md_build()) slab-alloced 'lmd': 136 at 0000000073d2f1f4. 00000100:00000040:3.0:1773946220.066707:0:191278:0:(niobuf.c:634:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9caa6b722800: using a timeout of 3313 usecs (4 jiffies) 00000400:00000010:2.0:1773946220.066709:0:191277:0:(lib-me.c:70:LNetMEAttach()) slab-alloced 'me': 88 at 000000005b20e160. 00000100:00000200:0.0:1773946220.066710:0:191276:0:(niobuf.c:94:ptl_send_buf()) Sending 224 bytes to portal 12, xid 1860117379162880, offset 0 00000100:00000001:3.0:1773946220.066713:0:191278:0:(niobuf.c:662:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 0 is 4299634692 00000100:00000001:3.0:1773946220.066719:0:191278:0:(niobuf.c:691:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9caa6b722800: PM QoS request active for cpu 0, keep current deadline 4299640375 00000400:00000010:2.0:1773946220.066719:0:191277:0:(lib-md.c:131:lnet_md_build()) slab-alloced 'lmd': 136 at 00000000c88a1048. 00000400:00000200:0.0:1773946220.066726:0:191276:0:(lib-move.c:5316:LNetPut()) LNetPut -> 12345-192.168.204.155@tcp 00000100:00000001:3.0:1773946220.066727:0:191278:0:(niobuf.c:662:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 1 is 4299634692 00000100:00000001:3.0:1773946220.066732:0:191278:0:(niobuf.c:691:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9caa6b722800: PM QoS request active for cpu 1, keep current deadline 4299640375 00000800:00000200:1.0:1773946220.066734:0:191269:0:(socklnd_cb.c:539:ksocknal_process_transmit()) send(0) 0 00000100:00000001:3.0:1773946220.066735:0:191278:0:(niobuf.c:662:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 2 is 4299634692 00000100:00000200:2.0:1773946220.066736:0:191277:0:(niobuf.c:1096:ptl_send_rpc()) Setup reply buffer: 512 bytes, xid 1860117379163008, portal 4 00000400:00000200:0.0:1773946220.066736:0:191276:0:(lib-move.c:2973:lnet_handle_send_case_locked()) Source ANY to MR: 192.168.204.155@tcp local destination 00000800:00000010:1.0:1773946220.066740:0:191269:0:(socklnd_cb.c:98:ksocknal_free_tx()) kfreed 'tx': 248 at 00000000828a44c4. 00000100:00000001:3.0:1773946220.066744:0:191278:0:(niobuf.c:691:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9caa6b722800: PM QoS request active for cpu 2, keep current deadline 4299640375 00000100:00000001:2.0:1773946220.066744:0:191277:0:(client.c:3244:ptlrpc_request_addref()) Process entered 00000400:00000200:0.0:1773946220.066746:0:191276:0:(lib-move.c:2603:lnet_find_best_ni_on_local_net()) selected best_lpn tcp 00000100:00000001:2.0:1773946220.066748:0:191277:0:(client.c:3246:ptlrpc_request_addref()) Process leaving (rc=18446634853835490176 : -109219874061440 : ffff9caa44344380) 00000400:00000200:1.0:1773946220.066748:0:191269:0:(lib-msg.c:1048:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000001:3.0:1773946220.066751:0:191278:0:(niobuf.c:662:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 3 is 4299634692 00000400:00000200:0.0:1773946220.066754:0:191276:0:(lib-move.c:1789:lnet_get_best_ni()) selected best_ni 192.168.204.55@tcp 00000400:00000200:0.0:1773946220.066761:0:191276:0:(lib-move.c:1343:lnet_select_peer_ni()) 192.168.204.55@tcp lpni_is_preferred = 0 00000100:00000040:2.0:1773946220.066763:0:191277:0:(niobuf.c:1116:ptl_send_rpc()) @@@ send flags=200 req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000400:00000200:1.0:1773946220.066763:0:191269:0:(lib-msg.c:824:lnet_health_check()) health check: 192.168.204.55@tcp->192.168.204.155@tcp: PUT: OK 00000100:00000001:3.0:1773946220.066766:0:191278:0:(niobuf.c:691:kick_cpu_latency()) lustre-MDT0000-mdc-ffff9caa6b722800: PM QoS request active for cpu 3, keep current deadline 4299640375 00000100:00000001:3.0:1773946220.066798:0:191278:0:(niobuf.c:1134:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1773946220.066802:0:191278:0:(client.c:1925:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:0.0:1773946220.066803:0:191276:0:(lib-move.c:1412:lnet_select_peer_ni()) sd_best_lpni = 192.168.204.155@tcp 00000100:00000001:3.0:1773946220.066809:0:191278:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:0.0:1773946220.066813:0:191276:0:(lib-move.c:1909:lnet_handle_send()) 192.168.204.55@tcp NI seq info: [74:74:254:4294967295] 192.168.204.155@tcp LPNI seq info [73:73:6:4294967295] 00000100:00000001:3.0:1773946220.066816:0:191278:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.066816:0:191277:0:(niobuf.c:69:ptl_send_buf()) Process entered 00000100:00000040:2.0:1773946220.066824:0:191277:0:(niobuf.c:72:ptl_send_buf()) peer_id 12345-192.168.204.155@tcp 00000400:00000010:2.0:1773946220.066829:0:191277:0:(lib-md.c:131:lnet_md_build()) slab-alloced 'lmd': 136 at 0000000048642243. 00000400:00000200:0.0:1773946220.066831:0:191276:0:(lib-move.c:2024:lnet_handle_send()) TRACE: 192.168.204.55@tcp(192.168.204.55@tcp:) -> 192.168.204.155@tcp(192.168.204.155@tcp:192.168.204.155@tcp) : PUT try# 0 00000100:00000200:2.0:1773946220.066836:0:191277:0:(niobuf.c:94:ptl_send_buf()) Sending 224 bytes to portal 28, xid 1860117379163008, offset 0 00000400:00000200:1.0:1773946220.066840:0:191269:0:(lib-md.c:46:lnet_md_unlink()) Unlinking md 00000000d733aa62 00000800:00000200:0.0:1773946220.066841:0:191276:0:(socklnd_cb.c:1001:ksocknal_send()) sending 224 bytes in 1 frags to 12345-192.168.204.155@tcp 00000400:00000010:1.0:1773946220.066844:0:191269:0:(lib-lnet.h:312:lnet_md_free()) slab-freed 'md': 136 at 00000000d733aa62. 00000800:00000010:0.0:1773946220.066849:0:191276:0:(socklnd_cb.c:41:ksocknal_alloc_tx()) alloc '(tx)': 248 at 000000009710ae9d. 00000100:00000001:1.0:1773946220.066851:0:191269:0:(events.c:34:request_out_callback()) Process entered 00000400:00000200:2.0:1773946220.066856:0:191277:0:(lib-move.c:5316:LNetPut()) LNetPut -> 12345-192.168.204.155@tcp 00000100:00000200:1.0:1773946220.066859:0:191269:0:(events.c:39:request_out_callback()) @@@ type 5, status 0 req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000800:00000200:0.0:1773946220.066860:0:191276:0:(socklnd.c:214:ksocknal_find_peer_locked()) got peer_ni [00000000ed7f6003] -> 12345-192.168.204.155@tcp (5) 00000400:00000200:2.0:1773946220.066870:0:191277:0:(lib-move.c:2973:lnet_handle_send_case_locked()) Source ANY to MR: 192.168.204.155@tcp local destination 00000800:00000200:0.0:1773946220.066875:0:191276:0:(socklnd_cb.c:748:ksocknal_queue_tx_locked()) Sending to 12345-192.168.204.155@tcp ip 192.168.204.155:988 00000100:00000001:1.0:1773946220.066877:0:191269:0:(client.c:2842:__ptlrpc_req_put()) Process entered 00000400:00000200:2.0:1773946220.066879:0:191277:0:(lib-move.c:2603:lnet_find_best_ni_on_local_net()) selected best_lpn tcp 00000800:00000200:0.0:1773946220.066882:0:191276:0:(socklnd_cb.c:767:ksocknal_queue_tx_locked()) Packet 000000009710ae9d type 1, nob 320 niov 1 nkiov 1 00000100:00000040:1.0:1773946220.066883:0:191269:0:(client.c:2850:__ptlrpc_req_put()) @@@ refcount now 1 req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000400:00000200:2.0:1773946220.066887:0:191277:0:(lib-move.c:1789:lnet_get_best_ni()) selected best_ni 192.168.204.55@tcp 00000100:00000001:0.0:1773946220.066891:0:191276:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:2.0:1773946220.066894:0:191277:0:(lib-move.c:1343:lnet_select_peer_ni()) 192.168.204.55@tcp lpni_is_preferred = 0 00000400:00000200:2.0:1773946220.066900:0:191277:0:(lib-move.c:1412:lnet_select_peer_ni()) sd_best_lpni = 192.168.204.155@tcp 00000100:00000040:0.0:1773946220.066903:0:191276:0:(niobuf.c:634:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9caa6b722800: using a timeout of 4307 usecs (5 jiffies) 00000400:00000200:2.0:1773946220.066910:0:191277:0:(lib-move.c:1909:lnet_handle_send()) 192.168.204.55@tcp NI seq info: [75:75:253:4294967295] 192.168.204.155@tcp LPNI seq info [74:74:5:4294967295] 00000100:00000001:0.0:1773946220.066911:0:191276:0:(niobuf.c:662:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 0 is 4299634694 00000100:00000001:1.0:1773946220.066914:0:191269:0:(client.c:2883:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1773946220.066918:0:191276:0:(niobuf.c:691:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9caa6b722800: PM QoS request active for cpu 0, keep current deadline 4299640375 00000100:00000001:1.0:1773946220.066921:0:191269:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:0.0:1773946220.066923:0:191276:0:(niobuf.c:662:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 1 is 4299634694 00000100:00000001:0.0:1773946220.066929:0:191276:0:(niobuf.c:691:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9caa6b722800: PM QoS request active for cpu 1, keep current deadline 4299640375 00000400:00000200:2.0:1773946220.066933:0:191277:0:(lib-move.c:2024:lnet_handle_send()) TRACE: 192.168.204.55@tcp(192.168.204.55@tcp:) -> 192.168.204.155@tcp(192.168.204.155@tcp:192.168.204.155@tcp) : PUT try# 0 00000100:00000001:0.0:1773946220.066934:0:191276:0:(niobuf.c:662:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 2 is 4299634694 00000100:00000001:0.0:1773946220.066940:0:191276:0:(niobuf.c:691:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9caa6b722800: PM QoS request active for cpu 2, keep current deadline 4299640375 00000800:00000200:2.0:1773946220.066946:0:191277:0:(socklnd_cb.c:1001:ksocknal_send()) sending 224 bytes in 1 frags to 12345-192.168.204.155@tcp 00000100:00000001:0.0:1773946220.066946:0:191276:0:(niobuf.c:662:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 3 is 4299634694 00000100:00000001:0.0:1773946220.066952:0:191276:0:(niobuf.c:691:kick_cpu_latency()) lustre-MDT0001-mdc-ffff9caa6b722800: PM QoS request active for cpu 3, keep current deadline 4299640375 00000800:00000200:1.0:1773946220.066954:0:191269:0:(socklnd_cb.c:539:ksocknal_process_transmit()) send(0) 0 00000800:00000010:2.0:1773946220.066955:0:191277:0:(socklnd_cb.c:41:ksocknal_alloc_tx()) alloc '(tx)': 248 at 00000000a1a54d4d. 00000100:00000001:0.0:1773946220.066956:0:191276:0:(niobuf.c:1134:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000800:00000010:1.0:1773946220.066958:0:191269:0:(socklnd_cb.c:98:ksocknal_free_tx()) kfreed 'tx': 248 at 0000000083731e81. 00000100:00000001:0.0:1773946220.066963:0:191276:0:(client.c:1925:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:1.0:1773946220.066965:0:191269:0:(lib-msg.c:1048:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000800:00000200:2.0:1773946220.066967:0:191277:0:(socklnd.c:214:ksocknal_find_peer_locked()) got peer_ni [00000000ed7f6003] -> 12345-192.168.204.155@tcp (5) 00000100:00000001:0.0:1773946220.066969:0:191276:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:1.0:1773946220.066976:0:191269:0:(lib-msg.c:824:lnet_health_check()) health check: 192.168.204.55@tcp->192.168.204.155@tcp: PUT: OK 00000100:00000001:0.0:1773946220.066976:0:191276:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000800:00000200:2.0:1773946220.066981:0:191277:0:(socklnd_cb.c:748:ksocknal_queue_tx_locked()) Sending to 12345-192.168.204.155@tcp ip 192.168.204.155:988 00000400:00000200:1.0:1773946220.066985:0:191269:0:(lib-md.c:46:lnet_md_unlink()) Unlinking md 000000002fa680cc 00000100:00000001:0.0:1773946220.066985:0:191276:0:(client.c:2614:ptlrpc_set_next_timeout()) Process entered 00000800:00000200:2.0:1773946220.066986:0:191277:0:(socklnd_cb.c:767:ksocknal_queue_tx_locked()) Packet 00000000a1a54d4d type 1, nob 320 niov 1 nkiov 1 00000400:00000010:1.0:1773946220.066989:0:191269:0:(lib-lnet.h:312:lnet_md_free()) slab-freed 'md': 136 at 000000002fa680cc. 00000100:00000001:2.0:1773946220.066993:0:191277:0:(niobuf.c:111:ptl_send_buf()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1773946220.066993:0:191276:0:(client.c:2642:ptlrpc_set_next_timeout()) Process leaving (rc=11 : 11 : b) 00000100:00000001:1.0:1773946220.066995:0:191269:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:0.0:1773946220.066998:0:191276:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1773946220.067000:0:191276:0:(client.c:1974:ptlrpc_check_set()) Process entered 00000100:00000040:2.0:1773946220.067001:0:191277:0:(niobuf.c:634:kick_cpu_latency()) lustre-OST0000-osc-ffff9caa6b722800: using a timeout of 2982 usecs (3 jiffies) 00000100:00000200:1.0:1773946220.067004:0:191269:0:(events.c:39:request_out_callback()) @@@ type 5, status 0 req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1773946220.067004:0:191276:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.067007:0:191277:0:(niobuf.c:662:kick_cpu_latency()) lustre-OST0000-osc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 0 is 4299634692 00000100:00000001:0.0:1773946220.067010:0:191276:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.067015:0:191277:0:(niobuf.c:691:kick_cpu_latency()) lustre-OST0000-osc-ffff9caa6b722800: PM QoS request active for cpu 0, keep current deadline 4299640375 00000100:00000001:2.0:1773946220.067021:0:191277:0:(niobuf.c:662:kick_cpu_latency()) lustre-OST0000-osc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 1 is 4299634692 00000100:00000001:1.0:1773946220.067025:0:191269:0:(client.c:2842:__ptlrpc_req_put()) Process entered 00000100:00000040:1.0:1773946220.067027:0:191269:0:(client.c:2850:__ptlrpc_req_put()) @@@ refcount now 1 req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1773946220.067033:0:191277:0:(niobuf.c:691:kick_cpu_latency()) lustre-OST0000-osc-ffff9caa6b722800: PM QoS request active for cpu 1, keep current deadline 4299640375 00000100:00000001:2.0:1773946220.067039:0:191277:0:(niobuf.c:662:kick_cpu_latency()) lustre-OST0000-osc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 2 is 4299634692 00000100:00000001:1.0:1773946220.067044:0:191269:0:(client.c:2883:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.067045:0:191277:0:(niobuf.c:691:kick_cpu_latency()) lustre-OST0000-osc-ffff9caa6b722800: PM QoS request active for cpu 2, keep current deadline 4299640375 00000100:00000001:1.0:1773946220.067048:0:191269:0:(events.c:68:request_out_callback()) Process leaving 00000100:00000001:2.0:1773946220.067051:0:191277:0:(niobuf.c:662:kick_cpu_latency()) lustre-OST0000-osc-ffff9caa6b722800: PM QoS new deadline estimation for cpu 3 is 4299634692 00000100:00000001:2.0:1773946220.067056:0:191277:0:(niobuf.c:691:kick_cpu_latency()) lustre-OST0000-osc-ffff9caa6b722800: PM QoS request active for cpu 3, keep current deadline 4299640375 00000100:00000001:2.0:1773946220.067058:0:191277:0:(niobuf.c:1134:ptl_send_rpc()) Process leaving via out (rc=0 : 0 : 0x0) 00000800:00000200:1.0:1773946220.067060:0:191269:0:(socklnd_cb.c:539:ksocknal_process_transmit()) send(0) 0 00000100:00000001:2.0:1773946220.067062:0:191277:0:(client.c:1925:ptlrpc_send_new_req()) Process leaving (rc=0 : 0 : 0) 00000800:00000010:1.0:1773946220.067063:0:191269:0:(socklnd_cb.c:98:ksocknal_free_tx()) kfreed 'tx': 248 at 000000009710ae9d. 00000100:00000001:2.0:1773946220.067069:0:191277:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000400:00000200:1.0:1773946220.067069:0:191269:0:(lib-msg.c:1048:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:1.0:1773946220.067073:0:191269:0:(lib-msg.c:824:lnet_health_check()) health check: 192.168.204.55@tcp->192.168.204.155@tcp: PUT: OK 00000100:00000001:2.0:1773946220.067076:0:191277:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000400:00000200:1.0:1773946220.067077:0:191269:0:(lib-md.c:46:lnet_md_unlink()) Unlinking md 0000000073d2f1f4 00000400:00000010:1.0:1773946220.067081:0:191269:0:(lib-lnet.h:312:lnet_md_free()) slab-freed 'md': 136 at 0000000073d2f1f4. 00000100:00000001:2.0:1773946220.067084:0:191277:0:(client.c:2614:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:1.0:1773946220.067086:0:191269:0:(events.c:34:request_out_callback()) Process entered 00000100:00000001:2.0:1773946220.067088:0:191277:0:(client.c:2642:ptlrpc_set_next_timeout()) Process leaving (rc=11 : 11 : b) 00000100:00000200:1.0:1773946220.067090:0:191269:0:(events.c:39:request_out_callback()) @@@ type 5, status 0 req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1773946220.067094:0:191277:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1773946220.067098:0:191277:0:(client.c:1974:ptlrpc_check_set()) Process entered 00000100:00000001:1.0:1773946220.067100:0:191269:0:(client.c:2842:__ptlrpc_req_put()) Process entered 00000100:00000001:2.0:1773946220.067102:0:191277:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:1.0:1773946220.067103:0:191269:0:(client.c:2850:__ptlrpc_req_put()) @@@ refcount now 1 req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1773946220.067107:0:191277:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1773946220.067110:0:191269:0:(client.c:2883:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1773946220.067113:0:191269:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000200:1.0:1773946220.067148:0:191269:0:(socklnd_cb.c:539:ksocknal_process_transmit()) send(0) 0 00000800:00000010:1.0:1773946220.067154:0:191269:0:(socklnd_cb.c:98:ksocknal_free_tx()) kfreed 'tx': 248 at 00000000a1a54d4d. 00000400:00000200:1.0:1773946220.067157:0:191269:0:(lib-msg.c:1048:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:1.0:1773946220.067159:0:191269:0:(lib-msg.c:824:lnet_health_check()) health check: 192.168.204.55@tcp->192.168.204.155@tcp: PUT: OK 00000400:00000200:1.0:1773946220.067161:0:191269:0:(lib-md.c:46:lnet_md_unlink()) Unlinking md 0000000048642243 00000400:00000010:1.0:1773946220.067163:0:191269:0:(lib-lnet.h:312:lnet_md_free()) slab-freed 'md': 136 at 0000000048642243. 00000100:00000001:1.0:1773946220.067166:0:191269:0:(events.c:34:request_out_callback()) Process entered 00000100:00000200:1.0:1773946220.067170:0:191269:0:(events.c:39:request_out_callback()) @@@ type 5, status 0 req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:Nr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1773946220.067177:0:191269:0:(client.c:2842:__ptlrpc_req_put()) Process entered 00000100:00000040:1.0:1773946220.067193:0:191269:0:(client.c:2850:__ptlrpc_req_put()) @@@ refcount now 1 req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 2 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1773946220.067200:0:191269:0:(client.c:2883:__ptlrpc_req_put()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:1.0:1773946220.067201:0:191269:0:(events.c:68:request_out_callback()) Process leaving 00000800:00000001:1.0:1773946220.069385:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000800:00000001:1.0:1773946220.069396:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000800:00000001:1.0:1773946220.069399:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000800:00000001:1.0:1773946220.069401:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000400:00000200:1.0:1773946220.069409:0:191271:0:(lib-move.c:4806:lnet_parse()) TRACE: 192.168.204.55@tcp(192.168.204.55@tcp) <- 192.168.204.155@tcp : PUT - for me 00000400:00000200:1.0:1773946220.069420:0:191271:0:(lib-ptl.c:584:lnet_ptl_match_md()) Request from 12345-192.168.204.155@tcp of length 224 into portal 25 MB=0x69bc453a02200 00000400:00000200:1.0:1773946220.069427:0:191271:0:(lib-ptl.c:210:lnet_try_match_md()) Incoming put index 19 from 12345-192.168.204.155@tcp of length 224/224 into md 0x431 [1] + 224 00000800:00000001:1.0:1773946220.069434:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000800:00000001:1.0:1773946220.069438:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000400:00000200:1.0:1773946220.069441:0:191271:0:(lib-msg.c:1048:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:1.0:1773946220.069444:0:191271:0:(lib-msg.c:824:lnet_health_check()) health check: 192.168.204.55@tcp->192.168.204.155@tcp: PUT: OK 00000100:00000001:1.0:1773946220.069448:0:191271:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000200:1.0:1773946220.069452:0:191271:0:(events.c:80:reply_in_callback()) @@@ type 2, status 0 req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1773946220.069462:0:191271:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:1.0:1773946220.069482:0:191271:0:(events.c:163:reply_in_callback()) Process leaving 00000800:00000001:1.0:1773946220.069489:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000800:00000001:1.0:1773946220.069494:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000800:00000001:1.0:1773946220.069502:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000800:00000001:1.0:1773946220.069505:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000400:00000200:1.0:1773946220.069511:0:191271:0:(lib-move.c:4806:lnet_parse()) TRACE: 192.168.204.55@tcp(192.168.204.55@tcp) <- 192.168.204.155@tcp : PUT - for me 00000400:00000200:1.0:1773946220.069519:0:191271:0:(lib-ptl.c:584:lnet_ptl_match_md()) Request from 12345-192.168.204.155@tcp of length 224 into portal 4 MB=0x69bc453a02380 00000100:00000001:2.0:1773946220.069525:0:191279:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000400:00000200:1.0:1773946220.069525:0:191271:0:(lib-ptl.c:210:lnet_try_match_md()) Incoming put index 4 from 12345-192.168.204.155@tcp of length 224/224 into md 0x449 [1] + 224 00000100:00000001:2.0:1773946220.069531:0:191279:0:(client.c:1974:ptlrpc_check_set()) Process entered 00000800:00000001:1.0:1773946220.069533:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000400:00000001:2.0:1773946220.069538:0:191279:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000800:00000001:1.0:1773946220.069539:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000400:00000001:2.0:1773946220.069542:0:191279:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000400:00000200:1.0:1773946220.069543:0:191271:0:(lib-msg.c:1048:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000010:2.0:1773946220.069548:0:191279:0:(lib-me.c:113:lnet_me_unlink()) slab-freed 'me': 88 at 00000000f7b94d97. 00000400:00000200:1.0:1773946220.069550:0:191271:0:(lib-msg.c:824:lnet_health_check()) health check: 192.168.204.55@tcp->192.168.204.155@tcp: PUT: OK 00000400:00000200:2.0:1773946220.069555:0:191279:0:(lib-md.c:46:lnet_md_unlink()) Unlinking md 0000000048355ba7 00000400:00000010:2.0:1773946220.069558:0:191279:0:(lib-lnet.h:312:lnet_md_free()) slab-freed 'md': 136 at 0000000048355ba7. 00000100:00000001:2.0:1773946220.069563:0:191279:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:1.0:1773946220.069564:0:191271:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000200:2.0:1773946220.069568:0:191279:0:(events.c:80:reply_in_callback()) @@@ type 6, status 0 req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000200:1.0:1773946220.069572:0:191271:0:(events.c:80:reply_in_callback()) @@@ type 2, status 0 req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000200:2.0:1773946220.069598:0:191279:0:(events.c:102:reply_in_callback()) @@@ unlink req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000040:1.0:1773946220.069600:0:191271:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1773946220.069617:0:191279:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1773946220.069619:0:191279:0:(client.c:2957:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1773946220.069621:0:191279:0:(client.c:1578:after_reply()) Process entered 02000000:00000001:2.0:1773946220.069622:0:191279:0:(sec.c:1113:do_cli_unwrap_reply()) Process entered 00000100:00000001:1.0:1773946220.069623:0:191271:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1773946220.069624:0:191279:0:(pack_generic.c:602:__lustre_unpack_msg()) Process entered 00000100:00000001:2.0:1773946220.069627:0:191279:0:(pack_generic.c:620:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 00000800:00000001:1.0:1773946220.069630:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 02000000:00000001:2.0:1773946220.069636:0:191279:0:(sec.c:1169:do_cli_unwrap_reply()) Process leaving (rc=0 : 0 : 0) 00000800:00000001:1.0:1773946220.069636:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000800:00000001:1.0:1773946220.069643:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000100:00001000:2.0:1773946220.069645:0:191279:0:(import.c:2136:obd_at_measure()) add 5 to 0000000071b16db0 time=6 v=5 (5 0 0 0) 00000800:00000001:1.0:1773946220.069648:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000100:00001000:2.0:1773946220.069650:0:191279:0:(import.c:2136:obd_at_measure()) add 1 to 000000005a06156e time=6 v=5 (1 0 0 0) 00000100:00000001:2.0:1773946220.069654:0:191279:0:(client.c:1480:ptlrpc_check_status()) Process entered 00000100:00000001:2.0:1773946220.069657:0:191279:0:(client.c:1518:ptlrpc_check_status()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1773946220.069659:0:191277:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000400:00000200:1.0:1773946220.069659:0:191271:0:(lib-move.c:4806:lnet_parse()) TRACE: 192.168.204.55@tcp(192.168.204.55@tcp) <- 192.168.204.155@tcp : PUT - for me 00010000:00000001:2.0:1773946220.069660:0:191279:0:(ldlm_request.c:1516:ldlm_cli_update_pool()) Process entered 00000100:00000001:3.0:1773946220.069664:0:191277:0:(client.c:1974:ptlrpc_check_set()) Process entered 00010000:00000001:2.0:1773946220.069665:0:191279:0:(ldlm_request.c:1520:ldlm_cli_update_pool()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.069669:0:191279:0:(client.c:1781:after_reply()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:3.0:1773946220.069670:0:191277:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000400:00000001:3.0:1773946220.069674:0:191277:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000400:00000200:1.0:1773946220.069674:0:191271:0:(lib-ptl.c:584:lnet_ptl_match_md()) Request from 12345-192.168.204.155@tcp of length 224 into portal 10 MB=0x69bc453a02280 00000100:00000040:2.0:1773946220.069675:0:191279:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 00000400:00000010:3.0:1773946220.069678:0:191277:0:(lib-me.c:113:lnet_me_unlink()) slab-freed 'me': 88 at 000000005b20e160. 00000400:00000200:3.0:1773946220.069684:0:191277:0:(lib-md.c:46:lnet_md_unlink()) Unlinking md 00000000c88a1048 00000400:00000010:3.0:1773946220.069687:0:191277:0:(lib-lnet.h:312:lnet_md_free()) slab-freed 'md': 136 at 00000000c88a1048. 00000100:00000001:2.0:1773946220.069689:0:191279:0:(client.c:2329:ptlrpc_check_set()) Process leaving via interpret (rc=0 : 0 : 0x0) 00000100:00000001:3.0:1773946220.069691:0:191277:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:2.0:1773946220.069692:0:191279:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1773946220.069696:0:191279:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000400:00000200:1.0:1773946220.069697:0:191271:0:(lib-ptl.c:210:lnet_try_match_md()) Incoming put index a from 12345-192.168.204.155@tcp of length 224/224 into md 0x439 [1] + 224 00000100:00000200:3.0:1773946220.069698:0:191277:0:(events.c:80:reply_in_callback()) @@@ type 6, status 0 req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000040:2.0:1773946220.069705:0:191279:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 00000800:00000001:1.0:1773946220.069710:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000800:00000001:1.0:1773946220.069716:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000100:00000200:3.0:1773946220.069721:0:191277:0:(events.c:102:reply_in_callback()) @@@ unlink req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000400:00000200:1.0:1773946220.069722:0:191271:0:(lib-msg.c:1048:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000400:00000200:1.0:1773946220.069733:0:191271:0:(lib-msg.c:824:lnet_health_check()) health check: 192.168.204.55@tcp->192.168.204.155@tcp: PUT: OK 00000100:00100000:2.0:1773946220.069739:0:191279:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@0000000024352b9e pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_03:8e4604cb-9f63-473b-92ae-61bdc80df14a:191279:1860117379162624:192.168.204.155@tcp:400: 00000100:00000001:1.0:1773946220.069742:0:191271:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000200:1.0:1773946220.069750:0:191271:0:(events.c:80:reply_in_callback()) @@@ type 2, status 0 req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1773946220.069752:0:191277:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:2.0:1773946220.069754:0:191279:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1773946220.069757:0:191277:0:(client.c:2957:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1773946220.069760:0:191279:0:(client.c:2842:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1773946220.069764:0:191277:0:(client.c:1578:after_reply()) Process entered 00000100:00000040:2.0:1773946220.069768:0:191279:0:(client.c:2850:__ptlrpc_req_put()) @@@ refcount now 0 req@0000000024352b9e x1860117379162624/t0(0) o400->MGC192.168.204.155@tcp@192.168.204.155@tcp:26/25 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1773946220.069769:0:191277:0:(sec.c:1113:do_cli_unwrap_reply()) Process entered 00000100:00000001:3.0:1773946220.069798:0:191277:0:(pack_generic.c:602:__lustre_unpack_msg()) Process entered 00000100:00000040:1.0:1773946220.069800:0:191271:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1773946220.069801:0:191277:0:(pack_generic.c:620:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:3.0:1773946220.069807:0:191277:0:(sec.c:1169:do_cli_unwrap_reply()) Process leaving (rc=0 : 0 : 0) 00000020:00000010:3.0:1773946220.069818:0:191277:0:(lprocfs_status.c:1408:lprocfs_stats_alloc_one()) alloc '(stats->ls_percpu[cpuid])': 4608 at 00000000031cfd3b. 00000100:00000001:2.0:1773946220.069822:0:191279:0:(client.c:2768:__ptlrpc_free_req()) Process entered 00000100:00001000:3.0:1773946220.069827:0:191277:0:(import.c:2136:obd_at_measure()) add 5 to 00000000c4acc2f2 time=6 v=5 (5 0 0 0) 02000000:00000001:2.0:1773946220.069827:0:191279:0:(sec.c:1877:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000010:2.0:1773946220.069830:0:191279:0:(sec_null.c:203:null_free_repbuf()) kfreed 'req->rq_repbuf': 512 at 0000000037440cf2. 00000100:00001000:3.0:1773946220.069833:0:191277:0:(import.c:2136:obd_at_measure()) add 1 to 00000000ed75db30 time=6 v=5 (1 0 0 0) 02000000:00000001:2.0:1773946220.069839:0:191279:0:(sec.c:1891:sptlrpc_cli_free_repbuf()) Process leaving 00000100:00000001:3.0:1773946220.069843:0:191277:0:(client.c:1480:ptlrpc_check_status()) Process entered 00000020:00000001:2.0:1773946220.069843:0:191279:0:(genops.c:1236:class_import_put()) Process entered 00000100:00000001:3.0:1773946220.069847:0:191277:0:(client.c:1518:ptlrpc_check_status()) Process leaving (rc=0 : 0 : 0) 00000020:00000040:2.0:1773946220.069847:0:191279:0:(genops.c:1242:class_import_put()) import 00000000bab840cb refcount=3 obd=MGC192.168.204.155@tcp 00010000:00000001:3.0:1773946220.069853:0:191277:0:(ldlm_request.c:1516:ldlm_cli_update_pool()) Process entered 00000020:00000001:2.0:1773946220.069854:0:191279:0:(genops.c:1249:class_import_put()) Process leaving 02000000:00000010:2.0:1773946220.069858:0:191279:0:(sec_null.c:173:null_free_reqbuf()) kvfree 'req->rq_reqbuf': 256 at 000000009502c4c6. 02000000:00000001:2.0:1773946220.069861:0:191279:0:(sec.c:480:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:1.0:1773946220.069861:0:191271:0:(events.c:163:reply_in_callback()) Process leaving 00010000:00080000:3.0:1773946220.069862:0:191277:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 02000000:00000001:2.0:1773946220.069865:0:191279:0:(sec.c:497:sptlrpc_req_put_ctx()) Process leaving 00000100:00000010:2.0:1773946220.069867:0:191279:0:(client.c:598:ptlrpc_request_cache_free()) slab-freed '(req)': 888 at 0000000024352b9e. 00000100:00000001:2.0:1773946220.069870:0:191279:0:(client.c:2824:__ptlrpc_free_req()) Process leaving 00000800:00000001:1.0:1773946220.069870:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000100:00000001:2.0:1773946220.069874:0:191279:0:(client.c:2883:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000800:00000001:1.0:1773946220.069877:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1773946220.069880:0:191278:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000800:00000001:1.0:1773946220.069884:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000100:00000001:2.0:1773946220.069885:0:191279:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1773946220.069886:0:191278:0:(client.c:1974:ptlrpc_check_set()) Process entered 00000800:00000001:1.0:1773946220.069890:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1773946220.069894:0:191279:0:(client.c:2614:ptlrpc_set_next_timeout()) Process entered 00000400:00000001:0.0:1773946220.069894:0:191278:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00010000:00000001:3.0:1773946220.069895:0:191277:0:(ldlm_request.c:1535:ldlm_cli_update_pool()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.069898:0:191279:0:(client.c:2642:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000400:00000001:0.0:1773946220.069899:0:191278:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000400:00000200:1.0:1773946220.069902:0:191271:0:(lib-move.c:4806:lnet_parse()) TRACE: 192.168.204.55@tcp(192.168.204.55@tcp) <- 192.168.204.155@tcp : PUT - for me 00000100:00000001:2.0:1773946220.069904:0:191279:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000400:00000010:0.0:1773946220.069905:0:191278:0:(lib-me.c:113:lnet_me_unlink()) slab-freed 'me': 88 at 00000000458d2b58. 00000100:00000001:2.0:1773946220.069910:0:191279:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1773946220.069911:0:191277:0:(client.c:3065:ptlrpc_free_committed()) Process entered 00000400:00000200:0.0:1773946220.069912:0:191278:0:(lib-md.c:46:lnet_md_unlink()) Unlinking md 0000000073d78dc2 00000100:00000040:3.0:1773946220.069914:0:191277:0:(client.c:3077:ptlrpc_free_committed()) lustre-OST0000-osc-ffff9caa6b722800: skip recheck: last_committed 0 00000400:00000200:1.0:1773946220.069919:0:191271:0:(lib-ptl.c:584:lnet_ptl_match_md()) Request from 12345-192.168.204.155@tcp of length 224 into portal 10 MB=0x69bc453a02300 00000400:00000010:0.0:1773946220.069919:0:191278:0:(lib-lnet.h:312:lnet_md_free()) slab-freed 'md': 136 at 0000000073d78dc2. 00000100:00000001:3.0:1773946220.069920:0:191277:0:(client.c:3078:ptlrpc_free_committed()) Process leaving 00000100:00000001:3.0:1773946220.069924:0:191277:0:(client.c:1781:after_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1773946220.069927:0:191278:0:(events.c:78:reply_in_callback()) Process entered 00000400:00000200:1.0:1773946220.069931:0:191271:0:(lib-ptl.c:210:lnet_try_match_md()) Incoming put index a from 12345-192.168.204.155@tcp of length 224/224 into md 0x441 [1] + 224 00000100:00000040:3.0:1773946220.069933:0:191277:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 00000100:00000200:0.0:1773946220.069936:0:191278:0:(events.c:80:reply_in_callback()) @@@ type 6, status 0 req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000800:00000001:1.0:1773946220.069943:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 00000800:00000001:1.0:1773946220.069952:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000400:00000200:1.0:1773946220.069958:0:191271:0:(lib-msg.c:1048:lnet_is_health_check()) health check = 1, status = 0, hstatus = 0 00000100:00000001:3.0:1773946220.069961:0:191277:0:(client.c:2329:ptlrpc_check_set()) Process leaving via interpret (rc=0 : 0 : 0x0) 00000400:00000200:1.0:1773946220.069968:0:191271:0:(lib-msg.c:824:lnet_health_check()) health check: 192.168.204.55@tcp->192.168.204.155@tcp: PUT: OK 00000100:00000001:3.0:1773946220.069969:0:191277:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000200:0.0:1773946220.069973:0:191278:0:(events.c:102:reply_in_callback()) @@@ unlink req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1773946220.069974:0:191277:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:1.0:1773946220.069974:0:191271:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000200:1.0:1773946220.069982:0:191271:0:(events.c:80:reply_in_callback()) @@@ type 2, status 0 req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000040:3.0:1773946220.069985:0:191277:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1773946220.070001:0:191278:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1773946220.070010:0:191278:0:(client.c:2957:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1773946220.070015:0:191278:0:(client.c:1578:after_reply()) Process entered 02000000:00000001:0.0:1773946220.070019:0:191278:0:(sec.c:1113:do_cli_unwrap_reply()) Process entered 00000100:00000040:1.0:1773946220.070021:0:191271:0:(events.c:149:reply_in_callback()) @@@ reply in flags=200 mlen=224 offset=224 replen=224 req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1773946220.070022:0:191277:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@00000000e7bddb4a pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:c8ed80ad-4461-452a-b4bd-ffd70870ac7a:191277:1860117379163008:192.168.204.155@tcp:400: 00000100:00000001:0.0:1773946220.070022:0:191278:0:(pack_generic.c:602:__lustre_unpack_msg()) Process entered 00000100:00000001:0.0:1773946220.070026:0:191278:0:(pack_generic.c:620:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1773946220.070031:0:191278:0:(sec.c:1169:do_cli_unwrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:3.0:1773946220.070034:0:191277:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00001000:0.0:1773946220.070036:0:191278:0:(import.c:2136:obd_at_measure()) add 5 to 000000005d3c167d time=6 v=5 (5 0 0 0) 00000100:00000001:3.0:1773946220.070040:0:191277:0:(client.c:2842:__ptlrpc_req_put()) Process entered 00000100:00001000:0.0:1773946220.070045:0:191278:0:(import.c:2136:obd_at_measure()) add 1 to 0000000026afc83b time=6 v=5 (1 0 0 0) 00000100:00000040:3.0:1773946220.070048:0:191277:0:(client.c:2850:__ptlrpc_req_put()) @@@ refcount now 0 req@00000000e7bddb4a x1860117379163008/t0(0) o400->lustre-OST0000-osc-ffff9caa6b722800@192.168.204.155@tcp:28/4 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1773946220.070050:0:191278:0:(client.c:1480:ptlrpc_check_status()) Process entered 00000100:00000001:0.0:1773946220.070053:0:191278:0:(client.c:1518:ptlrpc_check_status()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:0.0:1773946220.070057:0:191278:0:(ldlm_request.c:1516:ldlm_cli_update_pool()) Process entered 00000100:00000001:1.0:1773946220.070061:0:191271:0:(events.c:163:reply_in_callback()) Process leaving 00010000:00000001:0.0:1773946220.070061:0:191278:0:(ldlm_request.c:1546:ldlm_cli_update_pool()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1773946220.070066:0:191278:0:(client.c:3065:ptlrpc_free_committed()) Process entered 00000100:00000001:3.0:1773946220.070068:0:191277:0:(client.c:2768:__ptlrpc_free_req()) Process entered 00000100:00000040:0.0:1773946220.070069:0:191278:0:(client.c:3077:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff9caa6b722800: skip recheck: last_committed 0 00000800:00000001:1.0:1773946220.070070:0:191271:0:(socklnd_cb.c:345:ksocknal_receive()) Process entered 02000000:00000001:3.0:1773946220.070071:0:191277:0:(sec.c:1877:sptlrpc_cli_free_repbuf()) Process entered 00000100:00000001:0.0:1773946220.070072:0:191278:0:(client.c:3078:ptlrpc_free_committed()) Process leaving 02000000:00000010:3.0:1773946220.070077:0:191277:0:(sec_null.c:203:null_free_repbuf()) kfreed 'req->rq_repbuf': 512 at 00000000c47662ca. 00000100:00000001:0.0:1773946220.070077:0:191278:0:(client.c:1781:after_reply()) Process leaving (rc=0 : 0 : 0) 00000800:00000001:1.0:1773946220.070078:0:191271:0:(socklnd_cb.c:383:ksocknal_receive()) Process leaving (rc=1 : 1 : 1) 00000100:00000040:0.0:1773946220.070083:0:191278:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1773946220.070085:0:191277:0:(sec.c:1891:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:3.0:1773946220.070089:0:191277:0:(genops.c:1236:class_import_put()) Process entered 00000020:00000040:3.0:1773946220.070092:0:191277:0:(genops.c:1242:class_import_put()) import 00000000dcd78867 refcount=2 obd=lustre-OST0000-osc-ffff9caa6b722800 00000100:00000001:0.0:1773946220.070093:0:191278:0:(client.c:2329:ptlrpc_check_set()) Process leaving via interpret (rc=0 : 0 : 0x0) 00000020:00000001:3.0:1773946220.070097:0:191277:0:(genops.c:1249:class_import_put()) Process leaving 00000100:00000001:0.0:1773946220.070098:0:191278:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 02000000:00000010:3.0:1773946220.070101:0:191277:0:(sec_null.c:173:null_free_reqbuf()) kvfree 'req->rq_reqbuf': 256 at 0000000080043654. 00000100:00000001:0.0:1773946220.070102:0:191278:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000040:0.0:1773946220.070107:0:191278:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 02000000:00000001:3.0:1773946220.070109:0:191277:0:(sec.c:480:sptlrpc_req_put_ctx()) Process entered 00000100:00000001:2.0:1773946220.070111:0:191276:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 02000000:00000001:3.0:1773946220.070113:0:191277:0:(sec.c:497:sptlrpc_req_put_ctx()) Process leaving 00000100:00000001:2.0:1773946220.070116:0:191276:0:(client.c:1974:ptlrpc_check_set()) Process entered 00000100:00000010:3.0:1773946220.070117:0:191277:0:(client.c:598:ptlrpc_request_cache_free()) slab-freed '(req)': 888 at 00000000e7bddb4a. 00000100:00100000:0.0:1773946220.070119:0:191278:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@000000003782171d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_02:c8ed80ad-4461-452a-b4bd-ffd70870ac7a:191278:1860117379162752:192.168.204.155@tcp:400: 00000400:00000001:2.0:1773946220.070122:0:191276:0:(lib-msg.c:22:lnet_build_unlink_event()) Process entered 00000100:00000001:3.0:1773946220.070125:0:191277:0:(client.c:2824:__ptlrpc_free_req()) Process leaving 00000400:00000001:2.0:1773946220.070126:0:191276:0:(lib-msg.c:31:lnet_build_unlink_event()) Process leaving 00000100:00000001:0.0:1773946220.070128:0:191278:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:3.0:1773946220.070129:0:191277:0:(client.c:2883:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000400:00000010:2.0:1773946220.070130:0:191276:0:(lib-me.c:113:lnet_me_unlink()) slab-freed 'me': 88 at 00000000f10cd56f. 00000100:00000001:0.0:1773946220.070133:0:191278:0:(client.c:2842:__ptlrpc_req_put()) Process entered 00000100:00000001:3.0:1773946220.070135:0:191277:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000400:00000200:2.0:1773946220.070139:0:191276:0:(lib-md.c:46:lnet_md_unlink()) Unlinking md 000000005c421093 00000100:00000040:0.0:1773946220.070139:0:191278:0:(client.c:2850:__ptlrpc_req_put()) @@@ refcount now 0 req@000000003782171d x1860117379162752/t0(0) o400->lustre-MDT0000-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1773946220.070142:0:191277:0:(client.c:2614:ptlrpc_set_next_timeout()) Process entered 00000400:00000010:2.0:1773946220.070143:0:191276:0:(lib-lnet.h:312:lnet_md_free()) slab-freed 'md': 136 at 000000005c421093. 00000100:00000001:3.0:1773946220.070146:0:191277:0:(client.c:2642:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.070152:0:191276:0:(events.c:78:reply_in_callback()) Process entered 00000100:00000001:3.0:1773946220.070153:0:191277:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:0.0:1773946220.070155:0:191278:0:(client.c:2768:__ptlrpc_free_req()) Process entered 02000000:00000001:0.0:1773946220.070157:0:191278:0:(sec.c:1877:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000010:0.0:1773946220.070159:0:191278:0:(sec_null.c:203:null_free_repbuf()) kfreed 'req->rq_repbuf': 512 at 000000004a58fcba. 00000100:00000200:2.0:1773946220.070160:0:191276:0:(events.c:80:reply_in_callback()) @@@ type 6, status 0 req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQ/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:3.0:1773946220.070164:0:191277:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:0.0:1773946220.070165:0:191278:0:(sec.c:1891:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:0.0:1773946220.070168:0:191278:0:(genops.c:1236:class_import_put()) Process entered 00000020:00000040:0.0:1773946220.070176:0:191278:0:(genops.c:1242:class_import_put()) import 000000000f7b6da0 refcount=3 obd=lustre-MDT0000-mdc-ffff9caa6b722800 00000020:00000001:0.0:1773946220.070194:0:191278:0:(genops.c:1249:class_import_put()) Process leaving 02000000:00000010:0.0:1773946220.070196:0:191278:0:(sec_null.c:173:null_free_reqbuf()) kvfree 'req->rq_reqbuf': 256 at 0000000032c738f6. 02000000:00000001:0.0:1773946220.070199:0:191278:0:(sec.c:480:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:0.0:1773946220.070200:0:191278:0:(sec.c:497:sptlrpc_req_put_ctx()) Process leaving 00000100:00000010:0.0:1773946220.070201:0:191278:0:(client.c:598:ptlrpc_request_cache_free()) slab-freed '(req)': 888 at 000000003782171d. 00000100:00000001:0.0:1773946220.070203:0:191278:0:(client.c:2824:__ptlrpc_free_req()) Process leaving 00000100:00000200:2.0:1773946220.070205:0:191276:0:(events.c:102:reply_in_callback()) @@@ unlink req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:0.0:1773946220.070205:0:191278:0:(client.c:2883:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1773946220.070209:0:191278:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:0.0:1773946220.070215:0:191278:0:(client.c:2614:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:0.0:1773946220.070217:0:191278:0:(client.c:2642:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:0.0:1773946220.070220:0:191278:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1773946220.070223:0:191276:0:(events.c:163:reply_in_callback()) Process leaving 00000100:00000001:0.0:1773946220.070224:0:191278:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.070228:0:191276:0:(client.c:2957:ptlrpc_unregister_reply()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1773946220.070232:0:191276:0:(client.c:1578:after_reply()) Process entered 02000000:00000001:2.0:1773946220.070234:0:191276:0:(sec.c:1113:do_cli_unwrap_reply()) Process entered 00000100:00000001:2.0:1773946220.070236:0:191276:0:(pack_generic.c:602:__lustre_unpack_msg()) Process entered 00000100:00000001:2.0:1773946220.070238:0:191276:0:(pack_generic.c:620:__lustre_unpack_msg()) Process leaving (rc=0 : 0 : 0) 02000000:00000001:2.0:1773946220.070240:0:191276:0:(sec.c:1169:do_cli_unwrap_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00001000:2.0:1773946220.070244:0:191276:0:(import.c:2136:obd_at_measure()) add 5 to 00000000fd3f96db time=6 v=5 (5 0 0 0) 00000100:00001000:2.0:1773946220.070247:0:191276:0:(import.c:2136:obd_at_measure()) add 1 to 00000000c37578e4 time=6 v=5 (1 0 0 0) 00000100:00000001:2.0:1773946220.070249:0:191276:0:(client.c:1480:ptlrpc_check_status()) Process entered 00000100:00000001:2.0:1773946220.070250:0:191276:0:(client.c:1518:ptlrpc_check_status()) Process leaving (rc=0 : 0 : 0) 00010000:00000001:2.0:1773946220.070252:0:191276:0:(ldlm_request.c:1516:ldlm_cli_update_pool()) Process entered 00010000:00080000:2.0:1773946220.070256:0:191276:0:(ldlm_request.c:1534:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 00010000:00000001:2.0:1773946220.070264:0:191276:0:(ldlm_request.c:1535:ldlm_cli_update_pool()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.070266:0:191276:0:(client.c:3065:ptlrpc_free_committed()) Process entered 00000100:00000040:2.0:1773946220.070267:0:191276:0:(client.c:3077:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff9caa6b722800: skip recheck: last_committed 0 00000100:00000001:2.0:1773946220.070269:0:191276:0:(client.c:3078:ptlrpc_free_committed()) Process leaving 00000100:00000001:2.0:1773946220.070277:0:191276:0:(client.c:1781:after_reply()) Process leaving (rc=0 : 0 : 0) 00000100:00000040:2.0:1773946220.070281:0:191276:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from Rpc to Interpret req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1773946220.070288:0:191276:0:(client.c:2329:ptlrpc_check_set()) Process leaving via interpret (rc=0 : 0 : 0x0) 00000100:00000001:2.0:1773946220.070290:0:191276:0:(niobuf.c:473:ptlrpc_unregister_bulk()) Process entered 00000100:00000001:2.0:1773946220.070292:0:191276:0:(niobuf.c:487:ptlrpc_unregister_bulk()) Process leaving (rc=1 : 1 : 1) 00000100:00000040:2.0:1773946220.070294:0:191276:0:(lustre_net.h:2431:ptlrpc_rqphase_move()) @@@ move request phase from Interpret to Complete req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1773946220.070302:0:191276:0:(client.c:2389:ptlrpc_check_set()) Completed RPC req@00000000d4f3130d pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:c8ed80ad-4461-452a-b4bd-ffd70870ac7a:191276:1860117379162880:192.168.204.155@tcp:400: 00000100:00000001:2.0:1773946220.070306:0:191276:0:(client.c:2440:ptlrpc_check_set()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1773946220.070308:0:191276:0:(client.c:2842:__ptlrpc_req_put()) Process entered 00000100:00000040:2.0:1773946220.070310:0:191276:0:(client.c:2850:__ptlrpc_req_put()) @@@ refcount now 0 req@00000000d4f3130d x1860117379162880/t0(0) o400->lustre-MDT0001-mdc-ffff9caa6b722800@192.168.204.155@tcp:12/10 lens 224/224 e 0 to 0 dl 1773946236 ref 1 fl Complete:RNQU/200/0 rc 0/0 job:'' uid:0 gid:0 projid:4294967295 00000100:00000001:2.0:1773946220.070316:0:191276:0:(client.c:2768:__ptlrpc_free_req()) Process entered 02000000:00000001:2.0:1773946220.070318:0:191276:0:(sec.c:1877:sptlrpc_cli_free_repbuf()) Process entered 02000000:00000010:2.0:1773946220.070320:0:191276:0:(sec_null.c:203:null_free_repbuf()) kfreed 'req->rq_repbuf': 512 at 0000000069a065bb. 02000000:00000001:2.0:1773946220.070323:0:191276:0:(sec.c:1891:sptlrpc_cli_free_repbuf()) Process leaving 00000020:00000001:2.0:1773946220.070324:0:191276:0:(genops.c:1236:class_import_put()) Process entered 00000020:00000040:2.0:1773946220.070326:0:191276:0:(genops.c:1242:class_import_put()) import 0000000063e09825 refcount=3 obd=lustre-MDT0001-mdc-ffff9caa6b722800 00000020:00000001:2.0:1773946220.070328:0:191276:0:(genops.c:1249:class_import_put()) Process leaving 02000000:00000010:2.0:1773946220.070329:0:191276:0:(sec_null.c:173:null_free_reqbuf()) kvfree 'req->rq_reqbuf': 256 at 00000000c4aa4116. 02000000:00000001:2.0:1773946220.070332:0:191276:0:(sec.c:480:sptlrpc_req_put_ctx()) Process entered 02000000:00000001:2.0:1773946220.070333:0:191276:0:(sec.c:497:sptlrpc_req_put_ctx()) Process leaving 00000100:00000010:2.0:1773946220.070335:0:191276:0:(client.c:598:ptlrpc_request_cache_free()) slab-freed '(req)': 888 at 00000000d4f3130d. 00000100:00000001:2.0:1773946220.070337:0:191276:0:(client.c:2824:__ptlrpc_free_req()) Process leaving 00000100:00000001:2.0:1773946220.070338:0:191276:0:(client.c:2883:__ptlrpc_req_put()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1773946220.070340:0:191276:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=1 : 1 : 1) 00000100:00000001:2.0:1773946220.070344:0:191276:0:(client.c:2614:ptlrpc_set_next_timeout()) Process entered 00000100:00000001:2.0:1773946220.070345:0:191276:0:(client.c:2642:ptlrpc_set_next_timeout()) Process leaving (rc=0 : 0 : 0) 00000100:00000001:2.0:1773946220.070347:0:191276:0:(ptlrpcd.c:252:ptlrpcd_check()) Process entered 00000100:00000001:2.0:1773946220.070349:0:191276:0:(ptlrpcd.c:351:ptlrpcd_check()) Process leaving (rc=0 : 0 : 0) 00000080:00000001:3.0:1773946220.975528:0:194360:0:(llite_lib.c:4092:ll_show_options()) Process leaving (rc=0 : 0 : 0) 00000001:00000001:0.0:1773946221.088617:0:194396:0:(debug.c:644:libcfs_debug_mark_buffer()) ************************************************** 00000001:02000400:0.0:1773946221.088622:0:194396:0:(debug.c:645:libcfs_debug_mark_buffer()) DEBUG MARKER: conf-sanity test_162: @@@@@@ FAIL: MDT start failed 00010000:00000001:2.0:1773946221.089835:0:121148:0:(ldlm_pool.c:458:ldlm_cli_pool_recalc()) Process entered 00010000:00000001:2.0:1773946221.089838:0:121148:0:(ldlm_pool.c:462:ldlm_cli_pool_recalc()) Process leaving (rc=0 : 0 : 0) 00000020:00000001:3.0:1773946221.089902:0:193418:0:(genops.c:1914:obd_stale_export_get()) Process entered 00000020:00000001:3.0:1773946221.089905:0:193418:0:(genops.c:1928:obd_stale_export_get()) Process leaving (rc=0 : 0 : 0) 00010000:00010000:3.0:1773946221.089908:0:193418:0:(ldlm_lockd.c:2797:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:00000001:0.0:1773946221.090315:0:194396:0:(debug.c:647:libcfs_debug_mark_buffer()) ************************************************** 00000080:00000001:0.0:1773946221.487564:0:194496:0:(llite_lib.c:4092:ll_show_options()) Process leaving (rc=0 : 0 : 0) Debug log: 700 lines, 700 kept, 0 dropped, 0 bad.