[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 786350532 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002499] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004012] kvm-guest: setup PV IPIs [ 0.007736] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010122] LSM: Security Framework initializing [ 0.011047] Yama: becoming mindful. [ 0.012045] SELinux: Initializing. [ 0.013090] *** VALIDATE selinux *** [ 0.027536] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.031328] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.032136] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.033091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.034141] *** VALIDATE tmpfs *** [ 0.035381] *** VALIDATE proc *** [ 0.036168] *** VALIDATE cgroup *** [ 0.037005] *** VALIDATE cgroup2 *** [ 0.038220] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.042009] Spectre V2 : User space: Vulnerable [ 0.043008] Speculative Store Bypass: Vulnerable [ 0.045910] debug: unmapping init [mem 0xffffffffa8059000-0xffffffffa8060fff] [ 0.048312] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.049758] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.050030] ... version: 2 [ 0.051010] ... bit width: 48 [ 0.052011] ... generic registers: 4 [ 0.053008] ... value mask: 0000ffffffffffff [ 0.054009] ... max period: 00007fffffffffff [ 0.055009] ... fixed-purpose events: 3 [ 0.056007] ... event mask: 000000070000000f [ 0.057350] rcu: Hierarchical SRCU implementation. [ 0.059659] smp: Bringing up secondary CPUs ... [ 0.060600] x86: Booting SMP configuration: [ 0.061021] .... node #0, CPUs: #1 #2 #3 [ 0.068317] smp: Brought up 1 node, 4 CPUs [ 0.070014] smpboot: Max logical packages: 1 [ 0.071017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.101052] node 0 deferred pages initialised in 25ms [ 0.104481] devtmpfs: initialized [ 0.105187] x86/mm: Memory block size: 128MB [ 0.108057] gcov: version magic: 0x41383552 [ 0.110224] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.111074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.112229] pinctrl core: initialized pinctrl subsystem [ 0.113176] [ 0.113694] ************************************************************* [ 0.114024] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.115011] ** ** [ 0.116016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.117009] ** ** [ 0.118012] ** This means that this kernel is built to expose internal ** [ 0.119010] ** IOMMU data structures, which may compromise security on ** [ 0.120012] ** your system. ** [ 0.121011] ** ** [ 0.122009] ** If you see this message and you are not debugging the ** [ 0.123020] ** kernel, report this immediately to your vendor! ** [ 0.124011] ** ** [ 0.125010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.126023] ************************************************************* [ 0.127927] NET: Registered protocol family 16 [ 0.128408] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.129057] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.130053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.131627] cpuidle: using governor menu [ 0.134096] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.135494] PCI: Using configuration type 1 for base access [ 0.137172] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.148447] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.149000] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.151065] cryptd: max_cpu_qlen set to 1000 [ 0.153220] ACPI: Added _OSI(Module Device) [ 0.154000] ACPI: Added _OSI(Processor Device) [ 0.154000] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.155012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.161574] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.167973] ACPI: Interpreter enabled [ 0.170048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.172008] ACPI: Using IOAPIC for interrupt routing [ 0.174090] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.178508] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.192472] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.194037] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.198015] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.202074] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.209347] acpiphp: Slot [2] registered [ 0.211097] acpiphp: Slot [5] registered [ 0.213090] acpiphp: Slot [6] registered [ 0.215123] acpiphp: Slot [7] registered [ 0.216088] acpiphp: Slot [8] registered [ 0.218086] acpiphp: Slot [9] registered [ 0.219071] acpiphp: Slot [10] registered [ 0.220065] acpiphp: Slot [3] registered [ 0.221055] acpiphp: Slot [4] registered [ 0.222000] acpiphp: Slot [11] registered [ 0.222000] acpiphp: Slot [12] registered [ 0.223062] acpiphp: Slot [13] registered [ 0.224059] acpiphp: Slot [14] registered [ 0.225076] acpiphp: Slot [15] registered [ 0.226055] acpiphp: Slot [16] registered [ 0.227060] acpiphp: Slot [17] registered [ 0.228095] acpiphp: Slot [18] registered [ 0.229118] acpiphp: Slot [19] registered [ 0.231068] acpiphp: Slot [20] registered [ 0.232063] acpiphp: Slot [21] registered [ 0.233101] acpiphp: Slot [22] registered [ 0.234061] acpiphp: Slot [23] registered [ 0.235027] acpiphp: Slot [24] registered [ 0.235952] acpiphp: Slot [25] registered [ 0.237076] acpiphp: Slot [26] registered [ 0.238053] acpiphp: Slot [27] registered [ 0.240072] acpiphp: Slot [28] registered [ 0.241095] acpiphp: Slot [29] registered [ 0.243067] acpiphp: Slot [30] registered [ 0.245136] acpiphp: Slot [31] registered [ 0.246060] PCI host bridge to bus 0000:00 [ 0.248024] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.251020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.255024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.260023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.264022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.268040] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.270472] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.275235] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.280741] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.295011] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.302063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.306014] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.309027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.313018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.318929] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.323066] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.327046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.332258] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.339013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.361016] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.372017] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.373000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.392015] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.413015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.431000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.444015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.460017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.495017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.537106] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.552000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.559014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.589012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.603548] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.614014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.628015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.663014] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.679819] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.687000] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.689000] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.712016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.721000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.727015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.825012] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.840012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.850244] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.853403] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.856986] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.860218] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.862130] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.865292] iommu: Default domain type: Passthrough [ 0.868520] SCSI subsystem initialized [ 0.870129] ACPI: bus type USB registered [ 0.873099] usbcore: registered new interface driver usbfs [ 0.878310] usbcore: registered new interface driver hub [ 0.884117] usbcore: registered new device driver usb [ 0.886395] pps_core: LinuxPPS API ver. 1 registered [ 0.889023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.896105] PTP clock support registered [ 0.900203] EDAC MC: Ver: 3.0.0 [ 0.902708] PCI: Using ACPI for IRQ routing [ 0.908563] NetLabel: Initializing [ 0.910011] NetLabel: domain hash size = 128 [ 0.912009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.915077] NetLabel: unlabeled traffic allowed by default [ 0.918271] vgaarb: loaded [ 0.920494] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.923009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.929164] clocksource: Switched to clocksource kvm-clock [ 1.091950] VFS: Disk quotas dquot_6.6.0 [ 1.094317] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.096857] *** VALIDATE ramfs *** [ 1.098655] *** VALIDATE hugetlbfs *** [ 1.100375] pnp: PnP ACPI init [ 1.107822] pnp: PnP ACPI: found 6 devices [ 1.127213] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.131211] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.133817] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.136344] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.139546] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.142294] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.146700] NET: Registered protocol family 2 [ 1.149323] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.155857] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.159673] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.165341] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.169183] TCP: Hash tables configured (established 65536 bind 65536) [ 1.172435] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.175885] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.179231] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.182526] NET: Registered protocol family 1 [ 1.185487] RPC: Registered named UNIX socket transport module. [ 1.187428] RPC: Registered udp transport module. [ 1.189213] RPC: Registered tcp transport module. [ 1.190968] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.193222] NET: Registered protocol family 44 [ 1.194840] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.196425] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.198111] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.200267] PCI: CLS 0 bytes, default 64 [ 1.202165] Unpacking initramfs... [ 3.166967] debug: unmapping init [mem 0xffff9d7a3cc54000-0xffff9d7a3ffbffff] [ 3.171466] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.173501] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.176211] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.714747] Initialise system trusted keyrings [ 3.716565] Key type blacklist registered [ 3.718392] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.728067] zbud: loaded [ 3.731193] *** VALIDATE nfs *** [ 3.732400] *** VALIDATE nfs4 *** [ 3.734030] pstore: using deflate compression [ 3.738298] Platform Keyring initialized [ 3.855521] NET: Registered protocol family 38 [ 3.859296] Key type asymmetric registered [ 3.860772] Asymmetric key parser 'x509' registered [ 3.863202] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.866458] io scheduler mq-deadline registered [ 3.868218] io scheduler kyber registered [ 3.870153] io scheduler bfq registered [ 3.871927] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.875232] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.878452] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.881549] ACPI: Power Button [PWRF] [ 3.887540] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.895314] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.913571] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.923909] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.954175] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.984341] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.011436] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.015385] Non-volatile memory driver v1.3 [ 4.016806] Linux agpgart interface v0.103 [ 4.050721] virtio_blk virtio1: [vda] 145208 512-byte logical blocks (74.3 MB/70.9 MiB) [ 4.054284] vda: detected capacity change from 0 to 74346496 [ 4.074253] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.077837] vdb: detected capacity change from 0 to 1073741824 [ 4.101625] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.105946] vdc: detected capacity change from 0 to 2621440000 [ 4.129510] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.133878] vdd: detected capacity change from 0 to 2621440000 [ 4.154271] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.159903] vde: detected capacity change from 0 to 4294967296 [ 4.182398] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.185576] vdf: detected capacity change from 0 to 4294967296 [ 4.193505] libphy: Fixed MDIO Bus: probed [ 4.198968] usbcore: registered new interface driver usbserial_generic [ 4.201974] usbserial: USB Serial support registered for generic [ 4.204162] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.209417] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.211664] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.213989] mousedev: PS/2 mouse device common for all mice [ 4.216963] rtc_cmos 00:05: RTC can wake from S4 [ 4.220816] rtc_cmos 00:05: registered as rtc0 [ 4.224406] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.227539] intel_pstate: CPU model not supported [ 4.230410] hid: raw HID events driver (C) Jiri Kosina [ 4.231152] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.234434] usbcore: registered new interface driver usbhid [ 4.242505] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.244207] usbhid: USB HID core driver [ 4.244494] drop_monitor: Initializing network drop monitor service [ 4.244626] Initializing XFRM netlink socket [ 4.245788] NET: Registered protocol family 10 [ 4.251941] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.266442] Segment Routing with IPv6 [ 4.267942] NET: Registered protocol family 17 [ 4.270822] mpls_gso: MPLS GSO support [ 4.276149] RAS: Correctable Errors collector initialized. [ 4.277805] AVX version of gcm_enc/dec engaged. [ 4.279537] AES CTR mode by8 optimization enabled [ 4.393315] sched_clock: Marking stable (4393290995, 0)->(5879337426, -1486046431) [ 4.399253] registered taskstats version 1 [ 4.401706] Loading compiled-in X.509 certificates [ 4.404571] zswap: loaded using pool lzo/zbud [ 4.439550] Key type big_key registered [ 4.455073] Key type encrypted registered [ 4.457147] ima: No TPM chip found, activating TPM-bypass! [ 4.459461] ima: Allocated hash algorithm: sha1 [ 4.461257] ima: No architecture policies found [ 4.462826] evm: Initialising EVM extended attributes: [ 4.464524] evm: security.selinux [ 4.465983] evm: security.ima [ 4.467452] evm: security.capability [ 4.468920] evm: HMAC attrs: 0x1 [ 4.471619] rtc_cmos 00:05: setting system clock to 2026-07-04 03:55:27 UTC (1783137327) [ 4.479220] debug: unmapping init [mem 0xffffffffa9003000-0xffffffffa91fffff] [ 4.482591] debug: unmapping init [mem 0xffffffffa7d82000-0xffffffffa8058fff] [ 4.490190] Write protecting the kernel read-only data: 28672k [ 4.495506] debug: unmapping init [mem 0xffffffffa6403000-0xffffffffa65fffff] [ 4.499662] debug: unmapping init [mem 0xffffffffa6d14000-0xffffffffa6dfffff] [ 4.537866] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.546788] systemd[1]: Detected virtualization kvm. [ 4.551903] systemd[1]: Detected architecture x86-64. [ 4.555283] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.585869] systemd[1]: No hostname configured. [ 4.587636] systemd[1]: Set hostname to . [ 4.589455] random: systemd: uninitialized urandom read (16 bytes read) [ 4.591493] systemd[1]: Initializing machine ID from random generator. [ 4.721632] random: ln: uninitialized urandom read (6 bytes read) [ 4.904025] hrtimer: interrupt took 2608262 ns [ 5.150112] random: systemd: uninitialized urandom read (16 bytes read) [ 5.155715] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 5.166933] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 5.181954] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Timers. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.889332] device-mapper: uevent: version 1.0.3 [ 6.891113] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.957479] virtio_net virtio0 ens2: renamed from eth0 [ 10.119670] random: fast init done [ 11.468427] scsi host0: ata_piix [ 11.745120] scsi host1: ata_piix [ 11.748541] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 11.770996] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.033865] random: crng init done [ 18.035251] random: 7 urandom warning(s) missed due to ratelimiting [ 22.209936] dracut-initqueue[590]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 23.938590] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Sockets. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 28.166646] printk: systemd: 26 output lines suppressed due to ratelimiting [ 29.532881] SELinux: Disabled at runtime. [ 29.791434] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 29.827479] systemd[1]: Detected virtualization kvm. [ 29.829818] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 33.089364] systemd[1]: initrd-switch-root.service: Succeeded. [ 33.103837] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 33.144366] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 33.165894] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 33.180487] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 33.232545] systemd[1]: Starting Journal Service... Starting Journal Service... [ 33.413424] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on RPCbind Server Activation Socket. Starting Remount Root and Kernel File Systems... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd File Systems.[ 34.801154] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ 34.843907] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [ 34.918422] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. [ 34.995616] systemd[1]: Stopped target Initrd Root File System. [ OK ] Stopped target Initrd Root File System. [ 35.094954] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ 35.143649] systemd[1]: Reached target RPC Port Mapper. [ OK ] Reached target RPC Port Mapper. [ 35.206207] systemd[1]: Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. [ 35.259358] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 35.306580] systemd[1]: Listening on initctl Compatibility Named Pipe. [ OK ] Listening on initctl Compatibility Named Pipe. [ 35.343248] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ 35.372644] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Paths. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 37.978690] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 41.025774] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 41.247797] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 44.074318] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 44.303615] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…only root support (12s / no limit) [** ] A start job is running for Configur…only root support (12s / no limit) [*** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (14s / no limit) [ ***] A start job is running for Configur…only root support (14s / no limit) [ **] A start job is running for Configur…only root support (15s / no limit) [ *] A start job is running for Configur…only root support (15s / no limit) [ **] A start job is running for Configur…only root support (16s / no limit) [ ***] A start job is running for Configur…only root support (16s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit) [ *** ] A start job is running for Configur…only root support (17s / no limit) [*** ] A start job is running for Configur…only root support (18s / no limit) [** ] A start job is running for Configur…only root support (18s / no limit) [* ] A start job is running for Configur…only root support (19s / no limit) [** ] A start job is running for Configur…only root support (19s / no limit) [*** ] A start job is running for Configur…only root support (20s / no limit) [ *** ] A start job is running for Configur…only root support (21s / no limit) [ *** ] A start job is running for Configur…only root support (22s / no limit)[ 55.475656] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (22s / no limit) [ **] A start job is running for Configur…only root support (23s / no limit) [ *] A start job is running for Configur…only root support (23s / no limit) [ **] A start job is running for Configur…only root support (24s / no limit)[ 57.287912] NFS: Registering the id_resolver key type [ 57.290201] Key type id_resolver registered [ 57.314283] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (24s / no limit) [ *** ] A start job is running for Configur…only root support (25s / no limit) [ *** ] A start job is running for Configur…only root support (25s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [*** ] A start job is running for Rebuild …amic Linker Cache (32s / no limit) [** ] A start job is running for Rebuild …amic Linker Cache (33s / no limit) [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. Starting Hostname Service... [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 138.684652] libcfs: loading out-of-tree module taints kernel. [ 138.730365] Key type ._llcrypt registered [ 138.736140] Key type .llcrypt registered [ 138.834226] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 155.844491] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 157.344520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 157.364424] alg: No test for adler32 (adler32-zlib) [ 158.745126] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 159.662777] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 161.391396] Key type lgssc registered [ 163.225329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 178.558777] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 218.605356] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 231.980784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 232.024044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 233.329599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 233.376512] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 233.499846] Lustre: lustre-MDT0000: new disk, initializing [ 233.580119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 233.598414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 237.662905] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 250.978244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 251.089508] Lustre: 6512:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 251.117897] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 251.121550] Lustre: Skipped 1 previous similar message [ 251.206801] Lustre: lustre-MDT0001: new disk, initializing [ 251.282529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 251.324060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 251.339254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 255.504122] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 260.594292] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 269.585231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 269.840522] Lustre: lustre-OST0000: new disk, initializing [ 269.844286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 269.849753] Lustre: 8447:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 269.924755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 270.224489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 270.235097] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 270.325657] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 276.934863] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 292.438302] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 292.562667] Lustre: lustre-OST0001: new disk, initializing [ 292.566586] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 292.572839] Lustre: 9526:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 292.638798] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 299.392797] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 301.135570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 301.144484] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 301.249556] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 312.246393] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 323.691838] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 330.413669] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 335.387892] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 340.514495] Lustre: DEBUG MARKER: Client: 2.17.54.85 [ 343.351364] Lustre: DEBUG MARKER: MDS: 2.17.54.85 [ 346.602820] Lustre: DEBUG MARKER: OSS: 2.17.54.85 [ 348.117200] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Jul 4 00:01:10 EDT 2026 [ 365.760686] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 367.206805] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 383.456939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.460623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 383.489448] Lustre: Skipped 1 previous similar message [ 383.517771] Lustre: Skipped 3 previous similar messages [ 388.577544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 388.585954] Lustre: Skipped 3 previous similar messages [ 389.743460] Lustre: server umount lustre-MDT0000 complete [ 393.745181] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137716 with bad export cookie 13973020979241542962 [ 393.751566] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 393.759031] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 398.819391] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 398.823633] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 398.823836] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 398.841634] Lustre: Skipped 3 previous similar messages [ 398.883928] LustreError: 6520:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 400.340496] Lustre: server umount lustre-MDT0001 complete [ 405.026240] Lustre: server umount lustre-OST0000 complete [ 409.517258] Lustre: server umount lustre-OST0001 complete [ 416.831509] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 427.888376] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 466.626426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 466.872518] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 466.892406] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 467.004312] Lustre: lustre-MDT0000: new disk, initializing [ 467.110047] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 467.131062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 471.508807] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 482.439123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 482.584258] Lustre: 15094:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 482.667666] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 482.674380] Lustre: Skipped 1 previous similar message [ 482.793832] Lustre: lustre-MDT0001: new disk, initializing [ 482.940845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 483.029822] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 483.046793] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 487.813318] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 497.352955] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 503.892406] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 510.764409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 511.121383] Lustre: lustre-OST0000: new disk, initializing [ 511.133534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 511.153541] Lustre: 17060:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 511.265948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 513.181515] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 513.194334] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 513.315445] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 518.387597] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 528.021626] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 533.162923] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 533.432043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 538.848166] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 539.052568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 544.228217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 544.231937] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 544.275968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 544.285147] Lustre: Skipped 1 previous similar message [ 547.465547] Lustre: server umount lustre-OST0000 complete [ 554.469996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.476814] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 554.491553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.491562] Lustre: Skipped 1 previous similar message [ 554.498796] Lustre: Skipped 2 previous similar messages [ 564.715824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 564.724467] Lustre: Skipped 4 previous similar messages [ 566.242832] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 566.502358] Lustre: server umount lustre-MDT0000 complete [ 569.824988] LustreError: 15105:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.846610] LustreError: 15105:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 570.351974] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783137893 with bad export cookie 13973020979241544495 [ 570.353186] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.361153] LustreError: 15085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 570.814511] Lustre: server umount lustre-MDT0001 complete [ 575.632378] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:04:57 (1783137897) [ 578.167914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 580.249817] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:05:02 (1783137902) [ 587.819751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.385216] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 588.502342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 593.891790] LustreError: 19473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 594.186557] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 597.985940] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 603.959263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 610.356380] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 618.859704] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 624.860819] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 632.724813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 639.726416] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 648.020251] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 664.034610] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 664.043567] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 664.062405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 666.715410] Lustre: server umount lustre-OST0000 complete [ 671.200833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 675.313255] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.324264] Lustre: Skipped 2 previous similar messages [ 677.402087] Lustre: server umount lustre-MDT0000 complete [ 680.421930] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 680.456046] LustreError: 19472:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 681.108973] LustreError: 19454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138004 with bad export cookie 13973020979241545377 [ 681.115309] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 681.119458] LustreError: 19454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 681.591505] Lustre: server umount lustre-MDT0001 complete [ 694.182475] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 697.164181] Key type lgssc unregistered [ 697.538396] LNet: 23020:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 697.549754] LNetError: 23020:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 697.574273] LNet: Removed LNI 192.168.203.155@tcp [ 698.822179] Key type .llcrypt unregistered [ 698.825450] Key type ._llcrypt unregistered [ 724.102063] Key type ._llcrypt registered [ 724.109180] Key type .llcrypt registered [ 724.317950] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 725.769555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 725.807160] alg: No test for adler32 (adler32-zlib) [ 726.975925] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 727.209499] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 728.895169] Key type lgssc registered [ 729.960524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 738.964483] loop: module loaded [ 739.072244] loop0: detected capacity change from 0 to 204800000 [ 739.374287] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.394708] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.775113] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 746.025687] loop0: detected capacity change from 0 to 204800000 [ 746.177951] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 746.198858] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 746.223532] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 752.170809] loop0: detected capacity change from 0 to 204800000 [ 758.548033] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 758.594502] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 759.892821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 759.981436] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 760.162755] Lustre: lustre-MDT0000: new disk, initializing [ 760.297779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.353795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 765.238680] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 774.793094] loop1: detected capacity change from 0 to 204800000 [ 780.932177] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 781.246203] Lustre: lustre-OST07c6: new disk, initializing [ 781.250811] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 781.256584] Lustre: Skipped 1 previous similar message [ 781.263756] Lustre: 26215:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 781.344647] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 788.235567] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 790.616057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 790.622648] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 790.646355] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 793.155883] Lustre: 25204:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 793.170988] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 793.178691] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 793.187955] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 793.206938] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 793.222374] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 809.494397] Lustre: 26843:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 809.505383] Lustre: 26843:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 809.523442] Lustre: 26843:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 809.532926] Lustre: 26843:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 809.540607] Lustre: 26843:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 809.547229] Lustre: 26843:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 814.647680] Lustre: 25202:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 814.653702] Lustre: 25202:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 1 previous similar message [ 814.668957] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 814.682653] Lustre: 25202:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.701619] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 814.722432] Lustre: 25202:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.729370] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 814.736745] Lustre: 25202:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.744425] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 814.750821] Lustre: 25202:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.759869] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 814.772047] Lustre: 25202:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.944403] Lustre: 26843:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 817.956403] Lustre: 26843:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 1 previous similar message [ 817.964158] Lustre: 26843:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 817.970817] Lustre: 26843:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.976682] Lustre: 26843:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 817.982914] Lustre: 26843:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.987633] Lustre: 26843:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 817.992837] Lustre: 26843:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 817.999470] Lustre: 26843:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 818.011709] Lustre: 26843:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.024234] Lustre: 26843:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 818.031607] Lustre: 26843:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.869499] Lustre: 25204:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 822.885357] Lustre: 25204:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 1 previous similar message [ 822.891801] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 822.898012] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.909114] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 822.913880] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.921424] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 822.928082] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.936459] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 822.943587] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 822.949622] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 822.955070] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 834.618739] Lustre: 25204:0:(osd_internal.h:1444:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 834.630607] Lustre: 25204:0:(osd_internal.h:1444:osd_trans_exec_op()) Skipped 5 previous similar messages [ 834.636256] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 834.652950] Lustre: 25204:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 834.663209] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 834.678191] Lustre: 25204:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 834.689601] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 834.701364] Lustre: 25204:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 834.715681] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 834.722447] Lustre: 25204:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 834.736994] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 834.746157] Lustre: 25204:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 882.664260] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 882.667315] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 882.692167] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 887.990850] Lustre: server umount lustre-OST07c6 complete [ 897.879327] Lustre: server umount lustre-MDT0000 complete [ 919.145546] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 923.250288] Key type lgssc unregistered [ 923.872772] LNet: 28778:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 923.887240] LNetError: 28778:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 923.912955] LNet: Removed LNI 192.168.203.155@tcp [ 925.186527] Key type .llcrypt unregistered [ 925.188618] Key type ._llcrypt unregistered [ 946.583483] Key type ._llcrypt registered [ 946.585754] Key type .llcrypt registered [ 946.718732] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 947.876958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.081596] alg: No test for adler32 (adler32-zlib) [ 949.191506] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 949.427691] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 951.170575] Key type lgssc registered [ 952.381869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 959.693226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 959.719640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.250466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 965.952616] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 971.747961] LustreError: 30049:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 971.774829] LustreError: 30049:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 974.336777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.811465] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 979.800927] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 988.203675] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 994.561201] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1002.594532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1003.081711] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1006.202671] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1011.440593] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1020.579681] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1037.285170] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1037.297306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1037.322984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.369324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1040.375286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1040.378838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.405104] Lustre: Skipped 1 previous similar message [ 1043.595754] Lustre: server umount lustre-OST0000 complete [ 1050.594902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1050.597912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1050.610220] Lustre: Skipped 1 previous similar message [ 1054.522742] Lustre: server umount lustre-MDT0000 complete [ 1055.718775] LustreError: 31889:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1055.743862] LustreError: 31889:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1058.459058] LustreError: 30030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138381 with bad export cookie 9632584288928414323 [ 1058.473992] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1058.905837] Lustre: server umount lustre-MDT0001 complete [ 1070.864356] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1074.022031] Key type lgssc unregistered [ 1074.370591] LNet: 33789:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.377820] LNetError: 33789:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.397319] LNet: Removed LNI 192.168.203.155@tcp [ 1075.293147] Key type .llcrypt unregistered [ 1075.298174] Key type ._llcrypt unregistered [ 1094.393702] Key type ._llcrypt registered [ 1094.396608] Key type .llcrypt registered [ 1094.550822] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1095.894337] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1095.996772] alg: No test for adler32 (adler32-zlib) [ 1097.085747] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 1097.444377] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1099.103667] Key type lgssc registered [ 1100.400572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1109.426372] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1109.482497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1110.994406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1115.320346] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1120.226748] LustreError: 35061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1120.240326] LustreError: 35061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1123.257795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1123.801727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1128.146140] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1137.389525] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1144.134324] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1151.324446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1151.685230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1155.782053] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1158.172193] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1165.357307] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1181.664767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1181.671990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1181.689689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1183.713798] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1183.717224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1183.736843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.752938] Lustre: Skipped 1 previous similar message [ 1186.551774] Lustre: server umount lustre-OST0000 complete [ 1190.369802] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1190.385648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1190.406626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.955105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.956652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1196.435975] Lustre: server umount lustre-MDT0000 complete [ 1199.075691] LustreError: 35060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1199.105330] LustreError: 35060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1200.470893] LustreError: 36904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138523 with bad export cookie 12554868825864897105 [ 1200.487554] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1200.842038] Lustre: server umount lustre-MDT0001 complete [ 1212.786393] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1216.228788] Key type lgssc unregistered [ 1216.596834] LNet: 38614:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1216.614409] LNetError: 38614:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1216.641666] LNet: Removed LNI 192.168.203.155@tcp [ 1217.699689] Key type .llcrypt unregistered [ 1217.702134] Key type ._llcrypt unregistered [ 1243.969530] Key type ._llcrypt registered [ 1243.972192] Key type .llcrypt registered [ 1244.099946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1245.232174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1245.320714] alg: No test for adler32 (adler32-zlib) [ 1246.406627] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 1246.651212] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1248.319173] Key type lgssc registered [ 1249.466784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1256.742142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1256.761382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1258.189785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1262.795516] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1268.708927] LustreError: 40031:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.728528] LustreError: 40031:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1271.212700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.563831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1276.352770] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1283.228941] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1289.541329] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1296.395398] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1296.740763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.088376] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1303.945996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1310.139659] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1321.442352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1321.444532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1321.470295] Lustre: Skipped 1 previous similar message [ 1323.427582] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1323.435592] Lustre: Skipped 2 previous similar messages [ 1326.564465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.579751] Lustre: Skipped 2 previous similar messages [ 1326.875480] Lustre: server umount lustre-MDT0000 complete [ 1328.523275] LustreError: 40027:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1330.615057] LustreError: 40010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138653 with bad export cookie 11563756266442442903 [ 1330.622893] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1330.633707] LustreError: 40010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1331.681948] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.685929] LustreError: 40024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1331.693885] Lustre: Skipped 1 previous similar message [ 1331.721568] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1335.713942] LustreError: 41046:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1335.751623] LustreError: 41046:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1337.123345] Lustre: server umount lustre-MDT0001 complete [ 1352.278843] Lustre: server umount lustre-OST0000 complete [ 1367.480807] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1370.164795] Key type lgssc unregistered [ 1370.435376] LNet: 43892:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1370.451186] LNetError: 43892:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1371.493886] LNet: Removed LNI 192.168.203.155@tcp [ 1372.305857] Key type .llcrypt unregistered [ 1372.307789] Key type ._llcrypt unregistered [ 1395.401287] Key type ._llcrypt registered [ 1395.404017] Key type .llcrypt registered [ 1395.536490] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1396.772375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1396.810967] alg: No test for adler32 (adler32-zlib) [ 1397.871651] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 1398.091690] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1399.791233] Key type lgssc registered [ 1400.933890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1409.257148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1409.275476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1410.799867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1414.834732] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1420.264804] LustreError: 45311:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1420.307688] LustreError: 45311:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1423.844395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1424.252250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1428.699629] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1436.576268] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1441.861348] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1447.887544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1448.202969] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1454.370831] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1455.419593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1462.712318] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1469.867656] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1469.920495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1469.930955] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.951868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1470.960882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1470.962415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1470.982695] Lustre: Skipped 2 previous similar messages [ 1474.956228] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1480.078154] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1480.085220] Lustre: Skipped 2 previous similar messages [ 1483.233040] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1483.381650] Lustre: server umount lustre-OST0000 complete [ 1501.152259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1501.157181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1501.178217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1501.187618] Lustre: Skipped 2 previous similar messages [ 1507.184017] Lustre: server umount lustre-MDT0000 complete [ 1511.030882] LustreError: 45293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783138834 with bad export cookie 10836162411754179961 [ 1511.049566] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1511.050610] LustreError: 45293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1517.508059] Lustre: server umount lustre-MDT0001 complete [ 1528.439897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1531.523778] Key type lgssc unregistered [ 1531.841683] LNet: 49019:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1531.853587] LNetError: 49019:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1531.879955] LNet: Removed LNI 192.168.203.155@tcp [ 1532.831235] Key type .llcrypt unregistered [ 1532.837494] Key type ._llcrypt unregistered [ 1555.396832] Key type ._llcrypt registered [ 1555.400292] Key type .llcrypt registered [ 1555.492307] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1556.800540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1556.896640] alg: No test for adler32 (adler32-zlib) [ 1558.030395] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 1558.329386] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1560.047371] Key type lgssc registered [ 1561.358707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1570.361767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1570.420036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1572.036045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1577.218981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1581.536553] LustreError: 50439:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.578990] LustreError: 50439:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1586.657471] LustreError: 50438:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1587.125728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1587.478374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1592.007354] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1598.599907] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1603.247906] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1609.489695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1609.878841] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1616.056120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1616.437410] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1623.176428] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1628.606745] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1631.060643] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1642.807085] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-04 0h22m45s [ 1643.939610] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1646.499869] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1647.072785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1647.078293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.084904] Lustre: Skipped 1 previous similar message [ 1647.094940] Lustre: Skipped 2 previous similar messages [ 1648.168766] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1648.173928] Lustre: Skipped 3 previous similar messages [ 1652.617680] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1658.760338] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1658.772483] Lustre: Skipped 3 previous similar messages [ 1660.384799] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1660.659160] Lustre: server umount lustre-MDT0000 complete [ 1663.786558] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1664.907969] LustreError: 50435:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1669.465789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1669.611337] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1669.867460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.924135] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1670.890149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1673.677946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1676.893167] Lustre: DEBUG MARKER: Wait for df (36591) ... [ 1678.243807] Lustre: DEBUG MARKER: done [ 1685.476198] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1685.484420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1685.507346] Lustre: Skipped 1 previous similar message [ 1685.514250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1685.520527] Lustre: Skipped 3 previous similar messages [ 1685.999397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1687.691819] Lustre: server umount lustre-OST0000 complete [ 1691.616977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1691.630036] Lustre: Skipped 2 previous similar messages [ 1697.251201] Lustre: server umount lustre-MDT0000 complete [ 1697.762409] LustreError: 51460:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1697.785589] LustreError: 51460:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1700.408277] LustreError: 50420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139023 with bad export cookie 16951961811676919506 [ 1700.409347] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1700.429930] LustreError: 50420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1700.780430] Lustre: server umount lustre-MDT0001 complete [ 1710.960324] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1713.585563] Key type lgssc unregistered [ 1713.863912] LNet: 55363:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1713.886168] LNetError: 55363:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1713.897216] LNet: Removed LNI 192.168.203.155@tcp [ 1714.662177] Key type .llcrypt unregistered [ 1714.667845] Key type ._llcrypt unregistered [ 1749.755289] Key type ._llcrypt registered [ 1749.760914] Key type .llcrypt registered [ 1749.872887] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1750.747610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1750.797132] alg: No test for adler32 (adler32-zlib) [ 1751.863705] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 1752.105199] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1753.770397] Key type lgssc registered [ 1754.845531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1763.144864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1763.165670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1764.483529] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1764.515129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1764.720562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1769.793346] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1778.025706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1778.133937] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1778.191743] Lustre: 57574:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1778.436883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1782.286402] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1789.344588] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1795.126814] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1802.103812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1802.307669] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1802.546084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1808.014463] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1808.633268] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1815.153879] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1820.854723] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-04 0h25m43s [ 1822.382306] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1829.323532] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1831.682855] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1831.728755] LustreError: 57580:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1851.844126] Lustre: lustre-MDT0000: Client 89d179e1-79b7-46f7-a27c-976df2db2d16 (at 192.168.203.55@tcp) reconnecting [ 1871.759133] LustreError: 57580:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1873.661810] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1876.094219] Lustre: DEBUG MARKER: done [ 1884.640932] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1884.654490] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.668128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1885.158860] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1885.165531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1885.182113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1885.187244] Lustre: Skipped 1 previous similar message [ 1888.334873] Lustre: server umount lustre-OST0000 complete [ 1895.393599] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.418388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1895.424562] Lustre: Skipped 1 previous similar message [ 1898.285353] Lustre: server umount lustre-MDT0000 complete [ 1901.368649] LustreError: 57565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139224 with bad export cookie 17395422998850966106 [ 1901.371148] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1901.387761] LustreError: 57565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1901.745744] Lustre: server umount lustre-MDT0001 complete [ 1911.543956] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1914.432933] Key type lgssc unregistered [ 1914.661570] LNet: 61899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.667876] LNetError: 61899:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.688032] LNet: Removed LNI 192.168.203.155@tcp [ 1916.510560] Key type .llcrypt unregistered [ 1916.514515] Key type ._llcrypt unregistered [ 1958.949765] Key type ._llcrypt registered [ 1958.954748] Key type .llcrypt registered [ 1959.030321] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1959.936469] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1960.090945] alg: No test for adler32 (adler32-zlib) [ 1961.182440] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 1961.406302] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1963.135253] Key type lgssc registered [ 1964.347612] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1967.732069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1967.760124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.033612] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1969.052515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1969.222885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.134615] Lustre: Failing over lustre-MDT0000 [ 1970.369572] Lustre: server umount lustre-MDT0000 complete [ 1976.308259] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:28:18 (1783139298) [ 1982.436607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1982.891237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.092839] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1994.972738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1995.073779] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1995.101543] Lustre: 64696:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1995.374428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2000.102950] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2007.796122] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2013.487675] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2020.211055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2020.409587] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2020.707037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2026.962153] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2028.166766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2033.794074] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2039.799902] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2049.003183] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2049.009558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2049.032399] Lustre: Skipped 1 previous similar message [ 2049.052918] Lustre: Skipped 2 previous similar messages [ 2052.814072] Lustre: server umount lustre-MDT0000 complete [ 2056.397521] LustreError: 64690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139379 with bad export cookie 1975227653269131728 [ 2056.404512] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2056.421166] LustreError: 64690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2056.685465] Lustre: server umount lustre-MDT0001 complete [ 2059.682676] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2064.051417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2072.340775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.615758] Lustre: 63619:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139382/real 1783139382] req@00000000ecde257d x1869757057900032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2075.646856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2075.659556] Lustre: Skipped 1 previous similar message [ 2077.667068] Lustre: 63621:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139384/real 1783139384] req@00000000ea09ecfe x1869757057900160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2080.735356] Lustre: 63622:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139387/real 1783139387] req@0000000080152504 x1869757057900288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139403 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2081.824639] LustreError: 63618:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000078028bbe x1869757057901056/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2082.247845] LustreError: 66545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2082.267324] LustreError: 66545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2082.354964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.460311] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2083.361159] Lustre: 63620:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139389/real 1783139389] req@00000000aa63875b x1869757057900544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139405 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2084.389522] LustreError: 66546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.412858] LustreError: 68466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.440942] LustreError: 68466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2087.083704] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2088.416328] Lustre: 63619:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139394/real 1783139394] req@0000000031a8a23d x1869757057900672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139410 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2090.469879] LustreError: 68647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.481692] LustreError: 68647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2094.859260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2095.075096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2099.023517] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2100.216831] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2104.368932] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2106.447162] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2111.818638] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2119.654071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2119.659841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2119.672736] Lustre: Skipped 1 previous similar message [ 2119.685467] Lustre: Skipped 1 previous similar message [ 2124.254709] Lustre: server umount lustre-MDT0000 complete [ 2124.780563] LustreError: 68647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2124.805988] LustreError: 68647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2128.055254] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139451 with bad export cookie 1975227653269134003 [ 2128.057536] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2128.061140] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2128.445419] Lustre: server umount lustre-MDT0001 complete [ 2134.865943] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2143.838949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2145.248380] Lustre: 63620:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783139452/real 1783139452] req@000000007b21944e x1869757057927168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783139468 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2145.285798] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.303175] Lustre: Skipped 1 previous similar message [ 2153.506865] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1b696b15dfa2a016 [ 2153.523389] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [ 2153.830232] LustreError: 66546:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2153.933700] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2153.987441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2157.367804] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2164.528201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.808453] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2170.371126] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2174.464618] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2175.558705] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2180.514991] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2186.703776] Lustre: server umount lustre-MDT0000 complete [ 2189.796615] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2189.814140] Lustre: Skipped 2 previous similar messages [ 2189.818723] LustreError: 66545:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2189.831838] LustreError: 66545:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2189.926231] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139512 with bad export cookie 1975227653269135382 [ 2189.926987] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.935139] LustreError: 64688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2190.316400] Lustre: server umount lustre-MDT0001 complete [ 2195.949219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2199.498901] Lustre: DEBUG MARKER: files compared the same [ 2211.617905] Lustre: server umount lustre-OST0000 complete [ 2227.607390] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2231.386358] Key type lgssc unregistered [ 2231.750544] LNet: 74421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2231.763052] LNetError: 74421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2231.783367] LNet: Removed LNI 192.168.203.155@tcp [ 2233.000425] Key type .llcrypt unregistered [ 2233.006380] Key type ._llcrypt unregistered [ 2258.647779] Key type ._llcrypt registered [ 2258.649602] Key type .llcrypt registered [ 2258.742439] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2259.532259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2259.546432] alg: No test for adler32 (adler32-zlib) [ 2260.647175] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 2260.962451] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2262.703178] Key type lgssc registered [ 2263.731608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2271.864994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2271.892732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2273.334783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2277.549644] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2283.487894] LustreError: 75840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2283.523530] LustreError: 75840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2286.311920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2286.594869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2290.354533] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2297.031209] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2302.925798] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2311.257734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2311.612547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.672443] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2318.768658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2325.819452] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2339.314773] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2339.326194] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2339.353721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2341.354532] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.354822] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2341.363916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2345.491330] Lustre: server umount lustre-OST0000 complete [ 2351.591934] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2351.605472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.629864] Lustre: Skipped 1 previous similar message [ 2351.654479] Lustre: Skipped 2 previous similar messages [ 2355.832530] Lustre: server umount lustre-MDT0000 complete [ 2356.704122] LustreError: 75834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2356.713134] LustreError: 75834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2360.118874] LustreError: 75819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139683 with bad export cookie 12685791584066852420 [ 2360.126169] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2360.144974] LustreError: 75819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2360.489886] Lustre: server umount lustre-MDT0001 complete [ 2372.329989] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2375.241372] Key type lgssc unregistered [ 2375.542273] LNet: 79396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2375.552584] LNetError: 79396:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2375.574701] LNet: Removed LNI 192.168.203.155@tcp [ 2376.816416] Key type .llcrypt unregistered [ 2376.821499] Key type ._llcrypt unregistered [ 2399.680209] Key type ._llcrypt registered [ 2399.682474] Key type .llcrypt registered [ 2399.825948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2400.909128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2401.007162] alg: No test for adler32 (adler32-zlib) [ 2401.976784] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 2402.190126] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2403.887697] Key type lgssc registered [ 2404.892885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2412.315892] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2412.344994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2427.935495] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2433.055560] LustreError: 80789:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2459.744520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.689230] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2472.285170] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2479.070203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.177681] Lustre: *** cfs_fail_loc=706, val=0*** [ 2479.497114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.562407] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2483.449863] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2484.713464] LustreError: 80814:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.750970] LustreError: 80814:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2489.827456] LustreError: 80815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.973682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.346482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2495.079371] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2500.692509] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2505.113671] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2506.721365] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2506.729481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2506.740749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.794058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2509.800047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2512.757326] Lustre: server umount lustre-OST0000 complete [ 2516.960437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2516.969249] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2516.986336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2516.993759] Lustre: Skipped 1 previous similar message [ 2520.032278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2520.038773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2525.156719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2525.165556] Lustre: Skipped 2 previous similar messages [ 2530.271244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2530.423066] Lustre: server umount lustre-MDT0000 complete [ 2533.607032] LustreError: 80796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783139856 with bad export cookie 4293186349830896804 [ 2533.613229] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2533.616736] LustreError: 80796:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2533.923345] Lustre: server umount lustre-MDT0001 complete [ 2543.862426] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2546.215748] Key type lgssc unregistered [ 2546.547097] LNet: 84402:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2546.558973] LNetError: 84402:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2546.579489] LNet: Removed LNI 192.168.203.155@tcp [ 2547.374447] Key type .llcrypt unregistered [ 2547.377155] Key type ._llcrypt unregistered [ 2568.698285] Key type ._llcrypt registered [ 2568.707231] Key type .llcrypt registered [ 2568.853582] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2570.015216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2570.063359] alg: No test for adler32 (adler32-zlib) [ 2571.116847] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 2571.395722] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2573.111337] Key type lgssc registered [ 2574.574874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2581.074000] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2581.097351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2585.876838] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2592.435168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2592.887257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2596.705331] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2598.368992] LustreError: 86484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2598.385277] LustreError: 86484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2602.471293] LustreError: 86487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2602.493328] LustreError: 86487:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2603.813392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2604.152033] LustreError: 86484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2604.199236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2609.634624] LustreError: 86483:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.658131] LustreError: 86483:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2609.980346] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2614.759716] LustreError: 86484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.799308] LustreError: 86484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2618.339639] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2618.347919] Lustre: MGS: Client dc2f1d1f-42a1-4854-a3b3-7466890ce7e7 (at 0@lo) reconnecting [ 2618.359677] LustreError: 86498:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2618.539918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2618.610713] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2622.798635] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2639.328539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2639.338700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2639.349598] Lustre: Skipped 1 previous similar message [ 2639.369933] Lustre: Skipped 1 previous similar message [ 2641.719316] Lustre: server umount lustre-OST0000 complete [ 2649.570486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2649.573620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2649.589702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2652.313929] Lustre: server umount lustre-MDT0000 complete [ 2654.690122] LustreError: 86488:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2654.711401] LustreError: 86488:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2657.395805] Lustre: server umount lustre-MDT0001 complete [ 2661.451311] Lustre: server umount MGS complete [ 2673.103153] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:39:55 (1783139995) [ 2700.204660] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 2708.916493] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2745.733881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2750.858332] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2760.886078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.121792] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2761.162491] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2761.242878] Lustre: lustre-MDT0001: new disk, initializing [ 2761.320819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.340948] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2761.351966] Lustre: 93655:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2761.366452] LustreError: 93655:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2765.942491] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2776.311783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2776.615874] Lustre: lustre-OST0000: new disk, initializing [ 2776.620128] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2776.627776] Lustre: 94420:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2776.704575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2782.473126] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2786.918024] Lustre: 94896:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000008d5cb095 x1869757697531392/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783140125 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2792.082418] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2798.575271] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2798.608977] Lustre: MGS: Client 6baac161-bb43-4364-ad9a-d7bb898d26f1 (at 0@lo) reconnecting [ 2798.640611] LustreError: 93641:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2798.673379] Lustre: 92922:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2798.733663] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2798.838777] Lustre: lustre-MDT0000: new disk, initializing [ 2799.077991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2799.114726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2799.257817] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2800.165983] Lustre: 95489:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000c344c62f x1869757697538176/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1783140139 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2804.415603] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2807.278780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2807.296303] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2809.262401] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2809.269765] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2809.390178] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2835.425414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2835.434620] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2835.455571] Lustre: Skipped 1 previous similar message [ 2835.460897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2835.475787] Lustre: Skipped 1 previous similar message [ 2842.081520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2842.096549] Lustre: Skipped 3 previous similar messages [ 2847.203292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2847.214237] Lustre: Skipped 1 previous similar message [ 2849.247138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2849.428106] Lustre: server umount lustre-OST0000 complete [ 2857.442757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2857.460000] Lustre: Skipped 1 previous similar message [ 2857.479688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2857.486205] Lustre: Skipped 1 previous similar message [ 2867.168172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2867.373692] Lustre: server umount lustre-MDT0000 complete [ 2867.708072] LustreError: 93626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2867.734256] LustreError: 93626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2878.089317] Lustre: server umount lustre-MDT0001 complete [ 2882.075946] Lustre: server umount MGS complete [ 2893.971290] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:43:36 (1783140216) [ 2910.759795] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2913.752370] Key type lgssc unregistered [ 2914.159741] LNet: 98001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2914.166558] LNetError: 98001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2914.189208] LNet: Removed LNI 192.168.203.155@tcp [ 2915.107981] Key type .llcrypt unregistered [ 2915.109530] Key type ._llcrypt unregistered [ 2933.732272] Key type ._llcrypt registered [ 2933.734419] Key type .llcrypt registered [ 2933.830870] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2935.034542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2935.076501] alg: No test for adler32 (adler32-zlib) [ 2936.093866] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 2936.319102] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2937.969225] Key type lgssc registered [ 2939.044869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2946.806182] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2946.847443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.240052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.559634] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2958.817488] LustreError: 99222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2958.837588] LustreError: 99222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2960.894900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2961.289114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2965.514259] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2972.777761] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2977.578470] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2984.883755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2985.164399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2990.496505] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2993.393719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2997.731241] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3006.944310] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3006.957262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3006.972880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3008.994720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3009.006814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3009.015502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.053884] Lustre: Skipped 1 previous similar message [ 3012.442136] Lustre: server umount lustre-OST0000 complete [ 3017.696278] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3017.704619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3017.714689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3022.098230] Lustre: server umount lustre-MDT0000 complete [ 3024.359674] LustreError: 99222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3024.393388] LustreError: 99222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3025.597430] LustreError: 101063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140348 with bad export cookie 10425751549622416973 [ 3025.600464] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3025.616247] LustreError: 101063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3025.924933] Lustre: server umount lustre-MDT0001 complete [ 3036.051692] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3038.832831] Key type lgssc unregistered [ 3039.118154] LNet: 102751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3039.125340] LNetError: 102751:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3039.147923] LNet: Removed LNI 192.168.203.155@tcp [ 3039.972720] Key type .llcrypt unregistered [ 3039.974687] Key type ._llcrypt unregistered [ 3057.521697] Key type ._llcrypt registered [ 3057.525776] Key type .llcrypt registered [ 3057.616232] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3058.982571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3059.019255] alg: No test for adler32 (adler32-zlib) [ 3060.180430] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 3060.443651] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3062.135243] Key type lgssc registered [ 3063.176906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3069.177837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3069.182451] LustreError: 103831:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3069.193938] LustreError: 103835:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3069.204620] LustreError: 103831:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 3069.216595] LustreError: 103831:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3069.229611] LustreError: 103831:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3069.252322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3070.787414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3079.521406] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.767609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3084.583897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3090.752326] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lsmod [ 3093.381938] LustreError: 105036:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3093.402877] LustreError: 105042:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3093.411340] LustreError: 105036:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4978 [ 3093.419842] LustreError: 105036:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3093.434527] LustreError: 105036:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3093.449687] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3093.903107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3095.995609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3100.136796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3100.155603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.178261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.161872] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.163839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.191781] Lustre: Skipped 1 previous similar message [ 3101.198215] Lustre: Skipped 1 previous similar message [ 3104.291966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3104.312474] Lustre: Skipped 3 previous similar messages [ 3109.352074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3109.365755] Lustre: Skipped 2 previous similar messages [ 3111.391178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3111.715285] Lustre: server umount lustre-MDT0000 complete [ 3114.466516] LustreError: 103873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3114.491172] LustreError: 103873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3115.544270] LustreError: 105082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140438 with bad export cookie 25594455657568654 [ 3115.551800] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3115.564214] LustreError: 105082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3116.639376] LustreError: 104305:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000001e012ba5 x1869758210268544/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3116.917492] Lustre: server umount lustre-MDT0001 complete [ 3130.987063] Lustre: server umount lustre-OST0000 complete [ 3139.581613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.128314] LustreError: 106107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3140.195178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3144.581263] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3145.189553] LustreError: 106108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.308350] LustreError: 106107:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.432646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.857504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3158.292889] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3166.166343] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3171.679305] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3178.466415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3178.772801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3185.363852] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3185.986803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3192.434861] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3206.117105] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3206.124306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.139932] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3208.680372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3208.680536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3211.413744] Lustre: server umount lustre-OST0000 complete [ 3215.333590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3215.343725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3215.364164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3215.371027] Lustre: Skipped 2 previous similar messages [ 3221.225136] Lustre: server umount lustre-MDT0000 complete [ 3224.034753] LustreError: 108313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3224.050202] LustreError: 108313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3224.741080] LustreError: 106087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140547 with bad export cookie 25594455657569599 [ 3224.764493] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3225.092159] Lustre: server umount lustre-MDT0001 complete [ 3234.986976] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3237.650471] Key type lgssc unregistered [ 3237.951829] LNet: 109659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3237.966420] LNetError: 109659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3239.028124] LNet: Removed LNI 192.168.203.155@tcp [ 3239.884959] Key type .llcrypt unregistered [ 3239.892373] Key type ._llcrypt unregistered [ 3260.231297] Key type ._llcrypt registered [ 3260.233372] Key type .llcrypt registered [ 3260.324692] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3261.520416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3261.553439] alg: No test for adler32 (adler32-zlib) [ 3262.516157] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 3262.798562] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3264.511249] Key type lgssc registered [ 3265.759752] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3272.924711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3272.938523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3274.389486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3278.538089] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3284.963716] LustreError: 111057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.977932] LustreError: 111057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3285.935438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3286.218528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3289.996473] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3296.716142] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3302.239474] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3308.494456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3308.865253] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3314.265477] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3319.086205] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3320.100975] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3324.147978] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3326.527913] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3329.505359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3329.515706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3329.526194] Lustre: Skipped 1 previous similar message [ 3329.528622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3329.532354] Lustre: Skipped 1 previous similar message [ 3331.483887] Lustre: server umount lustre-OST0000 complete [ 3334.624785] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.626818] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3334.639318] Lustre: Skipped 1 previous similar message [ 3339.751075] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3339.761588] Lustre: Skipped 2 previous similar messages [ 3340.714786] Lustre: server umount lustre-MDT0000 complete [ 3343.554691] LustreError: 113642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140666 with bad export cookie 10243053129108414074 [ 3343.558944] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3343.570914] LustreError: 113642:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3343.812007] Lustre: server umount lustre-MDT0001 complete [ 3351.808604] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3353.946384] Key type lgssc unregistered [ 3354.186715] LNet: 114696:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3354.197971] LNetError: 114696:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3355.237594] LNet: Removed LNI 192.168.203.155@tcp [ 3355.820239] Key type .llcrypt unregistered [ 3355.823382] Key type ._llcrypt unregistered [ 3368.230249] Key type ._llcrypt registered [ 3368.232641] Key type .llcrypt registered [ 3368.308350] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3368.981347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3369.008110] alg: No test for adler32 (adler32-zlib) [ 3369.995557] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 3370.174219] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3371.791281] Key type lgssc registered [ 3372.561695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3377.980434] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3378.009939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3379.329169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3382.843422] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3389.571928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3389.795393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3393.288825] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3399.033484] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3402.986090] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3407.694828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3407.945640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.785571] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3418.095859] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3425.394216] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3436.005969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3436.006137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3436.020856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3436.033133] Lustre: Skipped 1 previous similar message [ 3439.013061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3439.019730] Lustre: Skipped 2 previous similar messages [ 3441.551132] Lustre: server umount lustre-OST0000 complete [ 3446.241735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3446.248414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3446.258087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.316423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3449.323601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3449.337276] Lustre: Skipped 1 previous similar message [ 3454.434560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3454.439129] Lustre: Skipped 1 previous similar message [ 3459.039984] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3459.177883] Lustre: server umount lustre-MDT0000 complete [ 3459.552655] LustreError: 116700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3459.567779] LustreError: 116700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3462.069535] LustreError: 115877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140785 with bad export cookie 15395317643373494435 [ 3462.073311] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3462.083219] LustreError: 115877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3462.358974] Lustre: server umount lustre-MDT0001 complete [ 3470.618881] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3472.851254] Key type lgssc unregistered [ 3473.107102] LNet: 119452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3473.112743] LNetError: 119452:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3474.148831] LNet: Removed LNI 192.168.203.155@tcp [ 3474.789762] Key type .llcrypt unregistered [ 3474.792907] Key type ._llcrypt unregistered [ 3491.976905] Key type ._llcrypt registered [ 3491.979357] Key type .llcrypt registered [ 3492.062557] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3492.935574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3492.975728] alg: No test for adler32 (adler32-zlib) [ 3493.933887] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 3494.108750] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3495.759217] Key type lgssc registered [ 3496.681839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3502.811510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3502.823668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3504.119443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3507.239069] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3513.705742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3513.938394] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3517.131096] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3522.721156] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3526.566437] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3531.695445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3532.031464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3536.801959] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3542.323867] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3550.450755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3558.514720] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3561.264494] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3561.269343] Lustre: Skipped 2 previous similar messages [ 3564.280542] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3564.288871] Lustre: Skipped 2 previous similar messages [ 3568.175676] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3568.198159] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3573.707454] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3573.718923] Lustre: Skipped 2 previous similar messages [ 3575.588091] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp [ 3579.953445] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.55/24@tcp in log lustre-MDT0000 [ 3579.960396] Lustre: Skipped 2 previous similar messages [ 3581.272227] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3581.285170] Lustre: Skipped 1 previous similar message [ 3589.601130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3589.606505] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3589.615159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3591.144902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3591.144917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3591.156525] LustreError: Skipped 1 previous similar message [ 3591.165951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.769278] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3596.780332] Lustre: Skipped 2 previous similar messages [ 3601.892578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3601.904160] Lustre: Skipped 1 previous similar message [ 3602.399273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3602.517442] Lustre: server umount lustre-OST0000 complete [ 3605.984753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3605.995303] LustreError: Skipped 1 previous similar message [ 3606.001263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.010837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3617.249628] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3617.254346] Lustre: Skipped 5 previous similar messages [ 3619.295235] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3619.455693] Lustre: server umount lustre-MDT0000 complete [ 3622.091205] LustreError: 120828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783140945 with bad export cookie 5978789025221122324 [ 3622.098241] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3622.102159] LustreError: 120828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3622.293705] Lustre: server umount lustre-MDT0001 complete [ 3629.441802] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3631.174562] Key type lgssc unregistered [ 3631.396447] LNet: 125660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3631.403335] LNetError: 125660:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3631.417909] LNet: Removed LNI 192.168.203.155@tcp [ 3632.008435] Key type .llcrypt unregistered [ 3632.013713] Key type ._llcrypt unregistered [ 3649.492782] Key type ._llcrypt registered [ 3649.495310] Key type .llcrypt registered [ 3649.580768] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3650.138540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3650.243111] alg: No test for adler32 (adler32-zlib) [ 3651.198778] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 3651.367073] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3652.991162] Key type lgssc registered [ 3653.846267] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3658.579358] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3658.590897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3659.869733] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3659.883559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.893535] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.55/24@tcp [ 3662.479691] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3667.929164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3668.125657] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3668.155164] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3668.202129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3670.928651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3675.914994] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3679.634189] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3684.140305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3684.416398] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3685.420622] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 3688.495716] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3693.386536] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3700.705120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3700.710905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3700.723907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3703.781973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3703.783243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3703.794613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3703.815273] Lustre: Skipped 1 previous similar message [ 3706.885987] Lustre: server umount lustre-OST0000 complete [ 3714.016557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3714.030719] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3714.036804] Lustre: Skipped 1 previous similar message [ 3715.816338] Lustre: server umount lustre-MDT0000 complete [ 3717.983206] LustreError: 127179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141041 with bad export cookie 8770806493855521537 [ 3717.985994] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3717.992115] LustreError: 127179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3718.156837] Lustre: server umount lustre-MDT0001 complete [ 3724.318951] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3726.037637] Key type lgssc unregistered [ 3726.251456] LNet: 130781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3726.258482] LNetError: 130781:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3726.269127] LNet: Removed LNI 192.168.203.155@tcp [ 3726.766946] Key type .llcrypt unregistered [ 3726.770707] Key type ._llcrypt unregistered [ 3742.864584] Key type ._llcrypt registered [ 3742.866257] Key type .llcrypt registered [ 3742.930367] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3752.104591] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3752.682854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3752.702294] alg: No test for adler32 (adler32-zlib) [ 3753.689033] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 3753.876572] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3755.553562] Key type lgssc registered [ 3756.399276] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3777.984755] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3778.002664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3779.146293] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3779.167573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3779.212939] Lustre: lustre-MDT0000: new disk, initializing [ 3779.255513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3779.266689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3781.676813] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3788.183946] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3788.263923] Lustre: 134157:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3788.307791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3788.312148] Lustre: Skipped 1 previous similar message [ 3788.365768] Lustre: lustre-MDT0001: new disk, initializing [ 3788.424113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3788.448579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3788.455877] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3790.976424] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3796.545263] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3799.921223] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3804.322511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3804.531319] Lustre: lustre-OST0000: new disk, initializing [ 3804.534428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3804.542806] Lustre: 136123:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3804.601815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3808.274936] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3812.374255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3812.381058] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3812.448666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3813.866381] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3816.483616] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3816.602588] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3818.875224] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3819.016245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3822.560616] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3822.566852] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3822.587064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.683803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3827.686906] Lustre: Skipped 2 previous similar messages [ 3832.804766] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3832.809160] Lustre: Skipped 1 previous similar message [ 3834.335130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3834.451114] Lustre: server umount lustre-OST0000 complete [ 3837.921139] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3837.922345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3837.931490] Lustre: Skipped 2 previous similar messages [ 3837.939577] Lustre: Skipped 1 previous similar message [ 3842.720185] Lustre: server umount lustre-MDT0000 complete [ 3844.776092] LustreError: 134150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141167 with bad export cookie 5681349711493162789 [ 3844.777209] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3844.784372] LustreError: 134150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3844.980641] Lustre: server umount lustre-MDT0001 complete [ 3849.341238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3849.573974] LustreError: 138340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3849.586467] LustreError: 138340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3849.624917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3851.830945] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3854.818020] LustreError: 138341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3856.342255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3856.579189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3858.916872] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3863.397666] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3866.677664] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3870.699834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3870.892919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3874.494042] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3878.495751] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3881.015370] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3881.159783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3883.650434] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3883.807830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3889.774759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3889.877351] Lustre: lustre-OST0001: new disk, initializing [ 3889.879846] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3889.882996] Lustre: 141709:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3889.946353] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3893.017055] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3897.395683] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3897.868444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3897.875269] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3897.916742] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3898.571160] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 3900.620520] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3900.728966] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3903.375669] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88c802da6000.ost_server_uuid 50 [ 3904.165514] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88c802da6000.ost_server_uuid in FULL state after 0 sec [ 3913.186748] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3913.189654] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3913.191385] LustreError: Skipped 2 previous similar messages [ 3913.192412] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3913.210901] Lustre: Skipped 1 previous similar message [ 3914.251554] Lustre: server umount lustre-OST0001 complete [ 3917.280747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3917.287397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3918.310209] LustreError: 140160:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3918.329167] LustreError: 140160:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3922.609811] Lustre: server umount lustre-OST0000 complete [ 3928.545237] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3928.546797] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3928.564405] Lustre: Skipped 1 previous similar message [ 3928.569661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3928.575532] Lustre: Skipped 5 previous similar messages [ 3930.871708] Lustre: server umount lustre-MDT0000 complete [ 3932.988371] LustreError: 138323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141256 with bad export cookie 5681349711493163671 [ 3932.989355] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3932.993642] LustreError: 138323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3933.155353] Lustre: server umount lustre-MDT0001 complete [ 3942.238546] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3943.671413] Key type lgssc unregistered [ 3943.827527] LNet: 144725:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3943.835264] LNetError: 144725:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3943.846923] LNet: Removed LNI 192.168.203.155@tcp [ 3944.243546] Key type .llcrypt unregistered [ 3944.245540] Key type ._llcrypt unregistered [ 3962.219929] Key type ._llcrypt registered [ 3962.221433] Key type .llcrypt registered [ 3962.279057] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3969.668647] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3970.253991] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3970.271917] alg: No test for adler32 (adler32-zlib) [ 3971.205797] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 3971.331212] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3972.927218] Key type lgssc registered [ 3973.478163] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3990.147983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3990.157572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3991.264488] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3991.279816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3991.319659] Lustre: lustre-MDT0000: new disk, initializing [ 3991.358198] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3991.368031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3993.229337] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3998.990728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.053887] Lustre: 148850:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3999.088262] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3999.090859] Lustre: Skipped 1 previous similar message [ 3999.137764] Lustre: lustre-MDT0001: new disk, initializing [ 3999.187555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3999.212727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3999.223825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4001.311098] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4006.182170] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4008.787428] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4011.980528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4012.118831] Lustre: lustre-OST0000: new disk, initializing [ 4012.121556] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4012.124895] Lustre: 150820:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4012.172228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4014.998980] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4019.820206] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4020.232802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4020.239351] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4020.287886] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4027.555791] Lustre: Failing over lustre-OST0000 [ 4027.654712] Lustre: server umount lustre-OST0000 complete [ 4041.701924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.880439] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4041.890477] Lustre: Skipped 1 previous similar message [ 4041.893416] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4041.909475] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4044.912253] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4045.983114] Lustre: 147077:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783141353/real 1783141353] req@0000000088ca631c x1869759165527040/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783141369 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4046.741539] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4046.799186] Lustre: Failing over lustre-MDT0000 [ 4046.919417] Lustre: server umount lustre-MDT0000 complete [ 4047.330185] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.335988] Lustre: Skipped 2 previous similar messages [ 4051.423215] Lustre: 147078:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783141358/real 1783141358] req@00000000108b560c x1869759165528576/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783141374 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4051.442161] Lustre: 147078:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4052.452079] LustreError: 149745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4052.463397] LustreError: 149745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 4056.609859] Lustre: 147078:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783141363/real 1783141363] req@00000000f1a72163 x1869759165529600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1783141379 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4056.967897] LustreError: 148859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4061.256702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4061.329405] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4061.337781] Lustre: 148855:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4061.443356] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4061.470765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4061.509588] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4062.089859] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4063.607781] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4066.791569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4066.815374] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4066.822542] Lustre: Skipped 1 previous similar message [ 4066.844219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4071.906656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4071.907193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4071.911977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4071.929298] Lustre: Skipped 1 previous similar message [ 4074.849088] Lustre: server umount lustre-OST0000 complete [ 4077.026620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4077.028765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4077.039390] Lustre: Skipped 1 previous similar message [ 4077.045134] Lustre: Skipped 1 previous similar message [ 4082.144802] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4082.150093] Lustre: Skipped 2 previous similar messages [ 4082.858110] Lustre: server umount lustre-MDT0000 complete [ 4084.908065] LustreError: 148844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141407 with bad export cookie 6325468049024973249 [ 4084.910620] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4084.914610] LustreError: 148844:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4085.071630] Lustre: server umount lustre-MDT0001 complete [ 4090.508864] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4091.981294] Key type lgssc unregistered [ 4092.141389] LNet: 154467:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4092.145707] LNetError: 154467:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4092.162169] LNet: Removed LNI 192.168.203.155@tcp [ 4092.659445] Key type .llcrypt unregistered [ 4092.661269] Key type ._llcrypt unregistered [ 4118.106310] Key type ._llcrypt registered [ 4118.108500] Key type .llcrypt registered [ 4118.167897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4118.769426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4118.804096] alg: No test for adler32 (adler32-zlib) [ 4119.726098] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 4119.881145] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4121.487708] Key type lgssc registered [ 4122.187461] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4126.660648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4126.674966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4127.930369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.175449] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4135.405419] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4137.441889] LustreError: 155983:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.451485] LustreError: 155983:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4138.422867] LustreError: 156011:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4138.432106] LustreError: 156011:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4138.437216] LustreError: 156011:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 11, retries 0, failed: rc = -5 [ 4138.656754] Lustre: server umount lustre-MDT0000 complete [ 4154.637625] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4167.678585] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4213.404804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4213.695204] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4213.726329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4213.823258] Lustre: lustre-MDT0000: new disk, initializing [ 4213.951429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4213.976060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4219.511879] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4232.928708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4233.125834] Lustre: 159320:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4233.177421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4233.180674] Lustre: Skipped 1 previous similar message [ 4233.255882] Lustre: lustre-MDT0001: new disk, initializing [ 4233.401206] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4233.446092] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4233.456068] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4239.429894] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4250.856145] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4256.697237] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4264.892064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4265.345898] Lustre: lustre-OST0000: new disk, initializing [ 4265.353923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4265.372214] Lustre: 161287:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4265.524850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4271.226316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4271.244831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4271.362392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4272.954845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4283.125934] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4286.748422] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4288.669789] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 01:06:51 (1783141611) [ 4519.906116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4519.917219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.940819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4522.471036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4522.473025] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4522.483733] Lustre: Skipped 1 previous similar message [ 4522.503312] Lustre: Skipped 1 previous similar message [ 4525.414377] Lustre: server umount lustre-MDT0000 complete [ 4527.586534] LustreError: 161693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4527.604400] LustreError: 161693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4528.770440] LustreError: 159311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783141851 with bad export cookie 7800912884963563529 [ 4528.776219] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4528.786470] LustreError: 159311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4532.704785] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4532.707253] LustreError: 159327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.723653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4532.723665] Lustre: Skipped 1 previous similar message [ 4535.135744] Lustre: server umount lustre-MDT0001 complete [ 4553.183413] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4553.395377] Lustre: server umount lustre-OST0000 complete [ 4561.613467] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4570.466319] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4604.605158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4604.901200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4604.931324] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4605.043121] Lustre: lustre-MDT0000: new disk, initializing [ 4605.139340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4605.163924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4609.286777] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4619.583772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4619.715467] Lustre: 165941:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4619.815080] LustreError: 165954:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4619.820796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4619.825680] LustreError: 165954:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4619.834280] Lustre: Skipped 1 previous similar message [ 4619.931639] Lustre: lustre-MDT0001: new disk, initializing [ 4620.032566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4620.077143] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4620.088412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4623.647486] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4633.259536] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4638.686542] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4644.374379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4644.597446] Lustre: lustre-OST0000: new disk, initializing [ 4644.600559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4644.606978] Lustre: 167912:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4644.674322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.885021] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4650.534627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4650.546476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4650.662356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4658.111678] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4662.873139] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4663.078284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4667.080794] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4667.233639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4670.952362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4670.960843] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4670.980813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4670.983921] Lustre: Skipped 1 previous similar message [ 4674.924582] Lustre: server umount lustre-OST0000 complete [ 4681.702361] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4681.705619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4681.710586] Lustre: Skipped 2 previous similar messages [ 4681.710974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4681.710978] LustreError: Skipped 1 previous similar message [ 4681.725296] Lustre: Skipped 3 previous similar messages [ 4684.512852] Lustre: server umount lustre-MDT0000 complete [ 4686.824941] LustreError: 168573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4686.851073] LustreError: 168573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4687.710659] LustreError: 167911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142010 with bad export cookie 7800912884963722366 [ 4687.714889] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4687.731543] LustreError: 167911:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4688.016753] Lustre: server umount lustre-MDT0001 complete [ 4696.243041] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:13:38 (1783142018) [ 4712.644508] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4720.990449] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4755.958955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4756.259161] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4756.292646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4756.372445] Lustre: lustre-MDT0000: new disk, initializing [ 4756.463383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4756.482303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4761.020348] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4771.200068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4771.310315] Lustre: 172666:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4771.565134] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4775.549643] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4783.404442] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4788.880765] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4795.395969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4795.638892] Lustre: lustre-OST0000: new disk, initializing [ 4795.642340] Lustre: Skipped 1 previous similar message [ 4795.646136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4795.651373] Lustre: Skipped 2 previous similar messages [ 4795.659345] Lustre: 174645:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4801.962133] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4804.172280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4804.184540] Lustre: Skipped 1 previous similar message [ 4804.198234] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4804.306975] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4810.873678] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4819.121544] Lustre: server umount lustre-OST0000 complete [ 4825.058898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4825.071816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4825.082881] Lustre: Skipped 1 previous similar message [ 4833.358278] Lustre: server umount lustre-MDT0000 complete [ 4835.232072] Lustre: 155297:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783142142/real 1783142142] req@000000006a584c3f x1869759321452544/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1783142158 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4835.291280] LustreError: 172674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.315515] LustreError: 172674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4837.000720] LustreError: 172660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142160 with bad export cookie 7800912884963723248 [ 4837.005961] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4837.017610] LustreError: 172660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4837.426880] Lustre: server umount lustre-MDT0001 complete [ 4846.504190] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 01:16:08 (1783142168) [ 4863.040186] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4872.323479] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4906.171296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4906.424292] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4906.460376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4906.594299] Lustre: lustre-MDT0000: new disk, initializing [ 4906.739323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4906.742418] Lustre: Skipped 2 previous similar messages [ 4906.797303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4911.226938] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4922.715037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4922.837626] Lustre: 178980:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4923.115276] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4927.181503] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4935.453788] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4941.378487] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4947.762128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4948.008345] Lustre: 180959:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4954.052893] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4955.177756] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4955.317571] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4963.207353] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4970.468603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4970.475130] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4970.486286] Lustre: Skipped 2 previous similar messages [ 4970.490439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4970.495341] Lustre: Skipped 2 previous similar messages [ 4984.287142] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4984.460906] Lustre: server umount lustre-OST0000 complete [ 4989.921215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4993.937810] Lustre: server umount lustre-MDT0000 complete [ 4996.577487] LustreError: 178992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4996.594658] LustreError: 178992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4997.117538] LustreError: 178973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142320 with bad export cookie 7800912884963724333 [ 4997.121103] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4997.135068] LustreError: 178973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4997.447958] Lustre: server umount lustre-MDT0001 complete [ 5006.972130] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5009.502644] Key type lgssc unregistered [ 5009.808792] LNet: 182816:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5009.820378] LNetError: 182816:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5009.845617] LNet: Removed LNI 192.168.203.155@tcp [ 5010.704332] Key type .llcrypt unregistered [ 5010.707745] Key type ._llcrypt unregistered [ 5032.206327] Key type ._llcrypt registered [ 5032.210846] Key type .llcrypt registered [ 5032.370245] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5034.078325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5034.108204] alg: No test for adler32 (adler32-zlib) [ 5035.159688] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 5035.360966] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5037.021708] Key type lgssc registered [ 5038.264349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5045.834594] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5045.876915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5047.334032] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5051.936134] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5057.509640] LustreError: 184233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5057.518706] LustreError: 184233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5059.103864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5059.487971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5063.204505] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5069.271417] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5074.043596] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5079.787903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5080.128769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5085.429873] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5092.402857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5092.667332] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5109.728491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5109.734224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.757668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5112.805148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5112.808764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5112.819055] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5115.096296] Lustre: server umount lustre-OST0000 complete [ 5120.992432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5120.999811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5121.018362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5121.027804] Lustre: Skipped 1 previous similar message [ 5123.042838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5123.043381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5124.397246] Lustre: server umount lustre-MDT0000 complete [ 5127.610735] LustreError: 184213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142450 with bad export cookie 14139033998499950413 [ 5127.611391] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5127.622071] LustreError: 184213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5127.904468] Lustre: server umount lustre-MDT0001 complete [ 5135.679858] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5137.663211] Key type lgssc unregistered [ 5137.894023] LNet: 187787:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5137.901270] LNetError: 187787:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5137.911133] LNet: Removed LNI 192.168.203.155@tcp [ 5138.538252] Key type .llcrypt unregistered [ 5138.544180] Key type ._llcrypt unregistered [ 5157.369491] Key type ._llcrypt registered [ 5157.372942] Key type .llcrypt registered [ 5157.464220] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5158.464513] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5158.492256] alg: No test for adler32 (adler32-zlib) [ 5159.594847] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 5159.861273] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5161.536867] Key type lgssc registered [ 5162.499994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5168.883632] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5168.900480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5170.264865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5173.902770] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5180.388155] LustreError: 189182:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5180.404502] LustreError: 189182:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5180.608364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5180.934397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5184.429768] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5190.503382] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5195.204993] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5200.754278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5200.969696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5206.196599] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5213.129550] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5215.349254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5223.861714] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 5225.958346] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5225.958695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5225.964062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5225.970305] LustreError: Skipped 1 previous similar message [ 5226.004138] Lustre: Skipped 1 previous similar message [ 5228.383124] Lustre: server umount lustre-OST0000 complete [ 5234.310865] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff88c807b53000.ost_server_uuid 50 [ 5235.711155] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5251.555084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5251.583089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5251.591427] Lustre: Skipped 2 previous similar messages [ 5255.111811] Lustre: server umount lustre-MDT0000 complete [ 5256.673131] LustreError: 191573:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5256.697955] LustreError: 191573:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5258.237883] LustreError: 189162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142581 with bad export cookie 14874632708057979394 [ 5258.241715] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5258.251860] LustreError: 189162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5258.580551] Lustre: server umount lustre-MDT0001 complete [ 5265.836596] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 01:23:08 (1783142588) [ 5270.761728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5271.106331] LustreError: 193099:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5271.169839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5274.545867] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5276.640646] LustreError: 193100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5280.290822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5280.559962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5283.539917] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5288.668804] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5292.984389] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5298.725472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5299.042475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5304.341769] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5309.769198] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5315.133787] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5315.286761] Lustre: lustre-OST0001: new disk, initializing [ 5315.290669] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5315.305357] Lustre: 195938:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5315.423529] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5316.471761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5319.844664] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5321.766711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5321.772388] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5321.828918] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5326.385343] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5330.913544] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5330.920418] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5330.930600] Lustre: Skipped 1 previous similar message [ 5330.934520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5331.939864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5334.389850] Lustre: server umount lustre-OST0000 complete [ 5336.973939] LustreError: 194918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5338.566831] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5338.774565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5342.445463] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5342.685531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5347.213635] LustreError: 194917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5347.237728] LustreError: 194917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5357.542369] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5357.555132] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.555142] Lustre: Skipped 1 previous similar message [ 5357.555790] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5357.571896] LustreError: Skipped 1 previous similar message [ 5357.602502] Lustre: Skipped 4 previous similar messages [ 5361.499863] Lustre: server umount lustre-OST0001 complete [ 5367.776858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5367.779441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5367.780341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5367.801489] Lustre: Skipped 2 previous similar messages [ 5370.506289] Lustre: server umount lustre-MDT0000 complete [ 5372.900620] LustreError: 193100:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5372.915699] LustreError: 193100:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 5373.083859] LustreError: 195937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142696 with bad export cookie 14874632708057980465 [ 5373.089422] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5373.096975] LustreError: 195937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5373.353195] Lustre: server umount lustre-MDT0001 complete [ 5390.213296] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:25:13 (1783142713) [ 5394.897865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5395.099604] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5395.123425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5395.261948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5398.796403] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5405.092873] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5405.196254] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5405.231862] Lustre: 199376:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5408.631650] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5414.107548] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5418.449558] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5424.084578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5424.230367] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5428.828162] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5429.821256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5434.074031] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5438.822586] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5438.871177] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5438.937575] LustreError: 201675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5443.869571] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5449.590497] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5451.755586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.770568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5451.771897] Lustre: Skipped 1 previous similar message [ 5451.786867] Lustre: Skipped 2 previous similar messages [ 5457.796862] Lustre: server umount lustre-OST0000 complete [ 5472.229593] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5472.232595] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5472.241310] Lustre: Skipped 1 previous similar message [ 5477.712794] Lustre: server umount lustre-OST0001 complete [ 5487.585602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5487.591741] Lustre: Skipped 10 previous similar messages [ 5494.239169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5494.454350] Lustre: server umount lustre-MDT0000 complete [ 5497.343171] LustreError: 199366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142820 with bad export cookie 14874632708057981865 [ 5497.346652] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5497.354244] LustreError: 199366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5497.633896] Lustre: server umount lustre-MDT0001 complete [ 5515.069414] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:27:17 (1783142837) [ 5525.855590] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5532.888166] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5557.135828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5557.283861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5557.307081] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5557.361651] Lustre: lustre-MDT0000: new disk, initializing [ 5557.405785] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5557.409098] Lustre: Skipped 3 previous similar messages [ 5557.418747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5560.223828] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5567.280366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5567.345853] Lustre: 207465:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5567.387521] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5567.392287] Lustre: Skipped 1 previous similar message [ 5567.444704] Lustre: lustre-MDT0001: new disk, initializing [ 5567.525699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5567.530976] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5570.183948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5576.331209] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5579.548870] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5583.454690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5583.598877] Lustre: lustre-OST0000: new disk, initializing [ 5583.602990] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5583.608521] Lustre: 209431:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5587.293265] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5593.113372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5593.118888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5593.213970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5593.775602] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5596.747225] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5596.900195] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5599.644659] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5599.798978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5603.301741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5603.302642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5603.309358] Lustre: Skipped 2 previous similar messages [ 5603.310997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5603.313967] LustreError: Skipped 1 previous similar message [ 5603.324069] Lustre: Skipped 3 previous similar messages [ 5607.415616] Lustre: server umount lustre-OST0000 complete [ 5615.701063] Lustre: server umount lustre-MDT0000 complete [ 5618.024548] LustreError: 207458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783142941 with bad export cookie 14874632708057983426 [ 5618.029263] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5618.031021] LustreError: 207458:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5618.243893] Lustre: server umount lustre-MDT0001 complete [ 5623.448242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5623.698164] LustreError: 211646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5623.712782] LustreError: 211646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5626.296340] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5631.622110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5634.766952] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5639.808332] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5643.415431] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5647.854895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5651.795367] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5656.491277] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5659.209072] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5659.362575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5662.123721] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5662.247809] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5664.737429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5664.746285] LustreError: Skipped 1 previous similar message [ 5669.512670] Lustre: server umount lustre-OST0000 complete [ 5673.633944] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5676.592748] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5701.979068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5702.211160] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5702.215934] Lustre: Skipped 5 previous similar messages [ 5705.885821] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5706.342338] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5706.350653] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5710.114913] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5716.583081] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5716.594858] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5723.943903] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 12 sec [ 5727.203076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.213758] Lustre: Skipped 6 previous similar messages [ 5732.331719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5732.336494] Lustre: Skipped 7 previous similar messages [ 5732.416865] Lustre: server umount lustre-OST0000 complete [ 5749.215134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5749.317773] Lustre: server umount lustre-MDT0000 complete [ 5751.611415] LustreError: 214755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143074 with bad export cookie 14874632708057984308 [ 5751.614348] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5751.621404] LustreError: 214755:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5751.832920] Lustre: server umount lustre-MDT0001 complete [ 5756.932596] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:31:20 (1783143080) [ 5760.451838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5760.745041] LustreError: 217630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5760.756126] LustreError: 217630:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5763.531163] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5768.251275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5770.586862] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5774.317395] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5777.192542] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5780.806887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5784.098106] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5787.801567] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5790.123397] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5790.250408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5792.551496] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5793.685306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 1 sec [ 5796.970290] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5797.071911] Lustre: lustre-OST0001: new disk, initializing [ 5797.074825] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5797.083319] Lustre: 221004:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5799.146505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5799.153694] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5799.216621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5800.201598] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5805.220726] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5807.490946] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5807.587914] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5809.593077] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5809.712159] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5814.760936] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5814.768132] LustreError: Skipped 4 previous similar messages [ 5816.742619] Lustre: server umount lustre-OST0001 complete [ 5820.080558] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5820.217459] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5822.574861] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5822.695700] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5848.136518] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5851.299341] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5853.670791] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5853.679766] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5853.682546] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5855.182350] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5856.097497] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5858.784626] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.805512] Lustre: Skipped 5 previous similar messages [ 5863.311680] Lustre: server umount lustre-OST0001 complete [ 5883.102556] Lustre: server umount lustre-OST0000 complete [ 5891.526673] Lustre: server umount lustre-MDT0000 complete [ 5893.637743] LustreError: 221003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143216 with bad export cookie 14874632708057985267 [ 5893.640391] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5893.644638] LustreError: 221003:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5899.850900] Lustre: server umount lustre-MDT0001 complete [ 5912.323155] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:33:55 (1783143235) [ 5915.466647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5915.611719] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5915.627166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5917.545585] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5921.544558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.606989] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5921.627439] Lustre: 226239:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5923.785920] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5927.234754] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5929.874324] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5933.280702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5933.408109] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5936.588645] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5940.503862] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5945.783422] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5945.822378] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5948.720478] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5952.266764] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5952.999924] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5955.124276] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5955.239291] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5957.413533] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5957.534129] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5960.141054] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff88c809850800.ost_server_uuid 50 [ 5960.820593] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff88c809850800.ost_server_uuid in FULL state after 0 sec [ 5961.358521] Lustre: Permanently deactivating lustre-OST0001 [ 5961.366136] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5962.949154] Lustre: Permanently reactivating lustre-OST0001 [ 5969.376483] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5969.380313] LustreError: Skipped 4 previous similar messages [ 5970.917303] Lustre: server umount lustre-OST0001 complete [ 5987.295211] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5987.366888] Lustre: server umount lustre-OST0000 complete [ 5989.859601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5989.863369] Lustre: Skipped 26 previous similar messages [ 5995.410568] Lustre: server umount lustre-MDT0000 complete [ 5997.117746] LustreError: 226232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143320 with bad export cookie 14874632708057986548 [ 5997.123523] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5997.131269] LustreError: 226232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5997.312727] Lustre: server umount lustre-MDT0001 complete [ 6009.053912] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:35:32 (1783143332) [ 6012.880861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.026912] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6013.044507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6013.049467] Lustre: Skipped 5 previous similar messages [ 6013.139117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6013.142874] Lustre: Skipped 9 previous similar messages [ 6015.018732] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6019.016014] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.079062] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6019.094432] Lustre: 232926:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6021.167615] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6024.514356] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6026.943588] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6029.953344] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6030.064233] Lustre: Permanently deactivating lustre-OST0000 [ 6030.070054] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6031.866759] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6032.833759] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6036.014269] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6038.988756] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6041.281292] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6041.851606] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6045.343709] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6046.223235] Lustre: Permanently reactivating lustre-OST0000 [ 6046.230625] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6046.234413] Lustre: Skipped 2 previous similar messages [ 6048.509557] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6048.514167] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.519848] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6048.524923] Lustre: Skipped 1 previous similar message [ 6048.539059] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6048.551301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6063.190695] Lustre: server umount lustre-OST0001 complete [ 6067.172611] LustreError: 234770:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6067.180226] LustreError: 234770:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 35 previous similar messages [ 6071.133055] Lustre: server umount lustre-OST0000 complete [ 6087.135116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6087.213889] Lustre: server umount lustre-MDT0000 complete [ 6088.870610] LustreError: 232918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143411 with bad export cookie 14874632708057988179 [ 6088.873029] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6088.876069] LustreError: 232918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6089.017670] Lustre: server umount lustre-MDT0001 complete [ 6093.239776] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6094.439154] Key type lgssc unregistered [ 6094.600498] LNet: 237779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6094.603865] LNetError: 237779:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6094.613532] LNet: Removed LNI 192.168.203.155@tcp [ 6094.965624] Key type .llcrypt unregistered [ 6094.967419] Key type ._llcrypt unregistered [ 6105.657558] Key type ._llcrypt registered [ 6105.659014] Key type .llcrypt registered [ 6105.713842] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6106.261370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6106.269115] alg: No test for adler32 (adler32-zlib) [ 6107.171145] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 6107.299493] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6108.903528] Key type lgssc registered [ 6109.413574] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6114.404978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6114.414986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6115.700930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6117.975123] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6122.911555] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6122.973725] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6122.990935] Lustre: 239212:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6123.001812] Lustre: Permanently deactivating lustre-MDT0001 [ 6123.019494] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6123.236033] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6123.491561] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6123.494980] LustreError: 239254:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6123.501990] LustreError: 239254:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6123.507610] LustreError: 239254:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 6125.934377] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6129.983718] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6132.783927] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6136.587598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6136.779058] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6139.687821] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6140.900754] LustreError: 241062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6140.905129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6140.912862] LustreError: 241062:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6143.625866] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6146.019836] LustreError: 241062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6147.324029] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6147.432584] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6148.463104] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6148.465386] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6148.466367] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6150.415113] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6154.336602] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6156.337489] Lustre: Permanently reactivating lustre-MDT0001 [ 6156.353314] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6156.358540] Lustre: Skipped 2 previous similar messages [ 6159.965188] Lustre: lustre-MDT0001: Client 7b76312f-f15f-4104-962e-8de13202776f (at 192.168.203.55@tcp) reconnecting [ 6163.918925] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6164.930635] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6168.137218] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6168.277862] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6171.033426] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6171.181246] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6172.723267] Lustre: Permanently deactivating lustre-MDT0001 [ 6172.734449] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6172.739771] Lustre: Skipped 2 previous similar messages [ 6178.319158] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6182.305814] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.317561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6182.320867] Lustre: Skipped 3 previous similar messages [ 6187.492483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6187.500861] Lustre: Skipped 3 previous similar messages [ 6187.773852] Lustre: server umount lustre-MDT0000 complete [ 6190.042413] LustreError: 242766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143513 with bad export cookie 1246113211493137701 [ 6190.045863] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6190.052486] LustreError: 242766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6190.264546] Lustre: server umount lustre-MDT0001 complete [ 6198.618279] Lustre: server umount lustre-OST0000 complete [ 6206.039095] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:38:48 (1783143528) [ 6213.460937] Lustre: server umount lustre-OST0001 complete [ 6218.031084] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 6224.581688] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6245.309642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.459749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6245.466904] Lustre: Skipped 2 previous similar messages [ 6245.487810] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6245.546170] Lustre: lustre-MDT0000: new disk, initializing [ 6245.609119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6245.620311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6247.759783] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6253.757720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6253.822608] Lustre: 247471:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6253.857866] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6253.863976] Lustre: Skipped 1 previous similar message [ 6253.924160] Lustre: lustre-MDT0001: new disk, initializing [ 6253.974572] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6253.999070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6254.005595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6256.095050] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6261.701472] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6264.742764] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6268.369116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6268.571080] Lustre: lustre-OST0000: new disk, initializing [ 6268.574566] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6268.579401] Lustre: 249440:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6272.247661] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6275.605852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6275.611094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6275.667479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6277.645948] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6278.836289] LustreError: 248274:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6284.419909] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6284.555791] Lustre: lustre-OST0001: new disk, initializing [ 6284.558484] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6284.562622] Lustre: 250554:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6284.615394] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6284.619628] Lustre: Skipped 1 previous similar message [ 6287.968414] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6288.935199] LustreError: 248274:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6288.955491] LustreError: 247478:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6293.749341] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6294.052361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6294.059465] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6294.118150] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6294.692831] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6298.975088] LustreError: 247478:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6300.569605] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 6304.227423] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6304.230864] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6304.236909] Lustre: Skipped 3 previous similar messages [ 6304.244305] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6304.246992] Lustre: Skipped 1 previous similar message [ 6306.252869] Lustre: server umount lustre-OST0001 complete [ 6309.349844] LustreError: 249424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6309.357943] LustreError: 249424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6314.474237] LustreError: 250017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6314.484418] LustreError: 250017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6319.587285] LustreError: 249422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6319.588795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.599509] LustreError: 249422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6319.611376] Lustre: Skipped 1 previous similar message [ 6319.617727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6319.621364] Lustre: Skipped 2 previous similar messages [ 6325.683309] Lustre: server umount lustre-OST0000 complete [ 6329.825086] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6329.826923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6329.833713] Lustre: Skipped 2 previous similar messages [ 6329.838921] Lustre: Skipped 3 previous similar messages [ 6334.944250] LustreError: 247481:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.954571] LustreError: 247481:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6335.739811] Lustre: server umount lustre-MDT0000 complete [ 6338.086556] LustreError: 247462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143661 with bad export cookie 1246113211493140508 [ 6338.087526] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6338.091510] LustreError: 247462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6344.360550] Lustre: server umount lustre-MDT0001 complete [ 6349.969147] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6351.416837] Key type lgssc unregistered [ 6351.619807] LNet: 252587:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6351.625390] LNetError: 252587:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6351.636872] LNet: Removed LNI 192.168.203.155@tcp [ 6352.068314] Key type .llcrypt unregistered [ 6352.070631] Key type ._llcrypt unregistered [ 6372.809388] Key type ._llcrypt registered [ 6372.811075] Key type .llcrypt registered [ 6372.870205] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6373.292924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6373.335459] alg: No test for adler32 (adler32-zlib) [ 6374.270976] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 6374.398852] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6376.015157] Key type lgssc registered [ 6376.554728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6380.983874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6380.997987] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6382.140705] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6382.156732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6382.245613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6384.439488] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6389.243960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6389.296352] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6389.316560] Lustre: 254768:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6389.490847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6392.363345] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6396.890377] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6400.392901] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6404.432889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6404.551532] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6404.712708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6408.357323] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6412.639505] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6413.914269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6459.875268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6459.875512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6459.876848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6459.882563] Lustre: Skipped 1 previous similar message [ 6461.559419] Lustre: server umount lustre-OST0000 complete [ 6463.807030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6470.762604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6470.972641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6474.349842] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6478.580801] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6480.167849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.178511] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6480.179130] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6480.192349] Lustre: Skipped 1 previous similar message [ 6516.195068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6516.195319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6516.199524] LustreError: Skipped 1 previous similar message [ 6516.208739] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6516.213983] Lustre: Skipped 2 previous similar messages [ 6520.057329] Lustre: server umount lustre-OST0000 complete [ 6522.848963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6522.854224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6522.860335] Lustre: Skipped 1 previous similar message [ 6522.864921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.432885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6526.433429] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6528.279902] Lustre: server umount lustre-MDT0000 complete [ 6530.306913] LustreError: 254763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143853 with bad export cookie 15304482774517595282 [ 6530.307657] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6530.317471] LustreError: 254763:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6530.515027] Lustre: server umount lustre-MDT0001 complete [ 6536.746603] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6538.448599] Key type lgssc unregistered [ 6538.610696] LNet: 260183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6538.617165] LNetError: 260183:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6538.625676] LNet: Removed LNI 192.168.203.155@tcp [ 6539.122742] Key type .llcrypt unregistered [ 6539.125574] Key type ._llcrypt unregistered [ 6553.370212] Key type ._llcrypt registered [ 6553.371832] Key type .llcrypt registered [ 6553.441359] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6554.070968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6554.088218] alg: No test for adler32 (adler32-zlib) [ 6555.030632] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 6555.177257] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6556.809077] Key type lgssc registered [ 6557.455535] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6561.833531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6561.847370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6563.160384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6565.673962] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6570.371197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.611594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6572.854161] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6577.043795] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6580.131674] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6583.752424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6583.949066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6587.078719] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6588.074094] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6591.106625] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6608.864071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6608.870132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.884592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6610.366256] Lustre: server umount lustre-OST0000 complete [ 6616.033830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6616.035322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6616.047744] Lustre: Skipped 1 previous similar message [ 6616.057956] Lustre: Skipped 1 previous similar message [ 6618.779967] Lustre: server umount lustre-MDT0000 complete [ 6620.840622] LustreError: 261557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783143943 with bad export cookie 10857594331860661636 [ 6620.842618] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6620.851390] LustreError: 261557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6630.438830] Lustre: server umount lustre-MDT0001 complete [ 6635.552975] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6637.064461] Key type lgssc unregistered [ 6637.251754] LNet: 265879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6637.263293] LNetError: 265879:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6637.272930] LNet: Removed LNI 192.168.203.155@tcp [ 6637.741869] Key type .llcrypt unregistered [ 6637.743930] Key type ._llcrypt unregistered [ 6647.560898] Key type ._llcrypt registered [ 6647.562587] Key type .llcrypt registered [ 6647.638153] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6648.191875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6648.199617] alg: No test for adler32 (adler32-zlib) [ 6649.119349] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 6649.251685] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6650.855181] Key type lgssc registered [ 6651.453837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6655.169549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6655.184870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6656.423696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6658.603754] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6663.005931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6663.219881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6665.332986] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6668.934672] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6671.875311] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6675.412277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6675.602749] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6678.888564] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6681.767115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6682.786747] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6694.372252] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6694.374665] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6694.383685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6694.387260] Lustre: Skipped 1 previous similar message [ 6695.547525] Lustre: server umount lustre-OST0000 complete [ 6698.976258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6698.979695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6698.985096] Lustre: Skipped 1 previous similar message [ 6698.987856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6703.799557] Lustre: server umount lustre-MDT0000 complete [ 6704.608257] LustreError: 267080:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6704.618705] LustreError: 267080:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6705.871671] LustreError: 267059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144028 with bad export cookie 17288022932242705965 [ 6705.874762] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6705.879437] LustreError: 267059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6706.076527] Lustre: server umount lustre-MDT0001 complete [ 6711.544412] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6713.097827] Key type lgssc unregistered [ 6713.266780] LNet: 270752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6713.270574] LNetError: 270752:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6714.342366] LNet: Removed LNI 192.168.203.155@tcp [ 6714.899853] Key type .llcrypt unregistered [ 6714.901434] Key type ._llcrypt unregistered [ 6728.230539] Key type ._llcrypt registered [ 6728.231963] Key type .llcrypt registered [ 6728.283701] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6728.674419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6728.721485] alg: No test for adler32 (adler32-zlib) [ 6729.650322] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 6729.793969] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6731.391279] Key type lgssc registered [ 6731.960624] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6736.055213] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6736.065559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6737.276937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6739.337437] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6743.914200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6744.160386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6746.565295] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6750.479212] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6753.538144] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6757.495648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6757.703857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6760.947623] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6762.861400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6764.831116] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6783.459460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6783.468067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6783.476166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6784.486906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6784.487169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6784.489023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6784.495215] Lustre: Skipped 1 previous similar message [ 6784.827163] Lustre: server umount lustre-OST0000 complete [ 6789.600909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6789.601716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6789.607605] Lustre: Skipped 1 previous similar message [ 6789.612831] Lustre: Skipped 1 previous similar message [ 6794.722553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6794.727077] Lustre: Skipped 1 previous similar message [ 6799.840779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6799.846123] Lustre: Skipped 1 previous similar message [ 6801.375130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6801.462277] Lustre: server umount lustre-MDT0000 complete [ 6803.451352] LustreError: 272128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144126 with bad export cookie 9809181348573409488 [ 6803.454129] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6803.457634] LustreError: 272128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6803.621812] Lustre: server umount lustre-MDT0001 complete [ 6808.986892] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6810.402930] Key type lgssc unregistered [ 6810.545473] LNet: 276444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6810.548447] LNetError: 276444:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6810.555377] LNet: Removed LNI 192.168.203.155@tcp [ 6811.051760] Key type .llcrypt unregistered [ 6811.057446] Key type ._llcrypt unregistered [ 6820.317067] Key type ._llcrypt registered [ 6820.318891] Key type .llcrypt registered [ 6820.383296] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6820.924322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6820.942917] alg: No test for adler32 (adler32-zlib) [ 6821.857224] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 6821.997385] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6823.616437] Key type lgssc registered [ 6824.104571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6828.199275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6828.212865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6829.468919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6831.591449] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6835.896817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6836.101865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6838.346411] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6842.180384] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6845.093966] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6848.177433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6848.366397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6851.146495] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6854.506157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 6854.593662] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6872.035483] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6872.038987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6872.039109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6872.044872] Lustre: Skipped 1 previous similar message [ 6876.262066] Lustre: server umount lustre-OST0000 complete [ 6882.273438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6882.278917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6882.279805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6882.279811] Lustre: Skipped 1 previous similar message [ 6882.295498] Lustre: Skipped 1 previous similar message [ 6884.474271] Lustre: server umount lustre-MDT0000 complete [ 6886.424857] LustreError: 277625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144209 with bad export cookie 7647705575745445848 [ 6886.427992] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6886.433116] LustreError: 277625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6886.643495] Lustre: server umount lustre-MDT0001 complete [ 6891.633864] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6892.951131] Key type lgssc unregistered [ 6893.094566] LNet: 281373:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6893.098815] LNetError: 281373:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6893.107606] LNet: Removed LNI 192.168.203.155@tcp [ 6893.478073] Key type .llcrypt unregistered [ 6893.479495] Key type ._llcrypt unregistered [ 6898.175562] Key type ._llcrypt registered [ 6898.177203] Key type .llcrypt registered [ 6898.225832] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 6905.508812] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 6913.911877] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6914.382341] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6914.441056] alg: No test for adler32 (adler32-zlib) [ 6915.354725] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 6915.490546] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6917.103175] Key type lgssc registered [ 6917.568631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6935.263128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6935.275477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6936.401615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6936.422342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6936.465832] Lustre: lustre-MDT0000: new disk, initializing [ 6936.503688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6936.514620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6938.607463] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6943.891165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6943.942830] Lustre: 284900:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6943.972850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6943.975647] Lustre: Skipped 1 previous similar message [ 6944.015229] Lustre: lustre-MDT0001: new disk, initializing [ 6944.071716] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6944.091904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6944.097568] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6946.168121] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6951.022860] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6953.576632] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6956.913493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6957.055899] Lustre: lustre-OST0000: new disk, initializing [ 6957.059419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6957.063248] Lustre: 286868:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6957.109448] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6959.972203] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6963.728033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6963.732864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6963.784496] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6965.060959] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6967.189233] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6967.286623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6969.367914] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6969.502396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6973.922963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6973.923126] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6973.928139] LustreError: Skipped 1 previous similar message [ 6973.931232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6973.934170] Lustre: Skipped 1 previous similar message [ 6973.943665] Lustre: Skipped 1 previous similar message [ 6976.552152] Lustre: server umount lustre-OST0000 complete [ 6979.041157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6979.043599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6979.047796] Lustre: Skipped 1 previous similar message [ 6979.061170] Lustre: Skipped 1 previous similar message [ 6984.160540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6984.163868] Lustre: Skipped 1 previous similar message [ 6984.724410] Lustre: server umount lustre-MDT0000 complete [ 6986.804519] LustreError: 284892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144309 with bad export cookie 16666549743734004878 [ 6986.811484] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6986.811817] LustreError: 284892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6987.002589] Lustre: server umount lustre-MDT0001 complete [ 6991.703669] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:51:54 (1783144314) [ 6995.259728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6995.544425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6997.919581] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7000.549086] LustreError: 289277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7000.563601] LustreError: 289277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7002.674360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7002.863743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7004.838808] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7008.342781] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7010.954285] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7014.267221] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7014.433376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7017.058492] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7020.172513] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7035.872044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7035.875045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7035.881977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7037.924029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7037.927163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7040.167764] Lustre: server umount lustre-OST0000 complete [ 7043.042468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7043.046252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7043.049238] Lustre: Skipped 1 previous similar message [ 7043.054086] Lustre: Skipped 3 previous similar messages [ 7047.935701] Lustre: server umount lustre-MDT0000 complete [ 7048.160828] LustreError: 289274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.171330] LustreError: 289274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7049.665266] LustreError: 289257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144372 with bad export cookie 16666549743734005746 [ 7049.668853] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7049.671517] LustreError: 289257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7049.821753] Lustre: server umount lustre-MDT0001 complete [ 7054.547651] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7055.833980] Key type lgssc unregistered [ 7055.991532] LNet: 292829:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7055.996088] LNetError: 292829:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7056.005672] LNet: Removed LNI 192.168.203.155@tcp [ 7056.401212] Key type .llcrypt unregistered [ 7056.403045] Key type ._llcrypt unregistered [ 7069.329836] Key type ._llcrypt registered [ 7069.333713] Key type .llcrypt registered [ 7069.388322] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7077.014840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7077.494436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7077.514197] alg: No test for adler32 (adler32-zlib) [ 7078.418086] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 7078.542136] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7080.151473] Key type lgssc registered [ 7080.653392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7101.462396] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7101.471198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7102.578737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7102.595667] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7102.635918] Lustre: lustre-MDT0000: new disk, initializing [ 7102.670398] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7102.680629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7104.328488] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7109.312467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7109.362188] Lustre: 296644:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7109.392414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7109.395079] Lustre: Skipped 1 previous similar message [ 7109.434044] Lustre: lustre-MDT0001: new disk, initializing [ 7109.476386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7109.491736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7109.503838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7111.151752] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7115.793670] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7118.060436] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7121.015827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7121.147135] Lustre: lustre-OST2710: new disk, initializing [ 7121.150311] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7121.153923] Lustre: 298608:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7121.195853] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7123.850641] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7125.206858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7125.213562] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7125.262715] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7128.569909] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7131.770565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7131.847216] Lustre: lustre-OST03e8: new disk, initializing [ 7131.849537] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7131.852886] Lustre: 299670:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7131.894480] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7134.292819] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7137.296304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7137.305657] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7137.361134] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7137.541246] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7140.795402] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7140.889364] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7142.758889] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7142.853988] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7144.783753] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7144.895200] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7146.749375] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7146.850356] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7150.560556] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7150.564377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7150.570601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7152.610488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7152.612892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7152.618103] Lustre: Skipped 2 previous similar messages [ 7152.627521] Lustre: Skipped 3 previous similar messages [ 7154.416835] Lustre: server umount lustre-MDT0000 complete [ 7156.105546] LustreError: 296635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144479 with bad export cookie 12462044017434368736 [ 7156.108854] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7156.111401] LustreError: 296635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7156.261778] Lustre: server umount lustre-MDT0001 complete [ 7167.059403] Lustre: server umount lustre-OST2710 complete [ 7179.061947] Lustre: server umount lustre-OST03e8 complete [ 7187.079419] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7191.405458] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7207.140880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7207.235476] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7207.252768] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7207.294371] Lustre: lustre-MDT0000: new disk, initializing [ 7207.323517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7207.332733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7209.080076] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7214.068732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7214.121669] Lustre: 304911:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7214.236443] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7215.800492] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7219.896626] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7222.122621] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7225.040745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7225.166185] Lustre: lustre-OST0000: new disk, initializing [ 7225.168034] Lustre: Skipped 1 previous similar message [ 7225.171889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7225.174747] Lustre: Skipped 2 previous similar messages [ 7225.179850] Lustre: 306879:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7225.213909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7225.216705] Lustre: Skipped 1 previous similar message [ 7227.667727] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7231.853143] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7231.999421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7232.003897] Lustre: Skipped 1 previous similar message [ 7232.006910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7232.043167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7233.452543] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7233.547992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7235.127701] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7235.209669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7237.090618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7237.091789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7237.094808] LustreError: Skipped 1 previous similar message [ 7237.100939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7237.105089] Lustre: Skipped 1 previous similar message [ 7242.074973] Lustre: server umount lustre-OST0000 complete [ 7245.279994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7245.283897] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7245.288275] Lustre: Skipped 1 previous similar message [ 7245.290453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7249.637041] Lustre: server umount lustre-MDT0000 complete [ 7251.120535] LustreError: 305705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144574 with bad export cookie 12462044017434370409 [ 7251.123687] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7251.126801] LustreError: 305705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7251.257386] Lustre: server umount lustre-MDT0001 complete [ 7254.807404] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:56:18 (1783144578) [ 7255.429773] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7256.156512] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:56:19 (1783144579) [ 7257.395812] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7268.297520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7268.428741] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7268.441190] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7268.507018] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7270.308805] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7273.763401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7273.821482] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7273.833264] Lustre: 310726:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7275.465781] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7291.746851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7291.828261] Lustre: Denying initial registration attempt from nid 192.168.203.155@tcp, specified as failover [ 7291.832170] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7291.836321] LustreError: 312182:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 7291.839410] LustreError: 312182:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7291.842636] LustreError: 312182:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7291.871823] Lustre: server umount lustre-OST0000 complete [ 7291.874118] LustreError: 312182:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7294.432036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7294.435757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.442833] Lustre: Skipped 1 previous similar message [ 7294.448464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7294.452783] Lustre: Skipped 2 previous similar messages [ 7300.447375] Lustre: server umount lustre-MDT0000 complete [ 7301.964280] LustreError: 310719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144624 with bad export cookie 12462044017434371284 [ 7301.968493] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7301.970491] LustreError: 310719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7302.097130] Lustre: server umount lustre-MDT0001 complete [ 7306.199789] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7307.329194] Key type lgssc unregistered [ 7307.471348] LNet: 313301:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7307.474979] LNetError: 313301:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7307.483444] LNet: Removed LNI 192.168.203.155@tcp [ 7307.830092] Key type .llcrypt unregistered [ 7307.831861] Key type ._llcrypt unregistered [ 7311.820240] Key type ._llcrypt registered [ 7311.821782] Key type .llcrypt registered [ 7311.865728] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7312.200562] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7312.252047] alg: No test for adler32 (adler32-zlib) [ 7313.110707] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 7313.208429] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7314.791236] Key type lgssc registered [ 7315.200346] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7339.482192] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7342.366832] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7342.374634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7343.462221] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7343.470062] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7343.531934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7344.839733] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7347.641313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7347.673992] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7347.684867] Lustre: 316048:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7347.784870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7349.214384] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7351.902830] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7353.794135] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7369.123433] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7369.201699] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7369.282901] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.227104] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7373.641863] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7374.422264] Lustre: server umount lustre-OST0000 complete [ 7377.901288] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.902147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.904645] Lustre: Skipped 1 previous similar message [ 7381.841286] Lustre: server umount lustre-MDT0000 complete [ 7383.070270] LustreError: 316789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144706 with bad export cookie 8413364939409676075 [ 7383.073359] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7383.075378] LustreError: 316789:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7383.197914] Lustre: server umount lustre-MDT0001 complete [ 7386.921726] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7387.918892] Key type lgssc unregistered [ 7388.043411] LNet: 319609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7388.046182] LNetError: 319609:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7388.056468] LNet: Removed LNI 192.168.203.155@tcp [ 7388.351890] Key type .llcrypt unregistered [ 7388.353700] Key type ._llcrypt unregistered [ 7397.860510] Key type ._llcrypt registered [ 7397.861796] Key type .llcrypt registered [ 7397.902644] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7398.224555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7398.246495] alg: No test for adler32 (adler32-zlib) [ 7399.116752] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 7399.210621] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7400.799171] Key type lgssc registered [ 7401.157017] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7403.847105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7403.853532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7405.012325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7406.383282] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7409.285609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7409.422819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7410.923696] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7413.362623] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7415.219673] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7419.555266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7419.661933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7421.635763] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7424.157914] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7439.759994] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 7440.352631] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7440.356048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7440.362546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7441.377078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7441.377794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7441.379492] Lustre: Skipped 1 previous similar message [ 7441.385161] Lustre: Skipped 1 previous similar message [ 7443.096575] Lustre: server umount lustre-MDT0000 complete [ 7444.420237] LustreError: 320983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144767 with bad export cookie 2695379283241084335 [ 7444.423568] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7444.424683] LustreError: 320983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7444.870499] LustreError: 321728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7444.879031] LustreError: 321728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7444.880330] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 7444.885422] Lustre: Skipped 1 previous similar message [ 7446.495826] LustreError: 320996:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7446.496124] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7449.503754] LustreError: 321728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7449.504056] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7449.513666] Lustre: Skipped 1 previous similar message [ 7450.604092] Lustre: server umount lustre-MDT0001 complete [ 7452.832221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 7455.714233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7469.023966] LustreError: 320312:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000045ce4196 x1869762760041088/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7469.142472] LustreError: 322831:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7469.150566] LustreError: 322831:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7469.170795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7469.193580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 7470.598899] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7473.434331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7473.562269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7474.597612] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7474.988290] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7477.434247] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7479.714364] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7482.124157] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 4 sec [ 7484.006824] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7484.584686] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7488.992247] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7488.995372] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7489.000447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7489.003267] Lustre: Skipped 1 previous similar message [ 7490.016631] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7500.255261] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7500.312641] Lustre: server umount lustre-OST0000 complete [ 7504.351821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7504.354844] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7504.358523] Lustre: Skipped 1 previous similar message [ 7505.375911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7505.378506] Lustre: Skipped 8 previous similar messages [ 7507.739157] Lustre: server umount lustre-MDT0000 complete [ 7509.093322] LustreError: 323803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144832 with bad export cookie 2695379283241091489 [ 7509.096098] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7509.097836] LustreError: 323803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7509.218209] Lustre: server umount lustre-MDT0001 complete [ 7513.012650] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7514.099734] Key type lgssc unregistered [ 7514.234490] LNet: 327282:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7514.237404] LNetError: 327282:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7514.246418] LNet: Removed LNI 192.168.203.155@tcp [ 7514.557138] Key type .llcrypt unregistered [ 7514.558543] Key type ._llcrypt unregistered [ 7524.009330] Key type ._llcrypt registered [ 7524.010965] Key type .llcrypt registered [ 7524.052320] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7524.359781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7524.401030] alg: No test for adler32 (adler32-zlib) [ 7525.255317] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 7525.344104] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7526.927167] Key type lgssc registered [ 7527.361074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7530.270541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7530.278884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7531.440023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7533.006359] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7536.172351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7536.305849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7537.792033] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7540.360656] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7542.168953] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7546.272910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7546.277247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7546.279085] Lustre: Skipped 1 previous similar message [ 7549.332393] Lustre: server umount lustre-MDT0000 complete [ 7550.502645] LustreError: 330302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144873 with bad export cookie 5158747560345273138 [ 7550.506842] LustreError: 330302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7550.507474] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7550.616411] Lustre: server umount lustre-MDT0001 complete [ 7555.568196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7555.679396] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7555.691094] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7555.771495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7557.122299] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7560.193043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7560.229284] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7560.240554] Lustre: 331271:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7560.343805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7561.701525] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7564.088939] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7565.921967] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7570.657763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7570.727978] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 7570.732501] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 7570.736447] LustreError: 333154:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 7570.739994] LustreError: 333154:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7570.743130] LustreError: 333154:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7570.771441] Lustre: server umount lustre-OST0000 complete [ 7570.773262] LustreError: 333154:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 7575.649506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7575.713208] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7575.771699] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7577.560349] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7579.845555] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7582.314365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7582.346314] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7582.380256] Lustre: lustre-OST0001: new disk, initializing [ 7582.382706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7582.386183] Lustre: 334685:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7582.447696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 7582.485774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7582.490520] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7582.502119] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000400 [ 7584.378464] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7586.855087] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7592.931682] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7592.931815] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7592.935648] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7592.939285] Lustre: Skipped 2 previous similar messages [ 7592.942467] Lustre: Skipped 1 previous similar message [ 7598.053917] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7598.057242] Lustre: Skipped 1 previous similar message [ 7602.143125] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7602.182496] Lustre: server umount lustre-OST0001 complete [ 7603.171755] LustreError: 333621:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7603.177512] LustreError: 333621:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7608.289526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7608.291330] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7608.291921] LustreError: 335646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7608.293515] LustreError: Skipped 2 previous similar messages [ 7608.301320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7608.312220] Lustre: Skipped 1 previous similar message [ 7609.618366] Lustre: server umount lustre-OST0000 complete [ 7611.871911] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7611.875800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7611.882278] Lustre: Skipped 1 previous similar message [ 7613.408794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7613.410788] Lustre: Skipped 2 previous similar messages [ 7623.648458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7623.651348] Lustre: Skipped 3 previous similar messages [ 7625.183174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7625.245170] Lustre: server umount lustre-MDT0000 complete [ 7626.662332] LustreError: 331263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783144949 with bad export cookie 5158747560345273677 [ 7626.666075] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7626.667780] LustreError: 331263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7626.782887] Lustre: server umount lustre-MDT0001 complete [ 7630.736687] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7631.884258] Key type lgssc unregistered [ 7632.026380] LNet: 336646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7632.029743] LNetError: 336646:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7632.038595] LNet: Removed LNI 192.168.203.155@tcp [ 7632.396437] Key type .llcrypt unregistered [ 7632.398227] Key type ._llcrypt unregistered [ 7664.940923] Key type ._llcrypt registered [ 7664.942444] Key type .llcrypt registered [ 7664.983088] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7671.729436] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7672.036652] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7672.053791] alg: No test for adler32 (adler32-zlib) [ 7672.925360] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 7673.031717] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7674.623176] Key type lgssc registered [ 7675.042911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7687.645163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7687.653025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7688.743600] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7688.755950] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7688.792378] Lustre: lustre-MDT0000: new disk, initializing [ 7688.816855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7688.823820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7690.303344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7694.786822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7694.838233] Lustre: 341559:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7694.868809] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7694.871795] Lustre: Skipped 1 previous similar message [ 7694.914313] Lustre: lustre-MDT0001: new disk, initializing [ 7694.954396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7694.968486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7694.973020] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7696.515886] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7700.410755] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7702.472393] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7704.863556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7704.972725] Lustre: lustre-OST0000: new disk, initializing [ 7704.975140] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7704.979744] Lustre: 343525:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7705.013166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7707.021744] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7707.131434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7707.135001] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7707.171882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7710.900330] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7712.405081] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7712.479088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7713.900614] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7713.986108] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7717.345311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7717.348175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7717.351858] Lustre: Skipped 1 previous similar message [ 7717.356498] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7720.790790] Lustre: server umount lustre-OST0000 complete [ 7722.464721] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7722.465376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7722.469944] Lustre: Skipped 1 previous similar message [ 7722.472808] Lustre: Skipped 1 previous similar message [ 7727.583949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7727.586438] Lustre: Skipped 2 previous similar messages [ 7728.279436] Lustre: server umount lustre-MDT0000 complete [ 7729.503135] LustreError: 342354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145052 with bad export cookie 2561867360574438255 [ 7729.505980] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7729.508470] LustreError: 342354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7729.615911] Lustre: server umount lustre-MDT0001 complete [ 7732.405548] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 02:04:15 (1783145055) [ 7734.975546] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 02:04:18 (1783145058) [ 7737.940759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7738.151563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7739.605824] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7742.555346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7742.680328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7744.055078] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7746.424755] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7748.282537] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7750.698355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7750.842646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7752.944949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7755.509090] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7769.420101] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 7769.973289] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7770.504679] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 7771.074643] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 7771.760505] Lustre: Failing over lustre-MDT0000 [ 7771.980172] Lustre: server umount lustre-MDT0000 complete [ 7773.664359] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7773.666667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7773.671541] LustreError: 347013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7773.678073] LustreError: 347013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7774.688280] LustreError: 346276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7774.695114] LustreError: 346276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7776.142207] LustreError: 348836:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7779.808749] LustreError: 346277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7779.815020] LustreError: 346277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7784.928098] LustreError: 347013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7784.933268] LustreError: 347013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7784.936145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7784.978139] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7785.063627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7785.090123] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7786.374402] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7786.432525] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7790.564720] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7790.572678] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 7790.588717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7792.264101] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7792.797427] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7794.672510] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 7795.411983] Lustre: Failing over lustre-MDT0000 [ 7795.649589] Lustre: server umount lustre-MDT0000 complete [ 7795.680379] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7795.682026] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7795.685104] LustreError: 349874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7795.688191] Lustre: Skipped 4 previous similar messages [ 7795.694145] LustreError: 349874:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7808.699717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7808.747769] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7808.864825] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 7810.219101] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7811.974049] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 7814.116732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7814.119079] Lustre: Skipped 2 previous similar messages [ 7814.126520] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 7814.146305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 7815.900797] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7816.480413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7819.233945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7819.234978] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7819.236585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7819.245641] Lustre: Skipped 2 previous similar messages [ 7824.352412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7824.354523] Lustre: Skipped 4 previous similar messages [ 7824.926201] Lustre: server umount lustre-MDT0000 complete [ 7826.205142] LustreError: 348116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145149 with bad export cookie 2561867360574440936 [ 7826.208468] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7826.209411] LustreError: 348116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7826.321696] Lustre: server umount lustre-MDT0001 complete [ 7837.795497] Lustre: server umount lustre-OST0000 complete [ 7842.930363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7843.085242] LustreError: 352627:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7843.091215] LustreError: 352627:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 7843.106297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7843.108175] Lustre: Skipped 1 previous similar message [ 7844.474714] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7847.340925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7848.900451] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7851.425264] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7853.340414] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7855.628197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7857.640725] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7860.086066] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7868.072611] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 7873.506156] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7873.506390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7873.510162] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7873.515039] Lustre: Skipped 1 previous similar message [ 7878.099914] Lustre: server umount lustre-OST0000 complete [ 7883.231892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7883.234777] LustreError: Skipped 1 previous similar message [ 7885.521981] Lustre: server umount lustre-MDT0000 complete [ 7886.769256] LustreError: 352609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145209 with bad export cookie 2561867360574441657 [ 7886.772297] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7886.773629] LustreError: 352609:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7886.883774] Lustre: server umount lustre-MDT0001 complete [ 7890.766383] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7891.819645] Key type lgssc unregistered [ 7891.952399] LNet: 356178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7891.955713] LNetError: 356178:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7891.962306] LNet: Removed LNI 192.168.203.155@tcp [ 7892.276657] Key type .llcrypt unregistered [ 7892.277783] Key type ._llcrypt unregistered [ 7901.517047] Key type ._llcrypt registered [ 7901.518509] Key type .llcrypt registered [ 7901.563784] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7902.083675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7902.108295] alg: No test for adler32 (adler32-zlib) [ 7902.966246] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 7903.056717] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7904.639151] Key type lgssc registered [ 7905.031036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7907.646948] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7907.654455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7908.807216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.067321] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7912.726506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7912.832881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7914.095552] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7916.290380] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7917.988677] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7920.212419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7920.339873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7922.188836] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7924.492371] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7933.669546] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 7939.041796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7939.042861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7939.046795] Lustre: Skipped 2 previous similar messages [ 7939.049833] Lustre: Skipped 1 previous similar message [ 7944.160810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7944.162964] Lustre: Skipped 3 previous similar messages [ 7944.726515] Lustre: server umount lustre-MDT0000 complete [ 7945.908307] LustreError: 357650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145268 with bad export cookie 8056704625482552108 [ 7945.911288] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7945.912324] LustreError: 357650:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7946.014366] Lustre: server umount lustre-MDT0001 complete [ 7957.614283] Lustre: server umount lustre-OST0000 complete [ 7961.860698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7962.002898] LustreError: 361400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7962.009798] LustreError: 361400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7962.024863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7963.301182] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7965.926348] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7966.035828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7967.295811] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7969.515698] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7971.192543] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7973.454530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7975.440109] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7977.688440] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7987.939099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 7996.896081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7996.899072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7996.903079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7998.433878] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8000.982093] Lustre: server umount lustre-MDT0000 complete [ 8002.202993] LustreError: 363241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145325 with bad export cookie 8056704625482553207 [ 8002.208503] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8002.208767] LustreError: 363241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8002.319875] Lustre: server umount lustre-MDT0001 complete [ 8013.849211] Lustre: server umount lustre-OST0000 complete [ 8018.899419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.057719] LustreError: 365330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8019.083220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.086200] Lustre: Skipped 1 previous similar message [ 8020.511703] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8023.385050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8024.861780] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8027.365488] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8029.220152] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8031.599460] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8033.735927] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8036.197542] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8044.011295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 8049.124945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8049.125872] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8049.127933] LustreError: Skipped 1 previous similar message [ 8049.132969] Lustre: Skipped 1 previous similar message [ 8049.133871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8049.137620] Lustre: Skipped 4 previous similar messages [ 8054.220411] Lustre: server umount lustre-OST0000 complete [ 8059.361718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8059.361828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8059.362468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8059.370218] Lustre: Skipped 2 previous similar messages [ 8061.582574] Lustre: server umount lustre-MDT0000 complete [ 8062.742197] LustreError: 365311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145385 with bad export cookie 8056704625482554215 [ 8062.744636] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8062.746056] LustreError: 365311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8062.843547] Lustre: server umount lustre-MDT0001 complete [ 8066.215443] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8067.244053] Key type lgssc unregistered [ 8067.376392] LNet: 368879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8067.379184] LNetError: 368879:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8067.386410] LNet: Removed LNI 192.168.203.155@tcp [ 8067.702560] Key type .llcrypt unregistered [ 8067.704232] Key type ._llcrypt unregistered [ 8076.659944] Key type ._llcrypt registered [ 8076.661087] Key type .llcrypt registered [ 8076.700326] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8077.012839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8077.074122] alg: No test for adler32 (adler32-zlib) [ 8077.931671] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8078.009825] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8079.583144] Key type lgssc registered [ 8079.945597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8082.443247] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8082.448130] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8082.450502] LustreError: 370296:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 8082.457796] LustreError: 370296:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 8082.459768] LustreError: 370296:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 8082.461666] LustreError: 370296:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 8082.463747] LustreError: 370296:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8084.996642] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8084.999935] LustreError: 370692:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 8085.006928] LustreError: 370692:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 8085.009713] LustreError: 370692:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 8085.012454] LustreError: 370692:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 8085.014850] LustreError: 370692:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 8091.337593] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8092.337549] Key type lgssc unregistered [ 8092.473676] LNet: 371694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8092.475879] LNetError: 371694:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8092.484268] LNet: Removed LNI 192.168.203.155@tcp [ 8092.789325] Key type .llcrypt unregistered [ 8092.790685] Key type ._llcrypt unregistered [ 8098.481499] Key type ._llcrypt registered [ 8098.482622] Key type .llcrypt registered [ 8098.527342] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8104.671094] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8105.045911] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8105.053969] alg: No test for adler32 (adler32-zlib) [ 8105.906603] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8106.016220] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8107.607170] Key type lgssc registered [ 8108.011183] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8118.854524] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8118.858984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8119.940390] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8119.949536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8119.979624] Lustre: lustre-MDT0000: new disk, initializing [ 8119.999036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8120.005382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8121.234856] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8125.160189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8125.200644] Lustre: 374824:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8125.227728] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8125.230840] Lustre: Skipped 1 previous similar message [ 8125.268796] Lustre: lustre-MDT0001: new disk, initializing [ 8125.305808] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8125.317974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8125.321531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8126.704179] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8130.226725] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8131.924248] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8134.083843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8134.179564] Lustre: lustre-OST0000: new disk, initializing [ 8134.181807] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8134.185389] Lustre: 376790:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8134.213846] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8135.096242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8135.101174] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8135.136818] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8136.048208] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8139.587165] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8140.831154] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8140.904178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8142.320590] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8142.392585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8145.107497] Lustre: server umount lustre-OST0000 complete [ 8150.496553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.497069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8150.500086] Lustre: Skipped 1 previous similar message [ 8155.615956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8155.617944] Lustre: Skipped 2 previous similar messages [ 8156.667974] Lustre: server umount lustre-MDT0000 complete [ 8157.872511] LustreError: 374815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145480 with bad export cookie 6348982987140790913 [ 8157.877446] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8157.877706] LustreError: 374815:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8167.437531] Lustre: server umount lustre-MDT0001 complete [ 8169.901732] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 8170.419191] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 02:11:33 (1783145493) [ 8172.531995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8172.694251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8173.913902] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8176.551141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8177.884806] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8180.006715] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8181.668861] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8183.721149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8183.822570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8183.824744] Lustre: Skipped 1 previous similar message [ 8185.563277] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8187.737141] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8189.808338] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8189.864069] Lustre: lustre-OST0001: new disk, initializing [ 8189.865635] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8189.868564] Lustre: 382081:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8191.635975] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8192.380105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8192.383679] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8192.405069] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8195.046114] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8197.602080] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8197.602096] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8197.602823] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8197.609960] Lustre: Skipped 1 previous similar message [ 8201.613046] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 8201.615728] Lustre: Skipped 3 previous similar messages [ 8202.877725] Lustre: server umount lustre-OST0001 complete [ 8203.747571] LustreError: 382541:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8203.752226] LustreError: 382541:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8208.867708] LustreError: 382548:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8208.873076] LustreError: 382548:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8213.990071] LustreError: 381066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8213.995570] LustreError: 381066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8217.568263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8217.571329] LustreError: Skipped 1 previous similar message [ 8217.573106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8217.578661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8219.104609] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8219.107023] LustreError: 381066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.111613] LustreError: 381066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8224.226788] LustreError: 381066:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8224.231560] LustreError: 381066:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8229.343111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8229.345890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8229.349021] Lustre: Skipped 5 previous similar messages [ 8229.398519] Lustre: server umount lustre-OST0000 complete [ 8232.927786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8232.930910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8232.935847] Lustre: Skipped 1 previous similar message [ 8236.758236] Lustre: server umount lustre-MDT0000 complete [ 8237.882968] LustreError: 379228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145560 with bad export cookie 6348982987140791788 [ 8237.885845] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8237.888797] LustreError: 379228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8252.383205] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8252.468027] Lustre: server umount lustre-MDT0001 complete [ 8255.997640] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8257.043052] Key type lgssc unregistered [ 8257.171508] LNet: 384069:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8257.174688] LNetError: 384069:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8257.183362] LNet: Removed LNI 192.168.203.155@tcp [ 8257.485254] Key type .llcrypt unregistered [ 8257.486965] Key type ._llcrypt unregistered [ 8274.201251] Key type ._llcrypt registered [ 8274.202339] Key type .llcrypt registered [ 8274.238713] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8274.549627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8274.630478] alg: No test for adler32 (adler32-zlib) [ 8275.483593] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8275.579382] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8277.159118] Key type lgssc registered [ 8277.532836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8280.111684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8280.119720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8281.213585] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8281.221677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8281.283334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8282.737959] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8285.656197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8285.689417] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8285.700338] Lustre: 386844:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8285.789253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8287.159185] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8289.470392] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8291.228566] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8293.857360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8293.858472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8293.863169] Lustre: Skipped 1 previous similar message [ 8298.065921] Lustre: server umount lustre-MDT0000 complete [ 8299.245203] LustreError: 386834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145622 with bad export cookie 15523898246051226347 [ 8299.247853] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8299.250748] LustreError: 386834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8299.357659] Lustre: server umount lustre-MDT0001 complete [ 8302.289246] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 02:13:45 (1783145625) [ 8304.484217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8304.637516] LustreError: 389352:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8304.642826] LustreError: 389352:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8304.655446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8305.982411] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8308.824187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8308.926488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8310.209877] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8312.442548] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8314.185528] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8316.215467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8316.280051] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8316.337981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8318.078303] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8320.228442] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8329.152926] Lustre: Permanently deactivating lustre-OST0000 [ 8329.159575] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 8330.477941] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8331.071713] LustreError: 392190:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 8331.074709] LustreError: 392190:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 8331.076620] LustreError: 392190:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8331.652341] LustreError: 392238:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 8331.655016] LustreError: 392238:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8338.322040] Lustre: server umount lustre-OST0000 complete [ 8339.219985] LustreError: 392487:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 8339.223730] LustreError: 392487:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 8339.935759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8339.938037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8339.942394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8339.944467] Lustre: Skipped 1 previous similar message [ 8343.520222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8343.520675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8348.640027] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8348.642083] Lustre: Skipped 2 previous similar messages [ 8353.760021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8353.762329] Lustre: Skipped 1 previous similar message [ 8354.271202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8354.332234] Lustre: server umount lustre-MDT0000 complete [ 8355.570646] LustreError: 391185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145678 with bad export cookie 15523898246051226914 [ 8355.574874] LustreError: 391185:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8355.575790] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8365.070871] Lustre: server umount lustre-MDT0001 complete [ 8367.934537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8369.381819] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8370.344829] LustreError: 393645:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 8372.232077] Lustre: 393983:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.155@tcp [ 8372.859209] Lustre: server umount MGS complete [ 8376.387613] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.546645] LustreError: 394705:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.550974] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8376.556136] Lustre: Skipped 1 previous similar message [ 8376.576520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8377.903100] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8380.552200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.632390] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 8381.811932] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8384.195577] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8386.091661] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8386.715756] Lustre: Permanently reactivating lustre-OST0000 [ 8386.722058] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 8386.724775] Lustre: Skipped 2 previous similar messages [ 8412.639096] LustreError: 396532:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 8414.972160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8415.095420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8415.097977] Lustre: Skipped 1 previous similar message [ 8417.012572] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8419.429223] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8432.610420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8432.610748] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8432.611326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8432.619889] Lustre: Skipped 1 previous similar message [ 8433.354087] Lustre: server umount lustre-OST0000 complete [ 8436.756796] Lustre: server umount lustre-MDT0000 complete [ 8437.215737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8437.218908] LustreError: Skipped 1 previous similar message [ 8437.221234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.226555] LustreError: 394724:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8438.011917] LustreError: 394732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145761 with bad export cookie 15523898246051227936 [ 8438.015315] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8438.015540] LustreError: 394732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8438.133932] Lustre: server umount lustre-MDT0001 complete [ 8442.053702] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8443.123514] Key type lgssc unregistered [ 8443.243439] LNet: 399371:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8443.245967] LNetError: 399371:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8443.261351] LNet: Removed LNI 192.168.203.155@tcp [ 8443.577322] Key type .llcrypt unregistered [ 8443.578311] Key type ._llcrypt unregistered [ 8449.348093] Key type ._llcrypt registered [ 8449.349759] Key type .llcrypt registered [ 8449.383758] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8455.153224] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8455.469788] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8455.508063] alg: No test for adler32 (adler32-zlib) [ 8456.351068] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8456.430741] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8458.007125] Key type lgssc registered [ 8458.333122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8468.903519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8468.908681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8469.988207] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8469.998484] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8470.033503] Lustre: lustre-MDT0000: new disk, initializing [ 8470.053185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8470.058718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8471.366799] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8475.166173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8475.207177] Lustre: 402498:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8475.230377] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8475.232043] Lustre: Skipped 1 previous similar message [ 8475.263248] Lustre: lustre-MDT0001: new disk, initializing [ 8475.293383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8475.305663] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8475.309659] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8476.563292] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8480.050206] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8481.771704] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8483.851753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8483.946993] Lustre: lustre-OST0000: new disk, initializing [ 8483.948538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8483.950786] Lustre: 404468:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8483.975981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8485.687205] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8486.518237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8486.521792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8486.551795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8489.220654] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8490.595906] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8490.678097] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8492.087119] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8492.179665] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8497.123910] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8497.124052] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8497.127152] LustreError: Skipped 1 previous similar message [ 8497.131586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8497.133845] Lustre: Skipped 1 previous similar message [ 8502.242061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8502.244349] Lustre: Skipped 1 previous similar message [ 8507.359218] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8507.360952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8507.364626] Lustre: Skipped 1 previous similar message [ 8507.413449] Lustre: server umount lustre-OST0000 complete [ 8511.455843] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8511.458579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.462445] Lustre: Skipped 1 previous similar message [ 8511.464641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8512.480953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8514.839138] Lustre: server umount lustre-MDT0000 complete [ 8516.003199] LustreError: 402491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145839 with bad export cookie 8901976083375098407 [ 8516.005750] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8516.006497] LustreError: 402491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8516.121926] Lustre: server umount lustre-MDT0001 complete [ 8518.705844] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 8519.181401] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 02:17:22 (1783145842) [ 8521.207485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8521.380976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.706266] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8525.521649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8525.646825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8526.997645] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8529.568886] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8531.308320] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8533.419968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8535.433909] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8537.852424] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8538.499666] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 8549.323888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 8550.879947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8550.882813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8550.887511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8550.890949] Lustre: Skipped 2 previous similar messages [ 8556.823578] Lustre: server umount lustre-OST0000 complete [ 8561.632076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8561.635090] LustreError: Skipped 1 previous similar message [ 8561.637288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8561.640669] Lustre: Skipped 1 previous similar message [ 8561.642513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8561.644250] Lustre: Skipped 2 previous similar messages [ 8572.383143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8572.444225] Lustre: server umount lustre-MDT0000 complete [ 8573.668190] LustreError: 406907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145896 with bad export cookie 8901976083375099275 [ 8573.670486] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8573.671962] LustreError: 406907:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8573.778588] Lustre: server umount lustre-MDT0001 complete [ 8577.242282] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8578.267406] Key type lgssc unregistered [ 8578.395410] LNet: 410671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8578.397554] LNetError: 410671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8578.404232] LNet: Removed LNI 192.168.203.155@tcp [ 8578.704393] Key type .llcrypt unregistered [ 8578.705754] Key type ._llcrypt unregistered [ 8585.787198] Key type ._llcrypt registered [ 8585.788421] Key type .llcrypt registered [ 8585.828527] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8585.918439] Key type .llcrypt unregistered [ 8585.919971] Key type ._llcrypt unregistered [ 8592.430451] Key type ._llcrypt registered [ 8592.432118] Key type .llcrypt registered [ 8592.468366] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8592.756690] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8592.793816] alg: No test for adler32 (adler32-zlib) [ 8593.641610] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8593.722564] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8595.295113] Key type lgssc registered [ 8595.620399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8598.081586] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8598.088119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8599.232145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8600.470741] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8603.067165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8603.175527] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8604.979498] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8607.294898] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8608.226358] LustreError: 412859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8608.232174] LustreError: 412859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8609.250322] LustreError: 412858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8609.255099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8609.256096] LustreError: 412858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8609.520127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8609.641358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8610.925124] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8619.487864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8619.491346] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8619.495857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8620.001598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8620.002343] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8620.006830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8624.014148] Lustre: server umount lustre-OST0000 complete [ 8628.704982] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8628.707974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8628.712833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8631.250075] Lustre: server umount lustre-MDT0000 complete [ 8632.403144] LustreError: 412841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783145955 with bad export cookie 17847534834921438303 [ 8632.406644] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8632.408656] LustreError: 412841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8632.521658] Lustre: server umount lustre-MDT0001 complete [ 8635.848707] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8636.855827] Key type lgssc unregistered [ 8636.977408] LNet: 416001:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8636.980387] LNetError: 416001:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8636.988324] LNet: Removed LNI 192.168.203.155@tcp [ 8637.308832] Key type .llcrypt unregistered [ 8637.310686] Key type ._llcrypt unregistered [ 8645.860282] Key type ._llcrypt registered [ 8645.861841] Key type .llcrypt registered [ 8645.894581] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8646.230771] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8646.294450] alg: No test for adler32 (adler32-zlib) [ 8647.155158] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8647.255273] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8648.831157] Key type lgssc registered [ 8649.161432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8651.550435] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8651.557813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8666.975448] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 8672.095288] LustreError: 417368:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 8698.719171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8700.645632] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8703.026358] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8705.323451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.496522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8705.517168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8706.856609] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8709.599626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8709.725227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8711.082805] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8713.569521] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8715.445211] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8719.330892] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8719.331860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8719.336287] Lustre: Skipped 1 previous similar message [ 8719.340310] Lustre: Skipped 1 previous similar message [ 8723.942352] Lustre: server umount lustre-OST0000 complete [ 8729.057238] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 8729.061196] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8729.067772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8731.418047] Lustre: server umount lustre-MDT0000 complete [ 8732.626319] LustreError: 419896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146055 with bad export cookie 11321709970302554917 [ 8732.629532] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8732.630548] LustreError: 419896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8732.743354] Lustre: server umount lustre-MDT0001 complete [ 8736.253673] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8737.225232] Key type lgssc unregistered [ 8737.350404] LNet: 420936:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8737.353634] LNetError: 420936:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8737.364311] LNet: Removed LNI 192.168.203.155@tcp [ 8737.671438] Key type .llcrypt unregistered [ 8737.672711] Key type ._llcrypt unregistered [ 8746.029439] Key type ._llcrypt registered [ 8746.030881] Key type .llcrypt registered [ 8746.065385] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8746.451660] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8746.460402] alg: No test for adler32 (adler32-zlib) [ 8747.310742] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8747.406144] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8748.991127] Key type lgssc registered [ 8749.404652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8751.926555] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8751.932705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8753.076915] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8754.344764] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8757.060818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8757.178233] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8758.524619] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8760.841572] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8762.623569] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8764.727049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8764.841825] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.710559] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8768.998675] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8778.150844] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 8782.815941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8782.818303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8782.822231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8783.328947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8783.329509] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.783598] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 8784.785449] Lustre: Skipped 2 previous similar messages [ 8786.711259] Lustre: server umount lustre-MDT0000 complete [ 8787.533538] LustreError: 422354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.55@tcp arrived at 1783146110 with bad export cookie 15706633434923100525 [ 8787.538991] LustreError: 422354:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8788.243630] LustreError: 422310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146111 with bad export cookie 15706633434923099874 [ 8788.247326] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8788.248042] LustreError: 422310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8788.448373] LustreError: 422328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8788.448738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8788.448862] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.448869] Lustre: Skipped 1 previous similar message [ 8788.449790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8788.456167] LustreError: 422328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8793.567945] LustreError: 422327:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8793.568285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8793.573845] LustreError: 422327:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8794.383994] Lustre: server umount lustre-OST0000 complete [ 8796.594494] Lustre: server umount lustre-MDT0001 complete [ 8800.159796] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8801.129948] Key type lgssc unregistered [ 8801.248427] LNet: 426030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8801.250777] LNetError: 426030:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8801.258293] LNet: Removed LNI 192.168.203.155@tcp [ 8801.552741] Key type .llcrypt unregistered [ 8801.555083] Key type ._llcrypt unregistered [ 8810.263568] Key type ._llcrypt registered [ 8810.265010] Key type .llcrypt registered [ 8810.303788] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8810.633724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8810.695063] alg: No test for adler32 (adler32-zlib) [ 8811.541519] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8811.625429] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8813.207129] Key type lgssc registered [ 8813.588359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8816.268650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8816.275434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8817.432479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8818.885876] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8821.582801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8821.688641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8822.919646] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8825.149307] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8826.896242] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8829.100338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8829.213944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8831.069122] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8833.332198] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8842.533772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 8847.327837] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8847.329801] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8847.333086] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8847.842289] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8847.848951] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8850.705581] Lustre: server umount lustre-MDT0001 complete [ 8852.960110] LustreError: 427422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8852.960580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8852.960680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8852.966947] LustreError: 427422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8852.971883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8852.977702] Lustre: Skipped 1 previous similar message [ 8858.080152] LustreError: 428162:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8858.080485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8858.382960] Lustre: server umount lustre-OST0000 complete [ 8859.748110] Lustre: server umount lustre-MDT0000 complete [ 8864.087111] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8865.037606] Key type lgssc unregistered [ 8865.172368] LNet: 431126:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8865.174212] LNetError: 431126:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8865.184229] LNet: Removed LNI 192.168.203.155@tcp [ 8865.459502] Key type .llcrypt unregistered [ 8865.460413] Key type ._llcrypt unregistered [ 8872.901508] Key type ._llcrypt registered [ 8872.903059] Key type .llcrypt registered [ 8872.939822] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8879.307692] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8879.689150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8879.727129] alg: No test for adler32 (adler32-zlib) [ 8880.592335] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8880.690043] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8882.279141] Key type lgssc registered [ 8882.654131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8895.384614] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8900.835863] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8900.840969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8901.928605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8901.939468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8901.978082] Lustre: lustre-MDT0000: new disk, initializing [ 8901.998407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8902.003837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8903.248678] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8908.594297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8908.689530] Lustre: lustre-OST0000: new disk, initializing [ 8908.691167] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8908.692817] Lustre: Skipped 1 previous similar message [ 8908.694526] Lustre: 436129:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8908.715666] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8910.082949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8910.086619] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8910.097174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8910.614910] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8914.397061] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8920.092716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8920.144280] Lustre: 435114:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8920.181846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8920.223947] Lustre: lustre-MDT0001: new disk, initializing [ 8920.276281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8920.294140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 8920.298971] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 8921.763976] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8925.473700] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 8925.549737] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 8926.893330] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 8926.964302] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 8930.701035] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 8931.297841] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8931.298142] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 8931.302621] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8931.306244] Lustre: Skipped 1 previous similar message [ 8934.607574] Lustre: server umount lustre-MDT0001 complete [ 8941.536624] LustreError: 436137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8941.546321] LustreError: 436137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8946.657043] LustreError: 435119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8946.661646] LustreError: 435119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8951.263982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8951.267535] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8951.273334] Lustre: Skipped 1 previous similar message [ 8951.276331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8951.776528] LustreError: 435124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8952.981234] Lustre: server umount lustre-OST0000 complete [ 8960.575098] Lustre: server umount lustre-MDT0000 complete [ 8965.186593] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8966.298785] Key type lgssc unregistered [ 8966.435363] LNet: 439875:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8966.437917] LNetError: 439875:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8966.444410] LNet: Removed LNI 192.168.203.155@tcp [ 8966.777498] Key type .llcrypt unregistered [ 8966.778725] Key type ._llcrypt unregistered [ 8979.661240] Key type ._llcrypt registered [ 8979.662329] Key type .llcrypt registered [ 8979.703346] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8979.807275] Key type .llcrypt unregistered [ 8979.808392] Key type ._llcrypt unregistered [ 8986.716650] Key type ._llcrypt registered [ 8986.717748] Key type .llcrypt registered [ 8986.758352] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8987.219031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8987.237460] alg: No test for adler32 (adler32-zlib) [ 8988.096822] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 8988.191136] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8989.775119] Key type lgssc registered [ 8990.187585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8999.011953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8999.017653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9000.106382] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9000.116539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9000.157500] Lustre: lustre-MDT0000: new disk, initializing [ 9000.179109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9000.185442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9001.487499] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9005.573114] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9005.617605] Lustre: 443221:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9005.646281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9005.648798] Lustre: Skipped 1 previous similar message [ 9005.686248] Lustre: lustre-MDT0001: new disk, initializing [ 9005.722861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9005.735321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9005.739199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9007.080246] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9010.701477] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9012.521836] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9014.837034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9014.947278] Lustre: lustre-OST0000: new disk, initializing [ 9014.948971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9014.951537] Lustre: 445187:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9014.983844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9016.312832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9016.315694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9016.352525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9017.015440] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9020.711789] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9026.529492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9026.530122] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9026.530991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9026.534016] Lustre: Skipped 1 previous similar message [ 9029.214019] Lustre: server umount lustre-MDT0000 complete [ 9030.447265] LustreError: 443213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146353 with bad export cookie 15821968997491907644 [ 9030.450496] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9030.451065] LustreError: 443213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9030.567995] Lustre: server umount lustre-MDT0001 complete [ 9042.020380] Lustre: server umount lustre-OST0000 complete [ 9045.431567] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 02:26:08 (1783146368) [ 9047.990874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9048.175243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9049.574167] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9052.468432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9052.592918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9053.948611] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9056.109431] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9057.715690] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9072.737799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9072.810555] Lustre: Found index 0 for lustre-OST0000, updating log [ 9072.870847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9073.895739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 9074.947486] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9077.380581] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9079.263870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9079.266111] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9079.269428] Lustre: Skipped 1 previous similar message [ 9079.271294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9079.274172] Lustre: Skipped 2 previous similar messages [ 9083.362551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9083.364912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9083.369364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9084.694318] Lustre: server umount lustre-OST0000 complete [ 9098.719942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9098.722145] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9098.725905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9098.727582] Lustre: Skipped 2 previous similar messages [ 9103.382277] Lustre: server umount lustre-MDT0000 complete [ 9104.677117] LustreError: 449710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783146427 with bad export cookie 15821968997491909023 [ 9104.680433] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9104.682922] LustreError: 449710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9104.810212] Lustre: server umount lustre-MDT0001 complete [ 9107.570295] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 02:27:11 (1783146431) [ 9137.695222] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [ 9687.089340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9687.249572] LustreError: 453612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.254988] LustreError: 453612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9687.275798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9688.567934] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9691.245178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9692.591960] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9694.754848] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9696.427084] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9711.521872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9711.610229] Lustre: Found index 0 for lustre-OST0000, updating log [ 9713.418410] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9714.789438] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 9715.660805] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9717.345042] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9717.348283] Lustre: Skipped 1 previous similar message [ 9717.349909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9717.351454] Lustre: Skipped 2 previous similar messages [ 9718.369846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9719.393068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9722.442214] Lustre: server umount lustre-OST0000 complete [ 9724.640800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9724.641756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9724.644254] Lustre: Skipped 1 previous similar message [ 9724.647155] Lustre: Skipped 1 previous similar message [ 9728.736139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9728.737958] Lustre: Skipped 7 previous similar messages [ 9729.748548] Lustre: server umount lustre-MDT0000 complete [ 9729.761240] LustreError: 453613:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9729.768815] LustreError: 453613:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9730.784387] LustreError: 454340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9730.791423] LustreError: 454340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9730.970229] LustreError: 453593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147053 with bad export cookie 15821968997491910080 [ 9730.973511] LustreError: 453593:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9730.974885] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9731.092252] Lustre: server umount lustre-MDT0001 complete [ 9738.783295] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [ 9748.045226] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9751.294767] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9762.323431] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:38:05 (1783147085) [ 9767.721867] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9768.622765] Key type lgssc unregistered [ 9773.855495] LNet: 460257:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9773.859480] LNetError: 460257:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9844.521296] LNet: Removed LNI 192.168.203.155@tcp [ 9844.865080] Key type .llcrypt unregistered [ 9844.866461] Key type ._llcrypt unregistered [ 9851.463150] Key type ._llcrypt registered [ 9851.464207] Key type .llcrypt registered [ 9851.503105] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9851.819553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9851.839215] alg: No test for adler32 (adler32-zlib) [ 9852.694955] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 9852.785651] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9854.367145] Key type lgssc registered [ 9854.741944] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9856.160261] LNet: Added LNI 192.168.203.155@tcp100 [8/256/0/180] [ 9856.443504] LNet: Added LNI 192.168.203.155@tcp101 [8/256/0/180] [ 9856.724111] LNet: Added LNI 192.168.203.155@tcp102 [8/256/0/180] [ 9857.008689] LNet: Added LNI 192.168.203.155@tcp103 [8/256/0/180] [ 9857.300478] LNet: Added LNI 192.168.203.155@tcp104 [8/256/0/180] [ 9857.595427] LNet: Added LNI 192.168.203.155@tcp105 [8/256/0/180] [ 9857.874966] LNet: Added LNI 192.168.203.155@tcp106 [8/256/0/180] [ 9858.171225] LNet: Added LNI 192.168.203.155@tcp107 [8/256/0/180] [ 9858.440885] LNet: Added LNI 192.168.203.155@tcp108 [8/256/0/180] [ 9858.707276] LNet: Added LNI 192.168.203.155@tcp109 [8/256/0/180] [ 9858.974625] LNet: Added LNI 192.168.203.155@tcp110 [8/256/0/180] [ 9859.250992] LNet: Added LNI 192.168.203.155@tcp111 [8/256/0/180] [ 9859.526690] LNet: Added LNI 192.168.203.155@tcp112 [8/256/0/180] [ 9859.810255] LNet: Added LNI 192.168.203.155@tcp113 [8/256/0/180] [ 9860.095520] LNet: Added LNI 192.168.203.155@tcp114 [8/256/0/180] [ 9860.382528] LNet: Added LNI 192.168.203.155@tcp115 [8/256/0/180] [ 9860.658071] LNet: Added LNI 192.168.203.155@tcp116 [8/256/0/180] [ 9860.930146] LNet: Added LNI 192.168.203.155@tcp117 [8/256/0/180] [ 9861.215477] LNet: Added LNI 192.168.203.155@tcp118 [8/256/0/180] [ 9861.490828] LNet: Added LNI 192.168.203.155@tcp119 [8/256/0/180] [ 9861.761370] LNet: Added LNI 192.168.203.155@tcp120 [8/256/0/180] [ 9862.051394] LNet: Added LNI 192.168.203.155@tcp121 [8/256/0/180] [ 9862.329836] LNet: Added LNI 192.168.203.155@tcp122 [8/256/0/180] [ 9862.612843] LNet: Added LNI 192.168.203.155@tcp123 [8/256/0/180] [ 9862.900027] LNet: Added LNI 192.168.203.155@tcp124 [8/256/0/180] [ 9863.184479] LNet: Added LNI 192.168.203.155@tcp125 [8/256/0/180] [ 9863.466214] LNet: Added LNI 192.168.203.155@tcp126 [8/256/0/180] [ 9863.748224] LNet: Added LNI 192.168.203.155@tcp127 [8/256/0/180] [ 9864.013828] LNet: Added LNI 192.168.203.155@tcp128 [8/256/0/180] [ 9864.301620] LNet: Added LNI 192.168.203.155@tcp129 [8/256/0/180] [ 9864.593851] LNet: Added LNI 192.168.203.155@tcp130 [8/256/0/180] [ 9864.870402] LNet: Added LNI 192.168.203.155@tcp131 [8/256/0/180] [ 9865.150199] LNet: Added LNI 192.168.203.155@tcp132 [8/256/0/180] [ 9865.422120] LNet: Added LNI 192.168.203.155@tcp133 [8/256/0/180] [ 9865.695806] LNet: Added LNI 192.168.203.155@tcp134 [8/256/0/180] [ 9865.978107] LNet: Added LNI 192.168.203.155@tcp135 [8/256/0/180] [ 9868.013334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9868.018916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9869.053323] Lustre: 463259:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [ 9869.086466] LNetError: 463279:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [ 9869.093224] LNetError: Protocol error connecting to 192.168.203.155@tcp117 at host 192.168.203.155:988: is it running a compatible version of Lustre? [ 9869.096161] LNetError: Skipped 2 previous similar messages [ 9869.144859] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9869.151081] LNet: 4 local NIs in recovery (showing 4): 192.168.203.155@tcp124, 192.168.203.155@tcp125, 192.168.203.155@tcp128, 192.168.203.155@tcp127 [ 9869.155501] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9869.192933] Lustre: lustre-MDT0000: new disk, initializing [ 9869.213705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9869.218818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9870.113087] LNetError: 463279:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [ 9870.118062] LNetError: 463279:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 6 previous similar messages [ 9870.122327] LNetError: Protocol error connecting to 192.168.203.155@tcp102 at host 192.168.203.155:988: is it running a compatible version of Lustre? [ 9870.126120] LNetError: Skipped 3 previous similar messages [ 9870.471559] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9872.226268] LNetError: 460722:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [ 9872.229260] LNetError: 460722:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 4 previous similar messages [ 9872.231484] LNetError: Protocol error connecting to 192.168.203.155@tcp127 at host 192.168.203.155:988: is it running a compatible version of Lustre? [ 9872.234262] LNetError: Skipped 4 previous similar messages [ 9874.404435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9874.451278] Lustre: 463275:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9874.476157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9874.477736] Lustre: Skipped 1 previous similar message [ 9874.514228] Lustre: lustre-MDT0001: new disk, initializing [ 9874.525407] Lustre: 464077:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [ 9874.527883] Lustre: 464077:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9874.567224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9874.578803] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9874.581617] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9875.785357] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9876.321317] LNetError: 463285:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [ 9876.324612] LNetError: Protocol error connecting to 192.168.203.155@tcp127 at host 192.168.203.155:988: is it running a compatible version of Lustre? [ 9877.488780] Lustre: 464461:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [ 9877.491179] Lustre: 464461:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 9879.240481] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9880.919918] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9882.971567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9883.085986] Lustre: lustre-OST0000: new disk, initializing [ 9883.087557] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9883.089953] Lustre: 465249:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9883.121105] Lustre: 465254:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [ 9883.123828] Lustre: 465254:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 9883.139115] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9884.485560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9884.490853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9884.512876] LNetError: 460724:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [ 9884.515635] LNetError: Protocol error connecting to 192.168.203.155@tcp127 at host 192.168.203.155:988: is it running a compatible version of Lustre? [ 9884.536757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9884.917837] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9888.538186] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9894.882727] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9894.884377] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9894.885060] LustreError: Skipped 1 previous similar message [ 9894.891769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9894.895045] Lustre: Skipped 1 previous similar message [ 9897.042375] Lustre: server umount lustre-OST0000 complete [ 9900.000656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9900.001385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9900.005267] Lustre: Skipped 2 previous similar messages [ 9900.008099] Lustre: Skipped 1 previous similar message [ 9900.895915] LNetError: 463282:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [ 9900.898723] LNetError: Protocol error connecting to 192.168.203.155@tcp127 at host 192.168.203.155:988: is it running a compatible version of Lustre? [ 9904.472915] Lustre: server umount lustre-MDT0000 complete [ 9905.632339] LustreError: 465247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147228 with bad export cookie 3876136944766666020 [ 9905.635810] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9905.635910] LustreError: 465247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9905.743398] Lustre: server umount lustre-MDT0001 complete [ 9909.040378] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9910.048553] Key type lgssc unregistered [ 9910.210392] LNet: 467160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9910.213254] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9910.215444] LNet: Removed LNI 192.168.203.155@tcp [ 9911.263231] LNet: Removed LNI 192.168.203.155@tcp100 [ 9911.264576] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9911.266780] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9911.268936] LNet: Removed LNI 192.168.203.155@tcp101 [ 9912.287176] LNet: Removed LNI 192.168.203.155@tcp102 [ 9912.288424] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9912.290351] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 9912.292458] LNet: Removed LNI 192.168.203.155@tcp103 [ 9913.311143] LNet: Removed LNI 192.168.203.155@tcp104 [ 9914.335115] LNet: Removed LNI 192.168.203.155@tcp105 [ 9914.336434] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9914.338330] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 9914.340376] LNet: Removed LNI 192.168.203.155@tcp106 [ 9915.359126] LNet: Removed LNI 192.168.203.155@tcp107 [ 9916.383123] LNet: Removed LNI 192.168.203.155@tcp108 [ 9917.407108] LNet: Removed LNI 192.168.203.155@tcp109 [ 9918.431116] LNet: Removed LNI 192.168.203.155@tcp110 [ 9918.433232] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9918.436735] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 9918.440566] LNet: Removed LNI 192.168.203.155@tcp111 [ 9919.455111] LNet: Removed LNI 192.168.203.155@tcp112 [ 9920.479186] LNet: Removed LNI 192.168.203.155@tcp113 [ 9921.503137] LNet: Removed LNI 192.168.203.155@tcp114 [ 9922.527097] LNet: Removed LNI 192.168.203.155@tcp115 [ 9923.551159] LNet: Removed LNI 192.168.203.155@tcp116 [ 9924.575141] LNet: Removed LNI 192.168.203.155@tcp117 [ 9925.599155] LNet: Removed LNI 192.168.203.155@tcp118 [ 9926.623151] LNet: Removed LNI 192.168.203.155@tcp119 [ 9926.625613] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9926.629708] LNetError: 467160:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 9926.634073] LNet: Removed LNI 192.168.203.155@tcp120 [ 9927.647166] LNet: Removed LNI 192.168.203.155@tcp121 [ 9928.671145] LNet: Removed LNI 192.168.203.155@tcp122 [ 9929.695264] LNet: Removed LNI 192.168.203.155@tcp123 [ 9930.719137] LNet: Removed LNI 192.168.203.155@tcp124 [ 9931.743117] LNet: Removed LNI 192.168.203.155@tcp125 [ 9932.767123] LNet: Removed LNI 192.168.203.155@tcp126 [ 9932.768350] LNet: Removed LNI 192.168.203.155@tcp127 [ 9933.791175] LNet: Removed LNI 192.168.203.155@tcp128 [ 9934.815202] LNet: Removed LNI 192.168.203.155@tcp129 [ 9934.817512] LNet: Removed LNI 192.168.203.155@tcp130 [ 9934.819271] LNet: Removed LNI 192.168.203.155@tcp131 [ 9934.820405] LNet: Removed LNI 192.168.203.155@tcp132 [ 9934.821412] LNet: Removed LNI 192.168.203.155@tcp133 [ 9934.822383] LNet: Removed LNI 192.168.203.155@tcp134 [ 9934.829529] LNet: Removed LNI 192.168.203.155@tcp135 [ 9935.118874] Key type .llcrypt unregistered [ 9935.119881] Key type ._llcrypt unregistered [ 9942.377084] Key type ._llcrypt registered [ 9942.378232] Key type .llcrypt registered [ 9942.414417] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9948.256625] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9948.569024] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9948.629680] alg: No test for adler32 (adler32-zlib) [ 9949.477202] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 9949.560680] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9951.143132] Key type lgssc registered [ 9951.493327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9962.147468] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:41:25 (1783147285) [ 9967.978801] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9969.029941] Key type lgssc unregistered [ 9969.160444] LNet: 471218:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9969.163806] LNetError: 471218:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9969.174396] LNet: Removed LNI 192.168.203.155@tcp [ 9969.496033] Key type .llcrypt unregistered [ 9969.497115] Key type ._llcrypt unregistered [ 9975.696196] Key type ._llcrypt registered [ 9975.697246] Key type .llcrypt registered [ 9975.735989] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9976.095976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9976.159911] alg: No test for adler32 (adler32-zlib) [ 9977.010086] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [ 9977.107857] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9978.687122] Key type lgssc registered [ 9979.046071] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9982.007917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9982.012844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9983.106825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9983.118744] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9983.153511] Lustre: lustre-MDT0000: new disk, initializing [ 9983.174844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9983.180313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9984.488577] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9988.428428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9988.469359] Lustre: 472408:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9988.496479] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9988.498099] Lustre: Skipped 1 previous similar message [ 9988.535301] Lustre: lustre-MDT0001: new disk, initializing [ 9988.571750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9988.583390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9988.586733] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9989.861245] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9993.371158] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9995.007971] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9997.032345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9997.128499] Lustre: lustre-OST0000: new disk, initializing [ 9997.131403] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9997.135673] Lustre: 474375:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9997.167403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9998.857448] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10001.400858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10001.404575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10001.437965] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10002.278170] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10003.356455] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lnet_if_list [10004.827537] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [10004.837198] LNet: Added LNI 192.168.203.155@tcp10 [8/256/0/180] [10006.385920] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [10006.395083] LNet: Added LNI 192.168.203.155@tcp11 [8/256/0/180] [10007.616569] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [10007.626961] LNet: Added LNI 192.168.203.155@tcp12 [8/256/0/180] [10008.911754] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [10008.921749] LNet: Added LNI 192.168.203.155@tcp13 [8/256/0/180] [10010.259964] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [10010.269849] LNet: Added LNI 192.168.203.155@tcp14 [8/256/0/180] [10011.595208] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [10011.607601] LNet: Added LNI 192.168.203.155@tcp15 [8/256/0/180] [10013.043561] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [10013.054197] LNet: Added LNI 192.168.203.155@tcp16 [8/256/0/180] [10014.559523] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [10014.569569] LNet: Added LNI 192.168.203.155@tcp17 [8/256/0/180] [10016.095641] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [10016.106341] LNet: Added LNI 192.168.203.155@tcp18 [8/256/0/180] [10017.663351] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [10017.673036] LNet: Added LNI 192.168.203.155@tcp19 [8/256/0/180] [10019.253898] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [10019.264684] LNet: Added LNI 192.168.203.155@tcp20 [8/256/0/180] [10020.840807] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [10020.851294] LNet: Added LNI 192.168.203.155@tcp21 [8/256/0/180] [10022.422483] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [10022.431324] LNet: Added LNI 192.168.203.155@tcp22 [8/256/0/180] [10023.991267] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [10024.002604] LNet: Added LNI 192.168.203.155@tcp23 [8/256/0/180] [10025.678100] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [10025.687523] LNet: Added LNI 192.168.203.155@tcp24 [8/256/0/180] [10027.424810] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [10027.434585] LNet: Added LNI 192.168.203.155@tcp25 [8/256/0/180] [10029.226850] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [10029.241061] LNet: Added LNI 192.168.203.155@tcp26 [8/256/0/180] [10031.058251] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [10031.068492] LNet: Added LNI 192.168.203.155@tcp27 [8/256/0/180] [10032.912783] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [10032.922687] LNet: Added LNI 192.168.203.155@tcp28 [8/256/0/180] [10034.795992] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [10034.806742] LNet: Added LNI 192.168.203.155@tcp29 [8/256/0/180] [10036.767513] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [10036.778798] LNet: Added LNI 192.168.203.155@tcp30 [8/256/0/180] [10038.784565] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [10038.797089] LNet: Added LNI 192.168.203.155@tcp31 [8/256/0/180] [10040.897408] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [10040.908977] LNet: Added LNI 192.168.203.155@tcp32 [8/256/0/180] [10042.981471] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [10042.991505] LNet: Added LNI 192.168.203.155@tcp33 [8/256/0/180] [10044.863800] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [10044.873609] LNet: Added LNI 192.168.203.155@tcp34 [8/256/0/180] [10046.884698] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [10046.896939] LNet: Added LNI 192.168.203.155@tcp35 [8/256/0/180] [10049.084495] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [10049.093864] LNet: Added LNI 192.168.203.155@tcp36 [8/256/0/180] [10051.099712] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [10051.109479] LNet: Added LNI 192.168.203.155@tcp37 [8/256/0/180] [10053.225311] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [10053.234276] LNet: Added LNI 192.168.203.155@tcp38 [8/256/0/180] [10055.239117] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [10055.249117] LNet: Added LNI 192.168.203.155@tcp39 [8/256/0/180] [10057.453744] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [10057.464454] LNet: Added LNI 192.168.203.155@tcp40 [8/256/0/180] [10059.835751] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [10059.849869] LNet: Added LNI 192.168.203.155@tcp41 [8/256/0/180] [10062.092607] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [10062.102080] LNet: Added LNI 192.168.203.155@tcp42 [8/256/0/180] [10064.258158] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [10064.268246] LNet: Added LNI 192.168.203.155@tcp43 [8/256/0/180] [10066.718848] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [10066.730768] LNet: Added LNI 192.168.203.155@tcp44 [8/256/0/180] [10069.220129] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [10069.233674] LNet: Added LNI 192.168.203.155@tcp45 [8/256/0/180] [10071.747829] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [10071.761084] LNet: Added LNI 192.168.203.155@tcp46 [8/256/0/180] [10074.122540] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [10074.131944] LNet: Added LNI 192.168.203.155@tcp47 [8/256/0/180] [10076.512971] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [10076.522389] LNet: Added LNI 192.168.203.155@tcp48 [8/256/0/180] [10078.855390] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [10078.865610] LNet: Added LNI 192.168.203.155@tcp49 [8/256/0/180] [10081.210181] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [10081.219806] LNet: Added LNI 192.168.203.155@tcp50 [8/256/0/180] [10088.417486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10088.419347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10088.421095] Lustre: Skipped 2 previous similar messages [10088.423227] Lustre: Skipped 1 previous similar message [10093.536780] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10093.539745] Lustre: Skipped 2 previous similar messages [10094.038492] Lustre: server umount lustre-MDT0000 complete [10095.173980] LustreError: 474374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147418 with bad export cookie 17716010944442069860 [10095.177260] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10095.177364] LustreError: 474374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10095.277504] Lustre: server umount lustre-MDT0001 complete [10101.895570] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10104.644055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10105.695066] LustreError: 498540:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [10105.697807] LustreError: 498540:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000439d5bc8 x1869765463321856/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1783147428 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10105.705270] LustreError: 498540:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [10105.823349] LustreError: 471679:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000456bc632 x1869765463322624/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10105.930053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10107.081809] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10109.613644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.700685] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.962337] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10113.117395] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10114.016076] Lustre: 471682:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783147421/real 1783147421] req@00000000e4601bff x1869765463322240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783147437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10114.875488] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10115.043961] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10116.063171] Lustre: 471683:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783147423/real 1783147423] req@00000000c9b78647 x1869765463322368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783147439 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10118.117339] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10119.135121] Lustre: 471680:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783147426/real 1783147426] req@000000002d32f713 x1869765463322496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783147442 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10121.183155] Lustre: 471681:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1783147428/real 1783147428] req@00000000d1f3c83a x1869765463322752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1783147444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10123.232915] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10123.235413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10123.236396] Lustre: Skipped 1 previous similar message [10123.241512] Lustre: Skipped 1 previous similar message [10128.272843] Lustre: server umount lustre-OST0000 complete [10130.399898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10130.402267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10130.405977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10135.567993] Lustre: server umount lustre-MDT0000 complete [10136.737384] LustreError: 472400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147459 with bad export cookie 17716010944442071869 [10136.740091] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10136.741453] LustreError: 472400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10136.848142] Lustre: server umount lustre-MDT0001 complete [10140.078841] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10141.109001] Key type lgssc unregistered [10141.264122] LNet: 501331:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10141.266904] LNetError: 501331:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10141.268957] LNet: Removed LNI 192.168.203.155@tcp [10141.270621] LNet: Removed LNI 192.168.203.155@tcp10 [10141.271735] LNet: Removed LNI 192.168.203.155@tcp11 [10141.272846] LNet: Removed LNI 192.168.203.155@tcp12 [10141.274305] LNet: Removed LNI 192.168.203.155@tcp13 [10141.275463] LNet: Removed LNI 192.168.203.155@tcp14 [10141.276565] LNet: Removed LNI 192.168.203.155@tcp15 [10141.277759] LNet: Removed LNI 192.168.203.155@tcp16 [10141.278837] LNet: Removed LNI 192.168.203.155@tcp17 [10141.280100] LNet: Removed LNI 192.168.203.155@tcp18 [10141.281225] LNet: Removed LNI 192.168.203.155@tcp19 [10141.282468] LNet: Removed LNI 192.168.203.155@tcp20 [10141.283647] LNet: Removed LNI 192.168.203.155@tcp21 [10141.284814] LNet: Removed LNI 192.168.203.155@tcp22 [10141.286115] LNet: Removed LNI 192.168.203.155@tcp23 [10141.287292] LNet: Removed LNI 192.168.203.155@tcp24 [10141.288357] LNet: Removed LNI 192.168.203.155@tcp25 [10141.289561] LNet: Removed LNI 192.168.203.155@tcp26 [10141.290680] LNet: Removed LNI 192.168.203.155@tcp27 [10141.291864] LNet: Removed LNI 192.168.203.155@tcp28 [10141.292931] LNet: Removed LNI 192.168.203.155@tcp29 [10141.294108] LNet: Removed LNI 192.168.203.155@tcp30 [10141.295268] LNet: Removed LNI 192.168.203.155@tcp31 [10141.296326] LNet: Removed LNI 192.168.203.155@tcp32 [10141.297564] LNet: Removed LNI 192.168.203.155@tcp33 [10141.298706] LNet: Removed LNI 192.168.203.155@tcp34 [10141.299758] LNet: Removed LNI 192.168.203.155@tcp35 [10141.300963] LNet: Removed LNI 192.168.203.155@tcp36 [10141.302086] LNet: Removed LNI 192.168.203.155@tcp37 [10141.303326] LNet: Removed LNI 192.168.203.155@tcp38 [10141.304416] LNet: Removed LNI 192.168.203.155@tcp39 [10141.305595] LNet: Removed LNI 192.168.203.155@tcp40 [10141.306756] LNet: Removed LNI 192.168.203.155@tcp41 [10141.307808] LNet: Removed LNI 192.168.203.155@tcp42 [10141.308867] LNet: Removed LNI 192.168.203.155@tcp43 [10141.309981] LNet: Removed LNI 192.168.203.155@tcp44 [10141.311093] LNet: Removed LNI 192.168.203.155@tcp45 [10141.312170] LNet: Removed LNI 192.168.203.155@tcp46 [10141.313397] LNet: Removed LNI 192.168.203.155@tcp47 [10141.314516] LNet: Removed LNI 192.168.203.155@tcp48 [10141.315657] LNet: Removed LNI 192.168.203.155@tcp49 [10141.322356] LNet: Removed LNI 192.168.203.155@tcp50 [10141.604949] Key type .llcrypt unregistered [10141.605933] Key type ._llcrypt unregistered [10147.636735] Key type ._llcrypt registered [10147.637798] Key type .llcrypt registered [10147.671749] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10153.318608] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10153.649590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10153.685057] alg: No test for adler32 (adler32-zlib) [10154.559388] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [10154.650232] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10156.231090] Key type lgssc registered [10156.568454] Lustre: Echo OBD driver; http://www.lustre.org/ [10167.354115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10167.361236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.442500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10168.452305] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10168.491630] Lustre: lustre-MDT0000: new disk, initializing [10168.512548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10168.517893] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10169.823191] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10173.729561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10173.773956] Lustre: 504502:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10173.797846] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10173.799478] Lustre: Skipped 1 previous similar message [10173.833994] Lustre: lustre-MDT0001: new disk, initializing [10173.867768] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10173.879234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10173.882679] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10175.073829] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10178.508334] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10180.136575] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10182.402565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10182.473728] Lustre: *** cfs_fail_loc=90f, val=0*** [10182.501838] Lustre: lustre-OST0000: new disk, initializing [10182.504844] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10182.509257] Lustre: 506516:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10182.541735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10184.268240] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10186.231104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10186.234250] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10186.263672] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10187.693212] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10191.329893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10191.330796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10191.331544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10191.332034] LustreError: Skipped 1 previous similar message [10191.338302] Lustre: Skipped 1 previous similar message [10195.924851] Lustre: server umount lustre-OST0000 complete [10199.519852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10199.522677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10199.527534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10199.529608] Lustre: Skipped 1 previous similar message [10201.568222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10201.568571] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10203.227698] Lustre: server umount lustre-MDT0000 complete [10204.432323] LustreError: 504493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147527 with bad export cookie 14691560581440406049 [10204.435304] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10204.437926] LustreError: 504493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10204.554561] Lustre: server umount lustre-MDT0001 complete [10207.925754] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10208.891951] Key type lgssc unregistered [10209.012378] LNet: 508418:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10209.014970] LNetError: 508418:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10209.022279] LNet: Removed LNI 192.168.203.155@tcp [10209.334327] Key type .llcrypt unregistered [10209.335767] Key type ._llcrypt unregistered [10217.050608] Key type ._llcrypt registered [10217.051692] Key type .llcrypt registered [10217.087535] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10217.477160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10217.483661] alg: No test for adler32 (adler32-zlib) [10218.320662] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [10218.399342] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10219.975309] Key type lgssc registered [10220.513543] Lustre: Echo OBD driver; http://www.lustre.org/ [10223.465819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10223.472750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10224.614205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10225.876981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10228.671677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10228.789205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10230.043087] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10232.760596] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10234.454815] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10236.869468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10237.000954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10239.097993] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10241.411341] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10256.489429] loop0: detected capacity change from 0 to 204800000 [10256.600223] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10256.603714] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10256.607213] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10258.856125] loop0: detected capacity change from 0 to 204800000 [10258.928070] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10258.931703] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10258.935584] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10261.130635] loop0: detected capacity change from 0 to 204800000 [10263.699630] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10263.787463] Lustre: fs15246-OST0000: new disk, initializing [10263.791889] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [10263.797959] Lustre: 513295:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10263.832523] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10266.310952] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10271.379529] loop1: detected capacity change from 0 to 204800000 [10273.892287] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10273.956728] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [10273.998525] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [10274.051069] Lustre: fs15246-MDT0000: new disk, initializing [10274.120824] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10274.138273] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10274.204552] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10275.232127] Lustre: 514414:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000009c69fcbb x1869765715985920/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1783147614 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [10275.756110] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10284.322865] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10284.326583] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10284.336323] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10286.060449] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.069522] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10291.170833] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [10291.629064] Lustre: server umount fs15246-MDT0000 complete [10296.095376] Lustre: server umount fs15246-OST0000 complete [10300.895917] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10300.900092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10300.907471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10306.531664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10306.536161] Lustre: Skipped 3 previous similar messages [10311.650513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10311.654588] Lustre: Skipped 1 previous similar message [10315.231127] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10315.309945] Lustre: server umount lustre-OST0000 complete [10320.863859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10320.867085] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10320.871336] Lustre: Skipped 1 previous similar message [10320.874604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10323.055785] Lustre: server umount lustre-MDT0000 complete [10324.744222] LustreError: 509661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147647 with bad export cookie 15274345855488232497 [10324.751971] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10324.752077] LustreError: 509661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10324.903389] Lustre: server umount lustre-MDT0001 complete [10328.736189] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10329.992414] Key type lgssc unregistered [10330.138700] LNet: 517511:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10330.143953] LNetError: 517511:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10330.155559] LNet: Removed LNI 192.168.203.155@tcp [10330.579105] Key type .llcrypt unregistered [10330.581551] Key type ._llcrypt unregistered [10338.762477] Key type ._llcrypt registered [10338.764380] Key type .llcrypt registered [10338.814864] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10339.177686] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10339.235816] alg: No test for adler32 (adler32-zlib) [10340.090696] Lustre: Lustre: Build Version: 2.17.54_85_gf55a92a [10340.184325] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10341.783158] Key type lgssc registered [10342.378510] Lustre: Echo OBD driver; http://www.lustre.org/ [10357.279498] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10360.450954] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10376.370400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10376.380511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10377.483123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10377.495339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10377.535244] Lustre: lustre-MDT0000: new disk, initializing [10377.558430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10377.566142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10379.279586] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10384.678949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10384.760931] Lustre: 521948:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10384.807628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10384.811683] Lustre: Skipped 1 previous similar message [10384.864348] Lustre: lustre-MDT0001: new disk, initializing [10384.914219] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10384.938258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10384.946940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10386.179105] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10390.297767] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10392.348208] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10395.069767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10395.182777] Lustre: lustre-OST0000: new disk, initializing [10395.184446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10395.187023] Lustre: 523915:0:(osd_compat.c:1271:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10395.218621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10397.375072] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10399.737083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10399.740176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10399.771927] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10401.760675] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10403.344897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10403.414125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10404.607717] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10404.701098] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10409.954228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10409.954528] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10409.956334] LustreError: Skipped 1 previous similar message [10409.960281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10409.962426] Lustre: Skipped 1 previous similar message [10411.662374] Lustre: server umount lustre-OST0000 complete [10415.072788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10415.073525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10415.078494] Lustre: Skipped 2 previous similar messages [10415.081588] Lustre: Skipped 1 previous similar message [10420.192275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10420.194127] Lustre: Skipped 1 previous similar message [10425.312182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10425.314267] Lustre: Skipped 1 previous similar message [10427.359090] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10427.420694] Lustre: server umount lustre-MDT0000 complete [10428.599265] LustreError: 522743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1783147751 with bad export cookie 14156167464590072038 [10428.603076] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10428.603475] LustreError: 522743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10428.718607] Lustre: server umount lustre-MDT0001 complete [10436.103367] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10439.132138] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10450.686497] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 10102 sec ========= 02:49:34 (1783147774) [10451.200331] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:49:34 (1783147774) === [10452.218925] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:49:35 (1783147775) === [10466.060718] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10467.315041] Key type lgssc unregistered [10467.456393] LNet: 530549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10467.460699] LNetError: 530549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10467.472529] LNet: Removed LNI 192.168.203.155@tcp [10467.836895] Key type .llcrypt unregistered [10467.838962] Key type ._llcrypt unregistered