[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 3.0.0 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 653568483 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f53f0-0x000f53ff] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5200 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1D87 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1C23 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001BE3 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1C97 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1D27 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1D5F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1c23-0xbffe1c96] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1c22] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1c97-0xbffe1d26] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1d27-0xbffe1d5e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1d5f-0xbffe1d86] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001016] APIC: Switch to symmetric I/O mode setup [ 0.002399] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003000] kvm-guest: setup PV IPIs [ 0.003000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.003000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.003021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.004011] pid_max: default: 32768 minimum: 301 [ 0.005145] LSM: Security Framework initializing [ 0.006065] Yama: becoming mindful. [ 0.008021] SELinux: Initializing. [ 0.009069] *** VALIDATE selinux *** [ 0.018586] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025221] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026166] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027123] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028146] *** VALIDATE tmpfs *** [ 0.029536] *** VALIDATE proc *** [ 0.030421] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032263] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.033155] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.034012] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.035035] Spectre V2 : User space: Vulnerable [ 0.036010] Speculative Store Bypass: Vulnerable [ 0.039148] debug: unmapping init [mem 0xffffffffb9459000-0xffffffffb9460fff] [ 0.041185] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.042703] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.043020] ... version: 2 [ 0.044019] ... bit width: 48 [ 0.045012] ... generic registers: 4 [ 0.046018] ... value mask: 0000ffffffffffff [ 0.047026] ... max period: 00007fffffffffff [ 0.048020] ... fixed-purpose events: 3 [ 0.049012] ... event mask: 000000070000000f [ 0.051208] rcu: Hierarchical SRCU implementation. [ 0.053559] smp: Bringing up secondary CPUs ... [ 0.055077] x86: Booting SMP configuration: [ 0.056039] .... node #0, CPUs: #1 #2 #3 [ 0.063535] smp: Brought up 1 node, 4 CPUs [ 0.065012] smpboot: Max logical packages: 1 [ 0.066012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.138673] node 0 deferred pages initialised in 69ms [ 0.144188] devtmpfs: initialized [ 0.147332] x86/mm: Memory block size: 128MB [ 0.154000] gcov: version magic: 0x41383552 [ 0.157294] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.162169] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165351] pinctrl core: initialized pinctrl subsystem [ 0.166600] [ 0.168013] ************************************************************* [ 0.172015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.175012] ** ** [ 0.178016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.181013] ** ** [ 0.184015] ** This means that this kernel is built to expose internal ** [ 0.190016] ** IOMMU data structures, which may compromise security on ** [ 0.193015] ** your system. ** [ 0.196017] ** ** [ 0.199014] ** If you see this message and you are not debugging the ** [ 0.201013] ** kernel, report this immediately to your vendor! ** [ 0.203012] ** ** [ 0.205018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.207012] ************************************************************* [ 0.210436] NET: Registered protocol family 16 [ 0.212504] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.213055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.216066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218735] cpuidle: using governor menu [ 0.223015] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.229076] PCI: Using configuration type 1 for base access [ 0.232139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.247089] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.251144] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.258086] cryptd: max_cpu_qlen set to 1000 [ 0.264336] ACPI: Added _OSI(Module Device) [ 0.265012] ACPI: Added _OSI(Processor Device) [ 0.266016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.267592] ACPI: Added _OSI(Processor Aggregator Device) [ 0.272230] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.280617] ACPI: Interpreter enabled [ 0.284069] ACPI: PM: (supports S0 S3 S4 S5) [ 0.287012] ACPI: Using IOAPIC for interrupt routing [ 0.289138] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.294766] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.310000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.313045] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.315018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.321101] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.328030] acpiphp: Slot [2] registered [ 0.330093] acpiphp: Slot [5] registered [ 0.333121] acpiphp: Slot [6] registered [ 0.335118] acpiphp: Slot [7] registered [ 0.338152] acpiphp: Slot [8] registered [ 0.342172] acpiphp: Slot [9] registered [ 0.344150] acpiphp: Slot [10] registered [ 0.347117] acpiphp: Slot [3] registered [ 0.351146] acpiphp: Slot [4] registered [ 0.357185] acpiphp: Slot [11] registered [ 0.358080] acpiphp: Slot [12] registered [ 0.360079] acpiphp: Slot [13] registered [ 0.361099] acpiphp: Slot [14] registered [ 0.362100] acpiphp: Slot [15] registered [ 0.364122] acpiphp: Slot [16] registered [ 0.366107] acpiphp: Slot [17] registered [ 0.369118] acpiphp: Slot [18] registered [ 0.370095] acpiphp: Slot [19] registered [ 0.372096] acpiphp: Slot [20] registered [ 0.374109] acpiphp: Slot [21] registered [ 0.375070] acpiphp: Slot [22] registered [ 0.376078] acpiphp: Slot [23] registered [ 0.377086] acpiphp: Slot [24] registered [ 0.379106] acpiphp: Slot [25] registered [ 0.381098] acpiphp: Slot [26] registered [ 0.382000] acpiphp: Slot [27] registered [ 0.382000] acpiphp: Slot [28] registered [ 0.383079] acpiphp: Slot [29] registered [ 0.384101] acpiphp: Slot [30] registered [ 0.385083] acpiphp: Slot [31] registered [ 0.386061] PCI host bridge to bus 0000:00 [ 0.387018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.388019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.390018] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.391548] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.394024] pci_bus 0000:00: root bus resource [mem 0x380000000000-0x38007fffffff window] [ 0.397024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.399172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.401940] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.405531] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.414012] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.418452] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.419018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.420016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.421016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.424215] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.426755] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.428033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.430711] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.434027] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.442017] pci 0000:00:02.0: reg 0x20: [mem 0x380000000000-0x380000003fff 64bit pref] [ 0.444000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.448000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.451015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.466018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.492020] pci 0000:00:05.0: reg 0x20: [mem 0x380000004000-0x380000007fff 64bit pref] [ 0.513669] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.527018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.534017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.565016] pci 0000:00:06.0: reg 0x20: [mem 0x380000008000-0x38000000bfff 64bit pref] [ 0.581624] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.594016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.599017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.623017] pci 0000:00:07.0: reg 0x20: [mem 0x38000000c000-0x38000000ffff 64bit pref] [ 0.643426] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.656017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.673018] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.692014] pci 0000:00:08.0: reg 0x20: [mem 0x380000010000-0x380000013fff 64bit pref] [ 0.707571] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.718016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.724013] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.742019] pci 0000:00:09.0: reg 0x20: [mem 0x380000014000-0x380000017fff 64bit pref] [ 0.750000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.756017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.767027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.785021] pci 0000:00:0a.0: reg 0x20: [mem 0x380000018000-0x38000001bfff 64bit pref] [ 0.798376] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.801484] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.804348] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.807380] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.810208] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.817010] iommu: Default domain type: Passthrough [ 0.817788] SCSI subsystem initialized [ 0.820107] ACPI: bus type USB registered [ 0.822124] usbcore: registered new interface driver usbfs [ 0.824075] usbcore: registered new interface driver hub [ 0.827072] usbcore: registered new device driver usb [ 0.829196] pps_core: LinuxPPS API ver. 1 registered [ 0.830008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.833074] PTP clock support registered [ 0.835833] EDAC MC: Ver: 3.0.0 [ 0.838093] PCI: Using ACPI for IRQ routing [ 0.839818] NetLabel: Initializing [ 0.841008] NetLabel: domain hash size = 128 [ 0.843014] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.845086] NetLabel: unlabeled traffic allowed by default [ 0.846490] vgaarb: loaded [ 0.848554] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.850006] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.859259] clocksource: Switched to clocksource kvm-clock [ 0.985911] VFS: Disk quotas dquot_6.6.0 [ 0.987590] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.990190] *** VALIDATE ramfs *** [ 0.991549] *** VALIDATE hugetlbfs *** [ 0.993282] pnp: PnP ACPI init [ 0.995838] pnp: PnP ACPI: found 6 devices [ 1.015165] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.019095] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.021716] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.024185] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.027097] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.029321] pci_bus 0000:00: resource 8 [mem 0x380000000000-0x38007fffffff window] [ 1.033981] NET: Registered protocol family 2 [ 1.040410] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.048145] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.057131] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.062482] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.065506] TCP: Hash tables configured (established 65536 bind 65536) [ 1.068326] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.071837] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.075084] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.078323] NET: Registered protocol family 1 [ 1.081798] RPC: Registered named UNIX socket transport module. [ 1.084406] RPC: Registered udp transport module. [ 1.086143] RPC: Registered tcp transport module. [ 1.088080] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.091344] NET: Registered protocol family 44 [ 1.093415] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.095743] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.097686] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.100082] PCI: CLS 0 bytes, default 64 [ 1.101517] Unpacking initramfs... [ 3.392956] debug: unmapping init [mem 0xffff89e6fcc54000-0xffff89e6fffbffff] [ 3.399172] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.402231] software IO TLB: mapped [mem 0x00000000a3e00000-0x00000000a7e00000] (64MB) [ 3.406475] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.983306] Initialise system trusted keyrings [ 3.984577] Key type blacklist registered [ 3.986676] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.994382] zbud: loaded [ 3.997104] *** VALIDATE nfs *** [ 3.998146] *** VALIDATE nfs4 *** [ 3.999338] pstore: using deflate compression [ 4.002275] Platform Keyring initialized [ 4.148835] NET: Registered protocol family 38 [ 4.153786] Key type asymmetric registered [ 4.156757] Asymmetric key parser 'x509' registered [ 4.158545] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.161510] io scheduler mq-deadline registered [ 4.163118] io scheduler kyber registered [ 4.164560] io scheduler bfq registered [ 4.166171] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.169380] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.172475] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.175228] ACPI: Power Button [PWRF] [ 4.353089] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.458283] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.748790] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.942910] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 5.086674] hrtimer: interrupt took 22085250 ns [ 5.414944] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 5.475463] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 5.665114] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 5.671910] Non-volatile memory driver v1.3 [ 5.673986] Linux agpgart interface v0.103 [ 5.716765] virtio_blk virtio1: [vda] 133424 512-byte logical blocks (68.3 MB/65.1 MiB) [ 5.719860] vda: detected capacity change from 0 to 68313088 [ 5.745153] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 5.748401] vdb: detected capacity change from 0 to 1073741824 [ 5.771361] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.773805] vdc: detected capacity change from 0 to 2621440000 [ 5.787509] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 5.789930] vdd: detected capacity change from 0 to 2621440000 [ 5.806287] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.808926] vde: detected capacity change from 0 to 4294967296 [ 5.824264] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 5.826809] vdf: detected capacity change from 0 to 4294967296 [ 5.832119] libphy: Fixed MDIO Bus: probed [ 5.856087] usbcore: registered new interface driver usbserial_generic [ 5.860900] usbserial: USB Serial support registered for generic [ 5.865873] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 5.875988] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 5.879686] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 5.884905] mousedev: PS/2 mouse device common for all mice [ 5.890909] rtc_cmos 00:05: RTC can wake from S4 [ 5.900382] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 5.904164] rtc_cmos 00:05: registered as rtc0 [ 5.910266] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 5.913337] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.918273] intel_pstate: CPU model not supported [ 5.923673] hid: raw HID events driver (C) Jiri Kosina [ 5.933107] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.934147] usbcore: registered new interface driver usbhid [ 5.934151] usbhid: USB HID core driver [ 5.934279] drop_monitor: Initializing network drop monitor service [ 5.934420] Initializing XFRM netlink socket [ 5.936880] NET: Registered protocol family 10 [ 5.958655] Segment Routing with IPv6 [ 5.961244] NET: Registered protocol family 17 [ 5.964217] mpls_gso: MPLS GSO support [ 5.972135] RAS: Correctable Errors collector initialized. [ 5.975346] AVX version of gcm_enc/dec engaged. [ 5.976851] AES CTR mode by8 optimization enabled [ 6.090644] sched_clock: Marking stable (6090328903, 0)->(7215616078, -1125287175) [ 6.093386] registered taskstats version 1 [ 6.095225] Loading compiled-in X.509 certificates [ 6.097687] zswap: loaded using pool lzo/zbud [ 6.125088] Key type big_key registered [ 6.140243] Key type encrypted registered [ 6.141801] ima: No TPM chip found, activating TPM-bypass! [ 6.144641] ima: Allocated hash algorithm: sha1 [ 6.145840] ima: No architecture policies found [ 6.147407] evm: Initialising EVM extended attributes: [ 6.148944] evm: security.selinux [ 6.150359] evm: security.ima [ 6.151255] evm: security.capability [ 6.152164] evm: HMAC attrs: 0x1 [ 6.155315] rtc_cmos 00:05: setting system clock to 2025-10-22 13:51:30 UTC (1761141090) [ 6.166762] debug: unmapping init [mem 0xffffffffba403000-0xffffffffba5fffff] [ 6.169490] debug: unmapping init [mem 0xffffffffb9182000-0xffffffffb9458fff] [ 6.178055] Write protecting the kernel read-only data: 28672k [ 6.181520] debug: unmapping init [mem 0xffffffffb7803000-0xffffffffb79fffff] [ 6.183913] debug: unmapping init [mem 0xffffffffb8114000-0xffffffffb81fffff] [ 6.230390] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 6.240919] systemd[1]: Detected virtualization kvm. [ 6.242693] systemd[1]: Detected architecture x86-64. [ 6.244268] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 6.272772] systemd[1]: No hostname configured. [ 6.274562] systemd[1]: Set hostname to . [ 6.276815] random: systemd: uninitialized urandom read (16 bytes read) [ 6.278983] systemd[1]: Initializing machine ID from random generator. [ 6.392304] random: ln: uninitialized urandom read (6 bytes read) [ 6.551364] random: systemd: uninitialized urandom read (16 bytes read) [ 6.553552] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 6.560474] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 6.569088] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Timers. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.683459] device-mapper: uevent: version 1.0.3 [ 7.687180] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.146471] virtio_net virtio0 ens2: renamed from eth0 [ 9.313233] scsi host0: ata_piix [ 9.330610] random: fast init done [ 9.425099] scsi host1: ata_piix [ 9.527379] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.532393] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.338146] dracut-initqueue[585]: RTNETLINK answers: File exists [ 14.663269] random: crng init done [ 14.664871] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 16.150572] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.296249] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.868107] SELinux: Disabled at runtime. [ 18.952049] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.960978] systemd[1]: Detected virtualization kvm. [ 18.963133] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 20.209983] systemd[1]: initrd-switch-root.service: Succeeded. [ 20.214642] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 20.221791] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 20.225260] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 20.228428] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 20.248792] systemd[1]: Starting Journal Service... Starting Journal Service... [ 20.269548] systemd[1]: Started Forward Password Requests to Wall Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ 20.428133] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Kernel Debug File System... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started udev Coldplug all Devices. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 21.632969] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 22.476782] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.484937] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 22.730493] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.786758] EDAC sbridge: Ver: 1.1.2 [ 26.282408] Key type dns_resolver registered [ 26.714681] NFS: Registering the id_resolver key type [ 26.717672] Key type id_resolver registered [ 26.719912] Key type id_legacy registered [* ] A start job is running for Configur…-only root support (7s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg134-server login: [ 53.133234] libcfs: loading out-of-tree module taints kernel. [ 53.143925] Key type ._llcrypt registered [ 53.145055] Key type .llcrypt registered [ 53.179647] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 60.101621] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 60.779348] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 60.786486] alg: No test for adler32 (adler32-zlib) [ 61.776799] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 62.082783] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 63.695199] Key type lgssc registered [ 64.363167] Lustre: Echo OBD driver; http://www.lustre.org/ [ 71.608036] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 72.943789] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 76.783513] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.463977] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 82.125394] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 87.304780] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 91.786495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 91.805659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 91.816335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 92.930654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 92.943322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 92.983855] Lustre: lustre-MDT0000: new disk, initializing [ 93.009655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.016165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 94.351425] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 99.368497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 99.393767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 99.416991] Lustre: 6483:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 99.426644] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 99.428183] Lustre: Skipped 1 previous similar message [ 99.460819] Lustre: lustre-MDT0001: new disk, initializing [ 99.478339] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 99.484493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 99.487902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 100.730137] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.974236] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 106.006281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 106.030896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 106.122066] Lustre: lustre-OST0000: new disk, initializing [ 106.124679] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 106.143713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 106.314351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 106.317465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 106.335849] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 108.049735] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 113.564793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.596753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 113.635889] Lustre: lustre-OST0001: new disk, initializing [ 113.638119] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 113.658603] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 115.781346] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.904420] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 122.345167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 122.350472] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 122.363746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 129.654884] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 136.728538] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing check_logdir /tmp/testlogs/ [ 138.189733] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing yml_node [ 139.845380] Lustre: DEBUG MARKER: Client: 2.16.59.1 [ 140.792766] Lustre: DEBUG MARKER: MDS: 2.16.59.1 [ 141.767451] Lustre: DEBUG MARKER: OSS: 2.16.59.1 [ 142.417467] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Oct 22 09:53:46 EDT 2025 [ 148.509188] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 149.028098] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 161.247621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 161.250445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 161.255207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 163.296314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.296374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 163.306290] Lustre: Skipped 1 previous similar message [ 163.309511] Lustre: Skipped 1 previous similar message [ 166.975194] LustreError: 12185:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 167.009208] Lustre: server umount lustre-MDT0000 complete [ 168.585653] LustreError: 6476:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141252 with bad export cookie 7280731061423290327 [ 168.587225] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 168.590311] LustreError: 6476:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 173.535831] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 173.536209] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 173.536617] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 173.544472] LustreError: 6490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 173.550519] Lustre: Skipped 2 previous similar messages [ 174.720596] LustreError: 12387:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 174.723014] LustreError: 12387:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 174.827521] Lustre: server umount lustre-MDT0001 complete [ 176.421766] LustreError: 12588:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 176.423885] LustreError: 12588:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 176.449159] Lustre: server umount lustre-OST0000 complete [ 178.074350] Lustre: server umount lustre-OST0001 complete [ 180.517621] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 184.664501] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 189.063603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 191.986306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 194.861678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 197.863352] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.877680] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.908381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.041733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 201.064303] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 201.120106] Lustre: lustre-MDT0000: new disk, initializing [ 201.163735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 201.176869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 202.951400] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 208.041218] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 208.067506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 208.110465] Lustre: 14967:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 208.242511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 209.923024] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 213.950389] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 215.612331] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 219.029549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 219.061681] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 219.203534] Lustre: lustre-OST0000: new disk, initializing [ 219.205882] Lustre: Skipped 1 previous similar message [ 219.210387] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 219.213741] Lustre: Skipped 2 previous similar messages [ 219.276167] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 219.280605] Lustre: Skipped 1 previous similar message [ 220.513624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 220.519559] Lustre: Skipped 1 previous similar message [ 220.523966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 220.588118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 221.886634] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 225.769734] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 227.624456] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 227.714310] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 229.518286] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 229.640038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 230.883723] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 230.884206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 230.885579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 230.885586] Lustre: Skipped 1 previous similar message [ 230.898387] Lustre: Skipped 1 previous similar message [ 236.004357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 236.007672] Lustre: Skipped 2 previous similar messages [ 236.524142] LustreError: 17904:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 236.529383] LustreError: 17904:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 236.609056] Lustre: server umount lustre-OST0000 complete [ 239.008683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 239.012187] Lustre: Skipped 1 previous similar message [ 244.192685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 244.194924] Lustre: Skipped 3 previous similar messages [ 244.292093] LustreError: 18106:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 244.296174] LustreError: 18106:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 244.340883] Lustre: server umount lustre-MDT0000 complete [ 245.823226] LustreError: 14959:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141330 with bad export cookie 7280731061423291846 [ 245.826127] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 245.828439] LustreError: 14959:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 246.009360] Lustre: server umount lustre-MDT0001 complete [ 247.628676] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 09:55:31 (1761141331) [ 248.589857] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 249.256249] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 09:55:33 (1761141333) [ 252.088647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 252.324996] LustreError: 18976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 252.360488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 254.243352] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 257.506825] LustreError: 18976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.359537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 260.468835] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 263.105101] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 264.804356] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 267.983751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 270.950475] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 273.428081] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 287.712028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 287.717022] LustreError: Skipped 1 previous similar message [ 287.719742] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 287.727724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 292.144747] LustreError: 21244:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 292.146982] LustreError: 21244:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 292.224860] Lustre: server umount lustre-OST0000 complete [ 294.367976] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 294.371350] LustreError: Skipped 1 previous similar message [ 300.199585] Lustre: server umount lustre-MDT0000 complete [ 301.986919] LustreError: 19696:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141386 with bad export cookie 7280731061423292728 [ 301.990904] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 301.992117] LustreError: 19696:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 302.177484] Lustre: server umount lustre-MDT0001 complete [ 307.218712] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 308.930770] Key type lgssc unregistered [ 309.166766] LNet: 22178:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 309.170760] LNetError: 22178:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 309.177563] LNet: Removed LNI 192.168.201.134@tcp [ 309.705375] Key type .llcrypt unregistered [ 309.707614] Key type ._llcrypt unregistered [ 321.616063] Key type ._llcrypt registered [ 321.618243] Key type .llcrypt registered [ 321.669936] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 322.089898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 322.140042] alg: No test for adler32 (adler32-zlib) [ 323.074612] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 323.223343] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 324.832381] Key type lgssc registered [ 325.435875] Lustre: Echo OBD driver; http://www.lustre.org/ [ 329.392464] loop: module loaded [ 329.425361] loop0: detected capacity change from 0 to 204800000 [ 329.568932] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 329.576313] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 329.744732] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 329.751946] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 329.780895] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 329.809862] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 329.834703] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 329.862085] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 332.470722] loop0: detected capacity change from 0 to 204800000 [ 332.618885] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 332.626301] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 332.631488] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 332.699433] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 335.124879] loop0: detected capacity change from 0 to 204800000 [ 337.840992] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 337.861739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 337.869765] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 338.988774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 339.005715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 339.051289] Lustre: lustre-MDT0000: new disk, initializing [ 339.090228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 339.102390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 341.021107] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 345.851855] loop1: detected capacity change from 0 to 204800000 [ 348.440417] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 348.468646] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 348.620788] Lustre: lustre-OST07c6: new disk, initializing [ 348.622885] Lustre: lustre-OST07c6: Not available for connect from 0@lo (not set up) [ 348.624772] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 348.629050] Lustre: Skipped 1 previous similar message [ 348.657180] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 351.027564] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 353.776077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 353.780571] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 353.793780] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 440.801164] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 440.801209] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 440.805886] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 445.920922] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 451.040313] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 452.063269] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 452.086088] LustreError: 26524:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 452.159923] Lustre: server umount lustre-OST07c6 complete [ 456.038047] LustreError: 27026:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 456.041314] LustreError: 27026:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 456.138585] Lustre: server umount lustre-MDT0000 complete [ 462.633165] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 463.947399] Key type lgssc unregistered [ 464.094380] LNet: 27858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 464.097526] LNetError: 27858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 464.104310] LNet: Removed LNI 192.168.201.134@tcp [ 464.464567] Key type .llcrypt unregistered [ 464.466434] Key type ._llcrypt unregistered [ 472.306463] Key type ._llcrypt registered [ 472.307303] Key type .llcrypt registered [ 472.344743] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 472.689729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 472.742056] alg: No test for adler32 (adler32-zlib) [ 473.640934] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 473.732856] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 475.319132] Key type lgssc registered [ 475.803407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 479.007852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 479.019668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 480.223673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 481.841949] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 485.351023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 485.527318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 487.124758] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 489.393730] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 490.930313] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 493.780960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 493.931370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.324793] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 498.454246] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 506.218359] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 511.451308] LustreError: 31305:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 515.553598] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.558181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.564914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.579636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 516.582546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 516.583095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 516.592969] Lustre: Skipped 1 previous similar message [ 520.879257] LustreError: 31557:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 520.965281] Lustre: server umount lustre-OST0000 complete [ 526.816491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 526.816494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 526.818099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 526.827536] Lustre: Skipped 1 previous similar message [ 528.769862] LustreError: 31758:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 528.773066] LustreError: 31758:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 528.804696] Lustre: server umount lustre-MDT0000 complete [ 530.301481] LustreError: 31760:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141614 with bad export cookie 7458394415645430812 [ 530.306853] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 530.307126] LustreError: 31760:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 530.371500] LustreError: 31961:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 530.375228] LustreError: 31961:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 530.466880] Lustre: server umount lustre-MDT0001 complete [ 535.057943] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 536.580263] Key type lgssc unregistered [ 536.749242] LNet: 32492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 536.755189] LNetError: 32492:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 536.764454] LNet: Removed LNI 192.168.201.134@tcp [ 537.149971] Key type .llcrypt unregistered [ 537.151284] Key type ._llcrypt unregistered [ 546.118430] Key type ._llcrypt registered [ 546.120021] Key type .llcrypt registered [ 546.186661] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 546.650813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 546.662116] alg: No test for adler32 (adler32-zlib) [ 547.555398] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 547.658970] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 549.255267] Key type lgssc registered [ 549.756439] Lustre: Echo OBD driver; http://www.lustre.org/ [ 553.169406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 553.179256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 554.364679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 555.944082] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 559.721935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 559.873601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 561.578976] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 563.840208] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 565.392720] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 568.329589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 568.515517] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.054888] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 573.398708] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 578.791828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 584.161845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 584.162757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 584.166182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 584.171077] Lustre: Skipped 1 previous similar message [ 588.973925] LustreError: 35980:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 589.045328] Lustre: server umount lustre-OST0000 complete [ 590.816181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 590.819884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 590.825807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 590.829329] Lustre: Skipped 1 previous similar message [ 594.401531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 594.401919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 596.673617] LustreError: 36182:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 596.676394] LustreError: 36182:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 596.716097] Lustre: server umount lustre-MDT0000 complete [ 598.276152] LustreError: 35338:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141682 with bad export cookie 12041002065498026661 [ 598.278762] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 598.280658] LustreError: 35338:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 598.351081] LustreError: 36384:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 598.353676] LustreError: 36384:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 598.455324] Lustre: server umount lustre-MDT0001 complete [ 602.922106] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 604.249720] Key type lgssc unregistered [ 604.411944] LNet: 36915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 604.416896] LNetError: 36915:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 604.425312] LNet: Removed LNI 192.168.201.134@tcp [ 604.825499] Key type .llcrypt unregistered [ 604.827326] Key type ._llcrypt unregistered [ 616.054332] Key type ._llcrypt registered [ 616.056550] Key type .llcrypt registered [ 616.110298] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 616.600423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 616.647065] alg: No test for adler32 (adler32-zlib) [ 617.535511] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 617.652758] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 619.255152] Key type lgssc registered [ 619.810271] Lustre: Echo OBD driver; http://www.lustre.org/ [ 623.254581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 623.264288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.478888] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 626.283129] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 630.110023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 630.336549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 632.090225] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 634.411275] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 635.908888] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 638.601738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 638.747646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 641.146189] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 643.702116] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 649.000703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 654.182149] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.34@tcp (stopping) [ 654.307070] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 654.313984] Lustre: Skipped 1 previous similar message [ 657.601583] LustreError: 40538:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 657.630297] Lustre: server umount lustre-MDT0000 complete [ 659.196764] LustreError: 38271:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141743 with bad export cookie 5901938041909192130 [ 659.200745] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.203120] LustreError: 38271:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 659.280499] LustreError: 39442:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 659.288483] LustreError: 39442:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 659.291124] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.34@tcp (stopping) [ 659.297608] Lustre: Skipped 3 previous similar messages [ 659.424649] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 659.430070] Lustre: Skipped 1 previous similar message [ 664.398717] LustreError: 38285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 664.398754] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.34@tcp (stopping) [ 664.405933] LustreError: 38285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 664.409035] Lustre: Skipped 1 previous similar message [ 665.353151] LustreError: 40741:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 665.356927] LustreError: 40741:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 665.459560] Lustre: server umount lustre-MDT0001 complete [ 692.191138] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 692.211412] LustreError: 40966:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 692.218661] LustreError: 40966:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 692.304266] Lustre: server umount lustre-OST0000 complete [ 700.141891] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 701.496809] Key type lgssc unregistered [ 701.658271] LNet: 41804:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 701.661932] LNetError: 41804:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 701.670754] LNet: Removed LNI 192.168.201.134@tcp [ 702.066076] Key type .llcrypt unregistered [ 702.067247] Key type ._llcrypt unregistered [ 713.569613] Key type ._llcrypt registered [ 713.571698] Key type .llcrypt registered [ 713.619235] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 714.054632] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 714.133651] alg: No test for adler32 (adler32-zlib) [ 715.017185] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 715.114855] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 716.711169] Key type lgssc registered [ 717.206614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 720.446075] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 720.458653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 721.651133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 723.255062] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 726.693796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 726.853480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 728.445671] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 730.817736] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 732.402409] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 735.029168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 735.182827] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 737.471180] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 739.784164] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 746.472082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 755.544030] Lustre: lustre-OST0000: Not available for connect from 192.168.201.34@tcp (stopping) [ 756.708368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 756.708615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.712019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 756.716357] Lustre: Skipped 1 previous similar message [ 758.833228] LustreError: 45446:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 758.899678] Lustre: server umount lustre-OST0000 complete [ 771.984863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 771.991850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 771.994733] Lustre: Skipped 1 previous similar message [ 777.120304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 777.122077] Lustre: Skipped 1 previous similar message [ 782.304108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 782.306320] Lustre: Skipped 1 previous similar message [ 786.399105] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 786.438338] LustreError: 45823:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 786.442024] LustreError: 45823:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 786.479301] Lustre: server umount lustre-MDT0000 complete [ 787.423885] LustreError: 43180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 787.429921] LustreError: 43180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 787.907347] LustreError: 43160:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141872 with bad export cookie 3047984862823056790 [ 787.910918] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 787.912772] LustreError: 43160:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 802.271259] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 802.310892] LustreError: 46025:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 802.314676] LustreError: 46025:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 802.412792] Lustre: server umount lustre-MDT0001 complete [ 806.772039] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 808.036400] Key type lgssc unregistered [ 808.193926] LNet: 46558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 808.198351] LNetError: 46558:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 808.211825] LNet: Removed LNI 192.168.201.134@tcp [ 808.607222] Key type .llcrypt unregistered [ 808.608947] Key type ._llcrypt unregistered [ 818.852771] Key type ._llcrypt registered [ 818.854738] Key type .llcrypt registered [ 818.912109] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 819.309109] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 819.335049] alg: No test for adler32 (adler32-zlib) [ 820.214790] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 820.311423] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 821.903178] Key type lgssc registered [ 822.337112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 825.307783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 825.319140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 826.501766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 828.006750] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 831.296328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 831.464131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 833.211822] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 835.704225] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 837.406664] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 840.238600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 840.398099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 842.965912] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 845.207331] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 851.689305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 858.170591] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 859.519031] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 870.369927] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-10-2210h05m53s [ 871.041706] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 872.419031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 872.420187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 872.421099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 872.424292] Lustre: Skipped 2 previous similar messages [ 873.440698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 873.444080] Lustre: Skipped 4 previous similar messages [ 876.383186] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.34@tcp (stopping) [ 879.585123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 879.587799] Lustre: Skipped 2 previous similar messages [ 885.730824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 885.733439] Lustre: Skipped 3 previous similar messages [ 886.239236] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 886.289218] LustreError: 50515:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 886.323718] Lustre: server umount lustre-MDT0000 complete [ 887.896648] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 888.654474] LustreError: 47928:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.663211] LustreError: 47928:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 890.769508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 890.837121] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 890.958909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 890.992590] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 892.006844] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 892.711173] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 894.133943] Lustre: DEBUG MARKER: Wait for df (35957) ... [ 894.929465] Lustre: DEBUG MARKER: done [ 897.508367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 897.510416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 897.514847] Lustre: Skipped 1 previous similar message [ 902.254169] LustreError: 51565:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 902.257953] LustreError: 51565:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 902.327338] Lustre: server umount lustre-OST0000 complete [ 907.744903] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 907.745428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 907.756695] Lustre: Skipped 1 previous similar message [ 909.961072] LustreError: 51772:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 909.965422] LustreError: 51772:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 910.001151] Lustre: server umount lustre-MDT0000 complete [ 911.690545] LustreError: 47911:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761141996 with bad export cookie 13827319272338488183 [ 911.694921] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 911.697311] LustreError: 47911:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 911.880291] Lustre: server umount lustre-MDT0001 complete [ 916.649836] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 918.072719] Key type lgssc unregistered [ 918.237678] LNet: 52505:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 918.242424] LNetError: 52505:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 918.257372] LNet: Removed LNI 192.168.201.134@tcp [ 918.662949] Key type .llcrypt unregistered [ 918.664640] Key type ._llcrypt unregistered [ 933.276470] Key type ._llcrypt registered [ 933.278633] Key type .llcrypt registered [ 933.344258] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 940.840684] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 941.401718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 941.420718] alg: No test for adler32 (adler32-zlib) [ 942.314569] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 942.420944] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 944.015316] Key type lgssc registered [ 944.487918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 948.599832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 951.469390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 954.133259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 956.730685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 959.583205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 959.606209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 959.614885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.716911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 960.736460] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 960.782707] Lustre: lustre-MDT0000: new disk, initializing [ 960.818317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 960.826278] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 962.448788] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 967.309399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 967.337792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 967.386073] Lustre: 56428:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 967.415371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 967.417520] Lustre: Skipped 1 previous similar message [ 967.457887] Lustre: lustre-MDT0001: new disk, initializing [ 967.496964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 967.511685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 967.516396] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 969.169502] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 972.890268] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 974.520497] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 977.584471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 977.616438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 977.755317] Lustre: lustre-OST0000: new disk, initializing [ 977.758717] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 977.797224] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 979.271062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 979.276887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 979.322625] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 980.230862] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 983.546306] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 985.146270] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 985.232395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 986.671136] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 986.760930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 989.667180] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 989.667992] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 989.671665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 989.675799] Lustre: Skipped 1 previous similar message [ 993.584603] LustreError: 59367:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 993.670407] Lustre: server umount lustre-OST0000 complete [ 998.368117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 998.372484] LustreError: Skipped 1 previous similar message [ 998.374345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 998.380482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 998.385222] Lustre: Skipped 1 previous similar message [ 999.904917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 999.905269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1001.281846] LustreError: 59570:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1001.284619] LustreError: 59570:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1001.318965] Lustre: server umount lustre-MDT0000 complete [ 1002.746723] LustreError: 56418:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142087 with bad export cookie 5469664226084295391 [ 1002.749684] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1002.750585] LustreError: 56418:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1002.837719] LustreError: 59773:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1002.841309] LustreError: 59773:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1002.933273] Lustre: server umount lustre-MDT0001 complete [ 1006.189992] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 10:08:10 (1761142090) [ 1008.801932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1008.994800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1010.416680] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1013.374210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1013.508336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1014.869591] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1016.954296] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1018.413827] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1021.183437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1023.559120] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1025.617811] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1038.578581] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-10-2210h08m42s [ 1039.129913] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1041.548122] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1042.392637] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1042.410631] LustreError: 60439:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1062.751742] Lustre: lustre-MDT0000: Client f9a47b37-b008-4b53-99f9-de6aa877c3b5 (at 192.168.201.34@tcp) reconnecting [ 1082.455111] LustreError: 60439:0:(mdt_handler.c:2334:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1083.129685] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1083.963865] Lustre: DEBUG MARKER: done [ 1086.946394] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1086.947246] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1086.947678] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1086.947682] Lustre: Skipped 1 previous similar message [ 1086.952325] Lustre: Skipped 1 previous similar message [ 1091.499155] LustreError: 63466:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1091.501052] LustreError: 63466:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1091.561296] Lustre: server umount lustre-OST0000 complete [ 1095.647826] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1095.650101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1095.656139] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1095.658125] Lustre: Skipped 1 previous similar message [ 1098.808484] LustreError: 63668:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1098.810623] LustreError: 63668:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1098.834597] Lustre: server umount lustre-MDT0000 complete [ 1100.026246] LustreError: 60423:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142184 with bad export cookie 5469664226084296259 [ 1100.029973] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1100.030907] LustreError: 60423:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1100.163333] Lustre: server umount lustre-MDT0001 complete [ 1103.554947] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1104.642203] Key type lgssc unregistered [ 1104.775347] LNet: 64400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1104.777392] LNetError: 64400:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1104.784520] LNet: Removed LNI 192.168.201.134@tcp [ 1105.068822] Key type .llcrypt unregistered [ 1105.070512] Key type ._llcrypt unregistered [ 1116.220425] Key type ._llcrypt registered [ 1116.221935] Key type .llcrypt registered [ 1116.264644] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 1122.389845] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1122.841799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1122.856686] alg: No test for adler32 (adler32-zlib) [ 1123.720739] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1123.810352] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1125.391154] Key type lgssc registered [ 1125.767757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1128.818430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1131.065668] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1132.965933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1134.943069] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1137.282419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1137.302509] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1137.308066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1138.386168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1138.395957] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1138.427484] Lustre: lustre-MDT0000: new disk, initializing [ 1138.452211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.459141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1139.827568] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1143.921872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1143.941328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1143.974314] Lustre: 68323:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1143.995069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1143.996804] Lustre: Skipped 1 previous similar message [ 1144.030337] Lustre: lustre-MDT0001: new disk, initializing [ 1144.065138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1144.078424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1144.082385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1145.319696] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1148.332936] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1149.722926] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1152.158582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1152.185027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1152.284590] Lustre: lustre-OST0000: new disk, initializing [ 1152.286275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1152.316375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1153.723580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1153.728707] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1153.775481] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1154.471009] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1157.682635] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1159.227951] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1159.314624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1160.862153] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1160.964384] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1164.261103] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1164.261280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1164.264982] LustreError: Skipped 1 previous similar message [ 1164.271955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1164.275625] Lustre: Skipped 1 previous similar message [ 1169.378458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1169.381014] Lustre: Skipped 1 previous similar message [ 1174.499135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1174.500931] Lustre: Skipped 1 previous similar message [ 1176.031145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1176.045089] LustreError: 71261:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 1176.103759] Lustre: server umount lustre-OST0000 complete [ 1179.616801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1179.617620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1179.620461] Lustre: Skipped 2 previous similar messages [ 1179.623604] Lustre: Skipped 1 previous similar message [ 1183.424437] LustreError: 71464:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1183.428175] LustreError: 71464:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1183.463697] Lustre: server umount lustre-MDT0000 complete [ 1184.641356] LustreError: 70068:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142268 with bad export cookie 15980123803744989924 [ 1184.644123] LustreError: 70068:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1184.644824] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1184.712664] LustreError: 71666:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1184.716242] LustreError: 71666:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1184.810038] Lustre: server umount lustre-MDT0001 complete [ 1187.530595] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 10:11:11 (1761142271) [ 1188.030121] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1188.574436] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 10:11:12 (1761142272) [ 1192.602661] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1194.153932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.314801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1194.655366] Lustre: Failing over lustre-MDT0000 [ 1194.657094] LustreError: 72740:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1194.661193] LustreError: 72740:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1194.664502] LustreError: 72740:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 1, retries 0, failed: rc = -5 [ 1194.725543] LustreError: 72791:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1194.728304] LustreError: 72791:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1194.810877] Lustre: server umount lustre-MDT0000 complete [ 1196.991419] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:11:20 (1761142280) [ 1199.234694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1199.395902] LustreError: 73359:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1199.403144] LustreError: 73359:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1199.413253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1200.858392] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1203.874648] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1205.545411] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1207.529777] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1208.841815] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1211.297531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1211.415919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1211.419086] Lustre: Skipped 1 previous similar message [ 1213.636855] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1215.605889] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1228.217101] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1233.889803] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1233.891376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1233.894276] Lustre: Skipped 2 previous similar messages [ 1233.896021] Lustre: Skipped 1 previous similar message [ 1235.967938] LustreError: 75726:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1235.970833] LustreError: 75726:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1235.997750] Lustre: server umount lustre-MDT0000 complete [ 1237.294265] LustreError: 75728:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142321 with bad export cookie 15980123803744991065 [ 1237.296184] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1237.299693] LustreError: 75728:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1237.444362] Lustre: server umount lustre-MDT0001 complete [ 1238.594831] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1240.241173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1243.246357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1247.135586] LustreError: 66536:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000020bc4c57 x1846691451956608/t0(0) o250->MGC192.168.201.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1247.251105] LustreError: 74965:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1247.281215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.305087] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1248.620558] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1251.613796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1251.712774] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1253.203309] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1253.798452] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1254.815174] Lustre: 66538:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761142323/real 1761142323] req@00000000551b5153 x1846691451956224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761142339 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1255.256889] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1256.722356] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1257.871129] Lustre: 66538:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761142326/real 1761142326] req@0000000067a05fde x1846691451956352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761142342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1258.977461] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1258.979145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1258.982526] Lustre: Skipped 2 previous similar messages [ 1258.985017] Lustre: Skipped 2 previous similar messages [ 1259.999182] Lustre: 66540:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761142328/real 1761142328] req@00000000127514e9 x1846691451956480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761142344 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1263.071558] Lustre: 66539:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761142331/real 1761142331] req@00000000982fe82b x1846691451956736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761142347 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1264.833758] LustreError: 78186:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1264.836535] LustreError: 78186:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1264.865304] Lustre: server umount lustre-MDT0000 complete [ 1266.040689] LustreError: 73340:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142350 with bad export cookie 15980123803744992955 [ 1266.043478] LustreError: 73340:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1266.044542] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1266.192408] Lustre: server umount lustre-MDT0001 complete [ 1268.331846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1271.485676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.383653] LustreError: 66536:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009ef42f59 x1846691451975680/t0(0) o250->MGC192.168.201.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1276.503717] LustreError: 76806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.550944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1277.919617] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1280.915185] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1281.012690] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1281.047941] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1281.050923] Lustre: Skipped 2 previous similar messages [ 1282.451663] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1284.302119] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1285.087137] Lustre: 66537:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761142353/real 1761142353] req@000000008abf40b0 x1846691451975296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761142369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1285.555614] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1286.116163] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1286.118192] Lustre: Skipped 1 previous similar message [ 1289.192121] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1294.307080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1294.310313] Lustre: Skipped 5 previous similar messages [ 1298.748259] LustreError: 80638:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1298.750147] LustreError: 80638:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1298.775675] Lustre: server umount lustre-MDT0000 complete [ 1299.425975] LustreError: 74964:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1299.433587] LustreError: 74964:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1300.150353] LustreError: 73339:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142384 with bad export cookie 15980123803744994327 [ 1300.154920] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.155648] LustreError: 73339:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1300.296041] Lustre: server umount lustre-MDT0001 complete [ 1302.591563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1303.724816] Lustre: DEBUG MARKER: files compared the same [ 1314.748336] Lustre: server umount lustre-OST0000 complete [ 1320.618333] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1321.872783] Key type lgssc unregistered [ 1322.025374] LNet: 82269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1322.028521] LNetError: 82269:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1322.039339] LNet: Removed LNI 192.168.201.134@tcp [ 1322.401986] Key type .llcrypt unregistered [ 1322.403555] Key type ._llcrypt unregistered [ 1332.105041] Key type ._llcrypt registered [ 1332.106523] Key type .llcrypt registered [ 1332.148693] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1332.525523] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1332.570669] alg: No test for adler32 (adler32-zlib) [ 1333.437851] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1333.546823] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1335.135181] Key type lgssc registered [ 1335.574990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1338.341615] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1338.350671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1339.524217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1341.010257] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug +malloc all [ 1344.150476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1344.258278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1345.805602] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug +malloc all [ 1347.769162] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1349.108903] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1351.782796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1351.896612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1354.137051] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug +malloc all [ 1356.292561] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1364.194803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1369.536071] LustreError: 85908:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1369.606213] Lustre: server umount lustre-OST0000 complete [ 1374.688135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1374.688813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1374.694068] Lustre: Skipped 1 previous similar message [ 1379.807638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1379.809828] Lustre: Skipped 2 previous similar messages [ 1380.834608] LustreError: 86110:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1380.837566] LustreError: 86110:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1380.855947] Lustre: server umount lustre-MDT0000 complete [ 1382.083694] LustreError: 83626:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142466 with bad export cookie 16590612222778520298 [ 1382.084601] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1382.087125] LustreError: 83626:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1392.610588] LustreError: 86312:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1392.613594] LustreError: 86312:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1392.710447] Lustre: server umount lustre-MDT0001 complete [ 1396.531660] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1397.809205] Key type lgssc unregistered [ 1397.960458] LNet: 86845:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1397.963157] LNetError: 86845:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1397.973315] LNet: Removed LNI 192.168.201.134@tcp [ 1398.282192] Key type .llcrypt unregistered [ 1398.283777] Key type ._llcrypt unregistered [ 1407.927668] Key type ._llcrypt registered [ 1407.929207] Key type .llcrypt registered [ 1407.978145] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1408.501651] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1408.516599] alg: No test for adler32 (adler32-zlib) [ 1409.394454] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1409.488800] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1411.071172] Key type lgssc registered [ 1411.550954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1414.667099] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1414.675698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1430.175411] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1435.295300] LustreError: 88193:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.134@tcp: failed processing log, type 4: rc = -110 [ 1461.919163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1464.131976] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1466.268582] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1469.129889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1469.176499] Lustre: *** cfs_fail_loc=706, val=0*** [ 1469.369077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1469.399928] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1471.008449] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1474.350471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1474.517343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1476.150790] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1478.257110] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1479.680807] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1482.722200] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.729618] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1482.731961] Lustre: Skipped 1 previous similar message [ 1486.507075] LustreError: 90518:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1486.571642] Lustre: server umount lustre-OST0000 complete [ 1489.887848] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1489.890035] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1489.893367] Lustre: Skipped 1 previous similar message [ 1489.894901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1492.960740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1492.961138] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1494.078643] LustreError: 90721:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1494.080404] LustreError: 90721:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1494.107284] Lustre: server umount lustre-MDT0000 complete [ 1495.361150] LustreError: 88201:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142579 with bad export cookie 12282441844368282269 [ 1495.364769] LustreError: 88201:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1495.364912] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1495.429626] LustreError: 90922:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1495.432176] LustreError: 90922:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1495.518947] Lustre: server umount lustre-MDT0001 complete [ 1499.313229] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1500.428132] Key type lgssc unregistered [ 1500.555357] LNet: 91454:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1500.558171] LNetError: 91454:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1500.564533] LNet: Removed LNI 192.168.201.134@tcp [ 1500.882655] Key type .llcrypt unregistered [ 1500.883575] Key type ._llcrypt unregistered [ 1510.416798] Key type ._llcrypt registered [ 1510.417940] Key type .llcrypt registered [ 1510.455280] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1510.880836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1510.894363] alg: No test for adler32 (adler32-zlib) [ 1511.745672] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1511.842144] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1513.431165] Key type lgssc registered [ 1513.784558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1516.418252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1516.425367] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1518.782562] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1521.393593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1521.599505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1522.792544] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1525.619931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1525.732355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1527.591772] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1530.547657] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.551712] Lustre: MGS: Client 7f93f1fa-2bb0-4ba2-9994-092d973fca6e (at 0@lo) reconnecting [ 1530.639978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1530.666830] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1531.987466] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1541.088224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1541.090332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1541.093767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.113260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.115376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1542.116282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1545.899100] LustreError: 95458:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1545.968919] Lustre: server umount lustre-OST0000 complete [ 1552.352680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1552.352753] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1552.353441] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1552.353446] Lustre: Skipped 1 previous similar message [ 1552.356057] Lustre: Skipped 1 previous similar message [ 1553.404708] LustreError: 95660:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1553.406391] LustreError: 95660:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1553.415820] Lustre: server umount lustre-MDT0000 complete [ 1554.682361] LustreError: 95862:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1554.684028] LustreError: 95862:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1554.754175] Lustre: server umount lustre-MDT0001 complete [ 1555.901372] Lustre: server umount MGS complete [ 1559.468950] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:17:23 (1761142643) [ 1568.835708] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 1572.117558] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1575.551913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1577.706489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1579.731956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1581.879587] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1584.114821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1584.139583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1585.523157] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1589.419266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1589.437608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1589.511570] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1589.531083] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1589.567307] Lustre: lustre-MDT0001: new disk, initializing [ 1589.593937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1589.602507] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1589.605313] Lustre: 100570:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1589.607869] LustreError: 100570:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1590.936694] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1595.169305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1595.192502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1595.286514] Lustre: lustre-OST0000: new disk, initializing [ 1595.288329] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1595.312335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1597.220322] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1600.287903] Lustre: 101824:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@000000002cf5de55 x1846691858777088/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761142700 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1600.303244] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1602.593671] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1602.600448] Lustre: MGS: Client 0928d6d6-2d18-4c59-9f72-5acd7818d797 (at 0@lo) reconnecting [ 1602.616283] Lustre: 99856:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1602.626159] LustreError: 99856:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1602.636718] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1602.678471] Lustre: lustre-MDT0000: new disk, initializing [ 1602.734228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1603.743891] Lustre: 102285:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@0000000083a3bf91 x1846691858782208/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1761142704 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1603.807548] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1604.072664] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1608.801989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1608.804764] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1608.872547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1629.153547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.154768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1629.157718] Lustre: Skipped 1 previous similar message [ 1629.159725] Lustre: Skipped 1 previous similar message [ 1634.221068] LustreError: 102807:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1634.223336] LustreError: 102807:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1634.288102] Lustre: server umount lustre-OST0000 complete [ 1639.392781] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.393257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1639.393759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1639.393764] Lustre: Skipped 1 previous similar message [ 1639.397228] Lustre: Skipped 1 previous similar message [ 1641.664609] LustreError: 103008:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1641.667439] LustreError: 103008:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1641.679421] Lustre: server umount lustre-MDT0000 complete [ 1657.311131] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1657.350409] LustreError: 103211:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1657.352926] LustreError: 103211:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1657.432031] Lustre: server umount lustre-MDT0001 complete [ 1658.706991] Lustre: server umount MGS complete [ 1662.465108] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:19:06 (1761142746) [ 1668.750807] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1669.885837] Key type lgssc unregistered [ 1670.023358] LNet: 104749:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1670.027262] LNetError: 104749:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.035434] LNet: Removed LNI 192.168.201.134@tcp [ 1670.359385] Key type .llcrypt unregistered [ 1670.360696] Key type ._llcrypt unregistered [ 1677.473180] Key type ._llcrypt registered [ 1677.474793] Key type .llcrypt registered [ 1677.516272] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1677.864248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1677.889354] alg: No test for adler32 (adler32-zlib) [ 1678.752697] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1678.834830] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1680.415177] Key type lgssc registered [ 1680.781922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1683.399243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1683.405618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.576281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1685.833321] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1688.711358] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1688.847726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1690.178579] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1691.943199] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1693.208115] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1695.490564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1695.607070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.502358] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1699.350541] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1708.902962] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1714.145675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1714.146106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.148086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.148534] LustreError: Skipped 1 previous similar message [ 1714.152897] Lustre: Skipped 1 previous similar message [ 1718.637245] LustreError: 108196:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1718.714552] Lustre: server umount lustre-OST0000 complete [ 1724.384748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.385370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1724.388108] Lustre: Skipped 1 previous similar message [ 1724.391762] Lustre: Skipped 2 previous similar messages [ 1726.078522] LustreError: 108399:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1726.080263] LustreError: 108399:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1726.108162] Lustre: server umount lustre-MDT0000 complete [ 1727.357081] LustreError: 105910:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142811 with bad export cookie 10685696869060211408 [ 1727.360444] LustreError: 105910:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1727.360608] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1727.431894] LustreError: 108601:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1727.435015] LustreError: 108601:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1727.517293] Lustre: server umount lustre-MDT0001 complete [ 1731.428329] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1732.584891] Key type lgssc unregistered [ 1732.725553] LNet: 109133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1732.728695] LNetError: 109133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1732.736494] LNet: Removed LNI 192.168.201.134@tcp [ 1733.077064] Key type .llcrypt unregistered [ 1733.078341] Key type ._llcrypt unregistered [ 1739.620845] Key type ._llcrypt registered [ 1739.622650] Key type .llcrypt registered [ 1739.665198] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1740.012764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1740.062727] alg: No test for adler32 (adler32-zlib) [ 1740.922638] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1741.013119] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1742.591133] Key type lgssc registered [ 1742.952982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1744.973459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1744.975965] LustreError: 110171:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1744.979916] LustreError: 110174:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1744.982394] LustreError: 110171:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1744.984822] LustreError: 110171:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1744.989359] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1744.990185] LustreError: 110171:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1746.152827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1748.800184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1748.893946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1750.174221] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1751.918432] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing lsmod [ 1752.925177] LustreError: 111349:0:(tgt_mount.c:2435:server_fill_super()) cfs_race id 716 sleeping [ 1752.926061] LustreError: 111350:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 waking [ 1752.931211] LustreError: 111349:0:(tgt_mount.c:2435:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1752.933289] LustreError: 111349:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1752.937135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1752.937266] LustreError: 111349:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1753.062594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.080175] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1754.082287] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.087496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.177978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.178303] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.180280] Lustre: Skipped 1 previous similar message [ 1758.184900] Lustre: Skipped 1 previous similar message [ 1759.202773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1759.204823] Lustre: Skipped 2 previous similar messages [ 1760.130406] LustreError: 111540:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1760.167139] Lustre: server umount lustre-MDT0000 complete [ 1761.454347] LustreError: 111542:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142845 with bad export cookie 16765277386799169818 [ 1761.457689] LustreError: 111542:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1761.458849] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1763.295194] LustreError: 110647:0:(client.c:1370:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c56cc330 x1846692098882944/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1763.305517] LustreError: 111744:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1763.308486] LustreError: 111744:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1763.382696] Lustre: server umount lustre-MDT0001 complete [ 1773.876196] LustreError: 111945:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1773.879119] LustreError: 111945:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1773.946744] Lustre: server umount lustre-OST0000 complete [ 1776.837078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1776.989893] LustreError: 112422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1776.997170] LustreError: 112422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1777.017939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.450951] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1781.433127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1781.562497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1782.833767] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1784.564528] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1785.739689] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1787.797442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1789.577915] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1791.015040] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1791.188935] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1796.066090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1796.066438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1796.067179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1796.069126] Lustre: Skipped 1 previous similar message [ 1799.404089] LustreError: 114673:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1799.407300] LustreError: 114673:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1799.464041] Lustre: server umount lustre-OST0000 complete [ 1801.184501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1801.185100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1801.187898] Lustre: Skipped 1 previous similar message [ 1801.189591] Lustre: Skipped 1 previous similar message [ 1806.716503] LustreError: 114875:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1806.718502] LustreError: 114875:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1806.739663] Lustre: server umount lustre-MDT0000 complete [ 1807.927841] LustreError: 112403:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142892 with bad export cookie 16765277386799170490 [ 1807.932093] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1808.100936] Lustre: server umount lustre-MDT0001 complete [ 1811.674894] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1812.701745] Key type lgssc unregistered [ 1812.827358] LNet: 115609:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1812.829322] LNetError: 115609:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1812.836312] LNet: Removed LNI 192.168.201.134@tcp [ 1813.140226] Key type .llcrypt unregistered [ 1813.141813] Key type ._llcrypt unregistered [ 1821.917618] Key type ._llcrypt registered [ 1821.918464] Key type .llcrypt registered [ 1821.954665] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1822.377698] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1822.406096] alg: No test for adler32 (adler32-zlib) [ 1823.261663] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1823.346166] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1824.927127] Key type lgssc registered [ 1825.287229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1827.866714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1827.873182] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.028696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1830.326885] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1833.138482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1833.256275] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.587156] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1836.371196] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1837.577838] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1839.870440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1839.983286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1841.831108] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1843.521368] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1854.310242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 1860.773918] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1861.507192] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1863.647861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1863.650526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1863.654928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.673378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.673497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1864.673875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1864.682875] Lustre: Skipped 1 previous similar message [ 1868.523074] LustreError: 119350:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1868.577613] Lustre: server umount lustre-OST0000 complete [ 1869.791992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.796451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.399654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1874.403037] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1874.408740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1880.031978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1880.034224] Lustre: Skipped 2 previous similar messages [ 1884.127071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1884.156740] LustreError: 119556:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1884.158510] LustreError: 119556:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1884.184594] Lustre: server umount lustre-MDT0000 complete [ 1885.151879] LustreError: 116982:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.155595] LustreError: 116982:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1885.321132] LustreError: 116962:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761142969 with bad export cookie 17944189948992001389 [ 1885.324235] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1885.326048] LustreError: 116962:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1885.393578] LustreError: 119760:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1885.395121] LustreError: 119760:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1885.468763] Lustre: server umount lustre-MDT0001 complete [ 1888.728882] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1889.748204] Key type lgssc unregistered [ 1889.887329] LNet: 120291:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1889.889485] LNetError: 120291:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.895301] LNet: Removed LNI 192.168.201.134@tcp [ 1890.192861] Key type .llcrypt unregistered [ 1890.193777] Key type ._llcrypt unregistered [ 1897.133155] Key type ._llcrypt registered [ 1897.134693] Key type .llcrypt registered [ 1897.177523] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1897.531557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1897.568088] alg: No test for adler32 (adler32-zlib) [ 1898.443306] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1898.536151] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1900.127148] Key type lgssc registered [ 1900.496211] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1903.095750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1903.102090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1904.267167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1905.559262] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1908.478233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1908.599537] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1909.894761] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1911.707135] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1912.934387] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1915.166044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1915.274170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1917.101700] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1918.876950] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1929.637860] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1938.911795] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1938.913961] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.917498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.936848] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1939.937940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1939.938201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.948590] Lustre: Skipped 1 previous similar message [ 1943.341154] LustreError: 123738:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1943.404315] Lustre: server umount lustre-OST0000 complete [ 1945.056775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.057648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1945.059992] Lustre: Skipped 1 previous similar message [ 1950.176385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1950.179458] Lustre: Skipped 2 previous similar messages [ 1950.781579] LustreError: 123940:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1950.784030] LustreError: 123940:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1950.812235] Lustre: server umount lustre-MDT0000 complete [ 1952.041593] LustreError: 121452:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143036 with bad export cookie 16287307859640927073 [ 1952.044607] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1952.046822] LustreError: 121452:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1952.113933] LustreError: 124142:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1952.116703] LustreError: 124142:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1952.200391] Lustre: server umount lustre-MDT0001 complete [ 1955.964843] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 1957.009944] Key type lgssc unregistered [ 1957.147405] LNet: 124672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1957.149456] LNetError: 124672:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1957.158441] LNet: Removed LNI 192.168.201.134@tcp [ 1957.521192] Key type .llcrypt unregistered [ 1957.522432] Key type ._llcrypt unregistered [ 1966.540771] Key type ._llcrypt registered [ 1966.542418] Key type .llcrypt registered [ 1966.586516] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 1966.996367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1967.020930] alg: No test for adler32 (adler32-zlib) [ 1967.890986] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 1967.983235] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 1969.575218] Key type lgssc registered [ 1970.000237] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1972.679927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1972.687416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.851941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.178689] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1978.090407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1978.221502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1979.573255] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1981.423410] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1982.661020] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1985.045505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1985.171539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1987.183255] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 1989.016886] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1999.529863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2002.237889] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2003.514568] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2003.517254] Lustre: Skipped 2 previous similar messages [ 2004.948845] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2004.951683] Lustre: Skipped 2 previous similar messages [ 2009.594663] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2009.604164] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2012.547830] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.34@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2012.551569] Lustre: Skipped 2 previous similar messages [ 2015.293385] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.34@tcp 192.168.0.[2,10]@tcp [ 2022.741599] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.34/24@tcp in log lustre-MDT0000 [ 2022.745734] Lustre: Skipped 2 previous similar messages [ 2030.587991] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.34/24@tcp [ 2030.591136] Lustre: Skipped 1 previous similar message [ 2034.656519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2034.659065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2034.662472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2035.681365] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2035.681638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2035.682141] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.690376] Lustre: Skipped 1 previous similar message [ 2037.998040] LustreError: 129908:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2038.061746] Lustre: server umount lustre-OST0000 complete [ 2039.775828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2039.778358] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2039.781805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2045.501627] LustreError: 130110:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2045.503845] LustreError: 130110:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2045.528532] Lustre: server umount lustre-MDT0000 complete [ 2045.920110] LustreError: 126047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2045.925680] LustreError: 126047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2046.775285] LustreError: 126029:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143131 with bad export cookie 5625574768869067216 [ 2046.777939] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2046.780647] LustreError: 126029:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2046.844789] LustreError: 130313:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2046.847849] LustreError: 130313:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2046.925875] Lustre: server umount lustre-MDT0001 complete [ 2050.453627] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2051.548754] Key type lgssc unregistered [ 2051.681305] LNet: 130844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2051.684819] LNetError: 130844:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2051.695316] LNet: Removed LNI 192.168.201.134@tcp [ 2051.994802] Key type .llcrypt unregistered [ 2051.996158] Key type ._llcrypt unregistered [ 2062.462769] Key type ._llcrypt registered [ 2062.464495] Key type .llcrypt registered [ 2062.504951] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2062.877420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.911976] alg: No test for adler32 (adler32-zlib) [ 2063.778549] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 2063.888415] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2065.479129] Key type lgssc registered [ 2065.862262] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2068.539931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2068.547767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2069.713846] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2069.719228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2069.723031] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.34/24@tcp [ 2071.052102] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2073.918816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2074.026368] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2074.036926] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.34/24@tcp [ 2074.053814] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2075.417400] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2077.240577] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2078.453633] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2080.741325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2080.879108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2082.842678] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2084.688805] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2095.207287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2099.680539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2099.683238] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.688967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2100.706201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2100.706702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2100.707035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.714523] Lustre: Skipped 1 previous similar message [ 2104.494082] LustreError: 134689:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2104.559397] Lustre: server umount lustre-OST0000 complete [ 2105.824456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2105.829447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.831530] Lustre: Skipped 1 previous similar message [ 2110.944149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2110.946150] Lustre: Skipped 1 previous similar message [ 2116.064251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.066280] Lustre: Skipped 1 previous similar message [ 2120.159094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2120.191886] LustreError: 134890:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2120.194886] LustreError: 134890:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2120.226882] Lustre: server umount lustre-MDT0000 complete [ 2121.185149] LustreError: 132368:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.190844] LustreError: 132368:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2121.507173] LustreError: 132350:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143205 with bad export cookie 7329173697115130276 [ 2121.510316] LustreError: 132350:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2121.510594] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2121.572796] LustreError: 135092:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2121.576144] LustreError: 135092:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2121.661547] Lustre: server umount lustre-MDT0001 complete [ 2125.269488] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2126.405238] Key type lgssc unregistered [ 2126.542356] LNet: 135623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2126.544489] LNetError: 135623:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2126.557305] LNet: Removed LNI 192.168.201.134@tcp [ 2126.875884] Key type .llcrypt unregistered [ 2126.877700] Key type ._llcrypt unregistered [ 2135.362956] Key type ._llcrypt registered [ 2135.364057] Key type .llcrypt registered [ 2135.403161] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 2141.295691] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2141.738304] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2141.750424] alg: No test for adler32 (adler32-zlib) [ 2142.614145] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 2142.701811] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2144.287182] Key type lgssc registered [ 2144.652422] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2147.637813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2149.805748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2151.824246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2153.929319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2156.188961] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2156.209054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2156.213960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.300758] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2157.314140] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2157.353675] Lustre: lustre-MDT0000: new disk, initializing [ 2157.378310] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2157.386182] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2158.748910] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2162.913853] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2162.936454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2162.974349] Lustre: 138968:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2162.995883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2162.997490] Lustre: Skipped 1 previous similar message [ 2163.032793] Lustre: lustre-MDT0001: new disk, initializing [ 2163.065538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2163.078411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2163.081908] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2164.325933] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2167.454712] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2168.681202] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2170.923087] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2170.942767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2171.043539] Lustre: lustre-OST0000: new disk, initializing [ 2171.046108] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2171.075742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2172.863590] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2174.528739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2174.532551] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2174.565098] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2175.859036] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2177.198212] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2177.272558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2178.531442] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2178.611294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2179.553854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.554473] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2179.555062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.557816] Lustre: Skipped 1 previous similar message [ 2181.294101] LustreError: 141908:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2181.362587] Lustre: server umount lustre-OST0000 complete [ 2183.647830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2183.649823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2183.653816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2183.656105] Lustre: Skipped 1 previous similar message [ 2184.672277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2184.672894] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.731123] LustreError: 142109:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2188.733723] LustreError: 142109:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2188.767087] Lustre: server umount lustre-MDT0000 complete [ 2189.987335] LustreError: 138961:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143274 with bad export cookie 7069618093584016592 [ 2189.990437] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2189.992450] LustreError: 138961:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2190.057104] LustreError: 142310:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2190.060102] LustreError: 142310:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2190.136216] Lustre: server umount lustre-MDT0001 complete [ 2192.926667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2193.094312] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.387301] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2197.199797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2197.314918] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2198.687511] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2200.495301] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2201.721734] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2204.018030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2206.080889] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2207.893132] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2209.245086] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2219.546795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 2220.846434] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2220.931920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2224.268762] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2224.291770] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2224.347096] Lustre: lustre-OST0001: new disk, initializing [ 2224.348818] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2224.377598] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2224.380168] Lustre: Skipped 1 previous similar message [ 2226.197502] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2229.414845] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2232.831613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2232.834208] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2232.862766] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2233.562674] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 2234.783138] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2234.856852] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2235.975395] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa049c54c5800.ost_server_uuid 50 [ 2236.461172] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa049c54c5800.ost_server_uuid in FULL state after 0 sec [ 2243.041040] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2243.043110] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2243.045108] LustreError: Skipped 1 previous similar message [ 2243.049272] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2243.052543] Lustre: Skipped 2 previous similar messages [ 2244.332064] LustreError: 147128:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2244.334268] LustreError: 147128:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2244.351330] Lustre: server umount lustre-OST0001 complete [ 2248.162415] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2248.163470] LustreError: 144380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2248.166757] Lustre: Skipped 1 previous similar message [ 2248.171904] LustreError: 144380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2248.173193] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2248.173495] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2251.628661] LustreError: 147329:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2251.631687] LustreError: 147329:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2251.690894] Lustre: server umount lustre-OST0000 complete [ 2258.400302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2258.404092] Lustre: Skipped 4 previous similar messages [ 2259.032844] Lustre: server umount lustre-MDT0000 complete [ 2260.254154] LustreError: 142766:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143344 with bad export cookie 7069618093584017460 [ 2260.256929] LustreError: 142766:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2260.258091] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2260.321176] LustreError: 147733:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 2260.322878] LustreError: 147733:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 2260.409983] Lustre: server umount lustre-MDT0001 complete [ 2266.599555] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2267.584950] Key type lgssc unregistered [ 2267.709387] LNet: 148720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2267.711798] LNetError: 148720:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2267.722314] LNet: Removed LNI 192.168.201.134@tcp [ 2268.024273] Key type .llcrypt unregistered [ 2268.025225] Key type ._llcrypt unregistered [ 2278.591958] Key type ._llcrypt registered [ 2278.592947] Key type .llcrypt registered [ 2278.627836] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 2284.568257] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2284.992119] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2285.000964] alg: No test for adler32 (adler32-zlib) [ 2285.862731] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 2285.949240] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2287.527161] Key type lgssc registered [ 2287.899768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2291.031317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2293.675769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2295.870889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2297.972471] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2300.225734] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2300.245205] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2300.250889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2301.334583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2301.347444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2301.397460] Lustre: lustre-MDT0000: new disk, initializing [ 2301.421680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2301.427992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2302.794311] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2307.173369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2307.198163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2307.236092] Lustre: 152642:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2307.256791] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2307.258270] Lustre: Skipped 1 previous similar message [ 2307.293848] Lustre: lustre-MDT0001: new disk, initializing [ 2307.330401] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2307.341373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2307.344037] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2308.610522] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2311.704332] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2312.905843] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2315.061484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2315.083862] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2315.177172] Lustre: lustre-OST0000: new disk, initializing [ 2315.180312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2315.209615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2317.043472] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2318.845828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2318.848450] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2318.881630] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2320.202788] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2321.562116] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2321.640946] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2323.039412] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2323.132543] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2323.942054] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.942100] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2323.943492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2323.948608] Lustre: Skipped 1 previous similar message [ 2323.953770] LustreError: Skipped 1 previous similar message [ 2325.871410] LustreError: 155579:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2325.955745] Lustre: server umount lustre-OST0000 complete [ 2328.032637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2328.037227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2328.045253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2328.049729] Lustre: Skipped 1 previous similar message [ 2329.056743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2329.057478] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2333.504737] LustreError: 155780:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2333.506671] LustreError: 155780:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2333.536853] Lustre: server umount lustre-MDT0000 complete [ 2334.795118] LustreError: 153423:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143419 with bad export cookie 7725064245179271604 [ 2334.798221] LustreError: 153423:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2334.798884] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2334.863666] LustreError: 155982:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2334.865685] LustreError: 155982:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2334.946695] Lustre: server umount lustre-MDT0001 complete [ 2337.666498] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 10:30:21 (1761143421) [ 2343.804124] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 2346.884301] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2350.623134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2353.009782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2355.637729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2358.080450] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2360.504727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2360.526656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2360.606524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2360.618144] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2360.658236] Lustre: lustre-MDT0000: new disk, initializing [ 2360.681862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2360.687827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2361.937143] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2365.905703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2365.930861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2365.966286] Lustre: 158969:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2365.989484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2365.991090] Lustre: Skipped 1 previous similar message [ 2366.025216] Lustre: lustre-MDT0001: new disk, initializing [ 2366.058661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2366.069254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2366.073162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2367.328131] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2370.339684] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2371.461815] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2373.816372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2373.843954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2376.361965] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2377.022959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2377.027442] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2377.071166] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2379.629996] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2381.663453] Lustre: Failing over lustre-OST0000 [ 2381.701101] LustreError: 161411:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2381.704239] LustreError: 161411:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2381.768287] Lustre: server umount lustre-OST0000 complete [ 2394.678535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2394.816976] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2394.823492] Lustre: Skipped 1 previous similar message [ 2394.826531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2394.828857] Lustre: Skipped 1 previous similar message [ 2394.833657] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2395.262374] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2396.720872] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2397.919092] Lustre: 150857:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761143466/real 1761143466] req@000000009aa4e8e6 x1846692670396800/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1761143482 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2397.928455] Lustre: 150857:0:(client.c:2468:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2397.965413] Lustre: Failing over lustre-MDT0000 [ 2398.029546] LustreError: 162384:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2398.031172] LustreError: 162384:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2398.057106] Lustre: server umount lustre-MDT0000 complete [ 2399.508530] LustreError: 158974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2399.512379] LustreError: 158974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2400.224786] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2400.226370] LustreError: 158973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.228640] Lustre: Skipped 2 previous similar messages [ 2400.234101] LustreError: 158973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2403.295082] Lustre: 150857:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761143471/real 1761143471] req@00000000d2c7b32e x1846692670398848/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761143487 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2404.686232] LustreError: 158973:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2404.690600] LustreError: 158973:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2408.415140] Lustre: 150857:0:(client.c:2468:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761143476/real 1761143476] req@00000000c539144b x1846692670399872/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1761143492 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2409.806106] LustreError: 159003:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2409.812841] LustreError: 159003:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2410.962130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2411.003874] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2411.007567] Lustre: 158971:0:(mgc_request.c:1910:mgc_process_log()) MGC192.168.201.134@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2411.087433] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2411.104531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.124631] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2412.365974] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2414.927065] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2416.612539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2416.620057] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 2416.626091] Lustre: Skipped 1 previous similar message [ 2416.645739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2421.730606] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2421.730926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.731674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2421.731678] Lustre: Skipped 1 previous similar message [ 2421.733438] LustreError: Skipped 1 previous similar message [ 2421.737450] Lustre: Skipped 1 previous similar message [ 2423.724099] LustreError: 163268:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2423.727103] LustreError: 163268:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2423.800017] Lustre: server umount lustre-OST0000 complete [ 2426.849656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2426.849951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2426.854555] Lustre: Skipped 2 previous similar messages [ 2431.187098] Lustre: server umount lustre-MDT0000 complete [ 2431.967946] LustreError: 158979:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2431.971732] LustreError: 158979:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2432.377325] LustreError: 161860:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143516 with bad export cookie 7725064245179273662 [ 2432.380131] LustreError: 161860:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2432.380193] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2432.535435] Lustre: server umount lustre-MDT0001 complete [ 2435.892058] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2436.873900] Key type lgssc unregistered [ 2436.997319] LNet: 164202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2436.999296] LNetError: 164202:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2437.009303] LNet: Removed LNI 192.168.201.134@tcp [ 2437.326657] Key type .llcrypt unregistered [ 2437.328149] Key type ._llcrypt unregistered [ 2457.415167] Key type ._llcrypt registered [ 2457.416214] Key type .llcrypt registered [ 2457.450523] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2457.777921] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2457.787081] alg: No test for adler32 (adler32-zlib) [ 2458.631945] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 2458.710104] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2460.287092] Key type lgssc registered [ 2460.667036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2463.257457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2463.265352] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2464.414973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.687078] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2468.720533] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2470.757885] LustreError: 165729:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2470.761211] LustreError: 165729:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2470.763851] LustreError: 165729:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2470.836739] LustreError: 166251:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2470.926121] Lustre: server umount lustre-MDT0000 complete [ 2475.905079] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 2478.829130] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2482.342172] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2484.543578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2486.537369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2488.648753] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2490.927630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2490.951505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2491.039380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2491.052533] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2491.091042] Lustre: lustre-MDT0000: new disk, initializing [ 2491.115521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2491.121775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2492.426487] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2496.588296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2496.613264] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2496.650605] Lustre: 168889:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2496.683841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2496.685598] Lustre: Skipped 1 previous similar message [ 2496.724618] Lustre: lustre-MDT0001: new disk, initializing [ 2496.765301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2496.779531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2496.784817] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2498.095848] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2501.159360] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2502.329148] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2504.587580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2504.614029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2504.709701] Lustre: lustre-OST0000: new disk, initializing [ 2504.711559] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2504.742355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2505.466036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2505.468806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2505.510573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2506.669258] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2509.653372] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2511.446977] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2512.010144] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 10:33:15 (1761143595) [ 2577.375606] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2577.375788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2577.379048] Lustre: Skipped 2 previous similar messages [ 2577.380499] Lustre: Skipped 1 previous similar message [ 2580.669576] LustreError: 172384:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2580.672810] LustreError: 172384:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2580.714232] Lustre: server umount lustre-MDT0000 complete [ 2581.900527] LustreError: 168880:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143666 with bad export cookie 10474486730329966866 [ 2581.901419] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2581.903353] LustreError: 168880:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2582.495421] LustreError: 168924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2582.495458] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2582.499174] LustreError: 168924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2582.507371] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2582.508803] Lustre: Skipped 1 previous similar message [ 2586.527474] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2586.527474] LustreError: 168896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.711418] LustreError: 168896:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.711441] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2596.319138] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2596.353542] LustreError: 172586:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2596.355955] LustreError: 172586:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2596.420624] Lustre: server umount lustre-MDT0001 complete [ 2603.752482] LustreError: 172791:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2603.755426] LustreError: 172791:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2603.813698] Lustre: server umount lustre-OST0000 complete [ 2606.662535] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 2609.758579] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2613.462906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2615.847533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2618.302693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2621.061395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2623.786639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2623.810480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2623.936559] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2623.959672] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2624.020900] Lustre: lustre-MDT0000: new disk, initializing [ 2624.063906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.074807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2625.653389] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2630.015166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2630.046533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2630.079717] Lustre: 175123:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2630.107643] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2630.109532] Lustre: Skipped 1 previous similar message [ 2630.147631] Lustre: lustre-MDT0001: new disk, initializing [ 2630.196369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2630.199823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2631.736898] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2635.062717] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2636.319272] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2638.734619] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2638.760253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2638.875998] Lustre: lustre-OST0000: new disk, initializing [ 2638.877959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2638.910547] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2638.912552] Lustre: Skipped 1 previous similar message [ 2638.987294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2638.990441] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2639.040968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2641.110853] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2644.528204] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2646.027694] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2646.100814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2647.464724] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2647.532069] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2649.569700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2649.571222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2649.571590] LustreError: Skipped 1 previous similar message [ 2649.575189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2649.577400] Lustre: Skipped 2 previous similar messages [ 2659.809843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2659.811497] Lustre: Skipped 3 previous similar messages [ 2662.367111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2662.382379] LustreError: 178062:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2662.383930] LustreError: 178062:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2662.440173] Lustre: server umount lustre-OST0000 complete [ 2664.928326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.931709] Lustre: Skipped 2 previous similar messages [ 2669.795399] Lustre: server umount lustre-MDT0000 complete [ 2670.048271] LustreError: 175919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2670.055160] LustreError: 175919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2671.032193] LustreError: 175114:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143755 with bad export cookie 10474486730330125682 [ 2671.035595] LustreError: 175114:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2671.035821] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2671.102576] LustreError: 178465:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2671.104229] LustreError: 178465:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2671.180948] Lustre: server umount lustre-MDT0001 complete [ 2674.459103] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:35:58 (1761143758) [ 2679.755287] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 2683.111723] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2686.712973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2688.967486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2691.721780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2694.062926] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2696.566332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2696.598889] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2696.696867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2696.712330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2696.766534] Lustre: lustre-MDT0000: new disk, initializing [ 2696.797620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2696.804347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2698.156628] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2702.613594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2702.644331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2702.699364] Lustre: 181449:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2702.867636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2704.100774] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2707.520526] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2708.716475] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2710.976750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2711.009582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2712.927307] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2715.997477] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2717.174399] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2717.208590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2727.394720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2727.395638] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2727.400454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2727.401681] LustreError: Skipped 1 previous similar message [ 2727.405809] Lustre: Skipped 3 previous similar messages [ 2729.517111] LustreError: 184000:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2729.521712] LustreError: 184000:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2729.580066] Lustre: server umount lustre-OST0000 complete [ 2745.311114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2745.378391] Lustre: server umount lustre-MDT0000 complete [ 2746.701540] LustreError: 182235:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143831 with bad export cookie 10474486730330126550 [ 2746.705491] LustreError: 182235:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2746.706674] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2746.851412] Lustre: server umount lustre-MDT0001 complete [ 2749.639945] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 10:37:13 (1761143833) [ 2755.465653] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 2759.148773] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2762.876643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2765.732852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2768.112891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2770.193617] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2773.067553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2773.106489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.235504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2773.250813] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2773.252702] Lustre: Skipped 3 previous similar messages [ 2773.297260] Lustre: lustre-MDT0000: new disk, initializing [ 2773.298891] Lustre: Skipped 2 previous similar messages [ 2773.330387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.332318] Lustre: Skipped 2 previous similar messages [ 2773.338588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2773.344477] Lustre: Skipped 2 previous similar messages [ 2774.823268] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2779.266860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2779.285931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2779.323162] Lustre: 187388:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2779.443082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2780.775754] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2784.019179] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2785.247427] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2787.605380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2787.632402] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2789.443449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2790.014980] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2793.358184] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2799.586568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2799.586928] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2799.587522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2799.587526] Lustre: Skipped 6 previous similar messages [ 2799.593668] Lustre: Skipped 4 previous similar messages [ 2799.599307] LustreError: Skipped 1 previous similar message [ 2801.325506] LustreError: 189934:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2801.327353] LustreError: 189934:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2801.395290] Lustre: server umount lustre-OST0000 complete [ 2808.796585] Lustre: server umount lustre-MDT0000 complete [ 2809.824112] LustreError: 188191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2809.828605] LustreError: 188191:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2810.095328] LustreError: 187380:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143894 with bad export cookie 10474486730330127621 [ 2810.099156] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2810.101937] LustreError: 187380:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2810.244913] Lustre: server umount lustre-MDT0001 complete [ 2814.448492] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2815.778009] Key type lgssc unregistered [ 2815.937539] LNet: 190867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2815.943414] LNetError: 190867:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2815.953446] LNet: Removed LNI 192.168.201.134@tcp [ 2816.328816] Key type .llcrypt unregistered [ 2816.330716] Key type ._llcrypt unregistered [ 2825.555509] Key type ._llcrypt registered [ 2825.556489] Key type .llcrypt registered [ 2825.593829] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2826.103362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2826.110169] alg: No test for adler32 (adler32-zlib) [ 2826.973523] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 2827.053614] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2828.631206] Key type lgssc registered [ 2829.050697] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2831.524368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2831.530364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.690642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.957594] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2836.755991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2836.885266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2838.129955] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2839.863108] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2840.954762] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2843.200333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2843.326107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2845.197782] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2846.938923] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2857.707447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2878.433152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2878.434063] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2878.435332] LustreError: Skipped 1 previous similar message [ 2878.436291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2878.439734] Lustre: Skipped 1 previous similar message [ 2880.043251] LustreError: 194512:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2880.102676] Lustre: server umount lustre-OST0000 complete [ 2883.039956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2883.043112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2883.048347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2883.049902] Lustre: Skipped 1 previous similar message [ 2887.417176] LustreError: 194715:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2887.418962] LustreError: 194715:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2887.442535] Lustre: server umount lustre-MDT0000 complete [ 2888.596419] LustreError: 192221:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761143972 with bad export cookie 1870285805425241041 [ 2888.599751] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2888.602157] LustreError: 192221:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2888.671628] LustreError: 194919:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2888.673309] LustreError: 194919:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2888.756881] Lustre: server umount lustre-MDT0001 complete [ 2892.330486] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 2893.339977] Key type lgssc unregistered [ 2893.461382] LNet: 195450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2893.465046] LNetError: 195450:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2893.473355] LNet: Removed LNI 192.168.201.134@tcp [ 2893.752717] Key type .llcrypt unregistered [ 2893.753781] Key type ._llcrypt unregistered [ 2902.279278] Key type ._llcrypt registered [ 2902.280415] Key type .llcrypt registered [ 2902.314347] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 2902.637805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2902.661872] alg: No test for adler32 (adler32-zlib) [ 2903.508364] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 2903.586877] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 2905.167210] Key type lgssc registered [ 2905.510556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2908.015209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2908.023700] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.192048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2910.402426] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2913.022792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2913.139107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2914.355000] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2916.134238] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2917.433945] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2919.591209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2919.724615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.484159] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2923.183480] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2934.054699] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2943.455876] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2943.459158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2943.464603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2944.481586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2944.482958] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.483225] Lustre: Skipped 1 previous similar message [ 2944.488444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2946.862156] LustreError: 199092:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2946.929782] Lustre: server umount lustre-OST0000 complete [ 2948.621364] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa049c69d0800.ost_server_uuid 50 [ 2949.088201] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffffa049c69d0800.ost_server_uuid in DISCONN state after 0 sec [ 2949.566608] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2959.507447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.512659] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.447859] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2964.449943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.453740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2965.627127] LustreError: 199489:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2965.628920] LustreError: 199489:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2965.653341] Lustre: server umount lustre-MDT0000 complete [ 2966.885554] LustreError: 196804:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144051 with bad export cookie 12479446151814692814 [ 2966.890349] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2966.954605] LustreError: 199693:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2966.958119] LustreError: 199693:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2967.039977] Lustre: server umount lustre-MDT0001 complete [ 2969.757070] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 10:40:53 (1761144053) [ 2971.927662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.072605] LustreError: 200363:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2972.079193] LustreError: 200363:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2972.094981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2973.336413] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2976.119225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2976.232241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2977.405989] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2979.057806] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2980.183212] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2982.341876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2982.368940] LustreError: 201961:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2984.237194] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2985.821207] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2986.534623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2987.940312] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2987.973832] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2988.038499] Lustre: lustre-OST0001: new disk, initializing [ 2988.041647] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2988.078804] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2988.080923] Lustre: Skipped 1 previous similar message [ 2988.260492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2988.266188] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2988.304364] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2989.891544] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 2992.816341] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2996.704391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2996.704812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2996.706930] LustreError: Skipped 1 previous similar message [ 2996.710631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2996.712750] Lustre: Skipped 2 previous similar messages [ 2999.724129] LustreError: 203570:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2999.727583] LustreError: 203570:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2999.745577] Lustre: server umount lustre-OST0000 complete [ 3001.576458] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3001.651961] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3002.909808] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3002.987369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3003.215826] LustreError: 202607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.335611] LustreError: 202609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.340139] LustreError: 202609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3014.113365] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3014.113628] LustreError: 202609:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3014.114623] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.114629] Lustre: Skipped 1 previous similar message [ 3014.115054] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3014.115058] Lustre: Skipped 3 previous similar messages [ 3014.116859] LustreError: Skipped 1 previous similar message [ 3014.129741] LustreError: 202609:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3019.053247] LustreError: 204272:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3019.054949] LustreError: 204272:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3019.117848] Lustre: server umount lustre-OST0001 complete [ 3022.303787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3026.466543] Lustre: server umount lustre-MDT0000 complete [ 3027.656206] LustreError: 204186:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144112 with bad export cookie 12479446151814693878 [ 3027.659635] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3027.660197] LustreError: 204186:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3027.727247] LustreError: 204676:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3027.729254] LustreError: 204676:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3027.813611] Lustre: server umount lustre-MDT0001 complete [ 3029.701541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3030.950726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3032.199384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3033.485468] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3035.623802] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 10:41:59 (1761144119) [ 3037.819826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3037.917124] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3037.928461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3037.998615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.189512] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3041.829569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3041.864270] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3041.877535] Lustre: 206165:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3043.187928] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3045.165726] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3046.432300] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3048.933046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3049.021755] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3050.928727] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3052.604184] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3054.785711] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3054.806424] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3056.160601] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 3056.169381] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 3056.663752] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3058.371125] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3061.219618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3061.220033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.222348] LustreError: Skipped 1 previous similar message [ 3061.222964] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3061.226213] Lustre: Skipped 4 previous similar messages [ 3061.228040] Lustre: Skipped 4 previous similar messages [ 3065.324902] LustreError: 209356:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3065.327006] LustreError: 209356:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3065.342065] Lustre: server umount lustre-OST0000 complete [ 3066.338671] LustreError: 208930:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3066.343702] LustreError: 208930:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3082.039510] Lustre: server umount lustre-OST0001 complete [ 3089.375239] Lustre: server umount lustre-MDT0000 complete [ 3090.599644] LustreError: 206157:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144174 with bad export cookie 12479446151814695236 [ 3090.603148] LustreError: 206157:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3090.604550] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3090.750089] Lustre: server umount lustre-MDT0001 complete [ 3092.563568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3093.809204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3095.044500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3096.281307] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3098.407441] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 10:43:02 (1761144182) [ 3103.538616] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 3106.263225] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3109.354328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3111.570184] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3113.563857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3115.620439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3117.844320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3117.866913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.952119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3117.963411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3117.999175] Lustre: lustre-MDT0000: new disk, initializing [ 3118.023266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3118.025824] Lustre: Skipped 3 previous similar messages [ 3118.031378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3119.303464] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3123.370036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3123.395781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3123.425839] Lustre: 213778:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3123.448925] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3123.451378] Lustre: Skipped 1 previous similar message [ 3123.485277] Lustre: lustre-MDT0001: new disk, initializing [ 3123.528544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3123.531982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3124.862244] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3127.931697] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3129.071414] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3131.224355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3131.248331] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3131.337359] Lustre: lustre-OST0000: new disk, initializing [ 3131.339353] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3132.981112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3132.983680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3133.015353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3133.231128] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3136.169305] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3137.452166] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3137.518163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3138.747224] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3138.825512] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3141.483134] LustreError: 216714:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3141.486111] LustreError: 216714:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3141.546576] Lustre: server umount lustre-OST0000 complete [ 3143.137041] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3143.137510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3143.140129] Lustre: Skipped 5 previous similar messages [ 3143.143532] Lustre: Skipped 9 previous similar messages [ 3153.401802] Lustre: server umount lustre-MDT0000 complete [ 3154.623518] LustreError: 214558:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144238 with bad export cookie 12479446151814696909 [ 3154.627592] LustreError: 214558:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3154.627741] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3164.202589] Lustre: server umount lustre-MDT0001 complete [ 3167.012488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3167.163872] LustreError: 217594:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3167.172867] LustreError: 217594:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 3168.518192] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3171.340173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3172.761960] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3174.504096] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3175.654889] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3178.094056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3180.259241] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3182.148449] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3183.664958] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3194.065006] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3195.499392] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3195.574342] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3196.896017] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3196.899150] LustreError: Skipped 3 previous similar messages [ 3202.338697] Lustre: server umount lustre-OST0000 complete [ 3204.134447] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3205.401953] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3228.918450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3229.023462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3229.027210] Lustre: Skipped 5 previous similar messages [ 3230.925746] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3232.747697] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3233.323104] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3236.332107] LustreError: 222036:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3236.334663] LustreError: 222036:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3236.394704] Lustre: server umount lustre-OST0000 complete [ 3238.368062] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3238.373027] Lustre: Skipped 2 previous similar messages [ 3238.375325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3238.377131] Lustre: Skipped 7 previous similar messages [ 3239.764760] Lustre: server umount lustre-MDT0000 complete [ 3241.046579] LustreError: 217619:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144325 with bad export cookie 12479446151814697770 [ 3241.049969] LustreError: 217619:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3241.051600] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.211443] Lustre: server umount lustre-MDT0001 complete [ 3243.979528] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 10:45:27 (1761144327) [ 3246.236232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.393485] LustreError: 223115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3247.694665] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3250.491072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3251.913131] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3253.713472] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3254.937211] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3257.315155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3259.179666] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3260.961568] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3262.253772] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3271.539885] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3272.800248] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3272.881129] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3275.193273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3275.221419] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3275.277008] Lustre: lustre-OST0001: new disk, initializing [ 3275.279593] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3275.421757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3275.426162] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3275.452509] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3277.326929] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3280.494108] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3281.824829] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3281.928270] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3283.294312] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3283.366463] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3285.991374] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3285.999074] LustreError: Skipped 3 previous similar messages [ 3298.271139] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3298.317195] Lustre: server umount lustre-OST0001 complete [ 3300.397968] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3300.465984] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3301.637494] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3301.699792] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3311.591359] LustreError: 225218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3311.603751] LustreError: 225218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3325.248328] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3327.183528] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3329.388062] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3330.002792] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3330.535347] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3330.540521] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3330.540724] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3330.547229] Lustre: Skipped 1 previous similar message [ 3336.757679] Lustre: server umount lustre-OST0001 complete [ 3363.295150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3363.385083] Lustre: server umount lustre-OST0000 complete [ 3366.369635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.374751] Lustre: Skipped 8 previous similar messages [ 3366.377259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3366.379799] Lustre: Skipped 18 previous similar messages [ 3379.167127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3379.201728] LustreError: 229459:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 3379.203699] LustreError: 229459:0:(obd_class.h:479:obd_check_dev()) Skipped 29 previous similar messages [ 3379.231417] Lustre: server umount lustre-MDT0000 complete [ 3380.495504] LustreError: 224732:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144464 with bad export cookie 12479446151814698708 [ 3380.500233] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3386.709681] Lustre: server umount lustre-MDT0001 complete [ 3388.736975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3389.926310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3391.054385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3392.235738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3394.301768] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 10:47:58 (1761144478) [ 3396.394801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3396.483825] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3396.492585] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3396.566389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3396.568304] Lustre: Skipped 5 previous similar messages [ 3397.882366] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3400.543615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3400.578983] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3400.588040] Lustre: 231152:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3401.876665] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3403.727573] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3404.969416] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3407.489557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3407.602623] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3409.663443] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3411.833230] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3415.867025] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3415.896575] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3418.402978] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3420.269114] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3420.831179] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3422.333845] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3422.422175] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3424.136632] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3424.263049] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3425.892743] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa049cb1da800.ost_server_uuid 50 [ 3426.407020] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa049cb1da800.ost_server_uuid in FULL state after 0 sec [ 3426.767872] Lustre: Permanently deactivating lustre-OST0001 [ 3426.775337] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3427.606265] Lustre: Permanently reactivating lustre-OST0001 [ 3431.396596] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3431.398866] LustreError: Skipped 3 previous similar messages [ 3435.013890] Lustre: server umount lustre-OST0001 complete [ 3441.635623] LustreError: 232835:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3441.641504] LustreError: 232835:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 21 previous similar messages [ 3442.412626] Lustre: server umount lustre-OST0000 complete [ 3458.015248] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3458.104436] Lustre: server umount lustre-MDT0000 complete [ 3459.510715] LustreError: 231145:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144543 with bad export cookie 12479446151814699982 [ 3459.515294] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3459.518931] LustreError: 231145:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3459.685519] Lustre: server umount lustre-MDT0001 complete [ 3461.642888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3463.051277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3464.432866] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3465.692055] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3468.040134] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 10:49:11 (1761144551) [ 3468.443260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3470.720809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3470.808359] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3470.817175] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3470.819667] Lustre: Skipped 5 previous similar messages [ 3472.164193] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3474.998466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3475.028914] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3475.038039] Lustre: 237299:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3476.394712] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3478.419063] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3479.674623] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3482.036745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3482.059127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3482.132397] Lustre: Permanently deactivating lustre-OST0000 [ 3482.136565] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3484.631989] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3486.440329] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3486.707560] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3486.710518] LustreError: 239504:0:(osp_precreate.c:651:osp_precreate_send()) lustre-OST0000-osc-MDT0001: can't precreate: rc = -5 [ 3486.714282] LustreError: 239504:0:(osp_precreate.c:1355:osp_precreate_thread()) lustre-OST0000-osc-MDT0001: cannot precreate objects: rc = -5 [ 3488.739457] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3490.992497] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3492.948269] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3497.018150] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3498.755204] Lustre: Permanently reactivating lustre-OST0000 [ 3498.759901] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 3498.761712] Lustre: Skipped 2 previous similar messages [ 3507.188240] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3507.190220] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3507.193280] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3507.205502] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3507.211663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3510.976118] Lustre: server umount lustre-OST0001 complete [ 3514.287430] Lustre: server umount lustre-OST0000 complete [ 3521.943864] Lustre: server umount lustre-MDT0000 complete [ 3523.307446] LustreError: 238028:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144607 with bad export cookie 12479446151814701641 [ 3523.311252] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3523.314707] LustreError: 238028:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3523.470491] Lustre: server umount lustre-MDT0001 complete [ 3527.041054] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 3528.200955] Key type lgssc unregistered [ 3528.337623] LNet: 241689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3528.341503] LNetError: 241689:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3528.352489] LNet: Removed LNI 192.168.201.134@tcp [ 3528.741192] Key type .llcrypt unregistered [ 3528.743465] Key type ._llcrypt unregistered [ 3538.272677] Key type ._llcrypt registered [ 3538.273657] Key type .llcrypt registered [ 3538.315142] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3538.713484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3538.744909] alg: No test for adler32 (adler32-zlib) [ 3539.605438] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 3539.697724] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 3541.279155] Key type lgssc registered [ 3541.650915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3542.769546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3545.027875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3545.035822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3546.199692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.433238] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3550.254174] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3550.277558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3550.306036] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3550.316236] Lustre: 243103:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3550.322516] Lustre: Permanently deactivating lustre-MDT0001 [ 3550.328381] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3550.434522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3551.700718] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3552.749431] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3552.751696] LustreError: 243148:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3552.759979] LustreError: 243148:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3552.767217] LustreError: 243148:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 3553.508217] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3554.675637] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3556.984219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3557.114875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3558.981976] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3560.163344] LustreError: 245237:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3560.167722] LustreError: 245237:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3560.654275] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3563.052558] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3563.125156] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3565.046799] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3566.815492] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3568.614229] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3571.692166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3571.697168] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3572.444822] Lustre: Permanently reactivating lustre-MDT0001 [ 3572.449851] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3572.452324] Lustre: Skipped 2 previous similar messages [ 3572.477641] Lustre: lustre-MDT0001: Client df543e44-db43-4aa7-a0ab-2d20c3779e68 (at 192.168.201.34@tcp) reconnecting [ 3573.630885] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3574.166876] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3575.777559] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3575.850292] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3577.092955] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3579.221323] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 2 sec [ 3580.069529] Lustre: Permanently deactivating lustre-MDT0001 [ 3580.075738] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3580.077681] Lustre: Skipped 2 previous similar messages [ 3585.028652] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3588.065487] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3588.067364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3588.069959] Lustre: Skipped 3 previous similar messages [ 3588.071745] Lustre: Skipped 2 previous similar messages [ 3593.184616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3593.186939] Lustre: Skipped 3 previous similar messages [ 3593.982245] LustreError: 247242:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3594.013285] Lustre: server umount lustre-MDT0000 complete [ 3595.259455] LustreError: 243840:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144679 with bad export cookie 10650730738023358219 [ 3595.263477] LustreError: 243840:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3595.264158] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3595.336779] LustreError: 247444:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3595.339325] LustreError: 247444:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3595.432216] Lustre: server umount lustre-MDT0001 complete [ 3602.732067] LustreError: 247644:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3602.734104] LustreError: 247644:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3602.746841] Lustre: server umount lustre-OST0000 complete [ 3606.245350] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 10:51:30 (1761144690) [ 3609.566731] LustreError: 248649:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3609.570192] LustreError: 248649:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3609.633998] Lustre: server umount lustre-OST0001 complete [ 3611.665273] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 3614.483585] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3617.968330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3620.071298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3622.065037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3624.125353] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3626.467072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3626.497476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3626.577202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3626.580376] Lustre: Skipped 2 previous similar messages [ 3626.590622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3626.627925] Lustre: lustre-MDT0000: new disk, initializing [ 3626.654652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3626.661266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3627.970439] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3632.013571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3632.035483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3632.064503] Lustre: 250830:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3632.087452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3632.089053] Lustre: Skipped 1 previous similar message [ 3632.123827] Lustre: lustre-MDT0001: new disk, initializing [ 3632.166842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3632.170937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3633.394862] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3636.433120] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3637.610239] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3639.861879] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.885762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3639.990885] Lustre: lustre-OST0000: new disk, initializing [ 3639.993690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3640.023602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3640.027281] Lustre: Skipped 1 previous similar message [ 3641.973839] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3642.485395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3642.489121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3642.517147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3644.974660] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3645.668873] LustreError: 250837:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3650.011755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3650.038146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3650.088510] Lustre: lustre-OST0001: new disk, initializing [ 3650.090214] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3652.321994] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3655.487552] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3655.673129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3655.676748] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3655.705434] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3655.759099] LustreError: 250837:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3655.765576] LustreError: 250837:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3656.004239] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3665.855082] LustreError: 250837:0:(mdt_handler.c:3254:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3668.310749] Lustre: lustre-OST0001: Not available for connect from 192.168.201.34@tcp (stopping) [ 3671.009139] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3671.011084] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3671.012826] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3671.013838] Lustre: Skipped 1 previous similar message [ 3671.016653] Lustre: Skipped 1 previous similar message [ 3672.493875] LustreError: 254325:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3672.495849] LustreError: 254325:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3672.514308] Lustre: server umount lustre-OST0001 complete [ 3676.131782] LustreError: 252562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3676.135914] LustreError: 252562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3681.250913] LustreError: 252562:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3681.254715] LustreError: 252562:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3686.373966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3686.374871] LustreError: 252560:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3686.375977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3686.378942] Lustre: Skipped 1 previous similar message [ 3686.391030] LustreError: 252560:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3690.540077] LustreError: 254548:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3690.541807] LustreError: 254548:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3690.622469] Lustre: server umount lustre-OST0000 complete [ 3693.535809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3693.538117] LustreError: Skipped 1 previous similar message [ 3693.539359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3696.608175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3696.609970] Lustre: Skipped 3 previous similar messages [ 3706.335117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3706.394065] Lustre: server umount lustre-MDT0000 complete [ 3707.673388] LustreError: 250822:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144792 with bad export cookie 10650730738023360816 [ 3707.677180] LustreError: 250822:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3707.677846] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3713.798756] LustreError: 254953:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3713.801168] LustreError: 254953:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3713.902725] Lustre: server umount lustre-MDT0001 complete [ 3717.911170] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 3719.156644] Key type lgssc unregistered [ 3719.294416] LNet: 255484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3719.299365] LNetError: 255484:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3719.307516] LNet: Removed LNI 192.168.201.134@tcp [ 3719.613966] Key type .llcrypt unregistered [ 3719.614874] Key type ._llcrypt unregistered [ 3731.437667] Key type ._llcrypt registered [ 3731.439517] Key type .llcrypt registered [ 3731.478655] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 3738.594379] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3739.019085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3739.032915] alg: No test for adler32 (adler32-zlib) [ 3739.901398] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 3739.989591] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 3741.567141] Key type lgssc registered [ 3741.935880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3745.173345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3747.557981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3750.202878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3752.658223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3755.366656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3755.397504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3755.406641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3756.497294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3756.511962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3756.551596] Lustre: lustre-MDT0000: new disk, initializing [ 3756.578476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.587877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3757.848129] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3761.925790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3761.950554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3761.981103] Lustre: 259405:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3762.009615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3762.011174] Lustre: Skipped 1 previous similar message [ 3762.049775] Lustre: lustre-MDT0001: new disk, initializing [ 3762.084858] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3762.097409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3762.099931] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3763.275851] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3766.237575] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3767.291742] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3769.369250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3769.395610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3769.490409] Lustre: lustre-OST0000: new disk, initializing [ 3769.492117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3769.521581] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3769.805456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3769.809826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3769.847410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3771.302868] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3774.257342] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3775.506495] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3775.578149] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3776.810059] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3776.886586] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3780.068103] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3780.068560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3780.071025] LustreError: Skipped 1 previous similar message [ 3780.075973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3780.078434] Lustre: Skipped 1 previous similar message [ 3783.598145] LustreError: 262343:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3783.689533] Lustre: server umount lustre-OST0000 complete [ 3785.184822] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.185513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3785.188130] Lustre: Skipped 2 previous similar messages [ 3785.191081] Lustre: Skipped 1 previous similar message [ 3790.304113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3790.306236] Lustre: Skipped 1 previous similar message [ 3791.105685] LustreError: 262545:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3791.107575] LustreError: 262545:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3791.139149] Lustre: server umount lustre-MDT0000 complete [ 3792.363215] LustreError: 259397:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761144876 with bad export cookie 18150279380169789195 [ 3792.366373] LustreError: 259397:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3792.367075] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3792.435290] LustreError: 262747:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3792.436976] LustreError: 262747:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3792.521547] Lustre: server umount lustre-MDT0001 complete [ 3795.090531] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 10:54:39 (1761144879) [ 3797.331354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3797.511802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3798.800740] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3801.510273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3801.642046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3802.940592] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3804.677387] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3805.806913] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3808.079852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3810.124138] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3811.896211] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3867.616600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3867.621059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3867.627909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3869.666357] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3869.669890] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3872.045516] LustreError: 265761:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3872.047203] LustreError: 265761:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3872.116110] Lustre: server umount lustre-OST0000 complete [ 3873.625022] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3877.862963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3877.999170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3878.000877] Lustre: Skipped 1 previous similar message [ 3879.892647] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3881.628133] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3890.275194] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3890.278543] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3895.396622] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3895.400217] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3916.259985] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3916.260968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3916.262559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3916.262567] Lustre: Skipped 2 previous similar messages [ 3916.264278] LustreError: Skipped 1 previous similar message [ 3916.267740] Lustre: Skipped 1 previous similar message [ 3920.815060] LustreError: 267428:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3920.817183] LustreError: 267428:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3920.893683] Lustre: server umount lustre-OST0000 complete [ 3924.448050] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3924.450026] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3924.453341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3924.456274] Lustre: Skipped 1 previous similar message [ 3928.210519] Lustre: server umount lustre-MDT0000 complete [ 3929.398151] LustreError: 263397:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145013 with bad export cookie 18150279380169790063 [ 3929.401450] LustreError: 263397:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3929.401659] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3929.463757] LustreError: 267830:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3929.465468] LustreError: 267830:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3929.545604] Lustre: server umount lustre-MDT0001 complete [ 3933.206234] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 3934.290177] Key type lgssc unregistered [ 3934.422496] LNet: 268361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3934.424403] LNetError: 268361:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3934.435461] LNet: Removed LNI 192.168.201.134@tcp [ 3934.789574] Key type .llcrypt unregistered [ 3934.790743] Key type ._llcrypt unregistered [ 3943.954306] Key type ._llcrypt registered [ 3943.955213] Key type .llcrypt registered [ 3943.991479] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 3944.374451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3944.413195] alg: No test for adler32 (adler32-zlib) [ 3945.270879] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 3945.361277] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 3946.943204] Key type lgssc registered [ 3947.263794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3950.031672] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3950.045281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3951.232919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3952.538197] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3955.552986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3955.677755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3956.982762] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3958.755693] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3959.911239] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3962.118948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3962.240101] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.105726] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 3965.907237] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3976.616044] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3991.008105] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3991.012177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3991.015638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.033914] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3992.034580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3992.037393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.117869] LustreError: 272774:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3995.187129] Lustre: server umount lustre-OST0000 complete [ 3996.639861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3996.641890] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3996.645713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3996.647545] Lustre: Skipped 1 previous similar message [ 4002.272202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4002.274297] Lustre: Skipped 3 previous similar messages [ 4002.626671] LustreError: 272976:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4002.628443] LustreError: 272976:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4002.652093] Lustre: server umount lustre-MDT0000 complete [ 4003.851676] LustreError: 269716:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145088 with bad export cookie 14207800549176187920 [ 4003.855058] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4003.856134] LustreError: 269716:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4003.915634] LustreError: 273178:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4003.918889] LustreError: 273178:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4004.005538] Lustre: server umount lustre-MDT0001 complete [ 4007.514482] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 4008.547599] Key type lgssc unregistered [ 4008.678374] LNet: 273709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4008.680486] LNetError: 273709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4008.689363] LNet: Removed LNI 192.168.201.134@tcp [ 4009.007623] Key type .llcrypt unregistered [ 4009.008581] Key type ._llcrypt unregistered [ 4016.330369] Key type ._llcrypt registered [ 4016.332063] Key type .llcrypt registered [ 4016.374628] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4016.740449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4016.758532] alg: No test for adler32 (adler32-zlib) [ 4017.605711] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4017.682415] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 4019.255158] Key type lgssc registered [ 4019.657313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4022.115593] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4022.123344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4023.320913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.079705] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4028.723361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4028.874616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4030.369904] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4032.721692] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4034.019713] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4036.627296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4036.759864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4039.008202] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4041.034344] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4048.041489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4058.594195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4058.594509] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4058.596249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4058.599699] Lustre: Skipped 1 previous similar message [ 4061.227109] LustreError: 277300:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4061.289437] Lustre: server umount lustre-OST0000 complete [ 4063.712757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4063.713572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4063.715973] Lustre: Skipped 1 previous similar message [ 4063.719354] Lustre: Skipped 2 previous similar messages [ 4068.605255] LustreError: 277502:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4068.608269] LustreError: 277502:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4068.636737] Lustre: server umount lustre-MDT0000 complete [ 4068.831920] LustreError: 275610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4068.838341] LustreError: 275610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4069.898476] LustreError: 276505:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145154 with bad export cookie 7221380479670152547 [ 4069.901169] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4069.901423] LustreError: 276505:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4069.966968] LustreError: 277703:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4069.968789] LustreError: 277703:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4070.055627] Lustre: server umount lustre-MDT0001 complete [ 4073.633984] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 4074.755208] Key type lgssc unregistered [ 4074.896312] LNet: 278234:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4074.898566] LNetError: 278234:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4074.907463] LNet: Removed LNI 192.168.201.134@tcp [ 4075.255188] Key type .llcrypt unregistered [ 4075.256371] Key type ._llcrypt unregistered [ 4083.919662] Key type ._llcrypt registered [ 4083.920586] Key type .llcrypt registered [ 4083.954948] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4084.369326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4084.380912] alg: No test for adler32 (adler32-zlib) [ 4085.218771] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4085.292327] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 4086.871166] Key type lgssc registered [ 4087.206362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4089.657441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4089.664620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4090.826855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4092.075615] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4094.824618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4094.944817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4096.206865] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4097.974808] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4099.100076] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4101.360223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4101.489861] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.386242] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4105.187149] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4115.818970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4128.737435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4128.739703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4128.742883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4131.296972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4131.297703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4131.300845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4134.123204] LustreError: 282650:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4134.179313] Lustre: server umount lustre-OST0000 complete [ 4135.903972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4135.905919] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4135.909285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4135.910662] Lustre: Skipped 1 previous similar message [ 4141.497725] LustreError: 282851:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4141.499483] LustreError: 282851:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4141.525692] Lustre: server umount lustre-MDT0000 complete [ 4141.537084] LustreError: 279607:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4141.541494] LustreError: 279607:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4142.694294] LustreError: 279588:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145227 with bad export cookie 1451339176720981615 [ 4142.697110] LustreError: 279588:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4142.699391] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4142.756745] LustreError: 283052:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4142.758436] LustreError: 283052:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4142.842961] Lustre: server umount lustre-MDT0001 complete [ 4146.081464] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 4147.165132] Key type lgssc unregistered [ 4147.298427] LNet: 283583:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4147.300467] LNetError: 283583:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4147.309415] LNet: Removed LNI 192.168.201.134@tcp [ 4147.635108] Key type .llcrypt unregistered [ 4147.636047] Key type ._llcrypt unregistered [ 4154.229246] Key type ._llcrypt registered [ 4154.230251] Key type .llcrypt registered [ 4154.268658] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4154.605529] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4154.612305] alg: No test for adler32 (adler32-zlib) [ 4155.477018] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4155.560330] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 4157.143180] Key type lgssc registered [ 4157.494721] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4159.947398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4159.955899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4161.125447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4162.346963] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4165.173147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4165.285521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4166.551101] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4168.266320] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4169.527572] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4171.860704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4171.985333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4173.850581] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4175.819850] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4187.367326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4195.808141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4195.811089] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.816456] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4197.859243] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4197.860992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4197.864555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4201.196080] LustreError: 287227:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4201.253678] Lustre: server umount lustre-OST0000 complete [ 4202.976303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4202.977029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4202.983290] Lustre: Skipped 1 previous similar message [ 4202.984685] Lustre: Skipped 1 previous similar message [ 4208.097298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.100593] Lustre: Skipped 2 previous similar messages [ 4208.594372] LustreError: 287428:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4208.599269] LustreError: 287428:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4208.634382] Lustre: server umount lustre-MDT0000 complete [ 4210.370729] LustreError: 285520:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145294 with bad export cookie 949160161284579592 [ 4210.375888] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4210.378852] LustreError: 285520:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4210.439316] LustreError: 287629:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4210.441431] LustreError: 287629:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4210.536616] Lustre: server umount lustre-MDT0001 complete [ 4214.036234] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 4215.043204] Key type lgssc unregistered [ 4215.177413] LNet: 288160:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4215.180264] LNetError: 288160:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4215.190318] LNet: Removed LNI 192.168.201.134@tcp [ 4215.560894] Key type .llcrypt unregistered [ 4215.561890] Key type ._llcrypt unregistered [ 4219.012340] Key type ._llcrypt registered [ 4219.013281] Key type .llcrypt registered [ 4219.046077] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4224.356201] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 4230.337247] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4230.696667] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4230.738755] alg: No test for adler32 (adler32-zlib) [ 4231.585347] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4231.667122] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 4233.255127] Key type lgssc registered [ 4233.622791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4236.533806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4238.764295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4240.710313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4242.746188] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4244.914801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4244.940281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4244.944862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4246.031813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4246.045771] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4246.079799] Lustre: lustre-MDT0000: new disk, initializing [ 4246.102957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4246.108455] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4247.402379] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4251.427306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4251.455570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4251.489943] Lustre: 291642:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4251.531559] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4251.532536] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4251.536308] Lustre: Skipped 1 previous similar message [ 4251.573625] Lustre: lustre-MDT0001: new disk, initializing [ 4251.612039] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4251.627368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4251.630938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4252.860340] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4255.834773] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4256.986232] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4259.266656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4259.292796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4259.410643] Lustre: lustre-OST0000: new disk, initializing [ 4259.412462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4259.447855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4261.311479] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4264.300804] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4265.608915] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4268.023290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4268.026022] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4268.054883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4268.741757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4270.010344] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4270.082363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4273.124230] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4273.126039] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4273.128650] LustreError: Skipped 1 previous similar message [ 4273.136689] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4273.141807] Lustre: Skipped 1 previous similar message [ 4276.779105] LustreError: 294605:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4276.838984] Lustre: server umount lustre-OST0000 complete [ 4278.240415] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4278.241049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4278.244938] Lustre: Skipped 2 previous similar messages [ 4278.247786] Lustre: Skipped 1 previous similar message [ 4283.360156] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4283.361725] Lustre: Skipped 1 previous similar message [ 4284.347811] LustreError: 294807:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4284.349813] LustreError: 294807:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4284.374054] Lustre: server umount lustre-MDT0000 complete [ 4285.816343] LustreError: 291631:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145370 with bad export cookie 11288893189278184666 [ 4285.819340] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4285.820427] LustreError: 291631:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4285.877454] LustreError: 295008:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4285.879281] LustreError: 295008:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4285.952254] Lustre: server umount lustre-MDT0001 complete [ 4288.951994] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 11:02:52 (1761145372) [ 4291.576148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4291.789574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.002711] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4295.960302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4296.072591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4297.337352] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4299.177222] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4300.806993] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4303.293257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4305.145268] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4306.823719] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4321.764184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4321.764378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4321.765828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4321.774594] Lustre: Skipped 1 previous similar message [ 4326.637965] LustreError: 297944:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4326.639681] LustreError: 297944:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4326.694169] Lustre: server umount lustre-OST0000 complete [ 4330.977267] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4330.979163] LustreError: Skipped 1 previous similar message [ 4330.980187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4330.983504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4330.985165] Lustre: Skipped 1 previous similar message [ 4334.156979] LustreError: 298146:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4334.161064] LustreError: 298146:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4334.203266] Lustre: server umount lustre-MDT0000 complete [ 4335.399142] LustreError: 295702:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145419 with bad export cookie 11288893189278185548 [ 4335.401069] LustreError: 296401:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4335.402020] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4335.402142] LustreError: 295702:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4335.557253] Lustre: server umount lustre-MDT0001 complete [ 4339.400933] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 4340.449882] Key type lgssc unregistered [ 4340.589721] LNet: 298878:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4340.594288] LNetError: 298878:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4340.602767] LNet: Removed LNI 192.168.201.134@tcp [ 4340.976867] Key type .llcrypt unregistered [ 4340.977891] Key type ._llcrypt unregistered [ 4349.020408] Key type ._llcrypt registered [ 4349.022364] Key type .llcrypt registered [ 4349.060370] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 4354.615378] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4355.006141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4355.012114] alg: No test for adler32 (adler32-zlib) [ 4355.853735] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4355.942311] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 4357.519102] Key type lgssc registered [ 4357.902528] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4360.829876] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4362.826439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4364.780509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4366.759589] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4368.296489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4369.892389] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4372.033416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4372.059312] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4372.064433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4373.146761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4373.158392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4373.196400] Lustre: lustre-MDT0000: new disk, initializing [ 4373.223317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4373.229334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4374.612577] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4378.656207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4378.684228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4378.712203] Lustre: 302654:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4378.731486] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4378.734187] Lustre: Skipped 1 previous similar message [ 4378.768416] Lustre: lustre-MDT0001: new disk, initializing [ 4378.798351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4378.809417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4378.812116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4379.997802] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4382.939221] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4384.067917] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4386.423769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4386.454277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4386.562316] Lustre: lustre-OST2710: new disk, initializing [ 4386.564253] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4386.597714] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4388.444730] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4388.473493] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4388.476847] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4388.513122] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4391.566893] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4393.896360] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4393.925920] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4394.000027] Lustre: lustre-OST03e8: new disk, initializing [ 4394.002969] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4394.046109] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4396.143248] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4397.089573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4397.095150] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4397.143418] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4398.245775] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4400.712534] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4400.781393] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4402.058252] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4402.130078] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4403.422023] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4403.504336] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4404.814738] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4404.888545] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4407.266317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4407.268348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4407.270165] Lustre: Skipped 1 previous similar message [ 4412.157693] LustreError: 306994:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4412.181873] Lustre: server umount lustre-MDT0000 complete [ 4413.398261] LustreError: 302646:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145497 with bad export cookie 7428434830005535206 [ 4413.401816] LustreError: 302646:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4413.403146] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4413.465267] LustreError: 307196:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4413.468823] LustreError: 307196:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4413.555884] Lustre: server umount lustre-MDT0001 complete [ 4424.691103] LustreError: 307396:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4424.694099] LustreError: 307396:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4424.713694] Lustre: server umount lustre-OST2710 complete [ 4436.080250] LustreError: 307598:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4436.083203] LustreError: 307598:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4436.152995] Lustre: server umount lustre-OST03e8 complete [ 4441.817597] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 4444.966991] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4448.050904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4450.146657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4452.003281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4454.099950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4456.530535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4456.562091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4456.656414] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4456.667060] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4456.701592] Lustre: lustre-MDT0000: new disk, initializing [ 4456.728119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4456.733696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4458.213821] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4462.773153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4462.810326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4462.859087] Lustre: 310388:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4462.994290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4464.190291] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4467.537064] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4469.121122] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4471.564140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4471.605119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4471.737833] Lustre: lustre-OST0000: new disk, initializing [ 4471.739294] Lustre: Skipped 1 previous similar message [ 4471.741064] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4471.742960] Lustre: Skipped 2 previous similar messages [ 4471.786855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4471.790330] Lustre: Skipped 1 previous similar message [ 4473.949528] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4477.157363] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4477.961136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4477.967768] Lustre: Skipped 1 previous similar message [ 4477.970664] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4478.016793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4478.705804] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4478.780531] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4480.287554] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4480.373664] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4483.041205] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4483.042825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4483.043076] LustreError: Skipped 1 previous similar message [ 4483.046081] Lustre: Skipped 2 previous similar messages [ 4483.046529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4483.052018] Lustre: Skipped 4 previous similar messages [ 4487.084684] LustreError: 313326:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4487.086181] LustreError: 313326:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4487.147447] Lustre: server umount lustre-OST0000 complete [ 4488.671944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4488.673914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.676803] Lustre: Skipped 1 previous similar message [ 4488.678316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4493.280324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4493.280724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4494.489127] Lustre: server umount lustre-MDT0000 complete [ 4495.703640] LustreError: 311169:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145580 with bad export cookie 7428434830005536872 [ 4495.708096] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4495.771500] LustreError: 313728:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4495.773304] LustreError: 313728:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4495.884564] Lustre: server umount lustre-MDT0001 complete [ 4498.487284] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 11:06:22 (1761145582) [ 4498.943440] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4499.481801] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 11:06:23 (1761145583) [ 4500.465134] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4503.282494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4504.542761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4505.815427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4507.108891] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4507.470229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4509.882944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4509.993335] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4510.003089] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4510.069434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4511.580111] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4514.453563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4514.493665] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4514.507227] Lustre: 315853:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4515.869812] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4517.515813] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4518.743766] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4521.049909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4521.085059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4521.176163] Lustre: Denying initial registration attempt from nid 192.168.201.134@tcp, specified as failover [ 4521.179138] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4521.181966] LustreError: 317469:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -99 [ 4521.183830] LustreError: 317469:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4521.185418] LustreError: 317469:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4521.239404] Lustre: server umount lustre-OST0000 complete [ 4521.240962] LustreError: 317469:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4525.024334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4525.028076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4525.034448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4525.035950] Lustre: Skipped 1 previous similar message [ 4529.607094] LustreError: 317836:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4529.609154] LustreError: 317836:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4529.645587] Lustre: server umount lustre-MDT0000 complete [ 4530.656196] LustreError: 315864:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4530.660555] LustreError: 315864:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4531.013309] LustreError: 315888:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145615 with bad export cookie 7428434830005537740 [ 4531.016728] LustreError: 315888:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4531.017548] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4531.169379] Lustre: server umount lustre-MDT0001 complete [ 4535.226265] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 4536.324570] Key type lgssc unregistered [ 4536.487492] LNet: 318570:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4536.490686] LNetError: 318570:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4536.501476] LNet: Removed LNI 192.168.201.134@tcp [ 4536.866274] Key type .llcrypt unregistered [ 4536.867280] Key type ._llcrypt unregistered [ 4540.801740] Key type ._llcrypt registered [ 4540.802819] Key type .llcrypt registered [ 4540.835795] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4541.117956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4541.151105] alg: No test for adler32 (adler32-zlib) [ 4542.011264] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4542.105387] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 4543.687212] Key type lgssc registered [ 4544.067958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4546.568698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4547.858541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4549.481253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4550.966897] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4551.295328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4557.954983] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4561.671171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4561.677549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4562.780457] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4562.790274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4562.855089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4564.203493] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4567.542020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4567.587073] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4567.601021] Lustre: 321301:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4567.725556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4569.378992] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4571.389636] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4572.576806] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4575.021968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4575.057330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4575.147187] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4575.235783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4577.721311] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4580.183808] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4582.371152] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4582.373846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.375650] LustreError: Skipped 1 previous similar message [ 4582.377640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4582.385737] Lustre: Skipped 1 previous similar message [ 4583.154196] LustreError: 323586:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4583.264970] Lustre: server umount lustre-OST0000 complete [ 4587.488769] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4587.489487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4587.492192] Lustre: Skipped 1 previous similar message [ 4587.493661] Lustre: Skipped 1 previous similar message [ 4592.609388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4592.615137] Lustre: Skipped 2 previous similar messages [ 4597.728039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4597.729559] Lustre: Skipped 1 previous similar message [ 4599.263094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4599.296073] LustreError: 323787:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4599.297697] LustreError: 323787:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4599.323489] Lustre: server umount lustre-MDT0000 complete [ 4600.807128] LustreError: 321294:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145685 with bad export cookie 6870391594393888508 [ 4600.813095] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4600.813688] LustreError: 321294:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4600.904129] LustreError: 323990:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4600.907394] LustreError: 323990:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4601.003909] Lustre: server umount lustre-MDT0001 complete [ 4605.476575] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 4606.887304] Key type lgssc unregistered [ 4607.045455] LNet: 324521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4607.048651] LNetError: 324521:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4607.058616] LNet: Removed LNI 192.168.201.134@tcp [ 4607.569308] Key type .llcrypt unregistered [ 4607.571179] Key type ._llcrypt unregistered [ 4619.031665] Key type ._llcrypt registered [ 4619.032932] Key type .llcrypt registered [ 4619.073765] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4619.445133] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4619.488366] alg: No test for adler32 (adler32-zlib) [ 4620.417033] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4620.547416] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 4622.159192] Key type lgssc registered [ 4622.744395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4626.593233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4626.604531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.800639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4629.233636] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4633.262477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4633.504224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4635.283318] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4637.281579] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4639.102671] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4642.346247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4642.544710] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4645.015770] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4647.214603] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4658.013383] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.34@tcp (stopping) [ 4658.147939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4658.156553] Lustre: Skipped 2 previous similar messages [ 4660.987322] LustreError: 328143:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4661.015701] Lustre: server umount lustre-MDT0000 complete [ 4662.674896] LustreError: 325875:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145747 with bad export cookie 2663546170703536581 [ 4662.681262] LustreError: 325875:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4662.681507] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4663.119572] LustreError: 325890:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4663.129196] LustreError: 325890:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4663.136554] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.34@tcp (stopping) [ 4663.140332] Lustre: Skipped 3 previous similar messages [ 4663.264886] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4667.295856] LustreError: 328050:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4667.296090] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4667.299669] LustreError: 328050:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4667.301229] Lustre: Skipped 1 previous similar message [ 4668.794678] LustreError: 328345:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4668.796310] LustreError: 328345:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4668.878563] Lustre: server umount lustre-MDT0001 complete [ 4670.877597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4673.523208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4687.775963] LustreError: 325205:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000886cd9e4 x1846695117740672/t0(0) o250->MGC192.168.201.134@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4687.932369] LustreError: 327504:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4687.937896] LustreError: 327504:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4687.956514] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4687.974293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4689.334401] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4692.064688] LustreError: 329156:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4692.070018] LustreError: 329156:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4692.205015] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4692.371314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4693.953095] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4695.910458] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4697.512450] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4698.623252] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 2 sec [ 4700.076151] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4700.583670] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4702.690596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4702.695288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4702.695609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4702.701571] Lustre: Skipped 2 previous similar messages [ 4707.812404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4707.815590] Lustre: Skipped 1 previous similar message [ 4707.958162] LustreError: 330651:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4707.963690] LustreError: 330651:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4708.064211] Lustre: server umount lustre-OST0000 complete [ 4712.930956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4712.931379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4712.942749] Lustre: Skipped 2 previous similar messages [ 4715.655495] LustreError: 330852:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4715.658786] LustreError: 330852:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4715.696154] Lustre: server umount lustre-MDT0000 complete [ 4717.179796] LustreError: 325876:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145801 with bad export cookie 2663546170703543735 [ 4717.185117] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4717.373852] Lustre: server umount lustre-MDT0001 complete [ 4721.850983] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 4723.505677] Key type lgssc unregistered [ 4723.710672] LNet: 331585:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4723.716132] LNetError: 331585:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4723.727825] LNet: Removed LNI 192.168.201.134@tcp [ 4724.232108] Key type .llcrypt unregistered [ 4724.234213] Key type ._llcrypt unregistered [ 4734.752989] Key type ._llcrypt registered [ 4734.754315] Key type .llcrypt registered [ 4734.809333] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4735.210325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4735.234221] alg: No test for adler32 (adler32-zlib) [ 4736.085108] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4736.166175] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 4737.759130] Key type lgssc registered [ 4738.088286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4741.012658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4741.026124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4742.270428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4743.910715] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4747.819688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4747.938995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4749.566935] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4751.397435] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4753.090245] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4757.473297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.474610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4757.479174] Lustre: Skipped 1 previous similar message [ 4757.484217] Lustre: Skipped 1 previous similar message [ 4758.497026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4758.499580] Lustre: Skipped 1 previous similar message [ 4760.323315] LustreError: 334351:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4760.354230] Lustre: server umount lustre-MDT0000 complete [ 4761.689542] LustreError: 332941:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145846 with bad export cookie 711451443607026470 [ 4761.693244] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4761.695769] LustreError: 332941:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4761.766538] LustreError: 334556:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4761.769444] LustreError: 334556:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4761.862616] Lustre: server umount lustre-MDT0001 complete [ 4767.155401] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.286972] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4767.300584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4767.375608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4768.733756] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4771.838479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4771.874971] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4771.888171] Lustre: 335323:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4771.993445] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4773.393335] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4775.048263] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4776.093987] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4778.696206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4778.761407] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4778.764287] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4778.766564] LustreError: 336981:0:(tgt_mount.c:2479:server_fill_super()) Unable to start targets: -2 [ 4778.768381] LustreError: 336981:0:(tgt_mount.c:1973:server_put_super()) no obd lustre-OST0000 [ 4778.770325] LustreError: 336981:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4778.831453] Lustre: server umount lustre-OST0000 complete [ 4778.833728] LustreError: 336981:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4781.687694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4781.756803] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4781.832169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4783.657853] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4785.035631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 4785.449857] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4787.921380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4787.944638] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4787.966572] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4787.996663] Lustre: lustre-OST0001: new disk, initializing [ 4787.998677] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4789.918391] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4791.622280] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4791.625073] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4791.659521] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4791.740557] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4796.900729] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4796.900940] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.902517] LustreError: Skipped 1 previous similar message [ 4796.902978] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4796.906729] Lustre: Skipped 1 previous similar message [ 4796.910759] Lustre: Skipped 1 previous similar message [ 4798.764087] LustreError: 339098:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4798.765897] LustreError: 339098:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4798.784436] Lustre: server umount lustre-OST0001 complete [ 4800.480378] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4800.480592] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4800.482407] LustreError: Skipped 1 previous similar message [ 4800.489440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4800.492391] Lustre: Skipped 1 previous similar message [ 4802.022207] LustreError: 337447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4802.029469] LustreError: 337447:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4806.187078] LustreError: 339300:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4806.190553] LustreError: 339300:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4806.260154] Lustre: server umount lustre-OST0000 complete [ 4808.159918] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4808.161856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4808.164704] Lustre: Skipped 1 previous similar message [ 4808.166106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4808.170007] Lustre: Skipped 2 previous similar messages [ 4812.256685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.693568] LustreError: 339501:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4813.696622] LustreError: 339501:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4813.723842] Lustre: server umount lustre-MDT0000 complete [ 4814.976150] LustreError: 335314:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145899 with bad export cookie 711451443607027009 [ 4814.979154] LustreError: 335314:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4814.980314] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4815.139552] Lustre: server umount lustre-MDT0001 complete [ 4818.682690] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 4819.872515] Key type lgssc unregistered [ 4820.029417] LNet: 340233:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4820.031877] LNetError: 340233:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4820.040514] LNet: Removed LNI 192.168.201.134@tcp [ 4820.459396] Key type .llcrypt unregistered [ 4820.460648] Key type ._llcrypt unregistered [ 4831.856091] Key type ._llcrypt registered [ 4831.857052] Key type .llcrypt registered [ 4831.910330] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 4838.870850] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4839.451321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4839.470804] alg: No test for adler32 (adler32-zlib) [ 4840.375892] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 4840.482628] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 4842.079163] Key type lgssc registered [ 4842.450385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4845.688725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4847.921826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4850.041761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4852.165042] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4854.463037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4854.485900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4854.492812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4855.585131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4855.597874] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4855.637446] Lustre: lustre-MDT0000: new disk, initializing [ 4855.666143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4855.672585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4857.073311] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4861.427350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4861.452626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4861.485446] Lustre: 344155:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4861.513514] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4861.515183] Lustre: Skipped 1 previous similar message [ 4861.550366] Lustre: lustre-MDT0001: new disk, initializing [ 4861.589932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4861.603568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4861.606849] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4863.058716] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4866.176215] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4867.438979] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4869.768985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4869.799704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4869.917419] Lustre: lustre-OST0000: new disk, initializing [ 4869.920481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4869.955477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4871.867884] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4873.273367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4873.276373] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4873.314667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4874.971543] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4876.361863] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4876.446863] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4877.815226] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4877.899759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4878.816197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4878.818634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4878.822565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4883.427700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4883.427899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4883.434090] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4884.587253] LustreError: 347093:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4884.659437] Lustre: server umount lustre-OST0000 complete [ 4887.520185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4887.523173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4887.527057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4887.529461] Lustre: Skipped 1 previous similar message [ 4893.664330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4893.666544] Lustre: Skipped 3 previous similar messages [ 4898.784170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4898.786010] Lustre: Skipped 1 previous similar message [ 4900.319187] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4900.350295] LustreError: 347295:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4900.352936] LustreError: 347295:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4900.382895] Lustre: server umount lustre-MDT0000 complete [ 4901.634473] LustreError: 347297:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761145985 with bad export cookie 12991048577440680978 [ 4901.637657] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4901.640279] LustreError: 347297:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4901.696894] LustreError: 347500:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4901.699264] LustreError: 347500:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4901.785198] Lustre: server umount lustre-MDT0001 complete [ 4904.532204] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 11:13:08 (1761145988) [ 4906.626032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4908.671419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4916.260856] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 4919.314035] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 4922.592874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4924.771302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4926.778378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4928.849678] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4931.124865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4931.150254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.233892] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4931.245483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4931.284789] Lustre: lustre-MDT0000: new disk, initializing [ 4931.309345] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4931.316308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4932.600879] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4936.742049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4936.763922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4936.796498] Lustre: 351266:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4936.823341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4936.826321] Lustre: Skipped 1 previous similar message [ 4936.860971] Lustre: lustre-MDT0001: new disk, initializing [ 4936.895361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4936.917386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4936.920290] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4938.267608] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4941.356703] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4942.551986] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4944.848381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4944.873680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4944.975135] Lustre: lustre-OST0000: new disk, initializing [ 4944.977429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4945.010075] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4945.511925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4945.517267] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4945.562822] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4946.893289] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4950.102364] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4951.417358] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4951.511797] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4952.944118] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4953.030611] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4956.129756] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4956.129872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4956.132023] LustreError: Skipped 1 previous similar message [ 4956.134917] Lustre: Skipped 1 previous similar message [ 4956.135554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4956.139757] Lustre: Skipped 1 previous similar message [ 4959.790119] LustreError: 354208:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4959.792693] LustreError: 354208:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4959.859196] Lustre: server umount lustre-OST0000 complete [ 4961.248634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4961.254200] Lustre: Skipped 2 previous similar messages [ 4967.228026] LustreError: 354409:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4967.229832] LustreError: 354409:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4967.259204] Lustre: server umount lustre-MDT0000 complete [ 4968.519294] LustreError: 351257:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146052 with bad export cookie 12991048577440681853 [ 4968.522655] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4968.525124] LustreError: 351257:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4968.687786] Lustre: server umount lustre-MDT0001 complete [ 4971.429744] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 11:14:15 (1761146055) [ 4973.945884] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 11:14:17 (1761146057) [ 4976.791682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4976.991052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4978.370564] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4981.644919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4983.251970] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4985.198857] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4986.348632] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4988.558300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4990.800777] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 4992.565300] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5006.441936] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5007.017413] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5007.568842] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5008.149796] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5008.830675] Lustre: Failing over lustre-MDT0000 [ 5008.992766] LustreError: 358081:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5008.995013] LustreError: 358081:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5009.031932] Lustre: server umount lustre-MDT0000 complete [ 5012.448964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5012.453031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5012.460238] LustreError: 355624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5012.468261] LustreError: 355624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5013.336184] LustreError: 357238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5018.448342] LustreError: 357238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.34@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5018.458268] LustreError: 357238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5022.501957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5022.556795] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5022.658617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5022.660480] Lustre: Skipped 2 previous similar messages [ 5022.680054] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5023.567294] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5024.138140] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5027.814328] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5027.827537] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5027.845271] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5029.029522] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5029.650465] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5031.554630] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5032.231742] Lustre: Failing over lustre-MDT0000 [ 5032.351951] LustreError: 359161:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5032.353731] LustreError: 359161:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5032.379659] Lustre: server umount lustre-MDT0000 complete [ 5032.927799] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5032.930593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.933677] Lustre: Skipped 4 previous similar messages [ 5032.933789] LustreError: 357238:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5032.942302] LustreError: 357238:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 5038.050233] LustreError: 355624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5038.059530] LustreError: 355624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5045.271313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5045.312153] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5045.425357] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5046.787055] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5049.168638] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5050.857878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5050.861720] Lustre: Skipped 2 previous similar messages [ 5050.870269] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5050.889734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5052.047666] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5052.606237] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5055.970263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5055.973332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5055.974792] Lustre: Skipped 6 previous similar messages [ 5061.046446] Lustre: server umount lustre-MDT0000 complete [ 5061.089035] LustreError: 355624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5061.093029] LustreError: 355624:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 5062.329213] LustreError: 358084:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146146 with bad export cookie 12991048577440684506 [ 5062.332867] LustreError: 358084:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5062.333484] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5062.474006] Lustre: server umount lustre-MDT0001 complete [ 5074.096250] LustreError: 360592:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5074.098815] LustreError: 360592:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5074.156582] Lustre: server umount lustre-OST0000 complete [ 5079.128043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5079.274141] LustreError: 361419:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5080.550947] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5083.498296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5084.936936] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5086.634096] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5087.904111] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5090.165222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5090.283288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5090.285261] Lustre: Skipped 3 previous similar messages [ 5092.206130] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5094.059842] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5104.613468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5113.823868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5113.828023] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5113.831481] Lustre: Skipped 3 previous similar messages [ 5113.832970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5116.897374] Lustre: server umount lustre-OST0000 complete [ 5124.373790] Lustre: server umount lustre-MDT0000 complete [ 5125.087929] LustreError: 362781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5125.091951] LustreError: 362781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5125.592352] LustreError: 361444:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146209 with bad export cookie 12991048577440685213 [ 5125.597280] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5125.752684] Lustre: server umount lustre-MDT0001 complete [ 5129.564505] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 5130.838490] Key type lgssc unregistered [ 5130.995739] LNet: 364620:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5131.000323] LNetError: 364620:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5131.009659] LNet: Removed LNI 192.168.201.134@tcp [ 5131.376585] Key type .llcrypt unregistered [ 5131.378669] Key type ._llcrypt unregistered [ 5141.541851] Key type ._llcrypt registered [ 5141.543035] Key type .llcrypt registered [ 5141.587285] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 5142.111785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5142.127198] alg: No test for adler32 (adler32-zlib) [ 5142.992836] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 5143.090105] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 5144.671094] Key type lgssc registered [ 5145.007911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5147.649601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5147.659258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5148.822058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5150.110025] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5153.038035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5153.152738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5154.464203] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5156.391512] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5157.527306] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5159.779566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5159.916807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5161.871847] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5163.685237] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5174.247422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5183.967966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5183.970071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5183.973585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5184.481060] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5184.483848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5184.487778] Lustre: Skipped 1 previous similar message [ 5189.183301] LustreError: 368363:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5189.219507] Lustre: server umount lustre-MDT0000 complete [ 5189.601545] LustreError: 366094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5189.608951] LustreError: 366094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5190.409351] LustreError: 367714:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146274 with bad export cookie 5509189433160701364 [ 5190.412610] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5190.412776] LustreError: 367714:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5190.478396] LustreError: 368566:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5190.480988] LustreError: 368566:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5190.555861] Lustre: server umount lustre-MDT0001 complete [ 5202.041410] LustreError: 368768:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5202.045148] LustreError: 368768:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5202.134533] Lustre: server umount lustre-OST0000 complete [ 5206.665980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5206.834053] LustreError: 369498:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5206.862510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5208.176772] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5211.015282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5211.138176] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5212.416284] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5214.175983] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5215.287783] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5217.528394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5219.381590] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5221.066957] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5231.971121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5246.944010] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5246.946858] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5246.950548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5246.952838] Lustre: Skipped 2 previous similar messages [ 5249.403737] LustreError: 371866:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5249.407074] LustreError: 371866:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5249.438690] Lustre: server umount lustre-MDT0000 complete [ 5250.680675] LustreError: 369479:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146335 with bad export cookie 5509189433160702463 [ 5250.686148] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5250.851282] Lustre: server umount lustre-MDT0001 complete [ 5262.384577] LustreError: 372271:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5262.388139] LustreError: 372271:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5262.473145] Lustre: server umount lustre-OST0000 complete [ 5267.691347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5267.844770] LustreError: 373098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5267.871656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5267.875507] Lustre: Skipped 1 previous similar message [ 5269.177331] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5272.132516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5273.653203] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5275.561118] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5276.830979] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5279.269673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5281.692518] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5283.564671] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5293.795268] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5298.146393] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5298.150980] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5298.157448] Lustre: Skipped 2 previous similar messages [ 5298.160475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5298.163972] Lustre: Skipped 3 previous similar messages [ 5301.102121] LustreError: 375365:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5301.103733] LustreError: 375365:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5301.162861] Lustre: server umount lustre-OST0000 complete [ 5303.264106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5303.266339] LustreError: Skipped 1 previous similar message [ 5303.267596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5303.271156] Lustre: Skipped 1 previous similar message [ 5303.273119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5303.275140] Lustre: Skipped 2 previous similar messages [ 5308.499330] Lustre: server umount lustre-MDT0000 complete [ 5309.409658] LustreError: 373116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5309.413459] LustreError: 373116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5309.666780] LustreError: 373079:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146394 with bad export cookie 5509189433160703471 [ 5309.669896] LustreError: 373079:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5309.671380] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5309.835118] Lustre: server umount lustre-MDT0001 complete [ 5313.113471] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 5314.185584] Key type lgssc unregistered [ 5314.321418] LNet: 376299:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5314.323440] LNetError: 376299:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5314.334360] LNet: Removed LNI 192.168.201.134@tcp [ 5314.638364] Key type .llcrypt unregistered [ 5314.639509] Key type ._llcrypt unregistered [ 5323.347469] Key type ._llcrypt registered [ 5323.348429] Key type .llcrypt registered [ 5323.390537] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 5323.840293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5323.879384] alg: No test for adler32 (adler32-zlib) [ 5324.741778] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 5324.831310] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 5326.415108] Key type lgssc registered [ 5326.800522] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5329.272125] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5329.277190] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5329.279684] LustreError: 377698:0:(osd_handler.c:8502:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5329.287953] LustreError: 377698:0:(obd_config.c:783:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5329.290229] LustreError: 377698:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5329.292028] LustreError: 377698:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5329.293971] LustreError: 377698:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5331.613334] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5331.615378] LustreError: 378095:0:(osd_handler.c:8502:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5331.624068] LustreError: 378095:0:(obd_config.c:783:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5331.625941] LustreError: 378095:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5331.627805] LustreError: 378095:0:(tgt_mount.c:2447:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5331.629784] LustreError: 378095:0:(super25.c:199:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5337.761252] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 5338.828297] Key type lgssc unregistered [ 5338.958362] LNet: 379078:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5338.961139] LNetError: 379078:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5338.971308] LNet: Removed LNI 192.168.201.134@tcp [ 5339.259947] Key type .llcrypt unregistered [ 5339.260945] Key type ._llcrypt unregistered [ 5344.937793] Key type ._llcrypt registered [ 5344.939383] Key type .llcrypt registered [ 5344.977204] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 5350.881477] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 5351.283661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5351.330216] alg: No test for adler32 (adler32-zlib) [ 5352.185584] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 5352.271586] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 5353.847149] Key type lgssc registered [ 5354.211214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5357.160072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5359.234954] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5361.186339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5363.181960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5365.378583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5365.398576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5365.404402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5366.499759] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5366.511430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5366.551683] Lustre: lustre-MDT0000: new disk, initializing [ 5366.577740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5366.584151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5367.821316] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5371.811128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5371.837423] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5371.872870] Lustre: 382182:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5371.897823] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5371.899356] Lustre: Skipped 1 previous similar message [ 5371.932800] Lustre: lustre-MDT0001: new disk, initializing [ 5371.970253] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5371.982113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5371.985454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5373.230603] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5376.272929] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5377.401465] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5379.466015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5379.495392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5379.614047] Lustre: lustre-OST0000: new disk, initializing [ 5379.616209] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5379.649221] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5381.434406] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5384.502263] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5385.863791] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5386.234734] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5386.237970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5386.275391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5386.970795] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5388.320080] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5388.394305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5391.330387] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5391.330996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5391.334194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5391.338201] Lustre: Skipped 1 previous similar message [ 5395.054345] LustreError: 385125:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5395.130423] Lustre: server umount lustre-OST0000 complete [ 5396.448434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.448984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5396.454701] Lustre: Skipped 1 previous similar message [ 5396.456211] Lustre: Skipped 1 previous similar message [ 5401.568518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5401.571604] Lustre: Skipped 2 previous similar messages [ 5402.428675] LustreError: 385326:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5402.431652] LustreError: 385326:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5402.457592] Lustre: server umount lustre-MDT0000 complete [ 5403.632297] LustreError: 382173:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146487 with bad export cookie 16581281504477417493 [ 5403.635472] LustreError: 382173:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5403.635555] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5403.699204] LustreError: 385528:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5403.700949] LustreError: 385528:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5403.796383] Lustre: server umount lustre-MDT0001 complete [ 5406.457721] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5406.993192] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 11:21:30 (1761146490) [ 5409.108940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5409.273679] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5410.492026] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5413.250069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5413.378958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5414.634606] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5416.255320] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5417.381590] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5419.537472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5421.338411] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5423.004489] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5425.071653] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5425.096294] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5425.160063] Lustre: lustre-OST0001: new disk, initializing [ 5425.161754] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5425.194560] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5425.196357] Lustre: Skipped 1 previous similar message [ 5427.002337] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5427.911147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5427.914511] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5427.955757] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5429.921148] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5433.318548] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5433.318723] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5433.320656] LustreError: Skipped 2 previous similar messages [ 5433.321231] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5433.323981] Lustre: Skipped 1 previous similar message [ 5437.746091] LustreError: 389445:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5437.748767] LustreError: 389445:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5437.764436] Lustre: server umount lustre-OST0001 complete [ 5439.463042] LustreError: 388351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5439.469697] LustreError: 388351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5444.579515] LustreError: 387841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5444.583954] LustreError: 387841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5449.705779] LustreError: 387841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5449.710368] LustreError: 387841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5453.792169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5453.794710] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5453.800067] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5453.801929] Lustre: Skipped 4 previous similar messages [ 5454.821343] LustreError: 388351:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5454.825153] LustreError: 388351:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5456.046169] LustreError: 389669:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5456.048215] LustreError: 389669:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5456.128037] Lustre: server umount lustre-OST0000 complete [ 5459.423888] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5459.425897] LustreError: Skipped 1 previous similar message [ 5459.427080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.429727] Lustre: Skipped 1 previous similar message [ 5463.590416] Lustre: server umount lustre-MDT0000 complete [ 5464.791362] LustreError: 386226:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146549 with bad export cookie 16581281504477418361 [ 5464.795069] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5464.796675] LustreError: 386226:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5479.391236] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5479.444747] LustreError: 390071:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 5479.447961] LustreError: 390071:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5479.546919] Lustre: server umount lustre-MDT0001 complete [ 5483.089223] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 5484.188525] Key type lgssc unregistered [ 5484.319374] LNet: 390603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5484.321343] LNetError: 390603:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5484.332408] LNet: Removed LNI 192.168.201.134@tcp [ 5484.642814] Key type .llcrypt unregistered [ 5484.643795] Key type ._llcrypt unregistered [ 5497.501395] Key type ._llcrypt registered [ 5497.502519] Key type .llcrypt registered [ 5497.536507] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 5504.403142] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 5504.800290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5504.810460] alg: No test for adler32 (adler32-zlib) [ 5505.661403] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 5505.748596] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 5507.327164] Key type lgssc registered [ 5507.710843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5511.145921] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5513.404318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5515.424868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5517.667488] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5520.062655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5520.096640] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5520.107149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5521.213410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5521.228338] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5521.266693] Lustre: lustre-MDT0000: new disk, initializing [ 5521.290992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5521.297534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5522.641933] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5526.615438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5526.636610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5526.664742] Lustre: 394525:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5526.685219] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5526.686696] Lustre: Skipped 1 previous similar message [ 5526.721287] Lustre: lustre-MDT0001: new disk, initializing [ 5526.751629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5526.763076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5526.766688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5528.016324] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5530.967048] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5532.097224] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5534.091154] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5534.111794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5534.207554] Lustre: lustre-OST0000: new disk, initializing [ 5534.209667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5534.235291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5535.877061] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5537.719840] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5537.724232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5537.758969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5538.780267] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5540.033653] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5540.100578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5541.265272] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5541.327698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5542.881242] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5542.882209] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.883310] LustreError: Skipped 1 previous similar message [ 5542.886945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5542.888728] Lustre: Skipped 1 previous similar message [ 5543.980158] LustreError: 397464:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5544.037375] Lustre: server umount lustre-OST0000 complete [ 5547.487737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5547.489906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5547.493069] Lustre: Skipped 1 previous similar message [ 5547.494569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5551.290226] LustreError: 397665:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5551.291770] LustreError: 397665:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5551.320659] Lustre: server umount lustre-MDT0000 complete [ 5552.461388] LustreError: 394516:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146636 with bad export cookie 7882525972991387811 [ 5552.465059] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5552.466586] LustreError: 394516:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5552.527529] LustreError: 397867:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5552.529338] LustreError: 397867:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5552.609436] Lustre: server umount lustre-MDT0001 complete [ 5555.250851] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 11:23:59 (1761146639) [ 5558.022911] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5560.550733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5560.682884] Lustre: lustre-MDT0000: new disk, initializing [ 5560.711426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.846562] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5564.373585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5565.731784] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5567.367324] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5568.505834] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5569.139154] LustreError: 399163:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5569.143314] LustreError: 399163:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5569.145847] LustreError: 399163:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 [ 5569.504390] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5569.504426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5569.504969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5569.504974] Lustre: Skipped 1 previous similar message [ 5569.516491] Lustre: Skipped 2 previous similar messages [ 5574.624185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5574.625614] Lustre: Skipped 2 previous similar messages [ 5575.288199] LustreError: 400474:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5575.289829] LustreError: 400474:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5575.313960] Lustre: server umount lustre-MDT0000 complete [ 5576.444279] LustreError: 399112:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146660 with bad export cookie 7882525972991388679 [ 5576.447078] LustreError: 399112:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5576.447906] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5576.568735] Lustre: server umount lustre-MDT0001 complete [ 5579.518776] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 11:24:23 (1761146663) [ 5581.600431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.747164] LustreError: 401395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5581.754789] LustreError: 401395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5581.768500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5581.771883] Lustre: Skipped 1 previous similar message [ 5582.961228] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5585.628587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5586.914840] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5588.643644] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5589.798150] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5591.901401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5592.010353] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5592.012048] Lustre: Skipped 1 previous similar message [ 5594.218565] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5596.439518] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5600.644195] Lustre: Permanently deactivating lustre-OST0000 [ 5600.652854] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5608.430685] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5608.930630] LustreError: 404134:0:(mgs_llog.c:1938:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5608.933295] LustreError: 404134:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5608.934982] LustreError: 404134:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5609.492832] LustreError: 404182:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5609.495027] LustreError: 404182:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5616.109997] LustreError: 404279:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5616.111770] LustreError: 404279:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5616.171647] Lustre: server umount lustre-OST0000 complete [ 5617.006083] LustreError: 404432:0:(mgs_llog.c:1992:mgs_replace_nids()) Only MGS is allowed to be started [ 5617.009791] LustreError: 404432:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5618.656423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5618.657050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5618.659433] Lustre: Skipped 1 previous similar message [ 5626.850429] LustreError: 404529:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5626.851928] LustreError: 404529:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5626.872609] Lustre: server umount lustre-MDT0000 complete [ 5627.958091] LustreError: 401376:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146712 with bad export cookie 7882525972991389211 [ 5627.960852] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5627.962771] LustreError: 401376:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5638.180747] Lustre: server umount lustre-MDT0001 complete [ 5640.740663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5641.878947] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5642.702848] LustreError: 405573:0:(mgs_handler.c:1168:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5644.673670] Lustre: 405912:0:(mgs_llog.c:1699:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.134@tcp [ 5645.308285] LustreError: 406009:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5645.309992] LustreError: 406009:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5645.335799] Lustre: server umount MGS complete [ 5648.933413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5649.093652] LustreError: 406636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5649.097983] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5649.105957] Lustre: Skipped 1 previous similar message [ 5649.123943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5650.227791] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5652.840425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5652.933406] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5654.142173] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5655.774770] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5656.840650] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5657.444146] Lustre: Permanently reactivating lustre-OST0000 [ 5657.448502] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5657.450576] Lustre: Skipped 2 previous similar messages [ 5679.071122] LustreError: 408048:0:(osp_dev.c:692:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5681.337152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5681.450812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5681.452676] Lustre: Skipped 1 previous similar message [ 5683.301110] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5684.822110] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5699.046348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5699.046686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5699.048801] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5699.052070] Lustre: Skipped 1 previous similar message [ 5699.053524] Lustre: Skipped 3 previous similar messages [ 5702.573222] LustreError: 409873:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5702.574894] LustreError: 409873:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5702.631130] Lustre: server umount lustre-OST0000 complete [ 5704.160779] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5704.163143] LustreError: Skipped 1 previous similar message [ 5717.983071] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5718.032537] Lustre: server umount lustre-MDT0000 complete [ 5719.138169] LustreError: 407987:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146803 with bad export cookie 7882525972991390156 [ 5719.141262] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5719.142641] LustreError: 407987:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5719.272581] Lustre: server umount lustre-MDT0001 complete [ 5722.573895] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 5723.460578] Key type lgssc unregistered [ 5723.577329] LNet: 410811:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5723.579359] LNetError: 410811:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5723.586278] LNet: Removed LNI 192.168.201.134@tcp [ 5723.854040] Key type .llcrypt unregistered [ 5723.855718] Key type ._llcrypt unregistered [ 5729.320228] Key type ._llcrypt registered [ 5729.321217] Key type .llcrypt registered [ 5729.354715] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 5735.348709] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 5735.636287] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5735.673558] alg: No test for adler32 (adler32-zlib) [ 5736.512013] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 5736.586742] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 5738.159128] Key type lgssc registered [ 5738.472981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5741.203084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5743.306230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5745.131487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5747.036551] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5749.222554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5749.247145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5749.252295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5750.340045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5750.352795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5750.392571] Lustre: lustre-MDT0000: new disk, initializing [ 5750.418088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5750.424507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5751.678485] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5755.617447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5755.638789] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5755.671448] Lustre: 413915:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5755.691483] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5755.692931] Lustre: Skipped 1 previous similar message [ 5755.727403] Lustre: lustre-MDT0001: new disk, initializing [ 5755.734448] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 5755.761970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5755.773583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5755.778700] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5756.908252] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5759.884674] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5760.944217] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5762.973993] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5762.995718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5763.081505] Lustre: lustre-OST0000: new disk, initializing [ 5763.083113] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5763.109491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5764.755788] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5767.624231] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5768.837905] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5769.205949] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5769.210165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5769.241455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5769.925935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5771.100746] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5771.164109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5774.306353] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5774.306592] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5774.307931] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5774.312234] Lustre: Skipped 1 previous similar message [ 5777.838086] LustreError: 416860:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5777.901221] Lustre: server umount lustre-OST0000 complete [ 5779.424356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5779.424878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5779.429261] Lustre: Skipped 1 previous similar message [ 5779.430693] Lustre: Skipped 1 previous similar message [ 5784.544148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5784.545800] Lustre: Skipped 2 previous similar messages [ 5785.210177] LustreError: 417062:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5785.211779] LustreError: 417062:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5785.242246] Lustre: server umount lustre-MDT0000 complete [ 5786.373307] LustreError: 414692:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146870 with bad export cookie 14297308537536721092 [ 5786.375432] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5786.378229] LustreError: 414692:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5786.439312] LustreError: 417263:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5786.442591] LustreError: 417263:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5786.519220] Lustre: server umount lustre-MDT0001 complete [ 5789.250819] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5789.799231] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 11:27:53 (1761146873) [ 5791.966811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.144347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5793.346489] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5796.009307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5797.347092] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5798.932869] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5800.049420] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5802.150212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5802.253735] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.255699] Lustre: Skipped 1 previous similar message [ 5804.037344] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5805.761700] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5806.347851] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5807.785162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5811.683982] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.685521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5811.689914] Lustre: Skipped 1 previous similar message [ 5811.693793] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5811.697370] LustreError: Skipped 1 previous similar message [ 5815.082129] LustreError: 420418:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5815.085275] LustreError: 420418:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5815.141314] Lustre: server umount lustre-OST0000 complete [ 5816.800546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5816.800646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5816.805371] Lustre: Skipped 1 previous similar message [ 5821.920170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5821.922028] Lustre: Skipped 4 previous similar messages [ 5822.394423] LustreError: 420620:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5822.396805] LustreError: 420620:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5822.422400] Lustre: server umount lustre-MDT0000 complete [ 5823.588531] LustreError: 417961:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146907 with bad export cookie 14297308537536721974 [ 5823.591081] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5823.592209] LustreError: 417961:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5823.740450] Lustre: server umount lustre-MDT0001 complete [ 5827.219565] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 5828.286139] Key type lgssc unregistered [ 5828.413375] LNet: 421351:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5828.415477] LNetError: 421351:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5828.426426] LNet: Removed LNI 192.168.201.134@tcp [ 5828.730865] Key type .llcrypt unregistered [ 5828.731800] Key type ._llcrypt unregistered [ 5836.051372] Key type ._llcrypt registered [ 5836.052417] Key type .llcrypt registered [ 5836.084555] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 5836.171294] Key type .llcrypt unregistered [ 5836.172319] Key type ._llcrypt unregistered [ 5842.837321] Key type ._llcrypt registered [ 5842.838249] Key type .llcrypt registered [ 5842.867771] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 5843.147709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5843.188039] alg: No test for adler32 (adler32-zlib) [ 5844.025361] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 5844.098860] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 5845.671131] Key type lgssc registered [ 5846.003299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5848.421343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5848.426872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5849.570314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5850.711997] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5853.329065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5853.431949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.058860] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5856.658592] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5858.750926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5858.784246] LustreError: 423523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5858.788063] LustreError: 423523:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5858.869320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5859.879428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5860.024298] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5863.904382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5863.906637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.910215] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.929595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5864.930682] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5864.931264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5864.933021] Lustre: Skipped 1 previous similar message [ 5868.844900] LustreError: 425555:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5868.900390] Lustre: server umount lustre-OST0000 complete [ 5870.049028] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5870.049830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5870.052395] Lustre: Skipped 1 previous similar message [ 5870.055376] Lustre: Skipped 1 previous similar message [ 5875.168108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5875.170562] Lustre: Skipped 1 previous similar message [ 5876.156346] LustreError: 425756:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5876.158098] LustreError: 425756:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5876.184141] Lustre: server umount lustre-MDT0000 complete [ 5877.326134] LustreError: 423480:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761146961 with bad export cookie 15071357285056315802 [ 5877.329043] LustreError: 423480:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5877.329571] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5877.388612] LustreError: 425958:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5877.391191] LustreError: 425958:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5877.472539] Lustre: server umount lustre-MDT0001 complete [ 5880.951585] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 5881.922416] Key type lgssc unregistered [ 5882.050399] LNet: 426489:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5882.052376] LNetError: 426489:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5882.059269] LNet: Removed LNI 192.168.201.134@tcp [ 5882.336964] Key type .llcrypt unregistered [ 5882.337903] Key type ._llcrypt unregistered [ 5890.742902] Key type ._llcrypt registered [ 5890.743824] Key type .llcrypt registered [ 5890.776438] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 5891.141580] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5891.163398] alg: No test for adler32 (adler32-zlib) [ 5892.005322] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 5892.082911] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 5893.655095] Key type lgssc registered [ 5893.983362] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5896.347140] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5896.352262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5911.775276] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5916.895138] LustreError: 427836:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.134@tcp: failed processing log, type 4: rc = -110 [ 5943.519154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5945.257918] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5946.893845] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5948.941676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5949.098715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5949.116607] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5950.247227] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5952.914865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5953.028437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5954.238055] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 5955.997303] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5957.111040] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5964.258116] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5964.259058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5964.261852] Lustre: Skipped 1 previous similar message [ 5969.379284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5969.381168] Lustre: Skipped 2 previous similar messages [ 5969.709146] LustreError: 430116:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 5969.766262] Lustre: server umount lustre-OST0000 complete [ 5973.472128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5973.473997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5973.477310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5974.498233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.089214] LustreError: 430318:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 5977.090909] LustreError: 430318:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5977.117389] Lustre: server umount lustre-MDT0000 complete [ 5978.322475] LustreError: 429473:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761147062 with bad export cookie 13304771879307122525 [ 5978.325369] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5978.325962] LustreError: 429473:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5978.396754] LustreError: 430520:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5978.400266] LustreError: 430520:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5978.487240] Lustre: server umount lustre-MDT0001 complete [ 5982.197359] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 5983.181803] Key type lgssc unregistered [ 5983.314305] LNet: 431051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5983.316476] LNetError: 431051:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5983.323336] LNet: Removed LNI 192.168.201.134@tcp [ 5983.589264] Key type .llcrypt unregistered [ 5983.590971] Key type ._llcrypt unregistered [ 5992.237978] Key type ._llcrypt registered [ 5992.239635] Key type .llcrypt registered [ 5992.270902] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 5992.535846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5992.552752] alg: No test for adler32 (adler32-zlib) [ 5993.391671] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 5993.468519] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 5995.047118] Key type lgssc registered [ 5995.364373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5997.624649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5997.630881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5998.777422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5999.881118] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6002.561887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6002.681939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6003.865273] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6005.450305] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6006.528538] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6008.518132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6008.621192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6009.639120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6010.291270] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6011.843261] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6016.993744] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6016.995131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6016.998801] Lustre: Skipped 2 previous similar messages [ 6017.001456] Lustre: Skipped 2 previous similar messages [ 6018.390780] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.34@tcp (stopping) [ 6019.704787] LustreError: 434664:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6019.726980] Lustre: server umount lustre-MDT0000 complete [ 6020.457148] LustreError: 432404:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.34@tcp arrived at 1761147104 with bad export cookie 11175848191380706048 [ 6020.460619] LustreError: 432404:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6021.068500] LustreError: 432404:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761147105 with bad export cookie 11175848191380705194 [ 6021.072411] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6022.113220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6022.113365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6022.113555] LustreError: 432420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6022.113565] LustreError: 432420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6022.127659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6026.144272] LustreError: 432423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.144851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6026.148612] LustreError: 432423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6027.182680] LustreError: 434870:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6027.184343] LustreError: 434870:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6027.248698] Lustre: server umount lustre-OST0000 complete [ 6029.354420] LustreError: 435223:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6029.355980] LustreError: 435223:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6029.451865] Lustre: server umount lustre-MDT0001 complete [ 6032.639236] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 6033.635711] Key type lgssc unregistered [ 6033.773486] LNet: 435754:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6033.775968] LNetError: 435754:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6033.785385] LNet: Removed LNI 192.168.201.134@tcp [ 6034.078860] Key type .llcrypt unregistered [ 6034.079761] Key type ._llcrypt unregistered [ 6042.658334] Key type ._llcrypt registered [ 6042.659214] Key type .llcrypt registered [ 6042.689890] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6042.927560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6042.939664] alg: No test for adler32 (adler32-zlib) [ 6043.774709] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 6043.846928] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 6045.423090] Key type lgssc registered [ 6045.707638] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6047.906226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6047.914048] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6049.058286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6050.143506] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6052.581314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6052.693398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6053.819691] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6055.416767] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6056.489436] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6058.552015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6058.653243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6059.685389] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6060.266267] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6061.816901] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6068.052631] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.34@tcp (stopping) [ 6068.195676] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6068.196235] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6069.689963] LustreError: 439371:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6069.715089] Lustre: server umount lustre-MDT0001 complete [ 6073.312980] LustreError: 437128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6073.314322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6073.317231] LustreError: 437128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6073.320196] Lustre: Skipped 1 previous similar message [ 6073.320796] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6073.325876] Lustre: Skipped 2 previous similar messages [ 6077.358458] LustreError: 439575:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6077.360208] LustreError: 439575:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6077.418665] Lustre: server umount lustre-OST0000 complete [ 6078.432731] LustreError: 437128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6078.678086] LustreError: 439776:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6078.679840] LustreError: 439776:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6078.770590] Lustre: server umount lustre-MDT0000 complete [ 6082.839926] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 6083.907353] Key type lgssc unregistered [ 6084.043530] LNet: 440460:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6084.045529] LNetError: 440460:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6084.052397] LNet: Removed LNI 192.168.201.134@tcp [ 6084.365701] Key type .llcrypt unregistered [ 6084.366710] Key type ._llcrypt unregistered [ 6091.930502] Key type ._llcrypt registered [ 6091.931447] Key type .llcrypt registered [ 6091.971769] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 6098.755197] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6099.284059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6099.302344] alg: No test for adler32 (adler32-zlib) [ 6100.193617] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 6100.296512] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 6101.911243] Key type lgssc registered [ 6102.592289] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6107.312821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6110.772369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6114.037284] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6117.424164] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6121.854295] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6126.062431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6129.542603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6129.578315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6129.590939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6130.708100] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6130.725507] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6130.790214] Lustre: lustre-MDT0000: new disk, initializing [ 6130.828307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6130.840354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6132.633234] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6137.398772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6140.617907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6140.649618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6140.794184] Lustre: lustre-OST0000: new disk, initializing [ 6140.797550] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6140.800374] Lustre: Skipped 1 previous similar message [ 6140.832131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6143.374837] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6144.458352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6144.462938] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6144.478072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6147.527266] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6152.250776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6155.652388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6155.693343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6155.748597] Lustre: 444408:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6155.795992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6155.853836] Lustre: lustre-MDT0001: new disk, initializing [ 6155.907616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6155.924580] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6155.929595] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6157.452722] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6162.186411] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6162.331936] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6163.981545] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6164.043609] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6167.521769] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6167.523636] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6167.526818] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6167.528429] Lustre: Skipped 1 previous similar message [ 6169.946515] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.34@tcp (stopping) [ 6171.519314] LustreError: 447832:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6171.545057] Lustre: server umount lustre-MDT0001 complete [ 6177.761600] LustreError: 444415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6177.765612] LustreError: 444415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6182.881509] LustreError: 444416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.885244] LustreError: 444416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6185.439809] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.441658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.444236] Lustre: Skipped 1 previous similar message [ 6185.445544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6188.001624] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6188.002844] LustreError: 444420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6193.121872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6193.122879] LustreError: 444420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6198.239191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6198.242992] LustreError: 444420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6198.272151] LustreError: 448106:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6198.275873] LustreError: 448106:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6198.357450] Lustre: server umount lustre-OST0000 complete [ 6206.282918] LustreError: 448308:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6206.286715] LustreError: 448308:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6206.393468] Lustre: server umount lustre-MDT0000 complete [ 6212.560311] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 6213.930597] Key type lgssc unregistered [ 6214.098522] LNet: 448992:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6214.101326] LNetError: 448992:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6214.109558] LNet: Removed LNI 192.168.201.134@tcp [ 6214.584339] Key type .llcrypt unregistered [ 6214.586387] Key type ._llcrypt unregistered [ 6231.681355] Key type ._llcrypt registered [ 6231.682733] Key type .llcrypt registered [ 6231.719856] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 6231.822320] Key type .llcrypt unregistered [ 6231.824100] Key type ._llcrypt unregistered [ 6240.567131] Key type ._llcrypt registered [ 6240.568786] Key type .llcrypt registered [ 6240.622757] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6240.945603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6240.956161] alg: No test for adler32 (adler32-zlib) [ 6241.810261] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 6241.896894] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 6243.495160] Key type lgssc registered [ 6243.990839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6247.796092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6250.757965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6253.555844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6256.757459] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6256.788390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6256.794597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6257.883145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6257.900265] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6257.949877] Lustre: lustre-MDT0000: new disk, initializing [ 6257.981141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6257.990635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6259.811405] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6265.375412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6265.418128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6265.473534] Lustre: 452311:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6265.527631] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6265.531359] Lustre: Skipped 1 previous similar message [ 6265.581636] Lustre: lustre-MDT0001: new disk, initializing [ 6265.645501] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6265.670254] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6265.676486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6267.547645] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6271.568950] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6273.425519] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6276.802069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6276.845271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6277.008723] Lustre: lustre-OST0000: new disk, initializing [ 6277.012335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6277.063748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6279.434778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6279.438968] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6279.488528] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6279.572054] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6283.727042] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6289.890856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.893536] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6289.899073] Lustre: Skipped 2 previous similar messages [ 6289.905570] Lustre: Skipped 1 previous similar message [ 6292.814193] LustreError: 454754:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6292.852783] Lustre: server umount lustre-MDT0000 complete [ 6294.644390] LustreError: 452303:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761147378 with bad export cookie 17104570176979911941 [ 6294.651443] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6294.653037] LustreError: 452303:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6294.729893] LustreError: 454958:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6294.732981] LustreError: 454958:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6294.817425] Lustre: server umount lustre-MDT0001 complete [ 6306.677798] LustreError: 455159:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6306.681898] LustreError: 455159:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6306.788571] Lustre: server umount lustre-OST0000 complete [ 6311.688490] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:36:35 (1761147395) [ 6312.213776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6315.609239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6315.813520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6317.542715] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6321.121425] LustreError: 455986:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6321.131460] LustreError: 455986:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6321.634032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6321.797736] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6323.483579] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6326.084912] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6327.854132] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6331.208909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6331.250580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6331.340499] Lustre: Found index 0 for lustre-OST0000, updating log [ 6331.413724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6333.712265] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6336.193682] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6341.669679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6351.843046] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.843676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6351.848324] Lustre: Skipped 1 previous similar message [ 6351.848602] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6351.850237] Lustre: Skipped 2 previous similar messages [ 6352.865902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6352.868077] Lustre: Skipped 1 previous similar message [ 6354.157070] LustreError: 458265:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6354.158748] LustreError: 458265:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6354.220521] Lustre: server umount lustre-OST0000 complete [ 6355.486362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6357.984807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6357.984874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6357.990580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6357.992019] Lustre: Skipped 1 previous similar message [ 6362.169634] LustreError: 458520:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6362.171365] LustreError: 458520:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6362.199465] Lustre: server umount lustre-MDT0000 complete [ 6363.611135] LustreError: 455966:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761147447 with bad export cookie 17104570176979913327 [ 6363.617741] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6363.618225] LustreError: 455966:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6363.768124] Lustre: server umount lustre-MDT0001 complete [ 6366.278967] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:37:30 (1761147450) [ 6366.590475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6366.932624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6367.369683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6367.836536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6368.315436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6368.775842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.223144] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6369.636173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.104899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6370.564941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.041406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.494115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6371.941371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.459888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6372.981855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6373.527372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6374.051205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6374.550813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6375.047470] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6375.581427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.117366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6376.639097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.155686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6377.703382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6378.217853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6378.675060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6379.190212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6379.665434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.101904] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6380.611964] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6381.100738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6381.602098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6382.059340] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6382.386052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6382.880985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6383.260236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6383.558659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6383.907790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.379275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6384.810454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.286081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6385.764313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6386.266170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6386.754351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.268257] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6387.754107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6388.202824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6388.502448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6388.802457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6389.110272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6389.427454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6392.734633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6392.970492] LustreError: 462104:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6393.007912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6394.239384] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6397.584275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6399.261186] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6401.899650] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6403.743273] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6406.227164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6406.282763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6406.381659] Lustre: Found index 0 for lustre-OST0000, updating log [ 6408.678600] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6410.366626] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6414.818095] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6414.818417] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6414.821520] Lustre: Skipped 1 previous similar message [ 6414.825254] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6414.826938] Lustre: Skipped 2 previous similar messages [ 6417.326317] LustreError: 464412:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6417.330047] LustreError: 464412:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6417.432774] Lustre: server umount lustre-OST0000 complete [ 6420.128787] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6420.132020] Lustre: Skipped 1 previous similar message [ 6423.201417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6423.204873] Lustre: Skipped 7 previous similar messages [ 6425.168478] Lustre: server umount lustre-MDT0000 complete [ 6425.248117] LustreError: 463313:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6425.252234] LustreError: 463313:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6426.406913] LustreError: 463782:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761147510 with bad export cookie 17104570176979914405 [ 6426.412618] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6426.413587] LustreError: 463782:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6426.604125] Lustre: server umount lustre-MDT0001 complete [ 6427.969731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6434.323906] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 6437.496655] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6441.225904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6444.503146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6447.520801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6450.726276] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6453.756908] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 11:38:57 (1761147537) [ 6469.935690] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 6471.423885] Key type lgssc unregistered [ 6471.584897] LNet: 468404:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6471.589930] LNetError: 468404:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6596.519347] LNet: Removed LNI 192.168.201.134@tcp [ 6596.937337] Key type .llcrypt unregistered [ 6596.939893] Key type ._llcrypt unregistered [ 6604.425389] Key type ._llcrypt registered [ 6604.426813] Key type .llcrypt registered [ 6604.482639] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6604.949875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6604.992541] alg: No test for adler32 (adler32-zlib) [ 6605.830588] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 6605.903088] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 6607.479136] Key type lgssc registered [ 6607.981385] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6610.114101] LNet: Added LNI 192.168.201.134@tcp100 [8/256/0/180] [ 6610.563278] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 6617.147140] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 6618.399720] Key type lgssc unregistered [ 6618.592908] LNet: 470273:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6618.598180] LNetError: 470273:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6618.603368] LNet: Removed LNI 192.168.201.134@tcp [ 6618.614403] LNet: Removed LNI 192.168.201.134@tcp100 [ 6619.008535] Key type .llcrypt unregistered [ 6619.011288] Key type ._llcrypt unregistered [ 6629.365124] Key type ._llcrypt registered [ 6629.366592] Key type .llcrypt registered [ 6629.424330] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 6638.112946] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6638.500748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6638.538481] alg: No test for adler32 (adler32-zlib) [ 6639.401017] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 6639.489637] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 6641.063172] Key type lgssc registered [ 6641.482403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6645.787569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6649.010509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6652.042957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6655.255501] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6658.490379] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 11:42:22 (1761147742) [ 6666.809455] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 6668.158637] Key type lgssc unregistered [ 6668.321493] LNet: 474285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6668.327600] LNetError: 474285:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6668.337573] LNet: Removed LNI 192.168.201.134@tcp [ 6668.788285] Key type .llcrypt unregistered [ 6668.789636] Key type ._llcrypt unregistered [ 6676.911190] Key type ._llcrypt registered [ 6676.913322] Key type .llcrypt registered [ 6676.971749] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6677.377459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6677.407421] alg: No test for adler32 (adler32-zlib) [ 6678.254831] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 6678.350718] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 6679.935222] Key type lgssc registered [ 6680.380585] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6685.020313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6685.062792] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6685.074316] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6686.210740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6686.231193] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6686.272988] Lustre: lustre-MDT0000: new disk, initializing [ 6686.312650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6686.321300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6688.359240] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6694.397583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6694.432652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6694.472422] Lustre: 475455:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6694.515746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6694.519648] Lustre: Skipped 1 previous similar message [ 6694.573591] Lustre: lustre-MDT0001: new disk, initializing [ 6694.638071] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6694.664051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6694.671204] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6696.513892] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6700.143829] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6701.404933] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6703.845902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6703.869048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6704.033243] Lustre: lustre-OST0000: new disk, initializing [ 6704.037504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6704.090007] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6705.155502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6705.158402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6705.196493] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6705.979064] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6709.155910] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6710.230470] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing lnet_if_list [ 6711.939426] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 6711.955199] LNet: Added LNI 192.168.201.134@tcp42 [8/256/0/180] [ 6720.484321] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6720.487324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6720.489238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6720.491527] Lustre: Skipped 1 previous similar message [ 6724.143280] LustreError: 478841:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6724.238473] Lustre: server umount lustre-OST0000 complete [ 6730.721824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.721937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6730.723381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6730.723386] Lustre: Skipped 1 previous similar message [ 6730.724957] Lustre: Skipped 1 previous similar message [ 6730.726925] LustreError: Skipped 1 previous similar message [ 6731.709330] LustreError: 479042:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6731.711265] LustreError: 479042:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6731.740685] Lustre: server umount lustre-MDT0000 complete [ 6733.069795] LustreError: 475447:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761147817 with bad export cookie 13331616995526161507 [ 6733.075813] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6733.177580] LustreError: 479243:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6733.182240] LustreError: 479243:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6733.293780] Lustre: server umount lustre-MDT0001 complete [ 6736.577338] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 6737.721931] Key type lgssc unregistered [ 6737.887904] LNet: 479781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6737.890124] LNetError: 479781:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6737.891906] LNet: Removed LNI 192.168.201.134@tcp [ 6737.896448] LNet: Removed LNI 192.168.201.134@tcp42 [ 6738.197520] Key type .llcrypt unregistered [ 6738.198518] Key type ._llcrypt unregistered [ 6747.113528] Key type ._llcrypt registered [ 6747.114428] Key type .llcrypt registered [ 6747.154635] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6747.677851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6747.693857] alg: No test for adler32 (adler32-zlib) [ 6748.549330] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 6748.644374] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 6750.223232] Key type lgssc registered [ 6750.815518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6753.998864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6754.011693] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6755.223783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6756.834059] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6759.634059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6759.836142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6761.256731] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6763.009168] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6764.468229] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6767.383846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6767.489374] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6769.092343] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6771.505827] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6786.396173] loop0: detected capacity change from 0 to 204800000 [ 6786.498765] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6786.502291] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6786.505715] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6786.509372] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6786.531726] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6786.548708] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6786.561179] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6786.582711] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6786.607668] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6786.639543] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6788.715652] loop0: detected capacity change from 0 to 204800000 [ 6788.775525] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6788.834028] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6791.058294] loop0: detected capacity change from 0 to 204800000 [ 6793.235642] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6793.257553] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6793.296988] Lustre: fs15246-OST0000: new disk, initializing [ 6793.298564] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 6793.313334] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6795.124341] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6799.251337] loop1: detected capacity change from 0 to 204800000 [ 6801.593650] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6801.621472] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6801.669433] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 6801.707403] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 6801.753378] Lustre: fs15246-MDT0000: new disk, initializing [ 6801.807455] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6801.819275] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6801.855534] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6802.911971] Lustre: 485400:0:(client.c:1627:after_reply()) @@@ resending request on EINPROGRESS req@00000000f9f7dff9 x1846697349119616/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1761147903 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 6803.476592] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6812.002329] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6812.007854] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6812.022917] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6818.788137] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6818.791748] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 6819.894940] LustreError: 486524:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 6819.918458] Lustre: server umount fs15246-MDT0000 complete [ 6823.910784] LustreError: 483164:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6823.919651] LustreError: 483164:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6824.049255] LustreError: 487026:0:(obd_class.h:479:obd_check_dev()) Device 32 not setup [ 6824.053814] LustreError: 487026:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6824.078455] Lustre: server umount fs15246-OST0000 complete [ 6829.025863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6829.026926] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6829.028850] Lustre: Skipped 1 previous similar message [ 6829.031830] Lustre: Skipped 1 previous similar message [ 6834.146370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6834.148198] Lustre: Skipped 1 previous similar message [ 6839.266431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6839.268200] Lustre: Skipped 1 previous similar message [ 6843.359133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6843.377865] LustreError: 487529:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6843.379863] LustreError: 487529:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6843.432480] Lustre: server umount lustre-OST0000 complete [ 6846.944374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6846.948767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6846.956569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6849.506326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6850.700073] LustreError: 487732:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6850.703768] LustreError: 487732:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6850.737761] Lustre: server umount lustre-MDT0000 complete [ 6852.548806] LustreError: 480983:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761147936 with bad export cookie 9603669488077884020 [ 6852.554858] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6852.554964] LustreError: 480983:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6852.745114] Lustre: server umount lustre-MDT0001 complete [ 6857.942382] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 6859.447405] Key type lgssc unregistered [ 6859.631548] LNet: 488464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6859.636331] LNetError: 488464:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6859.649531] LNet: Removed LNI 192.168.201.134@tcp [ 6860.018960] Key type .llcrypt unregistered [ 6860.021487] Key type ._llcrypt unregistered [ 6870.137058] Key type ._llcrypt registered [ 6870.138465] Key type .llcrypt registered [ 6870.188539] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6870.569278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6870.575723] alg: No test for adler32 (adler32-zlib) [ 6871.462729] Lustre: Lustre: Build Version: 2.16.59_1_gf12db73 [ 6871.558306] LNet: Added LNI 192.168.201.134@tcp [8/256/0/180] [ 6873.143199] Key type lgssc registered [ 6873.777402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6877.889582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6880.409972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6882.302158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6884.078976] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6892.254575] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 6897.242033] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6902.401381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6905.842620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6909.059500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6912.406766] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6915.905369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6915.943034] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6915.954608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6917.088109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6917.114864] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6917.169702] Lustre: lustre-MDT0000: new disk, initializing [ 6917.210462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6917.224590] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6919.049507] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6924.836724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6924.866149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6924.922718] Lustre: 492873:0:(mgs_llog.c:1346:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6924.967305] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6924.970535] Lustre: Skipped 1 previous similar message [ 6925.027620] Lustre: lustre-MDT0001: new disk, initializing [ 6925.083332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6925.105439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6925.112826] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6926.873044] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6931.043818] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6932.925207] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6936.352649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6936.390790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6936.548423] Lustre: lustre-OST0000: new disk, initializing [ 6936.552714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6936.602479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6939.210326] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_default_debug -1 all [ 6943.308996] Lustre: DEBUG MARKER: oleg134-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6945.292516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6945.297493] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6945.308065] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6945.345848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6945.405093] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6947.205232] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6947.331386] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6950.370331] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6950.371417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6950.374942] LustreError: Skipped 1 previous similar message [ 6950.376197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6950.383390] Lustre: Skipped 1 previous similar message [ 6954.354154] LustreError: 495812:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6954.453601] Lustre: server umount lustre-OST0000 complete [ 6960.610353] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6960.611923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6960.617546] Lustre: Skipped 1 previous similar message [ 6960.627349] Lustre: Skipped 2 previous similar messages [ 6962.443461] LustreError: 496013:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6962.448914] LustreError: 496013:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6962.486685] Lustre: server umount lustre-MDT0000 complete [ 6964.346770] LustreError: 492865:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761148048 with bad export cookie 4261596442738855227 [ 6964.352888] LustreError: 492865:0:(ldlm_lockd.c:2580:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6964.353382] LustreError: MGC192.168.201.134@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6964.436634] LustreError: 496216:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6964.439742] LustreError: 496216:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6964.540204] Lustre: server umount lustre-MDT0001 complete [ 6976.129397] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing set_hostid [ 6980.838955] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing load_modules_local [ 6986.366636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6990.489869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6993.874658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6997.440625] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6998.398662] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6855 sec ========== 11:48:01 (1761148081) [ 6999.278569] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:48:02 (1761148082) === [ 7000.995633] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:48:04 (1761148084) === [ 7020.255767] Lustre: DEBUG MARKER: oleg134-server.virtnet: executing unload_modules_local [ 7021.567377] Key type lgssc unregistered [ 7021.726532] LNet: 501067:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7021.730441] LNetError: 501067:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7021.743518] LNet: Removed LNI 192.168.201.134@tcp [ 7022.153153] Key type .llcrypt unregistered [ 7022.154888] Key type ._llcrypt unregistered