[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 538133772 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f5b30-0x000f5b3f] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F5950 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE1BB7 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE1A53 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 001A13 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE1AC7 000090 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE1B57 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE1B8F 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe1a53-0xbffe1ac6] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe1a52] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe1ac7-0xbffe1b56] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe1b57-0xbffe1b8e] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe1b8f-0xbffe1bb6] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002386] x2apic enabled [ 0.003000] Switched APIC routing to physical x2apic. [ 0.003014] kvm-guest: setup PV IPIs [ 0.006495] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007023] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009127] LSM: Security Framework initializing [ 0.010047] Yama: becoming mindful. [ 0.011036] SELinux: Initializing. [ 0.013070] *** VALIDATE selinux *** [ 0.022516] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027496] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029114] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030129] *** VALIDATE tmpfs *** [ 0.032215] *** VALIDATE proc *** [ 0.033283] *** VALIDATE cgroup *** [ 0.034020] *** VALIDATE cgroup2 *** [ 0.035000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035168] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037045] Spectre V2 : User space: Vulnerable [ 0.038013] Speculative Store Bypass: Vulnerable [ 0.042579] debug: unmapping init [mem 0xffffffffaba59000-0xffffffffaba60fff] [ 0.045373] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047215] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048047] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050010] ... generic registers: 4 [ 0.051010] ... value mask: 0000ffffffffffff [ 0.052019] ... max period: 00007fffffffffff [ 0.053029] ... fixed-purpose events: 3 [ 0.054029] ... event mask: 000000070000000f [ 0.055521] rcu: Hierarchical SRCU implementation. [ 0.057537] smp: Bringing up secondary CPUs ... [ 0.058616] x86: Booting SMP configuration: [ 0.059040] .... node #0, CPUs: #1 #2 #3 [ 0.080017] smp: Brought up 1 node, 4 CPUs [ 0.082014] smpboot: Max logical packages: 1 [ 0.083013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.123014] node 0 deferred pages initialised in 34ms [ 0.129163] devtmpfs: initialized [ 0.132389] x86/mm: Memory block size: 128MB [ 0.136235] gcov: version magic: 0x41383552 [ 0.139378] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.141198] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147339] pinctrl core: initialized pinctrl subsystem [ 0.151271] [ 0.151949] ************************************************************* [ 0.156019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.159018] ** ** [ 0.161016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.163018] ** ** [ 0.166020] ** This means that this kernel is built to expose internal ** [ 0.169019] ** IOMMU data structures, which may compromise security on ** [ 0.171018] ** your system. ** [ 0.173023] ** ** [ 0.176023] ** If you see this message and you are not debugging the ** [ 0.177000] ** kernel, report this immediately to your vendor! ** [ 0.178014] ** ** [ 0.180011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.182012] ************************************************************* [ 0.185995] NET: Registered protocol family 16 [ 0.187666] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.189079] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.192090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.197445] cpuidle: using governor menu [ 0.199169] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202782] PCI: Using configuration type 1 for base access [ 0.205155] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.217119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.220033] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.225059] cryptd: max_cpu_qlen set to 1000 [ 0.227352] ACPI: Added _OSI(Module Device) [ 0.229014] ACPI: Added _OSI(Processor Device) [ 0.230024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.232011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.235802] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.242009] ACPI: Interpreter enabled [ 0.243065] ACPI: PM: (supports S0 S3 S4 S5) [ 0.244015] ACPI: Using IOAPIC for interrupt routing [ 0.247181] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.248509] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.256754] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.258050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.261024] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.264095] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269311] acpiphp: Slot [2] registered [ 0.271162] acpiphp: Slot [3] registered [ 0.272095] acpiphp: Slot [4] registered [ 0.274107] acpiphp: Slot [5] registered [ 0.275176] acpiphp: Slot [6] registered [ 0.276116] acpiphp: Slot [7] registered [ 0.278146] acpiphp: Slot [8] registered [ 0.279133] acpiphp: Slot [9] registered [ 0.282198] acpiphp: Slot [10] registered [ 0.284960] acpiphp: Slot [11] registered [ 0.286110] acpiphp: Slot [12] registered [ 0.287119] acpiphp: Slot [13] registered [ 0.288155] acpiphp: Slot [14] registered [ 0.290121] acpiphp: Slot [15] registered [ 0.293170] acpiphp: Slot [16] registered [ 0.296128] acpiphp: Slot [17] registered [ 0.298081] acpiphp: Slot [18] registered [ 0.299073] acpiphp: Slot [19] registered [ 0.300096] acpiphp: Slot [20] registered [ 0.301000] acpiphp: Slot [21] registered [ 0.301000] acpiphp: Slot [22] registered [ 0.302118] acpiphp: Slot [23] registered [ 0.303077] acpiphp: Slot [24] registered [ 0.304072] acpiphp: Slot [25] registered [ 0.305084] acpiphp: Slot [26] registered [ 0.306000] acpiphp: Slot [27] registered [ 0.307031] acpiphp: Slot [28] registered [ 0.308631] acpiphp: Slot [29] registered [ 0.310101] acpiphp: Slot [30] registered [ 0.311084] acpiphp: Slot [31] registered [ 0.312053] PCI host bridge to bus 0000:00 [ 0.313018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.315019] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.318642] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.320034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.322023] pci_bus 0000:00: root bus resource [mem 0x180000000-0x1ffffffff window] [ 0.323128] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.324000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.326000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.329846] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.338665] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.343599] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.346317] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.348024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.352025] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.353000] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.355674] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.359050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.365700] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.371015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.393023] pci 0000:00:02.0: reg 0x20: [mem 0xfebe4000-0xfebe7fff 64bit pref] [ 0.397977] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.404283] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.412019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.420020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.441026] pci 0000:00:05.0: reg 0x20: [mem 0xfebe8000-0xfebebfff 64bit pref] [ 0.451940] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.458017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.463020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.478036] pci 0000:00:06.0: reg 0x20: [mem 0xfebec000-0xfebeffff 64bit pref] [ 0.491886] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.500016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.502022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.517029] pci 0000:00:07.0: reg 0x20: [mem 0xfebf0000-0xfebf3fff 64bit pref] [ 0.526502] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.532022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.538022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.549016] pci 0000:00:08.0: reg 0x20: [mem 0xfebf4000-0xfebf7fff 64bit pref] [ 0.555076] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.559017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.567048] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.583000] pci 0000:00:09.0: reg 0x20: [mem 0xfebf8000-0xfebfbfff 64bit pref] [ 0.591449] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.608019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.618017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.636016] pci 0000:00:0a.0: reg 0x20: [mem 0xfebfc000-0xfebfffff 64bit pref] [ 0.653650] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.655311] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.661358] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.665333] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.670227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.676109] iommu: Default domain type: Passthrough [ 0.677000] SCSI subsystem initialized [ 0.677000] ACPI: bus type USB registered [ 0.678099] usbcore: registered new interface driver usbfs [ 0.683225] usbcore: registered new interface driver hub [ 0.688305] usbcore: registered new device driver usb [ 0.689233] pps_core: LinuxPPS API ver. 1 registered [ 0.691030] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.696154] PTP clock support registered [ 0.701047] EDAC MC: Ver: 3.0.0 [ 0.702094] PCI: Using ACPI for IRQ routing [ 0.706792] NetLabel: Initializing [ 0.709015] NetLabel: domain hash size = 128 [ 0.711011] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.714141] NetLabel: unlabeled traffic allowed by default [ 0.717454] vgaarb: loaded [ 0.719341] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.724022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.737711] clocksource: Switched to clocksource kvm-clock [ 0.903718] VFS: Disk quotas dquot_6.6.0 [ 0.905396] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.908985] *** VALIDATE ramfs *** [ 0.910336] *** VALIDATE hugetlbfs *** [ 0.911886] pnp: PnP ACPI init [ 0.914829] pnp: PnP ACPI: found 6 devices [ 0.932199] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.935399] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.937510] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.939717] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.941893] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.944450] pci_bus 0000:00: resource 8 [mem 0x180000000-0x1ffffffff window] [ 0.947146] NET: Registered protocol family 2 [ 0.949409] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.954115] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.957224] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.962054] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.964986] TCP: Hash tables configured (established 65536 bind 65536) [ 0.967546] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.970203] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.972627] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.975309] NET: Registered protocol family 1 [ 0.978788] RPC: Registered named UNIX socket transport module. [ 0.980689] RPC: Registered udp transport module. [ 0.982395] RPC: Registered tcp transport module. [ 0.983833] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.985923] NET: Registered protocol family 44 [ 0.988203] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.990102] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.991912] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.995223] PCI: CLS 0 bytes, default 64 [ 0.997154] Unpacking initramfs... [ 2.662685] debug: unmapping init [mem 0xffff8ffb3cc54000-0xffff8ffb3ffbffff] [ 2.667572] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.669645] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.672377] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.203830] Initialise system trusted keyrings [ 3.205268] Key type blacklist registered [ 3.207209] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.216541] zbud: loaded [ 3.219196] *** VALIDATE nfs *** [ 3.220418] *** VALIDATE nfs4 *** [ 3.221600] pstore: using deflate compression [ 3.227718] Platform Keyring initialized [ 3.343525] NET: Registered protocol family 38 [ 3.345349] Key type asymmetric registered [ 3.346890] Asymmetric key parser 'x509' registered [ 3.348979] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.353161] io scheduler mq-deadline registered [ 3.354875] io scheduler kyber registered [ 3.356504] io scheduler bfq registered [ 3.358308] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.366672] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.370470] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.373409] ACPI: Power Button [PWRF] [ 3.481347] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.585395] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.797602] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.892406] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.078331] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.108828] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.139128] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.147647] Non-volatile memory driver v1.3 [ 4.150241] Linux agpgart interface v0.103 [ 4.218774] virtio_blk virtio1: [vda] 132768 512-byte logical blocks (68.0 MB/64.8 MiB) [ 4.222364] vda: detected capacity change from 0 to 67977216 [ 4.246778] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.252816] vdb: detected capacity change from 0 to 1073741824 [ 4.274668] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.277715] vdc: detected capacity change from 0 to 2621440000 [ 4.296846] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.299886] vdd: detected capacity change from 0 to 2621440000 [ 4.316260] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.318993] vde: detected capacity change from 0 to 4294967296 [ 4.334796] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.337878] vdf: detected capacity change from 0 to 4294967296 [ 4.343381] libphy: Fixed MDIO Bus: probed [ 4.351488] usbcore: registered new interface driver usbserial_generic [ 4.353899] usbserial: USB Serial support registered for generic [ 4.356276] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.360665] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.363295] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.366708] mousedev: PS/2 mouse device common for all mice [ 4.372641] rtc_cmos 00:05: RTC can wake from S4 [ 4.375931] rtc_cmos 00:05: registered as rtc0 [ 4.379192] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.384646] intel_pstate: CPU model not supported [ 4.392195] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.405104] hid: raw HID events driver (C) Jiri Kosina [ 4.407503] usbcore: registered new interface driver usbhid [ 4.409352] usbhid: USB HID core driver [ 4.410825] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.412254] drop_monitor: Initializing network drop monitor service [ 4.412374] Initializing XFRM netlink socket [ 4.412737] NET: Registered protocol family 10 [ 4.415834] Segment Routing with IPv6 [ 4.429350] NET: Registered protocol family 17 [ 4.432887] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.436266] mpls_gso: MPLS GSO support [ 4.444946] RAS: Correctable Errors collector initialized. [ 4.448629] AVX version of gcm_enc/dec engaged. [ 4.451167] AES CTR mode by8 optimization enabled [ 4.584491] sched_clock: Marking stable (4584007470, 0)->(5748156074, -1164148604) [ 4.588163] registered taskstats version 1 [ 4.591873] Loading compiled-in X.509 certificates [ 4.599516] zswap: loaded using pool lzo/zbud [ 4.642666] Key type big_key registered [ 4.662031] Key type encrypted registered [ 4.663282] ima: No TPM chip found, activating TPM-bypass! [ 4.664832] ima: Allocated hash algorithm: sha1 [ 4.669936] ima: No architecture policies found [ 4.671729] evm: Initialising EVM extended attributes: [ 4.673285] evm: security.selinux [ 4.674352] evm: security.ima [ 4.675288] evm: security.capability [ 4.676369] evm: HMAC attrs: 0x1 [ 4.679190] rtc_cmos 00:05: setting system clock to 2025-08-01 00:46:16 UTC (1754009176) [ 4.690185] debug: unmapping init [mem 0xffffffffaca03000-0xffffffffacbfffff] [ 4.695953] debug: unmapping init [mem 0xffffffffab782000-0xffffffffaba58fff] [ 4.702193] Write protecting the kernel read-only data: 28672k [ 4.706804] debug: unmapping init [mem 0xffffffffa9e03000-0xffffffffa9ffffff] [ 4.709780] debug: unmapping init [mem 0xffffffffaa714000-0xffffffffaa7fffff] [ 4.744868] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.753152] systemd[1]: Detected virtualization kvm. [ 4.754962] systemd[1]: Detected architecture x86-64. [ 4.756706] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.791180] systemd[1]: No hostname configured. [ 4.793230] systemd[1]: Set hostname to . [ 4.795302] random: systemd: uninitialized urandom read (16 bytes read) [ 4.797577] systemd[1]: Initializing machine ID from random generator. [ 5.001214] random: systemd: uninitialized urandom read (16 bytes read) [ 5.003431] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 5.007555] random: systemd: uninitialized urandom read (16 bytes read) [ 5.009984] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 5.021645] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Slices. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.058277] device-mapper: uevent: version 1.0.3 [ 6.060662] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 7.234985] virtio_net virtio0 ens2: renamed from eth0 [ 7.259770] random: fast init done [ 8.390653] scsi host0: ata_piix [ 8.440278] scsi host1: ata_piix [ 8.441618] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.444023] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 12.192890] random: crng init done [ 12.194170] random: 7 urandom warning(s) missed due to ratelimiting [ 15.056446] dracut-initqueue[591]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.948961] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.845855] printk: systemd: 26 output lines suppressed due to ratelimiting [ 18.248121] SELinux: Disabled at runtime. [ 18.333424] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 18.348440] systemd[1]: Detected virtualization kvm. [ 18.351410] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 19.289841] systemd[1]: initrd-switch-root.service: Succeeded. [ 19.292530] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 19.296513] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 19.307066] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 19.313373] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 19.322618] systemd[1]: Starting Journal Service... Starting Journal Service... [ 19.347037] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ 19.504621] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Initrd File Systems. Starting Create list of required st…ce nodes for the current kernel... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Created slice User and Session Slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Initrd Root File System. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 20.459307] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 21.379685] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 21.471656] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 22.618878] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 22.660063] EDAC sbridge: Ver: 1.1.2 [ 25.202585] Key type dns_resolver registered [ 25.577340] NFS: Registering the id_resolver key type [ 25.579471] Key type id_resolver registered [ 25.582094] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started dnf makecache --timer. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg324-server login: [ 53.683403] libcfs: loading out-of-tree module taints kernel. [ 53.694505] Key type ._llcrypt registered [ 53.696054] Key type .llcrypt registered [ 53.747616] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 61.950777] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 62.632579] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.641046] alg: No test for adler32 (adler32-zlib) [ 63.687087] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 64.045189] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 64.048425] LNet: Accept secure, port 988 [ 65.688565] Key type lgssc registered [ 66.270286] Lustre: Echo OBD driver; http://www.lustre.org/ [ 73.951646] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 75.038971] LDISKFS-fs (vdc): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 79.020544] LDISKFS-fs (vdd): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 81.999434] LDISKFS-fs (vde): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 85.285426] LDISKFS-fs (vdf): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 90.970065] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 96.050602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 96.075512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 96.096664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 97.214420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 97.230723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 97.275645] Lustre: lustre-MDT0000: new disk, initializing [ 97.318951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 97.330461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 98.939877] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 104.971670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 105.006353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 105.036597] Lustre: 6450:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 105.052292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 105.054667] Lustre: Skipped 1 previous similar message [ 105.099393] Lustre: lustre-MDT0001: new disk, initializing [ 105.123687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 105.135831] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 105.140601] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 106.632975] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 109.226605] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 113.247124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 113.279805] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 113.385473] Lustre: lustre-OST0000: new disk, initializing [ 113.388490] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 113.415481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 115.723246] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 121.492318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 121.515230] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 121.546908] Lustre: lustre-OST0001: new disk, initializing [ 121.548695] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 121.564311] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 121.873065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 121.876042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 121.904630] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 121.906345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 123.494070] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 129.916455] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 136.892666] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 144.075957] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing check_logdir /tmp/testlogs/ [ 145.585993] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing yml_node [ 147.307213] Lustre: DEBUG MARKER: Client: 2.16.57.19 [ 148.234891] Lustre: DEBUG MARKER: MDS: 2.16.57.19 [ 149.257361] Lustre: DEBUG MARKER: OSS: 2.16.57.19 [ 149.894468] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 31 20:48:41 EDT 2025 [ 156.299151] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 156.833527] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 172.000585] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 172.003067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 172.008296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 173.025183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 173.025724] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 173.029331] Lustre: Skipped 2 previous similar messages [ 173.034422] Lustre: Skipped 2 previous similar messages [ 174.720035] LustreError: 12111:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 174.760654] Lustre: server umount lustre-MDT0000 complete [ 176.234743] LustreError: 10190:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009348 with bad export cookie 11857215646088106379 [ 176.235814] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 176.239064] LustreError: 10190:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 178.144622] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 178.145139] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 178.150138] Lustre: Skipped 1 previous similar message [ 178.152342] Lustre: Skipped 1 previous similar message [ 181.152732] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 181.155134] Lustre: Skipped 2 previous similar messages [ 182.336717] LustreError: 12313:0:(obd_class.h:479:obd_check_dev()) Device 26 not setup [ 182.339435] LustreError: 12313:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 182.440417] Lustre: server umount lustre-MDT0001 complete [ 184.007195] LustreError: 12514:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 184.009593] LustreError: 12514:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 184.031556] Lustre: server umount lustre-OST0000 complete [ 185.570766] Lustre: server umount lustre-OST0001 complete [ 187.815957] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 191.491351] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 195.572982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 198.108214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 200.552888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 203.152967] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.199416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 206.230155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 206.330276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 206.349055] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 206.389204] Lustre: lustre-MDT0000: new disk, initializing [ 206.431049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 206.439619] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 206.442937] Lustre: Skipped 1 previous similar message [ 208.068557] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 213.233360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 213.266665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 213.312766] Lustre: 14877:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 213.454764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 213.460367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 213.466396] Lustre: Skipped 1 previous similar message [ 215.064798] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 218.703141] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 220.435646] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 223.452416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 223.490656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 223.606752] Lustre: lustre-OST0000: new disk, initializing [ 223.608808] Lustre: Skipped 1 previous similar message [ 223.611571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 223.613944] Lustre: Skipped 2 previous similar messages [ 223.651242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 223.654969] Lustre: Skipped 1 previous similar message [ 225.287678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 225.291237] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 225.340578] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 225.991900] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 229.798078] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 231.588267] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 231.691854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 233.656436] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 233.780460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 235.492320] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 235.493170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 235.496439] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 235.497232] LustreError: Skipped 1 previous similar message [ 235.501729] Lustre: Skipped 1 previous similar message [ 240.749154] LustreError: 17782:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 240.756095] LustreError: 17782:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 240.838471] Lustre: server umount lustre-OST0000 complete [ 244.193166] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 244.196687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 244.202432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 244.205238] Lustre: Skipped 3 previous similar messages [ 248.658256] LustreError: 17983:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 248.661015] LustreError: 17983:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 248.715032] Lustre: server umount lustre-MDT0000 complete [ 250.559980] LustreError: 17985:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009422 with bad export cookie 11857215646088107814 [ 250.563373] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 250.563787] LustreError: 17985:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 250.740196] Lustre: server umount lustre-MDT0001 complete [ 252.711602] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 20:50:23 (1754009423) [ 253.844602] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 254.663610] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 20:50:25 (1754009425) [ 258.070622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 258.289684] LustreError: 18855:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 258.299254] LustreError: 18855:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 258.340349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 260.241269] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 263.649830] LustreError: 18855:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 264.500862] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 266.464780] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 268.920470] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 270.514860] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 273.388375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 275.822962] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 278.052536] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 294.372295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.372316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 294.377373] Lustre: Skipped 1 previous similar message [ 294.382178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 294.382185] Lustre: Skipped 2 previous similar messages [ 297.452613] LustreError: 21103:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 297.455748] LustreError: 21103:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 297.524743] Lustre: server umount lustre-OST0000 complete [ 313.312146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 313.370593] Lustre: server umount lustre-MDT0000 complete [ 314.795653] LustreError: 21306:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009486 with bad export cookie 11857215646088108696 [ 314.798346] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 314.800528] LustreError: 21306:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 314.868214] LustreError: 21508:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 314.870664] LustreError: 21508:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 314.948757] Lustre: server umount lustre-MDT0001 complete [ 319.980264] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 321.142205] Key type lgssc unregistered [ 321.293412] LNet: 22034:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 322.342392] LNet: Removed LNI 192.168.203.124@tcp [ 322.724150] Key type .llcrypt unregistered [ 322.725484] Key type ._llcrypt unregistered [ 332.243524] Key type ._llcrypt registered [ 332.245167] Key type .llcrypt registered [ 332.289213] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 332.773852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 332.788384] alg: No test for adler32 (adler32-zlib) [ 333.652046] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 333.747638] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 333.749607] LNet: Accept secure, port 988 [ 335.336153] Key type lgssc registered [ 335.722772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 338.807352] loop: module loaded [ 338.829488] loop0: detected capacity change from 0 to 204800000 [ 338.953675] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 338.958282] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.086903] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.091660] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.119472] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.147370] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.164982] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 339.197506] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 341.387584] loop0: detected capacity change from 0 to 204800000 [ 341.459156] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.464561] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.469979] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 341.530586] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 343.465941] loop0: detected capacity change from 0 to 204800000 [ 345.645970] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 345.668173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 345.672839] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 346.765486] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 346.783569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 346.823458] Lustre: lustre-MDT0000: new disk, initializing [ 346.850730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 346.858780] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 348.313801] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 352.171578] loop1: detected capacity change from 0 to 204800000 [ 354.282571] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 354.307796] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 354.421340] Lustre: lustre-OST07c6: new disk, initializing [ 354.425124] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 354.428359] Lustre: Skipped 1 previous similar message [ 354.455330] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 356.555375] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 363.009977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 363.013556] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 363.025320] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 450.017524] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 450.021066] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.027474] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 455.137728] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 460.257260] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 461.280105] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 461.296087] LustreError: 26379:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 461.359434] Lustre: server umount lustre-OST07c6 complete [ 465.028739] LustreError: 26880:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 465.031622] LustreError: 26880:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 465.123882] Lustre: server umount lustre-MDT0000 complete [ 472.400234] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 473.575910] Key type lgssc unregistered [ 473.738534] LNet: 27708:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 474.794628] LNet: Removed LNI 192.168.203.124@tcp [ 475.147787] Key type .llcrypt unregistered [ 475.149510] Key type ._llcrypt unregistered [ 482.666707] Key type ._llcrypt registered [ 482.667942] Key type .llcrypt registered [ 482.705532] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 483.050398] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 483.107436] alg: No test for adler32 (adler32-zlib) [ 483.978625] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 484.064158] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 484.066206] LNet: Accept secure, port 988 [ 485.648274] Key type lgssc registered [ 486.026231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 488.844146] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 488.851974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.031365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 491.396639] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 494.332880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 494.463263] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 495.847246] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 497.789363] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 499.168461] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 501.618833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 501.757556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 503.867516] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 505.826068] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 515.107758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 518.363979] LustreError: 31122:0:(qsd_reint.c:618:qqi_reint_delayed()) lustre-MDT0000: Delaying reintegration for qtype:2 until pending updates are flushed. [ 523.747197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 523.747242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.750060] LustreError: Skipped 1 previous similar message [ 523.755442] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.282021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.282641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 525.284357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.022481] LustreError: 31373:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 527.078629] Lustre: server umount lustre-OST0000 complete [ 530.401608] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.401874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.404800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.407607] Lustre: Skipped 1 previous similar message [ 530.409953] Lustre: Skipped 2 previous similar messages [ 534.461369] LustreError: 31574:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 534.463386] LustreError: 31574:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 534.489860] Lustre: server umount lustre-MDT0000 complete [ 535.520947] LustreError: 28924:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.524788] LustreError: 28924:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 535.766627] LustreError: 28906:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009707 with bad export cookie 13123930047082435914 [ 535.769585] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 535.770327] LustreError: 28906:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 535.825332] LustreError: 31776:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 535.826976] LustreError: 31776:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 535.908460] Lustre: server umount lustre-MDT0001 complete [ 540.574195] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 541.698282] Key type lgssc unregistered [ 541.837439] LNet: 32303:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 542.886468] LNet: Removed LNI 192.168.203.124@tcp [ 543.202060] Key type .llcrypt unregistered [ 543.203528] Key type ._llcrypt unregistered [ 550.439023] Key type ._llcrypt registered [ 550.439833] Key type .llcrypt registered [ 550.474497] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 550.828251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 550.875809] alg: No test for adler32 (adler32-zlib) [ 551.738316] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 551.826399] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 551.828580] LNet: Accept secure, port 988 [ 553.408147] Key type lgssc registered [ 553.768268] Lustre: Echo OBD driver; http://www.lustre.org/ [ 556.630887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 556.637492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 557.824440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 559.204438] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 562.354082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 562.520763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 564.130661] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 566.100741] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 567.383455] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 569.808238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 569.931557] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 571.857141] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 573.790325] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 582.248439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 588.768925] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 588.771556] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.776272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.868180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.869564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.870053] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 594.606141] LustreError: 35769:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 594.667299] Lustre: server umount lustre-OST0000 complete [ 597.985631] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 597.986448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 597.990872] Lustre: Skipped 1 previous similar message [ 597.992637] Lustre: Skipped 1 previous similar message [ 603.105134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 603.107302] Lustre: Skipped 1 previous similar message [ 608.225503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 608.228116] Lustre: Skipped 1 previous similar message [ 610.272127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 610.305547] LustreError: 35971:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 610.308256] LustreError: 35971:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 610.337839] Lustre: server umount lustre-MDT0000 complete [ 611.669570] LustreError: 33500:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009783 with bad export cookie 9928491883585074934 [ 611.672970] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.674399] LustreError: 33500:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 611.741233] LustreError: 36173:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 611.743430] LustreError: 36173:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 611.824747] Lustre: server umount lustre-MDT0001 complete [ 616.784940] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 617.988937] Key type lgssc unregistered [ 618.137427] LNet: 36700:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 619.174370] LNet: Removed LNI 192.168.203.124@tcp [ 619.519119] Key type .llcrypt unregistered [ 619.520688] Key type ._llcrypt unregistered [ 628.476096] Key type ._llcrypt registered [ 628.477113] Key type .llcrypt registered [ 628.513828] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 628.906567] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 628.929560] alg: No test for adler32 (adler32-zlib) [ 629.797958] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 629.885030] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 629.886890] LNet: Accept secure, port 988 [ 631.473082] Key type lgssc registered [ 631.862444] Lustre: Echo OBD driver; http://www.lustre.org/ [ 634.651520] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 634.660463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.833243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 637.210929] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 640.188568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 640.336848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 641.673624] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 643.504287] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 644.714840] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 646.951318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 647.063591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 648.947753] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 650.749268] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 660.391391] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 665.571482] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 665.572028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 665.576123] Lustre: Skipped 1 previous similar message [ 666.479385] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 666.483056] Lustre: Skipped 2 previous similar messages [ 669.693505] LustreError: 40315:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 669.724806] Lustre: server umount lustre-MDT0000 complete [ 670.690581] LustreError: 38771:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 670.695976] LustreError: 38771:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 671.061502] LustreError: 38046:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009842 with bad export cookie 8182845792531263994 [ 671.063772] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 671.067021] LustreError: 38046:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 671.588877] LustreError: 38060:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 671.597162] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 675.808885] LustreError: 38061:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 675.809206] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 675.819741] Lustre: Skipped 1 previous similar message [ 675.822348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 677.188282] LustreError: 40518:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 677.190769] LustreError: 40518:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 677.271645] Lustre: server umount lustre-MDT0001 complete [ 695.213902] LustreError: 40741:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 695.215612] LustreError: 40741:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 695.278724] Lustre: server umount lustre-OST0000 complete [ 701.986884] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 703.150324] Key type lgssc unregistered [ 703.296400] LNet: 41573:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 704.358350] LNet: Removed LNI 192.168.203.124@tcp [ 704.679511] Key type .llcrypt unregistered [ 704.680348] Key type ._llcrypt unregistered [ 713.802103] Key type ._llcrypt registered [ 713.803483] Key type .llcrypt registered [ 713.843263] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 714.211737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 714.240447] alg: No test for adler32 (adler32-zlib) [ 715.090421] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 715.175308] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 715.177649] LNet: Accept secure, port 988 [ 716.760165] Key type lgssc registered [ 717.108962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 719.729718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 719.741996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 720.905153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 722.203891] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 724.944801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 725.063285] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 726.371441] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 728.139236] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 729.410541] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 731.731385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 731.852926] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 733.680168] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 735.364931] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 745.190603] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 750.563573] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 750.563830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 750.564335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 750.567828] Lustre: Skipped 1 previous similar message [ 755.682087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 755.684719] Lustre: Skipped 3 previous similar messages [ 760.802036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 760.804248] Lustre: Skipped 2 previous similar messages [ 762.336091] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 762.351182] LustreError: 45184:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 762.407062] Lustre: server umount lustre-OST0000 complete [ 775.145837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.151339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 775.153741] Lustre: Skipped 1 previous similar message [ 780.257024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 780.259532] Lustre: Skipped 2 previous similar messages [ 781.371965] LustreError: 45560:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 781.373405] LustreError: 45560:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 781.398374] Lustre: server umount lustre-MDT0000 complete [ 782.602211] LustreError: 42915:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754009954 with bad export cookie 1426710394249332464 [ 782.604709] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 782.605660] LustreError: 42915:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 788.732070] LustreError: 45761:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 788.733894] LustreError: 45761:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 788.804117] Lustre: server umount lustre-MDT0001 complete [ 793.401077] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 794.549677] Key type lgssc unregistered [ 794.686366] LNet: 46287:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 795.750365] LNet: Removed LNI 192.168.203.124@tcp [ 796.055987] Key type .llcrypt unregistered [ 796.057052] Key type ._llcrypt unregistered [ 804.954673] Key type ._llcrypt registered [ 804.956073] Key type .llcrypt registered [ 805.000937] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 805.488602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 805.501256] alg: No test for adler32 (adler32-zlib) [ 806.355846] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 806.441637] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 806.443568] LNet: Accept secure, port 988 [ 808.025392] Key type lgssc registered [ 808.427279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 811.231891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 811.239540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 812.396342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 813.650356] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 816.462496] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 816.577691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 817.818952] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 819.532384] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 820.674651] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 822.902031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 823.012696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.810130] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 826.499479] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 837.350434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 844.179806] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 845.331303] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 853.476199] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 853.478740] LNetError: 46955:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 853.481987] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 856.053460] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2025-07-3121h00m27s [ 856.520640] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 857.449038] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 857.568772] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 857.570923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 857.575776] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 857.578729] LNetError: 46958:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 857.582026] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 858.467830] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 858.469865] Lustre: Skipped 3 previous similar messages [ 863.356988] LustreError: 50216:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 863.386572] Lustre: server umount lustre-MDT0000 complete [ 863.712747] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 863.716247] LNetError: 46956:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 863.719072] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 864.451808] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 864.611742] LustreError: 47647:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 864.616577] LustreError: 47647:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 866.537263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 866.577385] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 866.657503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 866.684082] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 867.685146] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 867.687039] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 867.688323] LNetError: 46957:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 867.692620] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 867.924156] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 869.067107] Lustre: DEBUG MARKER: Wait for df (35961) ... [ 869.592733] Lustre: DEBUG MARKER: done [ 871.905149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 871.908548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 871.913771] Lustre: Skipped 2 previous similar messages [ 871.915616] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 871.917542] Lustre: Skipped 3 previous similar messages [ 872.931094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 876.587575] LustreError: 51257:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 876.589268] LustreError: 51257:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 876.643320] Lustre: server umount lustre-OST0000 complete [ 878.049795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 878.049884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 878.053764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 878.056142] Lustre: Skipped 1 previous similar message [ 878.057687] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 878.057946] Lustre: Skipped 3 previous similar messages [ 878.058989] LNetError: 46955:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 878.066028] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 884.129282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 884.132635] Lustre: Skipped 3 previous similar messages [ 892.384200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 892.415431] LustreError: 51462:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 892.418020] LustreError: 51462:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 892.445625] Lustre: server umount lustre-MDT0000 complete [ 893.409437] LustreError: 47651:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 893.416557] LustreError: 47651:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 893.746719] LustreError: 47633:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010065 with bad export cookie 1580263685443542339 [ 893.747866] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 893.751784] LustreError: 47633:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 893.900769] Lustre: server umount lustre-MDT0001 complete [ 898.563850] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 899.674464] Key type lgssc unregistered [ 899.815434] LNet: 52203:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.838705] LNet: Removed LNI 192.168.203.124@tcp [ 901.149580] Key type .llcrypt unregistered [ 901.151319] Key type ._llcrypt unregistered [ 912.159180] Key type ._llcrypt registered [ 912.160700] Key type .llcrypt registered [ 912.199615] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 918.315769] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 918.744421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 918.755423] alg: No test for adler32 (adler32-zlib) [ 919.610105] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 919.685781] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 919.687683] LNet: Accept secure, port 988 [ 921.264159] Key type lgssc registered [ 921.629776] Lustre: Echo OBD driver; http://www.lustre.org/ [ 924.618412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 926.852837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 928.947149] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 931.156331] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 933.444058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 933.461428] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 933.466476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 934.552271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 934.564743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 934.603577] Lustre: lustre-MDT0000: new disk, initializing [ 934.633141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 934.640915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 935.951042] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 940.111573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 940.137170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 940.172878] Lustre: 56109:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 940.197237] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 940.199348] Lustre: Skipped 1 previous similar message [ 940.234996] Lustre: lustre-MDT0001: new disk, initializing [ 940.266811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 940.278221] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 940.281668] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 941.560436] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 944.646592] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 945.843101] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 948.083645] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 948.107047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 948.201692] Lustre: lustre-OST0000: new disk, initializing [ 948.203979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 948.237169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 949.943594] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 949.947641] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 949.979448] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 950.180977] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 953.315555] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 954.677121] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 954.759691] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 956.132692] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 956.226260] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 960.482235] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 960.483298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 960.485106] LustreError: Skipped 1 previous similar message [ 960.485594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 960.488351] Lustre: Skipped 1 previous similar message [ 960.494166] Lustre: Skipped 1 previous similar message [ 965.602600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 965.604291] Lustre: Skipped 1 previous similar message [ 970.722897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 970.725079] Lustre: Skipped 1 previous similar message [ 971.232143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 971.251497] LustreError: 59020:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 971.309913] Lustre: server umount lustre-OST0000 complete [ 975.842134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 975.844416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 975.847475] Lustre: Skipped 1 previous similar message [ 978.813414] LustreError: 59228:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 978.816393] LustreError: 59228:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 978.852629] Lustre: server umount lustre-MDT0000 complete [ 980.181247] LustreError: 57846:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010152 with bad export cookie 9461055201704648156 [ 980.183450] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.185923] LustreError: 57846:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 980.250197] LustreError: 59430:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 980.252722] LustreError: 59430:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 980.331715] Lustre: server umount lustre-MDT0001 complete [ 983.143411] Lustre: DEBUG MARKER: == conf-sanity test 35b: Continue reconnection retries, if the active server is busy ========================================================== 21:02:34 (1754010154) [ 985.550078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 985.723728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 987.057455] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 990.066806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 990.195220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 991.699640] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 993.737315] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 995.059356] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 997.501250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 999.686928] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1001.638407] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1014.407381] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2025-07-3121h03m05s [ 1014.942565] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1017.208650] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1018.064789] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1018.080375] LustreError: 60808:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1022.948867] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 1022.952371] LNetError: 54317:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1022.955497] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1038.194428] Lustre: lustre-MDT0000: Client 1d28b5d9-060b-4c00-903b-028de39c3922 (at 192.168.203.24@tcp) reconnecting [ 1058.120073] LustreError: 60808:0:(mdt_handler.c:2329:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1058.745346] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1059.468628] Lustre: DEBUG MARKER: done [ 1061.344737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1061.346857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.351365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1061.352837] Lustre: Skipped 1 previous similar message [ 1062.370249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1066.862240] LustreError: 63103:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1066.864931] LustreError: 63103:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1066.916698] Lustre: server umount lustre-OST0000 complete [ 1072.096791] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1072.098652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1072.102204] Lustre: Skipped 1 previous similar message [ 1072.103639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1072.105366] Lustre: Skipped 2 previous similar messages [ 1072.107311] LNetError: Refusing connection from 192.168.203.124 for 127.0.0.2@tcp: No matching NI [ 1072.109115] LNetError: 54320:0:(socklnd_cb.c:1794:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 [ 1072.111096] LNetError: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? [ 1074.235287] LustreError: 63305:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1074.236942] LustreError: 63305:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1074.258243] Lustre: server umount lustre-MDT0000 complete [ 1075.428198] LustreError: 60082:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010247 with bad export cookie 9461055201704649017 [ 1075.430504] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1075.433243] LustreError: 60082:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1075.564323] Lustre: server umount lustre-MDT0001 complete [ 1079.789315] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1080.784225] Key type lgssc unregistered [ 1080.920428] LNet: 64034:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.958336] LNet: Removed LNI 192.168.203.124@tcp [ 1082.233086] Key type .llcrypt unregistered [ 1082.233884] Key type ._llcrypt unregistered [ 1093.089143] Key type ._llcrypt registered [ 1093.090076] Key type .llcrypt registered [ 1093.125990] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 1098.739330] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1099.012224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1099.020070] alg: No test for adler32 (adler32-zlib) [ 1099.870241] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1099.943917] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1099.945291] LNet: Accept secure, port 988 [ 1101.520132] Key type lgssc registered [ 1101.849525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1104.771136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1106.870075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1108.722069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1110.746125] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1113.022146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1113.039490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1113.046260] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1114.129246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1114.139339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1114.179312] Lustre: lustre-MDT0000: new disk, initializing [ 1114.202857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1114.210198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1115.511599] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1119.624756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1119.645994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1119.680426] Lustre: 67939:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1119.706371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1119.708176] Lustre: Skipped 1 previous similar message [ 1119.744136] Lustre: lustre-MDT0001: new disk, initializing [ 1119.774825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1119.786713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1119.789503] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1121.147233] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1124.228547] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1125.427970] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1127.740407] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1127.764256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1127.859412] Lustre: lustre-OST0000: new disk, initializing [ 1127.861873] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1127.891570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.295674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1128.300150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1128.352119] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1129.795022] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1132.745529] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1133.979361] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1134.052805] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1135.254388] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1135.333336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1138.659304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1138.659721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.662110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1138.667489] Lustre: Skipped 1 previous similar message [ 1138.671946] Lustre: Skipped 1 previous similar message [ 1141.995107] LustreError: 70845:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1142.055676] Lustre: server umount lustre-OST0000 complete [ 1143.777161] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1143.777888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1143.781200] Lustre: Skipped 1 previous similar message [ 1148.897592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1148.900695] Lustre: Skipped 2 previous similar messages [ 1149.375729] LustreError: 71046:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 1149.378779] LustreError: 71046:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1149.412710] Lustre: server umount lustre-MDT0000 complete [ 1150.775180] LustreError: 67930:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010322 with bad export cookie 5570632088982721957 [ 1150.778522] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.779106] LustreError: 67930:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1150.843397] LustreError: 71247:0:(obd_class.h:479:obd_check_dev()) Device 20 not setup [ 1150.846133] LustreError: 71247:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1150.934504] Lustre: server umount lustre-MDT0001 complete [ 1153.792873] Lustre: DEBUG MARKER: == conf-sanity test 36: df report consistency on OSTs with different block size ========================================================== 21:05:25 (1754010325) [ 1154.331623] Lustre: DEBUG MARKER: SKIP: conf-sanity test_36 remote OST [ 1154.914860] Lustre: DEBUG MARKER: == conf-sanity test 37: verify set tunables works for symlink device ========================================================== 21:05:26 (1754010326) [ 1158.754967] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1160.158357] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.311358] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.671278] Lustre: Failing over lustre-MDT0000 [ 1160.673136] LustreError: 72312:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1160.677362] LustreError: 72312:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1160.679463] LustreError: 72312:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -5 [ 1160.738110] LustreError: 72363:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1160.739949] LustreError: 72363:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1160.810389] Lustre: server umount lustre-MDT0000 complete [ 1162.940810] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:05:34 (1754010334) [ 1165.220066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1165.357974] LustreError: 72929:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.365248] LustreError: 72929:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1165.373458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1166.645201] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1169.684297] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1171.245877] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1173.069345] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1174.285663] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1176.579461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1176.687899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1176.690097] Lustre: Skipped 1 previous similar message [ 1178.551091] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1180.424958] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1197.885065] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1200.609179] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1200.611836] LustreError: Skipped 1 previous similar message [ 1200.614060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1200.619872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1205.560561] LustreError: 75274:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1205.562786] LustreError: 75274:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1205.589450] Lustre: server umount lustre-MDT0000 complete [ 1206.753844] LustreError: 72925:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1206.760111] LustreError: 72925:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1206.777969] LustreError: 72910:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010378 with bad export cookie 5570632088982723042 [ 1206.782399] LustreError: 72910:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1206.783096] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1206.910881] Lustre: server umount lustre-MDT0001 complete [ 1208.029544] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1209.520952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1212.249082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1216.992647] LustreError: 66157:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000c6125b31 x1839213073981056/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1217.100985] LustreError: 74523:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1217.101316] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.113146] Lustre: Skipped 2 previous similar messages [ 1217.133651] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1217.155389] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1218.323945] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1221.049942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1222.362602] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1224.069860] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1225.228980] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1226.213645] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1227.232476] Lustre: 66161:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754010383/real 1754010383] req@000000004ca99af2 x1839213073980800/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754010399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1231.337679] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 1233.248134] Lustre: 66158:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754010388/real 1754010388] req@00000000666b0946 x1839213073980928/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754010404 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1236.450730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1236.451192] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1236.451954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1236.451959] Lustre: Skipped 3 previous similar messages [ 1236.458808] Lustre: Skipped 1 previous similar message [ 1238.330776] LustreError: 77723:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1238.332731] LustreError: 77723:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1238.360480] Lustre: server umount lustre-MDT0000 complete [ 1239.556144] LustreError: 75277:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010411 with bad export cookie 5570632088982724932 [ 1239.557991] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1239.561077] LustreError: 75277:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1239.690717] Lustre: server umount lustre-MDT0001 complete [ 1241.739350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1244.681759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1249.696700] LustreError: 66157:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000067cfee59 x1839213074000640/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1249.807174] LustreError: 74523:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1249.812721] LustreError: 74523:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 1249.834400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1249.836408] Lustre: Skipped 1 previous similar message [ 1249.863513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1251.193546] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1254.031225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1254.117533] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.124791] Lustre: Skipped 1 previous similar message [ 1255.490513] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1256.229302] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to (at 0@lo) [ 1257.335974] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1257.952178] Lustre: 66161:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754010413/real 1754010413] req@000000000080067f x1839213074000256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754010429 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1258.498172] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1261.152157] Lustre: 66158:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754010416/real 1754010416] req@000000008bb87a6c x1839213074000384/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754010432 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1261.539376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1261.540906] Lustre: Skipped 4 previous similar messages [ 1265.632150] Lustre: 66160:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754010421/real 1754010421] req@00000000c93b0bd8 x1839213074000768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1754010437 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1265.644155] Lustre: 66160:0:(client.c:2453:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1266.305138] LustreError: 80167:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1266.308235] LustreError: 80167:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 1266.338602] Lustre: server umount lustre-MDT0000 complete [ 1266.658530] LustreError: 78779:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.665797] LustreError: 78779:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1267.690101] LustreError: 72909:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010439 with bad export cookie 5570632088982726304 [ 1267.693038] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1267.694232] LustreError: 72909:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1267.848285] Lustre: server umount lustre-MDT0001 complete [ 1270.036602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1271.140765] Lustre: DEBUG MARKER: files compared the same [ 1282.090428] Lustre: server umount lustre-OST0000 complete [ 1288.048660] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1289.072707] Key type lgssc unregistered [ 1289.211311] LNet: 81793:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1290.278283] LNet: Removed LNI 192.168.203.124@tcp [ 1290.552272] Key type .llcrypt unregistered [ 1290.553292] Key type ._llcrypt unregistered [ 1298.998371] Key type ._llcrypt registered [ 1298.999325] Key type .llcrypt registered [ 1299.034408] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1299.410872] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1299.435118] alg: No test for adler32 (adler32-zlib) [ 1300.271430] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1300.337813] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1300.339539] LNet: Accept secure, port 988 [ 1301.912158] Key type lgssc registered [ 1302.304668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1304.742929] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1304.749511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1305.901247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1307.273086] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 1310.065507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1310.166692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1311.427326] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 1313.119413] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1314.196391] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1316.291850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1316.379469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1318.174377] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug +malloc all [ 1319.910770] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1330.722755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1339.872534] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1339.874816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.877915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1340.896880] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1340.897633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1340.899727] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.908071] Lustre: Skipped 1 previous similar message [ 1344.299063] LustreError: 85407:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1344.353920] Lustre: server umount lustre-OST0000 complete [ 1346.016947] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1346.016949] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1346.017224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1346.026892] Lustre: Skipped 1 previous similar message [ 1351.136696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1351.140231] Lustre: Skipped 1 previous similar message [ 1351.679838] LustreError: 85609:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1351.682033] LustreError: 85609:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1351.708357] Lustre: server umount lustre-MDT0000 complete [ 1352.995815] LustreError: 85307:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010524 with bad export cookie 13329531387959472115 [ 1352.997456] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1353.000150] LustreError: 85307:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1353.058756] LustreError: 85811:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 1353.061056] LustreError: 85811:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1353.133507] Lustre: server umount lustre-MDT0001 complete [ 1357.832743] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1358.880784] Key type lgssc unregistered [ 1359.014357] LNet: 86337:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1360.038638] LNet: Removed LNI 192.168.203.124@tcp [ 1360.325660] Key type .llcrypt unregistered [ 1360.326852] Key type ._llcrypt unregistered [ 1368.910576] Key type ._llcrypt registered [ 1368.912013] Key type .llcrypt registered [ 1368.945013] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1369.354575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1369.360894] alg: No test for adler32 (adler32-zlib) [ 1370.208422] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1370.291504] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1370.293682] LNet: Accept secure, port 988 [ 1371.872187] Key type lgssc registered [ 1372.205509] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1374.766485] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1374.773395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1391.264255] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1397.408150] LustreError: 87672:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 1427.104159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1428.993078] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1430.740422] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1433.235847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1433.273694] Lustre: *** cfs_fail_loc=706, val=0*** [ 1433.404557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1433.428861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1434.709806] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1437.540881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1437.660635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1439.004193] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1440.905716] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1442.145650] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1443.808968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1443.811524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1443.816579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.886585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1446.887075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1446.894090] Lustre: Skipped 1 previous similar message [ 1452.002073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1452.004317] Lustre: Skipped 1 previous similar message [ 1455.010357] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1455.013073] Lustre: Skipped 1 previous similar message [ 1457.120165] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1457.138209] LustreError: 89979:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 1457.203551] Lustre: server umount lustre-OST0000 complete [ 1460.194681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.199735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1460.202200] Lustre: Skipped 1 previous similar message [ 1464.572244] LustreError: 90184:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1464.574377] LustreError: 90184:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1464.602641] Lustre: server umount lustre-MDT0000 complete [ 1465.313220] LustreError: 88642:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1465.317879] LustreError: 88642:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 1465.838178] LustreError: 87681:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010637 with bad export cookie 4588396892659445469 [ 1465.840995] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1465.841277] LustreError: 87681:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1465.904170] LustreError: 90386:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1465.906507] LustreError: 90386:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1465.986621] Lustre: server umount lustre-MDT0001 complete [ 1469.645603] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1470.724526] Key type lgssc unregistered [ 1470.863351] LNet: 90912:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1471.910389] LNet: Removed LNI 192.168.203.124@tcp [ 1472.235410] Key type .llcrypt unregistered [ 1472.236787] Key type ._llcrypt unregistered [ 1481.287159] Key type ._llcrypt registered [ 1481.288470] Key type .llcrypt registered [ 1481.326785] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1481.725592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1481.755465] alg: No test for adler32 (adler32-zlib) [ 1482.612639] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1482.689863] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1482.691731] LNet: Accept secure, port 988 [ 1484.280166] Key type lgssc registered [ 1484.639858] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1487.278763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1487.286455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1489.531557] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1491.895328] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1492.049064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1493.230540] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1495.929438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1496.055614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1497.875674] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1500.537913] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1500.541171] Lustre: MGS: Client fd479e8e-20da-4744-a64e-f7f72b9ec64a (at 0@lo) reconnecting [ 1500.619187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.655957] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1501.911817] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1510.881395] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1510.883617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1510.887831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1511.905872] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.906809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1511.907158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1511.914114] Lustre: Skipped 1 previous similar message [ 1515.692068] LustreError: 94880:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 1515.753067] Lustre: server umount lustre-OST0000 complete [ 1517.025588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1517.025596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.026262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1517.033328] Lustre: Skipped 1 previous similar message [ 1522.146085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1522.147701] Lustre: Skipped 2 previous similar messages [ 1527.265483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.267392] Lustre: Skipped 1 previous similar message [ 1531.360153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1531.394609] LustreError: 95082:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1531.396670] LustreError: 95082:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1531.409594] Lustre: server umount lustre-MDT0000 complete [ 1532.773820] LustreError: 95285:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1532.776235] LustreError: 95285:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1532.848745] Lustre: server umount lustre-MDT0001 complete [ 1534.051838] Lustre: server umount MGS complete [ 1537.756356] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:11:49 (1754010709) [ 1546.769431] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 1549.657779] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1552.654653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1554.758383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1556.614683] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1558.615863] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1560.706919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1560.727498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1561.933404] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1565.757632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1565.784046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1565.862736] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1565.879697] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1565.915582] Lustre: lustre-MDT0001: new disk, initializing [ 1565.942111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1565.951519] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1565.954527] Lustre: 99972:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1565.958694] LustreError: 99972:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1567.208495] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1571.272522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 1571.300588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1571.382944] Lustre: lustre-OST0000: new disk, initializing [ 1571.384922] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1571.404059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1573.373142] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1576.618634] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1578.978123] Lustre: 101474:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@00000000658ebf61 x1839213475563136/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1754010766 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1579.002644] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1579.006979] Lustre: MGS: Client 4e93cdf0-e1a2-44ba-94a6-53310736e32f (at 0@lo) reconnecting [ 1579.020697] Lustre: 99264:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1579.027787] LustreError: 99264:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 1579.037665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1579.076919] Lustre: lustre-MDT0000: new disk, initializing [ 1579.140166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1580.193067] Lustre: 101661:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@000000005df16525 x1839213475568256/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1754010768 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1580.199806] Lustre: 101661:0:(client.c:1611:after_reply()) Skipped 1 previous similar message [ 1580.381137] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1581.024590] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1588.067098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1588.070276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1589.282863] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1589.285810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1589.286734] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1609.696923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1609.700280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.704137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1609.706712] Lustre: Skipped 2 previous similar messages [ 1615.214423] LustreError: 102190:0:(obd_class.h:479:obd_check_dev()) Device 11 not setup [ 1615.216774] LustreError: 102190:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 1615.279134] Lustre: server umount lustre-OST0000 complete [ 1616.865849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1616.865917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1616.871823] Lustre: Skipped 2 previous similar messages [ 1622.783160] LustreError: 102394:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 1622.786247] LustreError: 102394:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1622.797934] Lustre: server umount lustre-MDT0000 complete [ 1638.368132] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1638.396509] LustreError: 102595:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 1638.398054] LustreError: 102595:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1638.467917] Lustre: server umount lustre-MDT0001 complete [ 1639.655258] Lustre: server umount MGS complete [ 1643.268707] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:13:34 (1754010814) [ 1650.193358] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1651.353297] Key type lgssc unregistered [ 1651.500341] LNet: 104126:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1652.518573] LNet: Removed LNI 192.168.203.124@tcp [ 1652.865081] Key type .llcrypt unregistered [ 1652.866957] Key type ._llcrypt unregistered [ 1660.045770] Key type ._llcrypt registered [ 1660.047582] Key type .llcrypt registered [ 1660.087876] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1660.374721] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1660.387663] alg: No test for adler32 (adler32-zlib) [ 1661.262506] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1661.354895] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1661.356542] LNet: Accept secure, port 988 [ 1662.944166] Key type lgssc registered [ 1663.329582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1666.092696] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1666.100455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1667.257641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.530542] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1671.343372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1671.461619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1672.702401] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1674.427730] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1675.586821] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1677.789225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1677.895123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1679.643478] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1681.292526] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1692.262793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1702.881908] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.883428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1702.885497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1702.888900] Lustre: Skipped 1 previous similar message [ 1705.773222] LustreError: 107545:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1705.836965] Lustre: server umount lustre-OST0000 complete [ 1707.488889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1707.491349] LustreError: Skipped 1 previous similar message [ 1707.492652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1707.496859] Lustre: Skipped 1 previous similar message [ 1707.498698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1713.121494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1713.124332] Lustre: Skipped 2 previous similar messages [ 1713.283711] LustreError: 107747:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1713.288049] LustreError: 107747:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1713.319916] Lustre: server umount lustre-MDT0000 complete [ 1714.701640] LustreError: 106009:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010886 with bad export cookie 108751803448062227 [ 1714.704232] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1714.706499] LustreError: 106009:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1714.780022] LustreError: 107948:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1714.783493] LustreError: 107948:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1714.870621] Lustre: server umount lustre-MDT0001 complete [ 1719.279168] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1720.238470] Key type lgssc unregistered [ 1720.372342] LNet: 108473:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1721.382256] LNet: Removed LNI 192.168.203.124@tcp [ 1721.671676] Key type .llcrypt unregistered [ 1721.672705] Key type ._llcrypt unregistered [ 1727.590386] Key type ._llcrypt registered [ 1727.591572] Key type .llcrypt registered [ 1727.630269] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1728.002359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1728.031667] alg: No test for adler32 (adler32-zlib) [ 1728.893115] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1728.982858] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1728.985652] LNet: Accept secure, port 988 [ 1730.568174] Key type lgssc registered [ 1730.948706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1733.079836] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1733.080709] LustreError: 109503:0:(tgt_mount.c:2375:server_fill_super()) cfs_race id 716 sleeping [ 1733.083697] Lustre: Skipped 1 previous similar message [ 1733.087185] LustreError: 109504:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 waking [ 1733.089112] LustreError: 109503:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1733.091461] LustreError: 109503:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1733.095510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1733.095888] LustreError: 109503:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1734.253701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1736.782698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1736.876986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1738.156461] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1739.786051] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing lsmod [ 1740.793086] LustreError: 110667:0:(tgt_mount.c:2375:server_fill_super()) cfs_race id 716 sleeping [ 1740.793456] LustreError: 110668:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 waking [ 1740.799389] LustreError: 110667:0:(tgt_mount.c:2375:server_fill_super()) cfs_fail_race id 716 awake: rc=4997 [ 1740.802121] LustreError: 110667:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1740.805677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1740.806604] LustreError: 110667:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1740.928895] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1741.914623] LustreError: 109573:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 1741.918577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1741.918993] LustreError: 109573:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 1741.919690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1741.922851] Lustre: Skipped 2 previous similar messages [ 1741.926568] LustreError: 109573:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 1747.042252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1747.045575] Lustre: Skipped 4 previous similar messages [ 1748.028717] LustreError: 110856:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1748.060739] Lustre: server umount lustre-MDT0000 complete [ 1749.305145] LustreError: 109519:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010921 with bad export cookie 10996449105308272933 [ 1749.308753] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1749.308876] LustreError: 109519:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1751.136232] LustreError: 109977:0:(client.c:1375:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008608dfe1 x1839213733493888/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1751.145123] LustreError: 111060:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1751.148050] LustreError: 111060:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1751.218631] Lustre: server umount lustre-MDT0001 complete [ 1762.801334] LustreError: 111261:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 1762.803507] LustreError: 111261:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1762.871582] Lustre: server umount lustre-OST0000 complete [ 1765.865753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1766.028640] LustreError: 111738:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.039411] LustreError: 111738:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 1766.056745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1767.433759] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1770.535053] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1770.668495] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1772.076701] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1773.794501] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1774.916708] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1776.981748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1778.874182] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1780.542353] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1790.440262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1795.553750] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1795.555867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1795.557413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1795.562622] Lustre: Skipped 1 previous similar message [ 1798.767090] LustreError: 113987:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1798.769980] LustreError: 113987:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1798.838082] Lustre: server umount lustre-OST0000 complete [ 1800.673530] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1800.674161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1800.680829] Lustre: Skipped 1 previous similar message [ 1800.683151] Lustre: Skipped 1 previous similar message [ 1805.793208] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1805.795653] Lustre: Skipped 1 previous similar message [ 1806.269583] LustreError: 114188:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1806.271826] LustreError: 114188:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1806.304478] Lustre: server umount lustre-MDT0000 complete [ 1807.602144] LustreError: 111718:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754010979 with bad export cookie 10996449105308273612 [ 1807.604988] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1807.605939] LustreError: 111718:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1807.750567] Lustre: server umount lustre-MDT0001 complete [ 1812.585213] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1813.625765] Key type lgssc unregistered [ 1813.766297] LNet: 114916:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1814.822281] LNet: Removed LNI 192.168.203.124@tcp [ 1815.108584] Key type .llcrypt unregistered [ 1815.110037] Key type ._llcrypt unregistered [ 1823.397958] Key type ._llcrypt registered [ 1823.399363] Key type .llcrypt registered [ 1823.435681] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1823.761576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1823.824259] alg: No test for adler32 (adler32-zlib) [ 1824.676997] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1824.753322] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1824.755190] LNet: Accept secure, port 988 [ 1826.336129] Key type lgssc registered [ 1826.672418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1829.135628] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1829.141582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1830.288917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1831.524129] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1834.190430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1834.305073] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1835.642643] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1837.369907] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1838.513493] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1840.664172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1840.761090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1842.490698] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1844.131883] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1856.103319] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1861.950761] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1862.778412] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1865.696830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1865.699359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1865.703549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.723278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1866.723959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1866.725664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1869.871044] LustreError: 118630:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1869.932573] Lustre: server umount lustre-OST0000 complete [ 1871.841364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1871.841800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1871.845132] Lustre: Skipped 1 previous similar message [ 1871.850248] Lustre: Skipped 2 previous similar messages [ 1876.961086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1876.963136] Lustre: Skipped 1 previous similar message [ 1877.370271] LustreError: 118832:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1877.371813] LustreError: 118832:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1877.396602] Lustre: server umount lustre-MDT0000 complete [ 1878.561126] LustreError: 118482:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011050 with bad export cookie 5265968170242652234 [ 1878.563790] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1878.565692] LustreError: 118482:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1878.632124] LustreError: 119037:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1878.633897] LustreError: 119037:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1878.704895] Lustre: server umount lustre-MDT0001 complete [ 1883.144937] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1884.240815] Key type lgssc unregistered [ 1884.378437] LNet: 119562:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1885.414303] LNet: Removed LNI 192.168.203.124@tcp [ 1885.729889] Key type .llcrypt unregistered [ 1885.731593] Key type ._llcrypt unregistered [ 1892.810838] Key type ._llcrypt registered [ 1892.812009] Key type .llcrypt registered [ 1892.856190] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1893.222439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1893.265462] alg: No test for adler32 (adler32-zlib) [ 1894.114536] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1894.191212] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1894.192658] LNet: Accept secure, port 988 [ 1895.768148] Key type lgssc registered [ 1896.145874] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1898.837471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1898.843582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1900.007943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1901.340565] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1904.243110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1904.377617] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1905.733823] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1907.561614] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1908.660911] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1910.821974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1910.933282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1912.645027] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1914.292273] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1924.267498] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1929.697719] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1929.698725] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1929.700455] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1929.708076] Lustre: Skipped 1 previous similar message [ 1933.678612] LustreError: 122982:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 1933.738777] Lustre: server umount lustre-OST0000 complete [ 1935.328767] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1935.330904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1935.334799] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1935.336529] Lustre: Skipped 1 previous similar message [ 1939.937104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.937509] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1941.049205] LustreError: 123185:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 1941.051776] LustreError: 123185:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 1941.071572] Lustre: server umount lustre-MDT0000 complete [ 1942.238103] LustreError: 120710:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011114 with bad export cookie 881378750239496259 [ 1942.241759] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.245207] LustreError: 120710:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1942.303372] LustreError: 123387:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 1942.305077] LustreError: 123387:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 1942.378232] Lustre: server umount lustre-MDT0001 complete [ 1946.850427] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 1947.924807] Key type lgssc unregistered [ 1948.051378] LNet: 123913:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1949.094298] LNet: Removed LNI 192.168.203.124@tcp [ 1949.386155] Key type .llcrypt unregistered [ 1949.387665] Key type ._llcrypt unregistered [ 1957.788415] Key type ._llcrypt registered [ 1957.789783] Key type .llcrypt registered [ 1957.825841] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 1958.171385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1958.206060] alg: No test for adler32 (adler32-zlib) [ 1959.060674] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 1959.144952] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 1959.146339] LNet: Accept secure, port 988 [ 1960.728173] Key type lgssc registered [ 1961.067454] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1963.532102] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1963.538829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1964.680628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1965.841544] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1968.490278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1968.609500] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1969.800620] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1971.583724] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1972.671944] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1974.852986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 1974.958550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1976.720537] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 1978.483639] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1989.286211] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1996.602770] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1997.843878] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1997.846348] Lustre: Skipped 2 previous similar messages [ 1999.394613] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1999.397166] Lustre: Skipped 2 previous similar messages [ 2003.452844] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2003.463558] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2008.736232] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.24@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2008.739671] Lustre: Skipped 2 previous similar messages [ 2013.687452] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.24@tcp 192.168.0.[2,10]@tcp [ 2015.388700] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.24/24@tcp in log lustre-MDT0000 [ 2015.392629] Lustre: Skipped 2 previous similar messages [ 2022.904253] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.24/24@tcp [ 2022.906582] Lustre: Skipped 1 previous similar message [ 2025.441726] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2025.442027] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2025.445095] LustreError: Skipped 1 previous similar message [ 2025.448089] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2025.450786] Lustre: Skipped 1 previous similar message [ 2030.561354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2030.563952] Lustre: Skipped 1 previous similar message [ 2030.764060] LustreError: 129167:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2030.819707] Lustre: server umount lustre-OST0000 complete [ 2035.169109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2035.171414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2035.174290] Lustre: Skipped 1 previous similar message [ 2035.175682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2038.142389] LustreError: 129368:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2038.144843] LustreError: 129368:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2038.172261] Lustre: server umount lustre-MDT0000 complete [ 2039.445267] LustreError: 125257:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011211 with bad export cookie 5831200833495296685 [ 2039.447711] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.449958] LustreError: 125257:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2039.515610] LustreError: 129570:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2039.517438] LustreError: 129570:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2039.598134] Lustre: server umount lustre-MDT0001 complete [ 2044.331158] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2045.401150] Key type lgssc unregistered [ 2045.534408] LNet: 130096:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2046.566309] LNet: Removed LNI 192.168.203.124@tcp [ 2046.862760] Key type .llcrypt unregistered [ 2046.863694] Key type ._llcrypt unregistered [ 2057.105939] Key type ._llcrypt registered [ 2057.107245] Key type .llcrypt registered [ 2057.144335] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2057.536138] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2057.559751] alg: No test for adler32 (adler32-zlib) [ 2058.412432] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2058.493792] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2058.496233] LNet: Accept secure, port 988 [ 2060.072130] Key type lgssc registered [ 2060.398712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2062.839625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2062.848079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2064.000718] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2064.005545] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2064.010897] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.24/24@tcp [ 2065.246728] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2067.975078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2068.088309] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2068.098261] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.24/24@tcp [ 2068.110751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2069.482615] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2071.244567] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2072.496664] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2074.860067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2074.987788] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.869851] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2078.604673] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2088.296325] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 2093.537648] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2093.538975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2093.540157] LustreError: Skipped 1 previous similar message [ 2093.541074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.543893] Lustre: Skipped 1 previous similar message [ 2094.445891] LustreError: 133926:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2094.521593] Lustre: server umount lustre-OST0000 complete [ 2098.658973] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.659692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.662597] Lustre: Skipped 1 previous similar message [ 2098.664431] Lustre: Skipped 1 previous similar message [ 2101.949488] LustreError: 134127:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2101.951299] LustreError: 134127:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2101.970314] Lustre: server umount lustre-MDT0000 complete [ 2103.285221] LustreError: 131604:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011275 with bad export cookie 3863590380083231018 [ 2103.288582] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2103.289807] LustreError: 131604:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2103.352854] LustreError: 134328:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 2103.354864] LustreError: 134328:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2103.435694] Lustre: server umount lustre-MDT0001 complete [ 2108.255262] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2109.483247] Key type lgssc unregistered [ 2109.626401] LNet: 134854:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2110.694457] LNet: Removed LNI 192.168.203.124@tcp [ 2111.043970] Key type .llcrypt unregistered [ 2111.045201] Key type ._llcrypt unregistered [ 2119.809761] Key type ._llcrypt registered [ 2119.811436] Key type .llcrypt registered [ 2119.855329] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2125.472637] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2125.778926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2125.787955] alg: No test for adler32 (adler32-zlib) [ 2126.630301] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2126.707182] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2126.708615] LNet: Accept secure, port 988 [ 2128.288154] Key type lgssc registered [ 2128.626691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2131.370821] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2133.424894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2135.243432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2137.088947] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2139.142414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2139.159085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2139.163458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2140.238583] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2140.248448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2140.280201] Lustre: lustre-MDT0000: new disk, initializing [ 2140.300717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2140.306605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2141.523907] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2145.772556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2145.797741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2145.830835] Lustre: 138184:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2145.856783] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2145.859527] Lustre: Skipped 1 previous similar message [ 2145.903574] Lustre: lustre-MDT0001: new disk, initializing [ 2145.940059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2145.952728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2145.956261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2147.303581] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2150.440308] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2151.630480] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2153.846901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2153.872307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2153.973830] Lustre: lustre-OST0000: new disk, initializing [ 2153.977111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2154.013608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2155.448427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2155.452148] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2155.491321] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2155.968481] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2159.177977] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2160.532442] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2160.601865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2162.027824] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2162.102941] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2165.729701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2165.731384] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2165.733604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2165.737929] Lustre: Skipped 1 previous similar message [ 2165.740945] Lustre: Skipped 1 previous similar message [ 2168.877945] LustreError: 141091:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2168.938826] Lustre: server umount lustre-OST0000 complete [ 2170.849389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.849961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.854068] Lustre: Skipped 1 previous similar message [ 2175.969495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2175.972112] Lustre: Skipped 2 previous similar messages [ 2176.317168] LustreError: 141292:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2176.319042] LustreError: 141292:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2176.344554] Lustre: server umount lustre-MDT0000 complete [ 2177.596184] LustreError: 138174:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011349 with bad export cookie 5986591063572864292 [ 2177.599499] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2177.600656] LustreError: 138174:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2177.674122] LustreError: 141493:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2177.675852] LustreError: 141493:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2177.758828] Lustre: server umount lustre-MDT0001 complete [ 2180.650355] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.847236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2182.225561] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2185.163693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2185.283171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2186.670842] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2188.572256] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2189.835815] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2192.179263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2194.319362] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2196.288823] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2197.767032] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2207.052412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 2208.395666] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2208.468914] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2213.203116] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2213.244454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2213.354385] Lustre: lustre-OST0001: new disk, initializing [ 2213.357840] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2213.443341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2213.448063] Lustre: Skipped 1 previous similar message [ 2217.236225] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2219.651724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2219.681360] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2219.775457] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2223.864977] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2224.050255] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2227.559486] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2227.734974] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2231.096688] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d3685d21000.ost_server_uuid 50 [ 2232.534254] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d3685d21000.ost_server_uuid in FULL state after 0 sec [ 2240.481064] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2240.484719] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2240.498644] LustreError: Skipped 2 previous similar messages [ 2240.502620] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2240.507702] Lustre: Skipped 1 previous similar message [ 2240.540233] Lustre: Skipped 1 previous similar message [ 2245.435230] LustreError: 146226:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 2245.438993] LustreError: 146226:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2245.604551] LustreError: 143552:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2245.619662] Lustre: server umount lustre-OST0001 complete [ 2245.634560] LustreError: 143552:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2250.723835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2250.728618] LustreError: 143554:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.728918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2250.739504] Lustre: Skipped 1 previous similar message [ 2250.782361] LustreError: 143554:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2255.095502] LustreError: 146427:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2255.115706] LustreError: 146427:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2255.251563] Lustre: server umount lustre-OST0000 complete [ 2260.962725] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.970754] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2260.973399] Lustre: Skipped 1 previous similar message [ 2272.224214] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2272.267235] LustreError: 146629:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 2272.269352] LustreError: 146629:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2272.301326] Lustre: server umount lustre-MDT0000 complete [ 2273.642458] LustreError: 141950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011445 with bad export cookie 5986591063572865160 [ 2273.645352] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2273.646805] LustreError: 141950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2273.830378] Lustre: server umount lustre-MDT0001 complete [ 2282.176925] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2283.282634] Key type lgssc unregistered [ 2283.420363] LNet: 147815:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2284.455208] LNet: Removed LNI 192.168.203.124@tcp [ 2284.777345] Key type .llcrypt unregistered [ 2284.779063] Key type ._llcrypt unregistered [ 2298.070876] Key type ._llcrypt registered [ 2298.072455] Key type .llcrypt registered [ 2298.119371] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2305.379490] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2305.820847] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2305.872512] alg: No test for adler32 (adler32-zlib) [ 2306.747471] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2306.851866] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2306.853862] LNet: Accept secure, port 988 [ 2308.440156] Key type lgssc registered [ 2308.883181] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2312.553292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2315.243046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2317.590795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2319.983247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.556585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2322.575336] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2322.581296] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.672958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2323.686236] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2323.727628] Lustre: lustre-MDT0000: new disk, initializing [ 2323.758290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.767855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2325.244678] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2329.636685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2329.661731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2329.699292] Lustre: 151720:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2329.724230] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2329.726573] Lustre: Skipped 1 previous similar message [ 2329.763517] Lustre: lustre-MDT0001: new disk, initializing [ 2329.798469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2329.812090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2329.815775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2331.193206] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2334.309336] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2335.662951] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2338.379031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2338.420584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2338.555380] Lustre: lustre-OST0000: new disk, initializing [ 2338.559178] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2338.598506] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2340.999750] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2343.422504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2343.427089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2343.456909] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2344.474095] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2346.096271] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2346.191743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2347.643679] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2347.727157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2348.514947] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2348.515567] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2348.516186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2348.518351] LustreError: Skipped 1 previous similar message [ 2348.523122] Lustre: Skipped 1 previous similar message [ 2350.458882] LustreError: 154625:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2350.563322] Lustre: server umount lustre-OST0000 complete [ 2353.634379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2353.637261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2353.642022] Lustre: Skipped 1 previous similar message [ 2353.648604] Lustre: Skipped 2 previous similar messages [ 2358.452393] LustreError: 154826:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2358.465264] LustreError: 154826:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2358.656485] Lustre: server umount lustre-MDT0000 complete [ 2362.095673] LustreError: 151711:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011533 with bad export cookie 1282223733436175732 [ 2362.100966] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.109978] LustreError: 151711:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2362.213455] LustreError: 155027:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2362.216336] LustreError: 155027:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2362.384437] Lustre: server umount lustre-MDT0001 complete [ 2367.864268] Lustre: DEBUG MARKER: == conf-sanity test 47a: server restart does not lose lru_resize ========================================================== 21:25:38 (1754011538) [ 2377.773760] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2381.903714] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2386.090835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2389.054146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2391.633811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2394.170428] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2397.140434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2397.189679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2397.291746] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2397.309974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2397.355494] Lustre: lustre-MDT0000: new disk, initializing [ 2397.411342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2397.420795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2399.118278] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2404.235041] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2404.264644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2404.307423] Lustre: 158000:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2404.335154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2404.337693] Lustre: Skipped 1 previous similar message [ 2404.376309] Lustre: lustre-MDT0001: new disk, initializing [ 2404.417437] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2404.433400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2404.437451] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2406.120127] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2409.893788] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2411.505439] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2414.462342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2414.501038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2414.619187] Lustre: lustre-OST0000: new disk, initializing [ 2414.621700] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2414.655912] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2417.197413] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2420.902429] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2424.846318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2424.851220] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2424.892888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2428.314901] Lustre: Failing over lustre-OST0000 [ 2428.351853] LustreError: 160421:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2428.354968] LustreError: 160421:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2428.455722] Lustre: server umount lustre-OST0000 complete [ 2442.229104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2442.376787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2442.381754] Lustre: Skipped 1 previous similar message [ 2442.392124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2442.401109] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2442.614842] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2444.841294] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2445.473124] Lustre: 149940:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754011601/real 1754011601] req@0000000061f99f99 x1839214339611392/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1754011617 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2446.304207] Lustre: 149942:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754011601/real 1754011601] req@0000000029b2b008 x1839214339611648/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1754011617 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2446.499884] Lustre: Failing over lustre-MDT0000 [ 2446.566553] LustreError: 161386:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2446.568753] LustreError: 161386:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2446.609678] Lustre: server umount lustre-MDT0000 complete [ 2447.733640] LustreError: 160325:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.743664] LustreError: 160325:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2447.841979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2447.848642] Lustre: Skipped 1 previous similar message [ 2450.912154] Lustre: 149943:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754011606/real 1754011606] req@00000000368bf487 x1839214339613312/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1754011622 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2452.838069] LustreError: 158005:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.845508] LustreError: 158005:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2456.032379] Lustre: 149943:0:(client.c:2453:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1754011611/real 1754011611] req@00000000ba56de07 x1839214339614336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1754011627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2457.965208] LustreError: 158005:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2457.974963] LustreError: 158005:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 2460.676871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2460.746786] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2460.756403] Lustre: 158003:0:(mgc_request.c:1876:mgc_process_log()) MGC192.168.203.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2460.869563] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2460.914994] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2462.621759] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2463.082700] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2466.279177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) [ 2466.287694] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2466.315674] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2471.396632] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2471.397462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.398560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.400452] LustreError: Skipped 1 previous similar message [ 2471.413013] Lustre: Skipped 1 previous similar message [ 2473.645068] LustreError: 162262:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2473.650024] LustreError: 162262:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2473.724332] Lustre: server umount lustre-OST0000 complete [ 2476.514994] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2476.515769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2476.524402] Lustre: Skipped 2 previous similar messages [ 2481.342543] Lustre: server umount lustre-MDT0000 complete [ 2481.634143] LustreError: 158011:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.641602] LustreError: 158011:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2483.024412] LustreError: 158775:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011654 with bad export cookie 1282223733436177790 [ 2483.025558] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2483.030485] LustreError: 158775:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2483.210926] Lustre: server umount lustre-MDT0001 complete [ 2488.560606] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2489.883882] Key type lgssc unregistered [ 2490.037356] LNet: 163192:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2491.111031] LNet: Removed LNI 192.168.203.124@tcp [ 2491.470184] Key type .llcrypt unregistered [ 2491.471975] Key type ._llcrypt unregistered [ 2513.027947] Key type ._llcrypt registered [ 2513.029014] Key type .llcrypt registered [ 2513.071858] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2513.463444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2513.491088] alg: No test for adler32 (adler32-zlib) [ 2514.378817] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2514.483592] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2514.485685] LNet: Accept secure, port 988 [ 2516.080291] Key type lgssc registered [ 2516.499240] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2519.639240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2519.649354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.877243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2522.399994] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2525.990429] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2528.136187] LustreError: 164708:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2528.140626] LustreError: 164708:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2528.143326] LustreError: 164708:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2528.214847] LustreError: 165225:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2528.320805] Lustre: server umount lustre-MDT0000 complete [ 2534.084817] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2537.781472] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2541.779767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2544.293533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2546.496538] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2548.933000] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.395979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2551.420749] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2551.510627] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2551.527511] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2551.566825] Lustre: lustre-MDT0000: new disk, initializing [ 2551.612568] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2551.621543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2553.157904] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2557.643228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2557.668135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2557.704098] Lustre: 167848:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2557.733293] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2557.735376] Lustre: Skipped 1 previous similar message [ 2557.777095] Lustre: lustre-MDT0001: new disk, initializing [ 2557.816550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2557.833905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2557.837785] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2559.231119] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2562.534141] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2563.903043] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2566.599502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2566.638663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2566.762900] Lustre: lustre-OST0000: new disk, initializing [ 2566.765938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2566.808943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2569.339196] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2572.804820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2572.808868] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2572.869702] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2573.032833] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2576.161854] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2576.912808] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 21:29:08 (1754011748) [ 2680.288891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.289122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2680.294994] Lustre: Skipped 2 previous similar messages [ 2680.297654] Lustre: Skipped 1 previous similar message [ 2684.796490] LustreError: 171338:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 2684.798919] LustreError: 171338:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2684.862858] Lustre: server umount lustre-MDT0000 complete [ 2685.409453] LustreError: 167858:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2685.416740] LustreError: 167858:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 2686.340119] LustreError: 167840:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011858 with bad export cookie 2812610944131910171 [ 2686.340686] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2686.344768] LustreError: 167840:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2690.528746] LustreError: 171232:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.528908] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.540083] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2690.542817] Lustre: Skipped 1 previous similar message [ 2691.552628] LustreError: 171232:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.552885] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2692.480901] LustreError: 171539:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2692.482921] LustreError: 171539:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2692.580944] Lustre: server umount lustre-MDT0001 complete [ 2700.268941] LustreError: 171740:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 2700.272132] LustreError: 171740:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2700.333630] Lustre: server umount lustre-OST0000 complete [ 2703.780595] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2707.303604] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2711.328796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2714.151961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2716.590049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2719.131905] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2721.938081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2721.965204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2722.059291] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2722.074591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2722.116289] Lustre: lustre-MDT0000: new disk, initializing [ 2722.147429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2722.156704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2723.613671] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2728.192662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2728.221460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2728.259637] Lustre: 174059:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2728.287427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2728.289840] Lustre: Skipped 1 previous similar message [ 2728.332445] Lustre: lustre-MDT0001: new disk, initializing [ 2728.388605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2728.392992] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2729.945731] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2733.577475] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2734.958338] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2737.441036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2737.469089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2737.570956] Lustre: lustre-OST0000: new disk, initializing [ 2737.572887] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2737.603360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2737.606195] Lustre: Skipped 1 previous similar message [ 2739.636259] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2740.100813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2740.105039] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2740.146665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2742.981721] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2744.518397] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2744.614155] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2746.207041] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2746.295550] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2750.434305] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2750.436304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.438351] LustreError: Skipped 1 previous similar message [ 2750.444232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2750.449467] Lustre: Skipped 1 previous similar message [ 2753.071115] LustreError: 176967:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2753.074119] LustreError: 176967:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2753.144177] Lustre: server umount lustre-OST0000 complete [ 2755.554127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2755.555376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2755.559164] Lustre: Skipped 2 previous similar messages [ 2755.562967] Lustre: Skipped 1 previous similar message [ 2760.735412] Lustre: server umount lustre-MDT0000 complete [ 2762.254396] LustreError: 174051:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754011934 with bad export cookie 2812610944132068987 [ 2762.257680] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2762.260357] LustreError: 174051:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2762.338873] LustreError: 177370:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2762.342251] LustreError: 177370:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 2762.471724] Lustre: server umount lustre-MDT0001 complete [ 2765.889415] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:32:17 (1754011937) [ 2771.832882] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2775.139529] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2778.758984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2781.241750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2783.546073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2786.011178] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2788.862184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2788.898826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2789.007046] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2789.027270] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2789.068091] Lustre: lustre-MDT0000: new disk, initializing [ 2789.100308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.109666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2790.648213] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2795.491030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2795.521993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2795.574567] Lustre: 180337:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2795.698851] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2797.277741] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2800.834216] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2802.481080] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2805.178995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2805.210294] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2807.042553] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2807.088742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2807.700727] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2811.140150] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2817.508236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2817.508328] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2817.511024] LustreError: Skipped 1 previous similar message [ 2817.516553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2817.519631] Lustre: Skipped 3 previous similar messages [ 2819.437883] LustreError: 182865:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2819.440342] LustreError: 182865:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 2819.508622] Lustre: server umount lustre-OST0000 complete [ 2821.601632] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2827.165513] Lustre: server umount lustre-MDT0000 complete [ 2827.745048] LustreError: 181123:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2827.751782] LustreError: 181123:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 2828.726641] LustreError: 180330:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012000 with bad export cookie 2812610944132069855 [ 2828.729521] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2828.731111] LustreError: 180330:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2828.895821] Lustre: server umount lustre-MDT0001 complete [ 2832.399690] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:33:23 (1754012003) [ 2838.444576] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 2841.819667] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2845.506785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2848.180101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2850.578418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2853.118777] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2855.883743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2855.913135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2856.026774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2856.043116] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2856.045902] Lustre: Skipped 3 previous similar messages [ 2856.088124] Lustre: lustre-MDT0000: new disk, initializing [ 2856.090088] Lustre: Skipped 2 previous similar messages [ 2856.126236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2856.129073] Lustre: Skipped 2 previous similar messages [ 2856.137205] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2856.141296] Lustre: Skipped 2 previous similar messages [ 2857.767714] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2862.524067] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2862.560233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2862.607133] Lustre: 186239:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2862.643184] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 2862.645827] Lustre: Skipped 3 previous similar messages [ 2862.743145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2864.266682] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2867.621276] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2869.032259] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2871.659618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 2871.685607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2873.947996] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2875.040208] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2877.349103] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2880.484134] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2880.485140] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2880.487454] LustreError: Skipped 1 previous similar message [ 2880.491972] Lustre: Skipped 3 previous similar messages [ 2885.548118] LustreError: 188777:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 2885.550526] LustreError: 188777:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 2885.619254] Lustre: server umount lustre-OST0000 complete [ 2888.673295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2893.143707] Lustre: server umount lustre-MDT0000 complete [ 2894.599922] LustreError: 187993:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012066 with bad export cookie 2812610944132070940 [ 2894.602798] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2894.605101] LustreError: 187993:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2894.760572] Lustre: server umount lustre-MDT0001 complete [ 2899.927500] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2901.151432] Key type lgssc unregistered [ 2901.313715] LNet: 189707:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2902.374420] LNet: Removed LNI 192.168.203.124@tcp [ 2902.739216] Key type .llcrypt unregistered [ 2902.740799] Key type ._llcrypt unregistered [ 2912.435813] Key type ._llcrypt registered [ 2912.437430] Key type .llcrypt registered [ 2912.481676] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2912.771549] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2912.809448] alg: No test for adler32 (adler32-zlib) [ 2913.686359] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2913.789646] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2913.792425] LNet: Accept secure, port 988 [ 2915.376168] Key type lgssc registered [ 2915.773178] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2918.840791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2918.849594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2920.038431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2921.538626] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2924.755692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2924.899222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2926.297199] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2928.239288] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2929.571872] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2932.105268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 2932.242820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2934.346858] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 2936.430184] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2944.548817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2959.843249] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2959.843826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2959.845789] LustreError: Skipped 1 previous similar message [ 2959.852557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2959.856641] Lustre: Skipped 1 previous similar message [ 2964.717171] LustreError: 193333:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 2964.776806] Lustre: server umount lustre-OST0000 complete [ 2970.081373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2970.081843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2970.085954] Lustre: Skipped 2 previous similar messages [ 2972.221512] LustreError: 193534:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 2972.224279] LustreError: 193534:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 2972.251496] Lustre: server umount lustre-MDT0000 complete [ 2973.558144] LustreError: 191063:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012145 with bad export cookie 5540749156422398888 [ 2973.561371] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2973.562035] LustreError: 191063:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2973.625259] LustreError: 193737:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 2973.628063] LustreError: 193737:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 2973.708039] Lustre: server umount lustre-MDT0001 complete [ 2978.463950] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 2979.577322] Key type lgssc unregistered [ 2979.720453] LNet: 194264:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2980.774690] LNet: Removed LNI 192.168.203.124@tcp [ 2981.140838] Key type .llcrypt unregistered [ 2981.142378] Key type ._llcrypt unregistered [ 2991.096207] Key type ._llcrypt registered [ 2991.097548] Key type .llcrypt registered [ 2991.145468] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 2991.472033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2991.486207] alg: No test for adler32 (adler32-zlib) [ 2992.343765] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 2992.435821] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 2992.438106] LNet: Accept secure, port 988 [ 2994.016150] Key type lgssc registered [ 2994.375970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2997.217981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2997.226484] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2998.410394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2999.756902] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3002.737188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3002.859108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3004.187254] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3006.123386] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3007.499067] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3010.071539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3010.210311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.466579] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3014.460882] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3023.529392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3027.937158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3027.940194] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.945469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.961570] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.961875] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.963116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.972164] Lustre: Skipped 1 previous similar message [ 3033.133118] LustreError: 197872:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3033.196480] Lustre: server umount lustre-OST0000 complete [ 3035.221472] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d3698142800.ost_server_uuid 50 [ 3035.868417] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3045.903230] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.903675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3045.907961] Lustre: Skipped 1 previous similar message [ 3045.910744] Lustre: Skipped 1 previous similar message [ 3050.977768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3050.980411] Lustre: Skipped 1 previous similar message [ 3052.030537] LustreError: 198220:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 3052.032415] LustreError: 198220:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3052.061098] Lustre: server umount lustre-MDT0000 complete [ 3053.375482] LustreError: 198222:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012225 with bad export cookie 15808265985369994809 [ 3053.379194] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3053.379528] LustreError: 198222:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3053.451518] LustreError: 198423:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 3053.454184] LustreError: 198423:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3053.547185] Lustre: server umount lustre-MDT0001 complete [ 3056.687693] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:37:08 (1754012228) [ 3059.193962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.369327] LustreError: 199093:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3059.377143] LustreError: 199093:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3059.394587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3060.863072] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3064.042608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3064.202356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3065.613810] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3067.634825] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3068.895509] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3071.217417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3073.303391] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3075.131594] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3077.517138] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3077.549455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3077.615963] Lustre: lustre-OST0001: new disk, initializing [ 3077.619078] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3077.652777] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3077.654905] Lustre: Skipped 1 previous similar message [ 3079.679112] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3082.884271] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3083.788286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3083.791942] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3083.827689] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3084.844488] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3089.889582] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3089.893467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3089.898195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3089.900167] Lustre: Skipped 1 previous similar message [ 3095.086094] LustreError: 202273:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3095.089214] LustreError: 202273:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3095.109300] Lustre: server umount lustre-OST0000 complete [ 3097.237693] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3097.323296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3098.468747] LustreError: 200678:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3098.723966] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3098.806859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3100.132536] LustreError: 200677:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3100.136886] LustreError: 200677:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3103.588919] LustreError: 200679:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3109.345244] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3109.345277] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3109.349078] LustreError: Skipped 1 previous similar message [ 3109.354646] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3109.357372] Lustre: Skipped 5 previous similar messages [ 3110.370531] LustreError: 200679:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3110.377092] LustreError: 200679:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3114.989101] LustreError: 202963:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3114.991847] LustreError: 202963:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3115.055186] Lustre: server umount lustre-OST0001 complete [ 3120.609425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3120.610098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.618052] Lustre: Skipped 2 previous similar messages [ 3122.457930] Lustre: server umount lustre-MDT0000 complete [ 3123.759296] LustreError: 199073:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012295 with bad export cookie 15808265985369995866 [ 3123.762982] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3123.764707] LustreError: 199073:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3123.834588] LustreError: 203365:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3123.838840] LustreError: 203365:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 3123.935882] Lustre: server umount lustre-MDT0001 complete [ 3125.893780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3127.267807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3128.589566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3129.908213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3132.146638] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:38:23 (1754012303) [ 3134.314491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3134.402446] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3134.413092] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3134.480605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.724620] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3138.485739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3138.516977] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3138.534841] Lustre: 204855:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3139.898379] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3141.624296] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3142.768752] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3144.977352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3145.041955] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3146.941431] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3147.467680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3148.642953] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3150.788194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3150.813297] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3152.706598] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3154.492606] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3157.985139] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3157.985286] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3157.987367] LustreError: Skipped 1 previous similar message [ 3157.991884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3157.995029] Lustre: Skipped 5 previous similar messages [ 3161.390099] LustreError: 208024:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3161.392960] LustreError: 208024:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 3161.414038] Lustre: server umount lustre-OST0000 complete [ 3164.642421] LustreError: 207937:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.646403] LustreError: 207937:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3174.884584] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3174.887475] LustreError: Skipped 1 previous similar message [ 3178.163230] Lustre: server umount lustre-OST0001 complete [ 3185.497616] Lustre: server umount lustre-MDT0000 complete [ 3186.744085] LustreError: 204845:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012358 with bad export cookie 15808265985369997252 [ 3186.746389] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3186.748284] LustreError: 204845:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3186.894448] Lustre: server umount lustre-MDT0001 complete [ 3188.647123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3189.858429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3191.048997] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3192.249472] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3194.326626] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:39:25 (1754012365) [ 3199.180283] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3201.847979] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3205.025656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3207.188192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3209.060656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3211.132631] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.242375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3213.259287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3213.333876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3213.345955] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3213.378121] Lustre: lustre-MDT0000: new disk, initializing [ 3213.399391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.401047] Lustre: Skipped 3 previous similar messages [ 3213.406023] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3214.658939] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3218.771539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3218.796136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3218.830151] Lustre: 212429:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3218.852181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3218.853987] Lustre: Skipped 1 previous similar message [ 3218.889853] Lustre: lustre-MDT0001: new disk, initializing [ 3218.937691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3218.941822] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3220.281343] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3223.510426] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3224.759607] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3227.181830] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3227.207901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3227.307893] Lustre: lustre-OST0000: new disk, initializing [ 3227.310237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3228.598837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3228.602364] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3228.639284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3229.268683] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3232.515123] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3233.933867] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3234.032315] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3235.344100] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3235.431381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3238.125078] LustreError: 215333:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3238.127560] LustreError: 215333:0:(obd_class.h:479:obd_check_dev()) Skipped 25 previous similar messages [ 3238.192372] Lustre: server umount lustre-OST0000 complete [ 3244.001623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3244.002464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3244.005074] Lustre: Skipped 6 previous similar messages [ 3244.007279] Lustre: Skipped 11 previous similar messages [ 3250.171426] Lustre: server umount lustre-MDT0000 complete [ 3251.417204] LustreError: 215334:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012423 with bad export cookie 15808265985369998799 [ 3251.420394] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3251.421291] LustreError: 215334:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3262.001974] Lustre: server umount lustre-MDT0001 complete [ 3264.873787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3265.025795] LustreError: 216213:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.031478] LustreError: 216213:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 6 previous similar messages [ 3266.398031] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3269.311425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3270.708422] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3272.509195] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3273.756803] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3276.057661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3277.992303] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3279.759524] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3281.117407] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3290.399363] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec [ 3291.712890] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3291.789339] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3294.689625] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3294.693019] LustreError: Skipped 2 previous similar messages [ 3298.597512] Lustre: server umount lustre-OST0000 complete [ 3300.360068] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3301.549872] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3325.253279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3325.371271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3325.373284] Lustre: Skipped 5 previous similar messages [ 3327.308148] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3329.045821] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3329.613142] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3332.588884] LustreError: 220598:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3332.591801] LustreError: 220598:0:(obd_class.h:479:obd_check_dev()) Skipped 21 previous similar messages [ 3332.651823] Lustre: server umount lustre-OST0000 complete [ 3334.946457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.950308] Lustre: Skipped 2 previous similar messages [ 3334.952022] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3334.954009] Lustre: Skipped 6 previous similar messages [ 3340.054173] Lustre: server umount lustre-MDT0000 complete [ 3340.257965] LustreError: 216213:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3340.263332] LustreError: 216213:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3341.301386] LustreError: 216192:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012513 with bad export cookie 15808265985369999667 [ 3341.304987] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3341.305263] LustreError: 216192:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3341.451913] Lustre: server umount lustre-MDT0001 complete [ 3344.251606] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:41:55 (1754012515) [ 3346.409757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.933911] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3350.791872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3352.187199] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3353.965226] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3355.074139] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3357.204217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3359.077537] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3360.775141] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3362.042084] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3372.311415] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec [ 3373.519254] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3373.596175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3375.750673] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3375.780550] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3375.842773] Lustre: lustre-OST0001: new disk, initializing [ 3375.845595] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3377.859679] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3380.909427] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3382.175423] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3383.805079] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3383.809178] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3383.835339] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3384.287819] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3385.553712] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3385.625089] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3388.900768] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3388.903779] LustreError: Skipped 1 previous similar message [ 3392.314681] Lustre: server umount lustre-OST0001 complete [ 3394.262145] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3394.342657] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3395.626078] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3395.694173] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3404.258673] LustreError: 223259:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3404.264083] LustreError: 223259:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 9 previous similar messages [ 3419.214564] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3421.162461] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3422.945766] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3423.466713] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3430.198171] Lustre: server umount lustre-OST0001 complete [ 3448.361950] Lustre: server umount lustre-OST0000 complete [ 3455.758137] Lustre: server umount lustre-MDT0000 complete [ 3456.933417] LustreError: 221653:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012628 with bad export cookie 15808265985370000584 [ 3456.938051] LustreError: 221653:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3456.938475] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3471.328169] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3471.365300] LustreError: 228165:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup [ 3471.367883] LustreError: 228165:0:(obd_class.h:479:obd_check_dev()) Skipped 37 previous similar messages [ 3471.447201] Lustre: server umount lustre-MDT0001 complete [ 3473.430556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3474.774327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3476.076501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3477.362198] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3479.578321] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:44:10 (1754012650) [ 3481.780775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.886084] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3481.898058] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3481.976107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3481.978637] Lustre: Skipped 5 previous similar messages [ 3483.323040] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3486.062928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3486.092208] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3486.101683] Lustre: 229658:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3487.427181] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3489.186735] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3490.392862] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3492.644454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3492.707894] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3494.624202] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3496.303473] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3505.668700] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3505.693289] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3507.727211] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3509.539115] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3510.074996] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3511.329892] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3511.409117] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3512.681489] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3512.763769] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3513.974507] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d36871e5800.ost_server_uuid 50 [ 3514.431733] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d36871e5800.ost_server_uuid in FULL state after 0 sec [ 3514.765137] Lustre: Permanently deactivating lustre-OST0001 [ 3514.769703] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3515.536240] Lustre: Permanently reactivating lustre-OST0001 [ 3521.508463] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3521.509859] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3521.511637] LustreError: Skipped 3 previous similar messages [ 3521.514691] Lustre: Skipped 7 previous similar messages [ 3521.515239] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3521.519945] Lustre: Skipped 16 previous similar messages [ 3522.552962] Lustre: server umount lustre-OST0001 complete [ 3536.872184] LustreError: 231825:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.872184] LustreError: 231269:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.872230] LustreError: 231825:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 17 previous similar messages [ 3536.878716] LustreError: 231269:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 17 previous similar messages [ 3537.888101] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3537.957549] Lustre: server umount lustre-OST0000 complete [ 3545.299645] Lustre: server umount lustre-MDT0000 complete [ 3546.564133] LustreError: 229650:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012718 with bad export cookie 15808265985370001844 [ 3546.567072] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3546.569246] LustreError: 229650:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3546.713042] Lustre: server umount lustre-MDT0001 complete [ 3548.606317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3549.850431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3551.076414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3552.316994] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3554.404279] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:45:25 (1754012725) [ 3554.731027] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3556.892198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3556.989683] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3557.002317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3557.005466] Lustre: Skipped 5 previous similar messages [ 3558.302853] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3561.028989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3561.059595] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3561.069051] Lustre: 235760:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3562.439274] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3564.245183] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3565.418968] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3567.647616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3567.669464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3567.748660] Lustre: Permanently deactivating lustre-OST0000 [ 3567.754167] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3568.281761] Lustre: lustre-OST0000_UUID: llog_process_th(pid=237477/ppid=2) setting import INACTIVE by administrator request [ 3569.694269] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3571.341592] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3573.448315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3575.067920] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3575.257637] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3576.960302] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3577.504423] Lustre: Permanently reactivating lustre-OST0000 [ 3582.455721] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3582.458022] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3582.461151] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.124@tcp (at 0@lo) [ 3582.476155] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3582.483394] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3593.406757] Lustre: server umount lustre-OST0001 complete [ 3609.056107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3609.124767] Lustre: server umount lustre-OST0000 complete [ 3616.467546] Lustre: server umount lustre-MDT0000 complete [ 3617.619353] LustreError: 235751:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012789 with bad export cookie 15808265985370003496 [ 3617.622663] LustreError: 235751:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3617.622909] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3617.759984] Lustre: server umount lustre-MDT0001 complete [ 3622.059988] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3623.138509] Key type lgssc unregistered [ 3623.282407] LNet: 240122:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3624.294359] LNet: Removed LNI 192.168.203.124@tcp [ 3624.588788] Key type .llcrypt unregistered [ 3624.590789] Key type ._llcrypt unregistered [ 3632.684399] Key type ._llcrypt registered [ 3632.685972] Key type .llcrypt registered [ 3632.722145] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3633.046860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3633.084210] alg: No test for adler32 (adler32-zlib) [ 3633.934420] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 3634.015253] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3634.017118] LNet: Accept secure, port 988 [ 3635.592173] Key type lgssc registered [ 3635.959949] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3637.075180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3639.161485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3639.167176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3640.315478] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.523342] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3644.253384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3644.276368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3644.306166] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3644.314398] Lustre: 241525:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3644.317902] Lustre: Permanently deactivating lustre-MDT0001 [ 3644.322729] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3644.423371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3645.638540] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3647.434525] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3648.750979] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3650.029454] Lustre: lustre-MDT0001_UUID: llog_process_th(pid=242969/ppid=2) setting import INACTIVE by administrator request [ 3651.048451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3651.157085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3652.992504] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3654.241146] LustreError: 243145:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3654.245095] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3654.247540] LustreError: 243145:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3654.858921] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3657.208462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3657.276901] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3659.162818] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3661.055427] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3662.311146] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3665.383530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3665.383544] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3666.719109] Lustre: Permanently reactivating lustre-MDT0001 [ 3666.725766] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3666.728300] Lustre: Skipped 2 previous similar messages [ 3670.889710] Lustre: lustre-MDT0001: Client fa1e1a45-116d-4493-a249-a9dbb9d8994b (at 192.168.203.24@tcp) reconnecting [ 3673.073317] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3673.608701] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3673.613736] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 3673.616761] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 3673.621689] LustreError: 244758:0:(llog_osd.c:2254:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 3673.622262] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 192.168.203.124@tcp (at 0@lo) [ 3673.625157] LustreError: 244758:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 3673.631574] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3673.632093] LustreError: 244758:0:(obd_config.c:1993:class_config_llog_handler()) MGC192.168.203.124@tcp: cfg command failed: rc = -108 [ 3673.639617] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 3673.639617] [ 3673.642561] LustreError: 241527:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 3675.337106] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3675.408071] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3676.721279] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3676.800385] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3677.606191] Lustre: Permanently deactivating lustre-MDT0001 [ 3677.612458] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3677.614964] Lustre: Skipped 2 previous similar messages [ 3682.816597] Lustre: lustre-MDT0001_UUID: llog_process_th(pid=245476/ppid=2) setting import INACTIVE by administrator request [ 3688.930318] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3688.932330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.935569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3690.977331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3690.978126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3690.979463] Lustre: Skipped 1 previous similar message [ 3690.982747] Lustre: Skipped 2 previous similar messages [ 3693.627985] LustreError: 245611:0:(obd_class.h:479:obd_check_dev()) Device 10 not setup [ 3693.651602] Lustre: server umount lustre-MDT0000 complete [ 3694.839921] LustreError: 241515:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012866 with bad export cookie 11703925895472908965 [ 3694.843744] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3694.846021] LustreError: 241515:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3694.907029] LustreError: 245813:0:(obd_class.h:479:obd_check_dev()) Device 18 not setup [ 3694.909562] LustreError: 245813:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3694.985419] Lustre: server umount lustre-MDT0001 complete [ 3702.252118] LustreError: 246013:0:(obd_class.h:479:obd_check_dev()) Device 23 not setup [ 3702.254520] LustreError: 246013:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3702.270945] Lustre: server umount lustre-OST0000 complete [ 3705.774508] Lustre: DEBUG MARKER: == conf-sanity test 50j: deactivate MDC for MDT0 ========= 21:47:57 (1754012877) [ 3706.298246] Lustre: DEBUG MARKER: conf-sanity test_50j: @@@@@@ FAIL: failed to block deactivating MDC0 [ 3708.925606] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:48:00 (1754012880) [ 3712.351592] LustreError: 247262:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 3712.353619] LustreError: 247262:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 3712.412804] Lustre: server umount lustre-OST0001 complete [ 3714.257162] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3717.026757] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3720.206636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3722.291380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3724.131832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3726.186526] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3728.280729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3728.308126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3728.382708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3728.385430] Lustre: Skipped 2 previous similar messages [ 3728.393364] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3728.426054] Lustre: lustre-MDT0000: new disk, initializing [ 3728.448266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3728.453954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3729.711801] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3733.657624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3733.680791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3733.711629] Lustre: 249428:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3733.730421] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3733.732047] Lustre: Skipped 1 previous similar message [ 3733.763517] Lustre: lustre-MDT0001: new disk, initializing [ 3733.803735] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3733.807017] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3734.968260] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3737.985832] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3739.054365] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3741.218903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3741.243469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3741.322977] Lustre: lustre-OST0000: new disk, initializing [ 3741.325779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3741.352875] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.354411] Lustre: Skipped 1 previous similar message [ 3743.047466] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3745.940664] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3746.294592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3746.297746] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3746.326843] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3751.764898] LustreError: 249433:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3756.056925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3756.082954] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3756.155406] Lustre: lustre-OST0001: new disk, initializing [ 3756.157873] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3758.259960] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3761.362544] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3761.669583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3761.673539] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3761.696091] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3761.856059] LustreError: 249433:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3761.862904] LustreError: 249433:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3761.873248] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3771.952124] LustreError: 249433:0:(mdt_handler.c:3249:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3773.290530] Lustre: lustre-OST0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 3777.000299] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3777.000673] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3777.003891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3777.008542] Lustre: Skipped 1 previous similar message [ 3777.012833] Lustre: Skipped 1 previous similar message [ 3778.795969] LustreError: 252909:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 3778.798466] LustreError: 252909:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3778.818771] Lustre: server umount lustre-OST0001 complete [ 3782.116045] LustreError: 251143:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3782.120228] LustreError: 251143:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3787.235518] LustreError: 251143:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3787.239557] LustreError: 251143:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3792.358105] LustreError: 251143:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3792.358675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3792.358737] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3792.360318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3792.360327] Lustre: Skipped 1 previous similar message [ 3792.362744] LustreError: 251143:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3792.365347] LustreError: Skipped 2 previous similar messages [ 3797.476573] LustreError: 251144:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3797.480928] LustreError: 251144:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 3802.595483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3802.597298] Lustre: Skipped 4 previous similar messages [ 3805.152125] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3805.173520] LustreError: 253133:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3805.176163] LustreError: 253133:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3805.246232] Lustre: server umount lustre-OST0000 complete [ 3807.713989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3807.718479] Lustre: Skipped 2 previous similar messages [ 3812.827809] Lustre: server umount lustre-MDT0000 complete [ 3812.834129] LustreError: 250216:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3812.840135] LustreError: 250216:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 3814.012138] LustreError: 249419:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754012985 with bad export cookie 11703925895472911779 [ 3814.014740] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3814.016912] LustreError: 249419:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3820.231910] Lustre: server umount lustre-MDT0001 complete [ 3825.061105] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 3826.158214] Key type lgssc unregistered [ 3826.282327] LNet: 254063:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3827.302367] LNet: Removed LNI 192.168.203.124@tcp [ 3827.609403] Key type .llcrypt unregistered [ 3827.610723] Key type ._llcrypt unregistered [ 3838.958889] Key type ._llcrypt registered [ 3838.959840] Key type .llcrypt registered [ 3838.992268] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 3845.153080] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 3845.611642] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3845.636175] alg: No test for adler32 (adler32-zlib) [ 3846.496771] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 3846.572900] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 3846.574478] LNet: Accept secure, port 988 [ 3848.152224] Key type lgssc registered [ 3848.467490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3851.281281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3853.514542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3855.534857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3857.782455] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3860.163991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3860.183181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3860.188762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3861.267505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3861.278392] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3861.317125] Lustre: lustre-MDT0000: new disk, initializing [ 3861.338883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3861.346039] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3862.520580] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3866.458466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3866.474224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3866.507443] Lustre: 257970:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3866.546286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3866.548489] Lustre: Skipped 1 previous similar message [ 3866.586083] Lustre: lustre-MDT0001: new disk, initializing [ 3866.613654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3866.625490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3866.628297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3867.854298] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3870.819222] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3871.957599] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3874.101737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 3874.128970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3874.239104] Lustre: lustre-OST0000: new disk, initializing [ 3874.241738] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3874.273157] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3876.075395] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3879.060406] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3880.270683] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3881.462435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3881.466280] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3881.495993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3882.378767] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 3883.633783] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3883.708144] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3886.561965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3886.562112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.565195] LustreError: Skipped 1 previous similar message [ 3886.565843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.570543] Lustre: Skipped 1 previous similar message [ 3886.575326] Lustre: Skipped 1 previous similar message [ 3890.413180] LustreError: 260892:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 3890.480331] Lustre: server umount lustre-OST0000 complete [ 3891.681458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3891.681910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3891.686679] Lustre: Skipped 1 previous similar message [ 3896.801308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3896.804158] Lustre: Skipped 2 previous similar messages [ 3897.728120] LustreError: 261094:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 3897.731122] LustreError: 261094:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 3897.761435] Lustre: server umount lustre-MDT0000 complete [ 3898.976329] LustreError: 258744:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013070 with bad export cookie 9968839895856464079 [ 3898.979194] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3898.979804] LustreError: 258744:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3899.035635] LustreError: 261295:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3899.037140] LustreError: 261295:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 3899.118596] Lustre: server umount lustre-MDT0001 complete [ 3901.870485] Lustre: DEBUG MARKER: == conf-sanity test 52: check recovering objects from lost+found ========================================================== 21:51:13 (1754013073) [ 3903.968525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3904.100606] LustreError: 261964:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3904.107462] LustreError: 261964:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 3904.123956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3905.347730] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3907.921128] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3909.189318] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3910.891101] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3912.172219] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3914.473679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3914.594355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3914.596170] Lustre: Skipped 1 previous similar message [ 3916.614616] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3918.445325] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3968.992845] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3968.995077] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3968.998723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3970.019475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3973.038217] LustreError: 264292:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 3973.041665] LustreError: 264292:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 3973.112907] Lustre: server umount lustre-OST0000 complete [ 3974.654083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3978.610220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 3978.724515] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3980.606994] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 3982.378416] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3989.991491] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3989.997071] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.203.124@tcp (at 0@lo) [ 3995.115881] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3995.120572] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.203.124@tcp (at 0@lo) [ 4020.706438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4020.707335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4020.709233] LustreError: Skipped 1 previous similar message [ 4020.712580] Lustre: Skipped 1 previous similar message [ 4020.713493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4020.717410] Lustre: Skipped 3 previous similar messages [ 4021.747063] LustreError: 265945:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4021.749437] LustreError: 265945:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4021.822313] Lustre: server umount lustre-OST0000 complete [ 4025.826611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4025.826677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4025.833893] Lustre: Skipped 2 previous similar messages [ 4029.135977] Lustre: server umount lustre-MDT0000 complete [ 4030.295093] LustreError: 261990:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013202 with bad export cookie 9968839895856464961 [ 4030.297624] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4030.300361] LustreError: 261990:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4030.365804] LustreError: 266348:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4030.368803] LustreError: 266348:0:(obd_class.h:479:obd_check_dev()) Skipped 9 previous similar messages [ 4030.445815] Lustre: server umount lustre-MDT0001 complete [ 4034.798146] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4035.863092] Key type lgssc unregistered [ 4036.009454] LNet: 266873:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4037.030303] LNet: Removed LNI 192.168.203.124@tcp [ 4037.306463] Key type .llcrypt unregistered [ 4037.307373] Key type ._llcrypt unregistered [ 4045.646610] Key type ._llcrypt registered [ 4045.648299] Key type .llcrypt registered [ 4045.684367] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4045.924565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4045.937064] alg: No test for adler32 (adler32-zlib) [ 4046.771588] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 4046.848546] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4046.850825] LNet: Accept secure, port 988 [ 4048.432132] Key type lgssc registered [ 4048.745081] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4051.238680] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4051.245399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.385211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.638197] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4056.303027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4056.402099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4057.632227] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4059.402184] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4060.605891] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4062.821978] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4062.940493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.723289] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4066.430340] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4077.285738] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4092.899777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4092.901119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4092.902337] LustreError: Skipped 1 previous similar message [ 4092.909450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4092.912373] Lustre: Skipped 1 previous similar message [ 4095.084123] LustreError: 271258:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4095.158648] Lustre: server umount lustre-OST0000 complete [ 4097.504983] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4097.507112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4097.510295] Lustre: Skipped 1 previous similar message [ 4097.511972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4102.529480] LustreError: 271461:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4102.531873] LustreError: 271461:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4102.558332] Lustre: server umount lustre-MDT0000 complete [ 4103.137046] LustreError: 269618:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4103.144085] LustreError: 269618:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4103.787106] LustreError: 268260:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013275 with bad export cookie 14908670217006112266 [ 4103.790022] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4103.790535] LustreError: 268260:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4103.855471] LustreError: 271664:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4103.857157] LustreError: 271664:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4103.945018] Lustre: server umount lustre-MDT0001 complete [ 4108.682906] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4109.780426] Key type lgssc unregistered [ 4109.918499] LNet: 272190:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4110.950347] LNet: Removed LNI 192.168.203.124@tcp [ 4111.285736] Key type .llcrypt unregistered [ 4111.286691] Key type ._llcrypt unregistered [ 4118.270884] Key type ._llcrypt registered [ 4118.271716] Key type .llcrypt registered [ 4118.308654] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4118.608079] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4118.614422] alg: No test for adler32 (adler32-zlib) [ 4119.458868] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 4119.535148] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4119.537371] LNet: Accept secure, port 988 [ 4121.120173] Key type lgssc registered [ 4121.454622] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4124.004610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4124.011299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4125.178930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4126.373872] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4129.043792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4129.153152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4130.455122] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4132.165377] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4133.337139] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4135.397052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4135.508808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4137.318233] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4138.997946] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4149.859868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4159.456959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4159.459378] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4159.463149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4160.483257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4160.484934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4160.488617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4164.399957] LustreError: 275753:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4164.478340] Lustre: server umount lustre-OST0000 complete [ 4170.209088] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4170.211138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4170.214189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4170.216116] Lustre: Skipped 1 previous similar message [ 4172.026568] LustreError: 275955:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4172.028215] LustreError: 275955:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4172.054593] Lustre: server umount lustre-MDT0000 complete [ 4173.323159] LustreError: 273338:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013345 with bad export cookie 1440656285262126434 [ 4173.326809] LustreError: 273338:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4173.326938] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4173.387461] LustreError: 276156:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4173.389416] LustreError: 276156:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4173.469984] Lustre: server umount lustre-MDT0001 complete [ 4178.025519] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4179.168250] Key type lgssc unregistered [ 4179.301448] LNet: 276683:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4180.326395] LNet: Removed LNI 192.168.203.124@tcp [ 4180.629210] Key type .llcrypt unregistered [ 4180.630521] Key type ._llcrypt unregistered [ 4189.502291] Key type ._llcrypt registered [ 4189.503205] Key type .llcrypt registered [ 4189.544849] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4189.943691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4189.961385] alg: No test for adler32 (adler32-zlib) [ 4190.819663] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 4190.901765] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4190.904599] LNet: Accept secure, port 988 [ 4192.480164] Key type lgssc registered [ 4192.894290] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4195.636116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4195.644029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4196.803966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4198.242851] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4201.232677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4201.367547] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4202.720863] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4204.461857] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4205.585679] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4207.818442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4207.939725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4209.870062] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4211.535239] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4221.287332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4231.650122] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4231.650870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4231.652728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4231.656502] Lustre: Skipped 1 previous similar message [ 4234.863063] LustreError: 281066:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4234.923900] Lustre: server umount lustre-OST0000 complete [ 4236.769790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4236.770367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4236.774364] Lustre: Skipped 1 previous similar message [ 4236.776431] Lustre: Skipped 1 previous similar message [ 4241.889377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4241.891947] Lustre: Skipped 1 previous similar message [ 4242.237273] LustreError: 281268:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4242.239200] LustreError: 281268:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4242.261546] Lustre: server umount lustre-MDT0000 complete [ 4243.481986] LustreError: 280980:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013415 with bad export cookie 5345844171123856836 [ 4243.485282] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4243.485628] LustreError: 280980:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4243.542146] LustreError: 281470:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4243.545471] LustreError: 281470:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4243.624536] Lustre: server umount lustre-MDT0001 complete [ 4248.074542] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4249.148374] Key type lgssc unregistered [ 4249.283502] LNet: 281996:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4250.342487] LNet: Removed LNI 192.168.203.124@tcp [ 4250.635934] Key type .llcrypt unregistered [ 4250.637096] Key type ._llcrypt unregistered [ 4257.447426] Key type ._llcrypt registered [ 4257.448541] Key type .llcrypt registered [ 4257.487701] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4257.818391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4257.825926] alg: No test for adler32 (adler32-zlib) [ 4258.681692] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 4258.778268] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4258.780575] LNet: Accept secure, port 988 [ 4260.368162] Key type lgssc registered [ 4260.698837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4263.212556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4263.218199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.372261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4265.609228] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4268.305534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4268.429216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4269.701786] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4271.499638] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4272.714079] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4274.892531] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4275.024092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4276.885273] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4278.631221] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4289.383539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4299.746181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.746275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4299.747512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4299.751209] Lustre: Skipped 1 previous similar message [ 4304.052104] LustreError: 285602:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4304.128977] Lustre: server umount lustre-OST0000 complete [ 4309.472863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4309.476502] LustreError: Skipped 1 previous similar message [ 4309.478943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4309.486017] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4309.488242] Lustre: Skipped 1 previous similar message [ 4311.484472] LustreError: 285804:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4311.487367] LustreError: 285804:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4311.516454] Lustre: server umount lustre-MDT0000 complete [ 4312.737976] LustreError: 283154:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013484 with bad export cookie 11119298399430438936 [ 4312.741378] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4312.741902] LustreError: 283154:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4312.812303] LustreError: 286006:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4312.814471] LustreError: 286006:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4312.906596] Lustre: server umount lustre-MDT0001 complete [ 4317.373280] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4318.418841] Key type lgssc unregistered [ 4318.572397] LNet: 286533:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4319.590295] LNet: Removed LNI 192.168.203.124@tcp [ 4319.891586] Key type .llcrypt unregistered [ 4319.892912] Key type ._llcrypt unregistered [ 4323.371927] Key type ._llcrypt registered [ 4323.373019] Key type .llcrypt registered [ 4323.412849] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4328.598698] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 4334.286947] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4334.548756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4334.556284] alg: No test for adler32 (adler32-zlib) [ 4335.405433] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 4335.479773] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4335.481759] LNet: Accept secure, port 988 [ 4337.064158] Key type lgssc registered [ 4337.396898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4340.163522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4342.124936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4344.002273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4345.953931] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4348.137551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4348.155116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4348.160169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4349.243498] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4349.257120] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4349.296295] Lustre: lustre-MDT0000: new disk, initializing [ 4349.322961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4349.330096] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4350.575504] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4355.044526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4355.072284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4355.107211] Lustre: 289991:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4355.130114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4355.131916] Lustre: Skipped 1 previous similar message [ 4355.173129] Lustre: lustre-MDT0001: new disk, initializing [ 4355.212241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4355.226891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4355.231456] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4356.574787] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4359.726857] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4360.915724] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4363.145930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4363.174762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4363.276633] Lustre: lustre-OST0000: new disk, initializing [ 4363.278932] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4363.311273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.196395] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4367.356505] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4367.361988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4367.397053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4368.424716] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4369.709046] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4369.785977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4371.134308] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4371.214819] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4372.450082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4372.452220] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.454039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4372.455869] Lustre: Skipped 1 previous similar message [ 4373.870104] LustreError: 292896:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4373.939413] Lustre: server umount lustre-OST0000 complete [ 4376.032660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4376.035322] LustreError: Skipped 1 previous similar message [ 4376.036675] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4376.041324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4376.043187] Lustre: Skipped 1 previous similar message [ 4377.569958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4377.570937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4381.311207] LustreError: 293097:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4381.313303] LustreError: 293097:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4381.343750] Lustre: server umount lustre-MDT0000 complete [ 4382.643158] LustreError: 289983:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013554 with bad export cookie 13655401666002060156 [ 4382.645837] LustreError: 289983:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4382.645896] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4382.706606] LustreError: 293299:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4382.709049] LustreError: 293299:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4382.801653] Lustre: server umount lustre-MDT0001 complete [ 4385.578732] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:59:17 (1754013557) [ 4387.682667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.854886] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4389.144737] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4392.024412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4392.152568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4393.574448] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4395.483116] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4396.775719] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4399.135843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4401.187598] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4403.070803] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4416.992773] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4416.995601] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4417.001040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4417.002897] Lustre: Skipped 1 previous similar message [ 4422.704082] LustreError: 296225:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4422.705981] LustreError: 296225:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4422.773768] Lustre: server umount lustre-OST0000 complete [ 4426.723907] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 4426.726862] LustreError: Skipped 1 previous similar message [ 4426.728672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4426.733028] Lustre: Skipped 1 previous similar message [ 4426.735210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4426.736820] Lustre: Skipped 2 previous similar messages [ 4430.208254] LustreError: 296426:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4430.209978] LustreError: 296426:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4430.237940] Lustre: server umount lustre-MDT0000 complete [ 4431.616188] LustreError: 293950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013603 with bad export cookie 13655401666002061024 [ 4431.617511] LustreError: 294686:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4431.619261] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4431.621301] LustreError: 293950:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4431.794479] Lustre: server umount lustre-MDT0001 complete [ 4436.624843] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4437.716633] Key type lgssc unregistered [ 4437.858863] LNet: 297154:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4438.886333] LNet: Removed LNI 192.168.203.124@tcp [ 4439.205190] Key type .llcrypt unregistered [ 4439.206595] Key type ._llcrypt unregistered [ 4448.135161] Key type ._llcrypt registered [ 4448.136209] Key type .llcrypt registered [ 4448.173657] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 4454.149247] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4454.516484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4454.529292] alg: No test for adler32 (adler32-zlib) [ 4455.391437] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 4455.478486] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4455.479723] LNet: Accept secure, port 988 [ 4457.064106] Key type lgssc registered [ 4457.419780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4460.316481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4462.457078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4464.423247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4466.488900] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4468.193410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4469.957484] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4472.198255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4472.215731] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4472.221648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4473.306670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4473.317503] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4473.357473] Lustre: lustre-MDT0000: new disk, initializing [ 4473.382349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4473.389729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4474.608424] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4478.482877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4478.511833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4478.545723] Lustre: 300912:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4478.569652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4478.571028] Lustre: Skipped 1 previous similar message [ 4478.604481] Lustre: lustre-MDT0001: new disk, initializing [ 4478.636907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4478.649907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4478.653241] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4479.857094] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4482.755087] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4483.922500] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4486.117673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4486.144971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4486.258268] Lustre: lustre-OST2710: new disk, initializing [ 4486.260813] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4486.294880] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4488.114106] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4489.082965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4489.087157] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4489.125131] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4491.093620] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4493.274454] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4493.305477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4493.369780] Lustre: lustre-OST03e8: new disk, initializing [ 4493.371528] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4493.408759] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4495.264345] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4497.011541] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4498.962100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4498.964918] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4499.009107] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4499.621691] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4499.698255] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4500.975775] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4501.056982] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4502.334924] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4502.415016] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4503.687196] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4503.762899] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4509.157580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4509.159957] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4509.162864] Lustre: Skipped 2 previous similar messages [ 4509.165548] Lustre: Skipped 1 previous similar message [ 4510.851120] LustreError: 305219:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4510.894045] Lustre: server umount lustre-MDT0000 complete [ 4512.176386] LustreError: 300903:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013683 with bad export cookie 11359276081063737833 [ 4512.179926] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4512.180675] LustreError: 300903:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4512.248370] LustreError: 305420:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4512.250888] LustreError: 305420:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4512.336797] Lustre: server umount lustre-MDT0001 complete [ 4523.442073] LustreError: 305620:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4523.444821] LustreError: 305620:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4523.464572] Lustre: server umount lustre-OST2710 complete [ 4534.897855] LustreError: 305822:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 4534.900209] LustreError: 305822:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4534.983751] Lustre: server umount lustre-OST03e8 complete [ 4540.661227] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 4543.663383] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4546.795835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4548.884529] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4550.764985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4552.833420] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4554.999603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4555.022438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4555.110946] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4555.120434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4555.153578] Lustre: lustre-MDT0000: new disk, initializing [ 4555.176819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.184558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4556.428598] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4560.445884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4560.468077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4560.501996] Lustre: 308597:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4560.597538] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4561.888215] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4564.972239] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4566.166695] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4568.413287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4568.441464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4568.559148] Lustre: lustre-OST0000: new disk, initializing [ 4568.560913] Lustre: Skipped 1 previous similar message [ 4568.563320] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4568.565856] Lustre: Skipped 2 previous similar messages [ 4568.592746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4568.594974] Lustre: Skipped 1 previous similar message [ 4570.408220] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4570.679630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4570.683375] Lustre: Skipped 1 previous similar message [ 4570.687176] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4570.718747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4573.467492] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4574.751908] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4574.818421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4576.103985] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4576.180293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4578.862102] LustreError: 311502:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4578.864680] LustreError: 311502:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4578.923591] Lustre: server umount lustre-OST0000 complete [ 4580.833448] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.833945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4580.836518] Lustre: Skipped 2 previous similar messages [ 4580.839848] Lustre: Skipped 1 previous similar message [ 4585.953308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4585.955044] Lustre: Skipped 1 previous similar message [ 4590.051549] LustreError: 311702:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4590.054470] LustreError: 311702:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4590.079410] Lustre: server umount lustre-MDT0000 complete [ 4591.313472] LustreError: 308589:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013763 with bad export cookie 11359276081063739492 [ 4591.318058] LustreError: 308589:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4591.318144] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4601.903175] Lustre: server umount lustre-MDT0001 complete [ 4604.594207] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:02:56 (1754013776) [ 4605.056379] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4605.574271] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:02:57 (1754013777) [ 4606.425695] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4608.790498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4610.014257] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4611.179836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.345817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4612.672901] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4614.761243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4614.855440] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4614.861964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4614.920669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4616.119315] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4618.747196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4618.777983] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4618.786757] Lustre: 314021:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4620.067132] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4621.710534] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4622.828263] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4624.964758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4624.993772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4625.064071] Lustre: Denying initial registration attempt from nid 192.168.203.124@tcp, specified as failover [ 4625.067896] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4625.072573] LustreError: 315626:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -99 [ 4625.075985] LustreError: 315626:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 4625.078704] LustreError: 315626:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4625.124409] Lustre: server umount lustre-OST0000 complete [ 4625.125968] LustreError: 315626:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4629.474922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4629.475107] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4629.475644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4629.479046] Lustre: Skipped 1 previous similar message [ 4633.403057] LustreError: 315994:0:(obd_class.h:479:obd_check_dev()) Device 15 not setup [ 4633.404683] LustreError: 315994:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 4633.426399] Lustre: server umount lustre-MDT0000 complete [ 4634.593256] LustreError: 314032:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4634.598457] LustreError: 314032:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 4634.629115] LustreError: 314014:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013806 with bad export cookie 11359276081063740360 [ 4634.631569] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4634.634401] LustreError: 314014:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4634.777148] Lustre: server umount lustre-MDT0001 complete [ 4639.251106] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4640.308995] Key type lgssc unregistered [ 4640.443496] LNet: 316724:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4641.510271] LNet: Removed LNI 192.168.203.124@tcp [ 4641.811957] Key type .llcrypt unregistered [ 4641.813060] Key type ._llcrypt unregistered [ 4645.013601] Key type ._llcrypt registered [ 4645.014436] Key type .llcrypt registered [ 4645.052783] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4645.375076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4645.422675] alg: No test for adler32 (adler32-zlib) [ 4646.280632] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 4646.364866] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4646.367304] LNet: Accept secure, port 988 [ 4647.952269] Key type lgssc registered [ 4648.309615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4650.604593] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4651.835814] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4653.102440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4654.403146] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4654.774124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4661.049402] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4664.265044] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4664.271116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.378140] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4665.386896] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4665.454518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4666.826820] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4669.699988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4669.729638] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4669.740149] Lustre: 319433:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4669.844167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4671.147749] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4672.852398] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4673.971307] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4676.184613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4676.212238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4676.279620] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4676.347568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4678.224833] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4679.920165] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4683.236870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4683.238247] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4683.239752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4683.243899] Lustre: Skipped 1 previous similar message [ 4683.246510] Lustre: Skipped 1 previous similar message [ 4686.700080] LustreError: 321694:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4686.773470] Lustre: server umount lustre-OST0000 complete [ 4688.354278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4688.355082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4688.357671] Lustre: Skipped 1 previous similar message [ 4693.473415] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4693.475152] Lustre: Skipped 2 previous similar messages [ 4698.592952] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4698.594824] Lustre: Skipped 1 previous similar message [ 4702.176118] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4702.220074] LustreError: 321897:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4702.223306] LustreError: 321897:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4702.253915] Lustre: server umount lustre-MDT0000 complete [ 4703.490120] LustreError: 319424:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013875 with bad export cookie 14532969768486871954 [ 4703.493733] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4703.493977] LustreError: 319424:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4703.556761] LustreError: 322101:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4703.559619] LustreError: 322101:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4703.669774] Lustre: server umount lustre-MDT0001 complete [ 4707.596513] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4708.815153] Key type lgssc unregistered [ 4708.975449] LNet: 322627:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4709.990388] LNet: Removed LNI 192.168.203.124@tcp [ 4710.340964] Key type .llcrypt unregistered [ 4710.341985] Key type ._llcrypt unregistered [ 4719.733344] Key type ._llcrypt registered [ 4719.734355] Key type .llcrypt registered [ 4719.785862] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4720.386184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4720.401959] alg: No test for adler32 (adler32-zlib) [ 4721.288218] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 4721.385807] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4721.387520] LNet: Accept secure, port 988 [ 4722.976192] Key type lgssc registered [ 4723.384902] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4726.245799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4726.255151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4727.422576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4728.835771] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4731.795812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4731.933782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4733.443971] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4735.269882] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4736.517629] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4738.856880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4738.973406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.846639] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4742.640384] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4757.474752] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4757.476402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4757.478154] Lustre: Skipped 1 previous similar message [ 4757.482310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4758.383891] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 4758.385723] Lustre: Skipped 2 previous similar messages [ 4761.470047] LustreError: 326241:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4761.494817] Lustre: server umount lustre-MDT0000 complete [ 4762.595683] LustreError: 325283:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.605684] LustreError: 325283:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 4762.733310] LustreError: 323973:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013934 with bad export cookie 9281813021593334936 [ 4762.735947] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4762.739468] LustreError: 323973:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4763.492658] LustreError: 323986:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.497630] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 4767.712840] LustreError: 323988:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.713176] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4767.722722] Lustre: Skipped 1 previous similar message [ 4767.724584] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4768.893098] LustreError: 326442:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4768.894948] LustreError: 326442:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4768.971462] Lustre: server umount lustre-MDT0001 complete [ 4771.043229] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4773.771295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4788.192898] LustreError: 323305:0:(client.c:1385:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000096eb8168 x1839216871890304/t0(0) o250->MGC192.168.203.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4788.319252] LustreError: 325587:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4788.326382] LustreError: 325587:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 4788.346166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4788.370416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 4789.616667] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4792.402155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4792.536952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4793.789705] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4795.590257] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4797.668695] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 4799.261928] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 4800.462062] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4801.051752] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4803.044056] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4803.044171] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4803.048924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4803.052279] Lustre: Skipped 2 previous similar messages [ 4808.369096] LustreError: 328735:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4808.371460] LustreError: 328735:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4808.434114] Lustre: server umount lustre-OST0000 complete [ 4813.281655] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4813.281702] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4813.288179] Lustre: Skipped 1 previous similar message [ 4813.293090] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4813.295554] Lustre: Skipped 3 previous similar messages [ 4815.808259] LustreError: 328937:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4815.810343] LustreError: 328937:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4815.837242] Lustre: server umount lustre-MDT0000 complete [ 4817.095398] LustreError: 323973:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754013988 with bad export cookie 9281813021593342090 [ 4817.098297] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4817.099607] LustreError: 323973:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4817.240770] Lustre: server umount lustre-MDT0001 complete [ 4822.087100] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4823.257570] Key type lgssc unregistered [ 4823.402394] LNet: 329666:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4824.422313] LNet: Removed LNI 192.168.203.124@tcp [ 4824.754406] Key type .llcrypt unregistered [ 4824.755910] Key type ._llcrypt unregistered [ 4834.048370] Key type ._llcrypt registered [ 4834.049797] Key type .llcrypt registered [ 4834.088289] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4834.483308] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4834.518814] alg: No test for adler32 (adler32-zlib) [ 4835.374330] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 4835.457296] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4835.458807] LNet: Accept secure, port 988 [ 4837.040125] Key type lgssc registered [ 4837.395453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4840.120239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4840.129246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.291621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4842.634613] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4845.605645] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4845.727091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4847.016542] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4848.792431] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4850.029460] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4851.088829] LustreError: 331057:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4851.094085] LustreError: 331057:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4851.098511] LustreError: 331057:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 0, failed: rc = -5 [ 4851.169596] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4851.169852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4851.170230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4851.174952] Lustre: Skipped 1 previous similar message [ 4855.265300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4855.267283] Lustre: Skipped 2 previous similar messages [ 4857.211877] LustreError: 332404:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 4857.240508] Lustre: server umount lustre-MDT0000 complete [ 4858.462256] LustreError: 331010:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014030 with bad export cookie 17600528793229329899 [ 4858.466254] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4858.466355] LustreError: 331010:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4858.530551] LustreError: 332610:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 4858.532381] LustreError: 332610:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4858.608172] Lustre: server umount lustre-MDT0001 complete [ 4863.602224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.709432] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4863.722350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4863.811327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4865.258869] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4868.221073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4868.258318] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4868.270971] Lustre: 333379:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4868.382915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4869.739304] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4871.554213] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4872.913913] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4875.574177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4875.651251] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4875.654543] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4875.659716] LustreError: 335026:0:(tgt_mount.c:2419:server_fill_super()) Unable to start targets: -2 [ 4875.663543] LustreError: 335026:0:(tgt_mount.c:1913:server_put_super()) no obd lustre-OST0000 [ 4875.666579] LustreError: 335026:0:(tgt_mount.c:117:server_deregister_mount()) lustre-OST0000 not registered [ 4875.736195] Lustre: server umount lustre-OST0000 complete [ 4875.737743] LustreError: 335026:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4878.654162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4878.733096] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4878.813607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4880.661758] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 4880.767189] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4882.571970] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4885.241722] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4885.268826] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4885.296875] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4885.329905] Lustre: lustre-OST0001: new disk, initializing [ 4885.332738] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4886.598573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4886.602597] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4886.635946] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4887.286343] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4889.182091] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4891.619981] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4891.625908] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4891.625982] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4891.630725] Lustre: Skipped 1 previous similar message [ 4892.271100] LustreError: 337132:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 4892.272858] LustreError: 337132:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 4892.292475] Lustre: server umount lustre-OST0001 complete [ 4896.225131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4896.225311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.228123] LustreError: Skipped 1 previous similar message [ 4896.232563] Lustre: Skipped 1 previous similar message [ 4896.233138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4896.236659] Lustre: Skipped 1 previous similar message [ 4896.739258] LustreError: 335491:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4896.745070] LustreError: 335491:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 4899.628089] LustreError: 337332:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4899.630054] LustreError: 337332:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4899.695903] Lustre: server umount lustre-OST0000 complete [ 4901.859637] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4901.860577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4901.863655] Lustre: Skipped 2 previous similar messages [ 4901.867257] Lustre: Skipped 1 previous similar message [ 4907.138097] LustreError: 337533:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 4907.140011] LustreError: 337533:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4907.167824] Lustre: server umount lustre-MDT0000 complete [ 4908.432548] LustreError: 333372:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014080 with bad export cookie 17600528793229330431 [ 4908.436168] LustreError: 333372:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4908.436180] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4908.598269] Lustre: server umount lustre-MDT0001 complete [ 4912.310886] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 4913.438500] Key type lgssc unregistered [ 4913.584511] LNet: 338260:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4914.598341] LNet: Removed LNI 192.168.203.124@tcp [ 4914.929190] Key type .llcrypt unregistered [ 4914.930422] Key type ._llcrypt unregistered [ 4925.999270] Key type ._llcrypt registered [ 4926.001048] Key type .llcrypt registered [ 4926.041637] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 4932.162684] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 4932.483347] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4932.515088] alg: No test for adler32 (adler32-zlib) [ 4933.361616] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 4933.439926] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 4933.441517] LNet: Accept secure, port 988 [ 4935.024129] Key type lgssc registered [ 4935.352434] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4938.325702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4940.475440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4942.405342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4944.477584] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4946.724946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4946.742681] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4946.747442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.832398] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4947.844237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4947.878323] Lustre: lustre-MDT0000: new disk, initializing [ 4947.902790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4947.908308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4949.310165] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4953.496824] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4953.520319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4953.558251] Lustre: 342164:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4953.583367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4953.585091] Lustre: Skipped 1 previous similar message [ 4953.619895] Lustre: lustre-MDT0001: new disk, initializing [ 4953.654139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4953.667977] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4953.671944] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4954.930472] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4957.974223] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4959.164886] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4961.488301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4961.520036] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 4961.625154] Lustre: lustre-OST0000: new disk, initializing [ 4961.626813] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4961.652815] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4963.600164] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 4966.706967] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4968.131755] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4971.514518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4971.517944] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4971.550190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4972.317985] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4973.648818] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4973.725980] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4976.611100] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4976.613358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.615260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4976.617876] Lustre: Skipped 1 previous similar message [ 4980.462753] LustreError: 345098:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 4980.534452] Lustre: server umount lustre-OST0000 complete [ 4984.288875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4984.290932] LustreError: Skipped 1 previous similar message [ 4984.292071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4984.295670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4984.297361] Lustre: Skipped 1 previous similar message [ 4986.849597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4986.850601] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4987.966256] LustreError: 345300:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 4987.968976] LustreError: 345300:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 4987.998859] Lustre: server umount lustre-MDT0000 complete [ 4989.288327] LustreError: 342939:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014161 with bad export cookie 7178808056795881095 [ 4989.290447] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4989.294488] LustreError: 342939:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4989.359960] LustreError: 345502:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 4989.362592] LustreError: 345502:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 4989.446492] Lustre: server umount lustre-MDT0001 complete [ 4992.267596] Lustre: DEBUG MARKER: == conf-sanity test 60a: check mkfs.lustre --mkfsoptions -E -O options setting ========================================================== 22:09:23 (1754014163) [ 4994.243047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 4996.307215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5003.743088] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 5006.904310] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5010.496177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5012.807771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5014.956786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5017.156245] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5019.477809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5019.503580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5019.590053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5019.602700] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5019.639596] Lustre: lustre-MDT0000: new disk, initializing [ 5019.664570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5019.671362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5020.968778] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5025.185060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5025.208083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5025.245471] Lustre: 349257:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5025.275686] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5025.278848] Lustre: Skipped 1 previous similar message [ 5025.320264] Lustre: lustre-MDT0001: new disk, initializing [ 5025.355615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5025.367281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5025.370570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5026.725395] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5029.968676] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5031.203779] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5033.489905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5033.514692] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5033.615734] Lustre: lustre-OST0000: new disk, initializing [ 5033.617441] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5033.644468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5035.260605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5035.264089] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5035.300198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5035.618713] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5038.817019] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5040.252911] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5040.344125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5041.748922] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5041.828826] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5045.733681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5045.734161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5045.737238] LustreError: Skipped 1 previous similar message [ 5045.737906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5045.740328] Lustre: Skipped 1 previous similar message [ 5045.746779] Lustre: Skipped 2 previous similar messages [ 5048.558582] LustreError: 352165:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5048.560764] LustreError: 352165:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5048.622399] Lustre: server umount lustre-OST0000 complete [ 5050.849244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5050.849810] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5050.850406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5050.857632] Lustre: Skipped 1 previous similar message [ 5061.089638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5061.091499] Lustre: Skipped 4 previous similar messages [ 5064.160168] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5064.197778] LustreError: 352366:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5064.199496] LustreError: 352366:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5064.228317] Lustre: server umount lustre-MDT0000 complete [ 5065.481370] LustreError: 349248:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014237 with bad export cookie 7178808056795881963 [ 5065.486171] LustreError: 349248:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5065.486532] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5065.640850] Lustre: server umount lustre-MDT0001 complete [ 5068.443900] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:10:39 (1754014239) [ 5070.879227] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:10:42 (1754014242) [ 5073.635038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.771531] LustreError: 353581:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5073.776044] LustreError: 353581:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5073.793349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5075.047149] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5077.765778] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5079.246221] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5080.941944] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5082.086649] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5084.328177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5086.205376] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5087.908124] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5106.806579] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5107.350748] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5107.911826] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5108.487579] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5109.150361] Lustre: Failing over lustre-MDT0000 [ 5109.218759] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5109.219427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5109.223280] Lustre: Skipped 1 previous similar message [ 5109.290821] LustreError: 356026:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5109.294291] LustreError: 356026:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5109.319954] Lustre: server umount lustre-MDT0000 complete [ 5113.705799] LustreError: 355722:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.820665] LustreError: 353575:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5118.827208] LustreError: 353575:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5122.103814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5122.140951] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5122.221574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5122.223622] Lustre: Skipped 2 previous similar messages [ 5122.245107] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5123.512635] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5123.940900] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5127.654335] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5127.663249] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 5127.680631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5128.861391] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5129.475300] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5131.314752] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5131.962940] Lustre: Failing over lustre-MDT0000 [ 5132.144992] LustreError: 357098:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5132.146563] LustreError: 357098:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5132.177861] Lustre: server umount lustre-MDT0000 complete [ 5132.770508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5132.771242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5132.771694] LustreError: 353577:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5132.771701] LustreError: 353577:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5132.776197] Lustre: Skipped 2 previous similar messages [ 5137.889263] LustreError: 353599:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5137.896955] LustreError: 353599:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages [ 5144.963473] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5144.998312] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5145.111411] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5146.388750] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5149.540391] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5150.183069] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) [ 5150.185970] Lustre: Skipped 2 previous similar messages [ 5150.192765] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 5150.210761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5151.342552] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5151.863960] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5155.297426] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5155.300573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5155.302105] Lustre: Skipped 4 previous similar messages [ 5160.279803] Lustre: server umount lustre-MDT0000 complete [ 5160.417189] LustreError: 353581:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5160.423558] LustreError: 353581:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 7 previous similar messages [ 5161.554196] LustreError: 353562:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014333 with bad export cookie 7178808056795884602 [ 5161.556853] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5161.559912] LustreError: 353562:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5161.720331] Lustre: server umount lustre-MDT0001 complete [ 5172.210111] LustreError: 358523:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5172.211879] LustreError: 358523:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages [ 5172.282552] Lustre: server umount lustre-OST0000 complete [ 5177.228984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5177.373494] LustreError: 359350:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5178.640111] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5181.282398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5182.654896] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5184.347196] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5185.553631] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5187.875098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5187.979433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5187.982287] Lustre: Skipped 3 previous similar messages [ 5189.875380] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5191.582171] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5202.343301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5212.642244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5212.642768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5212.644849] LustreError: Skipped 1 previous similar message [ 5212.649204] Lustre: Skipped 5 previous similar messages [ 5214.699329] Lustre: server umount lustre-OST0000 complete [ 5222.044764] Lustre: server umount lustre-MDT0000 complete [ 5222.881072] LustreError: 359350:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5222.886204] LustreError: 359350:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5223.297160] LustreError: 359330:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014395 with bad export cookie 7178808056795885330 [ 5223.299971] LustreError: 359330:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5223.300309] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5223.437677] Lustre: server umount lustre-MDT0001 complete [ 5228.301181] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5229.408899] Key type lgssc unregistered [ 5229.547482] LNet: 362525:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5230.566348] LNet: Removed LNI 192.168.203.124@tcp [ 5230.879577] Key type .llcrypt unregistered [ 5230.881464] Key type ._llcrypt unregistered [ 5240.520017] Key type ._llcrypt registered [ 5240.520997] Key type .llcrypt registered [ 5240.558390] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5240.851779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5240.876573] alg: No test for adler32 (adler32-zlib) [ 5241.728412] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 5241.804821] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5241.807101] LNet: Accept secure, port 988 [ 5243.384118] Key type lgssc registered [ 5243.708694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5246.302622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5246.310952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5247.458468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5248.702388] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5251.424707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5251.556734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5252.849180] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5254.638531] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5255.815146] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5258.029081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5258.145740] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5259.908445] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5261.591830] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5272.488455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5282.272907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5282.275048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.278821] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5282.786800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5282.787737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5282.790290] Lustre: Skipped 1 previous similar message [ 5282.793655] Lustre: Skipped 1 previous similar message [ 5287.036428] LustreError: 366242:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5287.062616] Lustre: server umount lustre-MDT0000 complete [ 5287.905338] LustreError: 363989:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.910099] LustreError: 363989:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 5288.308615] LustreError: 363970:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014460 with bad export cookie 8088687529832616229 [ 5288.312111] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5288.314522] LustreError: 363970:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5288.380695] LustreError: 366444:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 5288.382597] LustreError: 366444:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5288.474001] Lustre: server umount lustre-MDT0001 complete [ 5300.020710] LustreError: 366645:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5300.022606] LustreError: 366645:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5300.099103] Lustre: server umount lustre-OST0000 complete [ 5304.663499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.842967] LustreError: 367374:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.866185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5306.093471] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5308.912992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5309.045561] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5310.461083] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5312.295224] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5313.450506] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5315.741981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5317.724242] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5319.578456] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5330.221561] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5339.617181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5339.618991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5339.622263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5343.291253] LustreError: 369726:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5343.292968] LustreError: 369726:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5343.315352] Lustre: server umount lustre-MDT0000 complete [ 5344.492110] LustreError: 367353:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014516 with bad export cookie 8088687529832617328 [ 5344.494731] LustreError: 367353:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5344.496052] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5344.634973] Lustre: server umount lustre-MDT0001 complete [ 5356.081131] LustreError: 370128:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5356.082912] LustreError: 370128:0:(obd_class.h:479:obd_check_dev()) Skipped 11 previous similar messages [ 5356.149464] Lustre: server umount lustre-OST0000 complete [ 5361.192668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.337309] LustreError: 370955:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5361.360789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5361.362537] Lustre: Skipped 1 previous similar message [ 5362.636515] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5365.542339] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5367.068391] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5368.886703] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5370.136749] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5372.492527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5374.666412] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5376.413088] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5386.980283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5391.329783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5391.332266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5391.335640] Lustre: Skipped 2 previous similar messages [ 5391.337496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5391.340392] Lustre: Skipped 3 previous similar messages [ 5394.349095] LustreError: 373206:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5394.351257] LustreError: 373206:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5394.410591] Lustre: server umount lustre-OST0000 complete [ 5396.449094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5396.451475] LustreError: Skipped 1 previous similar message [ 5396.452743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5396.457081] Lustre: Skipped 1 previous similar message [ 5396.459903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5396.461409] Lustre: Skipped 2 previous similar messages [ 5401.750034] Lustre: server umount lustre-MDT0000 complete [ 5402.592963] LustreError: 370954:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5402.600860] LustreError: 370954:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5403.035159] LustreError: 370934:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014574 with bad export cookie 8088687529832618336 [ 5403.038623] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5403.039809] LustreError: 370934:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5403.181918] Lustre: server umount lustre-MDT0001 complete [ 5408.203149] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5409.275136] Key type lgssc unregistered [ 5409.404345] LNet: 374136:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5410.470304] LNet: Removed LNI 192.168.203.124@tcp [ 5410.789264] Key type .llcrypt unregistered [ 5410.790905] Key type ._llcrypt unregistered [ 5420.036685] Key type ._llcrypt registered [ 5420.037563] Key type .llcrypt registered [ 5420.073192] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5420.472942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5420.478727] alg: No test for adler32 (adler32-zlib) [ 5421.322082] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 5421.396449] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5421.397869] LNet: Accept secure, port 988 [ 5422.968167] Key type lgssc registered [ 5423.316332] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5425.722498] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5425.727840] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.732127] LustreError: 375523:0:(osd_handler.c:8372:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5425.743190] LustreError: 375523:0:(obd_config.c:779:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5425.746803] LustreError: 375523:0:(obd_mount.c:193:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5425.750469] LustreError: 375523:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5425.754637] LustreError: 375523:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5428.032147] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5428.035324] LustreError: 375922:0:(osd_handler.c:8372:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5428.043831] LustreError: 375922:0:(obd_config.c:779:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5428.046592] LustreError: 375922:0:(obd_mount.c:193:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5428.049421] LustreError: 375922:0:(tgt_mount.c:2387:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5428.052643] LustreError: 375922:0:(super25.c:181:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5434.595250] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5435.758992] Key type lgssc unregistered [ 5435.900465] LNet: 376901:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5435.909409] LNet: Removed LNI 192.168.203.124@tcp [ 5436.241419] Key type .llcrypt unregistered [ 5436.242389] Key type ._llcrypt unregistered [ 5442.049243] Key type ._llcrypt registered [ 5442.050122] Key type .llcrypt registered [ 5442.085348] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 5448.232531] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5448.601964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5448.639864] alg: No test for adler32 (adler32-zlib) [ 5449.498558] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 5449.577940] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5449.579309] LNet: Accept secure, port 988 [ 5451.152120] Key type lgssc registered [ 5451.499770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5454.370505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5456.498727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5458.496105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5460.607735] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5462.931021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5462.959246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5462.967081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5464.074463] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5464.087518] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5464.140599] Lustre: lustre-MDT0000: new disk, initializing [ 5464.172172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5464.179691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5465.490565] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5469.591126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5469.615145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5469.656952] Lustre: 379988:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5469.681539] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5469.684517] Lustre: Skipped 1 previous similar message [ 5469.734304] Lustre: lustre-MDT0001: new disk, initializing [ 5469.772428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5469.787449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5469.790647] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5471.130823] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5474.220304] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5475.429431] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5477.651391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5477.682152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5477.795414] Lustre: lustre-OST0000: new disk, initializing [ 5477.798118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5477.830399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5479.639223] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5481.338477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5481.341317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5481.374992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5482.643907] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5483.954776] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5484.026672] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5485.197152] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5485.262606] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5486.562972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5486.563476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5486.564582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5486.565260] LustreError: Skipped 1 previous similar message [ 5486.570267] Lustre: Skipped 1 previous similar message [ 5487.919406] LustreError: 382893:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5487.977204] Lustre: server umount lustre-OST0000 complete [ 5490.144902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5490.146799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5490.150086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5490.152843] Lustre: Skipped 1 previous similar message [ 5491.681270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5491.681931] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5495.226834] LustreError: 383094:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5495.228524] LustreError: 383094:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5495.251623] Lustre: server umount lustre-MDT0000 complete [ 5496.480178] LustreError: 379980:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014668 with bad export cookie 13064024530365719948 [ 5496.482900] LustreError: 379980:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5496.483226] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5496.553716] LustreError: 383295:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5496.556395] LustreError: 383295:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5496.634870] Lustre: server umount lustre-MDT0001 complete [ 5499.407571] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5499.952879] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:17:51 (1754014671) [ 5502.103935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.285115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5503.583906] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5506.480566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5506.597615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5507.915124] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5509.715185] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5510.952920] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5513.177053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5515.072201] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5516.811311] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5518.892825] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5518.913556] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5518.966315] Lustre: lustre-OST0001: new disk, initializing [ 5518.968153] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5518.998533] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5519.000592] Lustre: Skipped 1 previous similar message [ 5520.449548] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5520.452471] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5520.487259] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5520.745349] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5523.680287] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5525.984770] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5525.987099] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.990733] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5525.992211] Lustre: Skipped 1 previous similar message [ 5531.565103] LustreError: 387186:0:(obd_class.h:479:obd_check_dev()) Device 25 not setup [ 5531.567143] LustreError: 387186:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5531.584291] Lustre: server umount lustre-OST0001 complete [ 5533.157116] LustreError: 385599:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5533.164920] LustreError: 385599:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 5538.278049] LustreError: 385597:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5538.282649] LustreError: 385597:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5543.396483] LustreError: 386099:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5543.403802] LustreError: 386099:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5547.489503] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5547.491623] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5547.494439] Lustre: Skipped 1 previous similar message [ 5547.495970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5547.498958] Lustre: Skipped 3 previous similar messages [ 5548.516570] LustreError: 385597:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5548.520525] LustreError: 385597:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 5549.615085] LustreError: 387410:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5549.616841] LustreError: 387410:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5549.677557] Lustre: server umount lustre-OST0000 complete [ 5552.609408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5552.614793] LustreError: Skipped 1 previous similar message [ 5558.753308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5558.756546] Lustre: Skipped 6 previous similar messages [ 5565.408132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5565.440577] LustreError: 387611:0:(obd_class.h:479:obd_check_dev()) Device 28 not setup [ 5565.443431] LustreError: 387611:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5565.474871] Lustre: server umount lustre-MDT0000 complete [ 5566.638248] LustreError: 383992:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014738 with bad export cookie 13064024530365720816 [ 5566.643568] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5572.808354] Lustre: server umount lustre-MDT0001 complete [ 5577.441409] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5578.427810] Key type lgssc unregistered [ 5578.557324] LNet: 388340:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5579.622297] LNet: Removed LNI 192.168.203.124@tcp [ 5579.918908] Key type .llcrypt unregistered [ 5579.920478] Key type ._llcrypt unregistered [ 5592.214336] Key type ._llcrypt registered [ 5592.215327] Key type .llcrypt registered [ 5592.260009] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 5597.993767] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5598.413266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5598.425233] alg: No test for adler32 (adler32-zlib) [ 5599.275555] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 5599.355751] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5599.357452] LNet: Accept secure, port 988 [ 5600.944170] Key type lgssc registered [ 5601.294546] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5604.265715] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5606.768528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5608.997421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5611.181348] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5613.716276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5613.740399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5613.745663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.831670] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5614.841964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5614.877845] Lustre: lustre-MDT0000: new disk, initializing [ 5614.903064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5614.909550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5616.179360] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5620.369835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5620.391446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5620.420339] Lustre: 392244:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5620.447215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5620.448645] Lustre: Skipped 1 previous similar message [ 5620.486678] Lustre: lustre-MDT0001: new disk, initializing [ 5620.522187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5620.533938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5620.538415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5621.803522] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5625.267686] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5626.467346] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5629.097339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5629.122455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5629.238557] Lustre: lustre-OST0000: new disk, initializing [ 5629.240637] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5629.275876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5631.500202] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5634.059058] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5634.064566] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5634.118591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5635.167693] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5636.448356] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5636.511333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5637.703486] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5637.772343] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5639.137520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5639.139402] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5639.140703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5639.142225] Lustre: Skipped 1 previous similar message [ 5640.428093] LustreError: 395152:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5640.485262] Lustre: server umount lustre-OST0000 complete [ 5644.257729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5644.258239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5644.260658] Lustre: Skipped 1 previous similar message [ 5644.263098] Lustre: Skipped 2 previous similar messages [ 5647.742870] LustreError: 395353:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5647.745288] LustreError: 395353:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5647.770603] Lustre: server umount lustre-MDT0000 complete [ 5648.919174] LustreError: 393979:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014820 with bad export cookie 12616294019979953381 [ 5648.923028] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5648.924066] LustreError: 393979:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5648.981204] LustreError: 395557:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5648.982868] LustreError: 395557:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5649.056473] Lustre: server umount lustre-MDT0001 complete [ 5651.620494] Lustre: DEBUG MARKER: == conf-sanity test 65: re-create the lost last_rcvd file when server mount ========================================================== 22:20:23 (1754014823) [ 5654.510181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5657.328591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5657.471821] Lustre: lustre-MDT0000: new disk, initializing [ 5657.503684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5658.790188] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5661.570910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5661.695398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5663.068209] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5664.902582] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5666.110066] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5671.906492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.906658] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5671.907810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5671.913066] Lustre: Skipped 1 previous similar message [ 5671.915513] LustreError: Skipped 1 previous similar message [ 5678.049189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5678.051012] Lustre: Skipped 4 previous similar messages [ 5681.120193] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5681.171285] LustreError: 398151:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5681.174863] LustreError: 398151:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5681.209784] Lustre: server umount lustre-MDT0000 complete [ 5682.597366] LustreError: 396802:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014854 with bad export cookie 12616294019979954249 [ 5682.601910] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5682.747948] Lustre: server umount lustre-MDT0001 complete [ 5685.852285] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:20:57 (1754014857) [ 5688.081742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5688.227632] LustreError: 399074:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5688.234488] LustreError: 399074:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 5688.248207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.528858] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5692.521988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5694.015532] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5695.850908] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5697.225111] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5699.523281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5701.508527] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5703.555544] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5716.863298] Lustre: Permanently deactivating lustre-OST0000 [ 5716.868232] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5725.165404] Lustre: lustre-OST0000_UUID: llog_process_th(pid=401777/ppid=2) setting import INACTIVE by administrator request [ 5726.403772] LustreError: 401864:0:(mgs_llog.c:1937:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5726.406499] LustreError: 401864:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5726.408269] LustreError: 401864:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5726.972902] LustreError: 401912:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5726.974936] LustreError: 401912:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5733.611116] LustreError: 402009:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5733.613304] LustreError: 402009:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5733.674141] Lustre: server umount lustre-OST0000 complete [ 5734.538286] LustreError: 402161:0:(mgs_llog.c:1991:mgs_replace_nids()) Only MGS is allowed to be started [ 5734.540048] LustreError: 402161:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5738.976889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5738.980850] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5738.986142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5744.616607] LustreError: 402259:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5744.620254] LustreError: 402259:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5744.660519] Lustre: server umount lustre-MDT0000 complete [ 5745.946718] LustreError: 399055:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754014917 with bad export cookie 12616294019979954788 [ 5745.952797] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5745.954207] LustreError: 399055:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5755.433340] Lustre: server umount lustre-MDT0001 complete [ 5758.124697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.324232] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5760.162984] LustreError: 403297:0:(mgs_handler.c:1161:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5762.055090] Lustre: 403636:0:(mgs_llog.c:1698:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.124@tcp [ 5762.629742] LustreError: 403733:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 5762.632787] LustreError: 403733:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 5762.653254] Lustre: server umount MGS complete [ 5766.212054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.354583] LustreError: 404361:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5766.360836] Lustre: lustre-OST0000_UUID: llog_process_th(pid=404373/ppid=2) setting import INACTIVE by administrator request [ 5766.364031] Lustre: Skipped 1 previous similar message [ 5766.384231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5766.387507] Lustre: Skipped 2 previous similar messages [ 5767.603038] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5770.386769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5770.491855] Lustre: lustre-OST0000_UUID: llog_process_th(pid=405061/ppid=2) setting import INACTIVE by administrator request [ 5771.753595] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5773.400811] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5774.525206] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5775.107230] Lustre: Permanently reactivating lustre-OST0000 [ 5775.111452] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5775.113422] Lustre: Skipped 2 previous similar messages [ 5802.976287] LustreError: 405999:0:(osp_dev.c:694:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5805.487925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5805.609226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5805.613049] Lustre: Skipped 1 previous similar message [ 5807.467264] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5809.130932] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5822.945649] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5822.947971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5822.950880] Lustre: Skipped 2 previous similar messages [ 5822.950987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.954786] Lustre: Skipped 5 previous similar messages [ 5827.057096] LustreError: 407819:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5827.059030] LustreError: 407819:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 5827.123234] Lustre: server umount lustre-OST0000 complete [ 5832.160755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5832.163623] LustreError: Skipped 1 previous similar message [ 5832.165302] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5834.453890] Lustre: server umount lustre-MDT0000 complete [ 5835.630079] LustreError: 404341:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754015007 with bad export cookie 12616294019979955726 [ 5835.633509] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5835.635398] LustreError: 404341:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5835.764063] Lustre: server umount lustre-MDT0001 complete [ 5840.169591] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5841.134693] Key type lgssc unregistered [ 5841.261296] LNet: 408752:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5842.278282] LNet: Removed LNI 192.168.203.124@tcp [ 5842.572976] Key type .llcrypt unregistered [ 5842.574127] Key type ._llcrypt unregistered [ 5848.206352] Key type ._llcrypt registered [ 5848.207288] Key type .llcrypt registered [ 5848.242228] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 5853.823474] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5854.103764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5854.122115] alg: No test for adler32 (adler32-zlib) [ 5854.962220] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 5855.036378] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5855.037968] LNet: Accept secure, port 988 [ 5856.616134] Key type lgssc registered [ 5856.934042] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5859.641200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5861.660439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5863.595842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5865.832721] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5868.036536] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5868.058145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5868.063317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5869.143511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5869.154291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5869.197865] Lustre: lustre-MDT0000: new disk, initializing [ 5869.224934] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5869.231604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5870.480776] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5874.473615] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5874.493858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5874.521902] Lustre: 411839:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5874.541310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5874.542743] Lustre: Skipped 1 previous similar message [ 5874.577410] Lustre: lustre-MDT0001: new disk, initializing [ 5874.607499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5874.618498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5874.621033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5875.812655] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5878.839811] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5879.939557] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5882.062858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 5882.086836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5882.179272] Lustre: lustre-OST0000: new disk, initializing [ 5882.181456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5882.211908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5883.958039] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5884.349691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5884.353655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5884.395206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5886.914780] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5888.208965] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5888.287230] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5889.497467] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5889.573670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5894.627460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.627676] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5894.628416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5894.633546] Lustre: Skipped 1 previous similar message [ 5894.635985] LustreError: Skipped 1 previous similar message [ 5896.236092] LustreError: 414744:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 5896.291484] Lustre: server umount lustre-OST0000 complete [ 5899.745759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5899.746445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5899.750110] Lustre: Skipped 1 previous similar message [ 5899.752580] Lustre: Skipped 2 previous similar messages [ 5903.547964] LustreError: 414945:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 5903.550467] LustreError: 414945:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5903.575128] Lustre: server umount lustre-MDT0000 complete [ 5904.715480] LustreError: 412614:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754015076 with bad export cookie 1408574124960322058 [ 5904.717518] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5904.719168] LustreError: 412614:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5904.780825] LustreError: 415146:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5904.782432] LustreError: 415146:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 5904.863254] Lustre: server umount lustre-MDT0001 complete [ 5907.532779] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5908.131283] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:24:39 (1754015079) [ 5910.356944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5910.526776] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.749456] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5914.544582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5914.667841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5916.014608] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5917.755698] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5918.945996] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5921.159570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5923.145512] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5924.937086] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5925.570886] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5936.615589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5939.681500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5939.685861] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5939.693354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5940.709337] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5944.364172] LustreError: 418304:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 5944.365878] LustreError: 418304:0:(obd_class.h:479:obd_check_dev()) Skipped 7 previous similar messages [ 5944.426121] Lustre: server umount lustre-OST0000 complete [ 5945.825118] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5945.828881] Lustre: Skipped 1 previous similar message [ 5945.831414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5945.833142] Lustre: Skipped 2 previous similar messages [ 5950.433881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5950.440865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.452414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5951.762578] LustreError: 418509:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 5951.768907] LustreError: 418509:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 5951.804625] Lustre: server umount lustre-MDT0000 complete [ 5953.260960] LustreError: 415845:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754015125 with bad export cookie 1408574124960322926 [ 5953.263397] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5953.263672] LustreError: 415845:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5953.425791] Lustre: server umount lustre-MDT0001 complete [ 5958.379319] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5959.575293] Key type lgssc unregistered [ 5959.713416] LNet: 419236:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5960.742321] LNet: Removed LNI 192.168.203.124@tcp [ 5961.018545] Key type .llcrypt unregistered [ 5961.019421] Key type ._llcrypt unregistered [ 5968.311955] Key type ._llcrypt registered [ 5968.312831] Key type .llcrypt registered [ 5968.347897] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 5968.452210] Key type .llcrypt unregistered [ 5968.453154] Key type ._llcrypt unregistered [ 5975.214123] Key type ._llcrypt registered [ 5975.214965] Key type .llcrypt registered [ 5975.246722] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 5975.541310] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5975.560042] alg: No test for adler32 (adler32-zlib) [ 5976.399157] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 5976.482215] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 5976.483762] LNet: Accept secure, port 988 [ 5978.056136] Key type lgssc registered [ 5978.354645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5980.896976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5980.903131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5982.056574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5983.295218] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5986.164432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5986.295960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5988.073793] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 5989.723376] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5991.393917] LustreError: 421367:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5991.398313] LustreError: 421367:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 5991.399744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5991.875396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 5991.996102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5993.257573] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6001.632889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6001.634842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.638117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6002.147626] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6002.148243] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6002.153860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.637080] LustreError: 423404:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6006.691568] Lustre: server umount lustre-OST0000 complete [ 6010.852753] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6010.857924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.866823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6010.870983] Lustre: Skipped 1 previous similar message [ 6014.027157] LustreError: 423605:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6014.029423] LustreError: 423605:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6014.059935] Lustre: server umount lustre-MDT0000 complete [ 6015.675553] LustreError: 422974:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754015187 with bad export cookie 324415313851283443 [ 6015.681775] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6015.684712] LustreError: 422974:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6015.776875] LustreError: 423807:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6015.779789] LustreError: 423807:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6015.862497] Lustre: server umount lustre-MDT0001 complete [ 6020.875308] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6021.954483] Key type lgssc unregistered [ 6022.091709] LNet: 424334:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6023.142557] LNet: Removed LNI 192.168.203.124@tcp [ 6023.473372] Key type .llcrypt unregistered [ 6023.474471] Key type ._llcrypt unregistered [ 6033.326679] Key type ._llcrypt registered [ 6033.327612] Key type .llcrypt registered [ 6033.364404] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6033.749148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6033.761108] alg: No test for adler32 (adler32-zlib) [ 6034.623751] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 6034.711351] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6034.713332] LNet: Accept secure, port 988 [ 6036.304172] Key type lgssc registered [ 6036.653806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6039.457104] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6039.462360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6056.928574] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6062.048410] LustreError: 425669:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.124@tcp: failed processing log, type 4: rc = -110 [ 6092.768161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6094.752208] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6096.576279] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6098.856612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6099.037114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6099.067536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6100.328426] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6103.202480] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6103.328431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6104.644224] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6106.387783] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6107.570407] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6111.715461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.716815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.718962] Lustre: Skipped 1 previous similar message [ 6111.722502] Lustre: Skipped 1 previous similar message [ 6115.950116] LustreError: 427934:0:(obd_class.h:479:obd_check_dev()) Device 4 not setup [ 6116.027028] Lustre: server umount lustre-OST0000 complete [ 6118.880837] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6118.884045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.888917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6121.953341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6121.953910] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.392652] LustreError: 428135:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup [ 6123.395142] LustreError: 428135:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6123.429156] Lustre: server umount lustre-MDT0000 complete [ 6124.661133] LustreError: 425678:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754015296 with bad export cookie 14605987657109661172 [ 6124.664303] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6124.666855] LustreError: 425678:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6124.729579] LustreError: 428337:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6124.731696] LustreError: 428337:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6124.825257] Lustre: server umount lustre-MDT0001 complete [ 6129.519670] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6130.603384] Key type lgssc unregistered [ 6130.730291] LNet: 428863:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6131.750362] LNet: Removed LNI 192.168.203.124@tcp [ 6132.051077] Key type .llcrypt unregistered [ 6132.052074] Key type ._llcrypt unregistered [ 6142.021192] Key type ._llcrypt registered [ 6142.022515] Key type .llcrypt registered [ 6142.059281] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6142.337965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6142.344918] alg: No test for adler32 (adler32-zlib) [ 6143.204075] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 6143.279469] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6143.280892] LNet: Accept secure, port 988 [ 6144.856175] Key type lgssc registered [ 6145.281330] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6148.580006] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6148.592378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6149.825258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6151.413986] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6154.466300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6154.596104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6156.028656] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6158.027804] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6159.224509] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6161.745388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6161.860798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6163.673779] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6165.423993] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6175.212686] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6180.322711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6180.322723] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6180.324454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6180.329264] Lustre: Skipped 1 previous similar message [ 6181.229818] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.24@tcp (stopping) [ 6181.232287] Lustre: Skipped 2 previous similar messages [ 6183.291298] LustreError: 432472:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6183.316713] Lustre: server umount lustre-MDT0000 complete [ 6184.077768] LustreError: 430207:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.24@tcp arrived at 1754015355 with bad export cookie 15070919305654064993 [ 6184.083405] LustreError: 430207:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6184.721184] LustreError: 430207:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754015356 with bad export cookie 15070919305654064370 [ 6184.724677] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6185.441101] LustreError: 430227:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6185.441618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6185.441636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6185.441642] Lustre: Skipped 1 previous similar message [ 6185.442154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6185.445326] LustreError: 430227:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6189.472948] LustreError: 430227:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6189.473279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6189.480882] LustreError: 430227:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6190.831232] LustreError: 432675:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6190.833868] LustreError: 432675:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6190.912371] Lustre: server umount lustre-OST0000 complete [ 6193.341358] LustreError: 433029:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6193.343374] LustreError: 433029:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6193.424114] Lustre: server umount lustre-MDT0001 complete [ 6198.157475] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6199.367290] Key type lgssc unregistered [ 6199.508557] LNet: 433555:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6200.550354] LNet: Removed LNI 192.168.203.124@tcp [ 6200.921519] Key type .llcrypt unregistered [ 6200.923336] Key type ._llcrypt unregistered [ 6211.521934] Key type ._llcrypt registered [ 6211.524608] Key type .llcrypt registered [ 6211.583605] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6211.947385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6211.954251] alg: No test for adler32 (adler32-zlib) [ 6212.828354] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 6212.914983] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6212.916778] LNet: Accept secure, port 988 [ 6214.496295] Key type lgssc registered [ 6214.867274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6217.706601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6217.713972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6218.889073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6220.529477] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6223.483737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6223.631729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6224.946698] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6227.046679] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6228.348148] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6230.818357] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6230.978642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6232.893188] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6235.053753] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6243.314237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6248.419702] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6248.427054] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6248.430418] Lustre: Skipped 1 previous similar message [ 6250.864523] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 6253.540500] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6253.542156] Lustre: Skipped 1 previous similar message [ 6255.973420] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 6261.093050] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 6261.095464] Lustre: Skipped 2 previous similar messages [ 6261.216153] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6261.260559] LustreError: 437168:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup [ 6261.290556] Lustre: server umount lustre-MDT0001 complete [ 6263.777098] LustreError: 434918:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6263.777658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.777671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6263.780900] LustreError: 434918:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages [ 6263.783768] Lustre: Skipped 1 previous similar message [ 6268.898220] LustreError: 434918:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6269.423993] LustreError: 437373:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6269.427056] LustreError: 437373:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6269.504987] Lustre: server umount lustre-OST0000 complete [ 6271.215090] LustreError: 437574:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6271.218585] LustreError: 437574:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6271.311920] Lustre: server umount lustre-MDT0000 complete [ 6278.078528] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6279.199112] Key type lgssc unregistered [ 6279.326483] LNet: 438252:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6280.358534] LNet: Removed LNI 192.168.203.124@tcp [ 6280.798824] Key type .llcrypt unregistered [ 6280.800089] Key type ._llcrypt unregistered [ 6290.108393] Key type ._llcrypt registered [ 6290.109355] Key type .llcrypt registered [ 6290.144634] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 6296.529340] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6296.977909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6296.994151] alg: No test for adler32 (adler32-zlib) [ 6297.848982] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 6297.927397] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6297.929512] LNet: Accept secure, port 988 [ 6299.512179] Key type lgssc registered [ 6299.850409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6303.721264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6306.640874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6309.160133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6311.396273] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6314.766317] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6318.123253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6320.551313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6320.574348] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6320.578900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6321.658322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6321.668373] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6321.703977] Lustre: lustre-MDT0000: new disk, initializing [ 6321.728875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6321.734665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6322.887790] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6326.554941] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6329.543570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6329.581395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6329.728838] Lustre: lustre-OST0000: new disk, initializing [ 6329.732146] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6329.735143] Lustre: Skipped 1 previous similar message [ 6329.767906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6330.309904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6330.314363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6330.327352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6331.918745] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6334.890515] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6338.285352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6341.141251] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6341.169924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6341.211082] Lustre: 442172:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6341.239142] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6341.273928] Lustre: lustre-MDT0001: new disk, initializing [ 6341.318345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6341.334972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6341.339406] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6342.806306] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6346.222864] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6346.295173] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6347.620719] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6347.701824] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6351.843207] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6351.843401] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6351.847873] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6351.852129] Lustre: Skipped 1 previous similar message [ 6354.287752] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.24@tcp (stopping) [ 6355.452476] LustreError: 445562:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6355.475086] Lustre: server umount lustre-MDT0001 complete [ 6362.082469] LustreError: 442179:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6362.088753] LustreError: 442179:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages [ 6367.203075] LustreError: 442179:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6367.207033] LustreError: 442179:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6371.297073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6371.299360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6371.302800] Lustre: Skipped 1 previous similar message [ 6371.304314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6372.322192] LustreError: 444386:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.441681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6377.442020] LustreError: 444386:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.444854] Lustre: Skipped 1 previous similar message [ 6382.048146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6382.068124] LustreError: 445836:0:(obd_class.h:479:obd_check_dev()) Device 12 not setup [ 6382.071198] LustreError: 445836:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6382.145952] Lustre: server umount lustre-OST0000 complete [ 6382.563328] LustreError: 444386:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6389.822782] LustreError: 446038:0:(obd_class.h:479:obd_check_dev()) Device 13 not setup [ 6389.826011] LustreError: 446038:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6389.913035] Lustre: server umount lustre-MDT0000 complete [ 6395.338318] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6396.417209] Key type lgssc unregistered [ 6396.557316] LNet: 446718:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6397.606367] LNet: Removed LNI 192.168.203.124@tcp [ 6397.900143] Key type .llcrypt unregistered [ 6397.902684] Key type ._llcrypt unregistered [ 6410.777067] Key type ._llcrypt registered [ 6410.778616] Key type .llcrypt registered [ 6410.835376] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6410.999331] Key type .llcrypt unregistered [ 6411.000681] Key type ._llcrypt unregistered [ 6417.387116] Key type ._llcrypt registered [ 6417.387996] Key type .llcrypt registered [ 6417.427889] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6417.726716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6417.757197] alg: No test for adler32 (adler32-zlib) [ 6418.611365] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 6418.693066] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6418.694886] LNet: Accept secure, port 988 [ 6420.280116] Key type lgssc registered [ 6420.650553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6423.295067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6425.324630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6427.759336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6430.371152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6430.406905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6430.417268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6431.519976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6431.532872] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6431.581497] Lustre: lustre-MDT0000: new disk, initializing [ 6431.614912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6431.623461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6433.062802] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6437.176210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6437.199738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6437.248665] Lustre: 450017:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6437.279655] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6437.284038] Lustre: Skipped 1 previous similar message [ 6437.324726] Lustre: lustre-MDT0001: new disk, initializing [ 6437.358024] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6437.371082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6437.376126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6438.732917] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6442.147734] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6443.390828] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6445.677105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6445.699486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6445.802264] Lustre: lustre-OST0000: new disk, initializing [ 6445.805159] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6445.834729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6447.690338] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6450.726029] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6451.207808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6451.211950] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6451.267696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6461.411536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6461.413106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6461.415163] Lustre: Skipped 2 previous similar messages [ 6461.418352] Lustre: Skipped 2 previous similar messages [ 6464.315258] LustreError: 452440:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6464.344525] Lustre: server umount lustre-MDT0000 complete [ 6465.552431] LustreError: 451751:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754015637 with bad export cookie 9278004988916949317 [ 6465.555804] LustreError: 451751:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6465.556806] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6465.625983] LustreError: 452642:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6465.627480] LustreError: 452642:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6465.708031] Lustre: server umount lustre-MDT0001 complete [ 6477.042657] LustreError: 452843:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup [ 6477.044440] LustreError: 452843:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6477.111516] Lustre: server umount lustre-OST0000 complete [ 6480.504745] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 22:34:11 (1754015651) [ 6480.895429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6483.553664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.729660] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6485.264401] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6488.439681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6488.627030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6489.997683] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6491.802902] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6493.066360] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6495.757803] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6495.793121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6495.898828] Lustre: Found index 0 for lustre-OST0000, updating log [ 6498.465667] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6500.618697] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6509.291068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6514.659435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6514.659611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6514.660138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6514.662219] Lustre: Skipped 1 previous similar message [ 6514.666412] LustreError: Skipped 1 previous similar message [ 6515.682304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6515.685242] Lustre: Skipped 2 previous similar messages [ 6518.189083] LustreError: 455930:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6518.190936] LustreError: 455930:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6518.253104] Lustre: server umount lustre-OST0000 complete [ 6519.113514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6520.802833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6520.809436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6520.813136] Lustre: Skipped 1 previous similar message [ 6526.076671] LustreError: 456185:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup [ 6526.078754] LustreError: 456185:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6526.100451] Lustre: server umount lustre-MDT0000 complete [ 6526.945865] LustreError: 453671:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6526.950926] LustreError: 453671:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages [ 6527.223053] LustreError: 453650:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754015699 with bad export cookie 9278004988916950696 [ 6527.226295] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6527.226389] LustreError: 453650:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6527.385291] Lustre: server umount lustre-MDT0001 complete [ 6530.427647] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 22:35:01 (1754015701) [ 6530.812347] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6531.343970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6531.711246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6532.084777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6532.497158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6532.985800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6533.493277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6533.938041] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6534.406279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6534.831682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6535.313973] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6535.809790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6536.178051] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6536.482387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6536.870425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6537.381441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6537.905178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6538.317519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6538.816501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6539.334834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6539.808814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6540.284929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6540.685851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6541.106328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6541.507107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6541.817980] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6542.128216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6542.429025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6542.784969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6543.224305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6543.674172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6544.171593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6544.653365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6545.076579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6545.556305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6545.966072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6546.406476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6546.840270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6547.291563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6547.769079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6548.234649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6548.720065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6549.189223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6549.495370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6549.802928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6550.110493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6550.410208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6550.714126] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6551.035125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6551.483784] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6551.825184] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6554.656119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6554.910943] LustreError: 459771:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6554.942479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6554.944093] Lustre: Skipped 1 previous similar message [ 6556.494649] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6559.494543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6560.972519] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6562.805525] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6564.391532] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6566.831048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6566.875401] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6566.989767] Lustre: Found index 0 for lustre-OST0000, updating log [ 6569.619168] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6571.566622] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6574.563833] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6574.564270] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6574.571369] Lustre: Skipped 1 previous similar message [ 6574.573333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6574.575015] Lustre: Skipped 2 previous similar messages [ 6578.479132] LustreError: 462064:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6578.481692] LustreError: 462064:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages [ 6578.557507] Lustre: server umount lustre-OST0000 complete [ 6580.350603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6583.457323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6583.459008] Lustre: Skipped 6 previous similar messages [ 6586.525096] Lustre: server umount lustre-MDT0000 complete [ 6586.529598] LustreError: 459789:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6586.534604] LustreError: 459789:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message [ 6587.844438] LustreError: 459751:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754015759 with bad export cookie 9278004988916951767 [ 6587.846561] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6587.848661] LustreError: 459751:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6587.996520] Lustre: server umount lustre-MDT0001 complete [ 6588.926811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6595.037120] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 6597.808320] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6601.167960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6603.363788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6605.379883] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6607.457192] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6609.608866] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 22:36:21 (1754015781) [ 6635.257373] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6636.646910] Key type lgssc unregistered [ 6636.783409] LNet: 466028:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6763.750884] LNet: Removed LNI 192.168.203.124@tcp [ 6764.058386] Key type .llcrypt unregistered [ 6764.059430] Key type ._llcrypt unregistered [ 6770.347659] Key type ._llcrypt registered [ 6770.348903] Key type .llcrypt registered [ 6770.387933] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6770.790532] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6770.811838] alg: No test for adler32 (adler32-zlib) [ 6771.662695] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 6771.744454] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6771.745721] LNet: Accept secure, port 988 [ 6773.320146] Key type lgssc registered [ 6773.658311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6775.082727] LNet: Added LNI 192.168.203.124@tcp100 [8/256/0/180] [ 6775.575797] Lustre: DEBUG MARKER: SKIP: conf-sanity test_73c unable to configure net [ 6781.232393] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6782.368988] Key type lgssc unregistered [ 6782.546762] LNet: 467877:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6782.549332] LNet: Removed LNI 192.168.203.124@tcp [ 6782.560364] LNet: Removed LNI 192.168.203.124@tcp100 [ 6782.834484] Key type .llcrypt unregistered [ 6782.835645] Key type ._llcrypt unregistered [ 6790.133664] Key type ._llcrypt registered [ 6790.134546] Key type .llcrypt registered [ 6790.180152] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 6796.409808] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6796.808823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6796.849822] alg: No test for adler32 (adler32-zlib) [ 6797.742820] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 6797.849745] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6797.851224] LNet: Accept secure, port 988 [ 6799.448160] Key type lgssc registered [ 6799.897305] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6803.112097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6805.470096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6807.672367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6809.903939] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6812.157207] Lustre: DEBUG MARKER: == conf-sanity test 74: Test per-device adaptive timeout parameters ========================================================== 22:39:43 (1754015983) [ 6814.520777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6814.543344] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6814.550995] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6815.654623] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6815.666136] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6815.706942] Lustre: lustre-MDT0000: new disk, initializing [ 6815.732929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6815.739269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6817.022667] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6821.121832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6821.141167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6821.180816] Lustre: 471353:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6821.206432] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6821.207910] Lustre: Skipped 1 previous similar message [ 6821.243852] Lustre: lustre-MDT0001: new disk, initializing [ 6821.278104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6821.289361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6821.293770] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6822.574320] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6825.839499] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6827.013904] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6829.138512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6829.163472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6829.258209] Lustre: lustre-OST0000: new disk, initializing [ 6829.259973] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6829.288524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6830.965867] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6832.845764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6832.851137] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6832.901388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6834.405200] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6838.969390] loop0: detected capacity change from 0 to 204800000 [ 6839.031615] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6839.036507] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6839.043386] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6839.046898] blk_update_request: operation not supported error, dev loop0, sector 160240 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6839.067436] blk_update_request: operation not supported error, dev loop0, sector 160344 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6839.088258] blk_update_request: operation not supported error, dev loop0, sector 168552 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6839.105642] blk_update_request: operation not supported error, dev loop0, sector 173280 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6839.131907] blk_update_request: operation not supported error, dev loop0, sector 176912 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6839.169411] blk_update_request: operation not supported error, dev loop0, sector 185112 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6839.204760] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6841.163786] loop0: detected capacity change from 0 to 204800000 [ 6841.284235] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 6841.334985] LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6842.939392] loop0: detected capacity change from 0 to 204800000 [ 6844.700575] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6844.721660] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6844.761493] Lustre: fs15246-OST0000: new disk, initializing [ 6844.763192] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 6844.780173] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6846.495640] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6850.106151] loop1: detected capacity change from 0 to 204800000 [ 6851.820855] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6851.842963] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6851.868754] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 6851.886793] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 6851.921515] Lustre: fs15246-MDT0000: new disk, initializing [ 6851.964565] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6851.973501] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6852.018980] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 6853.024959] Lustre: 475897:0:(client.c:1611:after_reply()) @@@ resending request on EINPROGRESS req@0000000072900e3c x1839219048747392/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1754016040 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 6853.298016] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6861.287945] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 6862.112203] LustreError: 475897:0:(fid_request.c:213:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 6862.114655] LustreError: 475897:0:(fid_request.c:255:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 6862.116921] LustreError: 475897:0:(osp_precreate.c:461:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 6866.405192] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 6867.005918] LustreError: 477011:0:(obd_class.h:479:obd_check_dev()) Device 30 not setup [ 6867.033205] Lustre: server umount fs15246-MDT0000 complete [ 6870.043168] LustreError: 477516:0:(obd_class.h:479:obd_check_dev()) Device 29 not setup [ 6870.044703] LustreError: 477516:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6870.056543] Lustre: server umount fs15246-OST0000 complete [ 6874.081329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6874.081345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6874.083067] LustreError: Skipped 1 previous similar message [ 6874.087376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6874.091015] Lustre: Skipped 1 previous similar message [ 6876.643197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6876.644702] Lustre: Skipped 1 previous similar message [ 6879.659108] LustreError: 478016:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6879.660588] LustreError: 478016:0:(obd_class.h:479:obd_check_dev()) Skipped 1 previous similar message [ 6879.720043] Lustre: server umount lustre-OST0000 complete [ 6881.761854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6881.762637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6881.764730] Lustre: Skipped 2 previous similar messages [ 6886.971443] LustreError: 478218:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6886.973168] LustreError: 478218:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6887.004892] Lustre: server umount lustre-MDT0000 complete [ 6888.134961] LustreError: 472129:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754016059 with bad export cookie 5692451816342315470 [ 6888.137586] LustreError: 472129:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6888.137968] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6888.272252] Lustre: server umount lustre-MDT0001 complete [ 6892.558707] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 6893.581025] Key type lgssc unregistered [ 6893.710345] LNet: 478948:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6894.758328] LNet: Removed LNI 192.168.203.124@tcp [ 6895.055757] Key type .llcrypt unregistered [ 6895.056630] Key type ._llcrypt unregistered [ 6902.325569] Key type ._llcrypt registered [ 6902.326706] Key type .llcrypt registered [ 6902.358390] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6902.601442] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6902.626479] alg: No test for adler32 (adler32-zlib) [ 6903.466583] Lustre: Lustre: Build Version: 2.16.57_19_gb3839b8 [ 6903.544024] LNet: Added LNI 192.168.203.124@tcp [8/256/0/180] [ 6903.545655] LNet: Accept secure, port 988 [ 6905.120115] Key type lgssc registered [ 6905.468102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6907.940468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6909.444155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6911.057227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6912.606059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6917.663884] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 6920.444432] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6923.515414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6925.585375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6927.558608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6929.637663] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6931.873040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6931.894583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6931.901395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6932.978820] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6932.988719] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6933.021955] Lustre: lustre-MDT0000: new disk, initializing [ 6933.042239] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6933.047642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6934.236819] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6939.281075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6939.302618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6939.334141] Lustre: 483324:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6939.357568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6939.359530] Lustre: Skipped 1 previous similar message [ 6939.394655] Lustre: lustre-MDT0001: new disk, initializing [ 6939.428046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6939.440940] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6939.445636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6940.816223] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6944.520870] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6946.075254] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6948.951390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6948.988032] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc [ 6949.100917] Lustre: lustre-OST0000: new disk, initializing [ 6949.104050] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6949.135230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.411090] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_default_debug -1 all [ 6952.443174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6952.446083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6952.485516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6954.492382] Lustre: DEBUG MARKER: oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6955.801261] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6955.876222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6957.084925] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6957.150506] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6962.658768] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6962.659108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6962.663917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6962.664856] LustreError: Skipped 1 previous similar message [ 6962.668332] Lustre: Skipped 1 previous similar message [ 6967.779584] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6967.781133] Lustre: Skipped 1 previous similar message [ 6971.872212] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6971.887198] LustreError: 486229:0:(obd_class.h:479:obd_check_dev()) Device 19 not setup [ 6971.947563] Lustre: server umount lustre-OST0000 complete [ 6975.456801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6975.458593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6975.461407] Lustre: Skipped 1 previous similar message [ 6975.462798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6978.017063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6978.017709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6979.647993] LustreError: 486431:0:(obd_class.h:479:obd_check_dev()) Device 22 not setup [ 6979.649627] LustreError: 486431:0:(obd_class.h:479:obd_check_dev()) Skipped 3 previous similar messages [ 6979.674759] Lustre: server umount lustre-MDT0000 complete [ 6980.995649] LustreError: 484099:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1754016152 with bad export cookie 2412611702320369627 [ 6980.998772] LustreError: MGC192.168.203.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6981.000629] LustreError: 484099:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6981.059447] LustreError: 486632:0:(obd_class.h:479:obd_check_dev()) Device 21 not setup [ 6981.061880] LustreError: 486632:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages [ 6981.138811] Lustre: server umount lustre-MDT0001 complete [ 6988.887777] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing set_hostid [ 6992.152898] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing load_modules_local [ 6995.836305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 6998.508628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7000.481494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7002.637157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro [ 7003.200588] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 6853 sec ========== 22:42:54 (1754016174) [ 7003.789239] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 22:42:55 (1754016175) === [ 7004.897794] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 22:42:56 (1754016176) === [ 7017.523666] Lustre: DEBUG MARKER: oleg324-server.virtnet: executing unload_modules_local [ 7018.650474] Key type lgssc unregistered [ 7018.783479] LNet: 491477:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7019.814405] LNet: Removed LNI 192.168.203.124@tcp [ 7020.144731] Key type .llcrypt unregistered [ 7020.145721] Key type ._llcrypt unregistered